Cartoon shader improvements

Thanks for the info. The current code is now posted as a patch against 1.8.1 in the bug tracker:

bugs.launchpad.net/panda3d/+bug/1221546

Some of my own thoughts on the current state of the patch:

As mentioned in the original post, I’m not completely satisfied with the double resolution inking, so I’m still debating whether to keep that feature or not. Aside from hardly being a clean solution (and not completely getting rid of artifacts), I haven’t yet been able to get the parameters to respond in a visually identical way between the two resolution modes, which would be required to make this option visually independent from the others.

Currently, the blurring (ink line smoothing) of advanced_ink is incompatible with BlurSharpen, because it uses the same blur buffers to render the blur. I have chosen this route, because if I’ve understood correctly, there is a hardware-dependent limit on the number of supported TEXCOORDn parameters. Since the current “main” shader of CommonFilters (the generated one, rendering to finalQuad) already uses up to 5 (0…4), I didn’t want to bump the number used to 7.

A general solution would be to either reuse the current buffers or use separate ones, depending on whether both advanced_ink and BlurSharpen are enabled at the same time. This would enable both to work at the same time if enough buffers are available. But then the logic becomes more complicated, which increases the risk of bugs and makes future maintenance harder.

But wait? Maybe an even more general, and much simpler, solution would be to keep a list of allocated TEXCOORDns, and assign them dynamically to the different parameters of the shader when the full rebuild runs. This would keep their allocation optimal, and would be easy to implement in Python. It seems the only bookkeeping required is to match the TEXCOORDn numbers between the generated vshader and fshader. I could do this if it’s useful?

As for the other stuff in the improvements, I think it is pretty much feature-complete, but of course feedback is welcome.

Usage examples are still missing, as I’ve been testing this using a short test program of my own (related to the project I’m building). For the final version, I could prepare a cleaned-up “advanced basic” tutorial demonstrating how to use the added features.

Finally, I have one further idea that I haven’t tested yet, which I might still add before a final version. I’m thinking that the light/dark boundaries could be smoothed by a postprocessing pass, instead of smoothing them in lighting space like the currently proposed patch does. The lighting space smoothing has its uses however, and would be kept as a separate feature.

(For example, the Toon shader in Blender has separate diffuse/specular controls, and smoothing parameters for each. The current patch pretty much implements that for Panda. Blender’s “size” corresponds to the threshold value (larger size = lower threshold), and “smooth” to the smoothing value.)

For this purpose, I’m thinking of introducing another blur filter that detects pixels which have no normal map discontinuities, and blurs only those. What would happen, theoretically, is that in an area with a constant lighting value (generated by the light ramping), the result is a no-op, while on the light/dark boundaries applying a blur will smooth the boundary.

This would slightly blur the textures, but that is unavoidable in this approach. Objects shouldn’t bleed, because their outlines usually have a normal map discontinuity with whatever is behind them. The only case where this approach obviously fails is when a light/dark boundary is aligned with an edge in the mesh. I’ll need to test this to see how common that is.