Cartoon shader improvements

I have to admit I’m not perfectly satisfied with it either.

However, maybe there’s also a component of taste. My eye tends to be drawn to sudden contrasts occurring over a single pixel - the kind of artifact that is produced when drawing lines without any antialiasing. I’m not so much disturbed by slight blur. The original version did not have any antialiasing, so it had the sudden contrast everywhere. The outline-smoothing version removes this artifact in most of the outlines drawn, at the cost of adding a blur to the outlines.

Granted, the jagged edges of the side bangs look horrible (as they did before any changes). I’ll have to re-check the code one more time to figure out what is going on in this particular case.

I did try double-resolution inking (and then downsampling the result) in an earlier version of the patch, but ended up discarding that, as it didn’t look any better. Going higher than double resolution probably makes no sense for a fullscreen texture.

The problem that prevents proper antialiasing is that the inking filter has no concept of a line: it simply colours pixels depending on local information only. It looks for discontinuities in the normal map (to my understanding this is a standard inking technique).

At each pixel, it first gathers the normal data at four points: (-dx, 0), (+dx, 0), (0, -dy) and (0, +dy) relative to the current pixel. Here dx = dy is the offset parameter. The differences between the smallest and largest values of the x and y components of this normal data are then summed and compared to a threshold. This produces a number, which in the original filter was used to switch the ink on or off (for that pixel). In the modified filter, the hard on/off has been changed to a linear interpolation (with saturation) - under the assumption that “less discontinuity” should mean a more translucent line.

The blur used for the outlines is a selective one, which only adds more ink to pixels (never removing existing ink). The intent is to simulate the antialiasing that would be applied while drawing a line, but without knowing there is a line. The drawback is that long horizontal or vertical line segments will bleed. This may contribute to the fuzzy appearance.

So, I don’t have a better solution either - ideas from the community are welcome :slight_smile:

Thanks!

And thanks for your input!