I think the rest of the jagginess is probably a limitation of the 1x resolution normal map. The screenshot is basically saying that in the jaggy areas, there is not enough normal map discontinuity outside the “jags” to warrant inking the pixel.
I played around with the settings in my code last night and surprisingly, it turns out that the two “minor changes” seem to have a larger effect on output quality than the subpixel sampling. The subpixel sampling does help visibly (especially in problematic areas), but the half-pixel shift in the discontinuity detector and the use of all three components of the normal already bring a lot of improvement by themselves. I think I’ll make the number of samples configurable, as it affects performance.
That sounds nice.
My inker is a modified version of the inker in vanilla 1.8.1, which also used normals. Also, using the normals plays well with the existing shader generator infrastructure - the currently available textures are the final pixel colour (which in the general case cannot be used for edge detection), the depth buffer (which I think I tested and found useless for inking), and the normal map.
Ah, thanks! Now I see. Looks like the result could be quite nice.
And it indeed seems that this is nearly impossible to do on the GPU, at least unless there is some clever trick I haven’t thought of.
But this gives me another idea. What about if we instead look for short (e.g. max. 5 pixels) horizontal/vertical runs locally near each pixel? The “run length coordinate” would not be available (as it’s inherently a global piece of information), but it would be possible to smooth selectively based on the number of inked neighbours (and their placement).
This would do something like the subpixel inking already does, but ignoring the normal map and just filtering in the pixels neighbouring the generated lines.
It may either look blurry, or useful - I’ll have to test.
Yes.
State could be saved into a new texture in a multipass approach, but I think the actual computation of the desired information is the problematic part, as it relies on stepping along a varying-length run (which is basically a serial approach). Parallelization tends to cause things like this 
That’s one possibility I hadn’t thought of. Could be nice. However, it’s a lot of re-engineering and I’m not yet that familiar with Panda, so maybe not for 1.9.0 