Maybe some version of this would be worth a try. Would you happen to have any links, or pictures? I’m not sure I understood in which pixels the ink should be applied e.g. along a horizontal run.
The ink is rendered into a separate texture anyway, so it is easy to postprocess it separately.
Implementation-wise, this kind of algorithm sounds a bit tricky for the GPU. As you may know, the fragment shader program is a kernel that will be executed independently for each pixel in the output texture. Any pre-existing data fed in to the shader can be utilized in the computation, but ultimately the fragment shader must decide the output colour of each pixel independently, without knowledge of any other pixels being rendered in the same pass. This makes the computation “embarrassingly parallel”.
Maybe the best approach would after all be some kind of supersampling. Looking at what the inker would like to do, is somehow represent the location of the ink at a resolution higher than the target - i.e. subpixel rendering. From this perspective it is clear why it is difficult to smooth the lines in a postprocessing pass: the subpixel information is not present.
Which gives me an idea. It occurred to me just now that I haven’t yet tried doing supersampling properly, i.e. inside the inking algorithm. Instead of using the center of the pixel as the reference position for the offsets, we could repeat the calculation four times, using shifted positions that are off the center, some way toward each corner. Then we colour the pixel based on how many of these calculations returned that the pixel should be inked (using a hard on/off threshold in the subcalculations).
This is probably a better way to utilize subpixel information than simply doubling the resolution of the ink texture and downsampling. It still relies on linear interpolation of the 1x resolution normal map, so I’m not sure how well this will work. But at least it is worth a try.
(As I see it, the only practical way to get a higher-resolution normal map would be to render the whole scene at 2x resolution - which makes no sense in a realtime application.)