Wow! That's some difference!

Code refactor is scheduled for this week.

however I did notice that the original implementation does not need the antialiasing fader as the soft edge value almost never trigger the fader. When it does, the values converted are in the QRect edge.

I run the code removing the fader antialias code entirely and the test is passed succesfully.

That is strange because on sizes below 20px the brush mask generator uses the old rendering scalar engine, so it should be equal to the old version.

I tried to reproduce and found that if "spacing" is left to auto I can sometimes get your results, but only in 16-bit modes, In 8bit it behaves more predictable.

Turning off auto spacing gives consistent behaviour across versions, engines and bitdepths

The images show on the left 8-bit canvases, and on the right 16 bit canvases, both versions shows very similar strokes. Nevertheless I'll look into the conditions to see if everything is triggering as expected, just to be extra sure I don't introduce a regression :]

As an extra note, I did noticed that for 16-bit canvases the old model caused strange renderings for Gaussian (the easiest to test), and the vectorized rendering gives the correct result. Which possibly indicates there is a bug in the 16-bit rendering process. however this should not affect the stroke behavior as the rendering used for small brushes is still the Scalar one.

After some more testing I was able to reproduce it.

Hi Dmitry!

This issue seems to be generalized to all Gauss brushes using the applicator. I tested it on Gauss circular and I get the same differences. Im not sure what is happening as the vector process is not used in that size range. There must be something different a step behind or after, maybe on the processing of the data, but it is something that seems to affect all "KisBrushMaskApplicatorBase" users as the Default brush also behaves in equal ways.

Im going to look for the differences between both paths for Scalar creation routes. It is strange that it changes even if the function valueAt called is the same for calculated values.

Fixed.
rkrita/f77e393ca079ffaa587063a58cfa03cac04fece7

Now I know exactly what shouldSuperSample() does. I removed all supersample() calls from other brushes implementation as it is not needed.
I wonder why it is needed in the Default brush as It basically makes antialias checkbox unable to complely turn off antialias on brushes below 10px size.

The fix in action

Hi, @vanyossi!

Did you also change Circular gaussian mask? It look like it is broken in your branch now :(

Here are the test strokes for 2 and 4 px brushed for Gaussian Circle with 1.0 Fade and Auto Spacing set to 1.0:

master

your branch

At the same time, rectangular tip works perfectly fine in both branches and generates the same results.

PS:
Just a guess: I remember we had two types of fade makers: one for 1d and one for 2d case. And one of them was used for rectangular masks only. Perhaps it might be connected to the bug somehow?