Hi, @woltherav!

1. In the old filter, the two horizontal versions are subtracted from on another, same for the two vertical versions.

I guess the explanation is the following:

• the derivative can be both, positive and negative, but in most of the cases, the channels in the paint device can hold only positive values.
• therefore, old filter tried to apply right-to-left and left-to-right kernels and then sum them up somehow.

Speaking truly, I'm not sure if it is mathematically correct, because it normal subtraction of rtl-ltr kernels should always generate empty image...

As far as I can tell, you can skip this two-pass (rtl+ltr) approach and just modify the kernel accordingly using KisConvolutionKernel::offset() and KisConvolutionKernel::factor() somehow. Basically, you need to scale range [-255; 255] range into [0; 255], so the offset should be 128, and factor 0.5.

That will generate a denormalized device for you that can be reused later, when merging X and Y passes.

Then, the horizontal and vertical version have the pythagoras theorem applied to them for mysterious reasons. Should I copy this fully(Right now just using addition)? I would then need to make a seperate blending mode that does a pythagoras equation to the src and dst, would I not? Is this of any use at all?

Well, the necessity of Pythagorean theorem can be explained by the following:

• the derivative is basically a vector, which has length and orientation in space
• so the user might want to have four properties of that derivative (sobel, prewitt are just different methods of calculation of derivative in discrete space):
  • growth of luminosity in X direction (your l-t-r kernel)
  • fall of luminosity in X direction (your r-t-l kernel)
  • growth of luminosity in Y direction (your t-t-b kernel)
  • fall of luminosity in Y direction (your b-t-t kernel)
  • length of the derivative vector (length = sqrt(max(abs(device_after_ltr), abs(device_after_rtl))^2 + max(abs(device_after_ttb), abs(device_after_btt))^2))
  • direction of the vector in radians (see formula here, though I'm not sure the user wants it)
• in most of the cases the user wants the length of the derivative, that is the one calculated with Pythagorean theorem
• if you have two denormalized devices as described above then you can find the length using the formula: length = 2 * sqrt ( (denorm_x - 128)^2 + (denorm_y - 128)^2)

All these calculations can be done in a usual iteration using KisSequentialIterator. Creating a full-features composite op or transformation for this purpose is really an overkill. Though you will have to handle all the different colorspace format yourself.

2. The result is also divided by a certain value. I think I know where this comes from, it's mentioned at the end of the third answer in that stack overflow post. How would I go about applying this division? Make a colortransform? Or just abuse the division blending mode? (Like, fill a device with a color of value 8, apply division blending mode, see what comes out)

You can use KisConvolutionKernel::factor() for this multiplication. Though you should take care about the normalization of the color spaces (quint8, quint16, half and float).

2. I would like to make a variation of this that instead of overwriting the whole device, it makes a grayscale version and writes the result into the alpha of the device. This is much more useful to artists, as it gives a thin outline to their colorpatches. How would I go about writing the grayscale result that is in a seperate paint device into the alpha channel of the main paint device when I only have kispainter and kispaintdevice to my disposal? (And, technically, is there a method for doing this with any of the channels? That'd be necessary for the normal map filter which I'll make later :) )

Check KisNodeManager::slotSplitAlphaWrite() about the example of implementation. I guess, you can even reuse some code from it :)

looking pretty good to me. This code looks quite mathematical, so I can't really speak to that. There is a flicker that is happening when changing settings, going between the original and the applied settings. Not sure if that could be reduced at all as it... That is pretty minor though

I am not sure what the results are supposed to look like, but they appear to make sense with the results and things don't crash.