Anti-spill/anti-overflow option for the brush tool

This task proposes a new option for the brush tool that can be found in other softwares: anti-spill or anti-overflow. Its purpose is to limit on the fly the area that the brush can paint on, usually used when coloring line art to avoid painting pass the lines.

Figure 1: Synthetic example of the feature created using a transparency mask

Why to implement this?

As can be seen in Figure 1 one can make use of selections and masks to achieve a similar effect, but having to reselect constantly can get in the way. For example, any closed region of the image that is not filled should be selected on its own to be painted without overflowing. With the brush anti-spill, the allowed region is computed on the fly, so thre's no need of changing tools or making auxiliary masks/selections.

Can't we just use the fill tool to fill the area without overflowing?

Yes, but using the brush tool allows to have more than flat colors or patterns. You can make use of textured or smudging brushes and have richer results.

Possible implementation

The implementation is kind of straightforward:

1. The brush engine must compute the dab as usual.
   
   
   
   Figure 2. The produced dab.

2. We then obtain a (rectangular?) bounding mask for the area that the dab is going to occupy in the canvas.
   
   
   
   Figure 3. The mask created from the bounds of the dab.

3. We then need to obtain a flood-fill selection where:
   • The reference sampling image can be an option in the anti-spill options page, similar to the one in the fill tools ("Current Layer", "All Layers", "Color Labeled Layers").
   • The mask obtained in point 2 is used as boundary selection. This way the flood-fill doesn't extend too much.
   • The seed point, were the flood-fill starts, is the center of the dab.
   • Some postprocessing options can be used, like "grow selection" and "feather".
   • Other options, like filling until an specific color (decribed in the enclose and fill task) can be used, and maybe they are even required here to avoid ending with a bad selection when the pixels under the cursor may not be uniform.
     
     
     
     Figure 4. The flood-fill mask obtained by using the center of the dab (the cross) as seed point, the bounding mask as boundary, and the layer with the black lines as reference.

4. Lastly, we just have to mask the dab with the newly created flood-fill selection.
   
   
   
   Figure 5. The dab masked by the flood-fill mask

Maybe for the smudge engine that mask needs to be used when sampling as well, so that the colors from the other side of the lines don't influence the mix.

Drawbacks

This step would reduce the performance of the brushes even more, although it is optional. I guess the brushes that would use this should be small and be used when painting close to the lines/contours.