Fix gamut mask rendering

quint8 will overflow if the view size is >255 pixels.
Since all QTransform related calculations use qreal anyway,
change the function arguments to qreal.

Also, Artistic Color Selector didn't update the transform
on resize, and the outer circle of Advanced Color Selector
was 1 pixel too small on odd dimensions due to integer math.

Bug:421142