Correctly populate max color values in server screen format
ClosedPublic

Authored by esjeon on Dec 11 2019, 10:50 AM.

Details

Summary

Compute max color values from color masks, instead of relying on unreliable bits_per_rgb_value provided by xcb_visualtype_t.

In some cases (e.g. nvidia?) bits_per_rgb_value contains wrong value. This results in wrong max color values, and causes weird color translation inside libvncserver. Clients will see a screen which is sorta recognizable but in a complete off-color, making krfb unusable.

This is probably a bug in drivers, but x11vnc does not use this value[1], so I guess it's fair to ignore it in krfb too.

[1]: https://github.com/LibVNC/x11vnc/blob/master/src/screen.c#L3442

Diff Detail

Repository
R437 Desktop Sharing
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
esjeon requested review of this revision.Dec 11 2019, 10:50 AM
esjeon created this revision.
esjeon edited the summary of this revision. (Show Details)
esjeon edited the summary of this revision. (Show Details)Dec 11 2019, 10:52 AM

Sorry for hailing, Alexey, but you're the original author, and it looks like you're the only one who already understands this part.

aacid added a comment.Jan 2 2020, 11:07 PM

Looks reasonable, just out of curiosity could you say which values of redMax/greenMax/blueMax you had before and which you have now?

esjeon added a comment.EditedJan 2 2020, 11:35 PM

bits_per_rgb_value was 11, and the max values were all set to 2047. Total 33 bits per pixel, this is just insane. Now it's all trimmed down to 255.

aacid accepted this revision.Jan 5 2020, 6:14 PM

Let's go for it, at least it doesn't seem to break stuff for me, and hopefully fixes the bug :)

This revision is now accepted and ready to land.Jan 5 2020, 6:14 PM
This revision was automatically updated to reflect the committed changes.