[wayland] Allow nested mode to have a fractional scale < 1

The bug reporter presents a use case of wanting to run a game and scale
down the result.

For everyone else kwin_wayland nested is just a test tool, so I don't
see any disadvantages.

It could be useful in testing.

BUG: 422141