virtualkeyboard: resize the focused window to make room for the keyboard
Needs RevisionPublic

Authored by mart on Thu, Feb 7, 4:55 PM.

Details

Summary

alternative approach: try to resize the winidow to make room for the keyboard.
the new input wayland protocol doesn't have anymore the overlap rectangle (and it would not be going to work with qwidget apps anyways)

in the future will probably be needed anextension to the input protocol v3 which partially gets back this, tough window resizing is needed regardless

what's missing: the resize should be "temporary" and the window should be restored to its previous geometry when the keyboard closes

Test Plan

tested with test QML code

Diff Detail

Repository
R108 KWin
Branch
mart/resizeforkeyboard
Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 8058
Build 8076: arc lint + arc unit
mart requested review of this revision.Thu, Feb 7, 4:55 PM
mart created this revision.
mart added a reviewer: romangg.Thu, Feb 7, 5:21 PM
mart updated this revision to Diff 51116.Thu, Feb 7, 5:44 PM
  • remove wrong line
mart abandoned this revision.Thu, Feb 7, 5:44 PM
mart reclaimed this revision.
mart updated this revision to Diff 51119.Thu, Feb 7, 6:09 PM
  • use geometryRestore
mart updated this revision to Diff 51125.Thu, Feb 7, 7:07 PM
  • move the window at the first keyboard show
mart updated this revision to Diff 51134.Thu, Feb 7, 9:53 PM
  • correctly restore
romangg added inline comments.Fri, Feb 8, 10:45 AM
virtualkeyboard.cpp
245

Can you put this and the next condition in the beginning and:

if(!condition) {
    return;
}
mart updated this revision to Diff 51179.Fri, Feb 8, 1:26 PM
  • correctly handle restoring
mart updated this revision to Diff 51181.Fri, Feb 8, 1:37 PM
  • clean logic with early returns
mart marked an inline comment as done.Sat, Feb 9, 9:36 AM
mart updated this revision to Diff 51240.Sat, Feb 9, 9:48 AM
  • remove useless includes
romangg accepted this revision.Sat, Feb 9, 10:09 AM

There are some open questions still:

  • How to deal with desktop containments in a sensible way. With this patch applets and the launcher are moved above the virtual keyboard, which is already a great improvement. But we might want to deal with them in a more nicer way.
  • Doe we want to expose the virtual keyboard position information somehow to the clients? The old text-input v2 protocol provided this information. We might want to let clients opt-in to not being size-changed, but placing the keyboard into their window instead at some proposed coordinates. I could imagine this could be nice on larger screens with multiple windows open. But this would need a text-input v4 protocol version or a separate protocol.
  • Talking about multiple windows open: currently only the input focus window is moved. What's with other windows? I might be interested in them as well on a large screen while typing into the focused window.

But anyway from the demo I saw this patch makes the virtual keyboard way more usable. So let's merge it and think about the open questions in the future again.

virtualkeyboard.h
30

Necessary?

This revision is now accepted and ready to land.Sat, Feb 9, 10:09 AM
graesslin requested changes to this revision.Sun, Feb 10, 4:14 PM

I'm not sure whether it's a good idea to resize windows for virtual keyboard. This can easily result in a feedback loop - the window resizes, the element with focus loses focus, keyboard closes, window resizes, element gets focus again and so on.

I would prefer if we could just move the window away. This could be done just by the compositor.

I'm setting to requires revision as I think from KWin point of view it's wrong to track the geometry for restore in VirtualKeyboard. For everything else the AbstractClient tracks it itself. E.g. there is a restoredGeometry for maximize and for fullscreen. I think the same should be done for virtual keyboard. This also ensures that the system is properly able to track state. The code now is dangerous. If the window is in maximized state, it will lose that state on resize and then the restore geometry gets broken.

This revision now requires changes to proceed.Sun, Feb 10, 4:14 PM

Please don't take it negative that I once again request changes here. I once run into this issue myself when I added quick tiling - oh that was a mess, because I didn't track in Client the state changes. Back then Thomas fixed all of it :-) It's just a lesson learned and this geometry handling is really complex. It's too many states and we need to ensure here that we don't jump out of maximized or fullscreen due to keyboard open. Now to make it even more complex one could think about the display getting rotated which would again change the geometries and all the windows changes.

I'll outline an idea how this could work: When virtual keyboard opens the focused window gets maximized and the virtual keyboard acts like a panel with strut - but just for the active window. Then we could use existing KWin functionality without having to track the geometry again.

I'll outline an idea how this could work: When virtual keyboard opens the focused window gets maximized and the virtual keyboard acts like a panel with strut - but just for the active window. Then we could use existing KWin functionality without having to track the geometry again.

On large enough screens (desktop, tablet) we want real window management or tiling and user might want to have a secondary window open for information. For example typing into text editor a summary of a page opened in browser. Just maximizing the active window wouldn't allow that.

Can you expand on the "like a panel with strut" part?

mart added a comment.Mon, Feb 11, 1:34 PM

On large enough screens (desktop, tablet) we want real window management or tiling and user might want to have a secondary window open for information. For example typing into text editor a summary of a page opened in browser. Just maximizing the active window wouldn't allow that.

Can you expand on the "like a panel with strut" part?

tough it could maximize only vertically...

mart added a comment.Mon, Feb 11, 1:39 PM

Please don't take it negative that I once again request changes here. I once run into this issue myself when I added quick tiling - oh that was a mess, because I didn't track in Client the state changes. Back then Thomas fixed all of it :-) It's just a lesson learned and this geometry handling is really complex.

none taken :) the more i look at this problem, the more complex and full of corner cases it looks like.

It's too many states and we need to ensure here that we don't jump out of maximized or fullscreen due to keyboard open. Now to make it even more complex one could think about the display getting rotated which would again change the geometries and all the windows changes.

I'll outline an idea how this could work: When virtual keyboard opens the focused window gets maximized and the virtual keyboard acts like a panel with strut - but just for the active window. Then we could use existing KWin functionality without having to track the geometry again.

I like that idea (with the caveat roman said, to perhaps maximize only vertically, but i don't have super strong opinions on this)

Implementation wise:

  • would all be done here still saving the old maximized state saved and restored from virtualkeyboard.cpp,
  • or having abstractclient itself knowing when there is a keyboard open and maximize itself when it has focus?

(or something else entirely?)

mart added a comment.Mon, Feb 11, 1:50 PM

I'm not sure whether it's a good idea to resize windows for virtual keyboard. This can easily result in a feedback loop - the window resizes, the element with focus loses focus, keyboard closes, window resizes, element gets focus again and so on.

I would prefer if we could just move the window away. This could be done just by the compositor.

you mean the window just to be painted as translated upwards, with parts of the contents outside the top of the screen?

Android has a possible virtual keyboard mode that does that, but is very rarely used in apps since it doesn't work that well and usability wise is kindof annoying.
getting back on Android, just to see what others do, it has 3 possible modes which the app can set in its manifest.xml (and never change again at runtime afaik)

  • sliding up the window enough to show the input field, as i said doesn't really seem to be used buch
  • window resize, the window is resized to make space for the keyboard, all the content that fits is still shown
  • the app makes room for the keyboard in itself: this is pretty much the approach of the previous patch which sets theoverlap region. Unfortunately this was removed from the last wayland input protocol revision, which is unfortunate (i think there are still use cases for it, even tough the security reasons used as motivation kinda make sense, knowing the screen size, its own size and the default keyboard size on the platform, a surface can infer its screen position with a good approximation)

in the end we will probably need some mechanism to select between the different approaches, some or all of the above...
at first tough i would like an approach which works in most of the cases (except on things like the desktop window) which shrinks the window.. if you think that setting a maximized state on the window is safer than just resizing, i can look into that.