Create Task
Maniphest T10462

Define input-output usage patterns
Open, NormalPublic
Actions

Description

We (Dorota from Purism, @mart, me) had some discussions at Plasma Mobile Sprint today about the question on how KWin should react to different input and output device classes. I'm looking for more feedback on our ideas.

Current state of affairs

  • On Plasma Mobile we install a kwinrc config file, which controls KWin's behavior, for example that the window placement strategy should always be "Maximizing".
  • In KWin we have the TabletModeManager class. Its purpose is to inform clients via DBus about the current usage mode of convertibles. Kirigami uses this information for example to change the size of visual elements to allow easier touch control.

Goals

  • Define a concept of determining and controlling the window placement and other window manager internal characteristic depending on current hardware and configuration.
  • Support multiple characteristics in the same session with different hardware devices (convergence).
  • Communicate the selected characteristic to interested clients.

Solution proposal

KWin internal

  • Define in KWin usage patterns describing the preferred way of arranging content and interacting with it. For example: Desktop, Phone, Tablet, TV, VR, ...
  • Set usage pattern per output.
  • Determine which usage pattern should be set on which output by evaluating information about the size of the output, the overall availability of input methods and manual system/user configuration.
  • Change window placement strategy, decoration visibility and other properties accordingly (should still be configurable, but per usage pattern).

Informing clients

  • Define a Wayland protocol to inform interested clients per surface, when this surface changes output. Set enums describing the current usage pattern on the output. Enums could be:
    • Directly named Desktop, Phone,... like the usage patterns
    • Enums named Mouse, Touch, GameController, VRGoggles,... resembling real hardware.
  • Add new names in new protocol versions.
  • Solution 1: Send a single enum
    • If client only supports older version, send the best fitting supported older enum as fallback.
  • Solution 2: Send a priority list of enum values
    • Fallback is integrated directly. Clients should just ignore values in the list if they can't interpret it.
    • By reading out multiple values, clients are able in theory to determine how best react to the usage pattern.
  • Solution 3: Send two enum values, one for input usage pattern and one for output usage pattern.
    • Combinations like Phone + Touch / Phone + Keyboard or VRGoggles + Gloves / VRGoggles + KnuckleControllers are easily possible
    • Might be redundant in certain cases: For example difference between Phone + Touch / Tablet + Touch is negligible client-side (in this example not with CSD).

Related Objects
Search...

romangg created this task.Feb 8 2019, 5:15 PM
romangg triaged this task as Normal priority.
romangg updated the task description. (Show Details)Feb 8 2019, 5:33 PM
davidedmundson added a subscriber: davidedmundson.Feb 8 2019, 5:44 PM

Change window placement strategy, decoration visibility and other properties accordingly

I'd quite like to improve our kwin scripting for initial placement.

Ideally we want it to run between XdgShellSurface being committed and us sending our initial configure.
Kwin scripts would give plasma mobile maximum flexibility, and help with all the kwin tiling scripts, and allow us to slowly clear out the horrible placement.cpp class.

Define a Wayland protocol to inform interested clients per surface, when this surface changes output.

wl_surface already has a protocol to informs clients when their surface changes output. It supports a surface being on N at once.
It should work right now.

From the sounds of it we just want to decorate either the wl_output/xdg_output/kde_output with some extra semantic values. Should be doable.

romangg added a comment.Feb 9 2019, 9:41 AM
In T10462#175694, @davidedmundson wrote:

Define a Wayland protocol to inform interested clients per surface, when this surface changes output.

wl_surface already has a protocol to informs clients when their surface changes output. It supports a surface being on N at once.
It should work right now.

From the sounds of it we just want to decorate either the wl_output/xdg_output/kde_output with some extra semantic values. Should be doable.

Yea, sounds spot on. What do you think how these semantic values should be defined? I.e. solution 1, 2, 3 above or something else?

dcz-purism added a subscriber: dcz-purism.Feb 9 2019, 12:15 PM

That doesn't sound quite right.

In T10462#175694, @davidedmundson wrote:

Change window placement strategy, decoration visibility and other properties accordingly

I'd quite like to improve our kwin scripting for initial placement.

Ideally we want it to run between XdgShellSurface being committed and us sending our initial configure.
Kwin scripts would give plasma mobile maximum flexibility, and help with all the kwin tiling scripts, and allow us to slowly clear out the horrible placement.cpp class.

Define a Wayland protocol to inform interested clients per surface, when this surface changes output.

wl_surface already has a protocol to informs clients when their surface changes output. It supports a surface being on N at once.
It should work right now.

From the sounds of it we just want to decorate either the wl_output/xdg_output/kde_output with some extra semantic values. Should be doable.

That's not quite right, there are many cases when the usage patterns change, but the output doesn't. Some examples:

  • tablet has a keyboard attached,
  • convertible switches modes,
  • a gamer picks up a controller and moves from the keyboard chair to a couch farther away.

I don't think the modes should be strongly attached only to display, there's more going on than that.

romangg renamed this task from Defining output-input usage patterns to Define input-output usage patterns.Feb 19 2019, 5:13 PM
romangg edited projects, added Plasma; removed KWin.Feb 19 2019, 6:38 PM
romangg added a comment.Feb 19 2019, 6:43 PM
In T10462#175716, @dcz-purism wrote:

...
That's not quite right, there are many cases when the usage patterns change, but the output doesn't. Some examples:

  • tablet has a keyboard attached,
  • convertible switches modes,
  • a gamer picks up a controller and moves from the keyboard chair to a couch farther away.

That's true. So the Wayland event message must be sent also on these hot-plug events or when the user does a manual override, what maybe warrants a separate protocol.

romangg added a comment.Feb 19 2019, 6:44 PM

I've created some sub-tasks to discuss definition of some common usage patterns we might want to concentrate first: Desktop, Phone, TV.

What's missing here is the Tablet use-case since I'm not sure if Phone and Tablet are just the same thing or should be treated as separate patterns.

nicolasfella added a comment.Feb 19 2019, 7:40 PM

What's missing here is the Tablet use-case since I'm not sure if Phone and Tablet are just the same thing or should be treated as separate patterns.

That depends a bit on your definition of Tablet.

A 7'' tablet is not much different than a phone, but a larger tablet with an attachable keyboard is a very different device

romangg added a comment.Feb 20 2019, 9:56 AM
In T10462#176648, @nicolasfella wrote:

What's missing here is the Tablet use-case since I'm not sure if Phone and Tablet are just the same thing or should be treated as separate patterns.

That depends a bit on your definition of Tablet.

A 7'' tablet is not much different than a phone, but a larger tablet with an attachable keyboard is a very different device

I've added now another IOUP, so we can keep tablets and phones separated for a start. Later we can still decide if they are the same after all. One important difference justifying the distinction is probably though that phones can be used in most cases with one hand, while tablets require both hands to hold and interact with it.