Define touch gestures in the HIG
Open, Needs TriagePublic

Description

There is a very long draft by Jens R and Martin F about touch gestures https://notes.kde.org/p/touchpadgesture

THIS IS A WORKING DOCUMENT!
From 1500 CET 161124 all edits will be made using "EDIT:" and be marked with date and seperated from other parts!

Touchpad Gestures, Design Document.

  1. Design Notes
  2. Possible Actions
  3. Possible Gestures
  4. Suggested Combinations
  5. Notes and planning
  6. (For reference) Touch Screen notes

--------------------------------1 Design Notes------------------------------------

Primary Goal: Make gestures usable and intuitive
This has to be the first goal, the risk we run with any new technology introduced is making it a collection of fun features that are activated more by mistake than by action. To avoid this we have to make sure that touchpad gestures are well understood, easily memorized and not something that will create huge issues for the user.
The last bit means that the gestures are either complex enough to not be activated by mistake or that they never create a destructive action. "Close Window" for example can never be something that would remotely be activated by mistake no matter how attractive a simple action for it would seem to us.

Secondary Goal: Make gestures editable.
So the second goal here should be to ensure that whatever the gesture does is representable as a shortcut that the user can edit if they want to. Meaning that if a user considers one gesture superflous he/she can then change it to another shortcut with another action connected to it in the shortcuts editor.

Tertiary Goal: Create a unified whole instead of new methods
What this means is that gestures on the touchpad have to lend themselves easily to touchscreens its closest relative. A touch screen isn't the same thing but the actions done on one, should in as much as possible mimic those of the other so that moving from one input to the other isn't confusing or something the user need to relearn.

Key Issues:
Palm Detection. At a certain point we have to insure that Palm Detection works spot on for any of this to be very usable. Without a good Palm Detection any form of gesture can be mistakenly activated by a clumsy swipe over the screen. If the option is "No Touch Gestures" or "Too much touch gesture happening constantly" - we have to go with "No Touch Gestures".
After that comes accessability. We know that certain gestures will be close to impossible depending on physical problems, like arthritis or something so clear as "

-> don't worry about palm detection, that's handled by libinput.

Assumptions:
A touchpad is ONLY present with a Keyboard. Unlike a touchscreen we can always assume a keyboard is as accessible as the touchpad, meaning that some actions and gestures, if they are too complex and are easily replaced with a keyboard shortcut will simply not be relevant. This may not be true in some very extreme cases since you can in theory by a USB touchpad and attach it but we can assume that those situations are so unique that they will hardly be of relevant to us.

Summary:
The main objective is to provide sane defaults, actually usable defaults that the user can, at a later date if he/she wants to. These defaults have to be CLEAR, SAFE and USABLE. A setting for an obscure movement on the touchpad to set off an arbitrary task by default is simply not acceptable.
The goal must be that when the user understands the movement he/she are then helped by this in all sitations. If this means IGNORING ideas to benefit usability that is ok.

--------------------------------2 Possible Actions------------------------------------

Actions and Action Groups:
Mouse Actions:
Single Click - One Finger Tap
Double Click - One Finger Tap x2
Right Click - Two Finger Tap
Middle Click - Three Finger Tap
Scrolling (Horizontal/Vertical) - Two Finger Drag
Dragging - One Finger Drag

Complex Actions:
Swapping Virtual Desktops
Cycling Through Windows
Swap to Next Window
Minimizing Windows
Closing Windows (DESTRUCTIVE)
Show Desktop/Minimize All
Start Launcher
Start Specific Application tied to the desktop (Yakuake)
Start Specific Application NOT tied to the desktop (Dolphin)
Moving Windows
Lock Desktop
Tiling Windows
Lower/Raise Audio Volume
Lower/Raise Screen Brightness
Lower/Raise Keyboard Brightness
Desktop Grid
Show All Windows
Log out options

--------------------------------3 Possible Gestures------------------------------------

LONG PRESS (holding down a finger or more for a longer period of time)

-> that's reported as a mouse press, not a touchpad gesture

PINCH/ZOOM (touch and then squeeze touch areas together, or expand them)
Two Finger
Three Finger
(Four Finger Pinch Excluded as it is too complex to mimic safely)
Five Finger

ROTATE (This action is probably not very intuitive, although the action lends itself well to lowering, raising a clear setting like Audio)
Two Finger Rotate
Three Finger Rotate

EDGE SWIPE (Slide fingers from outside of the touchpad to inside the touchpad)

--> we only have swipe gestures, no detection of edges

Two Finger Edge Swipe
Three Finger Edge Swipe
Four Finger Edge Swipe
From UP - Two Finger Edge Swipe
From UP - Three Finger Edge Swipe
From UP - Four Finger Edge Swipe
From LEFT - Two Finger Edge Swipe
From LEFT - Three Finger Edge Swipe
From LEFT - Four Finger Edge Swipe
From RIGHT - Two Finger Edge Swipe
From RIGHT - Three Finger Edge Swipe
From RIGHT - Four Finger Edge Swipe
From BOTTOM - Two Finger Edge Swipe
From BOTTOM - Three Finger Edge Swipe
From BOTTOM - Four Finger Edge Swipe

This could also be tied into the direction where the panel is placed. So for example "Swipe from the edge where the panel is" might kick off the launcher.

PEAKING (Move fingers from outside of the touchpad a short distance into the touchpad then stopping)

--> we only have swipe or pinch/rotate gesture, there is no such gesture as peaking)

One Finger
Two Finger
Three Finger
Four Finger
Five Finger

FLICKING (Move fingers quickly a short distance over the touchpad)

--> we only have swipe or pinch/rotate gesture, there is no such gesture as flicking)

Two Finger Flick
Three Finger Flick -
Four Finger Flick -
Five Finger Flick seems too complex to do safely

DRAGGING

--> dragging is not reported, this is a mouse press event followed by a mouse move. Cannot be determined

One Finger
Two Finger
Three Finger
Four Finger
Five Finger

CLICKING (Simply clicking/tapping press+release)

--> clicking is not reported, this is a mouse press/release event followed. Cannot be determined

One Finger
Two Finger
Three Finger
Four Finger
Five Finger

WIPE (Move Palm Over the entire touchpad)

--> we don't have any palm events

Left -> Right
Right -> Left
Top -> Bottom
Bottom - Top

HOLD N FINGERS + TAP N FINGERS (touch the touchpad with a certain set of fingers, while tapping it with another finger or fingers)
--> there is no such event reported
This is used in Gnome as a way to swap between windows, and its inclusion is simply to piggy back off of their usage to at least try to create some sense of standard. With a little hope we can use that as an excuse to perhaps make them look out whatever options we pick and use those in GNOME as well to enforce a standard.

MORE COMPLEX PATTERNS impossible but interesting:
Patterns and symbols drawn on the touchpad ... might be interesting as a way to unlock the device (wouldn't be very safe though)
Tracing - sliding one finger along the outer border of the touch pad or sliding two fingers on each border of the touch pad - perhaps as way to lock to make it too complex to do by mistake? Or one finger to lower raise volume or light?
Combination gestures + keyboard. Pressing Ctrl+Shift while Wipe could shut down the computer?

PROBABLY TECHNICALLY IMPOSSIBLE ON TOUCHPADS
Hover - close enough to the touchpad to make it react without touching it like on Samsung Phones Screens
Pressure sense - depth strength of depression creating level of pressure
--> That's a wacom tablet, it provides both hover events and pressure

--------------------------------4 Suggested Combination------------------------------------

(Note: these are all with the assumption that the old standards are kept in, as such two fingers to scroll in application is not used as an example and they are designed not to collide with those)

WIPE = SCREEN LOCK. Swiping side of the palm (which covers majority of touch surface) from right side of the touch pad to the left causes the screen to lock.
Logic: Screenlock isn't destructive but it can be frustrating, at the same time looking at a productive environment where you may have to leave your laptop ANY action to simplify and promote a quick screen lock is good. The massive SWIPE action is comparitively simple, it is fairly unique and the way it can be mistakenly executed is by having a bad palm detection in the touchpad (So this hinges on the palm detection being able to seperate an static palm press with one which starts from one side of the touchpad, moving over it, and ending beyond the other side)

FOUR FINGER TAP + DRAG = MOVING A WINDOW. Four finger actions could be used by the DE by default. So in this case tapping a window and dragging with four fingers make. Disables Four Finger scroll as that would easily be recognized as "drag"). The idea is that the tap of four fingers on a window means activating the drag, it will only stop until ALL fingers are released, meaning the user can tap with four fingers, release three and drag on with one.

FOUR FINGER FLICKING UP = Toggle Maximize Window for the active window

FOUR FINGER FLICKING LEFT/RIGHT = Quick Tile Window Left/Right

FOUR FINGER FLICKING DOWN = Minimize Window

FIVE FINGER PINCH = TOGGLE DISPLAY OPEN WINDOWS (Desktop Grid)
FIVE FINGER EXPAND = UNTOGGLE DISPLAY OPEN WINDOWS (Desktop Grid)

The desrciption "Toggle display open windows" sounds like Present Windows, the parenthesis says Desktop Grid, though. I'm confused.
The reason for not dictating it as "Four Fingers" is simply because the action is (during my limited testing with friends and family) not very pleasant or easily done.

(Sidenote: Desktop Grid really should be given the right to close windows as well as move them to different desktops - thereby merging the benefit from both Desktop Grid and Show All Windows. Initially an idea was to have an easy "close" system for Windows (like "Flicking") when in the Desktop Grid but as the user may want to move windows around that may be too close to other actions and since "close" is by nature destructive it makes it unsuitable. Without that the expand/pinch is not very effective or doesnt mesh well with the gesture. It is although included here since it wouldn't make sense to try to push it through later)
--> could we keep such side notes out of the document?

FOUR FINGER SWIPE FROM BORDER = Move to other Desktop (Swiping from right border moves the user to the right hand desktop, Swiping from bottom border moves the user to the desktop below)

THREE FINGER HOLD PLUS TAP = Alt-Tab, switching between open windows (this is a GNOME specific gesture which seems like a good thing to implement as it needs presence in Plasma as well and the solution is similar enough to the logic of "Four Fingers For Window Control"

In App Gestures:
ONE FINGER LONG PRESS = Right Clicking.
TWO FINGER ZOOM = Zooming in Applications
TWO FINGER ROTATE = Rotating in Applications
I have avoided putting to much information about in app gestures as that was not, to my understanding, relevant to write up and self explanatory. If that was not the case then we can write a further document about it. But safe to say it would simply be a repetition of the same information available in most other DE's who have already implemented touch gestures or for that matter HIGs for touch screens like android.

Idea but not at all clear:
THREE FINGER ROTATE CLOCKWISE = Raise Audio Volume
THREE FINGER ROTATE COUNTER-CLOCKWISE = Lower Audio Volume

--------------------------------Notes and Planning------------------------------------

What is needed to be done still:

  1. Make sure that all gestures are technically possible
  2. Proper testing as the gestures need to actually be usable in practice
  3. Ensure that the implementations are pleasant to use

The reason for this is so that we don't do testing before we know its even possible to make this gestures. The reason for testing is just as a way to ensure that these gestures aren't individual preference and the pattern we try to create with the whole project is meaningful.
That means that when the pattern is created we can actually have it makes sense as a whole instead of strange fun gestures used by a small subset of users and instead a "gesture language" that makes sense to most users. Unless we want a set of "fun features" like the bouncy ball, or the desktop cube.

What is a pattern?
A pattern in this case is a set of objects that function as a group. Specifically it is a set of gestures that not only make sense to the user as a human used to using his/her hands to create certain results but as someone sitting in front of Plasma. The pattern is relevant because by repeating the same "system" or "language" of usage we make the user feel clear about her actions, by repetition we create a sense of wholeness about the entire experience and make sure that the usage of Plasma isn't suddenly broken by something new and alien.
To that end the touch gestures are extremely conservative, I have looked at the notes me and Aleix took during a short conversation about touchscreen gestures and built on those (since they are interlinked in many ways) but have also looked at GNOME's usage of a touch gestures, touch gestures in android and touch gestures on Iphones.
The reason for its importance is that we want the pattern broken only in situations where we want the user to focus on the broken pattern. A bright red button on a flat grey surface demands the users attention and ensure that she will notice it. Since touch gestures are ment to be baked into normal usage we want there to be a common language of gestures without any breakage.

THIS in turn is relevant since IF for some reason one pattern is impossible to implement, we can not implement the others. Some are not at any risk in being affected but say that certain gestures need to be replaced with something else to start that action, our entirely pattern is broken. In those cases its always better to not implement at all.

After that comes tricky parts that need implementation to be tested, "how fast should you move your palm over the touchpad to lock the screen?" for example? If it isn't set correctly so that the action either triggers to easily or not at all - the entire pattern is broken. So we would have to be careful there too.
This makes this entire project rather difficult and have been from the start since the technical limitations are still unknown and the testing, being something entirely new - cannot be done as smoothly during production.

EDIT: 161124 - was told that this was ment to be experimental and sending it off would be more important than I thought. So we have to remember that at ANY point the language/pattern we are building doesn't make sense or when it starts breaking in informal tests - it needs to be reevaluated. The users have to be informed that it is experimental and a work-in-progress. We should also write tests for it now, so that we can get the users to help us with proper design tests.

--------------------------------6 Touch Screen Notes, for reference------------------------------------

These notes where taken during a short talk between Aleix and Jens.
Seeing the touchpad gestures it may be a good idea to mimic those. Leave three fingered actions to mimic mouseclicks (middle mouse button) and stay with the similar actions as above.

Focus: To make sure that the Plasma Desktop basic actions can be done with a touch screen
On Edgeswipes Kwin takes precedent over applications: That means that any edge swipe on an "active edge" for Kwin will be sent to Kwin and not the application.

Panel Interaction: A hidden panel on one edge will be activated when swiping from that edge. It will then remain in focus until something else is chosen (like the Application Launcher)
3 finger events are for KWin/Plasma features   (Theoretical Examples) 
Window Movement: Like drag&drop but with 3 fingers
Desktop Switching: Like edge swipe but with 3 fingers?
etc.

Mouse Actions:

Note: we cannot emulate mouse actions through touch screen. These are different input devices and the app needs to support touch events. Inside the application it can be translated, but it needs to be per application, it's not globally possible.

Clicking: Single Tap
Double Clicking: Two Taps (press release, press relase, fast)
Right Click: Long Press
Middle Click: STILL NOT SOLVED
Drag-and-drop: Tap + Movement/Drag (press, release, press, drag)

Bugs:

Auto-hide panel: https://bugs.kde.org/show_bug.cgi?id=370323
Plasmoid context menu: https://bugs.kde.org/show_bug.cgi?id=368772

This is a pretty good proposal. But I think there are some gestures we can't differentiate like the flick from the drag and also the edge swipe for 3 or 4 fingers seems to complex to determine and to be using it's good to have so many gestures so that the user can select them according to their will.

Is it possible that we can cut down some of the gestures?