WinInk: Simulate native mouse events for handled pen events
ClosedPublic

Authored by alvinhochun on Nov 13 2017, 8:17 PM.

Details

Summary

This changes the behaviour of the WIndows Pointer Input handling code. Instead of "ignoring" the pointer events and hoping that Windows will synthesize mouse events for us (which doesn't work properly on ver 1709), we inject the mouse events into Windows for handled pen events, and just let Windows synthesize touch+mouse events normally for unhandled pen events.

This also makes it act more like the WInTab mode.
(See also: https://bugs.kde.org/show_bug.cgi?id=386476)

This mainly fixes code that rely on QCursor::pos() to get the cursor position. QCursor::pos() calls GetCursorPos WinAPI. Since Windows doesn't update the cursor position if the pen pointer events are handled, certain elements would fail to follow the pen. (E.g. parallel ruler assistant: https://bugs.kde.org/show_bug.cgi?id=386475)

There is an issue where calling up the canvas popup palette would result in an unwanted click on the palette, but it also happens with WinTab so it's not new. It's a side-effect of mouse events being sent after the tablet events.

It is a hackish solution, but there is no way I can change QCursor::pos() to also use the pen position.

Commit message:

WinInk: Simulate native mouse events for handled pen events

The code now simulates native mouse events with `SetCursorPos` and
`SendInput` for pen pointer events whose QTabletEvents are handled by
the widgets, so that the cursor position is updated to get correct
results from QCursor::pos().

BUG: 386475
BUG: 386476
Differential Revision: https://phabricator.kde.org/D8801
Test Plan
  1. Make sure pointer input and Windows Ink are enabled.
  2. Make a document with parallel ruler assistants (e.g. one vertical and one horizontal)
  3. Draw on the canvas to check that it follows the cursor.
  4. Drag pen across the color selector and check that it works smoothly.

I've tested with Windows 10 ver 1709 with Surface Pro, and Wacom, also ver 1703 with Wacom, all seems to work fine.

Diff Detail

Repository
R37 Krita
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
alvinhochun created this revision.Nov 13 2017, 8:17 PM
alvinhochun edited the summary of this revision. (Show Details)Nov 14 2017, 2:17 PM

Hi, @alvinhochun!

I guess you can just replace QCursor::pos() with your own implementation, which asks either your winink implementation for the real cursor position (if winink is active), or just falls back to QCursor::pos() on other platforms. We already have KisCursor class, which can logically adopt this new function :)

Hi, @alvinhochun!

I guess you can just replace QCursor::pos() with your own implementation, which asks either your winink implementation for the real cursor position (if winink is active), or just falls back to QCursor::pos() on other platforms. We already have KisCursor class, which can logically adopt this new function :)

This might not be enough as I believe a lot of Qt code also use QCursor::pos()...

I can possibly send synthesized mouse events to the OS instead (with SendInput). It would seem to be too much but there doesn't seem to be another way other than having pointer input implemented in Qt and changing all the cursor handling code for it.

Sending mouse events with Windows API doesn't work very well :( There is always a delay at the start of a movement...

1diff --git a/libs/ui/input/wintab/kis_tablet_support_win8.cpp b/libs/ui/input/wintab/kis_tablet_support_win8.cpp
2index 53bb88a43d..69367c43c0 100644
3--- a/libs/ui/input/wintab/kis_tablet_support_win8.cpp
4+++ b/libs/ui/input/wintab/kis_tablet_support_win8.cpp
5@@ -38,6 +38,8 @@
6 #include <QWidget>
7 #include <QWindow>
8
9+#include <private/qguiapplication_p.h>
10+
11 #include <utility>
12
13 #include <kis_debug.h>
14@@ -624,7 +626,77 @@ bool sendProximityTabletEvent(const QEvent::Type eventType, const POINTER_PEN_IN
15 return ev.isAccepted();
16 }
17
18-bool sendPositionalTabletEvent(QWidget *target, const QEvent::Type eventType, const POINTER_PEN_INFO &penInfo, const PointerDeviceItem &device, const PenPointerItem &penPointerItem)
19+bool synthesizeMouseEvent(const QWidget *target, const QTabletEvent &ev, const POINTER_PEN_INFO &penInfo)
20+{
21+ BOOL result = SetCursorPos(penInfo.pointerInfo.ptPixelLocationRaw.x, penInfo.pointerInfo.ptPixelLocationRaw.y);
22+ if (!result) {
23+ qDebug() << "SetCursorPos failed, err" << GetLastError();
24+ // return true;
25+ }
26+ QPointF localPosF = ev.posF();
27+ QWindow *window = target->windowHandle();
28+ if (!window) {
29+ QWidget *nativeParent = target->nativeParentWidget();
30+ if (!nativeParent) {
31+ warnTablet << "KisTabletSupportWin8::synthesizeMouseEvent target->nativeParentWidget is null!";
32+ return false;
33+ }
34+ window = nativeParent->windowHandle();
35+ const QPoint globalPos = ev.globalPosF().toPoint();
36+ const QPoint localPos = nativeParent->mapFromGlobal(globalPos);
37+ const QPointF delta = ev.globalPosF() - globalPos;
38+ localPosF = localPos + delta;
39+ }
40+ QWindowSystemInterfacePrivate::MouseEvent fake(window, ev.timestamp(), localPosF,
41+ ev.globalPosF(),ev.buttons(), ev.modifiers(), Qt::MouseEventSynthesizedByQt);
42+ fake.flags |= QWindowSystemInterfacePrivate::WindowSystemEvent::Synthetic;
43+ QGuiApplicationPrivate::processMouseEvent(&fake);
44+ // DWORD inputDataFlags = 0;
45+ // switch (ev.type()) {
46+ // case QEvent::TabletPress:
47+ // switch (ev.button()) {
48+ // case Qt::LeftButton:
49+ // inputDataFlags = MOUSEEVENTF_LEFTDOWN;
50+ // break;
51+ // case Qt::RightButton:
52+ // inputDataFlags = MOUSEEVENTF_RIGHTDOWN;
53+ // break;
54+ // default:
55+ // return true;
56+ // }
57+ // break;
58+ // case QEvent::TabletRelease:
59+ // switch (ev.button()) {
60+ // case Qt::LeftButton:
61+ // inputDataFlags = MOUSEEVENTF_LEFTUP;
62+ // break;
63+ // case Qt::RightButton:
64+ // inputDataFlags = MOUSEEVENTF_RIGHTUP;
65+ // break;
66+ // default:
67+ // return true;
68+ // }
69+ // break;
70+ // case QEvent::TabletMove:
71+ // default:
72+ // inputDataFlags = MOUSEEVENTF_MOVE;
73+ // break;
74+ // }
75+ // INPUT inputData = {};
76+ // inputData.type = INPUT_MOUSE;
77+ // inputData.mi.dx = penInfo.pointerInfo.ptPixelLocationRaw.x;
78+ // inputData.mi.dy = penInfo.pointerInfo.ptPixelLocationRaw.y;
79+ // inputData.mi.dwFlags = inputDataFlags | MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_MOVE_NOCOALESCE;
80+ // //inputData.mi.dwExtraInfo =
81+ // UINT result2 = SendInput(1, &inputData, sizeof(inputData));
82+ // if (result2 != 1) {
83+ // qDebug() << "SendInput failed, err" << GetLastError();
84+ // return true;
85+ // }
86+ return true;
87+}
88+
89+bool sendPositionalTabletEvent(QWidget *target, const QEvent::Type eventType, const POINTER_PEN_INFO &penInfo, const PointerDeviceItem &device, const PenPointerItem &penPointerItem, const bool shouldSynthesizeMouseEvent = true)
90 {
91 KIS_SAFE_ASSERT_RECOVER_RETURN_VALUE(
92 eventType == QEvent::TabletMove || eventType == QEvent::TabletPress || eventType == QEvent::TabletRelease,
93@@ -634,7 +706,10 @@ bool sendPositionalTabletEvent(QWidget *target, const QEvent::Type eventType, co
94 ev.setAccepted(false);
95 ev.setTimestamp(penInfo.pointerInfo.dwTime);
96 QCoreApplication::sendEvent(target, &ev);
97- return ev.isAccepted();
98+ if (!shouldSynthesizeMouseEvent) {
99+ return true;
100+ }
101+ return synthesizeMouseEvent(target, ev, penInfo);
102 }
103
104 bool handlePenEnterMsg(const POINTER_PEN_INFO &penInfo)
105@@ -698,7 +773,7 @@ bool handlePenLeaveMsg(const POINTER_PEN_INFO &penInfo)
106 return false;
107 }
108
109-bool handleSinglePenUpdate(PenPointerItem &penPointerItem, const POINTER_PEN_INFO &penInfo, const PointerDeviceItem &device)
110+bool handleSinglePenUpdate(PenPointerItem &penPointerItem, const POINTER_PEN_INFO &penInfo, const PointerDeviceItem &device, const bool shouldSynthesizeMouseEvent)
111 {
112 QWidget *targetWidget;
113 if (penPointerItem.isCaptured()) {
114@@ -745,7 +820,7 @@ bool handleSinglePenUpdate(PenPointerItem &penPointerItem, const POINTER_PEN_INF
115 // penPointerItem.activeWidget = targetWidget;
116 }
117
118- bool handled = sendPositionalTabletEvent(targetWidget, QEvent::TabletMove, penInfo, device, penPointerItem);
119+ bool handled = sendPositionalTabletEvent(targetWidget, QEvent::TabletMove, penInfo, device, penPointerItem, shouldSynthesizeMouseEvent);
120 if (!handled) {
121 // dbgTablet << "Target widget doesn't want pen events";
122 }
123@@ -783,12 +858,13 @@ bool handlePenUpdateMsg(const POINTER_PEN_INFO &penInfo)
124 // The returned array is in reverse chronological order
125 const auto rbegin = penInfoArray.rbegin();
126 const auto rend = penInfoArray.rend();
127+ const auto rlast = rend - 1; // Only synthesize mouse event for the last one
128 for (auto it = rbegin; it != rend; ++it) {
129- handled |= handleSinglePenUpdate(*currentPointerIt, *it, *devIt); // Bitwise OR doesn't short circuit
130+ handled |= handleSinglePenUpdate(*currentPointerIt, *it, *devIt, it == rlast); // Bitwise OR doesn't short circuit
131 }
132 return handled;
133 } else {
134- return handleSinglePenUpdate(*currentPointerIt, penInfo, *devIt);
135+ return handleSinglePenUpdate(*currentPointerIt, penInfo, *devIt, true);
136 }
137 }
138
139@@ -979,9 +1055,7 @@ bool handlePointerMsg(const MSG &msg)
140 case WM_POINTERLEAVE:
141 return handlePenLeaveMsg(penInfo);
142 case WM_POINTERUPDATE:
143- // HACK: Force further processing to force Windows to generate mouse move events
144- handlePenUpdateMsg(penInfo);
145- return false;
146+ return handlePenUpdateMsg(penInfo);
147 case WM_POINTERCAPTURECHANGED:
148 // TODO: Should this event be handled?
149 dbgTablet << "FIXME: WM_POINTERCAPTURECHANGED isn't handled";

(Note to self:) SendInput works this way:

QRect virtualDesk(GetSystemMetrics(SM_XVIRTUALSCREEN), GetSystemMetrics(SM_YVIRTUALSCREEN), GetSystemMetrics(SM_CXVIRTUALSCREEN), GetSystemMetrics(SM_CYVIRTUALSCREEN));
INPUT inputData = {};
inputData.type = INPUT_MOUSE;
inputData.mi.dx = ((penInfo.pointerInfo.ptPixelLocationRaw.x - virtualDesk.x()) * 65536 + 32768) / virtualDesk.width();
inputData.mi.dy = ((penInfo.pointerInfo.ptPixelLocationRaw.y - virtualDesk.y()) * 65536 + 32768) / virtualDesk.height();
inputData.mi.dwFlags = inputDataFlags | MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_VIRTUALDESK /*|  MOUSEEVENTF_MOVE_NOCOALESCE*/;
// inputData.mi.dwExtraInfo = 0xFF515700 | 0x01; // https://msdn.microsoft.com/en-us/library/windows/desktop/ms703320%28v=vs.85%29.aspx
UINT result2 = SendInput(1, &inputData, sizeof(inputData));
alvinhochun retitled this revision from (WIP) WinTab: Synthesize mouse events with Qt internal function to WinInk: Simulate native mouse events for handled pen events.
alvinhochun edited the summary of this revision. (Show Details)
alvinhochun edited the test plan for this revision. (Show Details)

Changed to use Windows APIs to nudge the cursor for handled pen events...

alvinhochun edited the test plan for this revision. (Show Details)Dec 10 2017, 7:22 PM
rempt accepted this revision.Dec 12 2017, 8:00 AM
rempt added a subscriber: rempt.

I haven't had a chance to actually test this, but I'm fine with pushing it to master and 3.3 -- then we can make some builds and ask people to do tests?

This revision is now accepted and ready to land.Dec 12 2017, 8:00 AM
This revision was automatically updated to reflect the committed changes.