Text For Everyone: International Input Methods, Emoji πŸ‰, Word Completion
Open, Needs TriagePublic

Description

Description

Text input is the foundational means of human-computer interaction: We configure or systems, program them, and express ourselves through them by writing text. It's how our software is made, and it's how it's consumed. It's how users and developers connect with each other.

Text input in KDE software today faces many challenges:

  • Input methods for many languages are too hard to set up, requiring deep expert knowledge to install and configure relevant add-on software and third-party settings UI.
  • It's very hard to switch an English-language system to accepting input in other languages.
  • Support for typing acceleration/assistive features like word completion is very limited.
  • Support for easy input of text emoji is lacking.
  • Functionality and behavior between our physical and virtual (on-screen) keyboard input methods are inconsistent and far from being at parity.

The situation as described above makes using our software not as easy and enjoyable for many users as it should be, and some cannot use it at all.

It also holds KDE software back in many markets that are important potential growth markets for our software, particularly in East Asia.

Fixing this requires concerted action in our community. When done, it will add new fundental capabilities to our software that will help it reach new audiences and thereby grow our community of users and contributors.

Let's do it!

Proposed changes

  • Forming a strong team to tackle text input and related l10n challenges within the KDE community, leveraging many types of skills: UX, visual/icon assets, code, communication.
  • Working closely with the wider FOSS text input community by inviting relevant outside contributors to a KDE sprint.
  • Deep and non-optional integration of input method settings in Plasma's System Settings, building on a 2017 GSoC project (in detail: ibus and fcitx configuration in System Settings and a next-gen panel applet, evolving existing Keyboard settings and panel indicators beyond keyboard layout management).
  • Working on implementing more relevant technology standards in KDE software (in detail: supporting the Wayland text-input and input-method protocols in KWin and kwayland).
  • Doing pioneering work on aligning our virtual keyboard with our desktop input stack, achieving feature and behavior parity (in detail: an ibus backend for Qt Virtual Keyboard).
  • Designing and deploying improved emoji support throughout the system (in detail: a default emoji picker UI and keyboard shortcut, integrated via KF5/QPA into standard text UI controls).
  • Working upstream in our stack to fix relevant bugs (in detail: e.g. text underline style rendering in Qt Quick, behavior bugs in ibus, contributing to relevant Wayland protocols).
  • Working with downstream distributions of KDE software on packaging and installing the required dependencies. Addressing integration problems posed by new application container formats.
  • Educating on and documenting text input challenges and solutions/best practices throughout our community.
  • Promoting KDE's effort to take a leading role in this domain and the things we accomplish.

Relevant links

How we know we succeeded

  • Users stop asking how to setup input methods on forums and bug trackers. Tutorials become obsolete because the steps to configure a new IM are obvious and discoverable.
  • Our software sees increased uptake in new regions of the world that require input method support.
  • Plasma is the free system of choice for users that require input method support.

I am interested

I am willing to put work into this

Should we have input methods as a goal or maybe something a bit more broad such as Accessibility?

hein added a comment.EditedJun 11 2019, 11:03 AM

I submitted the same goal last year:

https://phabricator.kde.org/T6838

I also mentored the GSoC project that implemented a lot of this.

So naturally, my full support. This is incredibly necessary work.

Some the details of the proposal are a bit wrong though, although the brunt of it has the right goals. Here's my talk that lays out the todo:

https://youtu.be/wE0KDabPfsQ

! In T11054#187995, @hein wrote:
Some the details of the proposal are a bit wrong though, although the brunt of it has the right goals. Here's my talk that lays out the todo:

https://youtu.be/wE0KDabPfsQ

As the comment on Youtube points out the slides are not readable. Can you upload them somewhere and link here or in the video, so one can follow your talk better. I would also like to get an overview on current state on X11 in particular.

On Wayland there is the text-input and input-methods protocols. Mostly pushed by Purism at the moment because of their need in Librem 5 smartphones (for virtual keyboard). If these protocols lost focus in regards to traditional input-methods for internationalization and accessibility is yet to be seen when they are finally implemented somewhere or by a thorough inspection. For us there is a patch of the text-input protocol D16735, but without the input-methods protocol it does not make much sense to merge it yet.

GNOME uses some internal stuff, wlroots on the other side has implemented both protocols. There is also the question how this can be integrated with XWayland: https://gitlab.freedesktop.org/xorg/xserver/issues/820

In general I propose to concentrate on working on a user-friendly solution primarily on Wayland and only if possible try to rescue concepts over to X11.

Should we have input methods as a goal or maybe something a bit more broad such as Accessibility?

IMO Input methods could be part of accessibility. Think about Voice- , Braille-, OSK and similar. This could include input for different languages as well.

hein added a comment.Jun 11 2019, 2:35 PM

As the comment on Youtube points out the slides are not readable. Can you upload them somewhere and link here or in the video, so one can follow your talk better. I would also like to get an overview on current state on X11 in particular.

Talk slides: www.eikehein.com/kde/Input_Methods_in_Plasma_5.pdf

I've also contributed a bit to the Wayland protocols by working with dorota and the Sway people.

In T11054#187995, @hein wrote:

I submitted the same goal last year:

https://phabricator.kde.org/T6838

Your proposal was framed and justified so much better than mine!

Feel free to edit this task and merge your ideas into it (do I need to grant you access or can anyone edit? This is my first time using Phabricator).

hein updated the task description. (Show Details)Jun 12 2019, 9:25 AM

I would object a stronger integration with IBus (or any of the single input method). Even within the "common" CJK input method framework, Chinese/Japanese and Korean input methods have different requirements, which is a common source of frustration why Korean IM is sometimes not working out-of-the-box.

  • Chinese/Japanese input requires dictionaries to translate phonetic input into Hanzi/Kana, while Korean input is not heavily depending on the usage of dictionary for Hangul input (only Hanja requires it)
  • Chinese/Japanese input uses "explicit" commit of preedit text and when user moves focus away from a control it is okay to remove the preedit text, while Korean input uses "implicit" commit of preedit text and user expects automatic commit of preedit text when user moves focus away. This behavior is expected from Korean users from the beginning of the computer era, and it is not changeable. Period.
  • Chinese/Japanese IM includes several extra features regarding text input, while Korean IM doesn't have eyecandy features and users don't expect that much from an IM.

IBus once had (and maybe still?) a notorious bug for Korean users (and bad image as such) so called "trailing character bug"/"λκΈ€μž 버그", where preedit text is not committed properly and follows to another text area even after changing the focus. The asynchronous event loop design of IBus made the Korean input problem complicated, and it was counted as one of the motivation of developing another IM framework Nimf [1] (although I haven't personally evaluated it). While the IBus bug was said to be addressed, since my first task after installation is always removing IBus and installing something else to avoid that annoying bug, I couldn't confirm whether the bug is persistent yet. Some applications are also known to having such a bug, when it modifies preedit text handling behavior, not handling preedit text properly or interferes the events generated by IM (premature IM reset or such). This needs to be addressed by stable set of requirements and expected behaviors.

Also we need to make sure that common shortcuts used by applications should not collide with the IM's global shortcuts such as Ctrl+Space or Shift+Space. If that happens, users should be informed about the possibility and change either the app or the IM. This also applies to xkb options: when both Plasma and IM wants to access/modify xkb settings it should not collide againast each other.

Finally, cross-distro packages (AppImage, Snap, Flatpak) is another source of problem. Due to sandboxed design and limited integration with what is installed in the system, IMs are not available to the applications installed by those packages. This definitely is another source of user's confusion.

[1] https://gitlab.com/nimf-i18n/nimf/

pshinjo updated the task description. (Show Details)Jun 13 2019, 12:02 PM
ngraham updated the task description. (Show Details)Jun 13 2019, 1:40 PM

I would object a stronger integration with IBus (or any of the single input method). Even within the "common" CJK input method framework, Chinese/Japanese and Korean input methods have different requirements, which is a common source of frustration why Korean IM is sometimes not working out-of-the-box.

How do Gnome users cope with these issues?

Development appears to have stalled in older IM frameworks, and thus they don't have to don't support Wayland and desktop portals. Keeping them working for a few more years might be feasible, but we might not want to focus our future efforts on them.

pshinjo added a comment.EditedJun 14 2019, 11:45 AM

I would object a stronger integration with IBus (or any of the single input method). Even within the "common" CJK input method framework, Chinese/Japanese and Korean input methods have different requirements, which is a common source of frustration why Korean IM is sometimes not working out-of-the-box.

How do Gnome users cope with these issues?

Development appears to have stalled in older IM frameworks, and thus they don't have to don't support Wayland and desktop portals. Keeping them working for a few more years might be feasible, but we might not want to focus our future efforts on them.

Some tried to fix how Mutter on Wayland behaves [4], some avoid the bug by not using IBus at all, and switch to another IM framework. IBus and Korean didn't worked well for a long time and users have reported issues dating back from 10 years ago [1]. Even earlier Qt 5 IBus module had critical bug where all the spacing of words are garbled [2]. It was also pointed out that addressing the bugs within the asynchronous architecture of IBus is not possible [3 - please look inside the comments]. The root cause of these bugs sometimes could be traced to IMs not properly handling certain user's input habits (when they change focus? did they try to explicitly commit the preedit text by pressing non-character keys? etc.), but as everybody knows, these basics are impossible to change, and more importantly, if the IM's behavior changes between versions when user tries to do the same, they will try to change IM instead.

This is also the reason why certain users want to avoid Wayland at all, as their favorite IM framework doesn't support Wayland and/or IM framework available in the Wayland is not adequate for their workflow.

[1] https://github.com/ibus/ibus/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aclosed++korean
[2] https://github.com/ibus/ibus/issues/1742
[3] https://github.com/ibus/ibus/issues/1980
[4] https://gitlab.gnome.org/GNOME/mutter/issues/152

hein added a comment.EditedJun 14 2019, 1:07 PM

The work Geon and I did in GSoC had an fcitc5 backend with plans to add a second one.

I'll say firmly though that I don't like the "let's not solve our integration problems because we need to support all the things" position. If ibus has a bug, we should try to get the bug in ibus fixed. One of the problems with the Input Method community is this pattern of "there's a minor thing about XYZ that we don't like, so instead of working together we'll do our own thing". That way nothing ever gets great in the end! There's not enough manpower in this area to spread it so thinly.

In T11054#188801, @hein wrote:

The work Geon and I did in GSoC had an fcitc5 backend with plans to add a second one.

I'll say firmly though that I don't like the "let's not solve our integration problems because we need to support all the things" position. If ibus has a bug, we should try to get the bug in ibus fixed. One of the problems with the Input Method community is this pattern of "there's a minor thing about XYZ that we don't like, so instead of working together we'll do our own thing". That way nothing ever gets great in the end! There's not enough manpower in this area to spread it so thinly.

I do agree on "we need to solve integration problem" and having a reasonable default implementation. I might be biased against IBus due to its problematic history and people calling out "IBus is buggy". What I don't agree is making IBus or any other framework as a strong dependency so users can't change their IM framework at all. I interpreted the proposed changes as eliminating this possibility.

lydia updated the task description. (Show Details)Jun 15 2019, 7:10 AM
lydia added a subscriber: lydia.Jun 15 2019, 7:26 AM

I think it'd be a good idea to shamelessly borrow content from Eike's 2017 goal proposal and integrate it here ;-)

In T11054#188801, @hein wrote:

"there's a minor thing about XYZ that we don't like, so instead of working together we'll do our own thing". That way nothing ever gets great in the end! There's not enough manpower in this area to spread it so thinly.

+ a bazillion

I would even go so far as to say this is a general problem in the FOSS world: not enough cooperation for tools that aim to solve the same problem in roughly the same way. When the available development resources are limited, all this does is produce half a dozen buggy incomplete products that are all very similar to one another, rather than one or two really good ones. It's fine (and great) to have multiple products that aim to solve a problem in fundamentally different ways or for different target users, but otherwise It's better to cooperate than fork or work separately IMO.

I think it'd be a good idea to shamelessly borrow content from Eike's 2017 goal proposal and integrate it here ;-)

Yes, let's! @hein

hein added a comment.Jun 16 2019, 12:45 PM

Currently on airports, will do from the Plasma sprint :)

ervin added a subscriber: ervin.Jun 23 2019, 1:11 PM

I think it'd be a good idea to shamelessly borrow content from Eike's 2017 goal proposal and integrate it here ;-)

+100 @hein

mart added a subscriber: mart.Jun 25 2019, 10:49 AM

+100
that's the most useful and high quality proposal i've seen so far in this year's list

hein updated the task description. (Show Details)EditedJul 3 2019, 2:32 AM
hein updated the task description. (Show Details)

I've edited the proposal, taking some of the text from my 2017 proposal but mostly trying to do a fresh take. Please join and help us edit it further!

hein updated the task description. (Show Details)Jul 3 2019, 2:34 AM
hein renamed this task from Make Input Methods Just Work to Working with Text Modern & Global: Input Methods, Emoji, Word Completion.Jul 3 2019, 4:13 AM
hein renamed this task from Working with Text Modern & Global: Input Methods, Emoji, Word Completion to Text, Modern & Global: Input Methods, Emoji, Word Completion.
hein renamed this task from Text, Modern & Global: Input Methods, Emoji, Word Completion to Text For Everyone: International Input Methods, Emoji, Word Completion.
hein renamed this task from Text For Everyone: International Input Methods, Emoji, Word Completion to Text For Everyone: International Input Methods, Emoji πŸ‰, Word Completion.Jul 3 2019, 4:16 AM
hein updated the task description. (Show Details)Jul 3 2019, 4:18 AM
hein updated the task description. (Show Details)
hein updated the task description. (Show Details)
hein updated the task description. (Show Details)Jul 3 2019, 4:22 AM
hein updated the task description. (Show Details)
hein updated the task description. (Show Details)Jul 3 2019, 4:25 AM
gpark updated the task description. (Show Details)Jul 3 2019, 5:23 AM
IlyaBizyaev updated the task description. (Show Details)Jul 7 2019, 1:11 PM
IlyaBizyaev added a subscriber: IlyaBizyaev.
kchoi awarded a token.Sat, Aug 3, 12:21 PM
kchoi added a subscriber: kchoi.
lydia added a comment.Sat, Aug 3, 3:49 PM

Are we ready here? Looking good! Do you want to move this to the next column so we can consider it ready for voting?

lydia updated the task description. (Show Details)Sat, Aug 3, 4:13 PM

I'm pretty sure this is ready!

hein added a comment.Sun, Aug 4, 4:33 PM

Yup, seems ready to move :)

lydia added a comment.Tue, Aug 13, 8:48 PM

Quick reminder that there are two days left before the voting starts. Please make any changes you still want to make soon.

Quick reminder that there are two days left before the voting starts. Please make any changes you still want to make soon.

Has voting already started? Who can vote, and how? I couldn't find an announcement on kde-community@ nor elsewhere.

alexde added a subscriber: alexde.Mon, Aug 19, 2:07 PM
alexde removed a subscriber: alexde.Mon, Aug 19, 2:11 PM
lydia added a comment.Tue, Aug 20, 5:33 PM

Sorry I ran into some issues. Working on it.

zzag added a subscriber: zzag.Fri, Aug 23, 9:29 PM

@hein @gpark Sorry for asking unrelated question, but are there some useful resources (besides Eike's Akademy talk) that one could use to learn more about IM, e.g. how exactly a client and IM daemon interact with each other, terminology (what the hell is "input method framework"), etc? Maybe something like "Input Methods for Dummies"? The whole IM thing is coated with a thick layer of mystery, confusion, and lack of documentation.

@zzag Hi, I'm not really sure if you're asking for more details about how IMEs work in general or about how IMEs work specifically in a KDE (i.e. Qt widgets et al.) context from a technical point of view. For the time being I can't really help with the latter, but I guess it won't do any harm to try and provide some more info on the former.

I'm new here so I don't know the proper etiquette, but I'm pretty sure I shouldn't be spamming the comments section with a 2000+ wall of text, so here it is in Google Docs format:

Input Methods for Dummies

Sorry I ran into some issues. Working on it.

Hello @lydia, I saw your announcement that voting has begun, but didn't receive a ballot. Should I have signed up somewhere?

lydia added a comment.Sat, Aug 24, 3:39 PM

Vote invite were sent to everyone subscribed to the KDE community mailing list and everyone with a developer account. Any contributor who didn't receive one please subscribe to the mailing list to not miss future announcements and send me a quick email (lydia@kde.org) and I'll send you a vote invite.