Improve Thumbnail Aside effect
Closed, WontfixPublic

Description

Various other operating systems offer a Picture-in-Picture effect, and I was under the impression that we didn't--but in fact we have one too! It's the "Thumbnail Aside" effect, which is off by default. It just needs some love to make it competitive and discoverable. Proposals:

  • Rename to "Picture-in-Picture" (that's what it's called everywhere else)
  • Improve default settings: 100% opacity and ≥300px wide
  • When activated, allow the PIP window to be moved, resized, and closed with the mouse
  • When the effect is enabled, make it discoverable by automatically adding a window decoration button to window titlebars that will activate the effect for that window when clicked
  • Enable it by default (after all other changes are made)

With those changes, I think KWin's PIP effect could compete with the best of them. :)

Then, the promo team could promote the heck out of it so that people don't have the impression that we're lacking this feature anymore.

ngraham created this task.Nov 6 2018, 4:12 PM
ngraham triaged this task as Wishlist priority.
abetts added a subscriber: abetts.Nov 6 2018, 4:33 PM

I agree with all the changes. PIP naming is also much more recognizable.

zzag added a comment.Nov 6 2018, 6:23 PM

In general, I like the idea.

I'm afraid that until KWin has QML based effects API, such effect won't be implemented.

Thanks Vlad. Correct me if I'm wrong, but there are some things we could trivially change right now: the name, default settings, and whether or not it's enabled by default. What are the technical barriers to allowing mouse interaction and dynamically showing a titlebar button?

zzag added a comment.Nov 6 2018, 7:11 PM

the name, default settings, and whether or not it's enabled by default

IMHO, It won't change things too much, that still will be Thumbnails Aside, only with a different name.

What are the technical barriers to allowing mouse interaction

As far as I know, effects can't redirect mouse input right now.

Another issue is how we have to render "pictures". For example, if you play a video without decoration/frame, then we /probably/ need to draw shadows, otherwise it won't look good. Currently, if you try to do that in C++, I bet you go crazy.

The QML API would let us to implement such effects with no effort.

and dynamically showing a titlebar button?

I'm not sure what you mean with this.

zzag added a comment.Nov 6 2018, 7:13 PM

IMHO, It won't change things too much, that still will be Thumbnails Aside, only with a different name.

Also, it would confuse users because they would expect typical PiP.

zzag added a comment.EditedNov 6 2018, 7:19 PM

Also, I have a question: such effect would be most likely useful only for videos. How will KWin detect them?

FWIW, Google Chrome supports PiP on Linux.

zzag added a comment.Nov 6 2018, 7:56 PM

FWIW, Google Chrome supports PiP on Linux.

What is a "typical PiP"?

zzag added a comment.Nov 7 2018, 10:04 AM

What is a "typical PiP"?

The one that you can move, resize, and interact with mouse (e.g. press the pause button).

Even if you can't interact with the content, isn't is still potentially useful for other reasons?

Even if you can't interact with the content, isn't is still potentially useful for other reasons?

Can you please give me an example where such feature would be useful besides watching videos? (As far as I know, the sole purpose of PiP is to watch videos on top of other windows)

Frankly, I don't think that KWin is a good place where such a feature should be implemented. Just to be clear, I like the idea, but not how it should be implemented. At least, there should be some standardized way to create pictures-in-pictures, otherwise if you have several tabs in the browser, then everything will be messed up.

Anyway, that's not up to one man decide whether such feature should be implemented so I'm CC'ing other KWin folks.

If there's a better way to support watching videos in a little frame in the corner, I'm all for it. Alas, I lack the technical understanding to imagine how that would be implemented in an optimal and video-source-agnostic manner.

paulb added a subscriber: paulb.Nov 7 2018, 9:10 PM

Can you please give me an example where such feature would be useful besides watching videos? (As far as I know, the sole purpose of PiP is to watch videos on top of other windows)

Follow a long shell process, such s as a compile or dd copy out of the corner of your eye. I have used it recently to follow a long AUR installation and have known when to switch to konsole when I needed to input my user password.

zzag added a comment.EditedNov 7 2018, 9:18 PM
In T9996#166680, @paulb wrote:

Follow a long shell process, such s as a compile or dd copy out of the corner of your eye. I have used it recently to follow a long AUR installation and have known when to switch to konsole when I needed to input my user password.

This argument is not convincing (reading text at small scale is really hard).

paulb added a comment.Nov 7 2018, 9:44 PM

Oh. I don't read it. I just see out of the corner of my eye when it stops scrolling. Then I know I have to switch over and check if it has bombed out our needs an input.

zzag added a comment.EditedNov 7 2018, 9:55 PM
In T9996#166683, @paulb wrote:

Oh. I don't read it. I just see out of the corner of my eye when it stops scrolling. Then I know I have to switch over and check if it has bombed out our needs an input.

The existing Thumbnail Aside effect could be used for such tasks, but I think there are other better ways to get notified when a long running command is completed.

paulb added a comment.Nov 7 2018, 10:13 PM

Maybe. There are always multiple ways of doing lots of things. But that was not your question.

I think you can look at how it was implemented in gala: https://github.com/elementary/gala
In their implementation, PiP simply takes a window or part of a window and makes a preview that can be moved and closed, without mouse interaction support. I think this is quite enough.

zzag added a comment.Dec 15 2018, 8:17 AM

I think you can look at how it was implemented in gala: https://github.com/elementary/gala
In their implementation, PiP simply takes a window or part of a window and makes a preview that can be moved and closed, without mouse interaction support. I think this is quite enough.

The way it's implemented in Gala is not quite correct. What if you switch tabs in the web browser?

In T9996#170223, @zzag wrote:

I think you can look at how it was implemented in gala: https://github.com/elementary/gala
In their implementation, PiP simply takes a window or part of a window and makes a preview that can be moved and closed, without mouse interaction support. I think this is quite enough.

The way it's implemented in Gala is not quite correct. What if you switch tabs in the web browser?

I just put the tab in a separate window. I don't think that the window manager can somehow get around this without a browser plugin. But it's not much needed, this is quite an obvious behavior, that the so-made PiP will change content when the tab is switched.

zzag added a comment.Dec 15 2018, 8:23 AM

Anyway, I suggest to close this task. Compositor is not a good place to implement such a feature.

zzag added a comment.Dec 15 2018, 8:27 AM

I just put the tab in a separate window.

Seems like a workaround.

this is quite an obvious behavior, that the so-made PiP will change content when the tab is switched.

No, it's not.

I think I agree with you that a "real" PiP feature should be implemented in another way. But even aside from that, I think that the other ideas expressed in this task make sense and would improve the utility of the effect for other uses too. If there are no objections, I can begin to submit patches to, for example, tweak the default settings to make it more useful without needing to adjust it first.

zzag added a comment.Dec 16 2018, 7:12 PM

I think I agree with you that a "real" PiP feature should be implemented in another way. But even aside from that, I think that the other ideas expressed in this task make sense and would improve the utility of the effect for other uses too. If there are no objections, I can begin to submit patches to, for example, tweak the default settings to make it more useful without needing to adjust it first.

Personally, I would like to drop this effect. I think we have it just to show our compositing capabilities.

Regarding the defaults, I don't think that will improve something, for example, thumbnails can't "eat" input events so they [thumbnails] have to be translucent.

And we can't the thumbnail consume mouse events?

In T9996#170225, @zzag wrote:

Anyway, I suggest to close this task. Compositor is not a good place to implement such a feature.

Where is a good place then? The composer can do this with any window, and all applications will never do such a function. This is a very cool feature, even with such flaws as switching tabs in the browser and you should not completely abandon the idea because of this. Moreover, I now switched to using gala with KDE only because of this feature about a month ago, because I really did not have enough of it before, so I really would like to see it in my favorite kwin.

zzag added a comment.Dec 17 2018, 1:35 PM

And we can't the thumbnail consume mouse events?

No, currently, they can't.

Where is a good place then? The composer can do this with any window, and all applications will never do such a function. This is a very cool feature, even with such flaws as switching tabs in the browser and you should not completely abandon the idea because of this. Moreover, I now switched to using gala with KDE only because of this feature about a month ago, because I really did not have enough of it before, so I really would like to see it in my favorite kwin.

Applications should implement this feature (one such example is Chromium/Google Chrome). For what it's worth, with applications like mpv or vlc, PiP doesn't make any sense (at least on desktop).

In T9996#170387, @zzag wrote:

Applications should implement this feature (one such example is Chromium/Google Chrome). For what it's worth, with applications like mpv or vlc, PiP doesn't make any sense (at least on desktop).

I use PiP with vlc - it is very convenient.

zzag added a comment.EditedDec 17 2018, 1:56 PM

I use PiP with vlc - it is very convenient.

I wouldn't say that. Are window decorations really that big so you need PiP? If you want to have PiP-ified vlc, then change the keep above state and remove borders, that's all.

ilya-fedin added a comment.EditedDec 17 2018, 2:00 PM
In T9996#170389, @zzag wrote:

I use PiP with vlc - it is very convenient.

I wouldn't say that. Are window decorations really that big so you need PiP? If you want to have PiP-ified vlc, then change the keep above state and remove borders, that's all.

It's not about decorations, but about the convenience of management: resizing, moving and closing the PiP.

zzag added a comment.Dec 17 2018, 2:07 PM

It's not about the scenery, but about the convenience of management: resizing, moving and closing the PiP.

You can use alt+lmb/rmb for that. But even without that, PiP for vlc/mpv/ffplay/etc doesn't make sense, it's overkill.

In T9996#170391, @zzag wrote:

It's not about the scenery, but about the convenience of management: resizing, moving and closing the PiP.

You can use alt+lmb/rmb for that. But even without that, PiP for vlc/mpv/ffplay/etc doesn't make sense, it's overkill.

Do you understand the word "convenience"? Opening menu and adjusting "on top of all windows" and decorations, as well as moving/resizing through key shortcuts is inconvenient, but PiP in gala implementation is convenient.
If you want to improve the user experience, then you shouldn't reject ideas just because it looks like the result of several existing functions.

zzag added a comment.Dec 17 2018, 2:37 PM

If you want to improve the user experience, then you shouldn't reject ideas just because it looks like the result of several existing functions.

All what I wanted to say is that you don't need PiP for video players. Anyway, as I said earlier, the compositor is not a good place to implement such a feature.

In T9996#170395, @zzag wrote:

If you want to improve the user experience, then you shouldn't reject ideas just because it looks like the result of several existing functions.

All what I wanted to say is that you don't need PiP for video players. Anyway, as I said earlier, the compositor is not a good place to implement such a feature.

Why is compositor not a good place to implement this feature? In my opinion, just the opposite: PiP, working with any program is very convenient, I understood it with gala.

So reading all this, I'm rather lost.

The opening paragraph is basically "other people have $x. We should copy it".

The problem with that is:

  1. It doesn't explain the problem's we're trying to solve.
  2. At best we get a clone.

And now we have to backtrack to a point where people are on the same page.

So as I understand the main use case for videos: You're watching a video whilst you get on with some other tasks. You want a smaller version always on top.
Given you'd also be previewing unusable controls this sounds we're trying to optimise a hack.

If we are going to change it so that we large thumbnails, resizing + moving and full opacity, what's the advantage over just resizing the window and marking it always on top? That could be wrapped up in a kwin script with a shortcut that resizes back on focus if that's the actual goal.


A relevant thing to bring up was there was a concept a while ago for more relevant thumbnails for the taskbar; the problem to be solved was that on wayland we can't just grab the windows, we need to do something special for the taskmanager we may as well do something good.

Thumbnail aside also appears to be pretty buggy as-is.

Fixing that is obviously a pre-requisite to any changes - and something we should look at doing given we offer it.

Given you'd also be previewing unusable controls this sounds we're trying to optimise a hack.

In most cases, controls can be hidden.

If we are going to change it so that we large thumbnails, resizing + moving and full opacity, what's the advantage over just resizing the window and marking it always on top? That could be wrapped up in a kwin script with a shortcut that resizes back on focus if that's the actual goal.

Convenient control + ability to make PiP from part of the window

zzag closed this task as Wontfix.Jan 23 2019, 9:29 PM

As I said earlier, the compositor is not a good place to implement such a feature.


Fixing that is obviously a pre-requisite to any changes - and something we should look at doing given we offer it.

I suggest to drop it.

kives added a subscriber: kives.Jan 25 2019, 4:28 AM

I don't agree with @zzag or @davidedmundson here, despite how much I love the changes they make to KDE and their work overall. The functionality is clearly desired based on the Reddit thread that brought this post back to life: https://www.reddit.com/r/kde/comments/ajeume/are_there_any_plans_to_improve_pip/

The workarounds right now aren't a good solution either. Just because one app (Chrome) has added support for a specific use case doesn't mean the entire concept should be abandoned. Also, if not done in the compositor where else would it be done? A regular application trying to grab the contents of minimized/off-screen windows likely will be much worse, whereas the compositor can correctly do these things.

This can be useful for more than just video streaming, such as monitoring a chat window, pizza delivery status, pretty much any use case where you would want to not have a full interface in front of you but want to look at it from your peripheral vision.

I would also point out I don't agree with the idea that we would only "clone" the functionality or that some other operating distro defines what Picture in Picture (PiP) means. It's a common-place term used in TVs, DVRs for over 40 years and the public very much understands the basic idea.

If we are going to change it so that we large thumbnails, resizing + moving and full opacity, what's the advantage over just resizing the window and marking it always on top? That could be wrapped up in a kwin script with a shortcut that resizes back on focus if that's the actual goal.

That sounds interesting. Could the KWin script ask the user to select a rectangle area and then do this within it? I currently use this workaround you mention but it has some irritations because not all content can neatly represent itself in a window without padding, borders or making a video much smaller because it thinks it's on a mobile screen.

I fully agree with Vlad and David: this is not a feature belonging into the compositor and we should not drive by trying to copy features (badly). KWin's mission statement discourages to do so. We had huge problems in the past trying to please too many and failing.

I'm in favor of removing the thumbnail aside effect.

@kives
To be clear, you can't "disagree" with my position as I don't have one. I've merely asked what it's for so we can come up with any solutions that actually deliver on what's actually wanted.

Also, if not done in the compositor where else would it be done?

https://api.kde.org/frameworks/plasma-framework/html/classPlasma_1_1WindowThumbnail.html

Same as how the taskbar previews works. I don't see how it's worse.
The minimized problem applies to both.

kives added a comment.Jan 25 2019, 8:16 AM

Thanks @davidedmundson I'll take a look.