[kwin] Screen recording in Wayland session
Closed, ResolvedPublic

Description

We want an interface for screen recording in Wayland. Since only the compositor KWin is able to take hold of arbitrary window surfaces, we need to do it here.

Hard requirements:

  • Capture the whole screen
  • Fast and without compression loss

Soft requirements:

  • Capture single window contents or predefined sections of the screen

The hard requirements should be fulfilled by using the frame buffers in the Drm platform plugin. The soft requirement could be solved by using the KWayland buffers directly and in the second case cutting the frame buffer of the Drm platform plugin. A first patch should be confined to fulfilling the hard requirements though.

There was already an effort of tackling this task by using the Gl pipeline: D4366. But for performance reasons we decided against going further in this direction.

romangg created this task.Mar 15 2017, 8:19 PM
romangg added a comment.EditedMar 27 2017, 10:56 AM

I talked to Jonas from the Gnome project in order to, if possible, align our approaches. He currently plans on doing the following for Mutter:
‎[11:57] ‎<‎jadahl‎>‎ the (at least my) idea/plan for gnome is to use pinos instead of a pre-encoded movie file to let the screen cast application/process deal with the rest
‎[12:01] ‎<‎romangg‎>‎ jadahl: what about live streaming without saving? so currently only save to a file on disk is possible? but I assume live streaming would be possible with pinos?
‎[12:01] ‎<‎jadahl‎>‎ romangg: live streaming without saving is one of the potential new possibilites with using something like pinos
‎[12:02] ‎<‎jadahl‎>‎ i.e. you'd do processing and encoding in RAM and send directly to some stream service
‎[12:03] ‎<‎jadahl‎>‎ romangg: in short, the idea is that you get a gstreamer stream of the desktop, and you'd do whatever you please with it
‎[12:03] ‎<‎romangg‎>‎ not heard of pinos before. looks interesting.
‎[12:05] ‎<‎jadahl‎>‎ romangg: i have only prototyped with an older version in mutter, but I could use it to translate the desktop framebuffer to VNC, and all that processing was done in an external process talking to mutter via D-Bus
‎[12:05] ‎<‎jadahl‎>‎ pinos was used for pixel buffers via shmem

An old POC of Jonas for a screen recording DBus API in Mutter: https://github.com/jadahl/mutter/blob/wip/remote-desktop/src/org.gnome.Mutter.RemoteDesktop.xml

Pinos: https://cgit.freedesktop.org/~wtay/pinos

With D1231 there was already a proposed patch for a screen recording interface via a Wayland protocol extension in KWayland. The development at one point stagnated though.

The work on D1231 has been continued, what is currently expected to be the way forward.

romangg moved this task from Backlog to Under Review on the KWin board.Oct 23 2017, 7:07 PM
romangg moved this task from Backlog to Under Review on the Plasma on Wayland board.

So, I updated patches to KWin and KWayland to latest master, what are the next steps?

Did you take a look at PipeWire? Just do an assessment of feasibility in the context of your patch series. If you think that it's not feasible to adjust your patches such that PipeWire can be used for the workspace-client communication or if you don't think it's usable in Plasma at all, we can finish up your current patches so they get merged.

I'm writing on my thesis in the next month though, so I have to postpone thorough reviews until December.

Kanedias added a comment.EditedOct 29 2017, 2:33 PM

I've taken a look at PipeWire. For now this looks like Wayland for media-streams to me. Globals, resources, proxies, registry, etc. etc. just with pw_ prefix instead of wl_.
Streams expect to get video buffers as bytes.

Speaking of GNOME, main Mutter patch is here. It calls clutter_stage_capture_into to catch a frame which in turn reads pixels from screen and then passes them.

So, we can pass video buffer objects as file descriptors to KDE Daemon via RemoteAccess protocol. Then it can extract byte data from BOs and pass it via PipeWire to whoever needs them.
How does it sound to you?

romangg added a comment.EditedOct 29 2017, 3:14 PM

And the buffer objects would be the GBM buffer handles sent to the KDE Daemon via your Wayland protocol extension RemoteAccess as in D1231, correct? So there shouldn't be much rewrite needed (or none at all) in your KWin and KWayland patches. Sounds nice.

Also if you don't make your RemoteAccess protocol exclusively usable by Plasmashell right now, applications could decide to directly use RemoteAccess instead of the daemon with PipeWire. Not sure how sensible allowing this would be in the long run, but at the beginning it makes sense to not confine ourselves to only being reachable by PipeWire and letting an application decide if it wants to use the daemon and PipeWire or directly get the GBM handles and doing the Gl convert itself (although we should probably encourage PipeWire usage as long as we don't find something intrinsically wrong with it).

If you don't make the protocol exclusive you could also leave the patches as they are, add another one for the daemon on top of it and a second backend in krfb to either use directly GBM buffers via your Wayland protocol or your daemon together with PipeWire. This way in krfb we could compare both solutions.

Kanedias added a comment.EditedOct 29 2017, 5:36 PM

And the buffer objects would be the GBM buffer handles sent to the KDE Daemon via your Wayland protocol extension RemoteAccess as in D1231, correct?

Yep, that's the plan.

exclusively usable by Plasmashell

Interesting point. I'm not against confining scope a bit, because lack of auth in screen capture protocol seems a bit... troubling.
How do I do that? Are there any guidelines on how to make KWayland protocols internal to Plasma?

This way in krfb we could compare both solutions.

I was planning to revive my old KRfb patches once this gets merged, so yep, first backend is already half-done.

How do I do that? Are there any guidelines on how to make KWayland protocols internal to Plasma?

The plasma window management protocol is internal to Plasma by only allowing one client at a time to bind it (which is then plasmashell as the earliest client after startup to try that).

Did you make some progress regarding the daemon/PipeWire @Kanedias?

Hi. I updated my patchsets to KWin/KWayland to latest master. It's hard to maintain more than one patchset, so I'm waiting for these to be merged to continue work on KRfb or Pipewire daemon.

romangg closed this task as Resolved.Mar 25 2018, 8:26 PM
romangg updated the task description. (Show Details)
romangg moved this task from Under Review to Done on the Plasma on Wayland board.Mar 25 2018, 8:29 PM

Hard requirements have been met by the two patches. Soft requirement was dropped, something to look into in the future.

romangg moved this task from Under Review to Done on the KWin board.Mar 25 2018, 8:29 PM