My suggestion in T5653 was to use a KDE daemon to proxy client requests and the conversion of GBM buffers to buffers suitable for PipeWire, but @davidedmundson said that he would prefer a process spawned by KWin for that. This would have security advantages in his opinion.

So the basic flow would then be:

• KWin spawns on launch a separate process, that listens to client requests over D-Bus (this interface is not shared with GNOME, but can maybe be aligned with it).
• When at least one client requests screen capture, the process asks KWin for GBM buffers via the Wayland interface from D1231.
• KWin delivers them via the same interface to the process.
• The process does a conversion to PipeWire ready buffers and sends them to all interested clients via PipeWire. The process does this as long as at least one client is still interested in screen capture.

Yeah, sounds good to me. I had some concerns with kded as it's already quite heavy with features.
I'll try to come up with smth in a week or so.

@Kanedias if you can make a small test app that acts as a client side and fetches a buffer (what you already have in krdc, but standalone) and put that somewhere, that would be awesome.
Then maybe I can hook up DBus and possibly pipewire from there. It'll split the workload.

@davidedmundson , I started to hack it together at https://gitlab.com/Kanedias/kwin-remote-receiver

Still some errors to be addressed, hope to do it today

• Bind Output interface so that RemoteAccess will really send us something
• Investigate GBM error:

amdgpu_device_initialize: amdgpu_query_info(ACCEL_WORKING) failed (-13)
amdgpu: amdgpu_device_initialize failed.
do_winsys_init: DRM version is 3.19.0 but this driver is only compatible with 2.12.0 (kernel 3.2) or later.

How to build it:

1. Apply KWayland and KWin patches (better use branch gbm-vnc)
2. Install KWayland to /usr (KWin finds KF5 in /usr, stops searching for KWayland in prefix)
3. Install KWin to the prefix
4. In VT do the following

$ eval $(dbus-launch)
$ export QT_LOGGING_RULES='*.debug=true;*.warn=true;*.critical=true'
$ export QT_QPA_PLATFORM=wayland
$ export KWIN_REMOTE=1
$ ./kwin_wayland --drm --xwayland --exit-with-session konsole &> kwin.out
$ # launch ./kwrr from the opened terminal

Got buffer passing working, the only remaining piece is GBM init, it fails with this:

amdgpu_device_initialize: amdgpu_query_info(ACCEL_WORKING) failed (-13)
amdgpu: amdgpu_device_initialize failed.

in strace on

ioctl(6, DRM_IOCTL_AMDGPU_INFO or DRM_IOCTL_SIS_FB_FREE, 0x7fffb69fca40) = -1 EACCES (Permission denied)
write(2, "amdgpu_device_initialize: amdgpu"..., 72amdgpu_device_initialize: amdgpu_query_info(ACCEL_WORKING) failed (-13)
) = 72

Yeah... it works! Fixed last error by using render node instead of messing with DRM Master.

Tested on amdgpu Radeon RX480, Mesa 17.3.
@davidedmundson : latest source code

I'll try adding PipeWire interface this weekend if nobody snatches this fruit before ;)

Is someone already doing anything on top of this?
I'm about to start hooking up PipeWire, post here in case something's just around the corner.

Not yet, but I plan to look into it tomorrow and attend PipeWire hackfest on tuesday so there might be some progress from my side.

Wow, we definitely need some kind of C++ wrapper around pipewire... jumping around with C-style linked maps and kernel-like arrays doesn't feel good at all

I need to work on some other stuff until FOSDEM, but I'm definitely following your progress and I hope we can maybe afterwards come up with some meeting on IRC or a video call where @jgrulich can give a small report regarding the PipeWire hackfest.

In T7785#125343, @Kanedias wrote:

Wow, we definitely need some kind of C++ wrapper around pipewire...

This would be a question @jgrulich could ask at the hackfest. Maybe there is already a lib to provide something like this planned by the PipeWire devs.

In T7785#125343, @Kanedias wrote:

Wow, we definitely need some kind of C++ wrapper around pipewire... jumping around with C-style linked maps and kernel-like arrays doesn't feel good at all

I don't honesly think that Wim (= PipeWire devs) will work on something like this, there is plenty of work on PipeWire missing and given that people around are Gnome devs, there is no need for a C++ wrapper, at least for them. We can then have a meeting on IRC if there will be anything to share or I can send it here.

I attended a small hackfest we had with Wim, Firefox guys and a guy from our team who will add Pipewire support to webrtc (for browser support). Wim told us what is the best approach to support Pipewire. Up until now Gnome applications used some Mutter interface, returning a file descriptor which then they used through Pipewire and get the content. Now there is support for this in xdg-desktop-portal, which was intended to be used for sandboxed apps but there is plan to use it even outside sandbox. This means that for applications we can use org.freedesktop.portal.ScreenCast interface and with this it will work for sandboxed and non-sandboxed applications. On the otherside to have support for KWin → Pipewire we need to implement backend part for xdg-desktop-portal, specifically in xdg-desktop-portal-kde. Also applications using Pipewire directly (not using xdg-desktop-portal) will work as well, because there is plan for Pipewire to internally call the portal if user specifies to Pipewire that he has interest in screencast. For the backend we have to implement similar interface. I already started working on this. My plan right now is to add this, write a simple example which will pass some random video (there is an example in pipewire) to test the portal backend and add also a test to my xdg-portal-test-kde application to attempt to consume a stream using this portal. Once this is done and working we can attempt to get the real screen content as it's not possible to do this when you have nothing to test this with.

Result of this should be a consistant way how to support screencasting, which will work for browsers and all applications using Pipewire directly or using xdg-desktop-portal. One solution for all.

The procedure as I see it:

1. KWin starts kwin-remote-receiver utility
2. KWRR obtains GBM fds from KWin and converts them to frames via DRM render nodes
3. KWRR initiates pw_remote call which connects to pipewire-0 socket and starts pipewire daemon server
4. KWRR connects to pipewire server registering as a source node and starts listening
5. xdg-desktop-portal-kde or clients themselves connect to pipewire daemon, get fd from it and start receiving frames from KWRR node

Is that right?

That's pretty much how I envisioned it.

Only potential slight thing I'm not sure about is:

Do we start KWRR on startup listening but idling, and then have clients talk to it

or

clients query kwin, kwin spawns KWRR on demand, gets an FD and returns that to the client

Latter has its advantages as you'd have one KWRR per client keeping it simpler and it keeps the auth in one place.

Thanks for that source sample. Looks exactly what I was after.

Personally I'd just use low level C, writing a wrapper means you have to maintain a whole wrapper..and have the code you need to write anyway and it's a nice modern C lib.

I might have some time this weekend, but maybe you'll beat me.

@davidedmundson ,

clients query kwin, kwin spawns KWRR on demand, gets an FD and returns that to the client

I'd go with this option but not sure how well does it fit the flow. Clients need to query KWin somehow but all they should see is DBus or Pipewire daemon
Right now I start KWRR manually, didn't integrate it with KWin codebase yet.

Personally I'd just use low level C

Yeah, that's what I'm doing right now, mainly copy-pasting from pipewire/examples folder ;)

I might have some time this weekend, but maybe you'll beat me.

I certainly won't as I'll be on vacation traveling to Paris with my wife :)
But will try to get KWRR -> PipeWire part working until then. For now it lives in pipewire branch

Doing KWin → (KWRR → Pipewire) → xdg-desktop-portal-kde should work, problem is that from xdg-desktop-portal-kde you still need a way how to tell kwin what window/screen we need. In xdg-desktop-portal-gtk (gnome) they use org.gnome.Mutter.ScreenCast interface, which is similar to org.freedesktop.Portal.ScreenCast interface (obviously as mutter had this first and then they moved it to xdg-desktop-portal) and that interfaces returns fd. With this approach, which means having KWRR, we either need a way how to talk to KWRR and ask for what we want or directly talk to Kwin, which will ask KWRR do the pipewire stuff and KWRR returns fd back to KWin and Kwin returns it back to the client or xdg-desktop-portal-kde.

Guys, in case everyone misses me I'm on vacation right now, will be back by weekend.

Yup, I'll be using raw C Pipewire headers. Compiling with -fpermissive gives me some criiiiinge but nothing to worry about :)

I have all the portal logic done, if you want to look at it I have it in a branch "screencast" in xdg-desktop-portal-kde. Only missing part is return Pipewire node_id back through the portal, for that we need the daemon done. There is also a minor blocker for us in Pipewire (master).

Wim already fixed issues regarding Pipewire and C++ in master, we can now build it without "-fpermissive".

@jgrulich , I think I found a way to make KWin send only needed buffers to support >1 monitor setup.

The remote access protocol sends buffers for all outputs bound by the client.
So instead of this:

connect(m_registry, &KWayland::Client::Registry::outputAnnounced, this, [this] (quint32 name, quint32 version) {
    qCDebug(XdgDesktopPortalKdeScreenCast) << "Binding output";

    if (!m_output) {
        m_output = m_registry->createOutput(name, version);
        connect(m_output, &KWayland::Client::Output::changed, this, [this] {
            qCDebug(XdgDesktopPortalKdeScreenCast) << "Output bound:" << m_output->manufacturer() << m_output->model();
        });
    }
});

you can wait for all outputs to be announced and collect them.
Then, after the user confirms its selection you bind the specified output and kwin starts sending you buffers associated with it.

@jgrulich , were you able to connect the stream to pipewire with the code from xdg-desktop-portal-kde?
It says only "Stream stage changed: connecting" and then "protocol-native: got connection error" to me.

I'll try to redo it like export-source.c does meanwhile...

@Kanedias, yes, I can connect with that code, you need pipewire running to be able to connect. I use it the same way mutter uses pipewire.

Ok, guess I'm missing something then, I'll take a closer look, thanks!

In T7785#128972, @Kanedias wrote:

@jgrulich , I think I found a way to make KWin send only needed buffers to support >1 monitor setup.

The remote access protocol sends buffers for all outputs bound by the client.
So instead of this:

connect(m_registry, &KWayland::Client::Registry::outputAnnounced, this, [this] (quint32 name, quint32 version) {
    qCDebug(XdgDesktopPortalKdeScreenCast) << "Binding output";
 
    if (!m_output) {
        m_output = m_registry->createOutput(name, version);
        connect(m_output, &KWayland::Client::Output::changed, this, [this] {
            qCDebug(XdgDesktopPortalKdeScreenCast) << "Output bound:" << m_output->manufacturer() << m_output->model();
        });
    }
});

you can wait for all outputs to be announced and collect them.
Then, after the user confirms its selection you bind the specified output and kwin starts sending you buffers associated with it.

Btw. this makes sense and should work in my opinion. Once I find a way how to pass buffers to pipewire streams I will change it to get screen cast for specified screen.

I have been finally able to stream data from KWin through xdg-desktop-portal-kde and receive them with xdg-portal-test-kde. There are still issues like to select just one screen which should be recorder or initialize pipewire stream based on screen resolution. Right now it streams content from all screens and has hardcoded resolution to 1920x1080, but basically it works. I will keep polishing that and implement missing functionality tomorrow or so.

Here is a prove

@jgrulich hooray!

I'll be researching crashes in GBM handlers you mentioned then.

Can't make it crash, unfortunately.
@jgrulich , seems I need some cooperation to make it reliably fail, let's compare our specs.

Desktop:

System:    Host: APOSIMZ Kernel: 4.15.3-2-ARCH x86_64 bits: 64 Desktop: KDE Plasma 5.12.2 Distro: Arch Linux
Machine:   Device: desktop Mobo: ASRock model: X370 Gaming K4 serial: N/A UEFI: American Megatrends v: P3.40 date: 11/07/2017
CPU:       8 core AMD Ryzen 7 1700X Eight-Core (-MT-MCP-) speed/max: 1742/3400 MHz
Graphics:  Card: Advanced Micro Devices [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580]
           Display Server: x11 (X.Org 1.19.6 ) drivers: modesetting,ati (unloaded: fbdev,vesa,radeon)
           Resolution: 2560x1440@59.95hz
           OpenGL: renderer: AMD Radeon RX 480 Graphics (POLARIS10 / DRM 3.23.0 / 4.15.3-2-ARCH, LLVM 5.0.1)
           version: 4.5 Mesa 17.3.5

Laptop:

System:    Host: BENISUZUME Kernel: 4.15.5-1-ARCH x86_64 bits: 64 Desktop: KDE Plasma 5.12.2 Distro: Arch Linux
Machine:   Device: laptop System: ASUSTeK product: UX305CA v: 1.0 serial: N/A
           Mobo: ASUSTeK model: UX305CA v: 1.0 serial: N/A
           UEFI: American Megatrends v: UX305CA.300 date: 06/08/2016
Battery    BAT0: charge: 30.1 Wh 76.1% condition: 39.5/45.0 Wh (88%)
CPU:       Dual core Intel Core m3-6Y30 (-MT-MCP-) speed/max: 600/2200 MHz
Graphics:  Card: Intel HD Graphics 515
           Display Server: x11 (X.Org 1.19.6 ) drivers: modesetting (unloaded: fbdev,vesa)
           Resolution: 3200x1800@60.00hz
           OpenGL: renderer: Mesa DRI Intel HD Graphics 515 (Skylake GT2) version: 4.5 Mesa 17.3.5

Steps I've tried (on both desktop and laptop):

• Launch kwin_wayland, launch konsole, launch kwrr, get buffers
• Launch kwin_wayland, launch konsole, switch to Xorg session, launch kwrr there, get buffers, switch VTs back and forth
• Variations of the above using multiple instances of kwrr
• Variations of the above, interrupting kwrr, starting it again

Can you share your specs and steps?

pw_remote_destroy - calls free(remote->error) even if it's not initialized, is this a bug?..

This is setup I use where the crash can be reproduced:

System:    Host: localhost.localdomain Kernel: 4.15.4-300.fc27.x86_64 x86_64 bits: 64 gcc: 7.3.1
           Desktop: KDE Plasma 5.12.2 (Qt 5.9.4) Distro: Fedora release 27 (Twenty Seven)
Machine:   Device: laptop System: LENOVO product: 20EGS0R60G v: ThinkPad W541 serial: N/A
           Mobo: LENOVO model: 20EGS0R60G serial: N/A UEFI [Legacy]: LENOVO v: GNET85WW (2.33 ) date: 12/07/2017
Battery    BAT0: charge: 87.9 Wh 100.0% condition: 87.9/99.5 Wh (88%) model: SANYO 45N1779 status: Full
CPU:       Quad core Intel Core i7-4810MQ (-MT-MCP-) arch: Haswell rev.3 cache: 6144 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 22348
           clock speeds: max: 3800 MHz 1: 2879 MHz 2: 2738 MHz 3: 2367 MHz 4: 3491 MHz 5: 3039 MHz 6: 2480 MHz
           7: 3277 MHz 8: 2736 MHz
Memory:    Using dmidecode: root required for dmidecode
Graphics:  Card-1: Intel 4th Gen Core Processor Integrated Graphics Controller bus-ID: 00:02.0
           Card-2: NVIDIA GK107GLM [Quadro K1100M] bus-ID: 01:00.0
           Display Server: wayland (X.org 119.6 ) drivers: modesetting (unloaded: fbdev,vesa)
           Resolution: 1920x1080@59.96hz, 1920x1200@59.88hz, 1920x1080@59.96hz
           OpenGL: renderer: Mesa DRI Intel Haswell Mobile version: 4.5 Mesa 17.3.5 Direct Render: Yes

And this is what I have in my older laptop where I cannot reproduce it

System:    Host: localhost.localdomain Kernel: 4.16.0-0.rc2.git0.1.fc28.x86_64 x86_64 bits: 64 gcc: 8.0.1
           Desktop: N/A Distro: Fedora release 28 (Twenty Eight)
Machine:   Device: laptop System: LENOVO product: 4174BH4 v: ThinkPad T420s serial: N/A
           Mobo: LENOVO model: 4174BH4 serial: N/A UEFI [Legacy]: LENOVO v: 8CET51WW (1.31 ) date: 11/29/2011
Battery    BAT0: charge: 6.0 Wh 100.0% condition: 6.0/44.0 Wh (14%) model: SANYO 42T4845 status: Full
CPU:       Dual core Intel Core i5-2540M (-MT-MCP-) arch: Sandy Bridge rev.7 cache: 3072 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 10365
           clock speeds: max: 3300 MHz 1: 819 MHz 2: 831 MHz 3: 828 MHz 4: 1084 MHz
Memory:    Using dmidecode: root required for dmidecode
Graphics:  Card: Intel 2nd Generation Core Processor Family Integrated Graphics Controller bus-ID: 00:02.0
           Display Server: wayland (X.org 119.6 ) drivers: modesetting (unloaded: fbdev,vesa)
           Resolution: 1600x900@59.95hz
           OpenGL: renderer: Mesa DRI Intel Sandybridge Mobile version: 3.3 Mesa 18.0.0-rc4 Direct Render: Yes

AFAIK Mesa 17.3.x had a regression that caused lockups on intel iGPUs. Many of them were finally nailed down only in 17.3.6, see here
Can you try new version on your Fedora 27 laptop?

Next thing I'd be suspicious about is two GPUs, didn't have this setup.
Also, I really hope card KWin takes from logind and /dev/dri/renderD128 belong to the same GPU as for now this render node is hardcoded in screencasting impl.

Yep, I managed to get remote stream too, though it doesn't look so good as yours yet, Jan :)
I'll be pushing changes with removing GSource and rewriting it with QSocketNotifier to kwin-remote-receiver, you can take a look if interested.

I think I know what's the problem with delete output. In KWayland in output.cpp: WaylandPointer<wl_output, wl_output_destroy> output. While this is technically correct it shouldn't destroy output right away.
It should call wl_output_release that will notify destruction to the server so it will know this client has no more bound outputs and RemoteAccess interface and stop sending buffers.

@graesslin can you shed some light on this?

In T7785#130844, @Kanedias wrote:

AFAIK Mesa 17.3.x had a regression that caused lockups on intel iGPUs. Many of them were finally nailed down only in 17.3.6, see here
Can you try new version on your Fedora 27 laptop?

I tried with Mesa 17.3.6 which landed into updates yesterday but the result is still same.

Next thing I'd be suspicious about is two GPUs, didn't have this setup.

I think that the second graphics card is not used and active, that shouldn't be problem.

Also, I really hope card KWin takes from logind and /dev/dri/renderD128 belong to the same GPU as for now this render node is hardcoded in screencasting impl.

How do I find out which card KWin takes? That render node should belong to Intel graphics card

[jgrulich@localhost ~]$ vainfo --display drm --device /dev/dri/renderD128
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
failed to open /usr/lib64/dri/hybrid_drv_video.so
Failed to wrapper hybrid_drv_video.so
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: Intel i965 driver for Intel(R) Haswell Mobile - 1.8.3

@jgrulich For situation where deleting KWayland::Client::Output doesn't help - please see D10929

Try doing it with this patch, helped on my side.

@romangg , @davidedmundson , @graesslin , should we merge D1230 & D1231? D10965 for xdg-desktop-portal depends on this code.

So for an overview we now have:

• D1230, that enables GBM buffer passing in KWin,
• D1231, that implements a remote access interface in KWayland,
• D10965, that enables xdg-desktop-portal-kde to talk to KWin via the remote access interface in KWayland.

So an application is now supposed to talk via DBus to xdg-desktop-portal to initiate a video transfer and the backend code in xdg-desktop-portal-kde automatically takes up querying KWin for the GBM buffers, converts them to Gl (what process does this for xdg-desktop-portal-kde?) and sends them via PipeWire to the application. Is this correct?

Three more questions:

• We don't need the kwin-remote-receiver utility anymore because this is in xdg-d-p-kde code, yes?
• Is xdg-d-p-kde only for video in this regard or can it also be a proxy for input (as in remote access)?
• Normal applications not in containers can use the xdg-d-p-kde interfaces as well, yes?

In T7785#131835, @romangg wrote:

So for an overview we now have:

• D1230, that enables GBM buffer passing in KWin,
• D1231, that implements a remote access interface in KWayland,
• D10965, that enables xdg-desktop-portal-kde to talk to KWin via the remote access interface in KWayland.
  
  So an application is now supposed to talk via DBus to xdg-desktop-portal to initiate a video transfer and the backend code in xdg-desktop-portal-kde automatically takes up querying KWin for the GBM buffers, converts them to Gl (what process does this for xdg-desktop-portal-kde?) and sends them via PipeWire to the application. Is this correct?

Correct.

Three more questions:

• We don't need the kwin-remote-receiver utility anymore because this is in xdg-d-p-kde code, yes?

Shouldn't be needed, unless you want to avoid using xdg-d-p-kde.

• Is xdg-d-p-kde only for video in this regard or can it also be a proxy for input (as in remote access)?

For remote access there is remote desktop portal, I proposed it to GSoC as a task.

GSoC task: https://community.kde.org/GSoC/2018/Ideas#Project:_Remote_desktop_portal

• Normal applications not in containers can use the xdg-d-p-kde interfaces as well, yes?

Yes, this is supposed to work for both sandboxed and non sandboxed applications. Also if an app is using Pipewire directly, then Pipewire will call xdg-d-p anyway, at least that's a plan for Pipewire I've heard from Wim Taymans when we spoke about it.

I'll be on PTO since tomorrow to Saturday so if you have any other questions I will answer them later.

Thanks for the answers. Sounds fine to me. Not sure if we only want to support the portals stuff and by that indirectly make it a standard in a broader sense, but if it works fine for everybody and there are no other solutions available, I'm certainly not against it.

One remaining question:

xdg-desktop-portal-kde automatically takes up querying KWin for the GBM buffers, converts them to Gl what process does this for xdg-desktop-portal-kde

In T7785#133501, @romangg wrote:

One remaining question:

xdg-desktop-portal-kde automatically takes up querying KWin for the GBM buffers, converts them to Gl what process does this for xdg-desktop-portal-kde

Xdg-desktop-portal-kde does it on its own, same way like kwrr (I actually took that part from kwrr).

This finally is ready to proceed, thanks everybody. D10965 to go. I'll begin working on KRfb integration with pipewire stream this week.
We should think about input support as now remote sessions are limited to view-only.

In T7785#134386, @Kanedias wrote:

We should think about input support as now remote sessions are limited to view-only.

Can the remote access protocol be extended in this way? It would make sense by its name to facilitate remote input support here as well.

In T7785#134387, @romangg wrote:

Can the remote access protocol be extended in this way? It would make sense by its name to facilitate remote input support here as well.

Sure, we can, but I was thinking more about fake-input protocol as it's already in KWin, kde-connect uses it to control the desktop remotely.
The problem is Pipewire integration, should we create ancillary stream for control? Does Pipewire support this? How does Mutter implement remote control?

It's just food for thought for now, we can think more of it when KRfb patches re-emerge.

As I mentioned to @Kanedias on IRC, there is also remote desktop portal which I proposed as a GSoC task, see https://community.kde.org/GSoC/2018/Ideas#Project:_Remote_desktop_portal.

If there is no student who would like to work on it then I will implement it myself.

What's the status of this?

Screen sharing works just fine with Plasma 5.13 and upcoming Plasma 5.14. There was a major issue in KWayland which will be fixed in KF5 5.51 so we might want to mention this to anyone trying screen sharing on Wayland. We still need some work to be done on KWin side to be able to share just a window, but this is not implement even in Mutter yet. I also still have to finish remote desktop support, which I didn't make to Plasma 5.14, but it will definitely be in Plasma 5.15. I did lots of rewrites in xdg-desktop-portal-kde recently and you can also get screen sharing done through remote desktop portal now. I also did changes in Krfb to support new PipeWire API and to use remote desktop portal to get screen content. What is missing is to handle input in Krfb and pass it to remote desktop portal in xdg-desktop-portal-kde where I miss to forward them to KWayland, this is still all in progress.