<Sho_> bshah: since there's no takers, i suggest you commandeer the revisions and finish them to the comments
<bshah> Okay, cool

Some good overview on what we need to do is given here: https://www.x.org/wiki/Events/XDC2017/widawsky_fb_modifiers.pdf

Ideally:

1. Query EGL with which formats + modifiers we can sample
2. Query libdrm with what formats + modifiers we can direct scanout
3. Since direct scanout or sampling can be done interchangeable at any point in time send the formats + modifiers in both results (intersection).

Current situation:
(2) and (3) are irrelevant right now since we don't do direct scanout at all so (1) would be enough. But we might still want to do it like above directly to have less work later on.

Regarding our rendering APIs:
This makes only sense when using OpenGl/EGL, not with QPainter.

Regarding our backends:

DRM

It's pretty obvious to assume that when we sample from some client buffer into our own scanout buffer we don't change the format of our scanout buffer and after sampling the client's format and modifiers are not relevant anymore for scanout. So we can use formats and modifiers on our own buffers either created via EGLStreams or GBM how we like to.

Currently in the GBM case we call gbm_surface_create with format GBM_FORMAT_XRGB8888. We could look into if and how we can profit from using different formats and modifiers (for modifiers we would need to call gbm_surface_create_with_modifiers) at some point later on. Maybe @fooishbar or @garg can give some input on this.

Wayland

As a client we can use the same zwp_linux_dmabuf_v1 protocol to ask the host Wayland server what formats and modifiers it supports and then pipe these to our nested clients intersected with the formats/modifiers we support for sampling.

X.Org

We could query via X11 protocol as in Wayland case above. Low priority though.

Virtual

Just send the supported EGL formats + modifiers.

Since direct scanout or sampling can be done interchangeable at any point in time send the formats + modifiers in both results (intersection).

To be honest, I don't think you really want to do this. On a surprising amount of hardware, the GPU supports compressed/tiled formats that the display controllers don't. Unless you're using planes, you're going to throw performance on the floor by not using them.

The zwp_linux_dmabuf_hints_v1 (or whatever it's called) extension is designed to help with this. For initial use, you just use the GPU-supported modifiers, without bothering to intersect with the KMS-supported modifiers. Then if you see from the KWin side that you could put a surface into a plane directly, you then send the intersection of the GPU and KMS modifiers⁰.

Currently in the GBM case we call gbm_surface_create with format GBM_FORMAT_XRGB8888. We could look into if and how we can profit from using different formats and modifiers (for modifiers we would need to call gbm_surface_create_with_modifiers) at some point later on. Maybe @fooishbar or @garg can give some input on this.

At the moment there's no way to select the 'most optimal' format + modifier combination. Both Weston and Mutter just pick whatever format they want (usually XRGB8888, falling back to RGB565 if not supported), then pass the complete set of supported modifiers for that format to gbm_surface_create_with_modifiers(). GBM then picks the most optimal modifier from that list.

As a client we can use the same zwp_linux_dmabuf_v1 protocol to ask the host Wayland server what formats and modifiers it supports and then pipe these to our nested clients intersected with the formats/modifiers we support for sampling.

Again this only makes sense if you're doing direct buffer passthrough.

We could query via X11 protocol as in Wayland case above. Low priority though.

Since there's no way to do direct buffer passthrough in X11 (well, unless you use DRI3 directly yourself), there's no benefit to doing this.

⁰: Why the intersection in this case? Because if you can no longer scan out the surface on a plane (or if you want to take a screenshot, or ...), you need to make sure that you can access the surface with the GPU.