I've been thinking about "node-based" scene approach, e.g.

• We would have to define several node types, e.g. ShadowNode, DecorationNode, and SurfaceNode;
• Nodes would compose a tree, e.g. a SurfaceNode could have several child SurfaceNode's, etc.

Scene would render non-transformed windows just by rendering nodes from bottom to top.

With transformed windows, things become a bit complex... The most obvious approach is to render nodes in an offscreen texture, and then render the offscreen texture according to the transformations.

I'm not sure how to properly cross-fade a client if it has subsurfaces and also what Scene should do when a window is closed.

Any other ideas?

WindowPixmaps already form a tree. I can imagine making the base more generic and having shadows and decos in there.
That might be best considered a cleanup exercise done after we've made rendering the WindowPixmap tree for subsurfaces already perfect.

The most obvious approach is to render nodes in an offscreen texture, and then render the offscreen texture according to the transformations.

It'd certainly be one of the fastest solutions to code.

IIRC you do this already for some github magic lamp fork for other reasons. If we are going to do it anyway we may as well do that and simplify all the surrounding code.
We can do that all in the effects themselves without really changing scene.

I wouldn't want to see this in the regular non-animating case as it introduces a lot of overhead.

I don't think it's the only option. We manage to wobble/split apart window decos and the main surface together by sending both in one big quad list. Subsurface trees are just a (much) more complex version of that.

For cross fade.

Scene::Window keeps the current WindowPixmap and the previous WindowPixmap. WindowPixmap is (in theory) a recursive tree. As long as that works all we need to do is cache the geometry inside the windowPixmap instead of looking it up from the subSurface.

Cross fade would be just rendering 2 windowpixmap trees as normal just with different alpha.

The main problem with subsurface and WindowPixmap is that the subsurface a are evaluated too late in the rendering process which makes it impossible to have quads.

My idea would be to move the ownership to Toplevel, so that we can update at the right point. And then I think everything what David wrote holds.

This is definitely an important mechanic preventing people from using Plasma Wayland. Firefox Wayland just does not work, and goes absolutely nuts at least partly because of this.