Did you test this? I think the way it was was correct. A transient window is for example a popup . Only the popup is transientOnly (capturemode window under cursor) and everything is transientwithparent. See also the help text
-u, --windowundercursor Capture the window currently under the cursor, including parents of pop-up menus -t, --transientonly Capture the window currently under the cursor, excluding parents of pop-up menus
and the gui version
Please revert this
We should consider renaming the enums or serialized values or adding a comment in the code though, because the way this is written is is confusing and looks wrong (even if it's right).