Makes the API more complex, but it's worth it as it allows
knowledgable users to opt into avoding costly work and/or
data copies in very common use cases.
Significantly speeds up a lot of work in the models in the
presence of X apps that can't be identified and fall back to
the window icon.
Based on profiling by David.