Respect the scaling when doing frame-by-frame

This needs some major work, the whole thing is extremely imperative
in a declarative language, it does not feel at all right...