Cache the transform unelegantly.
This does cause a significant speed-up, but it's a bit fiddly. Regardless, the speed-up is enough to make this usable.