Don't ignore custom font size from app configuration
Fixes regression from a1bdb206e213.
The delta created by using the increase/decrease is now stored seperately and applied to the configured font.