Simplify minimum size handling

Summary:
We don't need to track the minimum size separately anymore, so avoid this
getting out of sync by removing it.

Reviewers: svuorela

Reviewed By: svuorela

Subscribers: kde-frameworks-devel

Tags: Frameworks

Differential Revision: https://phabricator.kde.org/D27952