Fix lister unintended selection while scrolling.

Code has also been a bit refactored during the process. Added a lot of 'const', some deduplications, simplifications, etc.

Searching up was fixed by increasing cache size to 1MiB.

Differential Revision: https://phabricator.kde.org/D5734