When re-using runners when reloading, reload their configuration

When configuration is changed, runners are reloaded but their instances re-used so they never get notified of config changes.

BUG: 399621

Differential Revision: https://phabricator.kde.org/D16108