I have around ~1400 keys in my keyring and it takes ages to load the dialog, that makes it very slow to select a key.
The first improvment is done by Ingo to use the same cache, so that should be faster now.