Fix crash/assert due to empty list, this method is only to be called if sorting is manual.

Patch by Eike Hein over IRC, I'm just the gdb monkey.