[WIP/RFC] Please let me know if what I propose is sane
UNIX filenames can contain any bytes (except \0 and /). Qt's QFile::decodeName() calls QString::fromLocal8Bit(), assuming that all filesystems use the system's locale encoding. For filenames that have been created with a different encoding, and have not yet been converted (e.g. using convmv), this creates non-reversible U+FFFD (REPLACEMENT CHARACTER) code points in the filenames.
For example, some old-style archives might not contain any information about the encoding of the filenames, and even today archivers extract them without trying to convert to the locale's encoding.
While full support for those filenames is not needed, Dolphin should at least be able to delete, rename, and move those files. Since all actual (local) file handling is done inside the file kioslave, patching Dolphin will not help.
This code is a near verbatim copy of the code we had in kdelibs, written by Szókovács Róbert. Only minor adaptions to Qt5 were done. It decodes invalid bytes as U+10FExx from Plane 16 (Supplementary Private Use Area-B) to be able to encode them later.
Dolphin could detect filenames with those characters, and either mark them (by color or overlay icon), or even automatically offer to rename them.