Fix crash when switching color-filtered layers too quickly
Sometimes, during the removal process, the model emits currentChanged() signal, which may cause a double-recursion, which causes a crash.
BUG:377732