Prevent modifications of removed entities

We'd end up with entities that don't show up in queries but are
considered existing.