Just set volume for all of them.
Otherwise when just calling setMuted, we will send the original volume struct, which might have multiple channels, with a hardcoded channel map that only has one channel. PulseAudio then refuses to write the settings.

I can now just setMuted(true) and it mutes notification sounds fine