This massively reduces the amount of network requests, at very little
disk and memory cost (1Mb per 20+k of missing avatars with both backends,
60+k for just Gravatar).
It's even more important when having both backends enabled, in that case
we always did a request for the first backend even if the second one could
provide it from the local cache.