KDE gits only allow ~16 concurrent connections from one node. To not exceed this we have a semaphore daemon running and a fake git that semaphores network-bound actions. Currently we only have this on the jenkins masters, but maybe should also roll out to slaves to prevent them from exceeding the limit and having git commands randomly fail.
This is however substantially less likely with slaves, as slaves do not poll (I think?) and so the git command limit is limited by the build slots on the slaves, which globally do not get anywhere near 16 across all blue systems jenkinses. If one of the jobs runs git out of concurrent threads, that is a different story though. Then again most jobs run through docker, so they'd bypass the semaphore anyway.
Any votes for semaphore rollout?