Would it be possible to add some kind of test maybe? to make sure that they do something at least.

One could potentially write an autotest that fires a search to every site and checks for whether it responds and doesn't throw a 404. However, I fear that test would be quite flaky.

In D9456#182021, @broulik wrote:

One could potentially write an autotest that fires a search to every site and checks for whether it responds and doesn't throw a 404. However, I fear that test would be quite flaky.

And not automatically run; iirc the build environments of the distributions may be isolated (for good reasons).