CMake supports at least two build generators: make and ninja. By default the first one is used, the second one needs to be requested explicitly with -GNinja option.

Benchmarks show that ninja builds much faster compared to make. E.g. here, at the bottom of the page you can see pure comparison of make and ninja. And this page is even more interesting as it has comparisons of cmake-with-make and cmake-with-ninja specifically (spoiler: ninja wins).

So it may make sense to add -GNinja option to test-builds, and replace make with ninja.