AFAICS the only chance this can happen if the test is using a different library version, e.g. when accidentally using the lib from the system. This would be a test setup error and should be reported with a fail ...

I think that's my fault when I introduced the extractor for test. Apparently it is using the system libraries, and I'm looking for a solution.
I think a skip is still the right thing to do here, since we cannot determine if the tags are correctly written.

In D16716#357321, @astippich wrote:

I think that's my fault when I introduced the extractor for test. Apparently it is using the system libraries, and I'm looking for a solution.
I think a skip is still the right thing to do here, since we cannot determine if the tags are correctly written.

I think skip should only be used iff the test does not apply (e.g. running a Windows specific test on Linux) or when some of the required fixtures can not be provided (e.g. running a test requiring network access in an isolated environment).

In this case, it is "easy" to fulfill the requirements, just correct the test setup.

Imagine what may have happened if this were a QSKIP from the beginning - it would have skipped the tests, and you might have committed untested code. I am not quite sure, but I think skips are not even reported when you run the complete test suite via ctest.

Well, if it is not even signaled via ctest, that's pretty bad, I agree