Ok, I found something and implemented in D17980. @zzag can you please test it?

I investigated a little bit on what changed in the new mesa version. It introduces a EGL_MESA_device_software which seems to support what we do with vgem without needing vgem. So it might be that we need to adjust our code to support this new extension in addition to vgem.

In T10245#171947, @bcooksley wrote:

Note that per the investigation by @zzag above, it looks like this is only triggered in scenarios where VGem is present, so it's likely most environments wouldn't see this.

Note that per the investigation by @zzag above, it looks like this is only triggered in scenarios where VGem is present, so it's likely most environments wouldn't see this.

I'll ask our OpenSUSE packagers if something can be done. Given that the packaging for the older version is still around it should hopefully be fairly straight forward for them to provide something.

Pulling in @fvogt - our unit tests found a regression in Mesa shipped in Tumbleweed. Could you please trigger an investigation inside openSUSE how it could happen that this was not discovered prior to shipping the update.

So what should we do? We cannot keep the tests failing till Mesa got that fixed - if at all. Can we get another CI base system which is not constantly rolling and introducing issues?

I would love to, but unfortunately Tumbleweed has already withdrawn the older versions of Mesa from it's repositories, so we're unable to do so :(
The only repositories containing older Mesa's for Tumbleweed now are people's personal home:* repositories (per https://software.opensuse.org/package/Mesa)

Could you downgrade Mesa please on the CI system?

Guess that confirms it's a Mesa regression - will you and Martin handle sorting it with the Mesa devs?

I downgraded Mesa to 18.2.6 and tests pass again.

That would seem to support the theory that Mesa has a regression with regards to VGem support...

Oh, I forgot to load vgem. Now, tests fail.

In T10245#171890, @bcooksley wrote:

Okay. The CI system is currently using Mesa 18.3.1 - how does this compare to your local system?

Okay. The CI system is currently using Mesa 18.3.1 - how does this compare to your local system?

In T10245#171815, @bcooksley wrote:

When you try to reproduce locally, are you using the docker images we ship at Dockerhub, or a SUSE VM?

Neither, nor. I just use my normal system.

On another note - does KWin's test infrastructure start it's own display services or anything along those lines by any chance? (I wonder if the issue is because the CI Tooling starts Xvfb and then KWin does some additional stuff, and DRM/GBM/Mesa freaks out as a result)

When you try to reproduce locally, are you using the docker images we ship at Dockerhub, or a SUSE VM?

In T10245#171768, @bcooksley wrote:

So it's not liking connecting to our Xvfb instance?

So it's not liking connecting to our Xvfb instance?

According to the documentation: EGL_NOT_INITIALIZED is generated if display cannot be initialized.

New additional output:
libEGL debug: EGL user error 0x3001 (EGL_NOT_INITIALIZED) in eglInitialize
libEGL debug: EGL user error 0x3001 (EGL_NOT_INITIALIZED) in eglInitialize
libEGL debug: EGL user error 0x3001 (EGL_NOT_INITIALIZED) in eglDestroyContext

I've now re-run the builds - does that output give any clues?

There is one additional env variable which could be set: EGL_LOG_LEVEL=debug

With test coverage I found that we fail in: https://phabricator.kde.org/source/kwin/browse/master/platformsupport/scenes/opengl/abstract_egl_backend.cpp$99

For comparison: on my system the debug output is

It's a possibility

It would appear this is our cause:

MESA-LOADER: failed to retrieve device information gbm: failed to open any driver (search paths /usr/lib64/dri) gbm: Last dlopen error: /usr/lib64/dri/vgem_dri.so: cannot open shared object file: No such file or directory failed to load driver: vgem

Afaik there are some Mesa env variables which might provide more information. They are documented on mesa's web page

What we have from the tests is:

I've investigated this and it appears shortly before this regression occurred, we did a rebuild of our SUSE images.
So it would seem that this regression is due to something in the software stack (probably Mesa/X/Wayland)

In T3689#170799, @davidedmundson wrote:

kwayland

KWayland/Client/xdgforeign_v2.h -> missing include "xdgforeign.h"

This is now hopefully fixed. Please let me know if it's still an issue.

kwayland

KWayland/Client/xdgforeign_v2.h -> missing include "xdgforeign.h"

I have made sure that kde/pim is now cleanup in terms of the issue of "not runnig ABI", by triggering a rebuilt. So it is a good test if the patch is working.
With more tests I have detected, that builds still use the oldest commit instead of tags commits (fix D17745) .

I've made an experimental change which should resolve this issue if it is what is suspected to be the issue here.

In T3689#170601, @bcooksley wrote:

I checked execution of KDav on all four build nodes, and the process ran fine when done manually.
Given there is no output from check-abi.py, this is quite hard to diagnose.

In the last for l in libraries: loop, we get output for every item in that list, etiher because there are candidates or there are not.
That means the only way to produce no output at all is that libraries list must be empty. We do not remove any entry from that list, so the initial search do not find any matching entry.
And the only way the list can be empty is that the for key, entry in ourArchive.serverManifest.items() don't find any matching entry.

I checked execution of KDav on all four build nodes, and the process ran fine when done manually.
Given there is no output from check-abi.py, this is quite hard to diagnose.

In T3689#170284, @bcooksley wrote:

In regards to akonadi-mime I have now re-run it, and it seems to work fine.
As long as you're relying only on the master manifest.yaml, then it should be impossible for any publishing failure to cause issues.

Okay, there have been a number of comments here since i've last read it, so a bit to catch up on and go over.

@bcooksley for akonadi-search we need some special settings for the abi-create step. Where we should store such settings?

okay with merging D17534 CI now successfully builds the abis for the mentioned packages. (I modified the last comment)

I use the ci-docker images to test the create-abi, but I can't reproduce why those are failing:

I now looked at every repository that failed to build successfully a ABI dump
for Pim. Most of them build successful, but a few of them have issues:

I now looked at every repository that failed to build successfully a ABI dump for Frameworks.
As I'm not really deep into the Framworks code, so I'm unsure, if those are real issues or if those Frameworks need special handling.
Please give me any response you have to those issues. (@aacid , @dfaure, @kde-frameworks-devel)

Alas, scripty time is probably the most likely time for the current approach - of only waiting for idle at the start of the whole test run - will fail us.

During last scripty run we had one random failure again: https://build.kde.org/job/Plasma/job/kwin/job/kf5-qt5%20SUSEQt5.11/221/

Cool, thanks for the update.

Thanks a lot. This seems to be working - at least running the tests takes significantly more time and the last 4 build succeeded.

YEAH - the artifacts are now generated. Thaks a lot!

Looks like the report is being written to literally "$WORKSPACE/compat_reports/..." which is why this is happening.
I'll investigate tonight when I get home.

okay still the artifacts are not saved:

You can use a URL of the form of https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.9/lastCompletedBuild/consoleText to retrieve the logs.
You'd need to build a list of projects in each Product though.

Yes most parts are resolved. But still packages have build errors in the create-abi step. Like:
https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.9/50/console

This should now hopefully be addressed, let me know if you continue to see this happening. A load threshold of one might end up being a bit too aggressive, but we'll see how it goes (at worst you'll have to wait an enormous amount of time before it will report back)

Fixed in a79493119a1c3967956ac8c857df970254889fcb

Because of the high resource cost of the dependency build jobs they are only triggered manually.

@bcooksley: Thanks, the script now runs successfully *yeah* But the artifacts are not saved, because of a typo
compat_reports/*_compat_reports.html -> compat_reports/*_compat_report.html

Thanks a lot Ben. Compilation failed because of a hardcoded path in the build-dependencies script. I have pushed a fix in git master, the job will need to be restarted... or is it automatically restarted when the script file is changed? Would be very handy...

Changes to jobs only come live on the CI system (and Binary Factory) when the respective "DSL Job Seed" job is run.
I've now run it on build.kde.org - https://build.kde.org/job/Administration/job/DSL%20Job%20Seed/

The Dependency Build is now underway - https://binary-factory.kde.org/job/Kdenlive_Nightly_Appimage_Dependency_Build/3/console
Assuming this succeeds, the automatic triggering process should commence the build of the first Appimage on the Binary Factory sometime in the next 24 hours.

Okay I see R857:a2c2c8fc3f9f in the ci-tooling repo, but jenkins jobs were not updated. When will the jobs been updated?

All Linux containers on both the CI system and Binary Factory are started fresh for each job, so nothing will be carried over.

Ok so the Kdenlive build scripts should be ready for testing in our git master. I have submitted a patch for the dockerfile:

Okay. Please bear in mind that this does mean that if load were to spike during the middle of KWin's test suites there is nothing the CI Tooling can do about that and you'll still see failures.

As we want to be able to run ctest as developers I think it would be better if the CI would do the check.

@bcooksley sure, as said, a WIP project on my side. I hope people will see the advantages of using linuxdeploy.

We can certainly look into providing the newer tooling.

Okay. Would you prefer this be handled on an all-tests basis (where the CI would do the check, but if the load spiked midway through tests running it would just keep going) or on an individual tests basis (where KWin's own test tooling would do the check and act accordingly)?

Sitting around and waiting sounds totally fine to me.

@bcooksley it would be nice if you could provide the more modern linuxdeploy, which supersedes linuxdeployqt. It's plugin based (https://github.com/linuxdeploy/awesome-linuxdeploy), providing an extensive framework for plugin developers.

Okay now the last step: enabling check-abi for our builds: D17099