D16589 now saves the the reports into $WORKSPACE/compat_report/LIB_NAME_compat_report.html.
Before activating the check-abi.py script we need to cleanup the build-artifacts.kde.org ABIReference directory and I need to cleanup check-abi.py to not create the symlinks.

I've removed the old *.tar files from the ABIReference/ repository now.

Oh frameworks is built against different Qt versions (Qt5.9 and Qt5.10). We should to save the abidumps in separately for different Qt versions, as different Qt version may trigger different ABIs. Okay than we need to use platform in the abidumps, too.

Yes, you will need to split by that as well. This is handled normally by having one repository per platform but for ABI dumps you will have to handle that in the metadata.

In T3689#166764, @bcooksley wrote:

Yes, you will need to split by that as well. This is handled normally by having one repository per platform but for ABI dumps you will have to handle that in the metadata.

done in D16814.

fix missing parameter platform:

diff --git a/helpers/create-abi-dump.py b/helpers/create-abi-dump.py
index ef29138..cc9830b 100755
--- a/helpers/create-abi-dump.py
+++ b/helpers/create-abi-dump.py
@@ -312,7 +312,7 @@ for library in foundLibraries:
                 "branchGroup": arguments.branchGroup,
                 "platform": arguments.platform,
         }
-        packageName = "{name}_{scmRevision}_{platform}".format(name=library.name, scmRevision=scmRevision)
+        packageName = "{name}_{scmRevision}_{platform}".format(name=library.name, scmRevision=scmRevision, platform=arguments.platform)
         ourArchive.storePackage(packageName, fileName, scmRevision, extraMetadata)
     except subprocess.CalledProcessError as e:
         retval = e.returncode

otherwise waiting for build-artifacts to be filled :D Before adding the check-abi.py to the pipeline-templates.

I've now applied that fix.

Okay now the last step: enabling check-abi for our builds: D17099

Okay I see R857:a2c2c8fc3f9f in the ci-tooling repo, but jenkins jobs were not updated. When will the jobs been updated?

Changes to jobs only come live on the CI system (and Binary Factory) when the respective "DSL Job Seed" job is run.
I've now run it on build.kde.org - https://build.kde.org/job/Administration/job/DSL%20Job%20Seed/

@bcooksley: Thanks, the script now runs successfully *yeah* But the artifacts are not saved, because of a typo
compat_reports/*_compat_reports.html -> compat_reports/*_compat_report.html

https://build.kde.org/job/Applications/job/messagelib/job/stable-kf5-qt5%20SUSEQt5.9/154/console

[...]
14:32:02 Creating compatibility report ...
14:32:02 Binary compatibility: 100%
14:32:02 Source compatibility: 100%
14:32:02 Total binary compatibility problems: 0, warnings: 0
14:32:02 Total source compatibility problems: 0, warnings: 0
14:32:02 Report: $WORKSPACE/compat_reports/KF5WebEngineViewer_compat_report.html
[Pipeline] archiveArtifacts
14:32:02 Archiving artifacts
14:32:03 ‘compat_reports/*_compat_reports.html’ doesn’t match anything: even ‘compat_reports’ doesn’t exist
14:32:03 No artifacts found that match the file pattern "compat_reports/*_compat_reports.html". Configuration error?

Fixed in a79493119a1c3967956ac8c857df970254889fcb

That for the most part resolves this issue I think, aside from cleaning up the old ABI dumps?

Yes most parts are resolved. But still packages have build errors in the create-abi step. Like:
https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.9/50/console

is there a way to download the consolelogs automatically for all framework/application packages, to scan for errors?

You can use a URL of the form of https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.9/lastCompletedBuild/consoleText to retrieve the logs.
You'd need to build a list of projects in each Product though.

okay still the artifacts are not saved:

https://build.kde.org/job/Frameworks/job/baloo/job/kf5-qt5%20SUSEQt5.10/183/console

[Pipeline] sh
20:49:06 + python3 -u ci-tooling/helpers/check-abi.py --project baloo --branchGroup kf5-qt5 --platform SUSEQt5.10 --environment production
20:49:07 WARNING:root:No released version was found, just use the oldest commit.
20:49:11 INFO:root:Let's do a ABI check 09f07895434f53b3695d19a076e5b60b0d31ad6b against 149c8b2379df57c55f16de206a1cc4f0fcf7282d
20:49:11 Preparing, please wait ...
20:49:12 Comparing ABIs ...
20:49:12 Comparing APIs ...
20:49:12 Creating compatibility report ...
20:49:12 Binary compatibility: 100%
20:49:12 Source compatibility: 100%
20:49:12 Total binary compatibility problems: 0, warnings: 0
20:49:12 Total source compatibility problems: 0, warnings: 0
20:49:12 Report: $WORKSPACE/compat_reports/KF5Baloo_compat_report.html
[Pipeline] archiveArtifacts
20:49:12 Archiving artifacts
20:49:15 ‘compat_reports/*_compat_report.html’ doesn’t match anything, but ‘$WORKSPACE/compat_reports/*_compat_report.html’ does. Perhaps that’s what you mean?
20:49:15 No artifacts found that match the file pattern "compat_reports/*_compat_report.html". Configuration error?

you said, that I should move the output to $WORKSPACE for this step - but now it looks, like this not needed. Can you fix that?

Looks like the report is being written to literally "$WORKSPACE/compat_reports/..." which is why this is happening.
I'll investigate tonight when I get home.

YEAH - the artifacts are now generated. Thaks a lot!

I now looked at every repository that failed to build successfully a ABI dump for Frameworks.
As I'm not really deep into the Framworks code, so I'm unsure, if those are real issues or if those Frameworks need special handling.
Please give me any response you have to those issues. (@aacid , @dfaure, @kde-frameworks-devel)

From 79 packages inside Frameworks a lot run successfully. From those 5 packages I'm quite sure, that those are real issues, that needs to get fixed:

• kdelibs4support
  • missing dependency to frameworks/kitemmodels
  • failed to find #include <kns3/downloaddialog.h>

• kwayland
  • KWayland/Client/xdgforeign_v2.h -> missing include "xdgforeign.h"

• kjs
  • no include/KF5/kjs in INTERFACE_INCLUDE_DIRECTORIES
  • CompileState.h includes non exported opcodes.h and bytecodes/opcodes.h #401955

*plasma-framework

• no include/KF5/KF5PlasmaQuick in INTERFACE_INCLUDE_DIRECTORIES for KF5PlasmaQuick

compiles but has still error on the log:

• networkmanager-qt
  • failing building headers

And some repositories, that have no public library. Is this intended?

• KF5NewStuffQuick has no library (KF5NewStuff is fine)
• extra-cmake-modules
• kdesignerplugin
• kinit
• qqc2-desktop-style

How this ABI checker tool works. As input the tool uses the output made while installing the package.

• It scans this output for installs of ".*Config.cmake$" files
• runs cmake in trace-expand mode for "find_package({name} CONFIG REQUIRED)"
• scans this cmake output for
  • PACKAGE_VERSION -> this is the library version
  • IMPORTED_SONAME_DEBUG, -> to get the soversion
  • IMPORTED_LOCATION_DEBUG -> the the path of the lib
  • INTERFACE_INCLUDE_DIRECTORIES -> all needed include directories

• each exported header file is tried to be compiled and symbols are extracted.

The sourcecode of this script you find in sysadmin/ci-tooling/helpers/create-abi-dump.py

I now looked at every repository that failed to build successfully a ABI dump
for Pim. Most of them build successful, but a few of them have issues:

• kmime has an unexported header in kmime/kmime_newsarticle.h: #include <supertrait.h> that lives in src/supertrait.h

• akonadi-search is unhappy that xapian.h is not imported as first file.

• kitinerary has an unexported header in KPim/kitinerary/airportdb.h: #include "timezonedb.h" that lives in src/knowledgedb/timezonedb.h

• PimCommonAkonadi adds $<TARGET_PROPERTY:KF5::IMAP,INTERFACE_INCLUDE_DIRECTORIES> to target_include_directories, but KF5::IMAP is a private dependency. Can we remove this?

• kdepim-apps-libs: FollowupReminder adds $<TARGET_PROPERTY:KF5::AkonadiCore,INTERFACE_INCLUDE_DIRECTORIES> to target_include_directories. But both files in src/libfollowupreminder/src/*h don't use relative links to AkonadiCore. Can we remove this?

I use the ci-docker images to test the create-abi, but I can't reproduce why those are failing:

• messagelib
• kdav
• libksieve
• akonadi-import-wizard
• kpkpass
• libkgapi

maybe those are also fixed by D17534 otherwise I need more details.

@bcooksley there is also an issue with akonadi-mime - that I don't understand.
Build #2 is fine:

• create-abi-dump.py is run successful
• check-abi.py is happy: "Let's do a ABI check 32025ac9cc671b7adf956d117555336703725ee4 against 1dae8da6ab3ebe1f8f5acf5cd0c563557cd3cbda"

but at Build #3:

• create-abi-dump.py is run successful
• check-abi does NOT check anything.

running this step locally machine is successful:
"Let's do a ABI check 327ed59cd15e6038bf3303ca5f30b4b51b2f61f0 against 1dae8da6ab3ebe1f8f5acf5cd0c563557cd3cbda "
May this be an issue of publishing the buildArtifacts?

okay with merging D17534 CI now successfully builds the abis for the mentioned packages. (I modified the last comment)

@bcooksley for akonadi-search we need some special settings for the abi-create step. Where we should store such settings?

This is a more general solution, that we can also modify the settings for other platforms etc.

ci-tooling/local-metadata/abi-complience-checker.yaml:

gcc_options:
    - "-std=c++11"
    - "-fPIC"
skip_include_paths:
    - /usr/lib/python3.6/site-packages/utils/fake_libc_include
    - /usr/include/clang/AST
    - /usr/lib64/clang/6.0.1/include
add_include_paths:
    - /usr/lib64/qt5/mkspecs/linux-g++

# "SUSEQt5.10":    # platform specific
# gcc_options:
#    - "-DSUSE5.10"

#"kde/*":    # special settings for all repos in kde (fnmatch)
#   gcc_options:
#       - "-DKDE"

#"kde/pim/*":    # special settings for all repos in kde/pim
#   gcc_options:
#       - "-DKDEPIM"
#  "SUSEQt5.10":   # for one specific platform
#     gcc_options:
#         - "-DKDEPIMSUSEQT5.10"

"kde/pim/akonadi-search":     # for one repo
  "SUSEQt5.10":   # for one specific platform
    gcc_options:
      - "-std=c++11"
      - "-fPIC"
      - "-DQT_NO_KEYWORDS"

and the Python code to handle this:
rules are:

• anthing overwrites defauls (outer level)
• most matching path overwrites less matching path
• platform specific overwrites non platform stuff

general -> specific
default -> "basepath/*" -> "basepath/* - platform" -> "basepath/repo" -> "basepath/* - platform"

import yaml, fnmatch, copy

metadata = yaml.load(open('local-metadata/abi-complience-checker.yaml'))

def getSettings(path, platform):
    
    entries = []
    for key in metadata:
        if fnmatch.fnmatch(path, key):
            entries.append(key)

    # sorting all entries - the entries at the end overwrites the former settings
    # len is efficient, but would fail if we have single letter repo names
    entries.sort(key=len)
    entries.insert(0, platform) # add platform settings at the start

    d = copy.copy(metadata)     # load default settings
    for entry in entries:
        try:
          content = metadata[entry]
          d.update(content)
          d.update(content[platform]) # try to load platform specific settings
        except KeyError:
          pass
    return d    

getSettings("kde/pim/akonadi-search",  "SUSEQt5.10")

Okay, there have been a number of comments here since i've last read it, so a bit to catch up on and go over.

In regards to akonadi-mime I have now re-run it, and it seems to work fine.
As long as you're relying only on the master manifest.yaml, then it should be impossible for any publishing failure to cause issues.

In terms of the akonadi-search issue, using something like that YAML file should be fine yes and is historically how we've solved such issues.

In T3689#170284, @bcooksley wrote:

In regards to akonadi-mime I have now re-run it, and it seems to work fine.
As long as you're relying only on the master manifest.yaml, then it should be impossible for any publishing failure to cause issues.

Well if if would be just akonadi-mime, I would not care. but it is actually a lot more failing with the same pattern.

What I did for checking:
*download this consoleText: https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.10/lastCompletedBuild/consoleText

• than do some simple regex:
  • (Successfully created a dump (^.*Dump path: abi_dumps/.*/ABI\.dump\s*$)
  • search for "Comparing ABIs"

And the outcome is overwhelming - a lot of is broken. I checked a lot locally via docker and could not reproduce this bug. That's why I think it is something with publishing the packages. Is it just one server, or is there any load balancing come into account so request are answered by a different sever, that isn't synced?

for kde/pim/*:

• kblog
• mailcommon
• kmbox
• kontactinterface
• kpimtextedit
• akonadi-import-wizard
• kpkpass
• kldap
• kalarmcal
• libgravatar
• libkgapi
• libkleo
• grantleetheme
• kcontacts
• akonadi-notes
• kitinerary
• ktnef
• akonadi-mime
• kcalutils

for frameworks:

• kcrash
• kwallet
• kpackage
• kholidays
• kparts
• syndication
• frameworkintegration
• khtml
• krunner
• knotifyconfig
• kidletime
• kitemviews
• kjobwidgets
• kmediaplayer
• kpeople
• prison
• kxmlrpcclient
• sonnet
• kcoreaddons
• kunitconversion
• kfilemetadata
• baloo
• kitemmodels
• kpty
• kconfig
• kxmlgui
• kconfigwidgets
• kdbusaddons
• kjsembed
• kplotting
• kactivities
• kdesu
• ki18n
• knotifications
• kactivities-stats
• attica
• ktexteditor
• solid
• kirigami
• syntax-highlighting
• kauth

In terms of the akonadi-search issue, using something like that YAML file should be fine yes and is historically how we've solved such issues.

okay create D17672 for this.

I checked execution of KDav on all four build nodes, and the process ran fine when done manually.
Given there is no output from check-abi.py, this is quite hard to diagnose.

Can you propose some debug output which might point out why it's doing nothing?

In T3689#170601, @bcooksley wrote:

I checked execution of KDav on all four build nodes, and the process ran fine when done manually.
Given there is no output from check-abi.py, this is quite hard to diagnose.

In the last for l in libraries: loop, we get output for every item in that list, etiher because there are candidates or there are not.
That means the only way to produce no output at all is that libraries list must be empty. We do not remove any entry from that list, so the initial search do not find any matching entry.
And the only way the list can be empty is that the for key, entry in ourArchive.serverManifest.items() don't find any matching entry.

Can you propose some debug output which might point out why it's doing nothing?

I added D17710 to retry to reload the serverManiphest file again after 5secs to sleep. The added output is hopefully enough to spot the issue.

I've made an experimental change which should resolve this issue if it is what is suspected to be the issue here.

I have made sure that kde/pim is now cleanup in terms of the issue of "not runnig ABI", by triggering a rebuilt. So it is a good test if the patch is working.
With more tests I have detected, that builds still use the oldest commit instead of tags commits (fix D17745) .

kwayland

KWayland/Client/xdgforeign_v2.h -> missing include "xdgforeign.h"

This is now hopefully fixed. Please let me know if it's still an issue.

In T3689#170799, @davidedmundson wrote:

kwayland

KWayland/Client/xdgforeign_v2.h -> missing include "xdgforeign.h"

This is now hopefully fixed. Please let me know if it's still an issue.

Thx, it looks fine so far. You see that now KF5Wayland_compat_report.html is created:

https://build.kde.org/job/Frameworks/job/kwayland/job/kf5-qt5%20SUSEQt5.11/6/

YEAH so far the abi-compatibility-results.yaml for pim and Frameworks, as far as they got update the last days.
So we are surly hitting the end of this and catching up the loosen tails of all of this. I'll focus currently on other stuff, will look at it and fix stuff in a week or so.

I get ABI error on use of QPair and QSet in kdiagram:
https://build.kde.org/job/Calligra/job/kdiagram/job/kf5-qt5%20SUSEQt5.10/8/artifact/compat_reports/KGantt_compat_report.html#Type_Binary_Problems_Medium
Afaics these lines have not been changed since last release.
I have added a public getter to the class, but it does not complain about that, so...
Also it does not complain about similar use in calligra (although not *exactly* the same, calligra uses other datatypes)
Can anybody shed any light on this?

I'm not a expert in abi-complience-checker, so we need to find out together...

In T3689#175022, @danders wrote:

The ABI error started with build 4:
https://build.kde.org/job/Calligra/job/kdiagram/job/kf5-qt5%20SUSEQt5.10/4/

but yes I can't see why the symbols are changed. Maybe the Qt version was switched in between. But from the error type it sounds like, that acc do not get the needed information of the underlying structure of QSet/QPair and can't garantee that this structure will be the same in future.

Afaics these lines have not been changed since last release.

Well we don't have a released version with ABI, so we just check them against a470d77be5a33529a95eb89243ddfaef6ee27ee7. This will change if the next version is released.

Also it does not complain about similar use in calligra (although not *exactly* the same, calligra uses other datatypes)

Can anybody shed any light on this?

well you can dig more into it by getting the abi dumps:
https://build-artifacts.kde.org/production/ABIReference/

after you downloaded two of the dumps you can check them again locally:

abi-compliance-checker -l KGrantt --old <old.dump> --new <new.dump>

than you can look to all outputs yourself.

The abidumps look different:

for a470d77be5a33529a95eb89243ddfaef6ee27ee7:

'90987' => {
             'Algn' => 1,
             'Header' => 'qpair.h',
             'Line' => '49',
             'Memb' => {
                         '0' => {
                                  'algn' => 1,
                                  'name' => 'first',
                                  'type' => '11416'
                                },
                         '1' => {
                                  'algn' => 1,
                                  'name' => 'second',
                                  'type' => '11416'
                                }
                       },
             'Name' => 'QPair<struct QDateTime, struct QDateTime>',
             'TParam' => {
                           '0' => {
                                    'name' => 'struct QDateTime'
                                  },
                           '1' => {
                                    'name' => 'struct QDateTime'
                                  }
                         },
             'Type' => 'Struct'
           },

and for f7f9ca1a7a8bfd550022ca4aafe3bb2985a1bee4

'90223' => {
             'Algn' => 1,
             'Header' => 'qpair.h',
             'Line' => '49',
             'Name' => 'QPair<struct QDateTime, struct QDateTime>',
             'TParam' => {
                           '0' => {
                                    'name' => 'struct QDateTime'
                                  },
                           '1' => {
                                    'name' => 'struct QDateTime'
                                  }
                         },
             'Type' => 'Struct'
           },

so yeah somehow it does not get as much information out of qpair.h as before.

Hmmm. do not know how to proceed with this without the actual libs available.
I have tried building against a new lib and runing with old lib, with no problems.
I was unsuccesful using abi-dumper on the old lib, so cannot check if it gives the same output as for f7f9ca1a7a8bfd550022ca4aafe3bb2985a1bee4 above.
I guess I have to live with this atm and see how things shape up in the furure...

To create a dump by hand this isn't that easy. see sysadmins/ci-tooling/helpers/create-abi-dump.py.

I was unsuccesful using abi-dumper

So far I understood abi-dumper is something different - I use abi-complience-checker to create those dumps.

As you need some more environment for this script, I ended up using docker-composer and a script to test this locally.

In T3689#175818, @knauss wrote:

So far I understood abi-dumper is something different - I use abi-complience-checker to create those dumps.

Afaiu abi-dumper is the new way of creating dumps. From home page:
"This new way is based on the analysis of the debug-info from binary objects. It's more reliable, faster and simple way. "
Any particular reason for not to use abi-dumper? (Just curious)

Anyway, I compiled both stable and unstable versions myself, created dumps with abi-dumper and got a clean bill.
So I'm pretty certain the problem reported on CI is not correct (but I have been wrong before).

In T3689#175858, @danders wrote:

Afaiu abi-dumper is the new way of creating dumps. From home page:
"This new way is based on the analysis of the debug-info from binary objects. It's more reliable, faster and simple way. "
Any particular reason for not to use abi-dumper? (Just curious)

IMO the homepage is not that clear about this. It tells me yes I can use abi-dumper for the library dump, but abi-complience-checker can also check the header files, that is so far not possible to check with abi-dumper.

Anyway, I compiled both stable and unstable versions myself, created dumps with abi-dumper and got a clean bill.
So I'm pretty certain the problem reported on CI is not correct (but I have been wrong before).

Thanks for checking. So we need to give acc the information to not check internal Qt structures.

@bcooksley: Did Qt 5.10 got a minor update recently, that is compiled differently?

Build logs from our Docker images can be found at https://build.kde.org/job/Administration/
In this case, the image last changed 26 days ago.

Please merge the commits in following order:
D19219
D19220
D19221
D19222
D19218

I don't want to merge the different patches, as it is a lot easier to review and to understand what is going on. I did not find a way to bundle different patches.

With the exception of D19222 all of those have now been integrated.