Appstreamercli tests all fail due on copying some badly named file (appdatafile::desktopfile)
Closed, ResolvedPublic

Description

Currently a lot of builds have in the build report log a test of appstreamercli failing, with the summary:
"Unable to reach remote location 'https://homepage.x.y.z' - does it"

Looking at the details of that test though, one can see that actually some file copy operation fails, so the report summary is completely misguiding. E.g. "01 Copying the source file 'org.kde.okteta.appdata.xml:org.kde.okteta.desktop:474' from the workspace to the build folder 'e5e5a8f9.tmp' on the Jenkins master failed."
or
" 01 Copying the source file 'org.kde.kdeconnect.kcm.appdata.xml:org.kde.kdeconnect.kcm.desktop:112' from the workspace to the build folder '70cedf1a.tmp' on the Jenkins master failed."

Which hints that there is some filename generation broken. The same strange filename with appdata file and desktop file names being concatenated is also shown in the Appstreamercli result overview page, cmp. e.g. "org.kde.okteta.appdata.xml:org.kde.okteta.desktop" in https://build.kde.org/view/Applications/job/Applications%20okteta%20stable-kf5-qt5%20SUSEQt5.9/17/warnings3Result/ & "org.kde.kdeconnect.kcm.appdata.xml:org.kde.kdeconnect.kcm.desktop" in https://build.kde.org/view/Extragear/job/Extragear%20kdeconnect-kde%20kf5-qt5%20SUSEQt5.9/68/warnings3Result/

Cannot remember to have seen that before, so possibly some recent regression in the CI scripts?

kossebau created this task.Mar 18 2018, 6:30 PM
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptMar 18 2018, 6:30 PM

The Copying the source file error has been around for a while - it's a bug in the regex used by Jenkins to pick up on the warnings (and couldn't cause the actual validation errors in any case). I've now fixed this.

In regards to the unable to reach remote location error, we recently rebuilt the SUSE images, so this is probably a regression or other issue in the Appstream tools themselves (like wanting specific headers, or lacking SSL support). The CI scripting has not changed recently, at least not in regards to the Appstream components.

Thoughts?

Any comments on the above?

Not in the appstream data business myself. @apol or @mak do you have an idea?

mak added a comment.Mar 21 2018, 6:11 PM

Which version of AppStream do these SUSE images use? In older versions, AppStream was making only header requests, which some webservers did not allow. So newer versions try to download the first byte of a document instead, which should always work.

The package version should be visible in the Docker image build log on Jenkins. Given this has regressed recently though it's more likely a newer version is the cause here as it only showed up when we rebuilt the image.

mak added a comment.Mar 25 2018, 4:31 AM

The particular code hasn't been touched in any recent AppStream version, and as far as I can see (at least from the log I found when searching on Jenkins), your AppStream is up to date.
So, I guess it's something else in the image that is broken (especially since this seems to work on other images, I would look for the images generally having no network access, or for some recent curl updates). If appstreamcli is run with the --nonet flag, all actions that require network access are disabled, which can be used as a temporary workaround.

bcooksley closed this task as Resolved.Mar 25 2018, 7:08 AM
bcooksley claimed this task.

I have completed investigating this.

Turns out that Appstream does not like the Bot Detection responses served up by Incapsula, even when they are served with a HTTP 200 response code.
Disabling suspected bot detection for Userbase allows this test to pass without issue. Given that the correct HTTP response code is being provided I do not entirely understand why Appstream was declaring that the page did not exist. I'm assuming the check requires the page be sufficiently bulky content wise as well?

Either way, this is now resolved.

jenkins@2911b8bc82f1:/home/jenkins/workspace/Extragear okteta stable-kf5-qt5 SUSEQt5.9> wget userbase.kde.org/Okteta/
--2018-03-25 06:56:38--  http://userbase.kde.org/Okteta/
Resolving userbase.kde.org (userbase.kde.org)... 149.126.77.38, 2a02:e980:d::26
Connecting to userbase.kde.org (userbase.kde.org)|149.126.77.38|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 718 [text/html]
Saving to: 'index.html'

index.html                                                  100%[=====================>]     718  --.-KB/s    in 0s      

2018-03-25 06:56:38 (184 MB/s) - 'index.html' saved [718/718]

jenkins@2911b8bc82f1:/home/jenkins/workspace/Extragear okteta stable-kf5-qt5 SUSEQt5.9> cat index.html 

<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"></head><body style="margin:0px;height:100%"><iframe src="/_Incapsula_Resource?CWUDNSAI=9&xinfo=14-121183288-0%200NNN%20RT%281521960997859%200%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B12%284%2c315%2c0%29%20U18&incident_id=727000460149229883-440777061818630894&edet=12&cinfo=04000000" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 727000460149229883-440777061818630894</iframe></body></html>