Autopkgtests HOWTO
As a precusor to staging
First install the 'autopkgtests' package. Once installed, write a small script to set up the repositories. One way to do this is copy/pasting the code below and save it as /usr/share/autopkgtest/setup-commands/kstaging:
#!/bin/sh
apt-get -y install software-properties-common
add-apt-repository -sy ppa:kubuntu-ppa/staging-frameworks
add-apt-repository -sy ppa:kubuntu-ppa/staging-plasma
add-apt-repository -sy ppa:kubuntu-ppa/staging-kdeapplicationsOnce the script is saved, to set up the virtualization environment. In this example, I use LXD, on of the 'virtualization' backends for autopkgtests running. To set up the LXD container where the tests are going to be executed, run this command as root:
autopkgtest-build-lxd ubuntu-daily:focal/amd64
Once your container is created, you may run any staging autopkgtests by running this command as root:
autopkgtest akonadi-search --setup-commands=kstaging -U --apt-pocket=proposed -- lxd autopkgtest/ubuntu/focal/amd64
If you suspect the tests are going to fail, pass -s/--shell-fail to the above command, so it would be:
autopkgtest akonadi-search -s --setup-commands=kstaging -U --apt-pocket=proposed -- lxd autopkgtest/ubuntu/focal/amd64
That should open a shell into the LXD container where the tests are being executed.
Testing solutions to proposed failures
Summary
Included below is a complete, real world example to illustrate how autopkgtest's lxd backend can be used to test a solution to failures with packages in proposed due to a test failure. This can be run locally or on the cloud. No GUI required.
Introduction
As discussed in the documentation on proposed migration, autopkgtest is run against proposed packages. The results of those tests, including failures, can be found on the excuses status page. After finding a potential solution, the best next step is to recreate the test environment. This can be done with virtual machine as documented, but using a container is far more portable and less resource intensive. This was performed in a Kubuntu development container (I prefer the term "kontainer"). With one of those, you could even do this on a phone by sshing to it— how cool is that?
The problem
With that in mind, we found a failure in ktexteditor. Looking at the logs, we can see the whole testsuite begin at line 9717. At line 9846, it becomes clear that a test called katesyntaxtest has failed:
61/66 Test #61: katesyntaxtest ..........................***Failed 0.74 sec
Scrolling down a bit more to line 10098, we see that 14/16 steps to this test failed:
Totals: 2 passed, 14 failed, 0 skipped, 0 blacklisted, 676ms
Above this line are the details of the failure. You can see the two passed steps are the setup and cleanup ones. An example failure begins with a diff between the "reference" results (those defined in the testsuite) and the "current" results (the results of the test on the proposed package):
DIFF: --- /tmp/autopkgtest.yuFMos/build.JG3/src/autotests/input/syntax/j/results/test.ijs.reference.html 2017-12-02 20:08:18.000000000 +0000 +++ /tmp/autopkgtest.yuFMos/build.JG3/src/autotests/input/syntax/j/results/test.ijs.current.html 2017-12-12 12:59:20.885249118 +0000 @@ -8,7 +8,7 @@ </head> <!-- Highlighting: "J" --> <body> -<pre style='color:#31363b;background-color:#fcfcfc;'> +<pre style='color:#232627;background-color:#fcfcfc;'> Comment <span style='color:#7f8c8d;'>NB. This is a single line comment, check regions overlapping priority: 1 2.3 +/ &. 'quoted text'</span> <span style='color:#7f8c8d;'>NB.(</span> FAIL! : KateSyntaxTest::testSyntaxHighlighting(/tmp/autopkgtest.yuFMos/build.JG3/src/autotests/input/syntax/j/test.ijs) Compared values are not the same Actual (QString::fromLocal8Bit(out)): "--- /tmp/autopkgtest.yuFMos/build.JG3/src/autotests/input/syntax/j/results/test.ijs.reference.html\t2017-12-02 20:08:18.000000000 +0000\n+++ /tmp/autopkgtest.yuFMos/build.JG3/src/autotests/input/syntax/j/results/test.ijs.current.html\t2017-12-12"... Expected (QString()) : "" Loc: [/tmp/autopkgtest.yuFMos/build.JG3/src/autotests/src/katesyntaxtest.cpp(132)]
Looking at the fail being on KateSyntaxTest::testSyntaxHighlighting, we can presume this has to do with syntax highlighting. What's particularly interesting looking at the diff is that the only thing that seems to have changed is the color.
Finding the fix
So we swim upstream to KDE git and find a commit that looks remarkably like it will fix our tests:
diff --git a/autotests/input/syntax/j/results/test.ijs.reference.html b/autotests/input/syntax/j/results/test.ijs.reference.html index 2954cd6..c766d44 100644 --- a/autotests/input/syntax/j/results/test.ijs.reference.html +++ b/autotests/input/syntax/j/results/test.ijs.reference.html @@ -8,7 +8,7 @@ </head> <!-- Highlighting: "J" --> <body> -<pre style='color:#31363b;background-color:#fcfcfc;'> +<pre style='color:#232627;background-color:#fcfcfc;'> Comment <span style='color:#7f8c8d;'>NB. This is a single line comment, check regions overlapping priority: 1 2.3 +/ &. 'quoted text'</span> <span style='color:#7f8c8d;'>NB.(</span>
That said, now we need to test it out. First grab the patch, which you can do by copy/pasting the text or by clicking the "patch" button.
Collecting tools
Now we need to set up our test environment. To begin, we'll need some packages:
$ sudo apt update && sudo apt install autopkgtest lxd lxd-clientNext we'll need to set up lxd, just accepting defaults (though we had better luck with the dir storage pool than the default (note this is the response from lxd 2.20-0ubuntu4 running on bionic; YMMV otherwise):
$ sudo lxd init Do you want to configure a new storage pool (yes/no) [default=yes]? Name of the new storage pool [default=default]: Name of the storage backend to use (dir, lvm, btrfs) [default=btrfs]: dir Would you like LXD to be available over the network (yes/no) [default=no]? Would you like stale cached images to be updated automatically (yes/no) [default=yes]? Would you like to create a new network bridge (yes/no) [default=yes]? What should the new bridge be called [default=lxdbr0]? What IPv4 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]? What IPv6 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]?
Next, we'll need to prepare our testing container:
$ autopkgtest-build-lxd images:ubuntu/bionic/amd64If you have apt-cacher-ng set up, as our kontainer does, you'll likely run into strange errors. You can fix this fairly easily by using your container's IP address, the port the cacher is running on (3128 by default), and setting it to be the proxy for http:
$ export LXD_ADDRESS=$(ip -4 -o address show dev lxdbr0 | awk -F'[ /]*' '{print $4}') $ lxc profile set default environment.http_proxy "http://$LXD_ADDRESS:3128"
Putting it all together
We should probably first confirm our results match the test results from the official Canonical machinery, you can just go ahead and run a test:
$ autopkgtest ktexteditor --apt-pocket=proposed --apt-upgrade --build-parallel=4 -- lxd autopkgtest/ubuntu/bionic/amd64As you could tell in that log above, this is going to go on for quite a while setting up the test bed. Eventually you'll get to the test. Sudo make me a sandwich while you wait. Having got the results we expect back, now we can work on implementing our fix.
Assuming you have the Kubuntu Automation tooling, let's get a copy of our repo. For the purposes of illustration, we'll assume this is all done in ~/autopkgtests:
$ git-clone-all -s ktexteditor -b kubuntu_bionic_archiveNow let's import our patch. It's important that you're in the same level as the debian folder for this to work right. In our example, the patch is within the ~/autopkgtests/ktexteditor folder and we gave it a nice name:
$ cd ktexteditor/git/ $ quilt import ../fix_syntax_hilite_test.patch
Now we need to build our source package, as if we were preparing it for upload to a PPA (note that as is, this requires a GPG key):
$ gbp-ppa -b kubuntu_bionic_archiveThis will create a source package as described in a .dsc file in ../build-area. Now we can use that to rerun our test:
$ autopkgtest../build-area/ktexteditor_5.41.0-0ubuntu1~ubuntu18.04~ppa2.dsc ktexteditor --apt-pocket=proposed --apt-upgrade --build-parallel=4 -- lxd autopkgtest/ubuntu/bionic/amd64And success! Well now we need to get it put in a PPA and get a sponsor to upload it and we're set!
- Last Author
- joselema
- Projects
- None
- Subscribers
- None