Old Kubuntu Packaging Notes

Updated 2,628 Days AgoPublic

Actions

This is an attempt at rewriting the Kubuntu Documentation for packaging
It MUST be written for people with absolutely no coding or any experience with managing software, but explain what needs to be explained, and link to the rest. (e.g. give links to git tutorials, don't explain what git is or how to use it)

Getting set up: http://packaging.ubuntu.com/html/getting-set-up.html~~
Current documentation: https://community.kde.org/Kubuntu#Packaging~~
Debian doc: https://www.debian.org/doc/debian-policy/~~
Maint guide: https://www.debian.org/doc/manuals/maint-guide/ (less up-to-date than the policy, but has more useful~~
information for working on packages)
~~Library packaging guide: https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html~~

Kubuntu Automation (KA) Packaging
status colours on the build status page: Red=failed, green=OK, blue=not yet built for various reasons, orange=has issues, but
nothing that completely breaks the package such as linitian errors, new symbols, stuff like that.
(would be good to say here what to do to fix red, and also turn orange to green)

What you need:
Git commit access to debian git http://anonscm.debian.org/cgit/pkg-kde (only for committing package changes to Debian)
Member of Kubuntu Ninjas Launchpad team https://launchpad.net/~kubuntu-ninjas (for uploading to the Staging PPAs)
Member of Kubuntu Members Launchpad team https://launchpad.net/~kubuntu-members (for access to make changes in the Kubuntu CI Jenkins)
Member of Kubuntu Packagers Launchpad team https://launchpad.net/~kubuntu-packagers (for LP packaging branches and commit to KA) [either direct or through ~kubuntu-members]
KDE Identity Account for access to the team notepads on https://notes.kde.org, KDE Wikis, Forums, phabricator (for patches
sent upstream) https://techbase.kde.org/Contribute/Get_a_Contributor_Account~~
Separate e-mail based authentication to file upstream bugs to https://bugs.kde.org/~~

Index, Glossary:
LP = Launchpad (Ubuntu Project management software) https://launchpad.net~~
KA = Kubuntu Automation https://git.launchpad.net/ka~~
KCI = Kubuntu Continuous Integration http://kci.pangea.pub~~
PPA = Personal Package Archive managed by Launchpad (repository for packages)~~
https://userbase.kde.org/Kubuntu/Advanced#PPAs_.28Personal_Package_Archive.29~~
git = Versioning system tool http://git-scm.com~~
QA = Quality Assurance http://qa.kubuntu.co.uk~~
MP = merge proposal https://dev.launchpad.net/UsingMergeProposals (known as a pull request on GitHub, but~~
e.g. FreeBSD calls bug reports PRs, or problem reports, so that's confusing)

What is that? (make me pretty)
<ovidiu-florin> how does okular get split in okular-dbg -src -dev ..... etc... ?
<ovidiu-florin> when? where? by whom?
<yofel> there is no -src package, the source package is what you make yourself by hand and upload to launchpad
<yofel> the thing that consists of the .orig.tar + .debian.tar + .dsc
<yofel> the binary packages are then created from information inside the control file, and information from the various .install etc. files in the debian/ folder
<yofel> ovidiu-florin: I really recommend that you read https://www.debian.org/doc/debian-policy/
<yofel> that's in some sense the technical specification how the packages are structured, and what each of the files in debian/ is for and all the files in them etc.
<yofel> at least the first 5 chapters are elemental knowledge

PPA Naming scheme: https://community.kde.org/Kubuntu/PPAs <-- needs new graph of package flow <-- How do you like my proposed treatment?
Staging: (before Testing) [DO NOT USE ON REGULAR SYSTEMS]
staging-plasma (sudo add-apt-repository ppa:kubuntu-ppa/staging-plasma)
staging-frameworks (sudo add-apt-repository ppa:kubuntu-ppa/staging-frameworks)
staging-kdeapplications (sudo add-apt-repository ppa:kubuntu-ppa/staging-kdeapplications)
staging-misc (sudo add-apt-repository ppa:kubuntu-ppa/staging-misc)

Testing: (This is usually packaged before a call to !testers in #kubuntu-devel)
ppa-landing (sudo add-apt-repository ppa:kubuntu-ppa/ppa-landing)
backports-landing (sudo add-apt-repository ppa:kubuntu-ppa/backports-landing)

Public:(The release of the software after it has been tested in Testing and ready for our users to have)
Backports: (new upstream releases)
backports (sudo add-apt-repository ppa:kubuntu-ppa/backports)
beta-backports
Updates: (point release bug fixes)
ppa (sudo add-apt-repository ppa:kubuntu-ppa/ppa)

Set-up: http://packaging.ubuntu.com/html/getting-set-up.html
Environment:
in ~/.bashrc add the following:
export EMAIL="your@email.com"
export DEBEMAIL="your@email.com"
export DEBFULLNAME="My full name is Foo Bar"
<yofel> ovidiu-florin: something I just realized for the packaging docs, I strongly recommend every packager to have a ~/.dput.cf with this content:
<yofel> [DEFAULT]
<yofel> default_host_main = ppa
<yofel> this isn't that important for beginners, but really important for every person with primary archive upload permissions
SSH:
Your ~/.ssh/config should be set up to use your SSH keys. Example (needed by KA):
Host depot.kde.org
User ftpubuntu
IdentityFile ~/.ssh/my_id
For pushing/pulling Kubuntu Packaging git repos on Launchpad:
Host git.launchpad.net
User yourlaunchpadid
IdentityFile /path/to/your/private/ssh/key
pbuild: (https://wiki.ubuntu.com/PbuilderHowto)
Install these: sudo apt install pbuilder debootstrap devscripts pigz eatmydata bzr
get lp:~kubuntu-packagers/pbuilder/pbuilder-hooks and lp:kubuntu-dev-tools (bzr branch lp: ...)
save http://paste.ubuntu.com/14286057 as .pbuilderrc and set HOOKDIR to point to the pbuilder-hooks clone
Create animage (archive) for xenial: sudo -E dist=xenial pbuilder create
To login use: sudo -E dist=xenial pbuilder login
To update use: sudo -E dist=xenial pbuilder update
To test get the sources of a package (e.g. apt-get source pkg-kde-tools) and then pass the .dsc file to pbuilder build. You'll find the results in /var/cache/pbuilder/xenial-amd64/result/

<clivejo> you will need to configure pbuilder to use some Launchpad PPA's
<clivejo> if its plasma 5.5.2 you want, you would need to add the staging PPA's
<yofel> when you add PPA's to the chroot, beware of what you add. If the PPAs you add don't match the dependencies of the PPA on LP, then you will waste your time debugging weird failures
<yofel> always make sure you know what environment you're building in
OR try sbuild: https://wiki.ubuntu.com/SimpleSbuild
To build using packages available in KCI, while in the repo folder you want to build and with the upstream tarball in the folder up one level (use get-kci-tarball from KA):
sbuild -d name-of-schroot --chroot-setup-command="echo 'deb http://ppa.launchpad.net/kubuntu-ci/unstable/ubuntu $1 main' >> /etc/apt/sources.list && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 432062D1 && apt update"

For easy cloning of LP git repositories, put this in your ~/.gitconfig (where USER is your launchpad username):

[url "git+ssh://USER@git.launchpad.net/"]
insteadof = lp:
[url "git+ssh://USER@git.launchpad.net/~USER/+git/"]
insteadof = lpme:
[url "git+ssh://USER@git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/"]
insteadof = kubuntu:
[url "git+ssh://USER@git.launchpad.net/~USER/kubuntu-packaging/+git/"]
insteadof = kubuntume:

example of use:
git clone kubuntu:plasma-sdk
make some changes and commit them
git push kubuntume:plasma-sdk
submit a merge proposal:
go to https://code.launchpad.net/~USER/kubuntu-packaging/+git/plasma-sdk/+ref/BRANCH/+register-merge
use lp:~kubuntu-packagers/kubuntu-packaging/+git/plasma-sdk for your target repository
use BRANCH for target reference path
enter in enough information to convince reviewer it makes sense— be verbose and include links (e.g. KCI failures)

Full git access documentation can be found at https://help.launchpad.net/Code/Git

https://techbase.kde.org/Contribute/Get_a_Contributor_Account
https://wiki.debian.org/Alioth/Git
http://packaging.ubuntu.com/html/getting-set-up.html

Process (Packaging a New Release of frameworks, plasma or apps):

Overview: https://drive.google.com/open?id=1SoO83PepBkU_kkbLL2eIeVDYYDzv6OEA1npVoCp7wHo

Check KDE release cycle and make sure the tarballs are available on depot.kde.org
(If you don't have access to depot yet, ask one of the senior packagers to add your ssh key)
Check KCI and fix any outstanding packaging problem (This will save you a lot of time fixing problems later) (This should be done on a regular basis, regardless of the KDE release cycle)
Check in IRC channel #kubuntu-devel to make sure nobody else is working on the same release (often mentioned in the topic and on https://notes.kde.org/p/kubuntu-ninjas)
Clone KA Tools, all tools implement --help for their parameter documentation and perform the steps in the README https://git.launchpad.net/ka/tree/README
KCI needs to be paused whilst executing some of the KA tools, those tools will warn you and fail if KCI is still running ( http://kci.pangea.pub/job/mgmt_pause_integration )
Execute the script staging-upload, an example follows: (Why? What does this script do?) This script grabs the tarball source code from depot and the packaging from debian git. It added new entries in the changelog and runs a debuild -S for all the packages in the suite. Problem packages are moved to /manual to be fixed by hand and the staged packages are moved to /upload ready for upload to LP.
./staging-upload -d xenial -v 5.5.2 -r plasma -t ~/workspace/plasma
Where -d is the distribution (xenial in this case), -v is the version, and -r is the release type (frameworks, plasma, applications), -t is the working directory
Provided all is well, the script will complete. If the script encounters a problem it moves the problem package to a folder called manual in your working directory. These packages will have to be manually fixed and moved to the upload folder.
Upload the packages to LP. Example: (script from step 7 is called staging-upload, but it does no uploading?)
dput ppa:kubuntu-ppa/staging-plasma *.changes
Run this command and don't ask why or what it does, it's magic. Also it only works if you work with plasma.

git-push-all -t  ~/workspace/plasma/

To help with automated QA of the LP package builds, update the appropriate config file in KA build-status-conf/ and push the change, that will then generate a status overview page in http://qa.kubuntu.co.uk/ppa-status/ that will show you what packages need fixing.
As packages build in LP they may fail because packages they depend on havent been published yet. You can retry the builds via LaunchPad's web interface or use one of the KA Tools. To retry PPA builds use "while true; do ./kubuntu-retry-builds -r plasma --ppa=kubuntu-ppa --ppaname=staging-plasma --force; sleep 1200; done" Which will basically loop retrying the builds forever!

Build status legend:
Blue | not built yet (needs building, depwait, building, uploading, ...)
Green | OK (or at least auto-qa ok)
Orange | minor things like lintian warnings, new symbols, etc.
Red | failed to build from source or otherwise very broken

Fixing a package (example):
Get a failing package from http://qa.kubuntu.co.uk/ppa-status/applications/build_status_15.12.0_xenial.html~~
Check the logs, find why it failed.~~
Knavalbattle failed due to missing dependencies.
Create a work directory for this package and cd into it
Download the source for the packaging from: http://anonscm.debian.org/cgit/pkg-kde/applications/knavalbattle.git/https://git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/knavalbattle and cd into it
Switch to the branch kubuntu_xenial_archive as this is the branch we're following here.~~

In case of dependency issues:
Add the missing dependencies to debian/control
Do not use virtual packages (ask yofel why)

In case of Symbols problems: (https://pkg-kde.alioth.debian.org/symbolfiles.html)~~
get the build logs from QA and extract them~~
run pkgkde-symbolshelper batchpatch -v 4:15.12.0 -c amd64,i386 buildlog_ubuntu-xenial-* to auto patch the symbols files (see man pkgkde-symbolshelper for more info)
<sgclark> README in kubuntu-automation has the command you need to batchpatch symbols. Run c++filt on any MISSING symbols. Then you need to look up the result on KDE API and see if it is a public symbol. If it is ( this is where I cry to someone more knowledgable like yofel

<ovidiu-florin> https://paste.kde.org/pcusi5k79/5sjxr4
<yofel> first of all, the (optional) ones are always ok to be removed
<yofel> that's usually internal GCC symbols that leak into the public namespace
<ovidiu-florin> ok, how do I remove them?
<ovidiu-florin> manual edit ?
<yofel> yes
<ovidiu-florin> yofel: first one is not optional
<ovidiu-florin> how do I check it?
<yofel> ovidiu-florin: it indeed is not, what you missed this time is that the soname version of libokular was changed from 6 to 7
<yofel> so as the interface version changed, any kind of changes are OK
<yofel> clive took care of that before you took over okular
<yofel> otherwise your best bet is to run c++filt on the symbol to look where it likely came from
<yofel> then look up the source history to find out what happened to it and/or talk to upstream
<clivejo> yofel: does that happen much?
<yofel> esp. the libs shipped in applications like to change the version every release, i.e. libmarblewidget, kdcraw, kipi, etc.
<yofel> frameworks and some other cases have an ABI stability guarantee, there you need to watch out
<yofel> and in the worst case, when upstream messes up we can still work around it by using the DebianAbiManager, but that's something for another day

<ovidiu-florin> it seems to be missing the vtable for Okular::ConfigInterface
<ovidiu-florin> search for _ZTVN6Okular15ConfigInterfaceE
<yofel> right, it does say so
<yofel> +#MISSING: 4:15.12.0-0ubuntu1~ubuntu16.04~ppa3# (optional)_ZTVN6Okular15ConfigInterfaceE@Base 4:14.12.0
<yofel> so, what's the question?
<ovidiu-florin> the class has virtual functions
<ovidiu-florin> https://lxr.kde.org/source/kde/kdegraphics/okular/interfaces/configinterface.h
<ovidiu-florin> so it should have a vtable
<yofel> and are they exported by gcc as part of the public interface? (That's all the symbol files care about)
<ovidiu-florin> they are public funtions
<yofel> and if you build the binary, "nm -DC <libfile>" will show the symbols?
<ovidiu-florin> what do I give as <libfile>
<ovidiu-florin> ?
<ovidiu-florin> and I assume I run that
<yofel> libokularcore or how it's called
<ovidiu-florin> in the pbuilder
<yofel> the symbol files don't care about the specifics of the C++ API, they only care about what symbols actually get exported by g++
<yofel> which is one reason why they're insufficient to do vtable tracking, you need at least abi-compliciance-checker for that

• ovidiu-florin researches "g++ export symbols"

<yofel> it's the symbols you see if you if you run "nm -D" on a lib, like, try "nm -D /usr/lib/x86_64-linux-gnu/libKF5Archive.so.5"
<yofel> if you handle symbol files, you don't need to go into API specifics. What you need to make sure is that your file matches the auto-generated output of dpkg-gensymbols
<yofel> that's why you see a diff in the log, it's the diff between the file in the packaging, and the auto-generated one

Test your changes:
Use the wrap-and-sort tool to sort and cleanup the debian/control file, if you changed it
Use uscan --download-current-version --destdir=../ to download the project sources
Now build the package with debuild -S (Check the debuild man page to understand what it does)
Now test your changes and build (compile) it in the pbuilder you've set up for xenial: sudo dist=xenial pbuilder build ../knavalbattle_15.12.0-0ubuntu1.dsc
If it fails read what it says (why it failed). Fix the issue and try again.
Usually if it fails, you are dropped down to a shell inside the pbuilder environment.

if you get an error like "cp: cannot stat 'debian/tmp/usr/share/applications/kde4/knavalbattle.desktop': No such file or directory" and you're building a KF5 app, it might be because the .desktop file for KDE4 does not exist anymore, So in this case comment the mention of that file in debian/*.install files. Ask a senior packager for guidance here.

Finishing:
When you're all done and all works, run dch to update the changelog file
Get a patch of your changes and send it to the devel team

• <yofel> the procedure for getting something uploaded that you cannot upload yourself is https://wiki.ubuntu.com/SponsorshipProcess

Merge fixes
This will display under "Merge FIX" on KCI. Usually it's a simple matter. The merge order is:
archive → backports → stable → unstable

Example:
git clone kubuntu:some-package
git checkout kubuntu_archive
git checkout kubuntu_backports
git merge
fix any merge problems
git checkout kubuntu_stable
git merge
fix any merge problems
git checkout kubuntu_unstable
git merge
fix any merge problems

IDEAS:
Perhaps we should update these pages: https://help.ubuntu.com/community/KMailGPGAgent https://userbase.kde.org/KMail/gpg and point to them for setting up gpg-agent (yofel)

Useful links:
Porting to KF5 status page http://developer.kde.org/~cfeck/portingstatus.html

Once we release, announce on: Kub.org/Kub-wire/KDE::Kub-Release, Social Media (please list the account-names and who has access), https://community.kde.org/Plasma/Packages#Kubuntu