That would be fantastic, Speaking for Kdenlive, that would relieve the developers and help us provide better releases since we have several faithful users that are waiting for this to happen to test & provide feedback on the development version.

This now has initial support. I hope to get my tooling moved over this week.

Any update on the status of AppImage CI ?
Thanks

We are getting the repos/tooling etc in place. Sorry I have been swamped and appimages are now free time work, which I have not had much of. Sincere apologies.

There has been some discussion at Akademy this year on how to implement this.

Based on my understanding of what is required, at the moment all we have to do is:

1. Build a project as normal, installing it and it's dependencies to a given install-prefix
2. Run the linuxdeploy command with the right parameters to generate an appimage from that.

In some instances we may need to hook parts of this process to allow for customisations, but that can be done fairly trivially on a project-by-project basis with a bit of shell script (and we can point to developers of individual apps how to do that and they can look after that bit)

About the only tricky problem we might have is getting newer versions of dependencies like Qt in an older distribution like CentOS 6 or Ubuntu 14.04 but if those end up being an issue we can always compile those ourselves as part of the image build process if a PPA or equivalent is not available.

Talking about Kdenlive, we need a recent Qt since we increasingly use Qml so Qt 5.11 would be best. We also depend on many multimedia libraries that need recent versions. Currently I have home made build scripts that build the following on CentOs 6:

Stage 1: base dependencies and requirements

libjpeg
jasper
libpng
libtiff
fft3w
qt
exiv2
gettext
ruby
yasm
nasm
KF5 frameworks
breeze

Stage 2: Multimedia libraries

libsndfile
libsamplerate
alsa-lib
sdl2
ladspa
libx264
libx265
libvpx
ffmpeg
cairo
harfbuzz
pango
gdk-pixbuf
gtk+
opencv + contrib
vid.stab

Stage 3: Kdenlive direct dependencies

frei0r
mlt
kdenlive

So on my side, the best would be to have a CentOs image or something similar with a recent Qt and KF5 version already build. Then, ideally some editable scripts (editable by the project developers) building and installing the stage 2 and 3 dependencies, with the possibility to manually trigger a rebuild.

About the only tricky problem we might have is getting newer versions of dependencies like Qt in an older distribution like CentOS 6 or Ubuntu 14.04 but if those end up being an issue we can always compile those ourselves as part of the image build process if a PPA or equivalent is not available.

We often recommend https://launchpad.net/~beineri for getting up to date Qt binaries. Unfortunately, Qt 5.11 is not available for trusty at the moment, but I guess if someone requested it by mail, perhaps the author will set up support for trusty again.

So on my side, the best would be to have a CentOs image or something similar with a recent Qt and KF5 version already build. Then, ideally some editable scripts (editable by the project developers) building and installing the stage 2 and 3 dependencies, with the possibility to manually trigger a rebuild.

I guess whatever you implement there, you should make sure local tests are possible without any major additional work.

As a baseline, Ubuntu trusty should suffice for now by the way. Nobody uses OSes from 2011 any more, at least not for the desktop. Most apps can still be built on trusty anyway, without the need for updating _all_ the dependencies.

As @bcooksley suggested, if you need to build more up to date dependencies, ideally, you can install all the software into a separate prefix, which can be used via pkg-config, CMake etc. again by exporting the respective environment variables before running the project's build scripts.

In any case, if you want to get in touch with the AppImage team, please feel free to join #appimage on Freenode.

I guess having a base AppImage with latest Qt and KDE frameworks would be save resources if there are several KDE Applications building AppImages, but otherwise I can for sure build everything and install in a prefix...

The plan at the moment is to have a base system image with the most common things already available yes.

I just saw that KMyMoney seems to have an AppImage built on binary.kde.org. How is it done ? from what I saw in their git repository, it seems like this is simply based on bash scripts ?
We really could benefit from some help on that side. Would it be possible to use a simple pipeline to build the AppImage on the CI ?
I had a quick look at Craft but honestly don't have time to learn how the system works, and since I already have some scripts to build my AppImage locally it would be much much simpler on my side if I could simply put some scripts on our git repository that are then executed on a docker container by the CI or something similar...

Any hint on how this works for KMyMoney ? Thanks a lot.

Sorry last year was madness for me,
With that said, I have more time now.
I failed miserably with craft even getting anything to build, much less integration with the appimage stuff. Python hates me.
I have resurrected my work from BS appimages. I am more than happy to use it on binary factory to produce appimages relatively easily , and will even help with the scripts. I am very close to ready should it be decided this is an acceptable solution.
Cheers, Scarlett

@scarlettclark I can provide assistance of course, just let us know what we can help you with. I'd offer to collaborate on developing the scripts for craft or whatever system you use, they might come in handy for non-KDE projects as well. Also, it's always good to see how people use our tools, as we then get an impression how we can improve them.

@azubieta might be able to help, too (maybe not this week).

@TheAssassin yes, I was using them for non KDE applications as well.
@ everyone - I guess what I am saying here is I am more than happy to use my existing tools ( ruby ) and help with appimages. I just simply don't have the time to learn python right now. Not to say this is never going to happen, just not now.
Cheers,
Scarlett

@scarlettclark since I'm a passionate Pythoneer, but have no clue about Ruby (I can read most of it, but like with PHP, I try to avoid it as good as I can), I guess we could team up to port your stuff to Python?

Sorry, grumpy warning...

I guess I wasn't clear - I just simply don't have the time to learn python right now. This has been the sole holdup on all of this.
I am offering up my current tooling and will help with recipes etc.
If someone wants to take over appimages in binary factory with a python solution, feel free. But please let me know ASAP so I don't spend any more time on it.

Thanks for understanding.
Scarlett

@scarlettclark is your (old?) code available somewhere so I could look a bit into it? I'd like to see how much work it'd be to maintain.

My current code is at https://anongit.kde.org/sysadmin/appimage-tooling

plus packaging is at http://anongit.kde.org/appimage-packaging

I have more packaging done in github that I will bring over should my solution be accepted.

I just want to say that I put many hours into this ( over a year + recent improvements )

I know I need to learn python, as it seems to be favorite around here, but this works and is available now.
I will maintain it, and eventually port it to python when I have some more experience in it. Ideally moving the build to craft.
Scarlett

Back to my original question, KMyMoney has AppImages built on the binary-factory right now, using what seems a very simple bash script system:
https://cgit.kde.org/kmymoney.git/commit/packaging/linux/appimage

It would really help us to have a working solution soon, whitout requiring to learn new languages/syntaxes or other complicated workflow.
So to achieve this goal for Kdenlive we need either :

• to be allowed to use the same bash script system as KMyMoney and have it run on binary-factory. I can probably easily adapt my own build scripts which are very similar and would not require too much investment
• or have strong support from the sysadmin team to help us understand, setup and maintain an alternative built system in the next weeks.

Can I please have some feedback on this from the sysadmin team ? Sorry I don't want to be rude but the workload on Kdenlive is huge and having automated AppImage builds would be a big relief.
Thanks

I completely understand your frustration. It has been a long while since we had working appimages for you. Create those bash scripts and I will either integrate them into my system or the current kmymoney way.
Scarlett

Having Kdenlive also use the same "simple" bash system as KMyMoney (and Krita) would be non-ideal mainly as it's fairly resource inefficient if a dependency needs to be rebuilt (because it requires a full rebuild of everything).

Given that with the exception of things Scarlett is working on, we don't have Appimage generation stuff at the moment, nor the Sysadmin time to look into it (hence my suggestion others look into it) the most immediate path forward would be to allow the same bash scripting to be used by Kdenlive as well.

Should more projects start to look into this though I would very much prefer that developers assist in building and maintaining generalised Appimage tooling, simply due to scalability concerns.

We run those scripts in a Docker container, which can be fetched from Dockerhub under the name kdeorg/appimage-base. It is Ubuntu 14.04 based (for maximum compatibility with older systems). To try to minimize scalability limitations as much as possible we break things up into two parts:

1. Building the Dependencies (Qt, other libraries, Frameworks, etc)
2. Building the actual application itself

If you can prepare and test everything in that environment, we can go from there.

Thanks for the answers. I am working on adapting my bash scripts to work on the Ubuntu 14.04 image mentionned and will ping you when I have it working locally.

Ok, I have now prepared the scripts and they produce a working AppImage using the kdeorg/appimage-base docker image (I still need to include a few additionnal dependencies).
The scripts are in kdenlive's git master, under packaging/appimage:
https://cgit.kde.org/kdenlive.git/tree/packaging/appimage

build-dependencies.sh
build-kdenlive.sh

I also have a script to build the AppImage package. Kdenlive's AppImage must be created with the AppImageTool utility since we need to set some environment variables to get all our stuff working. I didn't find the linuxdeployqt not the appimagetool binaries in the kdeorg/appimage-base docker container. However the KMyMoney build-image script calls linuxdeployqt. Where does this come from ?

My appimage build script is also here:
build-image.sh
But not sure where the linuxdeployqt/appimagetool binaries will come from.
Please let me know what I can do to move forwards.

Best regards

I've just had a quick look through the code and have noticed the following:

# Read in our parameters
#export BUILD_PREFIX=$1
#export KDENLIVE_SOURCES=$2
export BUILD_PREFIX=/build
export KDENLIVE_SOURCES=/kdenlive
export DEPS_INSTALL_PREFIX=/external

You'll need to reverse this as the Binary Factory needs to control the locations that are used as the build prefix, as well as let the script know where the source code will be found.

Also, I see you have added a pile of apt-get stuff at the top - this won't work as the script will not be executed as root.
Please instead send a patch for https://cgit.kde.org/sysadmin/ci-tooling.git/tree/system-images/appimage-base/Dockerfile and we'll get the necessary packages added.
(A large portion of the list you have there will already be present)

In regards to linuxdeployqt this is provided through the setup-utilities script.
Please see https://cgit.kde.org/sysadmin/ci-tooling.git/tree/system-images/appimage-base/setup-utilities

If you require appimagetool then please let me know how that can be easily sourced (I assume there is an equivalent for that in terms of being downloaded as an appimage) and we can get the installation for that integrated as part of setup-utilities

@bcooksley it would be nice if you could provide the more modern linuxdeploy, which supersedes linuxdeployqt. It's plugin based (https://github.com/linuxdeploy/awesome-linuxdeploy), providing an extensive framework for plugin developers.

Most important plugins:

• https://github.com/linuxdeploy/linuxdeploy-plugin-qt
• https://github.com/linuxdeploy/linuxdeploy-plugin-appimage

linuxdeploy's plugins are designed from ground up to be configurable. As the plugins are standalone executables, all configuration is done through environment variables. linuxdeploy's plugin system is supposed to eliminate the need for calling tools like appimagetool directly. If you need to perform intermediate steps, you can simply call linuxdeploy a few times, e.g., with different --plugins.

I'm pushing this tool a bit among several KDE applications, as I consider it to be easier to use and get started with.

There's a basic user guide in the AppImage documentation, and the packaging guide is using it.

The ultimate goal is to provide plugins for any additional resources you might need for your AppImage. We have a plugin setting up a portable miniconda environment in a standardized way in an AppDir, providing applications in AppImages with a reliable Python interpreter (can also be used to bundle apps written in Python, demonstrated by Pext and FreeCAD. Hence the idea to develop a KF5 plugin or e.g., a plugin that might replace the similar cp myqtplugin.so && patchelf ... myqtplugin.so calls, etc.

linuxdeploy plugins are distributed as self-containing AppImages (e.g., linuxdeploy-plugin-appimage contains a (maybe not so recent but stable) copy of appimagetool. So, by downloading those plugins from the release pages, you get all the tools you may need.

We can certainly look into providing the newer tooling.

The main reason why we haven't so far is because nobody has requested it, and I'm somewhat reluctant to make developers (in this case of Krita and Kmymoney) adjust their tooling when what we have currently works okay.

@mardelle is your tooling compatible with linuxdeploy as outlined above?

@bcooksley sure, as said, a WIP project on my side. I hope people will see the advantages of using linuxdeploy.

The syntax differs for obvious reasons, but we can help anyone to port their scripts. Most likely, less than 10 lines need to be changed, I'd say, probably resulting in less than before.

Ok so the Kdenlive build scripts should be ready for testing in our git master. I have submitted a patch for the dockerfile:

https://phabricator.kde.org/D17153

The build-image.sh script currently uses linuxdeployqt and appimagetool. appimagetool is embedded in the linuxdeployqt AppImage so it should be directly usable on the docker container without modifications. However reading the setup-utilities script, it extracts linuxdeployqt to /root/tools, which is not in the default paths so I don't fully understand how it can be directly accessed but if it linuxdeployqt works, appimagetool should work too.

I was not aware of linuxdeploy. I had a quick look but didn't find a way to define environment variable at the application startup, which is required for kdenlive (for example MLT requires an MLT_REPOSITORY variable at startup otherwise it won't find its libraries).

Current image build script:
https://cgit.kde.org/kdenlive.git/tree/packaging/appimage/build-image.sh

I am open to test linuxdeploy if you can give me some hints, but if possible would love to have my current scripts tested asap with the existing setup.

@bcooksley does the nightly job always run on the same container or does it start from a clean AppImage base each day ?

All Linux containers on both the CI system and Binary Factory are started fresh for each job, so nothing will be carried over.

In terms of linuxdeployqt/appimagetool, they're extracted to ~appimage/tools/ and the appropriate folder(s) are added into path as part of the Jenkins Pipeline which runs all of this.
See https://cgit.kde.org/sysadmin/binary-factory-tooling.git/tree/custom-jobs/kmymoney/KMyMoney_Nightly_Appimage_Build.pipeline#n56

The Dependency Build is now underway - https://binary-factory.kde.org/job/Kdenlive_Nightly_Appimage_Dependency_Build/3/console
Assuming this succeeds, the automatic triggering process should commence the build of the first Appimage on the Binary Factory sometime in the next 24 hours.

Thanks a lot Ben. Compilation failed because of a hardcoded path in the build-dependencies script. I have pushed a fix in git master, the job will need to be restarted... or is it automatically restarted when the script file is changed? Would be very handy...

Because of the high resource cost of the dependency build jobs they are only triggered manually.

I've now done this.

I suppose this task can be closed.