Problem definition
============
Since recently we adopted builds of ffmpeg, heif, webp and other heavy libraries in Krita. Which raised the problems we tried to ignore for ages.
1) Our deps builds are not "granular". We build them in one run/blob, and that is a problem now. The full build take 54 minutes on 16-core server (binary-factory). Performing these deps build locally to set up a development environment is even more problematic (since few people have 16-physical-core CPU).
2) The whole deps setup is really fragile. Our CMake scripts sets up different environment for build and install commands (hello, meson and its PYTHONPATH!). It means that when you try to build some dep manually, you can break the whole build tree very easily.
3) The whole setup is really hard to maintain. I spend about 10% of my time just fighting the deps build issues.
4) Ideally, we need multiple prebuilt versions of Qt available from CI: normal, debug and asan. It is impossible to achieve with the current setup because of the absence of granularity. Doing 3x2 54-minutes builds on CI would just paralyze the KDE's CI system.
Requirements for the deps management system
===============================
Here is the draft of the requirements to the management system:
1) [granular builds] The dependencies should be built in a granular way. That is, when we change something in Qt, we shouldn't rebuild FFmpeg.
2) [prebuilt deps] The prebuilt packages for the deps should be available for the developers. At least for Linux and Windows.
3) [granular downloads] Ideally, the deps pack for developers should be downloaded in a granular way as well, but one-blob solution will also work (since the whole pack of compressed deps is about 200MiB).
4) [work on deps] The developer should be able to rebuild one dep (e.g. Qt or FFmpeg) easily without manual composition of the configure/build commands. And it shouldn't break the dev-environment! ;)
5) [reuse build tree] Ideally, if the developer builds all the deps locally, he/she should be able to reuse the build trees for his local changes in a dependency. Obviously, we need to fix bugs in Qt quite often.
6) [multiple builds of qt] We need a way to quickly switch the version of Qt between normal, debug and asan.
Available options and evaluation
=====================
We have multiple options to solve these issues:
CMake-based scripts
----------------------
Our current CMake system is "almost perfect". It fits our workflow quite well, except that it is not granular and is quite hard to maintain.
Craft
-----
KDE has a home-grown package management system called [[ https://community.kde.org/Craft | Craft ]]. I tried to use it but faced with the fact that there is **extremely little** documentation about that. And the only manual about installing it required me to run PowerShell (which I'm not very familiar with) **with Administrator privileges**. I gave up on this step.
vcpkg package management system
----------------------------------------
I briefly looked into it. It seems like:
- it is quite MSVC-centric
- (it seems like) there is no readily available solution for storing the binary packages, we need to invent/host our own.
Conan package management system
----------------------------------------
I have managed to give [[ URL | a small test ]] to Conan system. And here is what I found:
1) It is quite easy to build packages in a granular way with Conan (obviously)
2) We may have multiple binary-compatible versions of the same package, just declare custom binary compatibility rules in `package_id()`
3) The main idea of Conan is:
- every package is first installed into **its own root folder**
- then Conan generates a custom CMake/Meson configuration file which adds paths to these packages. There are three (relevant) generators available:
* "CMakeDeps": generates `XXX-config.cmake` file for each package which can be consumed by CMake to find the package. Please take it into account that this method basically **overrides** the original `XXX-config.cmake` files of the packages. Which might be a problem in some cases.
* "CMakeToolchain": generates `conan_toolchain.cmake` and `CMakePresets.json` files. These files just adjust CMAKE_PREFIX_PATH (and other variables), so that CMake could find original `XXX-config.cmake` files of the packages themselves.
* "MesonToolchain": generates a the same toolchain file for Meson build system
- [note] these generators are usually enough to **build and link** the application, **but** they are usually not enough to **run** the application. The reasons are:
* Qt has a plugin system. These plugins are usually looked at a position relative to QtCore.dll or krita.exe (TODO: I'm not sure what locations are actually looked at)
* KDE has plugins, which should also be discoverable
* Qt and KDE have translations, which should be discoverable
* Krita installs its own plugins in its own location
* PyQt and Krita install .sip files into a common location
* libmlt has a plugin system. The plugins are searched relative to `libmlt.dll`. Currently, Krita's plugin is built alongside the MLT itself, but we might want to build it separately one day
- To solve the plugins discovery problem, we can merge all the deps into a single tree using "deploy" functionality of Conan. Basically, Conan can merge all the packages into one tree. See `deps-deploy/flat_deply.py` in the linked repo.
* the problem of `flat_deploy.py` approach is that Conan stops managing this merged folder. That is, if you change any deps (through Conan), you will have to regenerate the whole merged folder. Which might be a trouble in some cases (see below).
4) **[PROBLEM 1]** The first problem I noticed is that Conan moves/relocates package build trees when the build is done. It basically makes requirement `[reuse build tree]` broken. Theoretically, we could move the folders back into the temporary location or hack conan (it's raw python) to keep the build trees intact, but I'm not sure that would work.
5) To fulfil `[work on deps]` requirement, Conan has the concept of "editable packages". What it does is:
* it builds the package in a custom user's location **without installing or deploying** it (only `make all` without `make install`)
* then it generates `XXX-config.cmake` or `conan_toolchain.cmake` file that links the consumers to the **build directory** (not install/deploy directory) of the package
* **[PROBLEM 2]** Linking to the build folder is really a clumsy approach. Firstly, the plugins will not work. Secondly, there might be some weird issues with Qt generating headers on the install stage (though I guess Qt actually generates headers on the configure stage using that dreadful perl script).
* **[PROBLEM 3]** If we decide to use "deploy" strategy, this "editable" package will not link easily into the workflow. Because the we will have to **regenerate** the whole deploy folder after every change in the editable package (TODO: I'm actually not sure if deploy step is applicable to the editable packages).
That is, Conan somewhat solves the problem of building and deploying the deps in a granular way. But it has some issues with hacking on those deps. I have a feeling that these issues may be overcome somehow, though I'm not sure how at the moment.
Conclusion
=======
It seems like Conan fulfils most of the requirements, except of the "work on deps" ones. Which is quite bad. But I have a feeling that these problems can be resolved somehow, it just needs more investigations work...
| Requirements | CMake External Project | Conan | Something else? |
|--------------------------|------------------------|---------------------------------|-----------------|
| [granular builds] | no | yes | ? |
| [prebuilt deps] | yes | yes | ? |
| [granular downloads | no | yes | ? |
| [work on deps] | yes | complicated, can be improved? | ? |
| [reuse build tree] | yes | needs hacking conan? | ? |
| [multiple builds of qt] | no | yes | ? |