Sort out Python bindings generation
Open, Needs TriagePublic

Description

Based on an IRC conversation today; this applies to cmake-extra-modules (which contains the CMake-fu for Python bindings generation) and kitemmodels (which use that -fu to generate bindings). The idea is to sort out the toolchain for target distributions -- there seem to be Ubuntu-specific things in there.

Source material: https://steveire.wordpress.com/2016/05/18/generating-python-bindings-with-clang/

  • does ecm_generate_python_binding() work for everyone?
    • does it handle LLVM versioning?
    • does it handle Python versioning?

Related bugs:

Distro-raised issues:

  • properly detect the python install path and the clang library name via cmake-foo. // The hack on stackoverflow seems to work http://stackoverflow.com/questions/1242904/finding-python-site-packages-directory-with-cmake But I'd be surprised if that's really the least-awful way to do it. (doesn't *quite* work on Arch, because it assumes 'python' to be python2)
  • the clang code also seems to need fixing for FreeBSD, something on the lines of find_library(libclang_LIBRARY clang PATHS /usr/local/llvm3${_LIBCLANG3_FIND_VERSION}/lib )
adridg created this task.Jan 2 2017, 11:01 PM

Thanks for filing this.

I can be useful in the role of reviewer for this task, but not as the one to solve all the problems on all the distros. I would need to rely on people who are familiar with other distros and platforms to provide patches.

adridg added a comment.Jan 4 2017, 2:11 PM

I've massaged the code for finding libclang on FreeBSD a little; it now finds the library, but that's not a good way to do it because of the risk of mismatching libclang with the clang you're actually using (e.g. having llvm39 installed while using clang++38). So there would have to be more done to get this right:

  • check $(CXX) is clang >= 3.8
  • look for libclang in $(CXX) --print-search-dirs

Well, except that libclang doesn't live in the directory printed that way :| OTOH, llvm-config38 has the right information, but I'm not sure how to get from the clang version to the llvm-config version (in particular, system clang on 10.3 is clang 3.4, but there is no llvm-config34 or unversioned llvm-config). Getting to the right ClangConfig.cmake from the clang-version is tricksy, too.

That said, there's no libclang-python bindings in ports, so the next step in the toolchain which does ''import clang'' fails anyway. So that's something that would need to happen (independently of adequately searching for libclang) to move this forward for us.

Another issue that needs to be dealt with, now that more frameworks ship bindings, is the multiple copies of PyKF5/__init__.py, which will make frameworks packages conflict with each other.

skelly added a comment.Jan 9 2017, 8:19 PM

@arojas I raised that issue in June: http://comments.gmane.org/gmane.comp.kde.distributions/92 and it came up in November: https://www.mail-archive.com/kde-frameworks-devel@kde.org/msg39376.html

The suggestion from Harald is not quite clear to me. Do you have a suggestion?

skelly added a comment.Jan 9 2017, 9:35 PM

If anyone else here can reproduce the issue in https://bugs.kde.org/show_bug.cgi?id=374801 I would appreciate if you could dig into it and figure out what the problem is.

Update on bug https://bugs.kde.org/show_bug.cgi?id=374801 , myself and @arojas fixed it. Packagers may want to cherry-pick the 8aa6843404f9c6faef66cb9c76358158eafc1af1 commit in ECM, and ed1b9ce2bb2a2e51410e0a1754a72c110010a6a0 if you encounter other problems to diagnose.

Update on the installation directory issue. Instead of using 'dist-packages', the default is now 'site-packages', and packagers can now customize the install location for a package by passing

-DKDE_INSTALL_PYTHONBINDINGSDIR=dist-packages

for example to cmake

kitemmodels/build$ cmake . -DKDE_INSTALL_PYTHONBINDINGSDIR=dist-packages
kitemmodels/build$ make install
...
/home/stephen/dev/src/kitemmodels/build/prefix/lib/python2.7/dist-packages/PyKF5/KItemModels.so
...

I added https://commits.kde.org/extra-cmake-modules/29b9db8d67c67d4d1303e8dd9804f698fbaefbe1 in an attempt to resolve the issue of multiple packages installing the same __init__.py file.

Please try out and comment on that as a solution to the problem. My understanding is that you would be able to run that script while packaging ECM and be able to create a separate package for the file. That way, each framework would have a source-depends on ECM (which it already has anyway) instead of all frameworks having a source-depends on kitemmodels, or some other package which provides the 'blessed' __init__.py.

If that's a solution that works, perhaps the KDE tarball creation script could automatically create a separate tarball for that file either. Please comment on this.

In T5016#74679, @skelly wrote:

I added https://commits.kde.org/extra-cmake-modules/29b9db8d67c67d4d1303e8dd9804f698fbaefbe1 in an attempt to resolve the issue of multiple packages installing the same __init__.py file.

Uhhhh, good stuff.

Please try out and comment on that as a solution to the problem. My understanding is that you would be able to run that script while packaging ECM and be able to create a separate package for the file. That way, each framework would have a source-depends on ECM (which it already has anyway) instead of all frameworks having a source-depends on kitemmodels, or some other package which provides the 'blessed' __init__.py.

If that's a solution that works, perhaps the KDE tarball creation script could automatically create a separate tarball for that file either. Please comment on this.

Is the __init__.py actually needed at build time though? It occurs to me that it would be a binary dependency (i.e. runtime) rather than a source dependency (i.e. buildtime).

At any rate, I am a purist, I don't think ECM should generate the file at all ^^

To pick up where I left off with my suggestion on the mailing list... The way I imagine this to work in the easiest to grasp manner is: there is a python-kf5 source tarball which contains a CMakeLists.txt which ultimately uses GeneratePythonBindingUmbrellaModule to create the init.py and installs that. Each framework which generates bindings would then runtime depend on python-kf5 to get the __init__.py.
For simplicities sake, I'd actually put this in its own git repo, so from a release script POV it's like any other framework, albeit a very tiny one.

This does raise the problem of an implicit runtime dependency, so, in addition to the above I would also have python-kf5 generate a CMakeConfig and have other frameworks (possibly implicitly via FindPythonModuleGeneration) find_package() for it and set the package as RUNTIME so it actually will be reported in the feature summary as a runtime dependency. This should make it easier for both distros and source users to make sure they actually have python-kf5 installed.

From a packaging POV it resolves the file conflict, makes it super low effort to get the separate binary package for __init__.py, and codifies the runtime dependency so one doesn't have an excuse to forget about carrying that into the packaging.
From a source user POV there's an additional tarball to build, which is a tad meh, but probably not too bad given it's super fast and has 100% dependency overlap with the framework that the user wants to build bindings for.

In T5016#74679, @skelly wrote:

I added https://commits.kde.org/extra-cmake-modules/29b9db8d67c67d4d1303e8dd9804f698fbaefbe1 in an attempt to resolve the issue of multiple packages installing the same __init__.py file.

GeneratePythonBindingUmbrellaModule.cmake requires cmake 3.2, is it really needed? As a matter of policy it should probably follow the version required by the rest of extra-cmake-modules.

In order to allow clean splitting of the bindings in PKGBUILD-based distros, it would be great to have a make target to install the bindings, so one could simply do "make DESTDIR=foo install python_bindings". Would that be feasible?

@arojas Does that imply that 'make' alone should not build the bindings? I don't think that's generally desirable.

@skelly not at all, 'make' and 'make install' should still install everything, the python_bindings target would be a subset of it.

@arojas Is something comparable already done like that elsewhere in KDE? Like for documentation for example?