Make MacOS package creation work with plugins
ClosedPublic

Authored by arichardson on Aug 28 2018, 10:05 AM.

Details

Reviewers
vonreth
kfunk
Summary

This is a series of commits to fix the MacDMGPackager to create an
application bundle that will actually work. With these changes I can
create a kate.app that has a terminal view, etc.
I also added a drop link to /Applications to the generated .dmg


MacDMGPackager: Validate that plugins don't depend on absolute paths

All the plugin .so files in the kate DMG reference absolute paths from
the binary-factory builder and therefore can't be loaded after installation
Check for this case and output and error avoid this issue.

I will fix this problem in a follow-up commit.


MacDMG: Fix absolute paths in plugins

This currenlty requires a patched macdylibbundler so I will work on fixing
this wihthout using macdylibbundler (which should also be much faster)


Reimplement macdylibbundler in python as part of MacDMGPackager

This is much faster since the list of libraries only needs to be parsed
once and it also has the advantage that it handles the library name id
correctly.

With this change the DMG of kate.app finally has a working Konsole plugin.


Move the dylib bundling code into a separate class

This means two less arguments to every function call


MacDylibBundler: Handle already bundled libs with relative paths

Add a drop link to /Applications in the .dmg file


Test Plan

Terminal view plugin works after dropping kate.app from the DMG to /Applications

Diff Detail

Repository
R138 Craft
Branch
craft-reviews (branched from master)
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2299
Build 2317: arc lint + arc unit
arichardson requested review of this revision.Aug 28 2018, 10:05 AM
arichardson created this revision.

em wow, I'll need some time to review this

bin/Packager/MacDMGPackager.py
77

outdated comment

hm in general all files should be already copied to the archive dir as we use the CollectionPackagerBase.
So in theory we only need to fix the rpath etc and not need to deploy additional files from CraftRoot.
@kfunk thoughts?

bin/Packager/MacDMGPackager.py
77

ignore that one

127

my experience showed that we usually should not resolve symlinks and just copy them (with folow symlinks=false), as done by utils.copyFile)

arichardson added inline comments.Aug 28 2018, 11:14 AM
bin/Packager/MacDMGPackager.py
127

This is effectively what I do just a few lines further down. I just added some checks here that the symlink target is inside the bundle and not an absolute symlink. Not sure if it's required but it seemed like a good idea to me..

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

bin/Packager/MacDMGPackager.py
127

ok

Just as a general thought: It'd be really nice to have the pythonified MacDylibBundler as a separate executable (i.e. .py file). I would have required this a couple of times, also in other projects.

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

Initially I've used MacDylibBundler to pull in libs from packages installed via Brew. Nowadays we built (almost?) everything ourselves using Craft, right? There might still be some libs which need to be pulled into the .app, though, if it's not tracked by Craft... But I honestly do not know the current state of affairs re. Craft on macOS...

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

Initially I've used MacDylibBundler to pull in libs from packages installed via Brew. Nowadays we built (almost?) everything ourselves using Craft, right? There might still be some libs which need to be pulled into the .app, though, if it's not tracked by Craft... But I honestly do not know the current state of affairs re. Craft on macOS...

I noticed it pulled in at least libgit2.dylib and editorconfig.dylib from homebrew.

The one thing that is missing for any application to work on MacOS is the ability to override QStandardPaths::GenericDataLocation. The app bundle seems to work because it bundles all of share/ into Contents/Resources but if I try to run the kate.app created by craft in ~/kde/Applications it doesn't find anything.
I can (almost) get it to work by removing kate.app/Contents/Resources and turning that into a symlink to ~/kde/share. If I do that kate works fine when launched from the console ./Applications/KDE/kate.app/Contents/MacOS/kate foo.txt but if I double click on the .app or use open -a all icons are missing and lots of stuff is broken.

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

Initially I've used MacDylibBundler to pull in libs from packages installed via Brew. Nowadays we built (almost?) everything ourselves using Craft, right? There might still be some libs which need to be pulled into the .app, though, if it's not tracked by Craft... But I honestly do not know the current state of affairs re. Craft on macOS...

I noticed it pulled in at least libgit2.dylib and editorconfig.dylib from homebrew.

The one thing that is missing for any application to work on MacOS is the ability to override QStandardPaths::GenericDataLocation. The app bundle seems to work because it bundles all of share/ into Contents/Resources but if I try to run the kate.app created by craft in ~/kde/Applications it doesn't find anything.
I can (almost) get it to work by removing kate.app/Contents/Resources and turning that into a symlink to ~/kde/share. If I do that kate works fine when launched from the console ./Applications/KDE/kate.app/Contents/MacOS/kate foo.txt but if I double click on the .app or use open -a all icons are missing and lots of stuff is broken.

Hm are you using our qt build, we apply https://github.com/KDE/craft-blueprints-kde/commit/f7513dc802ee29db1c61dc6e3f9f4e729b418a4a

For libgit, libgit should be available in craft and is probably a missing dependency of kate then, editorconfig too.

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

Initially I've used MacDylibBundler to pull in libs from packages installed via Brew. Nowadays we built (almost?) everything ourselves using Craft, right? There might still be some libs which need to be pulled into the .app, though, if it's not tracked by Craft... But I honestly do not know the current state of affairs re. Craft on macOS...

I noticed it pulled in at least libgit2.dylib and editorconfig.dylib from homebrew.

The one thing that is missing for any application to work on MacOS is the ability to override QStandardPaths::GenericDataLocation. The app bundle seems to work because it bundles all of share/ into Contents/Resources but if I try to run the kate.app created by craft in ~/kde/Applications it doesn't find anything.
I can (almost) get it to work by removing kate.app/Contents/Resources and turning that into a symlink to ~/kde/share. If I do that kate works fine when launched from the console ./Applications/KDE/kate.app/Contents/MacOS/kate foo.txt but if I double click on the .app or use open -a all icons are missing and lots of stuff is broken.

Hm are you using our qt build, we apply https://github.com/KDE/craft-blueprints-kde/commit/f7513dc802ee29db1c61dc6e3f9f4e729b418a4a

Yes I am. It works for the .app from the generated .dmg, just not for the .app from craftroot. I wonder if we should just add another env var (e.g. QT_GENERIC_DATA_LOCATION) or even simpler the hardcoded path?

For libgit, libgit should be available in craft and is probably a missing dependency of kate then, editorconfig too.

Okay I'll add that dependency and see if it still builds if I make deps in /usr/local an error.

ok then the main question is, do we need to deploy extra libs? they should already be in the image, and removing this functionality would simplify the code?

Initially I've used MacDylibBundler to pull in libs from packages installed via Brew. Nowadays we built (almost?) everything ourselves using Craft, right? There might still be some libs which need to be pulled into the .app, though, if it's not tracked by Craft... But I honestly do not know the current state of affairs re. Craft on macOS...

I noticed it pulled in at least libgit2.dylib and editorconfig.dylib from homebrew.

The one thing that is missing for any application to work on MacOS is the ability to override QStandardPaths::GenericDataLocation. The app bundle seems to work because it bundles all of share/ into Contents/Resources but if I try to run the kate.app created by craft in ~/kde/Applications it doesn't find anything.
I can (almost) get it to work by removing kate.app/Contents/Resources and turning that into a symlink to ~/kde/share. If I do that kate works fine when launched from the console ./Applications/KDE/kate.app/Contents/MacOS/kate foo.txt but if I double click on the .app or use open -a all icons are missing and lots of stuff is broken.

Hm are you using our qt build, we apply https://github.com/KDE/craft-blueprints-kde/commit/f7513dc802ee29db1c61dc6e3f9f4e729b418a4a

Yes I am. It works for the .app from the generated .dmg, just not for the .app from craftroot. I wonder if we should just add another env var (e.g. QT_GENERIC_DATA_LOCATION) or even simpler the hardcoded path?

Hardcoding would again make testing harder, but we might just add a patch to use the relative path to qappDir/…./share as a fallback, or maybe we can even detect whether we are in a bundle or in a build root?

For libgit, libgit should be available in craft and is probably a missing dependency of kate then, editorconfig too.

Okay I'll add that dependency and see if it still builds if I make deps in /usr/local an error.

Hardcoding would again make testing harder, but we might just add a patch to use the relative path to qappDir/…./share as a fallback, or maybe we can even detect whether we are in a bundle or in a build root?

It should be easy to detect whether we are in a bundle or not. I can also check if the current executable is below qtpaths --install-prefix and if so add $(qtpaths --install-prefix)/share.

Is there a way to do an incremental build of qtbase with craft? Otherwise I'll try to test this with a separate Qt build.

craft --make --install --qmerge qtbase

vonreth accepted this revision.Sep 5 2018, 1:20 PM

We can reduce the complexity later :)
Can you also remove the blueprint for dylibbundler once its merged?

This revision is now accepted and ready to land.Sep 5 2018, 1:20 PM