Introduction of runtimes
ClosedPublic

Authored by apol on Apr 25 2017, 5:19 PM.

Details

Summary

When we develop our software, we usually don't develop the application just
for us. In the linux world we have a big problem with distributing our
software beyond software tarballs and this abstraction aims at integrating
the solutions created for deploying software so that it can be integrated
in our workflow when developing.

As you can see in the proposed IRuntime class, there's 2 main things
abstracted:

  • the file system, by converting file paths into the runtimes' back and

forth. (e.g. include directories)

  • processes, by running the processes we execute to interact and

introspect the system (i.e. cmake or gdb)

For a proof of what we're discussing, I implemented two plugins: flatpak
and docker.

  • docker: is more of a proof of concept nowadays, in general it just works

in terms of running processes but there we don't (yet?) have integration
with the file system.

  • flatpak: allows us a full-featured environment for creating and deploying

desktop applications, which offers many simplifications over how we've been
working so far.

Here's a couple of videos showing the kind of things we can do and a
proposal for a workflow.

Docker: https://youtu.be/7a4qbQ1rrYc
Flatpak: https://youtu.be/Gh9nT7tFI6g
Android: https://youtu.be/M7Hom4OUSHg

Diff Detail

Repository
R33 KDevPlatform
Branch
runtimes
Lint
No Linters Available
Unit
No Unit Test Coverage
apol created this revision.Apr 25 2017, 5:19 PM
Restricted Application added a subscriber: kdevelop-devel. · View Herald TranscriptApr 25 2017, 5:19 PM

In general this seems a good thing to get integrated into KDevelop.

Though with the current documentation and a personal lack of experience on developing with Docker or flatpack, myself I am a little lost about the exact problems which should be solved by the new classes (even more as I yet have to grasp the build/execute parts of KDevelop completely). Pardon my ignorance so far.
Could you give some more concrete examples what does not work yet, and how the new classes and their functionality will be a solution?

E.g. extending the API dox of the new class IRuntime might help. The current text leaves me puzzled like this,to give you an idea where more explanation could help me and then perhaps some people:

A runtime represents an environment that will be targetted

What is meant by "environment" exactly? A set of environment variables? Another virtual or real machine? A chroot environment? Any of that? What not?
By what will the "environment" be targetted? Build system? Installation? Execution? Packaging? Deployment? Testing? Only one or any of them?

It allows the IDE to interact with virtual systems that live in different namespaces.

Is a "virtual system" such an "environment" or a "runtime"? Or just another term for it? What is a "different namespace" in that context?

It allows to execute processes in them and translate the paths these runtimes offer into ones that will be visible to our process to interact with.

This sounds like a runtime plugin would map between filesystem structures. But what defines which filesystem a process would expect? And by what principal logic would the mapping be done?
I would need an example at least to get an idea.

Please, also give a short comment what exactly can be seen in the videos, so one knows what to look out for and where to spot the differences. Some dialogs and actions are done so fast and without explaining why, so for the clueless watcher just some wild build stuff happens :)

apol updated this revision to Diff 13851.Apr 27 2017, 12:35 AM
  • Use icon for docker build action
  • Improve procedure to create bundles that we can ship
  • Improve debug messages
apol updated this revision to Diff 13883.Apr 27 2017, 4:39 PM
  • Save the last used device in a configuration file
  • Extend API documentation
  • Remove the RuntimesModel
apol added a comment.Apr 27 2017, 4:48 PM

Please, also give a short comment what exactly can be seen in the videos, so one knows what to look out for and where to spot the differences. Some dialogs and actions are done so fast and without explaining why, so for the clueless watcher just some wild build stuff happens :)

Okay, yes, sorry:

docker video

On the docker video you can see how the project cannot be tested in the local environment because some dependencies are not present. If you notice the first time it's run it complains that an import cannot be resolved. Then it's builds a Dockerfile that provides the package and when we choose the runtime it works transparently.
I think it's specially interesting that KDevelop takes care about how the file we are running is named within the docker, so it doesn't need special tinkering.

flatpak video

It opens ark, the KDE archiving software and directly fetches the dependencies (i.e this file) being libarchive and libzip. Then an environment is created with these, on top of KDE's flatpak runtime with all the dependencies ark needs to be developed. Then it builds and installs the ark version opened in the IDE against this environment. It's important because it gives us a generic 3-step fetch, build, execute procedure that any developer can use to develop KDE applications without having to worry about Qt/KDE/libzip/lib* versions.

apol updated this revision to Diff 13938.Apr 28 2017, 3:54 PM
  • Remove unused code
  • Implement flatpak host path conversions
apol updated this revision to Diff 14062.May 2 2017, 1:53 AM
  • Implement flatpak host path conversions
anthonyfieroni added inline comments.
plugins/docker/dockerplugin.cpp
95

Better use !!qobject_cast<> because you get a pointer not a bool.

mwolff added a subscriber: mwolff.May 3 2017, 10:04 PM

first of all, sorry for the long delay Aleix. I haven't yet had time to look at the code, just watched your videos which look pretty neat. some high-level questions:

  • would one need to build from scratch every time, or do you support some way to do partial rebuilds inside the container?
  • did you actually manage to integrate the container environment into our language system? i.e. at work I'm working on one project inside a docker container that contains important thirdparty dependencies that are unavailable on my host system. in kdevelop, that leads to tons of "file not found" errors and so forth - how do you deal with that?
  • you mention GDB, so we can build/run/debug transparently from kdev within a container with this patch?
plugins/docker/dockerplugin.cpp
95

no need. we don't use this javascript style elsewhere in our codebase

apol added a comment.May 4 2017, 10:41 AM

first of all, sorry for the long delay Aleix. I haven't yet had time to look at the code, just watched your videos which look pretty neat. some high-level questions:

No worries!

  • would one need to build from scratch every time, or do you support some way to do partial rebuilds inside the container?

Not really. It depends on the tech used of course but it works just fine here. Both for flatpak and docker (and obviously Android) allow to keep their build directory so it can just be re-used. The catch is that you want a different build directory per runtime, but I'd say that's a given.

  • did you actually manage to integrate the container environment into our language system? i.e. at work I'm working on one project inside a docker container that contains important thirdparty dependencies that are unavailable on my host system. in kdevelop, that leads to tons of "file not found" errors and so forth - how do you deal with that?

That's the whole point of this approach, yes. And that's what the pathInRuntime/Host are supposed to achieve. (also see the kdev patch, it has some logic to translate include paths, although it's not working very well just yet, I'm assuming I'm failing to invalidate a cache somewhere).
For flatpak it should just work as the contained file system is easily available.
I haven't found a way to access the docker file system yet (without sshfs that is, which I consider a last resource approach).

  • you mention GDB, so we can build/run/debug transparently from kdev within a container with this patch?

I haven't tested it yet, but it should just work as long as flatpak (which commonly does) and docker provide gdb. In this use case we'll still be triggering a process with which we can interact through the GDBMI protocol.

mwolff added a comment.May 4 2017, 1:44 PM

neat, sounds awesome - I'll try to find some time to try this out over the next days. But yes, without sshfs it's going to be tricky to get access to docker contents. I access them usually via, in my case, these paths:

/var/lib/docker/devicemapper/mnt/*/rootfs/

but, afaik, this only works for me as a normal user b/c I nastily allowed me access to that folder...

apol added a comment.May 4 2017, 2:32 PM

but, afaik, this only works for me as a normal user b/c I nastily allowed me access to that folder...

Correct. Figuring out docker would be a homeround surely. I looked into using /var/lib/docker but it's not available. We should probably get in touch with the upstream community and measure how open to dreaming they are.

apol added a comment.May 5 2017, 7:21 PM

Here's a video transparently switching from local to Android runtimes:
https://youtu.be/M7Hom4OUSHg

apol edited the summary of this revision. (Show Details)May 5 2017, 9:55 PM
apol updated this revision to Diff 14280.May 8 2017, 11:48 AM
  • Use icon for docker build action
  • Improve procedure to create bundles that we can ship
  • Improve debug messages
  • Save the last used device in a configuration file
  • Extend API documentation
  • Remove the RuntimesModel
  • Remove unused code
  • Implement flatpak host path conversions
  • Fix debug statement, no need to check QProcess::program
  • List architectures as per the ones that are installed for the sdk
  • Readability
  • Fix relative path generation
apol updated this revision to Diff 14386.May 11 2017, 8:59 AM
  • Use icon for docker build action
  • Improve procedure to create bundles that we can ship
  • Improve debug messages
  • Save the last used device in a configuration file
  • Extend API documentation
  • Remove the RuntimesModel
  • Remove unused code
  • Implement flatpak host path conversions
  • Fix debug statement, no need to check QProcess::program
  • List architectures as per the ones that are installed for the sdk
  • Readability
  • Fix relative path generation
  • Also mount build directories as docker volumes
  • Pass the command as well when finalizing a flatpak build directory
apol updated this revision to Diff 14419.May 11 2017, 6:19 PM
  • Properly en/disable runtimes
  • Have docker make available the internal filesystem to the IDE
mwolff requested changes to this revision.May 14 2017, 10:58 AM

overall looks OK to me, mostly nitpicks. I can't comment on the docker/flatpack magic as this goes beyond my knowledge in that area. Also make sure that the docker/flatpak plugins are only installed on linux.

interfaces/iruntime.h
71 ↗(On Diff #14280)

const?

77 ↗(On Diff #14280)

const?

85 ↗(On Diff #14280)

how is this supposed to be used? the API isn't straight-forward, i.e. why is this protected? Why do I call it on this class instead of something like controller->setEnabled(theRuntime)?

interfaces/iruntimecontroller.h
51 ↗(On Diff #14280)

is ownership being transferred here?

53 ↗(On Diff #14280)

do we usually add multiple ones in a go? I'd expect when the user adds _a_ runtime, it's only one - so having an add function that does not take a vector would be good to have.

also, are we never removing runtimes?

language/backgroundparser/backgroundparser.cpp
592 ↗(On Diff #14280)

unrelated change(s) (also below)

outputview/outputexecutejob.cpp
109 ↗(On Diff #14280)

out of interest: why is this required? what state is the process in?

307 ↗(On Diff #14280)

unrelated

outputview/outputexecutejob.h
225 ↗(On Diff #14280)

please add api dox and link to the runtime stuff

plugins/docker/dockerplugin.cpp
60 ↗(On Diff #14280)

QOverload is 5.8 only, afaik we support older versions - please do the cast manually

76 ↗(On Diff #14280)

if you'd make this whole function a lambda and inline it above into the connect, you could capture the process directly, removing the need to call sender()

89 ↗(On Diff #14280)

so the runtime controller takes ownership?

97 ↗(On Diff #14280)

please add braces and a space after for

146 ↗(On Diff #14280)

this is why I was expecting - we want a addRuntime(new DockerRuntime(name)) here, i.e. no vector

plugins/docker/dockerplugin.h
32 ↗(On Diff #14280)

indent

plugins/docker/dockerpreferences.h
31 ↗(On Diff #14280)

from here on until below: deindent one level

plugins/docker/dockerruntime.cpp
54 ↗(On Diff #14280)

categorized logging, and output a warning when something goes wrong?

70 ↗(On Diff #14280)

can you add some comments that outline what this is doing?

73 ↗(On Diff #14280)

qgetenv(USER)? Don't we have KDE api for this? I doubt this would work on windows, but then again - most of the commands here don't work on windows. make sure this whole plugin isn't compiled on windows?

77 ↗(On Diff #14280)

braces please

96 ↗(On Diff #14280)

space

103 ↗(On Diff #14280)

braces

104 ↗(On Diff #14280)

concat manually instead of using %1:%2?

plugins/docker/dockerruntime.h
30 ↗(On Diff #14280)

I think having a good apidox here is crucial so that others understand what this is doing. i.e. some short explanation of the concepts to get the path mapping etc.

plugins/flatpak/flatpakruntime.cpp
76 ↗(On Diff #14280)

/me hopes this never deletes $HOME or anything like that accidentally... could this instead use a temporary dir to make sure it's not some pre-existing user-path?

plugins/flatpak/flatpakruntime.h
30 ↗(On Diff #14280)

dito for having apidox outlining the core concepts

41 ↗(On Diff #14280)

why both K and Q process?

49 ↗(On Diff #14280)

const?

shell/runtimecontroller.cpp
40 ↗(On Diff #14280)

remove the "error!!!", also below - it's already a warning. but do print the processes error string

This revision now requires changes to proceed.May 14 2017, 10:58 AM
apol marked 23 inline comments as done.May 14 2017, 8:00 PM
apol added inline comments.
interfaces/iruntime.h
85 ↗(On Diff #14280)

See DockerRuntime::setEnabled. There's some things we need to get ready before getting it announced as enabled.

interfaces/iruntimecontroller.h
53 ↗(On Diff #14280)

Yes, you are right, I added this in an iteration where I had a model. Probably doesn't make much sense now.

Some plugins add at bulk, such as docker at boot.

Will remove them on deletion.

outputview/outputexecutejob.cpp
109 ↗(On Diff #14280)

I'll revert it and debug again, I don't remember why I added it anymore.

plugins/docker/dockerplugin.cpp
76 ↗(On Diff #14280)

meh.

89 ↗(On Diff #14280)

Yes.

plugins/docker/dockerruntime.cpp
73 ↗(On Diff #14280)

I looked into KUser, I don't see how to create a user for the current user. Something to look into.

plugins/flatpak/flatpakruntime.h
41 ↗(On Diff #14280)
KProcess p;
p.setProgram({a, b, c});
p->QProcess::start();

^this doesn't do what it should do.

apol marked 4 inline comments as done.May 14 2017, 8:03 PM
apol added inline comments.
plugins/docker/dockerruntime.cpp
73 ↗(On Diff #14280)

Fixed ^^'

apol marked an inline comment as done.May 14 2017, 8:07 PM
apol updated this revision to Diff 14518.May 14 2017, 8:07 PM
apol edited edge metadata.
  • Have docker make available the internal filesystem to the IDE
  • Allow to expose environment variables for runtimes
  • Only compile flatpak and docker on Unix for now
  • Fix most of Milian's comments
apol updated this revision to Diff 14520.May 14 2017, 8:16 PM
  • Use icon for docker build action
  • Improve procedure to create bundles that we can ship
  • Improve debug messages
  • Save the last used device in a configuration file
  • Extend API documentation
  • Remove the RuntimesModel
  • Remove unused code
  • Implement flatpak host path conversions
  • Fix debug statement, no need to check QProcess::program
  • List architectures as per the ones that are installed for the sdk
  • Readability
  • Fix relative path generation
  • Also mount build directories as docker volumes
  • Pass the command as well when finalizing a flatpak build directory
  • Properly en/disable runtimes
  • Have docker make available the internal filesystem to the IDE
  • Allow to expose environment variables for runtimes
  • Only compile flatpak and docker on Unix for now
  • Fix most of Milian's comments
  • Remove unrelated change
apol updated this revision to Diff 14638.May 17 2017, 3:37 PM
  • Use icon for docker build action
  • Improve procedure to create bundles that we can ship
  • Improve debug messages
  • Save the last used device in a configuration file
  • Extend API documentation
  • Remove the RuntimesModel
  • Remove unused code
  • Implement flatpak host path conversions
  • Fix debug statement, no need to check QProcess::program
  • List architectures as per the ones that are installed for the sdk
  • Readability
  • Fix relative path generation
  • Also mount build directories as docker volumes
  • Pass the command as well when finalizing a flatpak build directory
  • Properly en/disable runtimes
  • Have docker make available the internal filesystem to the IDE
  • Allow to expose environment variables for runtimes
  • Only compile flatpak and docker on Unix for now
  • Fix most of Milian's comments
  • Remove unrelated change
  • Display the full command on an output job
  • Fix menu
  • Fix umount of docker environments
  • Don't decode and pass an empty directory
  • Fix code that figures out docker runtime host paths
apol updated this revision to Diff 14675.May 18 2017, 6:11 PM
  • Fix remote flatpak application execution
  • Don't rely on the CWD when calling flatpak-builder
  • Also pass the finishArgs when executing processes
  • Add test
  • Fix issues detected by the test
This revision was automatically updated to reflect the committed changes.