Create Calligra Framework by separating out applications and libraries
Open, Needs TriagePublic

Description

As mentioned on https://phabricator.kde.org/D26050#583223 (and discussed in person with a number of people), the current monolithic state of the Calligra suite is less advantageous than it was back when it lived in subversion (where if you needed you could pick out bits you wanted to work on without grabbing the entire thing). That is to say, what used to be a positive thing, which lowered the barrier of entry to potential new contributors, has turned into something of a stumbling block, as we ask people to please clone this gigantic repository before they can really do much of anything. So, with that in mind...

I would like to propose that we split out Calligra into several repositories, conceptually a set of framework repositories (for release along with KDE Frameworks), and a set of application repositories. As an initial suggestion, the list might look like so:

Calligra Frameworks:

  • calligra-frameworks-base (what is currently in libs/ - possibly wants splitting further)
  • calligra-components (the Qt Quick components)
  • calligra-parts (the various KParts - this would be a fair bit of work, and might also want splitting out)
  • calligra-plugins (currently in plugins/)
  • calligra-extra-filters (currently in filters/)

Calligra Suite (just in alphabetical order):

  • calligra-author (yeah, of course i have ulterior motives - i entirely intend to make the best creative writing on the planet out of this ;) )
  • calligra-braindump
  • calligra-gemini
  • calligra-karbon
  • calligra-sheets
  • calligra-stage
  • calligra-words
  • calligra-tools (commandline tools - one of many reasons having the widgetisms removed from the libraries would be good)

Finally, it would be highly advantageous if, while this work is undertaken, we could remove all the widgetisms from the libraries

This task would function as the nexus for the work, both discussion and tracking subtasks, and i would like to see us create these new repositories on Invent (to avoid a move later, and creating a clean break - that is, of course, unless the sysadmin team has reasons why that would be a bad idea)

As a part of this, it might very well be useful for those of us intending to spend effort on this to all to be together somewhere at some point, for some high-intensity hacking. It feels to me as though this would be something where at least part of our handshake funds would be well spent (though perhaps that'd want to happen once things regarding the current world health situation have calmed down just a touch).

I hope you like the idea - subscribe to this task, make all the words at eachother, and let's hammer out a plan, and then get it all sorted. We've got a little while until KF6, but if we can prepare beforehand... that seems like a worthy effort :)

Related Objects

leinir created this task.Mar 13 2020, 1:12 PM
ognarb added a subscriber: ognarb.Mar 13 2020, 1:15 PM

+1 on the general idea

Some remarks:

  • Do we want to have an abi and api stability in the Calligra frameworks? or do we prefer to do it like in PIM and only guaranty that all the package from a certain version are compatible? I think the second option is better at least at the beginning since we don't have many third parties users and the only one is using submodules with a list of patches anyway.
  • I don't think we need to wait for KF6 branching to split the repository. We could already start doing the split after the next release but more progressively by for example create application repository first and later split the rest.

I'm always in favor of moving shared content into frameworks, but just to be a miserable contrarian here (so feel free to ignore me), is the size of a git repo really an impediment to new contributors? In my experience, the opposite is true, that the more git repos one needs to check out to be productive, the more opportunities there are for failures due to missing dependencies, code compiled out of order, etc. I mean, with a huge repo, you just clone it once and it's done. If it takes 30 seconds or a minute or two, is that really a big deal?

rjvbb added a subscriber: rjvbb.Mar 13 2020, 8:56 PM

What's the size of the .git directory of the calligra repo these days? I seem to recall it was what I thought really huge already in KDE4 days. Last time I tried git wouldn't let you push from a clone that didn't have the full history. And yes, for me there's a point where I consider the value of my few and smallish contributions don't justify wasting "that much" disk space on what's literally old history.

FWIW, KDevelop moved in the opposite direction, pulled their "framework" bit (kdevplatform) with the other code into a single repo. Which, incidentally, made it harder to build only select parts (plugins), just like you cannot really build only a specific Calligra component against installed versions of the library components (last I checked).
Not that that's a necessary implication but it's a bit tricky to write a build system that can build against either just-build libraries or against the installed copies.

pino added a subscriber: pino.Mar 13 2020, 9:37 PM

What's the size of the .git directory of the calligra repo these days?

$ du -hcs .git/
778M    .git/

Few notes from a person that did few commits in the past, and current maintainer of Calligra in Debian:

  • seriously, 13 repositories? note that LO went in the opposite direction, and such a huge amount of sources just to get Calligra built is a hell of work
  • calligra-author is dead since Calligra 3.0... (not even in git/master)
  • calligra-braindump is sort of dead even in 2.9; "ported" to 3.0 as "unmaintained", hard-disabled it 3.1; better alternatives exist
  • why calligra-plugins and calligra-extra-filters? they would require all the other apps/libraries to build, and it would make sense to simply place the filters/plugins of each application together with each application
  • calligra-frameworks-base and calligra-components split? to me the latter is the QML version of the former, so it would make sense to make just one repository with the Calligra libraries

In addition, a more general question: what is the goal of all of this?
I think that splitting just for the sake of splitting is counter-productive (you need to invest a lot of resources to get the split parts to build and interact).

And, who is going to contribute to Calligra? No, I'm not trying to provoke or anything, I'm asking seriously:

  • there is barely somebody that did 3.0/3.1 releases in the last 3 years
  • not many commits not related to "making it build with newer poppler/gcc/qt/etc"
  • barely new features...

And who is going to maintain all of this?

It is a no from me for exactly the reasons pino and ngraham bring up.

Now if there was a thriving community, I'd be happy to step down and let the community do this if there was a big consensus, but as it stands it will just kill this project even more than it already is

dcaliste added a subscriber: dcaliste.EditedMar 14 2020, 9:31 AM

As a nobody, my opinion on the rightfullness to split or not the repository is of no interest. But what I like in the proposition of @leinir is the target to separate the "engine" from the UI (_i.e._ the widgetery). As an example (hopefully not outdated), the "find" classes, the one to search and replace, are deeply liked to the widgets that control them. It makes it impossible to reuse these class in a UI different from the original one.

Separating the repositories is a way to enforce this engine _vs._ UI design. But it can also be achieved in the monolithic repo, by removing the dependencies to kowidgets in the CMakeLists for instance. I would be happy to join any task that would goes in any of these directions.

Finally about te splitting itself, maybe a less fine grain splitting would already be enough ?

@leinir correct me if i'm wrong, but this can be beneficial if some of Calligra Suite want to have different release circle, like Krita before. Or you want Krita to returns back to Calligra framework?

rempt added a subscriber: rempt.Mar 14 2020, 10:10 AM

@leinir Or you want Krita to returns back to Calligra framework?

That's not going to happen... We're happy hacking out all the flexibility from the libraries that was only needed to support so many different applications.

As an aside, I think splitting up kdepim into so many repositories was a huge mistake. Listening to David Faure at the onboarding sprint only confirmed that opinion. Having to work on half a dozen repos to add a single feature to kmail sounds like hell to me. I think it would be a mistake for Calligra, too, but don't pay attention to me :-)

As an aside, I think splitting up kdepim into so many repositories was a huge mistake.

Ah yes, that was definitely one of the reasons why I never updated my kdepim4 packaging. That said, I also appreciate the fact that the few apps I use that depend on kdepim features now require only to maintain packaging for smallish libraries (that often also don't evolve that quickly).

But somehow I doubt that latter argument can be transferred to Calligra's libraries...

As an aside, I think splitting up kdepim into so many repositories was a huge mistake. Listening to David Faure at the onboarding sprint only confirmed that opinion. Having to work on half a dozen repos to add a single feature to kmail sounds like hell to me. I think it would be a mistake for Calligra, too, but don't pay attention to me :-)

I sometimes find myself feeling this way about Frameworks too. A recent experience at a hackathon where I helped 8 students set up complete development environments from scratch reinforced this viewpoint. Not that I'm seriously recommending re-merging the frameworks, but I would like to challenge the notion that splitting a monolithic codebase across multiple repos is a boon to onboarding; I don't think it is. It may have other benefits, but I don't think onboarding is one of them.

As I have some free time due to the virus and I'm trying to get up to date.

In the SQL word there is the notion of tables, queries and views, where a view can show de values of several tables (combined, filtered or raw).

Do you know if there is something similar for the SCM world? If it exists, there could be several repositories splitted in tiny pieces and also global views composed of several of them (or pieces of several of them).

rjvbb added a comment.Mar 14 2020, 5:40 PM
I sometimes find myself feeling this way about Frameworks too. A recent experience at a hackathon where I helped 8 students set up complete development environments from scratch reinforced this viewpoint. Not that I'm seriously recommending re-merging the frameworks, but I would like to challenge the notion that splitting a monolithic codebase across multiple repos is a boon to onboarding; I don't think it is. It may have other benefits, but I don't think onboarding is one of them.

Apple were first to start calling their system libraries "frameworks" and they do have a nice feature that would address those issues. Something like the Accelerate framework is actually a collection of frameworks. You only specify the "interface" framework, and I *think* that "the system" figures out what to link. Of course they had to rewrite the linker driver (or linker itself) in order to understand how to deal with their framework bundles so they had every occasion to add some additional logic.

ndavis added a subscriber: ndavis.Mar 15 2020, 11:39 PM

As a newcomer, and only occasional contributor, my experience of getting the build up and running is that it's already a daunting process, made tractable only because the instructions are very good. Anything that simplifies these instructions is good, anything that complicates them is bad.

If I'm honest, having everything in one repository probably made things simpler. I'm sure separate repos could be made easy too (e.g. using submodules), but also introduces more scope for errors that are difficult for an occasional contributor (i.e. me) to solve.

But what I like in the proposition of @leinir is the target to separate the "engine" from the UI (_i.e._ the widgetery). As an example (hopefully not outdated), the "find" classes, the one to search and replace, are deeply liked to the widgets that control them. It makes it impossible to reuse these class in a UI different from the original one.

Having said all that above, as a developer for Sailfish OS (but speaking personally), I would completely second @dcaliste's point. Whatever the delivery structure, better decoupling between components would make Calligra more flexible and usable in other contexts. The split between the chrome and the document rendering is a very clear case, and if splitting into repos helps reinforce this, then I'd support it.

furiously attempts to put the worms back in the can - Oh dear! ;)

Right! That's a really good bunch of discussion up there (and very civil, thanks all!), and what we can read from this is that no, we really, really(, no seriously, for real really) don't want to split up the repositories, as that raises an already high barrier of entry even higher without any worthy gain. The other bits of this are still valid, though, from what i see, so... with that in mind, it sounds like what we might want to try and do is perhaps more a bit of code reorganisation and some refactoring in certain places...

  • Get rid of the widget requirements in the core libraries themselves. We can enforce this without much trouble at link time (like @dcaliste suggests). Having done this somewhat hackishly during the initial work for Sailfish, i realise this is not a simple task, but i firmly believe it's a worthwhile one.
  • Those libraries which really /require/ widgets (on account of being UI for some operation or another) can do so, but will want to be described as such in an explicit fashion (perhaps by being given a suitable name). We've already got a bunch of that done (widgets and widgetutils), but we still managed to have some of the UI sneak in where it perhaps wants not to be in some places.
  • Documentation - as @davidllewellynjones says, we've got a bunch already, and it really isn't terrible at all - but that said more could (and should) be done (also, water is wet, circles are round ;) ). Especially we've got a bit of a disconnect between what's on techbase and what's on api.kde.org, and if someone wants to jump in somewhere, that'd probably be a nice place to start (for example, there's rundowns of some of the codebase on techbase pointing to details on api.kde.org, which likely wants to be in the code instead, so we can avoid links randomly dying on us - techbase landing page would remain, but point to api.kde.org for that information).

Suddenly a much more contained effort, and one which... well, doesn't require quite so much lock-down coordination perhaps

On a personal note, I'm really glad to see everybody sounding so keen on Calligra in general, too, it's heartening to see more than just nostalgia :D

PS: I still personally intend to attempt to produce a new Author (the reason it was removed was because it quite literally was just a copy of Words, with nothing additional or new, since all the work had happened in filters or in Words itself, and so having that target in the source tree essentially served no purpose), but i am thinking this could be a useful test for an external application which makes use of the Calligra libraries, without being Calligra itself. Those who have used Scrivener will be aware of some of where i intend to go with that application, and also why that would not fit very well into a generic word processor :)

I think where leinir has landed is more doable than a frameworks.
Also if compiletime is a big hurdle, I see a couple of things that can be done:

  • Remove pigment, see T5738.
  • Make more finegrained productsets. Pt the only thing you can control (if I haven't missed anything) is which apps to build.

E.g. if you want to work on the innards of words for gemini, you select gemini and get words, sheets and stage. In addition you get almost all plugins and all relevant filters. Probably *some* plugins are necessary but you should be able to do without most plugins and no filters at all.

Another problem is how new users find out how to use productsets. A front-end would have been nice, but a bit of documantation should go a long way.

Just wondering, is there any use of karbon inside the suite?
Also the stuff used by okular could be split out?

Just wondering, is there any use of karbon inside the suite?

What kind of use?

Since it is a graphics related program unlike rest of the suite, i was thinking karbon could be split into its own repo. Similar to krita.