kde.org/applications rewrite
Open, Needs TriagePublic

Description

Harald did a rewrite of the kde.org/applications code using appstream data and modern infrastructure. Todo item to not forget about it:

extractor https://github.com/apachelogger/kde-applications-appstream
UI https://websvn.kde.org/?view=revision&revision=1512271

The main problems as I recall off the top of my head:

  • I was too opinionated in excluding certain types of apps (unmaintained/* repos comes to mind; there's more though)
  • For the same reason also some fields from the old format didn't get moved into the new format and so overall there was less data
  • Numerous urls broke because the human-made category didn't match the actual category of the appstream data
  • A number of applications had no, bad, or un-crawlable appstream data (I think there's a list in a comment in appstream.rb)

It may also not properly extract data just now due to bitrot and the fact that things were reshuffled on the CI...

The inner workings are fairly straight-forward. It gets a list of all projects from projects.kde.org API, then filters all the ones we have on build.kde.org as well, iters them and grabs the tarball created from the make install result from some server, crawls all the application appdata inside, runs them through appstreamcli to convert them to yaml, then converts that to json and done. Well, almost, projects which are not under CI get their git repo crawled for an actual appstream file as a last ditch effort. That's actually when appstream data is considered not crawlable... if the appdata is configure_file'd through cmake there's no reliable way of knowing what the output is unless it gets CI'd (being the CI fanboy that I am I'd argue that the solution of course is to CI them ;))
On the UI side it mostly changes the data frontend class to be backed by the new appstream json format.

Related Objects

StatusAssignedTask
OpenNone
OpenNone
jriddell created this task.Dec 6 2018, 3:39 PM
jriddell added a subscriber: GB_2.Dec 10 2018, 2:54 PM

I ran the script and got some output which I put at
http://apps.kde.org.uk/applications/
It needs some setup to mirror www.kde.org and I'm not sure what that is
probably https://cgit.kde.org/websites/capacity.git/ goes somewhere
dunno what else

There could be a bigger issue - rsibreak is not unmaintained.
https://cgit.kde.org/sysadmin/repo-metadata.git/tree/projects/extragear/utils/rsibreak

It doesn't have an appstream file which is easy enough to fix
And it's not in build.kde.org CI, which can either be fixed by adding it there or by doing a workaround to get the appstream files.

First task is presumably to make a list of such apps

Also todo:
The sidebar isn't loading
The /applications/spdx-to-html.js script doesn't exist
The jquery isn't loading cos it's marked as defer but the use of it isn't deferred

And it's not in build.kde.org CI, which can either be fixed by adding it there or by doing a workaround to get the appstream files.

IIRC projects without a build artifact get their git repo crawled. So, build.kde.org integration is not entirely necessary, it's only the preferred and most reliable means of extraction.

The only way an existing appstream file is not picked up is when the project is not under CI and the appstream file is generated at build time (e.g. org.kde.foo.appdata.xml.cmake which gets run through cmake to fill in some fields or whatever). In that case the only way to reliably pick up the data is to CI the project.

ngraham added a subscriber: ngraham.Wed, May 8, 3:46 PM

I added appstream to rsibreak master and on a following run of appstream.rb it gets picked up and added
http://apps.kde.org.uk/applications/utilities/org.kde.rsibreak
but now I'm not sure how to get it added to the Utilities page

Oh turns out I just needed to run appstream_mkindex.rb again and rsibreak now appears in the right place, yay

aacid added a subscriber: aacid.Wed, May 8, 9:48 PM
apol added a subscriber: apol.Wed, May 8, 10:54 PM

http://apps.kde.org.uk/applications/ now in a decent shape but it's still missing various apps. Umbrello and Konqueror for example. Needs working out why that is. I think the parser is getting confused by which appstream.xml files it should be using maybe since these apps have several.

Then all the meta-data needs checked for sanity and up-to-date ness.

We might want to exclude some obscure bits like pimsettingexporter.

currently they're just ignored but they should be unignored and get appstream files added and a hidden page will appear

aacid added a comment.Thu, May 9, 10:49 PM

That makes no sense, why would we commit appstream files to unmaintained applications?

That makes no sense, why would we commit appstream files to unmaintained applications?

So that they can be added to the kde.org/applications pages :)

I think that for unmaintained, we should have a graveyard, so the apps can't be forgotten, because they make part of KDE history.
Something like google graveyard.
https://gcemetery.co/

I am going to help on this task, I just need to pass on my college exams.

aacid added a comment.Fri, May 10, 8:17 PM

That makes no sense, why would we commit appstream files to unmaintained applications?

So that they can be added to the kde.org/applications pages :)

That seems the wrong way to do it if you ask me, it's cool that appstream files are the "database" for the web, but needing to add files to apps that are dead so they are still shown doesn't seem like the best way to do it. But then again, i'm not the one doing it.

aacid added a comment.Fri, May 10, 8:17 PM

I think that for unmaintained, we should have a graveyard, so the apps can't be forgotten, because they make part of KDE history.
Something like google graveyard.
https://gcemetery.co/

That's what https://kde.org/applications/unmaintained/ is, isn't it?

valorie added a subscriber: valorie.EditedSun, May 12, 2:28 AM

A year or two ago, we had a discussion about using data like this for each application (at least), reporting QC information like (as my fuzzy memories tell me) unit test coverage, devel activity like commits per month/year, etc. I suppose this would have to be done by analysis of the codebase in git -- how difficult is this to do? Do other projects do this, and how?

PS: KUDOS to the work so far! Anything that makes our webpages more up-to-date without requiring more regular work, the better.

apol added a comment.Sun, May 12, 10:16 PM

@valorie is this what you are thinking about? https://reports.kde.org https://reports.kde.org/en/projects/libs-playground-kirigami-components-framework/commits_report

I'm not sure it makes sense having this information on the product pages... :shrug:

If ind it quite a pitty that there are only so few informations about all those cool applications which are part of the great KDE project.

Just for reference:
Apps page from GNOME wiki: https://wiki.gnome.org/Apps/Calendar
Apps page from elementaryOS app store: https://appcenter.elementary.io/com.github.artemanufrij.playmymusic/

What I really like about the GNOME page is the structure of several "tabs" which enlight one topic per page.
We have a similar "Development Information" in the side panel but I always overlooked it.
And at least in my opinion it would be great to see how development is ongoing for those applications which could be seperated in one "development" tab ?

So the current view could be the "common" tab at the default one. But another "development" tab could hold all those nice informations whch could help interested people into contributing ?
There would be so much room for all kind of informations. Those commits report looks quite good. Maybe also get a list of latest commits or current task which are under development ?
The easiest, but maybe not the most appealing one would be an iframe from the corresponding phab page ?

We have the possibility to achive something great and get the most out of those application pages.

Let's not blow up the scope. The concern of this task is making the backend data maintainable and refreshing the page style a bit.