Create Task
Maniphest T10159

kde.org/applications rewrite
Closed, ResolvedPublic
Actions

Description

Harald did a rewrite of the kde.org/applications code using appstream data and modern infrastructure. Todo item to not forget about it:

extractor https://github.com/apachelogger/kde-applications-appstream
UI https://websvn.kde.org/?view=revision&revision=1512271

The main problems as I recall off the top of my head:

  • I was too opinionated in excluding certain types of apps (unmaintained/* repos comes to mind; there's more though)
  • For the same reason also some fields from the old format didn't get moved into the new format and so overall there was less data
  • Numerous urls broke because the human-made category didn't match the actual category of the appstream data
  • A number of applications had no, bad, or un-crawlable appstream data (I think there's a list in a comment in appstream.rb)

It may also not properly extract data just now due to bitrot and the fact that things were reshuffled on the CI...

The inner workings are fairly straight-forward. It gets a list of all projects from projects.kde.org API, then filters all the ones we have on build.kde.org as well, iters them and grabs the tarball created from the make install result from some server, crawls all the application appdata inside, runs them through appstreamcli to convert them to yaml, then converts that to json and done. Well, almost, projects which are not under CI get their git repo crawled for an actual appstream file as a last ditch effort. That's actually when appstream data is considered not crawlable... if the appdata is configure_file'd through cmake there's no reliable way of knowing what the output is unless it gets CI'd (being the CI fanboy that I am I'd argue that the solution of course is to CI them ;))
On the UI side it mostly changes the data frontend class to be backed by the new appstream json format.

TODO:

  • Also make a wiki page to explain all this
  • Check for apps which are maintained but missing appstream metadata
  • Deploy on kde.org https://phabricator.kde.org/T10896
  • Tidy KAlgebra (not Mobile)
  • Cantor screenshot
  • Metadata Review:
    • Development
    • Education
    • Games
    • Graphics (except skanlite)
    • Internet
    • Multimedia
    • Office
    • System
    • Utilities
  • Review old URLs forward to new ones e.g. https://kde.org/applications-ng/system/khelpcenter/
  • e-mail out differences in the build

Related Objects
Search...

jriddell created this task.Dec 6 2018, 3:39 PM
jriddell added a subscriber: GB_2.Dec 10 2018, 2:54 PM
jriddell added a comment.May 3 2019, 6:39 PM

I ran the script and got some output which I put at
http://apps.kde.org.uk/applications/
It needs some setup to mirror www.kde.org and I'm not sure what that is
probably https://cgit.kde.org/websites/capacity.git/ goes somewhere
dunno what else

jriddell added a comment.May 7 2019, 4:59 PM

There could be a bigger issue - rsibreak is not unmaintained.
https://cgit.kde.org/sysadmin/repo-metadata.git/tree/projects/extragear/utils/rsibreak

It doesn't have an appstream file which is easy enough to fix
And it's not in build.kde.org CI, which can either be fixed by adding it there or by doing a workaround to get the appstream files.

First task is presumably to make a list of such apps

Also todo:
The sidebar isn't loading
The /applications/spdx-to-html.js script doesn't exist
The jquery isn't loading cos it's marked as defer but the use of it isn't deferred

sitter added a comment.May 8 2019, 11:40 AM
In T10159#184274, @jriddell wrote:

And it's not in build.kde.org CI, which can either be fixed by adding it there or by doing a workaround to get the appstream files.

IIRC projects without a build artifact get their git repo crawled. So, build.kde.org integration is not entirely necessary, it's only the preferred and most reliable means of extraction.

The only way an existing appstream file is not picked up is when the project is not under CI and the appstream file is generated at build time (e.g. org.kde.foo.appdata.xml.cmake which gets run through cmake to fill in some fields or whatever). In that case the only way to reliably pick up the data is to CI the project.

ngraham added a subscriber: ngraham.May 8 2019, 3:46 PM
jriddell added a comment.May 8 2019, 9:14 PM

I added appstream to rsibreak master and on a following run of appstream.rb it gets picked up and added
http://apps.kde.org.uk/applications/utilities/org.kde.rsibreak
but now I'm not sure how to get it added to the Utilities page

jriddell added a comment.May 8 2019, 9:39 PM

Oh turns out I just needed to run appstream_mkindex.rb again and rsibreak now appears in the right place, yay

aacid added a subscriber: aacid.May 8 2019, 9:48 PM
apol added a subscriber: apol.May 8 2019, 10:54 PM
jriddell added a comment.May 9 2019, 10:50 AM

http://apps.kde.org.uk/applications/ now in a decent shape but it's still missing various apps. Umbrello and Konqueror for example. Needs working out why that is. I think the parser is getting confused by which appstream.xml files it should be using maybe since these apps have several.

Then all the meta-data needs checked for sanity and up-to-date ness.

We might want to exclude some obscure bits like pimsettingexporter.

jriddell added a comment.May 9 2019, 10:46 PM
In T10159#184518, @aacid wrote:

What's the plan for https://kde.org/applications/unmaintained/ ?

currently they're just ignored but they should be unignored and get appstream files added and a hidden page will appear

aacid added a comment.May 9 2019, 10:49 PM

That makes no sense, why would we commit appstream files to unmaintained applications?

jriddell added a comment.May 9 2019, 10:56 PM
In T10159#184520, @aacid wrote:

That makes no sense, why would we commit appstream files to unmaintained applications?

So that they can be added to the kde.org/applications pages :)

laysrodrigues added a comment.May 9 2019, 11:34 PM

I think that for unmaintained, we should have a graveyard, so the apps can't be forgotten, because they make part of KDE history.
Something like google graveyard.
https://gcemetery.co/

I am going to help on this task, I just need to pass on my college exams.

aacid added a comment.May 10 2019, 8:17 PM
In T10159#184521, @jriddell wrote:
In T10159#184520, @aacid wrote:

That makes no sense, why would we commit appstream files to unmaintained applications?

So that they can be added to the kde.org/applications pages :)

That seems the wrong way to do it if you ask me, it's cool that appstream files are the "database" for the web, but needing to add files to apps that are dead so they are still shown doesn't seem like the best way to do it. But then again, i'm not the one doing it.

aacid added a comment.May 10 2019, 8:17 PM
In T10159#184522, @laysrodrigues wrote:

I think that for unmaintained, we should have a graveyard, so the apps can't be forgotten, because they make part of KDE history.
Something like google graveyard.
https://gcemetery.co/

That's what https://kde.org/applications/unmaintained/ is, isn't it?

valorie added a subscriber: valorie.EditedMay 12 2019, 2:28 AM

A year or two ago, we had a discussion about using data like this for each application (at least), reporting QC information like (as my fuzzy memories tell me) unit test coverage, devel activity like commits per month/year, etc. I suppose this would have to be done by analysis of the codebase in git -- how difficult is this to do? Do other projects do this, and how?

PS: KUDOS to the work so far! Anything that makes our webpages more up-to-date without requiring more regular work, the better.

apol added a comment.May 12 2019, 10:16 PM

@valorie is this what you are thinking about? https://reports.kde.org https://reports.kde.org/en/projects/libs-playground-kirigami-components-framework/commits_report

I'm not sure it makes sense having this information on the product pages... :shrug:

mmustac added a subscriber: mmustac.May 14 2019, 6:47 AM

If ind it quite a pitty that there are only so few informations about all those cool applications which are part of the great KDE project.

Just for reference:
Apps page from GNOME wiki: https://wiki.gnome.org/Apps/Calendar
Apps page from elementaryOS app store: https://appcenter.elementary.io/com.github.artemanufrij.playmymusic/

What I really like about the GNOME page is the structure of several "tabs" which enlight one topic per page.
We have a similar "Development Information" in the side panel but I always overlooked it.
And at least in my opinion it would be great to see how development is ongoing for those applications which could be seperated in one "development" tab ?

So the current view could be the "common" tab at the default one. But another "development" tab could hold all those nice informations whch could help interested people into contributing ?
There would be so much room for all kind of informations. Those commits report looks quite good. Maybe also get a list of latest commits or current task which are under development ?
The easiest, but maybe not the most appealing one would be an iframe from the corresponding phab page ?

We have the possibility to achive something great and get the most out of those application pages.

sitter added a comment.May 14 2019, 9:07 AM

Let's not blow up the scope. The concern of this task is making the backend data maintainable and refreshing the page style a bit.

jriddell added a comment.May 22 2019, 11:32 AM

Groovy, I'm happy with the code now and its output
http://apps.kde.org.uk/applications/
Now we need to go over the metadata and make sure it's all valid.
Longer term tasks are tiding up the design of the pages in various ways.
Also adding in support for non-apps like icon themes, plasmoids, plugins etc
https://invent.kde.org/jriddell/kde-applications-appstream/blob/master/appstream.rb#L284
Also make a wiki page to explain all this

jriddell added a comment.May 22 2019, 11:55 AM

Also looking for what apps are maintained but missing appstream metadata
Also looking for what apps are unmaintained and getting them moved to unmaintained

jriddell updated the task description. (Show Details)May 22 2019, 12:28 PM
jriddell updated the task description. (Show Details)May 22 2019, 1:05 PM
ngraham added a comment.May 22 2019, 1:18 PM

Is the visual design of http://apps.kde.org.uk/applications/ within the scope of this task. or can/should that be worked on after it's already in production?

jriddell added a comment.May 22 2019, 2:14 PM

For Longer term issues I made some more tasks under the 'KDE Applications' tag/project in Phab
https://phabricator.kde.org/project/board/196/

e.g.
https://phabricator.kde.org/T10968 Make kde.org/applications more pretty

aacid added a comment.May 22 2019, 8:50 PM

What's the plan to not breaking all the links in the world to our applications pages?

jriddell added a comment.May 22 2019, 11:04 PM
In T10159#185906, @aacid wrote:

What's the plan to not breaking all the links in the world to our applications pages?

Without any further details I'm unable to respond

aacid added a comment.May 22 2019, 11:18 PM

Rephrase: What is the plan to keep https://kde.org/applications/graphics/okular working?

ognarb added a subscriber: ognarb.May 23 2019, 10:45 AM
jriddell added a comment.May 23 2019, 10:47 AM

Answering comment on T10967

In T10967#185904, @ognarb wrote:

Unmaintained apps in the list

  • Karbon is officially unmaintained (say the small popup when you open the app) but there is a small commit activity.

Added to unmaintained list kept locally in these scripts.

Maintained apps not in the list:

  • KDE Connect

It's a plasmoid but I added an appstream desktop app file as a work around for this.

  • Koko

It's in Playground with no sign of release

  • SymbolEditor

Icon needed adding to breeze

  • Rkward

Playground, no release imminent

  • Wacom tablet support

KCM see T10969

jriddell added a comment.May 23 2019, 11:07 AM
  • pimsettingexporter the name is not correct

Updated

Fixed!

  • Tellico has an outdated screenshot

Updated (but it's not the only one)

  • Apper is the only one in the setting category. And I'm not sure is the right category.

I removed this

  • Zanshin has a big TODO in the description

Well it is a TODO manager :)

jriddell updated the task description. (Show Details)May 23 2019, 1:13 PM
jriddell updated the task description. (Show Details)May 23 2019, 2:54 PM
jriddell updated the task description. (Show Details)May 23 2019, 5:01 PM
jriddell updated the task description. (Show Details)May 23 2019, 5:31 PM
jriddell updated the task description. (Show Details)May 24 2019, 1:18 PM
jriddell updated the task description. (Show Details)May 24 2019, 2:02 PM
jriddell updated the task description. (Show Details)May 24 2019, 2:35 PM
jriddell updated the task description. (Show Details)May 24 2019, 3:17 PM
jriddell updated the task description. (Show Details)May 24 2019, 3:52 PM
jriddell updated the task description. (Show Details)May 24 2019, 4:20 PM
jriddell updated the task description. (Show Details)May 24 2019, 7:00 PM
aacid added a comment.May 24 2019, 8:33 PM
In T10159#185920, @jriddell wrote:

Works fine here
http://apps.kde.org.uk/applications/graphics/okular

Ah, i didn't realize it would be working given the URL that the "main" page offers is http://apps.kde.org.uk/applications/graphics/org.kde.okular

aacid added a comment.May 24 2019, 8:36 PM

http://apps.kde.org.uk/applications/unmaintained/knode/ seems to be missing ?

should i try to tes all the other apps? have you?

jriddell added a comment.May 25 2019, 6:16 PM
In T10159#186074, @aacid wrote:

http://apps.kde.org.uk/applications/unmaintained/knode/ seems to be missing ?

should i try to tes all the other apps? have you?

Probably wait until we get it moved to the KDE servers to check closer

aacid added a comment.May 25 2019, 10:05 PM

I think it would be better to make sure all links are there, that way no regression would happen

jriddell added a comment.May 31 2019, 7:44 PM

Now up at https://kde.org/applications-ng/ do check it over

Issues I can see are
https://kde.org/applications-ng/system/kioskadmintool/ 404
No KHelpCentre (probably needs categories special cased)
No Konqueror
No Kexi
No Krita
Build at https://binary-factory.kde.org/view/Websites/job/Website_kde-org-applications/

jriddell added a comment.May 31 2019, 7:47 PM

Also check why Install button on applicationpage is not styled.

Then check the appstream app ID is right and works in Discover with all the apps. I changed Krita's launchable-id to not have the .desktop but I'm told it needs it so review that.

aacid added a comment.Jun 1 2019, 9:52 AM

Nice work in general, some little things i found.

Do you think "Page Setup Info" should be there? Feels weird to me, we don't really have that in the rests of the web.

To me it seems that adding some spacing between the text and the icon would make it breathe a bit better, see Spectable for example at https://kde.org/applications/graphics/ and at https://kde.org/applications-ng/utilities/

No Kst
no kftpgrabber

KMid is described as "blogging client". Too much copy&paste?

The list of things is translated but not sorted by translation, which maybe feels a bit weird? See how "Batalla Naval" is not the first in https://i.imgur.com/vDyo7Cw.png

Actually what is the sorting pattern? https://i.imgur.com/yN6gZLp.png Seems strange that symbol editor is the first one

For apps that are unmaintained on subversion like kappfinder the links at https://kde.org/applications-ng/unmaintained/org.kde.kappfinder/development are broken. Not huge important, but since the old page has it right, would be cool for it to be right (or at least don't show anything at all so it's not wrong?)

jriddell added a comment.Jun 1 2019, 10:42 AM
In T10159#186751, @aacid wrote:

Do you think "Page Setup Info" should be there? Feels weird to me, we don't really have that in the rests of the web.

Yeah it's a complex site and there will be 100 people with an interest in keeping it up to date so I think an obvious link on how to do that is useful. I've moved it to float: right now so it's more out the way.

To me it seems that adding some spacing between the text and the icon would make it breathe a bit better, see Spectable for example at https://kde.org/applications/graphics/ and at https://kde.org/applications-ng/utilities/

Yes, added.

No Kst
no kftpgrabber

typo in kftpgrabber and kst has no release since 2014 so put both in unmaintained now

KMid is described as "blogging client". Too much copy&paste?

Fixed

The list of things is translated but not sorted by translation, which maybe feels a bit weird? See how "Batalla Naval" is not the first in https://i.imgur.com/vDyo7Cw.png
Actually what is the sorting pattern? https://i.imgur.com/yN6gZLp.png Seems strange that symbol editor is the first one

Dunno, as with the rest of kde.org I consider translations to be an incomplete afterthought which someone should think about and complete more.

For apps that are unmaintained on subversion like kappfinder the links at https://kde.org/applications-ng/unmaintained/org.kde.kappfinder/development are broken. Not huge important, but since the old page has it right, would be cool for it to be right (or at least don't show anything at all so it's not wrong?)

I just added an unmaintained message now

jriddell updated the task description. (Show Details)Jun 1 2019, 10:43 AM
jriddell added a comment.Jun 2 2019, 9:45 AM

kst needs to get into the right category
kexi needs to display, currently it has an icon set to 'kexi-3.3' which does not exist, e-mailed maintainer to ask what he wants to do

aacid added a comment.Jun 2 2019, 9:57 AM
In T10159#186753, @jriddell wrote:
In T10159#186751, @aacid wrote:

Do you think "Page Setup Info" should be there? Feels weird to me, we don't really have that in the rests of the web.

Yeah it's a complex site and there will be 100 people with an interest in keeping it up to date so I think an obvious link on how to do that is useful. I've moved it to float: right now so it's more out the way.

I understand you feel this is a complex site, but the 99.9999% of people looking at it should just be users, for them that information is unwanted and potentially confusing. If you want to let people how this page is created, email kde-devel, blog about it, add a HTML comment on top of the page code, I still don't think having the information shown to the regular visitor makes sense.

jriddell updated the task description. (Show Details)Jun 2 2019, 8:44 PM
jriddell added a comment.Jun 4 2019, 10:03 AM

The Page Setup Link doesn't affect the 99.9% of visitors who look at the page, but for those who are part of KDE and want to help keep app details up to date that should be made easy to understand how to do that. Currently the website is largely abandoned and bitrotting because of the social and technical difficulty in making changes to it. This is a way to help fix that.

ognarb added a comment.EditedJun 4 2019, 12:48 PM

There is still some 404 links.

wget --recursive --no-parent https://kde.org/applications-ng/ -nv 2>&1 >/dev/null | grep -e "404" | wc -l is 5

and for example kde.org/applications-ng/games/org.kde.gcompris redirecto to applications/educations and not kde.org/applications-ng/education/org.kde.gcompris

jriddell closed this task as Resolved.Jun 5 2019, 11:01 AM
ngraham added a comment.Jun 5 2019, 1:26 PM

Fantastic work. Looks and feels really good!

elvisangelaccio added a comment.Jun 9 2019, 9:20 AM

@jriddell Would it be possible to have https://kde.org/applications/utilities/ark instead of https://kde.org/applications/utilities/org.kde.ark?

i.e. add some .htaccess magic that rewrites the URLs without the org.kde. prefix?
I feel like this prefix is just an appstream implementation detail that shouldn't be exposed in the app URL.

jriddell added a comment.Jun 10 2019, 10:46 AM
In T10159#187560, @elvisangelaccio wrote:

@jriddell Would it be possible to have https://kde.org/applications/utilities/ark instead of https://kde.org/applications/utilities/org.kde.ark?

i.e. add some .htaccess magic that rewrites the URLs without the org.kde. prefix?
I feel like this prefix is just an appstream implementation detail that shouldn't be exposed in the app URL.

@sitter any comment on why this URL scheme was chosen?

ngraham added a comment.Jun 10 2019, 2:09 PM
In T10159#187560, @elvisangelaccio wrote:

I feel like this prefix is just an appstream implementation detail that shouldn't be exposed in the app URL.

+1

sitter added a comment.Jun 11 2019, 10:44 AM
In T10159#187658, @jriddell wrote:
In T10159#187560, @elvisangelaccio wrote:

@jriddell Would it be possible to have https://kde.org/applications/utilities/ark instead of https://kde.org/applications/utilities/org.kde.ark?

i.e. add some .htaccess magic that rewrites the URLs without the org.kde. prefix?
I feel like this prefix is just an appstream implementation detail that shouldn't be exposed in the app URL.

@sitter any comment on why this URL scheme was chosen?

It's the same scheme as before except instead of an arbitrary ID it now has the appstream ID.

• fusitter added a subscriber: • fusitter.Aug 29 2019, 4:00 PM

[spam comment removed by sysadmin]