Add new categories for Cantor for documentation files
Closed, ResolvedPublic
Actions

Description

In T9980 categories for example projects were added for Cantor. As next we want to allow to download the documentation files in QtHelp format for the varios systems supported by Cantor (Maxima, Octave, etc.).
Ideally we have the following structure on store.kde.org for Cantor:

Cantor
    \Examples
        \Maxima
        \Octave
        \R
        \Python
        ...
    \Documentation
        \Maxima
        \Octave
        \R
        \Python
        ...

Would it be possible to realize something like this?

For example projects we'd continue working with the already existing knscr file https://invent.kde.org/education/cantor/-/blob/master/src/cantor.knsrc and adjust the descriptions (use "Maxima" instead of "Cantor (Maxima)", etc.), for the documentation files I'll create a new knsrc file.

Related Objects

Mentioned Here: T9980: Creating categories for Cantor on kde.store for publish/load worksheets.

asemke created this task.May 30 2021, 7:37 PM

Hey there - this /should/ be doable yes. We have a couple of things to care for here to fully handle this on our side (which currently does not really know how to do the parent/child relationship the server returns right now, because it's outside of the OCS specification, but also that is something we should fix). That said, this only really affects some basics of the display, and since you're wanting to show them in two different locations anyway, you'll not need to worry about that part (and i guess i'm only really saying it here to have it on record that i need to get that done ;) )

So, basically the original categories would retain their existing names in the backend (meaning your knsrc file would have the same list in the Categories entry - this is one of those tricks where we've got a separate human readable name for things, to avoid trouble with suddenly ending up without store access in old Cantor installations), and then a set of new categories for the new documentation categories, which you probably want to give a similar category naming style to what you've got for Examples, just for a bit of consistency in your knsrc files Categories lists. The human-readable names then end up mapped from those, and don't (technically at least) have to be at all similar, which means you end up with something like the following, where the first is what you'd write in the knsrc file's Categories list, and the others are what the user will see in the UI. That is, these categories would then be shown like this on store.kde.org (and eventually also in the knewstuff dialog, and indeed in Discover, which already has some understanding of the parent/child relationship for categories, with some caveats similarly irrelevant to this situation):

Cantor -> Cantor
  Cantor Examples -> Examples
    Cantor (Maxima) -> Maxima
    Cantor (Octave) -> Octave
  Cantor Documentation -> Documentation
    Cantor Documentation (Maxima) -> Maxima
    Cantor Documentation (Python) -> Python

(etc)

The documentation part's mostly to give a suggestion that retains the style of the examples entries, with a bit of extra flavour to make it distinguishable.

All this, really, to give you a bit of an idea for what the request here would want to probably look like in the end - a list of the existing categories, with new human-readable names in the style above, which then is what is shown in the dropdown in the NewStuff categories pickers, and in Discover.

Also, sorry it took so long to get back to you on this, entirely missed this!

This is currently the structure we have:

Cantor
 Cantor (Maxima)
 Cantor (Sage)
 Cantor (KAlgebra)
 Cantor (Qalculate)
 Cantor (Python 2)
 Cantor (Python 3)
 Cantor (Scilab)
 Cantor (Octave)
 Cantor (R) 
 Cantor (Lua)
 Cantor (Julia)

If you need a different categories, then that is quite possible to implement something like that. We are careful to keep the naming of the categories as clear as possible (unique names).
Although this is not necessary from the source code, it makes it easier for us to deal with the categories.

Is it correct, that only one of the existing category so far shows a
product?

Those existing categories should all become or rather stay "Examples" for
Cantor, right?

Then the new categories should be Documentation ones?

Greetings, Clemens.

In T14532#258271, @starbuck wrote:

Is it correct, that only one of the existing category so far shows a
product?

If you mean with "product" that one example project file for Octave then yes, this is correct.

Those existing categories should all become or rather stay "Examples" for
Cantor, right?

Then the new categories should be Documentation ones?

Yes, exactly. So, we have Cantor supporting different "backends" like Maxima, Octave, R, etc. and for every backend we want to have examples and documentation.

In T14532#258182, @alexanderschmidt wrote:
This is currently the structure we have:
Cantor
 Cantor (Maxima)
 Cantor (Sage)
 Cantor (KAlgebra)
 Cantor (Qalculate)
 Cantor (Python 2)
 Cantor (Python 3)
 Cantor (Scilab)
 Cantor (Octave)
 Cantor (R) 
 Cantor (Lua)
 Cantor (Julia)
If you need a different categories, then that is quite possible to implement something like that.

The request here is to have new categories as children of the same and already existing parent. If I understood @leinir correctly, this should be doable/possible.

We are careful to keep the naming of the categories as clear as possible (unique names).
Although this is not necessary from the source code, it makes it easier for us to deal with the categories.

Having

Cantor
    Maxima
    Sage
    ...

is more clear than having

Cantor
    Cantor (Maxima)
    Cantor (Sage)
    ...

Ideally we can differentiate between the technical IDs and user friendly description texts.

Restructured it in the suggested way:
https://store.kde.org/browse/cat/649/order/latest/

For Cantor Documentation, I only added one subgroup so far for testing,
then if all is fine, we can add the rest:
https://store.kde.org/browse/cat/651/order/latest/

Greetings, Clemens.

@starbuck thanks, Clemens! I'm trying now with the following knsrc file

[KNewStuff3]
Name=Documentation
ProvidersUrl=https://autoconfig.kde.org/ocs/providers.xml
Categories=Cantor Documentation (Maxima)
TargetDir=cantor/documentation
Uncompress=archive

It seems to work but I'm getting the same category three times:

Do I see it correctly that we cannot use and differentiate between the technical IDs for the categories (for example "cantor_documentation_maxima") and the description texts exposed to the users (for example "Maxima") and this is the reason why we need to use the full text "Cantor Documentation (Maxima)" here

instead of a simple "Maxima" to get a unique name?

I'd like to upload the documentation for maxima now. I registered on store.kde.org but I don't see how to add files to a category. Can you please help with this?

That you see the category 3xtimes is weird, I think @leinir should maybe have an idea as it is from within KDE.

As for uploading stuff to the categories, once you logged in you should see a point "Add Product" in the dropdown menu when you click on your avatar in the upper right corner.

Since we allow basically any user to upload things also be shared with Gnome-Look, XFCE, etc. we grouped everything (hopefully) in a logical way, so you find Cantor hierachially here:

After filling out at least the mandatory fields with *, you can advance the tabs and hopefully upload anything under "3. Files" tab section.

Let me know how it works or if you have any more questions.

Also let me know if the other Documentation categories should be added in the same way, or if you like to change anything.

In T14532#258398, @starbuck wrote:

That you see the category 3xtimes is weird, I think @leinir should maybe have an idea as it is from within KDE.

As for uploading stuff to the categories, once you logged in you should see a point "Add Product" in the dropdown menu when you click on your avatar in the upper right corner.

Thanks, that worked:

Something is wrong here with the time, it's showing "added 3 hours ago" (it was done 1 minute ago or so) but this shouldn't block me now I guess :-)

I don't see this product yet on https://store.kde.org/browse/cat/651/order/latest/. Is there any delay between publishing on pling and getting it visible on store.kde.org?

Great! There is a slight buffer for store.kde.org if i am correct, but it
shows up already by now over here :)

In T14532#258421, @starbuck wrote:

Great! There is a slight buffer for store.kde.org if i am correct, but it
shows up already by now over here :)

Yes, I see it now, too :-)

I still see the category for Maxima three times and there is still no content shown in KDE's "get new stuff" dialog but I think this can be addressed independently of the registration of the new categories, right? If yes, let's please go ahead and register for all backends supported by Cantor:

Cantor Documentation (Sage),
Cantor Documentation (KAlgebra),
Cantor Documentation (Qalculate)
Cantor Documentation (Python)
Cantor Documentation (Scilab)
Cantor Documentation (Octave)
Cantor Documentation (R)
Cantor Documentation (Lua)
Cantor Documentation (Julia)

I assume it will be sorted automatically.

In T14532#258395, @asemke wrote:

It seems to work but I'm getting the same category three times:

Just had a quick look, and the api server is returning three entries for that category - as in, there are three entries for the category with id 651 here: https://api.kde-look.org/ocs/v1/content/categories :)

In T14532#258428, @leinir wrote:

Just had a quick look, and the api server is returning three entries for that category - as in, there are three entries for the category with id 651 here: https://api.kde-look.org/ocs/v1/content/categories :)

can you fix this?

We are looking into this, yes.

Is it still 3 entries? It seems a hard to find issue, so hopefully some
change we did may already solved it on the api end.

At least over here i only see one time 651 now.

Yup, looks like just the one 651 now in the /content/categories output :)

Great. I will add the new categories then, assuming it all goes well.

@leinir I'm testing with the documentation file I uploaded for Maxima. It's a zip archive with two files inside that is unpacked upon downloading it. KSN::Entry::installedFiles() is returning

"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/*"

instead of a list with two file names that I would expect

"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/help.qch"
"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/help.qhc"

I can work around this in the code now but is it really the correct behavior or am I rather doing something wrong?

Also, when debugging today and trying to download again I ran into this problem

After closing that "An Error Occurred" pop-up, the tile still had the status "Working" and that icon continued spinning. Looks like the KNS dialog doesn't handle this kind of errors correctly.

I've just downloaded this documentation through the web interface without issue. I have also downloaded Cantor to test but I am assuming this KNHS feature isn't released yet? Is it in master or a MR I can grab and test with?

justinzobel moved this task from To Do to Awaiting User Feedback on the KDE Store board.Jun 21 2021, 1:31 AM

In T14532#258514, @justinzobel wrote:

I've just downloaded this documentation through the web interface without issue.

yes, it's working now without problems but yesterday the server had issues and such cases are not handled correctly in KNS::dialog.

I have also downloaded Cantor to test but I am assuming this KNHS feature isn't released yet? Is it in master or a MR I can grab and test with?

It's in the branch gsoc20_documentation. The relevant logic is in QtHelpConfig::knsUpdate() (src/backends/qthelpconfig.cpp). Here, the size of the list is 1. To get to this dialog, go to the application settings, select Maxima backend and navigate to the tab "Documentation". Thanks.

Update:
All Documentation categories have now been added and should show up within
24 hours or earlier.

If anything else needs to be done, just let us know.

In T14532#258488, @asemke wrote:
@leinir I'm testing with the documentation file I uploaded for Maxima. It's a zip archive with two files inside that is unpacked upon downloading it. KSN::Entry::installedFiles() is returning
"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/*"
instead of a list with two file names that I would expect
"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/help.qch"
"/home/alex/.local/share/cantor/documentation/Maxima_v5.44/help.qhc"
I can work around this in the code now but is it really the correct behavior or am I rather doing something wrong?

This is indeed the expected behaviour - what you are seeing is that it will list only the root entries of the decompressed archive, not the children (and the /* bit basically just means "this is a directory"). i can see how that's a little odd, but it came out of situations where caches were built inside the installed location, which then in turn would result in a complete inability to fully uninstall stuff, since knewstuff would remove the entries, leaving behind those cache bits, and then report that it had failed to uninstall things, when in reality it was an application not leaving installed data sufficiently alone. So, we had to fix it in knewstuff, and what you see there is the result. It's suboptimal, but kind of the best tradeoff we could come up with at the time. Hope that makes sense :)

Also, when debugging today and trying to download again I ran into this problem

After closing that "An Error Occurred" pop-up, the tile still had the status "Working" and that icon continued spinning. Looks like the KNS dialog doesn't handle this kind of errors correctly.

Yes, it was only the other day that i merged the plumbing to enable handling those more gracefully than what you see there - but yes, handling some of the various ways in which downloading and whatnot can fail more gracefully is on the docket :)

In T14532#258564, @starbuck wrote:

Update:
All Documentation categories have now been added and should show up within
24 hours or earlier.

If anything else needs to be done, just let us know.

@starbuck thanks. Can you please rename "Cantor Documentation (Python 2)" to "Cantor Documentation (Python)". Also, the categories for Sage, Qalculate, Scilab and R are missing.

Will rename then also the Example one to Python accordingly?
I did add all of the categories inc. Scilab, R, ... so I wonder why they
are missing. Will investigate.

They show up under store.kde.org fine?

In T14532#258616, @starbuck wrote:

Will rename then also the Example one to Python accordingly?

Yes, let's please rename it also for examples. In the past we supported python2 and python3 but after the sunset of python2 there is only "Python" now.

In T14532#258617, @starbuck wrote:

They show up under store.kde.org fine?

Not for me...

I think its the resolution of that cut-out? like can you just scroll down in your browser and then see Python 3,... and the rest?

In T14532#258620, @starbuck wrote:

I think its the resolution of that cut-out? like can you just scroll down in your browser and then see Python 3,... and the rest?

I can see all categories now, must have been a caching issue somewhere maybe...

We still need to correct the category for python. In the past we supported python2 and python3 and had to differentiate between these two major version. Now we support only Python, being internally Python3, but we talk about Python everwhere without refering to any version similarly to how we don't refer to any version of any other languages. So, we only need one single category for Python - "Cantor Documentation (Python)". And similar for examples. Can you please remove that Python3 specific categories?

So Python 2 stays Python 2 and Python 3 should be simply Python then?

In T14532#258656, @starbuck wrote:

So Python 2 stays Python 2 and Python 3 should be simply Python then?

No. We should have only one single category for Python without any mentioning of any versions. So, it should be just "Cantor Examples (Python)" and "Cantor Documentation (Python)".

Okay, now I renamed both Python 3 to simply Python.
Can you just doublecheck and confirm I should remove the Python (2) ones
then altogether?

If you still need to distinguish some older Python 2 files, we can also add
a dropdown selectbox via an attribute, so people can, but dont need to pick
2 or 3 or both to signal compatibility per File.

In T14532#258809, @starbuck wrote:

Okay, now I renamed both Python 3 to simply Python.
Can you just doublecheck and confirm I should remove the Python (2) ones
then altogether?

Yes, please remove it completely. We should only have "Python".

For the documentation I still see "Python 2" and no "Python":

Is it showing correctly for you and you only have "Python"?

In T14532#258582, @leinir wrote:

This is indeed the expected behaviour
[...]

Ok, thanks for the explanations. I adjusted the code accordingly and everything is working now.

One more question related to KSN. In Cantor the settings dialog looks like

Here, when openning KNS3::DownloadDialog I'd like to automatically select the proper category. So, if I'm modifying the settings for Maxima and want to add a new documentation for it, I click on "Get New Documentation" and I want to see the content for the category "Cantor Documentation (Maxima)" only. The content from other categories doesn't make much sense. Can we add to the APIs of KNS3::DownloadDialog functions to pre-select the category and to hide that ComboBox?

Also, the deletion of the installed documentation files should be done via that delete button you see on the screenshot. We copied this widget and logic from KDevelop to provide a similar user experience across multiple applications. The deletion via this button is not possible now since this functionality is not exposed in KNewStuffCore. We have to ask the user to open the DownloadDialog, to uninstall from there and then to parse the uninstalledEntries() and to remove the corresponding row in the table seen on the screenshot. This is not so optimal from the UX perspective... Do we have a chance to delete via KNewStuffCore direclty?

In T14532#259056, @asemke wrote:

In T14532#258582, @leinir wrote:

This is indeed the expected behaviour
[...]

Ok, thanks for the explanations. I adjusted the code accordingly and everything is working now.

Great! :)

Here, when openning KNS3::DownloadDialog I'd like to automatically select the proper category. So, if I'm modifying the settings for Maxima and want to add a new documentation for it, I click on "Get New Documentation" and I want to see the content for the category "Cantor Documentation (Maxima)" only. The content from other categories doesn't make much sense. Can we add to the APIs of KNS3::DownloadDialog functions to pre-select the category and to hide that ComboBox?

Hmm... That sounds like a good idea, yeah, could you create a wishlist entry for knewstuff on bugs.kde.org for that? There's no api for that right now, but it's definitely a good idea :)

Also, the deletion of the installed documentation files should be done via that delete button you see on the screenshot. We copied this widget and logic from KDevelop to provide a similar user experience across multiple applications. The deletion via this button is not possible now since this functionality is not exposed in KNewStuffCore. We have to ask the user to open the DownloadDialog, to uninstall from there and then to parse the uninstalledEntries() and to remove the corresponding row in the table seen on the screenshot. This is not so optimal from the UX perspective... Do we have a chance to delete via KNewStuffCore direclty?

You don't really need to do it through KNewStuff (it's of course an issue you run into with a lot of things), but since it's a behavioural change, when we implemented the automatic dead entries removal, we left it disabled by default because it'd break the way some apps handle it. But, in short, you should be able to just add RemoveDeadEntries=true to your knsrc (https://api.kde.org/frameworks/knewstuff/html/index.html shows the details, we recently polished the docs there a fair bit)

In T14532#259058, @leinir wrote:

In T14532#259056, @asemke wrote:

In T14532#258582, @leinir wrote:

This is indeed the expected behaviour
[...]

Ok, thanks for the explanations. I adjusted the code accordingly and everything is working now.

Great! :)

Here, when openning KNS3::DownloadDialog I'd like to automatically select the proper category. So, if I'm modifying the settings for Maxima and want to add a new documentation for it, I click on "Get New Documentation" and I want to see the content for the category "Cantor Documentation (Maxima)" only. The content from other categories doesn't make much sense. Can we add to the APIs of KNS3::DownloadDialog functions to pre-select the category and to hide that ComboBox?

Hmm... That sounds like a good idea, yeah, could you create a wishlist entry for knewstuff on bugs.kde.org for that? There's no api for that right now, but it's definitely a good idea :)

Done in https://bugs.kde.org/show_bug.cgi?id=439210.

You don't really need to do it through KNewStuff (it's of course an issue you run into with a lot of things), but since it's a behavioural change, when we implemented the automatic dead entries removal, we left it disabled by default because it'd break the way some apps handle it. But, in short, you should be able to just add RemoveDeadEntries=true to your knsrc

I think it's more cleaner from the architectural point of view to delete the content that was downloaded from KnewStuff also via KNewStuff than delete the file on the hard disk directly and rely on some cleanup mechanisms happening in this framework later. This discussion is going beyond the actual request for new categories in this ticket. Since all categories are available now and everything seems to work now in Cantor, should we close this ticket and continue the discussion on Matrix maybe?

(https://api.kde.org/frameworks/knewstuff/html/index.html shows the details, we recently polished the docs there a fair bit)

The documentation is in a much better shape now, indeed!

asemke closed this task as Resolved.Jul 31 2021, 4:49 PM

Add new categories for Cantor for documentation filesClosed, ResolvedPublicActions

Description

Related Objects

Add new categories for Cantor for documentation files
Closed, ResolvedPublic
Actions