Something like this is outside of Sysadmin's scope - as a general rule we don't provide migration services between different data formats.
Once you have everything in Sphinx form we can certainly discuss further though.

I'll now detach this from Sysadmin and reassign it to Digikam

Hi Ben,

For the migration of current data, the digiKam team will do it as well, as the format to host handbooks is a little bit different than DocBook, but so far more simple to understand/learn/teach/maintain. I waste a lot of time trying to convince contributors to use DocBook, but it's very complicated and most of the time without success.

About the infrastructure, I think the list below must be done by the system-admin to quickly work on the migration:

• The core Sphinx must be initialized in the git repository,
• The base empty documentation web site must be initialized. Something based on KDenlive will be enough, as i can see the Krita version as a more complex approach,
• A CI must be configured to be able to check integrity of documentation code,
• The relevant i18n/i10n scripts must be adapted to the new Sphinx documentation to export strings to translators,
• A Jenkins job must be add to publish the documentation,
• A new https://docs.digikam.org web site must be created to host Sphinx based documentation.

Let me know if I'm wrong here...

Best regards

Gilles Caulier

I'm afraid we don't do the work of preparing the repository - that will need to be something undertaken by the DIgikam team (it was done by Kdenlive/Krita folks for theirs).

With regards to jobs for CI and website deployment, they require the repository be minimally functional for building with Sphinx first - so we'll come in once you have an initial base in place.

I'm afraid we don't do the work of preparing the repository - that will need to be something undertaken by the >Digikam team (it was done by Kdenlive/Krita folks for theirs).

Well i suppose that i can clone the KDenlive documentation repository and just import the Sphinx code as well in digiKam documentation repository.

I will tag current implementation based on DocBook for archiving and backport KDenlive contents instead in master. Of course some tune needs to be done to work the rest of infrastructure as i18n/i10n.

I'm fine ?

Gilles

digiKam git documentation based on DocBook is now tagged with v7.9.0:

https://invent.kde.org/documentation/digikam-doc/-/tree/v7.9.0

Gilles

Eugen, Julius,

As you works on the KDenlive Documentation repository, Perhaps you can guide us a little bit on the currrent digiKam DocBook repository to Sphinx migration ?

Thanks in advance

Best regards

Gilles Caulier

I think Julius can advise for the technical part: migrate from docBook -> Sphinx (we migrated from KDE-UserBase-Wiki to Sphinx).

Eugen

Thanks Eugen,

I will probably plan to create one by one the new Handook parts based on Sphinx container (reStructuredText) using strings already present in the DocBook. This will limit certainly the job from translators. This part is not really a problem, just a question of time.

I would like to talk in this part about the infrastructure :

• Sphinx configuration,
• connection with i18n/i10n world,
• Website dedicated to host documentation,
• Jenkins jobs,
• Unit tests,
• etc.

In fact the technical arcane to initiate the project. You have certainly talk with Krita team about this point to start the KDenlive doc project. There are few knowledgment to take a care for people who will discover Sphinx framework ?

Best

Gilles Caulier

Hi Wolthera,

As you work on Krita documentation project, have you any guidance to share on the digiKam documentation port to Sphinx framework, as explained before in this thread ?

Thanks in advace

Gilles Caulier

Hi Gilles,

first you should have a minimal working version locally. Ones thats the case you can start to fill it with content think about all the infrastructural parts.

For the basic setup you basically need to care about:

• Makefiles to simplifiy build (not strictly necessary). Kdenlive has two a "normal" one and a make.bat for windows
• The sphinx configuration in conf.py. With the comments there it should be mostly self explaining.
• As you can see in the config we have some custom css/js on top of the sphinx_rtd_theme theme in the resources folder. These customizations are mainly for the language switcher and for the panels on the start page:
  
  The favicons do also live here too. We also customzie the footer and add a header for matomo tracking with https://stats.kde.org/ (if you are interested in this, you need to coordinate with sysadmins. You should not use the same code as kdenlive as eg. the side ID it is specific to the kdenlive manual)
• .gitignore to avoid commiting build results of local builds
• For translations a StaticMessages.sh file. See these commits: https://invent.kde.org/documentation/docs-kdenlive-org/-/commit/87b06ee5e854cf18599f26d46ead69964cc50ccf, https://invent.kde.org/documentation/docs-kdenlive-org/-/commit/a5e8ea28416e7f8986d3f77cd4a0e22fe7a27aca
• As Eugen said we migrated from the wiki and I wrote some python scripts to automatically convert the content. They live in https://invent.kde.org/documentation/docs-kdenlive-org/-/tree/master/tools. However since you migrate form DocBook you either need to tweak the scripts or to it manually.
• Add a 404handler.php this will also take care of choosing the currect default language based on the browser language for the user.
• Concerning tests, we only have one to make sure that file paths are not to long as this will cause problems with Scripty.

On the infrastuture part it is:

• Ask Sysadmins to setup the domain
• Ask sysadmins to configure on the serverside to use 404handler.php
• Make sure the repo is correctly registered (especally the i18n part). Sysadmins now about details. See this commit
• Enable Scripty for translations on the documentation repository as done in this commit (in dough ask the i18n team for help)
• Add Jenkins job that takes care of building and publishing with a sniped like

	{
		"name": "docs-kdenlive-org",
		"repository": "documentation/docs-kdenlive-org",
		"branch": "master",
		"cron": "",
		"type": "sphinx-app-docs",
		"deploylangs": "en de fr nl es tr da pt_BR ru uk_UA zh_CN zh_TW ja ko ca sk sv it",
		"deploypath": "/srv/www/generated/docs.kdenlive.org/"
	}

added to https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/staticweb/custom-jobs.json

Not sure if I forgot anything, but thats all I can recall right now. Hope that helps.

Thanks Julius for this excellent feedback. Now i know how to start with the migration.

Best

Gilles

Hi Ben,

What's about the system admin tasks listed by Julius previously ?

Best

Gilles

As stated previously, the initial work for Sphinx to be able to run must be in place within the repository first before anything from a Sysadmin perspective can be actioned.

Ok, Must i use master or a devel branch in git repository ?

The system has flexibility as to the branch that we use.

Thanks Ben. I will backport and configure KDenlive files in "sphinx" branch from digiKam documentation repository.

I start to back-port KDenlive implementations:

https://invent.kde.org/documentation/digikam-doc/commit/1245c681a2bc4fd693d4718871f9664e438a1b1e

All is not yet completed...

Hi Ben,

The minimum Sphinx configuration with the intro entries is now implemented in digiKam-doc repo, "sphinx" branch. It compiles and generates HTML versions properly on my computer.

Basic Setup

I review all points listed by jlskuz previously about Basic Setup. I commented on the Matomo tracking code as it uses KDenlive ID. It must be customized before to be re-enabled:

https://invent.kde.org/documentation/digikam-doc/-/blob/sphinx/src/resources/templates/layout.html

Notes :
 

• Unlike KDenlive, I stored the root of documentation implementations in src/ for better readability. People who will work on this doc will not be developers, so we need to simplify the future tasks.

• Also, we will store screenshots in each doc sub-sections, in images/ sub directory, not in a central directory. This will simplify the maintenance, and as I can see this does not have a side effect while compiling. Example for the doc section "getting_started":

https://invent.kde.org/documentation/digikam-doc/-/tree/sphinx/src/getting_started/images

Infrastructure Part

All these points still to do. I think all is ready to be exported to a dedicated domain. I use a similar typo than KDenlive, as docs.digikam.org.

For the translations, i use the same typo than KDenlive, as docs_digikam_org_ as prefix:

https://invent.kde.org/documentation/digikam-doc/-/blob/sphinx/StaticMessages.sh#L4

Voilà, I hope to forget nothing...
 
Best Regards

Gilles

The Binary Factory job to build the site has now been provisioned: https://binary-factory.kde.org/job/Website_docs-digikam-org/1/console
It fails however, likely because Digikam is not mirroring the format that Kdenlive and Krita have chosen. You need to match how they have their setup i'm afraid.

The web server configuration, complete with 404 handler, is now in place and DNS has been provisioned.

You will need to discuss i18n/scripty matters with Luigi/Albert.

Any form of CI is outside of scope at this time.

Hi Ben,

The web domain build is fixed now. Epub build work too from Phabricator.

Gilles

Ben,

The commented Matomo tracking code need to be re-enabled. A new ID must be used (actually it's 34 for KDenlive);

https://invent.kde.org/documentation/digikam-doc/-/blob/sphinx/resources/templates/layout.html#L13

Gilles

I've provisioned ID "38" for that now.

ID 38 enable for stats.kde.org :

https://invent.kde.org/documentation/digikam-doc/commit/5f906c377742c8c047b7a47792c597b15803e14c

Gilles

Hi Julius,

For info, Docbook can be converted to reST using Pandoc tool as explained in this slide :

https://www.slideshare.net/henrich_d/challenge-convert-policy-doc-from-docbook-to-sphinx-77172587

Gilles

Hi Ben,

I advance step by step from Docbook to ReStructuredText migration. There is no technical problem, it's easy and fast. I hope to compile the whole conversion before Christmas. ReStructuredText is a real pleasure to use compared to Docbook.

Now it still the translations point to solve to close this file. The script StaticMessages.sh in the git repository must be ready to export/import strings. We need to enable the i18n workflow...

Gilles

Albert, Luigi,

The infrastructure of digiKam handbook is under migrating from DocBook to RestructuredText/Sphinx. It miss the i18n to enable.
I plan to complete the manual conversion before Christmas, as we have a group of people ready to work on ported documentation to create missing entry in the documentation.

The i18n use the same mechanism than KDenLive and Krita documentation (also based on Restructured/Sphinx). Can you check the missing parts to complete the internationalization of this documentation.

Thanks in advance

Gilles caulier

Personally i know 0 about how that works, but if you're doing the same as the two other places, i don't see why it wouldn't work.

Same - if you start from the StaticMessages.sh script from the other repositories, it should be fine.

Side note: if you are rewriting the manuals from scratch you can use the new suggested license (CC BY SA 4.0), otherwise you need to keep the old GNU FDL license and list all the old contributors.

Hi Albert,

Yes sure, there is no technical blocker here.

See previous comment from jlskuz who works for KDenlive doc. On the infrastructure part it is:

Make sure the repo is correctly registered (especially the i18n part). Sysadmins know about details. See this commit :
https://invent.kde.org/sysadmin/repo-metadata/-/commit/e2391d59be276cb5e61571388b51bb0dd6a843e3

Enable Scripty for translations on the documentation repository as done in this commit :
https://invent.kde.org/sysadmin/l10n-scripty/-/commit/5f8833316aec9dc302f0776da3c55587732be216
(in dough ask the i18n team for help)

Add Jenkins job that takes care of building and publishing with a sniped like

{

		"name": "docs-kdenlive-org",
		"repository": "documentation/docs-kdenlive-org",
		"branch": "master",
		"cron": "",
		"type": "sphinx-app-docs",
		"deploylangs": "en de fr nl es tr da pt_BR ru uk_UA zh_CN zh_TW ja ko ca sk sv it",
		"deploypath": "/srv/www/generated/docs.kdenlive.org/"

}

added to https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/staticweb/custom-jobs.json

Gilles

Hi all,

About the pending tasks:

• "Make sure the repo is correctly registered (especially the i18n part)." --> this is already the case for digikam-doc repository :

https://invent.kde.org/sysadmin/repo-metadata/-/tree/master/projects-invent/documentation/digikam-doc

• "Enable Scripty for translations on the documentation repository" : in the get_paths script, digikam-doc is already registered and switch/case branch return "master" as for documentation-docs-kdenlive-org. It's not perfect as the digikam-doc must be moved to the documentation-docs-kdenlive-org branch, but logically, it will work.

• "Add Jenkins job that takes care of building" : this is already done : https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/staticweb/custom-jobs.json#L145

For this last point, the list of translated languages must be updated when all contents will be ported from DocBook to ReStructuredText (currently EN only).
Also the branch sets in this json config file for digikam-doc ust be changed to master when port will be completed (currently sphinx branch).

Gilles Caulier

Hi Luigi,

I take a look how StaticMessages.sh from this repository must be called daily. The i10n-scripty/process-static-messages.sh script do the job to deal with strings to extract from ReStructuredText files and update templates PO file, and later to push back the PO translations.

https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/staticweb/custom-jobs.json

So, my question is : why i cannot found template PO file generated from RestructuredText files hosted by the "sphinx" branch in the git repository. I can see the kdenlive-docs PO files here :

https://websvn.kde.org/trunk/l10n-kf5/templates/messages/documentation-docs-kdenlive-org/

... but digikam-doc directory still empty :

https://websvn.kde.org/trunk/l10n-kf5/templates/messages/digikam-doc/

It miss something ? Do the PO template files will be only generated when the "sphinx" branch will be merged to "master" ?

Note : I completed 85 % of the DocBook to Sphinx/ResStructuredText migration, and all will be ready for translations in a few days :

https://docs.digikam.org/en/index.html

Thanks in advance, and Merry Christmas to all.

Gilles Caulier

In T16036#285444, @cgilles wrote:

It miss something ? Do the PO template files will be only generated when the "sphinx" branch will be merged to "master" ?

Right now repo-metadata (and then get_paths inside l10n-scripty) says that the trunk_kf5 branch is "master":
{"trunk": "none", "stable": "none", "stable_kf5": "none", "trunk_kf5": "master"}

https://invent.kde.org/sysadmin/repo-metadata/-/blob/da84d34e684cfac038e0de4dee55730095b74487/projects-invent/documentation/digikam-doc/i18n.json

So of course it doesn't know anything about the sphinx branch.

Thanks Luigi for the feedback. So i need to complete first the conversion and merge back sphinx branch to master, before to check the translations workflow.

Best

Gilles

Hi all,

I 'm back here to said that the migration from DocBook to Sphinx/ReStructuredText is mostly complete:

https://docs.digikam.org/en/index.html#

I'm ready to merge back the "sphinx" branch to "master" in the git repository:

https://invent.kde.org/documentation/digikam-doc/-/tree/sphinx

So the docs-digikam-org section from this binary-factory-tooling json file can be patch to master :

https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/staticweb/custom-jobs.json#L148

... and we will see if the internationalized files will be generated properly by i10n-scripty.

My best

Gilles Caulier

Hi Ben,

Merge request created : https://invent.kde.org/sysadmin/binary-factory-tooling/-/merge_requests/327

Best

Gilles

Ben,

The job https://binary-factory.kde.org/view/Websites/job/Website_docs-digikam-org/ is not triggered anymore automatically when a commit is done in git/master.

There is one other setting somewhere for that ?

Note : Starting this job manually give the expected result in web site.

Best

Gilles

That will be due to the first run not being done - i've now done that.

Hi all,

Good news, the strings for internationalization are well exported at this place :

https://websvn.kde.org/trunk/l10n-kf5/templates/messages/digikam-doc/

Best

Gillres Caulier

Hi all,

The i10n workflow sound working fine as i can see.

Before to close this file, it still a question for Julius about the non responsive behaviors of the documentation in HTML format.

As you can see in this screenshot :

Why the right side of the page is mostly an empty space ?

Best

Gilles

That behaviour is governed by the Sphinx theme - you can see the exact same thing happening with Kdenlive.
Krita appears to be using a customised theme hence why they don't have the issue.

Yes, this is just the nature of the readthedocs theme. Not only Kdenlive all documentations using it, look like this.

See eg. https://opentimelineio.readthedocs.io/en/latest/ or https://docs.blender.org/manual/en/latest/

I guess with a bit of CSS it is not that hard to fill the full width or you can choose a different theme.

The ticket can't be closed just yet: the new documentation is stated to be CC BY SA 4.0 but the text is mostly the same as the old one, which is GFDL 1.2. Unless a proper relicensing is documented, where all contributors agree (or have already agreed though FLA and or https://invent.kde.org/sdk/kde-dev-scripts/-/blob/master/relicensecheck.pl), the license must stay GFDL 1.2.

Done with this commit https://invent.kde.org/documentation/digikam-doc/commit/22dfc978e6442ef34a0b5bea8a03fa4f2fc79634

Gilles

Hi all,

One question about po/ directory from the digiKam doc repository:

https://invent.kde.org/documentation/digikam-doc/-/tree/master/po

The contents is obsolete and are relevant of old DocBook implementation. With Sphinx, locale/directory is used instead for the internationalization.

The Q is : can i delete po/ dir without side effects ? Why kdenlive doc repository still have the po/ directory ?

Best

Gilles Caulier

Please leave them there for now. See T16036.

Hi Luigi,
T16036 ??? are you sure ?..

Sorry, wrong copy-and-paste: T16059

Hum, i don't have the right to access on T16059...

I have now unprotected that Sysadmin ticket.

I am starting to get incredibly and extremely frustrated with the numerous and excessive complaints from users of Sphinx concerning the po/ folders.
Would be nice if members of the wider community could please stop complaining about it.

Ok, i seen the problem.

If scriptly team prefer to see a po dir everywhere to be homogeneous, ir's logic.

If Sphinx needs a sub directory named locale and not po, why not.

So the solution can be to use po dir and create a symlink to po named locale...

Gilles

Hi all,

In Sphinx configuration we have this line :

https://invent.kde.org/documentation/digikam-doc/-/blob/master/conf.py#L120

Why "locale" is used instead "po". If this can be customized why "po" is not used as well ?

Note : this settings is based on kdenlive doc.

Gilles

Hi Ben,

A small feedback here: The Chinese zh_CN translations is not available at this url :

https://docs.digikam.org/zh_CN/index.html

... even if all is build and exported without problem in binary factory. Any ideas ?

Best

Gilles

PS: Other exported languages works as expected...

Reviewing the logs I see the following typo:

LANGUAGES=en es fr nl uk_UA ca de it zh_CH pt_PT

Probably needs a pipeline fix.

Right, i will prepare a Merge Request