Moving the manual from mediawiki to sphinx.
Closed, ResolvedPublic

Description

We've been discussing whether to move the manual to sphinx.

This is for two reasons:

  1. The most important one is that the current manual is hard to translate. We are using the wonderful plugin that userbase uses, but while having an excellent interface, it cannot translate URLs, and thus not translate categories. Furthermore, we have trouble getting search to work with the non-latin script translations.
  2. Offline copies.
  3. We are still not sure what to do with the python API docs. Sphinx might be an answer to help with this.

There's one major disadvantage to sphinx and that is that it is much more technical to use than mediawiki. It will need to have the docs stored in a git repository, while a building system builds it once in a while. We have asked sysadmin and were told that the technical bits are feasible. However, community-wise, we will need to write better contributor's guides for this reason as well as provide workflows so clear and easy that a twelve year old can contribute.

Finally, porting the wiki will take time, even if we use pandoc to translate to rst. Our previous porting from mediawiki to mediawiki took a month with 3 people to make sure everything was clean. We do not have time right now. I want to delay this to after at the least the 4.0 release.

Please feel free to write down your thoughts.

woltherav created this task.Aug 3 2017, 4:12 PM
woltherav updated the task description. (Show Details)

Sphinx/rst Links

http://www.sphinx-doc.org/en/stable/rest.html
http://build-me-the-docs-please.readthedocs.io/en/latest/Using_Sphinx/index.html#references
https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst
https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html
There are a ton of cheatsheets/primers on the net. Remember google is your friend.

Editors

Most any editor will work once the user gets the hang of the rst syntax, but most will prefer one with syntax-highlighting and a live-preview to ease typos/errors.
My python-based SourceCoder has a small docker/pane with basic code that uses the python docutils library to convert RST/MD/etc from the editor to HTML on the fly, but my code isn't as polished as these smaller projects which specialize in it.
https://github.com/retext-project/retext ReText is a simple but powerful editor for Markdown and reStructuredText markup languages. python-based.
https://github.com/ShiraNai7/rstpad reStructuredText editor with live preview. python-based conversion.

http://rst.ninjs.org/ Online reStructuredText editor

http://markdownpad.com/ specializes in markdown(.md) only and has free version. If you use offline github-flavored markdown enough to justify dev cost, a licence for the pro version isn't too much $15.

Free Project Doc hosting

https://readthedocs.org/

Docs as an example

Download the manual from https://docs.blender.org/manual/en/dev/ and look in the _sources directory.
Or various other projects on https://readthedocs.org/ most use Sphinx.

Metallicow added a comment.EditedAug 7 2017, 12:54 AM

Well, I've managed to compile a indented(like the wikitree) list of page links with a few options in a python script.
Good News:

  • It can export pages mediawiki syntax to filename.mediawiki files.
  • It can export pages to pdf.

Bad News:

  • pandoc conversion of the .mediawiki files was "half-assed" to say the least and only converted some. So it looks like Sphinx will have to be a semi-manual conversion.
  • The mediawiki exported pdfs have some images with artifacts/ugly. This may be because of wikisize/color profile/etc...?? I dunno. NEEDS Investigated.
  • Links in the pdfs don't work, but are readable. Apparently the mediawiki pdf exporter isn't the greatest.

Here is the GitHub repo I setup for this stuff: https://github.com/Metallicow/Krita-Docs-Utilities

Hi @Metallicow ,

My reply to you on the forum might have come across rude or aggressive , I apologize for that.
I appreciate your effort to help port the wiki. And thank you for the github repo.

Metallicow added a comment.EditedAug 10 2017, 1:57 AM

Hi @Metallicow ,

My reply to you on the forum might have come across rude or aggressive , I apologize for that.
I appreciate your effort to help port the wiki. And thank you for the github repo.

@kamathraghavendra
No harm done. I didn't take it personally.
There are great programmers out there, but they are not good documentarists. There are great documentarists out there, but they are not good programmers.
There are a few that can do both, but lack in the 'rocket science' catagory, which is the current issue... LOL
I myself write docs(mainly for SourceCoder), but do not profess to be a good documentarist. And learning a trade such as printing, takes time, so bear with me. I am a jack of all trades, but I am a qualified carpenter, that is for sure(my roof will never leak LOL). Amongs't other things, I engineer my SourceCoder software(help with other projects), play guitar, and enjoy art(I am not a natural drawer, like my brother tho... Mainly other art forms.)

Anyway....
I have updated my github repo with the first commit worth of conversion work.
Most everything is ready to preview(make html), exept for installing the REQz if you haven't already.
This is the result of a couple nights of work. My TODO.txt file is here, which outlines, some of the stuff I have come across sofar.
The current theme is a mixture of wxpython, embroidermodder2, and some blender docs I have tweaked throughout the years.
One question I have, that needs to be discussed is... Are we trying to imitate the mediawiki 100% or will this be a basic sphinxdoc project?

@woltherav Also, I have no experience with translations outside of python coding, so I'm guessing you have an idea for that. The 2 docs that are in mediawiki translation format(I cleaned one) look like a NIGHTMARE. I hope there wont be littering of translation syntax like that for sphinx(That stuff scares off people).
Lastly, when complete, I would hope the end-result repo stays on Github however(docs uploaded to site, built when a commit happens), since Github has a easy streamlined interface most people willing to contribute too, can live with.
Offsubject sorta: If I fork the Github Krita/mirror repo and work on python api stuff, can the devs here import changes easily enough from a github branch?

@Metallicow

No harm done. I didn't take it personally.

That is great, thanks :)

One question I have, that needs to be discussed is... Are we trying to imitate the mediawiki 100% or will this be a basic sphinxdoc project?

Imitate mediawiki in what way? Are you talking about the theme, the theme, colors icons for mouse buttons etc can be the same from our existing doc (improvement can be done) , this will make it easy to have it on site with consistency. This is my opinion.

If you are asking about the structure of the articles then it needs to be discussed.

Lastly, when complete, I would hope the end-result repo stays on Github however(docs uploaded to site, built when a commit happens), since Github has a easy streamlined interface most people willing to contribute too, can live with.

Offsubject sorta: If I fork the Github Krita/mirror repo and work on python api stuff, can the devs here import changes easily enough from a github branch?

While yes github will be great to get contributors on board easily, but i guess this needs to be discussed, In my opinion once it is out of experiment mode and working we should have it on kde servers because it will be easy for sysadmins to handle it and link it to existing automatic build services.

Offsubject sorta: If I fork the Github Krita/mirror repo and work on python api stuff, can the devs here import changes easily enough from a github branch?

While yes github will be great to get contributors on board easily, but i guess this needs to be discussed, In my opinion once it is out of experiment mode and working we should have it on kde servers because it will be easy for sysadmins to handle it and link it to existing automatic build services.

Thanks to the distributed nature of git, copying a branch from one tree to another is very simple since they have a shared history.

Interacting with the Phabricator review process would still require the use of the arc tool or manual uploading of the diff, winch is still kind of inconvenient for contributors because it's more work than simply pushing a commit and making a pull request on GitHub or a review with gerrit.

Ultimately, any changes that are accepted would have to be merged into the master branch on our main tree on KDE servers, which can be done by anyone with a KDE developer account. They will have to do a fetch with the contributor's github repo as the remote, but otherwise it's the same as any git merge process.

woltherav added a comment.EditedAug 10 2017, 8:57 AM

Uhm, sorry, I have been a bit busy with RL this week, but:

  1. The git repo will be on KDE servers (there will be a mirror on github, like all the other kde projects)
  2. The build system will be on KDE servers
  3. The final build results will be on KDE servers
  4. We are NOT starting porting while we haven't released 4.0 precisely because I do not have the time to oversee anyone doing any porting or even to start making choices.

Dude, I appreciate your enthusiasm, but at this rate we'll end up creating a big fat mess.

Edit: For the translations I was hoping to use sphinx's gettext integration, because that gives PO files and the KDE translation teams are well familiar with those. But like I said, I don't have physical time to do research into sphinx right now, so I haven't been able to look at how that is set up. This is why I wanted to delay after 4.0, because 4.0 will need docs written for it as well.

Okay, so we're slowly working on porting the manual.

Sysadmin wants us to stabilize on porting first before we get a git and build setup going.

The main repository is Scotts, and I have been porting to sphinx on here.

The whole reference and user manual are ported, meaning that the majority of important pages have been labeled and can be linked to. I would like some help with the general and tutorials categories.

What about using something like pandoc for converting to PDF?

Sphinx already has a system in place for epub and scott figured out how to get it to work for latex export (which is basically how pandoc makes pdfs too).

I got my build ready and I am starting to work on the tutorials section now.

I've done the general section too, so better update your fork :P

I will be busy with school for the next few days, so I proly won't be working on the manual until the sprint :)

I had did about 60% back when I started, then stopped at a point. I'll have to dig that out of the corner and revisit it now as to what else got updated from diff drop. ...So fork scotts repo and continue work and send pull requests, right? ...Thats the plan of action at the moment?

Yes, but only the tutorial section really needs to be ported anymore.

Oh ok. I just sent a pull request of the files that didnt change from my project. If that helps plug in missing pages, then it is there to review and work from. about 20% of what I did changed so I didnt include those files. Thanks for the work so far.

I sent a PR on wolthera's fork for common workflows and clipping mask tutorials, i am porting some more, will send PR one bye one

I think we should consolidate all forks into one so that there won't be a confusion.

woltherav added a subscriber: timotheegiet.EditedMay 20 2018, 7:45 PM

Okay, so we've managed to get the manual 100% ported, and I have added two simple pages to the contributors section. @timotheegiet has ported the French translation of the FAQ and updated it.

Resources and the main page is also properly ported, plus I added some indexing. It isn't complete, but a decent basis.

Optipng and Png quant were run over the images folder.

I also have the audio for animation and the reference tool page done, but I'll keep them on this drive till we get to the git.

Ok, the manual has been moved:

Here's the FAQ in French:

https://docs.krita.org/fr/KritaFAQ.html

Here's the git repository:

https://phabricator.kde.org/source/websites-docs-krita-org/

The next big thing is to figure out a 404 page, because all the mediawiki links are broken...

Reminder: add translation entry in readme, point at https://l10n.kde.org/

woltherav closed this task as Resolved.Feb 26 2019, 4:46 AM

Ok, with https://phabricator.kde.org/D19227 pushed, this is finally done!

Now we only need to write a blogpost :)