Moving kde.org to git
Closed, ResolvedPublic

Description

Kde.org is one of the last website hosted on svn. For the sake of consitency with the other website and remove all the non l10n content of svn, I think we should move it to git.

The biggest problem is the size of the repository. So I propose we split it in two git repositories.

  • /kdeslides (225MiB) kde-org-kdeslides
    • This contains old slides and can be safely moved to a separate repository that is deployed to kde.org/kdeslides to remain compatibility with the urls.
  • / the rest kde-org
    • We will need to rewrite the git history to remove kdeslides from it

The case of /announcements

Moving the announcements out of the main repository would be good to reduce the size of the main repository but if one day we want to have the entire website autogenerated from hugo or another static site generator we would like to also have the index page with the list of the latest announcements autogenerated (like in https://kde.org/announcements/releases/). But I'm not sure if this is an good enough reason :(

The case of /i18n:

I think the translations should be fetched at compile time in binary-factory and not pushed by scripty with a cron job. The i18n folder will be deleted and the git/svn history to. This will require coordination with the Localization team.

The main advantage of this is that the git history won't be filled by l10n commits and the translation process of kde.org will be a bit shorted (we won't need to wait for scripty to push the translations). The main disadvantage is that this will need some work to swith the infra.

The case of /applications

This is already a git repository 🎉 websites/kde-org-applications

The case of /css and /js

Delete from the repository and use version hosted in cdn. We can probably keep in the history, since there aren't many binary files in it.

ognarb created this task.Nov 16 2019, 11:53 PM
ognarb added a project: Websites.
ognarb added projects: Sysadmin, Localization.
ognarb added a subscriber: kde-www.
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptNov 16 2019, 11:53 PM

With regards to kdeslides/ these should be shifted to share.kde.org, which means they do not require a Git repository.

aacid added a subscriber: aacid.Nov 17 2019, 10:46 AM

I disagree with having magic setup unless there's clear instructions provided

magic setup = /applications and /i18n not being part of the website repository

ognarb added a comment.EditedNov 17 2019, 11:49 AM

@aacid Completely agree, it's the reason why I added a lot of information to https://community.kde.org/KDE.org/Local_Setup. Since there was only really outdated information until recently and I needed to figure the installation process alone. The documentation is not perfect but at least the main information are here.

If I move some directories to other git module, these instructions will be updated.

In terms of /applications, that needs to remain separate because the generator relies on being able to add it's files to the contents of websites/kde-org-applications prior to it being uploaded.

While this doesn't sound like too much of an issue, the generator for /applications takes quite a bit of time to run unfortunately (about 30 mintues if memory serves) which would be quite a pain if people were doing quick updates to other parts of the site. Therefore we'll have to keep /applications separate, unless there is another way of doing the generation of that part of the site which is more resource and time efficient.

aacid added a comment.Nov 17 2019, 5:28 PM

@aacid Completely agree, it's the reason why I added a lot of information to https://community.kde.org/KDE.org/Local_Setup.

ok, sounds good

Since there was only really outdated information until recently and I needed to figure the installation process alone. The documentation is not perfect but at least the main information are here.

Well, the old setup was easy, checkout and run, but it's good we have all the info needed in a wiki place now that's getting more complitated

I would like to add one more idea: https://kde.org/images/screenshots/ can be removed in favor of https://cdn.kde.org/screenshots/ that mirrors https://cgit.kde.org/websites/product-screenshots.git/

However some screenshots are not available in product-screenshots.git yet (e.g. simon.png), may need importing.

ognarb added a comment.Dec 9 2019, 2:51 PM

I would like to add one more idea: https://kde.org/images/screenshots/ can be removed in favor of https://cdn.kde.org/screenshots/ that mirrors https://cgit.kde.org/websites/product-screenshots.git/

However some screenshots are not available in product-screenshots.git yet (e.g. simon.png), may need importing.

I copied all the screenshots to the cdn, next steep is to change all the AppStream files and create a redirect

Is there any reason we can't proceed with putting in place the redirect?

In terms of other blockers, do we have anything else we need to sort out before we can convert the repository to Git?

ognarb moved this task from incoming to kde.org redesign on the Websites board.Jan 18 2020, 10:27 PM

Process wise, what is the status of this migration?

ognarb added a comment.EditedFeb 12 2020, 10:54 AM

The current problem, I'm facing is that the repository is still 3GB big :( and all my attempts to reduce it only improve the size from a few megabytes.

Given the folder on disk is 1.3GB, and in the past people were known to perform image size optimisation passes of Subversion, it ending up at 3GB in size isn't a huge surprise.
I'm not sure what options we have for eliminating those image size optimisations from the history.

The other option is to cut the history, much like was done with Calligra and Amarok when they were converted to Git.

Thoughts Nicolas?

We could try to remove all the images from the git history and add then again optimized in one commit at the end. It will be less radical than removing the complete history. Another solution would be to play with Git lfs (https://git-lfs.github.com) and have the images/videos/pdfs stored in another server.

Another problem is the i18n folder, I removed it from the history but I don't doubt it will grow again after the conversion. I guess having binary factory fetch the translations when building the website instead of scripty generating then could be a good solution.

Git LFS is certainly an option for larger files as Gitlab provides us with support for that.

I don't know how easy it is to integrate that with the conversion process though.

Have you had a a chance to look at this @nalvarez?

Recently a new git features was announced: partial-clone, this allow to clone only the binary files used by the current checkout.

https://about.gitlab.com/blog/2020/03/13/partial-clone-for-massive-repositories/
https://git-scm.com/docs/partial-clone

I'm not sure it is already available in gitlab and gitlab-ce, but if available it would be a lot faster to clone the repository containing kde-org decreasing the problem that we have aout the huge repository size. This could be also used for faster cloning in the CI.

I have checked and can confirm that Gitlab CE does support Git Partial Clones, subject to the support being enabled (it isn't by default).

This repo was created: https://invent.kde.org/carlschwan/kde-org

And has for the moment the same content as the svn repo. Missing is the kdeslides folder that I guess should be moved to share.kde.org or another localtion for historical interest and the i18n folder that contained only generated content.

Change that needs to be done:

  • Move repo to the websites namespace
  • Give access to the people who should have access (release team and others who edit the text often)
  • Change the scripty configuration
  • Change the ci configuration
  • Change the release scripts

With regards to kdeslides/ I believe that https://share.kde.org/f/749964 would be an appropriate folder to upload it to.
We can arrange for a redirect to a public version of that without too much issue I think.

Thoughts?

If everyone is okay with that, i'll proceed with moving that repository to the websites/ namespace.

Adding other release maintainers/managers to ensure they're all aware of this and have the opportunity to comment.

duffus added a subscriber: duffus.Apr 21 2020, 11:05 AM

Seems reasonable to move slides there, also makes it easier for people to add them

I started uploading the slides in share.

KDE.org moved to git :D I will work on updating the various release scripts later today :D

Are the git conversion rules in kde-ruleset.git?

Looks like from the Sysadmin side this is all resolved now?

ognarb closed this task as Resolved.Fri, May 22, 10:40 AM
ognarb claimed this task.