Thanks @subins2000 for adding a report of your experience and also some documentation. I added this information to the task description.

In T13514#237694, @woltherav wrote:

Thanks! Maybe we should make this a subtask of that as well?

EDIT: especially because we're literally having duplicate conversations in both tasks :D

In T13514#237638, @huftis wrote:

In T13514#237645, @pino wrote:

A "l10n" repository with submodules will not really work: git submodules are not like svn externals, and they point to a specific revision and not to a branch. Considering how often files in the repository of each language will change (at least once every day), it would mean constantly updating the submodules in the "l10n" repository... no please.

In T13514#237695, @dkazakov wrote:

In T13514#237630, @rempt wrote:

[...] to include the po files in the source repositories, for the following reasons:

In T13514#237630, @rempt wrote:

This will make things easier for developers, packagers and user’s who like to compile applications themselves (instead of using packages, or just for testing a new feature or a bug fix).
But having a non-central location of the PO files will make things much harder for the translators (for various reasons that I won’t get into now).

Can it be solved with git-submodules? It looks like submodules were invented exactly for this purpose, weren't they?

TBH I'm mostly ignoring this task, but i'm at least going to answer your initial points

In T13514#237630, @rempt wrote:

[...] to include the po files in the source repositories, for the following reasons:

Thanks! Maybe we should make this a subtask of that as well?

The mythical web-based translation task is T11070, plus the subtasks of it (especially T13311). Because this was proposed as a community goal, it initially wasn't tied to the Localization project and later on one forgot to change that.

Erm, maybe it is worth it to actually link to this mythical web-based translation task? Because it is not inside the localization team project.

In T13514#237685, @subins2000 wrote:

@pino The goal of doing the git migration is to solve "SVN is harder to use for new contributors". I meant this problem will be better solved by an online tool than a migration to git.

@pino The goal of doing the git migration is to solve "SVN is harder to use for new contributors". I meant this problem will be better solved by an online tool than a migration to git.

In T13514#237626, @nalvarez wrote:

FYI, I tried to convert Spanish translations to Git, only including /trunk/l10n-kde4/es/messages/ and /trunk/l10n-kf5/es/messages/ (no docs, no docmessages, no /branches/stable/, etc) and the result was 150MB.

@subins2000 this ticket is NOT about a web-based translation tool, there is a separate ticket for that. Please keep the topic of this focused ONLY on the svn -> git migration.

In T13514#237640, @huftis wrote:

But a giant Git repository with all the languages will be. So please consider using submodules, one for each language.

In T13514#237637, @aspotashev wrote:

In T13514#237629, @pino wrote:

Git should facilitate integration with external translation file trees maintained by Linux distributions (e.g. BaseALT)

@aspotashev can you please explain why these distros care about the layout of our translations? When packaging upstream releases they ought to use translations as available in release tarballs, not pick random files from upstream translation trees.

Our [upstream] translation team has limited capacity and we are unable to review new translations submitted by the distro on a timely manner. Therefore they want to use their own versions of translations when they are not upstreamed yet. They do it by injecting their custom translation files by the distro-specific packaging scripts.

Git should make it easier for them to merge upstreamed translations back into their custom tree when both repos will be in Git.

I came into the KDE Community last year as part of localizing KDE apps to Malayalam. As a beginner, it was difficult to start. I had previous experience with GNOME localization, and there, the process was to lock, download PO file from Damned Lies, and upload. As a beginner, that process was a little difficult cause the team was practically dead and had to contact the maintainer to review it. The existing maintainers are no more college students and is busy in their daily jobs, so review was difficult.

Additional comment (by me) from the mailing list:

In T13514#237626, @nalvarez wrote:

FYI, I tried to convert Spanish translations to Git, only including /trunk/l10n-kde4/es/messages/ and /trunk/l10n-kf5/es/messages/ (no docs, no docmessages, no /branches/stable/, etc) and the result was 150MB.

(initially svn2git made an unoptimized 6GB monster; after a 'git gc --aggressive' that ate 9GB of RAM, it reduced the repo to 150MB)

In T13514#237630, @rempt wrote:

As I've said before, and will keep saying, if the way translations are handled in KDE changes, the change should be to include the po files in the source repositories, for the following reasons:

• Developers will always build with translations
• Since we're using gitlab now, that would making a release is as simple as creating a tag, i.e., we would be using gitlab as it is meant to be used
• No chance of accidentally packaging the unstable translations with a release from the stable branch
• nightly builds will include translations

In T13514#237629, @pino wrote:

Git should facilitate integration with external translation file trees maintained by Linux distributions (e.g. BaseALT)

@aspotashev can you please explain why these distros care about the layout of our translations? When packaging upstream releases they ought to use translations as available in release tarballs, not pick random files from upstream translation trees.

As I've said before, and will keep saying, if the way translations are handled in KDE changes, the change should be to include the po files in the source repositories, for the following reasons:

Git should facilitate integration with external translation file trees maintained by Linux distributions (e.g. BaseALT)

Please keep the eventual integration of Weblate (or any other web-based translation system) out of this "migrate to git" roadmap. Adding a web-based translation system requires additional complexity and even workflow changes that would make doing this conversion a even bigger and risky task than what it already is.

FYI, I tried to convert Spanish translations to Git, only including /trunk/l10n-kde4/es/messages/ and /trunk/l10n-kf5/es/messages/ (no docs, no docmessages, no /branches/stable/, etc) and the result was 150MB.

In T13514#237614, @ltoscano wrote:

I strongly believe that people who need easy review needs something like weblate. All the others don't care whether it's direct git pushing or svn pushing.
I also think that the web solution have an higher priority than the move of the translations to git.

In T13514#237616, @ltoscano wrote:

In T13514#237615, @dkazakov wrote:

I strongly believe that people who need easy review needs something like weblate. All the others don't care whether it's direct git pushing or svn pushing.

Well, the main problem (in Krita) that newcomers are frightened away from doing translations because of too complicated process. Any solution that would make it simpler would work.

So we are basically in agreement, but not on having a git-based translation system directly exposed to the newcomers. That would make just things more complicated.

PS
Though SVN was actually the reason why my students didn't want to do the translations. It is too outdated and noone knows how to use it anymore.

That's sad, but I have to say that no one want to use it, because it's documented, and the basic commands maps directly to git commands (clone/checkout, commit+push/commit - even one less).
This does *not* mean I want to keep it, but I feel there is a bit too much bad advertisement.

In T13514#237618, @ltoscano wrote:

In T13514#237617, @aspotashev wrote:

I'm not sure what you mean here, can you please elaborate? The tooling to simplify moving of translations between directories is part of step #4 of the roadmap. Something like super-duper-tool mv kscreenshot/kscreenshot.po spectacle/spectable.po would create a commit in each of the Git translations repositories and git push all of them, possibly retrying this operation in a case of a merge conflict.

I don't want to have a tool to push things in all repositories first without reducing the number of occurrences when this can happen. Also, summit should get that support first.

In T13514#237617, @aspotashev wrote:

In T13514#237610, @ltoscano wrote:

The points can be dune until the moving of translations is not automated. Otherwise having to do the operations on a set of per-languages git repositories will be impossible.

I'm not sure what you mean here, can you please elaborate? The tooling to simplify moving of translations between directories is part of step #4 of the roadmap. Something like super-duper-tool mv kscreenshot/kscreenshot.po spectacle/spectable.po would create a commit in each of the Git translations repositories and git push all of them, possibly retrying this operation in a case of a merge conflict.

In T13514#237610, @ltoscano wrote:

The points can be dune until the moving of translations is not automated. Otherwise having to do the operations on a set of per-languages git repositories will be impossible.

In T13514#237615, @dkazakov wrote:

I strongly believe that people who need easy review needs something like weblate. All the others don't care whether it's direct git pushing or svn pushing.

Well, the main problem (in Krita) that newcomers are frightened away from doing translations because of too complicated process. Any solution that would make it simpler would work.

I strongly believe that people who need easy review needs something like weblate. All the others don't care whether it's direct git pushing or svn pushing.

In T13514#237612, @dkazakov wrote:

Hi, @aspotashev!

I'm not sure about this point, could you elaborate?

Disable GitLab merge requests feature for l10n/[language].git repositories unless we have an explicit approval from particular language team's coordinators.

One of the main advantages of migration to GitLab is that people could propose/review changes via some GUI interface, avoiding sending them via email or doing weird things like that. Sending files via email in 2020 just frightens new contributors.

I'm not sure about this point, could you elaborate?

Also, no parallel migration. When this will happen (after everything else is automated, including summit and the new website) it should be one shot.

The points can be dune until the moving of translations is not automated. Otherwise having to do the operations on a set of per-languages git repositories will be impossible.

As using summit globally was mentioned in T11070#198358 I am setting the parent task.

After a few days of usage, it seems everything is working as expected.

https://websvn.kde.org/?view=revision&revision=1574284

I have been updating the conversion every few days when I remember, but it's not automated. I just updated it again.

So what's missing apart from rechecking the history and adapting makemessages to checkout the various branches in order?

I feel like it would be a good moment to pick up this again, @ltoscano what do you think?

@IlyaBizyaev: true it shouldn't happen, but I already spend a few hours trying to debug that. The problem is that the errors don't appear on my local setup, there is no error in the production log and activating the debug mode on production hide the error. I'm not sure how I should debug that :/

I don’t think this is resolved. IIRC, the site used to use content negotiation to automatically show the content in the user’s preferred language (the ‘Accept-Language’ HTTP header sent by the browser). This doesn’t work anymore, and the user has to manually select a language from the language bar.

Will do, but I think it's not really resolved if it crashes unless that file is translated...

This is now resolved and a language bar was also added.

Currently localization doesn't work at all for this subsection, failing with a server error.
https://kde.org/applications/ru/education/org.kde.gcompris

Maybe, or maybe not. A web interface with reservation will be needed anyway, and if a web translation system won't provide it, we will still need this.

Related to https://phabricator.kde.org/T11070?

I think this is basically done at this point (thanks!)

Given the current state of instability we have due to the gitlab migration + different structure fallout, let's give us a few weeks to try to stabilize things and come back to this?

Ping?

Looks like from the Sysadmin side this is all resolved now?

Are the git conversion rules in kde-ruleset.git?

KDE.org moved to git :D I will work on updating the various release scripts later today :D

I just noticed l10n-support: https://websvn.kde.org/trunk/l10n-support/scripts/

I just noticed l10n-support: https://websvn.kde.org/trunk/l10n-support/scripts/

Last version: https://invent.kde.org/nalvarez/l10n-scripts-conversion

I started uploading the slides in share.

Seems reasonable to move slides there, also makes it easier for people to add them

With regards to kdeslides/ I believe that https://share.kde.org/f/749964 would be an appropriate folder to upload it to.
We can arrange for a redirect to a public version of that without too much issue I think.

This repo was created: https://invent.kde.org/carlschwan/kde-org

To clarify: I identified that we can't just migrate to Git and remove the scripty source code from SVN without changing makemessages. Migration can be done as follows:

In D26076#627394, @aacid wrote:

Just to be sure, this needs https://phabricator.kde.org/D25929 (or rather the repo resulting from it), right?

I have checked and can confirm that Gitlab CE does support Git Partial Clones, subject to the support being enabled (it isn't by default).

Recently a new git features was announced: partial-clone, this allow to clone only the binary files used by the current checkout.

Just to be sure, this needs https://phabricator.kde.org/D25929 (or rather the repo resulting from it), right?

Sure, i agree producing similar formatting to those of other tools is good.

It may be valid, but the "standard version" produced for example by msgfmt is the one where the spaces are at the end of the line, not at the beginning.
It is worth noting that KBabel first and Lokalize later, up to a certain version which I don't remember, did produce the "properly" formatted version.

When you say "incorrectly" you mean "not like i like it" or "not like gettext creates", right?

Git LFS is certainly an option for larger files as Gitlab provides us with support for that.

updated diff

In D10009#611209, @aacid wrote:

Can you share a .po file where this makes a difference?

And also can you provide a better description of what it does as a commit message? Because "Improvements for Gettext entries wordwrapping" is as generic as it gets.

updated diff

Can you share a .po file where this makes a difference?

We could try to remove all the images from the git history and add then again optimized in one commit at the end. It will be less radical than removing the complete history. Another solution would be to play with Git lfs (https://git-lfs.github.com) and have the images/videos/pdfs stored in another server.

Given the folder on disk is 1.3GB, and in the past people were known to perform image size optimisation passes of Subversion, it ending up at 3GB in size isn't a huge surprise.
I'm not sure what options we have for eliminating those image size optimisations from the history.

The current problem, I'm facing is that the repository is still 3GB big :( and all my attempts to reduce it only improve the size from a few megabytes.

Process wise, what is the status of this migration?