Move Capacity websites and kde.org to PHP7
Closed, ResolvedPublic

Description

I'm running since two months kde.org locally with PHP 7.3. For the moment I didn't saw any regression, but this will need more testing before switching.

So first step, I will be to write a script that is doing a diff between my local version with php 7.3 and the kde.org version with php 5.5.9.

ognarb created this task.Jul 30 2019, 11:03 AM
ognarb triaged this task as High priority.
ognarb added a project: Sysadmin.
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptJul 30 2019, 11:03 AM
bcooksley added a subscriber: bcooksley.

The underlying operating system running www.kde.org (along with around another 90 or so similar static/capacity sites) is on Ubuntu 14.04 i'm afraid, so making PHP 7 available isn't an option at the moment.

Upgrading this system (which is our last remaining Ubuntu 14.04 system) is on the todo list, but current Sysadmin resource focus is on the Gitlab migration.

ognarb lowered the priority of this task from High to Low.Jul 30 2019, 11:34 AM

Ok no problem, this can wait ;)

tomal added a subscriber: tomal.Jul 30 2019, 6:45 PM
toma added a subscriber: toma.Jul 30 2019, 7:35 PM
tomal removed a subscriber: tomal.Jul 30 2019, 7:36 PM
toma added a comment.Jul 31 2019, 5:24 PM

@bcooksley Would it be an idea if I setup/compile a fastcgi based php7 on Olios so we can move site-per-site from PHP5 to PHP7. Maybe @ognarb is able to help convert the website code if needed while I change the PHP version for each site.

If all sites are PHP7 we can easily upgrade the server or move the sites to another server and we can use distro based PHP7 again.

Just an idea, if you have other plans where i can help, let me know.

@toma if you can do the php7 sysadmin setup, I will do the necessary work to convert the websites to php7. :)

Given the age of the Ubuntu 14.04 system, i've asked the Board if we can get an additional IP to deploy a fresh, clean container (using Ubuntu 18.04) so we can start migrating sites there.
Much like with Edulis, we'll be able to run sites with either PHP 5.6 or PHP 7.2 (or another version of PHP 7 if needed) on an as needed basis.

I've now completed the deployment of the successor system to Olios, known as Nicoda.

Apache has been deployed, with a combined PHP 5.6/7.2 setup exactly the same as Edulis, and MySQL has been deployed to accept databases (matching the setup on Edulis as well).
We should now be able to start migrating sites on a one-by-one basis to Nicoda.

As Cano (the physical host of Olios) has a failing disk (which we'd like to ask for replacement of, but due to the age of the remaining disk it's possible it won't be able to come back up or survive the RAID rebuild) we should probably prioritize getting through as many sites as possible, while modernizing things where possible.

As KDE.org has the most complexity with it's setup, i'd like to propose that we work through most of the other smaller sites first, before tackling it.

Thoughts?

As KDE.org has the most complexity with it's setup, i'd like to propose that we work through most of the other smaller sites first, before tackling it.

Completely agree. I will test some smaller sites like https://kdesrc-build.kde.org/ on my local server, and report back then they work perfectly with php7.

toma added a comment.Aug 14 2019, 4:12 PM

Excellent. I am happy to help with sysadmin side of things. I'll be back from vacation in 2 weeks, if you want it faster due to cano, I can't help.

ognarb renamed this task from Move kde.org to PHP7 to Move Capacity websites and kde.org to PHP7.Aug 17 2019, 8:46 PM

As an update on this, I have essentially completed the vast majority of the preparation work to transfer all the sites on Olios to Nicoda now, which would allow us to start bringing sites on to PHP7.2.

Completed to date:

  • Deployment of site files (either directly from Git repositories, or through the appropriate Jekyll/Sphinx/Hugo/etc generator) to the server (accomplished via the Binary Factory)
  • Apache setup on the server, including copying across all the virtualhost configurations and adapting them for the new server, and completing the necessary PHP setup for Capacity

Items left to complete:

  • Issuing all the necessary certificates via LetsEncrypt on Nicoda so it can serve traffic

Once that is done we can begin migrating sites to Nicoda, and then in turn to PHP7 as needed.

Note that www.kde.org is one of the more complicated ones to move, so we'll probably move virtually all the other ones first.

Only complicated sites left to sort out:

  • collect.kde.org
  • planet.kde.org
  • networkcheck.kde.org
  • docs.kde.org
  • books.kde.org

We can resolve those once all the other ones are sorted out though.

Of the sites we are looking to shift, i'm also expecting us to hit issues with ev.kde.org.

I've now completed the SSL setup for all of the sites Nicoda will need to host and have transferred the vast bulk of the websites from Olios to Nicoda now.
The following sites are left to sort out:

  • autoconfig.kde.org
  • collect.kde.org.conf
  • docs.kde.org.conf
  • evolve.kde.org.conf
  • marble.kde.org.conf
  • planet.kde.org.conf
  • quality.calligra.org.conf
  • www.kde.org.conf
  • books.kde.org.conf
  • ev.kde.org.conf
  • jointhegame.kde.org.conf
  • networkcheck.kde.org.conf
  • plasma-mobile.org.conf
  • www-backstage.kde.org.conf

From the above list:

  • evolve.kde.org and plasma-mobile.org should be very straight forward to migrate once their Binary Factory builds are fixed
  • autoconfig.kde.org will need a custom pipeline to handle the integration of a DVB file which we host for Kaffeine
  • docs.kde.org will need all the necessary tools to be installed and tested, after which Library can start uploading to Nicoda instead
  • marble.kde.org is dependent on a database, the continued usage of this needs to be assessed
  • planet.kde.org is generated by Rawdog, we need to assess whether this should continue to be by Cronjob, or done via the Binary Factory
  • quality.calligra.org is to be decomissioned
  • books.kde.org needs further investigation
  • ev.kde.org needs the LDAP Tunnel with Identity setup, and may also need the LDAP PEAR packages installed on the system
  • jointhegame.kde.org has already turned into a redirect hosted by Edulis, this just needs the original site to be archived on Olios
  • networkcheck.kde.org contains just a single tiny file, would be a bit of a waste to set a repository up just for this...
  • www-backstage.kde.org should probably be archived, pending further discussion on how to replace www.kde.org

That will leave just www.kde.org to sort once the above is handled..

Some websites, for example docs.plasma-mobile.org and plasma-mobile.org used to be deployed using GitLab CI on invent. Will this option stay available or is the plan to migrate all websites to the Binary Factory?

Thanks :)

From the above list:

  • evolve.kde.org and plasma-mobile.org should be very straight forward to migrate once their Binary Factory builds are fixed
  • autoconfig.kde.org will need a custom pipeline to handle the integration of a DVB file which we host for Kaffeine
  • docs.kde.org will need all the necessary tools to be installed and tested, after which Library can start uploading to Nicoda instead

I already have a rewritten version using php7 (D17936), but I wasn't able to make sure xapian-omega worked (it didn't work in my Arch Linux setup). I need to try again in a vm with Ubuntu 18.04.

  • marble.kde.org is dependent on a database, the continued usage of this needs to be assessed

This website could probably be converted to Jekyll. I will need to contact the marble devs and add point in my todo list ;).

  • planet.kde.org is generated by Rawdog, we need to assess whether this should continue to be by Cronjob, or done via the Binary Factory
  • quality.calligra.org is to be decomissioned
  • books.kde.org needs further investigation

books.kde.org redirect to the community wiki for me and I didn't find any source code in cgit.

  • ev.kde.org needs the LDAP Tunnel with Identity setup, and may also need the LDAP PEAR packages installed on the system

LDAP PEAR is not maintained anymore so we will need to switch to php building ldap functionality. I already took a look at the documentation and it shouldn't be too difficult. I will submit a patch in the coming days. Since this website need to be moved away from capacity, and we can't use Jekyll, another solution would be to move it to Symfony or another web framework but this will takes more time

  • jointhegame.kde.org has already turned into a redirect hosted by Edulis, this just needs the original site to be archived on Olios
  • networkcheck.kde.org contains just a single tiny file, would be a bit of a waste to set a repository up just for this...
  • www-backstage.kde.org should probably be archived, pending further discussion on how to replace www.kde.org

    That will leave just www.kde.org to sort once the above is handled..

plasma-mobile.org seems to be the only domain that is not covered by *.kde.org (or at least one of the few). It probably needs to be added to the new certificate as well.

@jbbgameich All websites will need to be handled by the Binary Factory yes.
With regards to plasma-mobile.org, it already has a separate certificate issued for this which is ready to go (which will be visible if you visit docs.plasma-mobile.org)

The site books.kde.org was hosting three files which were generated from the kf5book repository about 4 years ago. I've now placed copies of those three files on share.kde.org, and updated the single wiki page that was linking to them to point to the new home of the files on share.kde.org.

With regards to quality.calligra.org, i've now shut that one down as discussed earlier.

I've also moved autoconfig.kde.org to Nicoda now, and it appears I already archived jointhegame.kde.org a while back, so there is nothing left to action on that item.
That leaves:

  1. plasma-mobile.org and evolve.kde.org (site generation to fix)
  2. docs.kde.org (needs the tooling installed and tested)
  3. marble.kde.org (site database usage to be evaluated, and if seen to not be necessary, discontinued)
  4. planet.kde.org (we need to determine whether we want to run Rawdog directly there)
  5. ev.kde.org (needs porting from Net_LDAP2)
  6. networkcheck.kde.org (needs a decision made given it has just a single static file whose contents are literally just 'OK')
  7. www-backstage.kde.org (needs archival on Olios)

I contacted the marble dev about the marble website.

I've shifted Planet and networkcheck over this evening, which now leaves:

  1. docs.kde.org
  2. marble.kde.org
  3. ev.kde.org
  4. www.kde.org

We still need to archive www-backstage.kde.org as well.

I'll start looking into what is needed for docs.kde.org and the porting of ev.kde.org next.

I've now staged a migration of docs.kde.org and have set that up on Nicoda. As far as I can tell all is working as intended, so i've asked Luigi and the others involved in Documentation to please check everything. If it all checks out, we'll switch it over.

Ok I got a response from the marble devs in irc/matrix. The website is still needed and should like it is.

Docs has now moved over.

All sites are now transferred to Nicoda.

toma added a comment.Sep 28 2019, 8:40 PM

@ognarb, I think we can start converting sites.

On nicoda i see the following PHP5.6 sites currently:

  • conference2004.kde.org.conf
  • conference2005.kde.org.conf
  • conference2006.kde.org.conf
  • conference2007.kde.org.conf
  • conference2008.kde.org.conf
  • conference2009.kde.org.conf
  • czechia.kde.org.conf
  • docs.kde.org.conf
  • edu.kde.org.conf
  • events.kde.org.conf
  • ev.kde.org.conf
  • extragear.kde.org.conf
  • freebsd.kde.org.conf
  • games.kde.org.conf
  • il.kde.org.conf
  • jp.kde.org.conf
  • kdemail.net.conf
  • kdesrc-build.kde.org.conf
  • kmplayer.kde.org.conf
  • kmymoney.org.conf
  • konqueror.org.conf
  • konsole.kde.org.conf
  • konversation.kde.org.conf
  • kpdf.kde.org.conf
  • kphotoalbum.org.conf
  • krusader.org.conf
  • kst-plot.kde.org.conf
  • lakademy.kde.org.conf
  • marble.kde.org.conf
  • multimedia.kde.org.conf
  • okular.kde.org.conf
  • pe.kde.org.conf
  • ro.kde.org.conf
  • umbrello.kde.org.conf
  • utils.kde.org.conf
  • vvave.kde.org.conf
  • wiki.kde.org.conf

Let me know if you want one or more switched to 7.2.

ognarb added a comment.EditedSep 28 2019, 9:05 PM

konsole.kde.org, wiki.kde.org and vvave.k.or are static website and should not use php.

http://kdemail.kde.org/ is disabled

kmplayer.k.o, kst-plot.k.o, ro.kde.org, and czechia.k.o should probably be killed

For the others websites, I for the moment only had the time to test the kdesrc-build and the kde ev websites.

I think most of the websites won't need any change to be compatible with php7 (we just need to test it), the only problematic websites are events.kde.org, edu.kde.org, marble.kde.org and maybe the conference200x.kde.org websites.

duffus added a subscriber: duffus.Sep 28 2019, 9:25 PM

the old akademy websites are just capacity

toma added a comment.Sep 28 2019, 9:27 PM

konsole.kde.org, wiki.kde.org and vvave.k.or are static website and should not use php.

Great, PHP handler removed.

http://kdemail.kde.org/ is disabled

This is about www.kdemail.net

Let me know how you want to proceed.

Is there anything further we need to do to help this continue Carl?

Sorry for the delay, University started again and I didn't have time available.

I created D25087, it would be good if we could merge this today and switch the PHP version for all capacity based website to PHP 7.

This should work, but I tested with only some capacity based websites. But I have time this weekend (3 days weekend in Germany) to fix any possible error I find.

I've now switched all the Capacity sites to PHP 7.2.

That completes this task I believe?

ognarb closed this task as Resolved.Sat, Nov 2, 8:06 PM

Yes :)