Create Task
Maniphest T4532

jenkins master -> 16.04
Closed, ResolvedPublic
Actions

Description

master needs rebuilding with 16.04

prep

  • request new server (4 cores - down from 8)
  • wire neon-sftp-bridge into a cookbook so the user is set up through chef and it does an initial deployment (may warrant separate task)
  • change neon-sftp-bridge to use systemd for service management
  • make sure we have a cookbook that sets journald to persistent mode so we have persistent logs
  • request new block storage. the new server would be on our DO team so it needs a new storage we have to rsync into
  • new cookbooks for scrapper-ci role: pangea-ruby (replaces rvm, also drop rvm attributes), neon-sftp-bridge

cook

  • create server (name: do-neon-jenkins)
  • create volume for server (name: volume-neon-jenkins)
  • cook new server same as current drax
  • copy ssh cert of jenkins
  • make sure to have an rsync command that preserves ownership or make sure to run as relevant user
  • rsync /var/lib/jenkins (take care of not syncing rvm or .*rc in particular .bashrc)
  • restart jenkins and make sure it is disabled in every possible form and fashion! best disconnect all slaves, reduce master to one slot and set it to only accept explicitly associated builds and run some placeholder?)
  • make sure mounts are set up correctly
  • rsync block storage (i.e. mounts)
  • rsync /var/www/* (metadata and so forth)
  • deploy tooling and rebuild docker images via jenkins job

move

  • remove (NOT purge!) jenkins on old server
  • remove (NOT purge!) apache on old server
  • make sure block storage works on new server
  • have drax.kde.org point to new ip
  • have neon.pangea.pub point to new ip
  • re-enable jenkins on new server to actually resume CI
  • have old server shut down and turned into a digital ocean image in case we need it again to recover something we forgot (can be deleted after ~3 months)

Related Objects
Search...

sitter created this task.Nov 14 2016, 8:22 AM
sitter updated the task description. (Show Details)Nov 14 2016, 8:31 AM
sitter added a comment.Nov 28 2016, 1:14 PM

Given drax was excessively upgraded to a $80/month droplet I'd argue that rebuilding is the way to go so we can go back to a $40/month machine. It also will highlight stuff we haven't chefd so that's an added bonus.

bshah added a subscriber: bshah.Dec 7 2016, 10:45 AM

Some more stuff to look forward to when you upgrade to 16.04

publisher cookbook needs to be changed to use systemd units instead of upstart unit, otherwise this part will fail,

service "#{uid}_aptly" do
  action :restart
end

Also,

gpg2 seems to have problem with bsw_gpg cookbook, see Github issue

though I believe drax is not hosting aptly repository, right?

sitter added a comment.Dec 7 2016, 10:52 AM
In T4532#70260, @bshah wrote:

though I believe drax is not hosting aptly repository, right?

It's my understanding that we currently don't cook aptly at all, so it shouldn't be an issue. For gpg you could use another cookbook I suppose.

sitter triaged this task as Normal priority.Jan 12 2017, 12:26 PM
sitter updated the task description. (Show Details)
sitter moved this task from Discussing to Ready To Do on the Neon board.
sitter updated the task description. (Show Details)Mar 2 2017, 3:30 PM
sitter updated the task description. (Show Details)Mar 7 2017, 6:51 AM

blocked on T5505 currently

sitter updated the task description. (Show Details)Mar 10 2017, 11:28 AM
sitter moved this task from Ready To Do to Blocked on the Neon board.Mar 14 2017, 11:31 AM
bshah closed this task as Resolved.Jul 17 2018, 4:36 AM
bshah claimed this task.
root@do-neon-jenkins:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
...

This is done now.