migrate server
Closed, ResolvedPublic

Description

prep

  • deploy neon-sftp-bridge onto new server (needs user system services)
  • deploy git semaphore onto new server (may require adjustments to code?; runs as jenkins user I think; probably also user systemd service?)

access validation

  • neon-sftp-bridge can access racnoss
  • jenkins user can access semaphore (test by git clone via the git in the sempahore)
  • jenkins user can access persistent blue systems nodes (openqa, do-builder e.g.)

jenkins prep

  • adjust jenkins job updater to not generate mgmt_docker! (should be pushed to master I suppose?; simple check if user != jenkins maybe?)
  • adjust paths inside jenkins to point to correct directories if necessary? (grep the xml maybe)
  • shutdown jenkins and prevent it from doing aaaaaaaaaaanything. make doubly sure it doesn't switch out of this state!
  • best start a script to rsync; this script should be run at least twice. once for the initial sync and then again when we are ready to roll over
  • rsync entire home into jenkins data dir
  • make sure to also rsync jobs.baks from the mount on drax
  • copy mgmt_tooling to a temporary other name (so job numbers don't get messed up) and try to deploy tooling
  • make sure all tests pass/ed
  • reload jenkins configs

roll over

  • put jenkins into shutdown on drax as well
  • stop sftp-bridge and semaphore service on drax
  • run final rsync
  • fully restart jenkins on charlotte
  • switch cname of domains metadata.neon.kde.org, build.neon.kde.org, download.kde.internal.neon.kde.org, images.neon.kde.org, images.neon.kde.org.uk
  • ALSO update our internal DNS?
  • pull jenkins out of shutdown mode on charlotte once DNS has propagated a bit

cleanup

  • clean out drax' mount and jenkins home (workspaces, jobs)
  • snapshot drax
  • delete drax
  • keep mount and snapshot for a while longer
sitter created this task.Nov 19 2018, 3:40 PM
sitter triaged this task as Low priority.
bshah moved this task from Ready To Do to Doing on the Neon board.Nov 20 2018, 9:07 AM
bshah added a subscriber: bshah.

Work done:

  • neon-sftp-bridge deployed on the charlotte (user neonsftpproxy)
  • Changes for this pushed in bshah/charlotte branch
bshah updated the task description. (Show Details)Nov 20 2018, 9:10 AM

Work done:

  • neon-sftp-bridge deployed on the charlotte (user neonsftpproxy)
  • Changes for this pushed in bshah/charlotte branch

please note that this
https://cgit.kde.org/sysadmin/neon-sftp-bridge.git/commit/?h=bshah/charlotte&id=60a47961b87b27a87d4803999a002c96044f7457
also requires changes in the main.go where the interface addresses get foolishly hardcoded

Work done:

  • git-semaphore service made available to the neon user on charlotte
  • Changes for it pushed to git-semaphore repo
    • Make git-semaphore.service systemd user unit
    • Get rid of hardcoded paths
    • Get rid of old upstart crap.
bshah added a comment.Nov 21 2018, 6:35 AM

Access validation done:

  • jenkins user can access persistent blue systems nodes (openqa, do-builder e.g.)
  • jenkins user can access semaphore (test by git clone via the git in the sempahore)
  • neon-sftp-bridge can access racnoss (http service not yet enabled at this point)
bshah added a comment.Nov 21 2018, 7:00 AM

For mgmt_docker job, my idea is to remove it entirely. Opened pull request for it.

bshah added a comment.Nov 21 2018, 7:07 AM

Work done:

  • adjust paths inside jenkins to point to correct directories if necessary? (grep the xml maybe)

There are no instances of hardcoded path in config.xml

Work done:

  • adjust paths inside jenkins to point to correct directories if necessary? (grep the xml maybe) [ only location where this was needed was git path to point to correct semaphore location ]
  • adjust jenkins job updater to not generate mgmt_docker! (should be pushed to master I suppose?; simple check if user != jenkins maybe?) [ We realized that it won't be needed as neon user on charlotte doesn't have access to docker daemon anyway ]
  • shutdown jenkins and prevent it from doing aaaaaaaaaaanything. make doubly sure it doesn't switch out of this state!
  • Initial sync of data from the drax to charlotte
  • copy mgmt_tooling to a temporary other name (so job numbers don't get messed up) and try to deploy tooling

Work in progress:

  • Fix the problems with the mgmt_tooling

Isn't this done?

bshah closed this task as Resolved.Fri, Mar 29, 12:03 PM
bshah claimed this task.