weekly merge from debian
Closed, ResolvedPublic

Description

once a week (early monday) merge master into unstable

do it all at once so we can fail the entire merge if it fails, rather than have some bits merged and others not

job1
  find all frameworks ➡ clone one repo ➡ get latest tag on master ➡ split version ➡ ls-remote all frameworks, verify they have version tagged ➡ abort or continue ➡ trigger job2

⬇

job2 [parameter tag; parameter list]
  merge all parameter tag for all parameter list of frameworks ➡ if merge successful push to Neon/pending-merge
    if all successful ➡ trigger job3
    else ➡ abort

⬇

job3 (can be triggered at any time by anyone!)
  for all frameworks ➡ merge Neon/pending-merge if it exists ➡ push all merged ➡ delete Neon/pending-merge for all pushed

This 3 part split allows fairly efficient and consistent handling. Automerge does not start unless ALL repos have a consistent tag mainline version (or supposedly excluding an override). The pending-merge temporary branch allows us to manually refine and prepare merges if automerge should fail. pending-merge would be available for everything that merged, regardless of whether something else did not merge (low human overhead). The final push being detached means a human can do partial merges out of order but still have them integrated all at once (e.g. adopting a package migration in kio.git and plasma-framework.git which requires substantial changes to the two).

Todo

  1. make sure coverage is near 100%
  2. refactor Repo classes to share code
  3. template jobs
  4. rubocop cleanup
  5. move tagdetective to module NCI in code (namespace)
  6. make tagdetective use Data class for writing
  7. remove future observer from merger

Related Objects

sitter created this task.Jun 9 2016, 10:35 AM
sitter triaged this task as Low priority.Jun 10 2016, 9:57 AM
sitter moved this task from Backlog to Ready To Do on the Neon board.
sitter claimed this task.Jun 21 2016, 8:39 AM
sitter removed sitter as the assignee of this task.Jul 28 2016, 10:25 AM

We have mergers in tooling and it could be done for debian master into neon/unstable

we must only merge tags (debian tags uploads and this stops us uploading broken stuff)

could be done daily only acted on once a week

done as 1 job or as 1 job per package?

sitter moved this task from Ready To Do to Doing on the Neon board.Sep 26 2016, 1:23 PM
sitter claimed this task.
sitter updated the task description. (Show Details)Sep 26 2016, 3:38 PM

Using the existing merger code is actually no good since the requirements are vastly different. We could base the actual git merge on the existing code (for which the code should be very well suited. The larger amount of logic is in the surrounding rigging that decides if we merge and what we merge though.

code in tooling repo, job1 built and working job2 is a clusterfuck for code reasons

a) threading seem to screw over the git gem although it's not clear why
b) rugged was built without ssh and the ssh credentials code is fairly shitty shit

job3 can't be implemented becuase the scripty shares the ssh problems of job2

Additionally I just noticed that we have no reverse-merge feature (i.e. before pushing job3 should attempt to automerge unstable such that changes that were made in the meantime get automatically picked up, currently we'd have to manually merge them into pending!)

alternatively we could add pending as a merge target?

finalizer now also working.

Code is a major mess. Templetifciation still missing as well.

sitter updated the task description. (Show Details)Nov 2 2016, 2:17 PM
sitter updated the task description. (Show Details)Nov 4 2016, 2:31 PM
sitter updated the task description. (Show Details)Nov 8 2016, 8:56 AM
sitter updated the task description. (Show Details)Nov 8 2016, 9:25 AM
sitter moved this task from Doing to Review on the Neon board.Nov 8 2016, 9:27 AM

Fully rolled out

src in nci/debian-merge/*

uses a pipeline job which unfortunately right now can't do IRC notifications nor downstream linking (i.e. block frameworks). assuming we'll want to go from multijob to pipelines for builds we'll need to sort the limitations though.

sitter closed this task as Resolved.Nov 22 2016, 11:10 AM