archive binary factory release artifacts for longer
Closed, ResolvedPublic

Description

Today I was trying to debug a crash in filelight 20.1202.522.0 for windows but couldn't because we don't have the artifacts anymore https://binary-factory.kde.org/view/Windows%2064-bit/job/Filelight_Release_win64/

It'd be really great if we could have a persistent archive of releases that cleans up only after 3-6 months, or maybe even only when space is getting tight?
Being able to only re-trace dumps that happend on a version of the last 7 days is really quite limiting.

sitter created this task.Apr 29 2021, 12:48 PM
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptApr 29 2021, 12:48 PM
bcooksley added a subscriber: bcooksley.

The Binary Factory currently consumes approximately 170GB of space on Charlotte with the 7 days of history it retains.

Extrapolating this out we would end up with 4.3TB of data if we were to preserve those, along with corresponding overhead - Jenkins would take forever to start up. This seems like a bit of a waste given it is all essentially just nightly builds.

What i'd suggest instead is that rather than preserving all of our nightly builds (which in quite a few cases are just rebuilds of exactly the same thing sadly), is that we ensure that we upload all the necessary artifacts for a project somewhere (files.kde.org / download.kde.org) when we make a proper release (on the Store for instance).

Thoughts?

Yes I think when a maintainer is preparing a Windows store release, they should also preserve the other artifacts.
That way we would also publish the nsis installer and the plain archives.

Two possible options here for storing these artifacts:

  1. download.kde.org, which given the Store (and other places) should be receiving proper releases is the most logical place (whether this be a beta, release candidate or final release)
  2. files.kde.org

Any preference?

download.kde.org also seems the most logical to me. @vonreth ?

Hannah, could you please respond to the above?

1 sounds great.
But we can't enforce it because every maintainer manually uploads the stuff to the store.

bcooksley changed the visibility from "Custom Policy" to "Public (No Login Required)".May 18 2021, 11:26 AM
bcooksley changed the edit policy from "Custom Policy" to "All Users".

Thanks for confirming Hannah.

I've now sent an email to everyone with access to the Store asking them to please upload the files in question - hopefully this should solve everything (going forward at least)?

If you could confirm that would be appreciated @sitter

Thanks! This should do nicely.

Somewhat related for future reference: we could opt to automate this process somewhat - like a releaseme for binaries: download artifacts from jenkins + upload to store api + upload to our server + template sysadmin ticket.

(this may indeed also be interesting from a metadata POV as we could convert appstream metadata to submission metadata thereby reducing duplication AND increasing l10n)

rempt added a subscriber: rempt.May 18 2021, 12:42 PM

I'm not sure I get it? I'm making the windows store uploads from our binary releases from the binary factory -- and those are around on download.kde.org, too, of course. Do I need to do something more?

I'm not sure I get it? I'm making the windows store uploads from our binary releases from the binary factory -- and those are around on download.kde.org, too, of course. Do I need to do something more?

If you already upload the same files to the windows store and to download.kde.org, then you're already okay :)

bcooksley closed this task as Resolved.May 19 2021, 9:27 AM
bcooksley claimed this task.

@sitter That is definitely something we could look into in the future yes.

One of the things i'd like to see us look into in general would be some kind of automation around release workflows to help improve consistency there. Hopefully this will become more realistic once we have figured out how translations work with our repositories...