Windows CI roadmap
Open, Needs TriagePublic

Description

Hi, @bcooksley!

The Windows CI for Krita is almost ready, could you please help me with the following questions?

  1. What cache/ccache directories Krita-Windows-CI should use? On Linux we use:
KDECI_CC_CACHE: /mnt/caches/krita-appimage/
KDECI_CACHE_PATH: /mnt/artifacts/krita-appimage/
  1. How should I proceed with code-signing on Windows? Can you draft some kind of roadmap for that?
  1. Should we split package types for "release" and "per-commit-ci"? I mean we have a few optional features that we can technically live without in the normal CI builds (and e.g. build them daily instead):
    • Debugging symbols for the Windows packages (about 700MiB package)
    • GMic plugin (on Windows and Linux)
    • code signing (see problems below)
  1. Should we automatically start building CI packages for all the MRs, not depending on the author of the MR? Are there any security concerns about that? I saw that in intel-llvm's repository, the CIs for the MRs should be triggered manually by the trusted developer (though I should say, it is rather inconvenient for the contributors)
  1. Should we code-sign the packages in the MRs? Technically, they can contain random non-verified code, so it doesn't seem to be a good idea...

Related Objects

For the cache and artifact folders, you can use:

  • KDECI_CC_CACHE=C:/Gitlab/Caches/krita-windows/
  • KDECI_CACHE_PATH=C:/Gitlab/Artifacts/krita-windows/

Those folders, much like their Linux equivalents will be persisted across builds (but not across machines).

For code signing, that will be handled on a separate dedicated machine that Gitlab CI jobs will need to send artifacts to for signing (and will receive signed artifacts back). The existing logic you've got for signing will therefore not be usable. The exact mechanism for submitting files for signing is still being finalised and is what is holding up the closure of the Binary Factory at this time.

The ability to communicate with the signer will only be available on protected branches on our mainline repositories - so it will be unavailable to all forks, work branches and merge requests. Code located outside of protected branches on our mainline repositories is considered untrusted / unreviewed, and therefore cannot be signed. This is also why signing can only happen on a dedicated isolated machine (as the builders themselves are "tainted" by virtue of the fact they perform builds of any repository on invent.kde.org)

In terms of the security of builds, all other projects (which make use of the generally provided Windows CI functionality) that support Windows already have CI on their merge requests, work branches and forks - so assuming you can keep the build times reasonable, there is no reason why Krita can't either.

Regarding the package types, it would be nice to keep the resource usage reasonable so:
a) Please set the expiry lifetime of the job artifacts to something (say 48 hours - certainly no more than a week)
b) For debugging symbols, if they are unlikely to be used i'd suggest discarding them as 700MiB is not a small amount of space per job if there are a number of builds every day.

Hi, @bcooksley!

Thank you for your replies!

...so assuming you can keep the build times reasonable, there is no reason why Krita can't either

The problem that the MR can build (unsigned) executables. And, theoretically, we may get a DDoS attack by opening too many MRs by bad people. Though that is purely hypothetical.

a) Please set the expiry lifetime of the job artifacts to something (say 48 hours - certainly no more than a week)

Currently, I've set the expiry time to 1h and checked the checkbox "keep the latest build". So, theoretically, the old packages are removes as soon as the new appears. I'm not sure how that works for MRs though. Is it removed after the MR is merged?

Perhaps we could configure that in a way that the CI is run only on protected branches and MRs only? That would reduce the possibility of the artifacts being stored forever for the private abandoned branches in the forks...

For debugging symbols...

We, I'm not yet sure they are useless :) Theoretically, if we keep only the latest package, it should be somewhat acceptable...

There are two different sets of expiry times we're referring to here i'm afraid:

  1. Being the job timeout (which is what you've referred to) - being the maximum amount of time the job is allowed to run for
  2. Being the artifact expiry time (which is what I was referring to) - being the maximum amount of time Gitlab will keep the artifacts of a given CI job around before it schedules them for deletion.

In terms of limiting the amount of CI worker time being abused, this is best mitigated by ensuring the CI jobs themselves complete as quickly as possible by optimising the build process (through caching where appropriate, etc). This has worked quite well for Linux so far.

For artifact cleanup, setting the expiry time in the .gitlab-ci.yml configuration should be enough for now - we can look into situations where it isn't working as it is supposed to as this rolls out.

Not sure how building executables specifically is going to cause a problem vs. just testing if it compiles and tests pass?

There are two different sets of expiry times we're referring to here i'm afraid

Yes, I have set "artifact expiration time" to 1h actually :) The latest artifact is still kept indefinitely by the project configuration settings

artifacts:
  name: krita-$CI_COMMIT_REF_SLUG
  expire_in: 1 hour

Not sure how building executables specifically is going to cause a problem vs. just testing if it compiles and tests pass?

Well, theoretically, someone could build a malicious MR and distribute the resulting binary from the KDE's infrastructure. Though it looks a bit improbable :)

The maliciously built binary is why we wouldn't sign them - in that case all we have done is provide hosting for the binary, but no actual legitimacy.

Okay, the Windows CI is now fully deployed. There is only one todo item that should be fixed before we move release builds to Gitlab:

  • enable RelWithDebInfo builds on Windows with debugging packages. Right now build type is hardcoded to Release in run-ci-build.py for all Windows hosts