Create Task
Maniphest T13263

Can opt-in to make the tests failing make the CI be marked as failed instead of as success?
Open, Needs TriagePublic
Actions

Description

In okular + gitlab CI we would like that if the tests fail the build is marked as failed instead of as succesful.

Is that possible?

aacid created this task.Jun 10 2020, 10:24 PM
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptJun 10 2020, 10:24 PM
bshah moved this task from External: Active to External: Awaiting Response on the Sysadmin board.Jun 11 2020, 8:05 AM
bshah added a subscriber: bshah.

Currently scripts used by Gitlab CI is shared with Jenkins setup,

https://invent.kde.org/sysadmin/ci-tooling/-/blob/master/helpers/run-tests.py is the script which runs the tests,

You can modify that to not exit with 0 status if tests are failing, but lot of project would not want this option. I am not sure what is correct solution.

Please note that this issue is not exclusive to Gitlab CI but rather to jenkins as well.

bshah changed the visibility from "Custom Policy" to "Public (No Login Required)".Jun 11 2020, 8:06 AM
bshah edited projects, added build.kde.org; removed Sysadmin.
bcooksley added a subscriber: bcooksley.Jun 11 2020, 8:17 AM

I already have plans to significantly revamp how CI works with Gitlab so it shouldn't be too much of an issue to delegate control over this as part of the changes.

aacid added a comment.Jun 11 2020, 5:35 PM

Yes i understand the scripts are shared between gitlab and jenkins and that lots of projects are not "very good" in keeping their tests clean, that's why i asked this to be opt-in for those projects that try to actually have passing tests :)