Having a dedicated KDE server/service/location for public gpg keys of tarballers & Co.
Closed, ResolvedPublic

Description

With public GPG key servers being under attack and shut-down (from what I understood), for the signing of files made available on the download servers the public keys need some other place to be made available.

Current approaches seen:

Myself not an experienced GPG users (despite having had my first signing party 2 decades ago ;) ). So also no idea how to help with the web of trust here to make the layer of the public key instance itself a more trust-worthy thing. Beyond persistence of same key used for some time.

So looking for a KDE standard how to deal with providing public keys

kossebau created this task.Jul 31 2019, 1:28 PM
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptJul 31 2019, 1:28 PM

For the upcoming kdevelop 5.4.0 release I would use the share.kde.org approach for my key for now.

bcooksley added a subscriber: bcooksley.

Were you thinking of a space where people could have their project/personal ASC files stored, or something different?

lbeltrame added a subscriber: lbeltrame.EditedAug 1 2019, 12:17 PM

Web Key Directory or Web Key Service would fit the bill for this, perhaps?

Info: https://wiki.gnupg.org/WKD

Although that is more for using the keys (e.g. in email) rather than spreading them.

@bcooksley It's about where to put public keys because the current way of distributing public keys through (SKS) key servers is a mess https://dkg.fifthhorseman.net/blog/openpgp-certificate-flooding.html & https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f

There's two sides to the problem:

Distribution of the key itself

To which the answer is what Luca said... If distributions actually commit to using that ;)
Otherwise you could just put your public key on share.kde.org or literally anywhere else that is somewhat reasonably trustworthy. Unless distributions actually implement automation around the WKD there's no point in us having a WKD. Well, besides the mail usecase itself of course, which is an argument for WKD either-way IMHO.

E.g. from the neon side we generally have a human do a manual verification dance which usually entails grabbing the key from a keyserver or file, verifying the fingerpinter/id, and possibly look at our web of trust before gluing the key into the verification tech. The reason we do it manually is because we do sign gpg keys every akademy and so there's a good chance we do in fact have whoever made the release in our web of trust.
We could easily make our process WKD-aware, but then neon is special in terms of how forcefully we employ gpg verification and I doubt we are the problem with making WKD worthwhile ;)

Distribution of signatures aka web of trust

The actual problem behind poisoning a certificate is that you can flood a public key with garbage signatures to the point where the key "breaks" so it can't be used anymore because it's essentially computationally too large or hits artificial limits inside client software. As I understand it there isn't really a solution for this at large. You can use an unaffected keyserver, but the reason they are unaffected is because they don't accept/serve the signatures to begin with, so they are useless from a web of trust POV.

By extension this also isn't a problem if you put your key on share.kde.org or elsewhere. But again, WKD+WKS may be able to allow great automation.

In lieu of a proper GPG community-wide solution I expect the only ways to reliably deal with this is to either run our own (safe) keyserver and/or use WKS+WKD. And let developers pop their keys into those. So, we effectively are a CA for the keys used in and around KDE. Which I guess would work something like

  • dev exports public key
  • sends key in signed mail to sysadmins
  • sysadmins verify mail was signed by key of developer
  • sysadmins export key

It's altogether unsatisfying because it makes things way harder to use for a person doing releases in extragear once in a blue moon :/

@aheinecke might have more knowledgeable input on the subject.

timotheegiet updated the task description. (Show Details)Aug 1 2019, 1:37 PM

Ideally we wouldn't end up running additional software in order to support the validation of GPG signatures which get bundled alongside tarballs we ship (and having another step for releases, especially one gated by Sysadmin is sub-optimal as well).

Looking at this from another angle, what do distributions require in order to perform the validation of signatures, and how does that process look currently?

If Andre (@aheinecke) could comment as well that would be appreciated.

How can KDE guarantee that malicious keys aren't being served?

Without end to end encryption (such as via onion services, which the sysadmins are against) this is just another single point of failure. An attacker that has access to a CA that your browser will accept (hundreds of states and companies, for example).

I guess Kazakh users are out of luck?

But hey security theater is more fun than actual security. KDE ♥ privacy

In the absence of an actionable item, we should close this or transfer it out of the Sysadmin task board.
This looks like it needs further discussion prior to us taking any action on this, should it be necessary.

Any objections to me taking this action @kossebau?

aheinecke added a comment.EditedAug 28 2019, 11:29 AM

Woops. Sorry for the late reply. This ticket got filtered into my general "kde phab" folder.

First: I am all for enabling WKD for KDE keys or even WKS. Ben If you want to look into that, I can help you. I wrote https://wiki.gnupg.org/WKDHosting and the scripts used in that.

Second: As long as you distribute the fingerprint through signatures or somewhere on the homepage Keyservers are fine.
Keyservers still work! It is only that the most recent gnupg versions will not fetch and verify third party signatures by default.

So basically the Web of Trust does not work anymore with Keyservers. That is it. Keyservers still work for key distribution.

P.S. Please give me a ping by mail to aheinecke@gnupg.org when adding @aheinecke somewhere. Otherwise I might miss it.

What a civil discussion from the sysadmins with no mention of how they are swamped and don't have any resources. It's almost like you're actually taken seriously if you have social capital, regardless of how well-founded what is being proposed is.

Weird how that goes, huh?

<snip>
Please stay focussed on the topic to help us here. If you are unhappy with sysadmin work, please open a separate issue to discuss this.

In the absence of an actionable item, we should close this or transfer it out of the Sysadmin task board.
This looks like it needs further discussion prior to us taking any action on this, should it be necessary.

Yes, also a bit unsure how to proceed. As said, myself I am just a mere sourcecode releaser, who sees that some seem to expect source tarballs being signed with gpg keys. And who found a situation where he is puzzled

  • how those "some" expect to get the public key
  • if they rely on any kind of web of trust (and which entry points into it)
  • if all the signing activity actually makes sense and properly protects against assumable attack vectors

Givem that despite my email to kde-distro-packagers@kde.org only people of 2 distributions commented here (for what I can attribute), but no-one told that their distributions actually makes use of the gpg signing keys, I now wonder why we do all this tarball signing dance.

So when it comes to my needs, I conform with the current level of activity now by having put my public key to share.kde.org and adding a note to kdevelop.org/download. But I have to tell, I do not feel like this actually is made us of by anyone, nor could I tell if this really improves protection against assumable attacks.

I mean, even for KDE machines we do not have a list of ssh fingerprints one could check. So I could have been MITM attacked already when uploading my tarballs to the download server, right? And if someone now replaced my tarballs and the signatures, I would not notice as I do not check them once uploaded. And the signature being replaced by one with a key done with my name and email address, who would notice it's not really mine?

So, to do more work here, I would like to see those take over which make use of signatures at the receiver end or those at the publisher end who know what makes sense. I only tried to get things rolling, but have no motivation/needs to keep it rolling.

So from my POV, this task could be closed due to no sufficient feedback. Perhaps I should try to trigger more discussion/activity by uploading future tarballs without any signature. Let's see who will be triggered by that from behind their stone ;)

(For now hoping on @sitter & the neon crew to pick up the ball, given they seem to make use of signatures, after all) ;)

@kossebau You say stay focused and then repeat my criticism of this being security theater, except needlessly wordy. That's... interesting. I'm very focused on what we can do to improve the security situation, I think pointing out others that actively work against it (some sysadmins to say the least) is a part of that focus. I think you ignoring my criticism and acting like you came up with it on your own is part of this social capital problem I'm talking about.

Stop disrupting the ticket. Your uninformed babbling is entirely irrelevant.

@sitter Uninformed babbling? Insulting people you don't even know? Uninformed about what? The security theater you want to set up without a threat model? Good one.

You have some amazing ways of not disrupting the ticket.

There's absolutely no social capital problem at all. KDE treats everyone with respect. I've learned it first hand /s

So when it comes to my needs, I conform with the current level of activity now by having put my public key to share.kde.org and adding a note to kdevelop.org/download. But I have to tell, I do not feel like this actually is made us of by anyone, nor could I tell if this really improves protection against assumable attacks.

What it protects against is the mirror network first and foremost IMO. We do not control our mirrors, they may get hacked or a particularly disgruntled Uni employee may poison our data, etc etc. By signing the release artifacts everyone can verify their authenticity. If people don't do that it's their thing. If distros don't it's a bit of a different thing because they are putting theirs and our users at potential risk.

I mean, even for KDE machines we do not have a list of ssh fingerprints one could check. So I could have been MITM attacked already when uploading my tarballs to the download server, right? And if someone now replaced my tarballs and the signatures, I would not notice as I do not check them once uploaded.

Conceivably possible but only the first time you release. This is more or less only an attack vector with trust on first use on both ends. I'll spare you the long yadayada... it's a fairly difficult attack to pull off.

And the signature being replaced by one with a key done with my name and email address, who would notice it's not really mine?

This is actually why our release guides say you should put your key fingerprint or id on a not-easily-editable website somewhere different than the tarball and ideally sign an announcement mail with it. It lessens the TOFU-ness a bit. It's also why we do a GPG key signing each Akademy, if a packager needs to pick up a new key and sees that the new key is signed by jriddell's or dfaure's key already they can assume it is trustworthy.

So, to do more work here, I would like to see those take over which make use of signatures at the receiver end or those at the publisher end who know what makes sense. I only tried to get things rolling, but have no motivation/needs to keep it rolling.

So from my POV, this task could be closed due to no sufficient feedback. Perhaps I should try to trigger more discussion/activity by uploading future tarballs without any signature. Let's see who will be triggered by that from behind their stone ;)

(For now hoping on @sitter & the neon crew to pick up the ball, given they seem to make use of signatures, after all) ;)

BTW, I am almost certain Debian and possibly Kubuntu use verification too. The packaging repos all have shared ancestry and get merged in all directions, so they at least do have the metadata to carry out the verification, to which extend they use it I couldn't say though.

Anyway...
What I would suggest we do is

a) Check if there's interest in WKD (I actually don't know how/where? ^^)
b) Sysadmins (and Andre) look into the possibility of setting up a WKD if there is interest and its deemed worthwhile (as mentioned it'd make sense for mail alone, but we can spin tarball verification off of it)
c) Talk to packagers at Akademy and find out why they aren't doing GPG verification yet and get them excited for actually implementing it. Possibly fix pain points.

The way I looked at it originally was as a chicken-and-egg problem: without signatures distros had no reason to implement verification, and without verification we had no reason to implement signing. So we made signing a thing, but perhaps it wasn't a chicken-and-egg to begin with... 😒

@lavender I'm very sorry if I insulted you.

Now leave this task alone.

This is a total joke. Oh just put it on a "not easily editable website". And you have the audacity to say I'm uninformed 😂 There is no threat model. This is a joke.

This comment was removed by bcooksley.

Stop clogging the thread with your uninformed security theater nonsense. Come back with a threat model of what you actually want to do, because this is a joke.

So when it comes to my needs, I conform with the current level of activity now by having put my public key to share.kde.org and adding a note to kdevelop.org/download. But I have to tell, I do not feel like this actually is made us of by anyone, nor could I tell if this really improves protection against assumable attacks.

What it protects against is the mirror network first and foremost IMO. We do not control our mirrors, they may get hacked or a particularly disgruntled Uni employee may poison our data, etc etc. By signing the release artifacts everyone can verify their authenticity.

But only if they have a trustable reference, right? Which is the main question here for me how this trust can be always ensured to be above the minimal required level, and which triggered this very ticket.

I mean, even for KDE machines we do not have a list of ssh fingerprints one could check. So I could have been MITM attacked already when uploading my tarballs to the download server, right? And if someone now replaced my tarballs and the signatures, I would not notice as I do not check them once uploaded.

Conceivably possible but only the first time you release. This is more or less only an attack vector with trust on first use on both ends. I'll spare you the long yadayada... it's a fairly difficult attack to pull off.

Some more context which makes me wonder

  • my currently used key's valid time ends in September, so I will use a new one -> how do recipients deal with the change, how do they deal with people having multiple keys in general, or changing them frequently? after all, the longer a key exists, the higher chance there is it has been leaked, no?
  • the old key was never signed, and the new also might not in the foreseeable time, due to missing chances for key signing with me

So "first use" might happen more often. And I am curious if we could have some defined protocol how to quickly & simple communicate a new key which gets to use, to a proper level of trust, where personal contact for key auth or via a web of trust is not an option. You hint later to some ideas, but I would like to know people are making use of t, instead me doing useless cargo dances.

(When it comes to "difficult attack"after all the years my conclusion is, "difficult" is just an attribute for things for those who do not know them yet ;) )

And the signature being replaced by one with a key done with my name and email address, who would notice it's not really mine?

This is actually why our release guides say you should put your key fingerprint or id on a not-easily-editable website somewhere different than the tarball and ideally sign an announcement mail with it.

Which is something I would like to see more formalized/standardized or tool-enforced. "not-easily-editable website" does not tell me a lot. For attackers with their tools any Drupal instance could be "easily-editable", I cannot tell. Signing the email BTW, is missing from https://community.kde.org/ReleasingSoftware , please add that bit there (with correct wording, instead of me writing something bogus perhaps :) ).

It lessens the TOFU-ness a bit. It's also why we do a GPG key signing each Akademy, if a packager needs to pick up a new key and sees that the new key is signed by jriddell's or dfaure's key already they can assume it is trustworthy.

But what to do for people who have a harder time to meet up with other's for key-signing, once a key is new? Both packagers and tarballers? How to ensure the minimum level of trust can be high enough from the start that people do not have to wait for any web-of-trust kicking in, before they think they can rely on the auth information?

I wonder if we should not also look into having some multiple factors deployed here, like explicitly mentioning the tag and the repository state id(s) (think git commit id or svn revision) the tarball is made from, so there could be also services trying to reproduce the tarball (and thus also protecting against evil behaving tarballl creators ;) ). After all some packagers seem to skip the source tarballs completely and directly fetch from the repositories. The latter of course is screwed where tarballs get injections like translations, which IMHO should be fixed as well, due to not being easily reproducable/repeatable in case of needs.

What I would suggest we do is

a) Check if there's interest in WKD (I actually don't know how/where? ^^)
b) Sysadmins (and Andre) look into the possibility of setting up a WKD if there is interest and its deemed worthwhile (as mentioned it'd make sense for mail alone, but we can spin tarball verification off of it)
c) Talk to packagers at Akademy and find out why they aren't doing GPG verification yet and get them excited for actually implementing it. Possibly fix pain points.

The way I looked at it originally was as a chicken-and-egg problem: without signatures distros had no reason to implement verification, and without verification we had no reason to implement signing. So we made signing a thing, but perhaps it wasn't a chicken-and-egg to begin with... 😒

To win people over for this, it might also be good to have some attack scenarious which we would protect against here and why gpg signing by people is the most reasonable reply. I could not tell, and would just do what they possibly call "cargo cult".

This is all getting a bit long :/

So when it comes to my needs, I conform with the current level of activity now by having put my public key to share.kde.org and adding a note to kdevelop.org/download. But I have to tell, I do not feel like this actually is made us of by anyone, nor could I tell if this really improves protection against assumable attacks.

What it protects against is the mirror network first and foremost IMO. We do not control our mirrors, they may get hacked or a particularly disgruntled Uni employee may poison our data, etc etc. By signing the release artifacts everyone can verify their authenticity.

But only if they have a trustable reference, right? Which is the main question here for me how this trust can be always ensured to be above the minimal required level, and which triggered this very ticket.

I am honestly not sure if always-trustable is a thing that can or should be sought after. The tricky thing of course is that humans are involved (on both sides - tarball creation and packaging), so the process needs to be "simple".

But, if 100% trust was the goal the answer would be that every packager needs to be able to span a high certainty web of trust to the release dude. Seeing as that would be nigh impossible to achievable, we opted for a TOFU with optional safety nets. Ultimately a packager (or any user who builds from source for that matter) can decide for themselves that this isn't enough and refuse to package the software. Or reach out on IRC or by email and ask you for additional authentication. For example by sending you an encrypted email that you have to reply to with an encrypted text file you have to push onto a scratch repo to proof that you are indeed the KDE dev you claim to be.

In the end whatever we do it's going to be some degree of TOFU unless the packager has actually met the KDE dev and signed their key. As such a packager could always opt to require more authentication, whether we define a process or not. So, we may well be talking about problems no one has either because distros aren't doing verification yet or what we provide is sufficient to them. I'd like to know which it is because that could inform where we need to spend our time.

I mean, even for KDE machines we do not have a list of ssh fingerprints one could check. So I could have been MITM attacked already when uploading my tarballs to the download server, right? And if someone now replaced my tarballs and the signatures, I would not notice as I do not check them once uploaded.

Conceivably possible but only the first time you release. This is more or less only an attack vector with trust on first use on both ends. I'll spare you the long yadayada... it's a fairly difficult attack to pull off.

Some more context which makes me wonder

  • my currently used key's valid time ends in September, so I will use a new one -> how do recipients deal with the change, how do they deal with people having multiple keys in general, or changing them frequently? after all, the longer a key exists, the higher chance there is it has been leaked, no?
  • the old key was never signed, and the new also might not in the foreseeable time, due to missing chances for key signing with me

    So "first use" might happen more often. And I am curious if we could have some defined protocol how to quickly & simple communicate a new key which gets to use, to a proper level of trust, where personal contact for key auth or via a web of trust is not an option. You hint later to some ideas, but I would like to know people are making use of t, instead me doing useless cargo dances.

The established practice is to create the new key and then sign the new with the old (and possibly the old with the new) thus authenticating the new key as you know have a web that consists of new and old. Additionally it's a good idea to write a quick message and sign it for sending via mail and/or blog post and/or mention it in your next announcement. That makes picking up the new one no longer TOFU.

Some good info https://www.apache.org/dev/key-transition.html

(When it comes to "difficult attack"after all the years my conclusion is, "difficult" is just an attribute for things for those who do not know them yet ;) )

And the signature being replaced by one with a key done with my name and email address, who would notice it's not really mine?

This is actually why our release guides say you should put your key fingerprint or id on a not-easily-editable website somewhere different than the tarball and ideally sign an announcement mail with it.

Which is something I would like to see more formalized/standardized or tool-enforced. "not-easily-editable website" does not tell me a lot. For attackers with their tools any Drupal instance could be "easily-editable", I cannot tell. Signing the email BTW, is missing from https://community.kde.org/ReleasingSoftware , please add that bit there (with correct wording, instead of me writing something bogus perhaps :) ).

You are of course right.
I don't think you appreciate just how huge the difference in understanding cryptography/security is within our community though. We've had people put their fingerprint on a wiki page. That's what the not-easily-editable is meaning to tell you... don't put your fingerprint reference on a wiki page that all the world can edit.
Having a WKD with a clear process of how to get your key in there would even prevent that, which is why I think it is a good idea. Also sysadmins could automate signature verification to the point where we could refuse to release artifacts which do not have a signature. A double-win as far as as I am concerned, but then I am also not sure if possibly getting the key into the WKD would be too big a hurdle (release engineering is already quite demanding for most people as-is).

It lessens the TOFU-ness a bit. It's also why we do a GPG key signing each Akademy, if a packager needs to pick up a new key and sees that the new key is signed by jriddell's or dfaure's key already they can assume it is trustworthy.

But what to do for people who have a harder time to meet up with other's for key-signing, once a key is new? Both packagers and tarballers?

One could do online key verification and signing (e.g. as I described above) with only marginal trust on the signature. Signature trust has levels, I am almost certain I've heard of people using a lower level to sign a key they only verified through a digital process. I am sure you, specifically, could meet up with someone to have a signing done though. Within Europe this shouldn't be an unsolvable problem if you shoot a mail to the community ML.

How to ensure the minimum level of trust can be high enough from the start that people do not have to wait for any web-of-trust kicking in, before they think they can rely on the auth information

I wonder if we should not also look into having some multiple factors deployed here, like explicitly mentioning the tag and the repository state id(s) (think git commit id or svn revision) the tarball is made from, so there could be also services trying to reproduce the tarball (and thus also protecting against evil behaving tarballl creators ;) ). After all some packagers seem to skip the source tarballs completely and directly fetch from the repositories. The latter of course is screwed where tarballs get injections like translations, which IMHO should be fixed as well, due to not being easily reproducable/repeatable in case of needs.

Tags should be v$version and should be signed by the same key as the tarball. If a packager wants quick reliable automated authentication today they can perfectly well do it by doing a three point comparison between the key on webpage/email, the key used to sign the tarball, and the key used to sign the tag (which implies control over both the SSH key for a dev account AND the GPG key in question). If the keys are all the same and the tarball signature validates, then the tarball is authenticated as coming from someone with access to a KDE dev account. If the signatures do not align something is fishy and investigation needs to go underway.

To win people over for this, it might also be good to have some attack scenarious which we would protect against here and why gpg signing by people is the most reasonable reply. I could not tell, and would just do what they possibly call "cargo cult".

Mh, dunno, if you use releaseme.... RTFM? ;) https://cgit.kde.org/releaseme.git/about/GPG.md
Quite possibly should be moved to the wiki though and linked to from the release page.

This comment was removed by nalvarez.
toma added a subscriber: toma.Aug 29 2019, 4:07 PM

(not read everything. we could ask pgp key in identity.kde.org and publish all the keys on a key store/webpage. identity.kde.org is also maintaining the ssh-keys of developers, so if that is hacked, you can already commit stuff, so all is lost by then. On the release tarball bit, publication is mostly done via sysadmin ticket, in that form we do check the the given sha-sum in the sysadmin ticket with the presented tarball, so an attack there also requires phabricator/identity access).

Hi,

I think @sitter represents my Opinions well in this thread, too. Thank you.
Honestly I think this thread has digressed and can be closed.

Let me give you an example how it works in my distribution.

https://dev.gnupg.org/source/gpg4win/browse/master/packages/packages.current

We have a policy that you need two sources for a checksum or a signature from a key that was already included in the last release.

Ideally, and this is how I work with KDE. I grab the checksum from an https protected server _and_ check the signature. If both are valid I can be 99.(lots of 9es) sure that I have the Package that the maintainers intended to release.
This has nothing to do with web of trust, keyservers providing signatures etc. I just note the fingerprint of the key I used to verify the current / last release.


I would say:

  1. Let's open a ticket to provide a WKD / WKS for all @kde.org keys this is a bit of sysadmin work so I would like the sysadmins / Ben to decide if they want to do this.
  2. Let's close this ticket. I don't see how the recent keyserver attacks are specific to KDE release keys and are hurting them. The mitigations in GnuPG should be enough.

Just confirming Harald that the outcome of the BoF on this at Akademy was that we should publish these keys in a Git repository controlled/curated by Sysadmin?

I was there in the BoF but don't really remember if that was what we agreed on or not.

Should have taken notes :/

But yes seems like a solution we may have reached

@sitter @jriddell did you take notes by any chance?

cfeck added a subscriber: cfeck.Sep 16 2019, 10:54 PM

I'll write a mail to release team list later this week as I am still catching up with my inbox. What we came up with is still pending further discussion anyway. The short description is that we want a keyring of "trusted" release keys, which would be curated by sysadmins in a git repo. (Source) releases would have to be signed by a key in the keyring and the keyring would act as validation source for all concerned parties. This would also solve the task at hand in that the keyring can also be used by distros. It would also allow a future move to more tightly controlled signing procedures (i.e. no source releases without signature) which would in turn allow for greater automation possibilities for sysadmins as well, as instead of manual checksum checks an automated signature verification could be employed.

Okay, in that case should this be closed until the final proposal is confirmed (which can be opened as a fresh ticket)?

sitter closed this task as Resolved.Sep 17 2019, 2:14 PM
sitter claimed this task.

Fine by me.

Thanks for driving this Harald - it's appreciated.