<rdonkin>
(Before my injury) I was an Apache Software Foundation member
<yorickpeterse>
Ah. Well, right now we're mostly fighting about whether or not to go with a CA based system for signing Gems or a more Web of Trust oriented system
<rdonkin>
If there's anyone has any questions about Apache's experience with webs of trust etc, if I can't answer I can probably push you in the direction of someone who can
<yorickpeterse>
(it's quite the read)
havenwood has quit [Remote host closed the connection]
<yorickpeterse>
Well yeah, what does Apache use?
<rdonkin>
Now?
<rdonkin>
For what purpose?
<yorickpeterse>
Yes, though what they used in the past is interesting to know as well.
<rdonkin>
(is a better question)
<yorickpeterse>
For signing stuff btw
<yorickpeterse>
wait, I should probably start from the beginning
<rdonkin>
What are you trying to achieve by signing releases...?
<rdonkin>
(Dave has given me a brief outlined)
<yorickpeterse>
Ah
<yorickpeterse>
Well, the core idea is that we provide the means for Ruby packages to be signed to ensure that they can not be tampered with without people knowing about it
<yorickpeterse>
Such as how you sign an Email using PGP for example
<rdonkin>
The NSA and the PRC will be able to tamper with any gem without your knowledge
<yorickpeterse>
heh
<rdonkin>
Probably also GCHQ
<rdonkin>
Defense in depth is the most important lesson we've learned over the years at Apache
<rdonkin>
1024 RSA is now vulnerable to anyone with a decent bot net
<rdonkin>
or rather than hash is
<rdonkin>
Signing doesn't require PGP
<rdonkin>
PGP is just a very useful standard
<rdonkin>
Which people do you care about knowing that a package is tampered with...?
<yorickpeterse>
End users (= other developers). To give an example:
<rdonkin>
Users != developers
<rdonkin>
Users may have minimal knowledge of cryptography
<yorickpeterse>
The hack of last week would allow the hacker to execute arbitrary code. As a result he/she could've used that to inject malicious code into a popular project (e.g. Rails). Because there's literally zero validation this could've wreaked havoc on many developers' systems in a matter of hours
<yorickpeterse>
e.g. by something as simple as `rm -rf $HOME`
<raggi>
rdonkin: developers have minimal knowledge of crypto too
<rdonkin>
you can force developers to care about crypto
<raggi>
care != understand
<rdonkin>
Apache has failed to make users care about crypto
<raggi>
understanding crypto takes years
<rdonkin>
WOT is easy
<rdonkin>
I've explained it to LUGs in 15 minutes
<raggi>
that's not crypto, so i guess we're talking about different things
<raggi>
ouch
<rdonkin>
Implementing crypto is very tough
<yorickpeterse>
The least you can teach people is to not blindly click "ACCEPT!" whenever they see the text "This seems fishy, are you sure you want to continue?"
<rdonkin>
yes
<rdonkin>
:-)
<raggi>
yorickpeterse: do you have an example of some place where a group of people have learned not to blindly click accept on things, outside of /maybe/ some lawyers and really obsesive individuals?
<rdonkin>
A package signed by a release manager allows other project developers to verify that an artifact hosted on a mirror has been signed by the release manager
<rdonkin>
A package signed by every project developer is very hard to fake
<raggi>
rdonkin: or at least signed by that key, which makes it very likely to be the release manager, yes
<rdonkin>
But not impossible
Antiarc has quit [Remote host closed the connection]
<yorickpeterse>
raggi: yes: rolling release Linux distributions
<rdonkin>
Over at Apache we work hard to ensure that we have a critical mass of F2F meetings with key signing parties
Antiarc has joined #rubygems-trust
jstr has joined #rubygems-trust
<raggi>
rdonkin: we will deal with more credential theft than crypto breaches, more than likely, unless the impl. is totally hosed
<rdonkin>
raggi: yes
<rdonkin>
Defense in depth :-)
jstr has quit [Client Quit]
<rdonkin>
I'm out of the loop (injured for the last couple of years)
<rdonkin>
But first line of defense at Apache is storing releases on a fully versioned file system
<raggi>
yorickpeterse: i think i missed something
<rdonkin>
Plus very, very clueful SysOps
<raggi>
rdonkin: right, and /backups/
<raggi>
rdonkin: which was our biggest failing
<rdonkin>
raggi: defense in depth
<rdonkin>
Before Apache used smart file systems and clever backups, we used to collect hashes
<raz>
defense in depth translates to end-to-end signing :P
<yorickpeterse>
raggi: the OS I use, Arch Linux (rolling release) very quickly teaches you to be careful because if not it will hit you in the balls with a sledgehamer
<raggi>
rdonkin: yep, and that would have saved us ~80 man hours too
<yorickpeterse>
* hammer
<raggi>
yorickpeterse: still not sure i follow exactly
<raz>
raggi: a backup does not help you when you don't know when you were compromised
<rdonkin>
raggi: only 80 hours? IIRC the first time we had to check every release by hand took more than that
<raggi>
rdonkin: it took between 2 and 4 people the best part of 2 workign days to validate all the .gem files using external sources
<rdonkin>
raggi: that's quick
<raggi>
rdonkin: we were lucky
<rdonkin>
be glad :-)
<raggi>
rdonkin: oh i am
<kseifried>
that's assuming this was the first guy to break into rubygems.org =)
<raggi>
kseifried: sure, but equally, there could be a ghost looking over your shoulder right now.
<raz>
perhaps he just accidentally published his teaching notes... :P
<rdonkin>
a versioned file system allows you to 1. rollback 2. diff 3. prioritize manual verification 4. publish vulnerabilities fast
<kseifried>
raggi, : nope. I have a ghost repelling stone on my desk. Also works for tigers.
<raggi>
kseifried: i'll make a note to steal that one day then
<kseifried>
rdonkin, : assuming the attacker doesn't pwn it ;P
<tarcieri>
it's pretty scary how easily someone could write a gem worm right now
<yorickpeterse>
rdonkin: out of curiosity, what would your opinion be on using X509 for signing and such? Things seem to be leaning towards using that at the moment
<raz>
tarcieri: i'm sadly waiting for it to happen..
<rdonkin>
Solaris is a very secure host
* kseifried
coughs
<raggi>
rdonkin: correct, and s3 is versioned, but we couldn't necessarily trust it
* raz
chokes
<rdonkin>
breaking out from a zone and then compromising the file system is tough
Antiarc has quit [Remote host closed the connection]
<tarcieri>
with .gem/credentials sitting around in the clear, you could install a gem and it could run extconf.rb or whatever, search your home directory for gems, add a payload, and try to push them all to gemcutter
Antiarc has joined #rubygems-trust
<rdonkin>
IIRC The last time we had a major issue at Apache, they used a day zero exploit to gain root
<kseifried>
I think at this point rubygems.org is pretty clearly a high value target
<raggi>
tarcieri: yep
<tarcieri>
then anyone who installs any of those gems would get the same payload added to their gems
<raz>
tarcieri: that's sort of inherent - gems run code. ;)
<rdonkin>
Not sure any attacker would waste a day zero file system exploit for solaris on anything other than a bank
<raz>
although it would be a hilarious DDoS; a gem that rolls and pushes a new gem upon installation :D
<tarcieri>
raz: well, I mentioned that possibility on whatever issue was added for adding a passphrase to ~/.gem/credentials
<raz>
tarcieri: yup.. it all comes down to burdening the user with it.
Antiarc has quit [Remote host closed the connection]
<raggi>
rdonkin: opportunistic attacks are generally constrained by context
Antiarc has joined #rubygems-trust
<rdonkin>
raggi: true
<raggi>
rdonkin: i.e. if you dont' have access to that filesystem, you won't be using the opportunity *there*
<rdonkin>
Defense in depth :-)
<raz>
erm, can we just stick to the premise "host exists = host will be compromised", kthx :)
<rdonkin>
Compromising a file system implementation is non-trivial
<rdonkin>
But let's not argue
<rdonkin>
Back to the point
<kseifried>
who cares. even one hour of traffic people pulling compromised gems is enough to be a rather large issue
<rdonkin>
GPG is great
<kseifried>
GPG is also well understood for package signing (e.g. RPM)
<rdonkin>
But WOT is a lot of effort
<raz>
kseifried: cough, ssh-style, cough :P
<raggi>
see, apache products are mostly vendored
<raggi>
so there, the loose cert distribution is handled by vendor process
<kseifried>
I don't get the point of WOT in this case, users are not savvy enough generally enough to actually make proper use of it
<rdonkin>
Maven
<raggi>
and very occasionaly, very, very occasionally, end users pay attention if they use source packages (but mostly not)
<kseifried>
shit we can't even make WOT work for email GPG keys
<raz>
kseifried: it's not about unsavvy users, it's about making it *possible* for savvy users to validate.
<raggi>
kseifried: confirm
<yorickpeterse>
kseifried: doesn't matter, at minimum you only need two people
<kseifried>
yeah. we don't need WOT for that
<yorickpeterse>
The author and somebody that trusts him
<yorickpeterse>
given that most of the code comes from the popular devs that's mostly a non issue
<rdonkin>
That's all GPG does
<kseifried>
and we're back to .. ah nevermind
<kseifried>
raggi: get writing =)
<kseifried>
rdonkin, : huh?
<raz>
kseifried: unsavvy users will just disable anything that breaks their omakase-zem.
<tarcieri>
kseifried: strongly agree re dumb users
<raz>
zen*
<raggi>
kseifried: yeah, i need to send you those emails first, i had to clear my inbox this morning :-p
<rdonkin>
Unless you have a trust link, GPG is worse than useless
<kseifried>
mine keeps getting bigger. I need a trash compactor
<kseifried>
rdonkin, : same is tryue of CA/X.509 model
<rdonkin>
Most lilnux distros get round this point by distributing trusted keys
<kseifried>
rdonkin, : I'm gheussing vendors will not have a major problems hipping the signing keys/etc
<yorickpeterse>
also ugh, I should cook myself some food at this hour
<raggi>
kseifried: ja mijn liniaal
<kseifried>
so for rubygems for example maybe we decide policy is X of Y (e.g. 4 of 5) signing keys are needed, or whatever. or maybe 1. or 4.5. whatever.
<rdonkin>
Back to my point of origin: what are you trying to achieve..?
<kseifried>
rdonkin, : that's why we're doing a requirements doc first
<kseifried>
rdonkin: nobody really seems sure at this point
<rdonkin>
But requirements need to informed by intention
<kseifried>
yeah
<kseifried>
hence we're figuring it out
<rdonkin>
Protecting files on the server is do-able
<raz>
requirement #1: i want to be able to verify my gem comes from where it claims it's coming from (from = a private key)
<kseifried>
define "From"
<kseifried>
original dev? rubygems.org?
<raz>
didn't i just do that?
<rdonkin>
:-)
<kseifried>
no
<rdonkin>
No
<kseifried>
you didn't
<raz>
ah right i didn't
<kseifried>
see. this shit isn't easy
<raz>
i mean the author of course :)
<kseifried>
raz: ok problem. 50,000 gems
<rdonkin>
The author or the release manager
<kseifried>
I'm guessing you could get 1% signing compliance
<raz>
rdonkin: the author(ized person to release that gem) :)
<kseifried>
and half of them will lose the keys before the next release
<kseifried>
define who that is please.
<raz>
kseifried: that's okay, i will refrain from using those gems
<kseifried>
"I am the king of gem-foo"
<rdonkin>
Some defense in depth could be gained by automated signed
<kseifried>
and we validate how?
qmx|away is now known as qmx
qmx has quit [Excess Flood]
<raz>
kseifried: the user validates, not "we" :)
<kseifried>
rdonkin: yeah we're focussing more on rubygems/etc site signing
<raz>
the inclined user that is
<kseifried>
raz: and what happens when all your needed gems aren't signed?
<raz>
kseifried: signing is enforced in my model
<rdonkin>
How?
<kseifried>
raz: hahahahaha
<raz>
by not accepting gem-pushes that are not signed
<rdonkin>
By whom...?
<kseifried>
ok you just lost 99% of your gems
<raz>
signing costs the author two commands...
<kseifried>
raz, : key generation
<kseifried>
secure key storage/management
<raz>
rdonkin: by the author.. who else? :)
<kseifried>
you really, really haven't thought this through
qmx has joined #rubygems-trust
<rdonkin>
enforcing signing turns out to be tricky
<kseifried>
raz: how do you enforce key security for the author BTW?
<kseifried>
tricky?
<kseifried>
landing on the moon is tricky. this is way worse.
<raz>
kseifried: actually i really have, and you are just trying to solve impossible problems.
<rdonkin>
corner cases :-/
<kseifried>
raz: uhmm.. no. these are commnon problems
<raz>
kseifried: you keep forgetting where we are coming from: nothing at all
<kseifried>
raz: which honestly is better than a broken ass system
<kseifried>
trust in a broken system is worse then knowing the system isn't trust worthy
<raz>
kseifried: as said, i'm eagerly looking forward to your proposal
<raz>
kseifried: especially how you are going to resolve what is not solvable
<kseifried>
raz: yeah, well since we have to answer all these Q's and much more it's gonna take more than a few hours
<kseifried>
raz: uhm no. this stuff is quite solvable
<rdonkin>
defense in depth
* kseifried
hands rdonkin some free beer
<rdonkin>
don't go for one perfect system
<dstufft>
Btw, here's a fun problem: Roll back attacks :3
<rdonkin>
build layers
<raz>
yup, what i described would be the lowest layer
<raz>
in fact, it will pretty much inevitably be, unless kseifried reinvents logic :)
<kseifried>
and it intrduces more problems than it solves, hence my concerns
<dstufft>
By which I mean, *hand waves* you have end to end signing and your users are perfectly validating that packages are signed. Now someone comes and takes control over rubygems.org, and rolls back some core packages to insecure versions
<kseifried>
dstufft: solvable to a large degree thankfully
<kseifried>
assuming the software client is semi intelligent
<rdonkin>
raz: too tough in plain GPG
<rdonkin>
but doable with lower level crypto
<dstufft>
kseifried: Interested to hear how you'd do it when the repo is telling a client that version 2.3 (INSECURE) is the latest when 2.4 (SECURE) is actually it. (Not being snarky, genuinely interested)
<rdonkin>
authors should generate code signing keys that are just for that purpose and clearly labelled
<rdonkin>
FWIW I also carry cards with me
<raz>
postmodern: what i want to know is *NOT* "dhh met this guy at a conference some years ago". what i want to know is "does DHH trust the gem signed with his key right now" :)
<kseifried>
raz: btw you still haven't told us how you deal with non compliant gem authors
<raz>
kseifried: what do you mean by non-compliant?
<kseifried>
thwy donb't sign things
<rdonkin>
keifried: it should be possible to generate a private key as part of the release process
<raz>
kseifried: well, they don't get to push to rubygems.org
<rdonkin>
keep the private key local and push the public one
<kseifried>
raz: great. so you break 99% of gems
drbrain has joined #rubygems-trust
<kseifried>
raz: your cure to this problem sucks
<kseifried>
raz: we need solutions that actually work
<raz>
kseifried: existing gems will still exist. when the author wants to push an update he'll be asked to generate a key. do you really think this causes most authors to say "ah, nahh, i'll stop this shit and switch to python"?
<kseifried>
yah
<kseifried>
actually I do
<raz>
lol
<kseifried>
I can't get people to reliably sign their email when they send me CVE requests
<raz>
ok.. then we just have to disagree there :)
qmx is now known as qmx|away
<kseifried>
I regularily misign meial on purpose
<kseifried>
guess how many people notice and complain
<kseifried>
0
<rdonkin>
kseifried: once you're talking about using low level crypto, automating key creation and signing is possible as part of the release process
havenwood has joined #rubygems-trust
<rdonkin>
the essential problem is distribution of the public key
<rdonkin>
raz: how would you solve that...?
<kseifried>
rdonkin, : what we'll probably end up with is rubygems.org/repo signing of gems, and optional dev signing/third party signing
<kseifried>
rdonkin: so we fic the "is this gem what rubygems.org should have" and optionally add the ability to verify actual gems/etc
havenwood has quit [Remote host closed the connection]
<kseifried>
rdonkin, : wee little magic elves!
<raz>
rdonkin: see my proposal, i don't think it's solvable. i'd just display the URL to them (embedded in the gem) and trust the user to do his due diligence. obviously a tampered gem can display *any* url - but.. oh well, read the proposal :)
<rdonkin>
kseifried: automated author signing adds a layer of security, at the cost of some custom development, given a process that copied and securely stores the public keys
<rdonkin>
raz: back to my question about trusting DNS
<raz>
rdonkin: yes, if the user is MITMed he is fucked
<kseifried>
rdonkin, : I'm talking optional, not mandatory author signing which is a big difference =)
<raz>
rdonkin: but frankly, it the user he is MITMed then he is fucked in pretty much any but the very most exotic scenarios :)
<raz>
if*
<raz>
my key concern is preventing an attacker from compromising the entire repository at once
<raz>
where repository == any mirror
<rdonkin>
raz: defense in depth
<raz>
yup, it doesn't get any deeper than end-to-end (author->user).
<raz>
convenience may be layered on top :)
<rdonkin>
raz: automated signing and update on the author's machine would add some defense, at the cost of developing the software required
<rdonkin>
FWIW
<raz>
rdonkin: if you mean the signing tools then that actually happens to be the one variant that already *is* implemented ;)
<rdonkin>
:-)
<rdonkin>
The more defense the better
<rdonkin>
But convenience is king
<raz>
it's those pie-in-the-sky CA machineries who have yet to realize what amount (and complexity) of code they're looking at when they start implementing
<rdonkin>
CA repositories are non-trivial
<raz>
rdonkin: i'm fine with anything that lets me disable any convenience and verify by hand
<rdonkin>
IIRC Apache has a couple of partial CA implementations hanging around
<rdonkin>
GPG requires WOT
<rdonkin>
Generate the public and private key as part of the release process, output the private key to the command line and commit the public one to the repository
<rdonkin>
Any author who cares will keep the key
<kseifried>
gpg does not rwquire WOT
<kseifried>
GPG can also be used in a CA model
<rdonkin>
Yes and no
<rdonkin>
Yes, it can
<rdonkin>
But WOT is baked in deep
<kseifried>
and not needed in all cases
<rdonkin>
Into the verification model
<kseifried>
nice option to have but I doubt we'll use it
<rdonkin>
easier to use the low level crypto directly
<kseifried>
since we plan to have key signing parties how exactly? :P
<raz>
i'll probably be in NY in september
<rdonkin>
I'm talking about automated key signing during releases
<raz>
perhaps you could schedule one by then? :P
<rdonkin>
I'm all in favour of optional GPG as well
<kseifried>
raz: sure. send me $2000 so I can travel there
<raz>
kseifried: i'm from germany, i pay about that much for my own travel ;)
<rdonkin>
But automated signing by authors would be possible
<rdonkin>
(Just correcting myself)
<raz>
i think rdonkin only talked about making the gem-signing process mostly transparent to the gem-author
<raz>
which i violently agree with (it's a non-issue)
<rdonkin>
Yes, and it's possible
<raz>
we're not talking about grandmas on their ipad here, it's gem-developers ffs
<rdonkin>
but transparency is tough with GPG
<rdonkin>
Keeping keys safe is tough
<raz>
yes, keys get lost left and right
<kseifried>
raz: so you'll educate al 50,000 gem projects on security/proper key use/storage? thanks
<kseifried>
problem solved
<rdonkin>
Just saying. I'm not implementing, so anyone who wants to give transparent GPG a go is very welcome
<raz>
gem publishers who keep changing their keys will just notice how their userbase magically shrinks
<raz>
kseifried: they already do that with ssh and various other things
<raz>
kseifried: it's not that hard a concept
<rdonkin>
I'm almost out of typing time. Any questions I might be able to help with...?
<kseifried>
hahahaha
<raz>
kseifried: and it's inherent to any model. as said, it's one of the unsolvable dilemmas that you seem to claim you know a solution for.
<raz>
the author will have a key and he will have to keep it secure
<kseifried>
raz: actually yes. we make these things optional so we don't create more problems than they solve.
<raz>
if you can remove *that* requirement then i'll listen very closely :)
<rdonkin>
which requirement...?
<raz>
and optional private key? oO
havenwood has joined #rubygems-trust
<raz>
rdonkin: the requirement of the gem author needing a private key to sign his gems
<rdonkin>
you can generate a private key then throw it away
<kseifried>
read my above comments from back when
<rdonkin>
that turns out to be very safe
<rdonkin>
just not very useful
<raz>
rdonkin: ;)
<rdonkin>
for some purposes
<rdonkin>
but okay for others
<raz>
kseifried: i'll just wait for your proposal where this is going to be elaborately explained :)
<rdonkin>
I'm going to hop off now. Hope my presence hasn't been completely useless :-)
<rdonkin>
have fun :-)
<raz>
no absolutely not, nice meeting you! :)
<yorickpeterse>
rdonkin: thank you for your time!
<rdonkin>
the Apache devops hang around on freenode
<rdonkin>
on a public channel
rdonkin has quit [Quit: be seeing you]
teancom has quit [Remote host closed the connection]
Antiarc has quit [Remote host closed the connection]
Antiarc has joined #rubygems-trust
geal has joined #rubygems-trust
drbrain_ has joined #rubygems-trust
drbrain has quit [Ping timeout: 272 seconds]
geal has quit [Ping timeout: 276 seconds]
geal has joined #rubygems-trust
workmad3 has joined #rubygems-trust
billdingo-afk is now known as billdingo
workmad3 has quit [Ping timeout: 264 seconds]
kseifried has quit [Read error: Operation timed out]
billdingo is now known as billdingo-afk
kseifried has joined #rubygems-trust
teancom has joined #rubygems-trust
drbrain_ has quit [Remote host closed the connection]
drbrain has joined #rubygems-trust
geal has quit [Ping timeout: 255 seconds]
havenwood has quit [Remote host closed the connection]