<noddy>
in particular, no noise source can be blocking or depleting
<noddy>
and we do the best anyone knows how to do with whatever entropy is really in there
<avsm>
Yes, but we might be stuck with a self-seeded source due to no other entropy
<avsm>
These aren't quite noise sources
<avsm>
any single noise source must fundamentally block sometimes (i.e. no interrupts are happening)
<noddy>
honestly, i'd just coalesce the two into a single one and say "the environmental noise on xen does not carry a lot of entropy, yet"
<avsm>
hm, no, I'm not happy with that — we're going to have 2 separate methods for some time (>1 year)
<avsm>
a split-channel one that's external, and a self-seeded one that does the best it can
<noddy>
yes and no; if you are for ex. hashing noise into accumulators, you read-reset the rolling hash and never block
<avsm>
until the former goes upstream
<avsm>
noddy: right, but we need to split that logic out into the Random device then
<noddy>
so again - the entropy device is not expected to provide strong random, right?
<avsm>
and then Entropy can never block
<noddy>
what it should give is *noise*, and noise varies in quality depending on the platform
<avsm>
yes, but what supplies random numbers then?
<noddy>
your own state-of-the-art PRNG
<avsm>
Fortuna is embedded in TLS atm isnt it?
<noddy>
in nocrypto
<avsm>
what i'm saying is — where is it for 2.0 :)
<noddy>
the thing is, these are cryptographically distinct things
<noddy>
entropy and rng
<noddy>
in an entropy device, you expect noise, not random numbers
<noddy>
in a random, you expect random numbers
<avsm>
I realise that — but in the absence of a noise source on Xen, we're sort of stuck and have to provide something
<noddy>
i thought that was the whole point of an "entropy" device; to expose noise as opposed to prng
<avsm>
right now, it errors out
<avsm>
so reverting to Random.self_init temporarily (i.e. the current behaviour), which is then seeded upstream by Fortuna seems reasonable to unblock it
<noddy>
ok, what i'm trying to say: given the desired semantics, i think we can coalesce weak/strong back b/c the entropy, as opposed to rng, is always somewhere in between the two
<noddy>
as in, with *entropy*, you have no guarantees on the actual distribution of output
<noddy>
but anyone implementing a prng needs to know you sole guarantee - that it's environmental
<avsm>
Isn't it worth distinguishing between external entropy and self-discovered ones though?
<noddy>
they are both environment-based
<noddy>
you can say "our xen noise is weak" but fundamentally they are the same
<avsm>
no, one's channeled in from outside, the other one is observed from within
<avsm>
and quite possibly clocked or otherwise aligned on something we can't observe
<avsm>
but also out interface isn't quite right then, since we do block until we can get the right amount of entropy
<noddy>
so - why does freebsd have symling /dev/urandom -> /dev/random but linux has two completely different things there?
<avsm>
well, that's not entropy; it's a strong prng
<noddy>
well /dev/random on linux is entropy
<noddy>
and all the prngs involved are crypto-strong
<noddy>
on both
<noddy>
still, linux makes the distinction between noise and the endless stream
<noddy>
but freebsd does not; why?
<avsm>
no, /dev/random mixes on linux too i think
<noddy>
it mixes, that's all
<noddy>
it has running hash of noise
<noddy>
a hash of noise still has the same amount of actual entropy, modulo size, as the noise itself
<avsm>
my key point though, is that we have two things: crypto strong entropy, and really bad and quite determinstic entropy in xen
<avsm>
i'd really like to expose that a unikernel might have a poor source
<noddy>
my point: there is only one kind of entropy
<noddy>
and several kinds of pseudorandom numbers
<noddy>
entropy being environmental noise
<noddy>
or something derived from it
<avsm>
i dont see how you can call observed entropy from interrupt counters in the same class as random noise though?
<noddy>
entropy is also never crypto strong because before mixing, its statistically skewed
<avsm>
yep, that's true
<noddy>
it is
<noddy>
it is random
<noddy>
'cept its skewed
<noddy>
and you need much more for the same unpredictability
<noddy>
but basically noise is noise
<noddy>
and some channels are noisier
<avsm>
yeah, i agree with that; but then we have the wrong interface entirely in Entropy
<avsm>
since it blocks
<noddy>
it doesn't actually
<noddy>
exactly because of this
<noddy>
( it uses urandom, not random, to emulate opportunistic noise on unix )
<avsm>
argh, but then this is no longer entropy
<noddy>
it is: anything derived from noise is at least as noisy as the noise
<avsm>
and so the xen self_init is reasonable
<noddy>
it has *more* properties we disregard
<noddy>
so the use case:
<noddy>
tls has a really effing good mixing thing for deriving strong crypto rng from _some_ noise
<noddy>
and other people might have the same
<noddy>
and now somehow the platform needs to expose the pure noise
<noddy>
of some quality
<avsm>
right, and how do we distinguish that quality?
<noddy>
without imposing other rng properties on it, because some people need just noise
<avsm>
yep, agreed
<noddy>
we say "the best we have"
<noddy>
for example, fortuna gracefully recovers in environment with low trickle of entropy
<avsm>
but the point of exposing it is that if the unikernel needs to have top-class noise, then we can abort at build time
<noddy>
the property is that it take a little longer, but even with interrupts, it converges onto a "good" state
<avsm>
this implies that the xen interrupt source needs to go through a mixer though
<noddy>
you don't ever need "top-class" noise if you know how to build a prng
<noddy>
yup
<noddy>
and our mixer is fortuna
<avsm>
so in other words, we need to merge the prng and entropy sources
<avsm>
they aren't standalone at all
<noddy>
i think the opposite
<noddy>
we need the pure entropy
<noddy>
and the prng
<noddy>
and nocrypto in particular needs just the entropy
<noddy>
just the noise from the environment
<avsm>
we don't have such a noise device at the moment...
<avsm>
we have:
<noddy>
timers
<avsm>
Unix: /dev/random
<noddy>
( urandom )
<avsm>
Xen: interrupt, timers
<noddy>
aha
<avsm>
Xen: split channel
<avsm>
(which proxies /dev/[u]random)
<noddy>
and as long as we agree on the desired semantics of that device, these are ok implementations of alleged "noise"
<noddy>
it's the blurb i'm worried about really: what is intended there?
<avsm>
with the existing Entropy interface, Xen *must* have fortuna to provide noise
<noddy>
and i'm saying we need it to intend to provide noise measurements, better or worse
<noddy>
you can provide noise to whoever
<avsm>
it has no polling interface to peek "give me 5 bytes if available"
<noddy>
you have to have it to convert noise to random numbers
<noddy>
yes
<noddy>
and we have a system that intentionally does not rely on underlying platform's ability to generate crypto-strong rngs
<avsm>
so in other words, we have to merge the Xen noise with the prng
<noddy>
and intentionally just needs noise
<noddy>
i think these are two complete separate issues here
<noddy>
some people needs random numbers
<noddy>
some people mix their own but need the environment
<noddy>
... so i was quite happy with the entropy device, but then we need it to remain an *entropy* device and not become a prng device
<noddy>
( given that a well-seeded prng such as urandom *is* of sufficient entropy to play that role when available )
<avsm>
From a code perspective, what do you want the Xen entropy device to expose?
<avsm>
Sample interrupts, feed that into a fortuna, and then reexpose that as Entropy, right?
<noddy>
*some* amount of bytes, without blocking, that are somewhat unpredictable
<noddy>
nono
<noddy>
entropy does not read fortuna
<noddy>
just the measurements
<noddy>
just *some* bytes read from interrupts and timers
<avsm>
but the interface is broken then!
<noddy>
why?
<noddy>
why do you think that?
<avsm>
entropy len must return the full len
<avsm>
it has to support a short read
<noddy>
that's why i asked about the semantics
<noddy>
we can agree it needs to return only as much as it has too
<noddy>
as long as the contract is noise, not prng
<avsm>
in that case even something as simple as useconds from gettimeofday would work in the xen timer
<noddy>
that is one of possible sources, indeed
<noddy>
and one of the suggested sources in the fortuna documents too
<noddy>
( as long as there are other ofc )
<noddy>
usecs from interrupts
<noddy>
usecs from latencies
<noddy>
*any* usecs we can get our hands on, really :)
<noddy>
if any of those events contains mere two unpredictable bits, that's good
<noddy>
and the prng recovers
<avsm>
hm, the interrupt one can only be used once per interrupt
<noddy>
still good
<avsm>
so i guess on wakeup, we set a ref
<avsm>
that can be Noned by the polling noise
<noddy>
for ex, yeah
<noddy>
so the interface can also be "*up to* N bytes of noise, as availble *now*"
<avsm>
i'll fix the Unix interface to return a shorter cstruct, first
<avsm>
i'm tempted to leave the Xen one doing self_init for now
<avsm>
to unblock the end-to-end build (with a printf noting its weak)
<noddy>
self_init is the old random thingie with time?
<avsm>
y
<noddy>
:(
<avsm>
why?
<noddy>
we basically need a couple of self_inits per sec, continuously
<avsm>
yeah, that's fine too
<noddy>
and i'd rather read the bytes directly
<noddy>
but the basic idea, still: entropy can make no statistical guarantees, the only property it has is that *some* bits are truly unpredictable even to a computationally-unbounded adversary
<noddy>
that's what i mean by "entropy"
<avsm>
can do that later one (read bytes directly). without an end to end build it's very hard to develop
<avsm>
is there a nocrypto/asn/x509 cut in opam btw?
<avsm>
that would make it easier to build the stack itself
<noddy>
there is the overlay currently
* avsm
needs to test talex5's patch, and conduit for vbmithr too...
<noddy>
we plan to make a cut, i think, today
<noddy>
look: it not really urgent
<noddy>
i just wanted to chime in before we desync on what it means
<avsm>
yeah, it's important
<noddy>
well we have urandom for now
<avsm>
not on xen...
<noddy>
is the overlay ok for you now?
<avsm>
overlay?
<avsm>
self_init, you mean?
<noddy>
we have an opam overlay
<noddy>
opam overlay
<noddy>
in mirleft/
<noddy>
for quick builds
<noddy>
also opam files in all projects for direct pins with git-opam
<avsm>
yep overlay fine for now
<noddy>
cool
<noddy>
ok, we agree on definition? "entropy" is something not entirely predictable even for computationally unbounded adversary. but its unpredictable space can be very small too.
<avsm>
i think this works on unix then:
<avsm>
let entropy { fd = fd } len =
<avsm>
let r = Cstruct.create len in
<avsm>
`Ok r
<avsm>
Lwt_cstruct.(complete (read fd) r) >|= fun () ->
<avsm>
so this never returns short, but the interface supports short reads
<noddy>
i never noticed the `complete` :D
djs551 has joined #mirage
<avsm>
djs551 just added it
<noddy>
sweet
<avsm>
the slightly subtle thing in Lwt_cstruct is that it leaves the rest of the cstruct nutouched
<avsm>
so the old entropy would have had a lot of zeros potentially in the cstruct ;-)
<noddy>
*maybe* use Cstruct.sub to signal how much there was
<noddy>
but then again, you can't ever measure entropy since the property is relative
<avsm>
i dont think /dev/urandom will ever return short in normal use anyway
<noddy>
urandom won't, ever
<noddy>
was thinking about xen equivalent
<noddy>
this code is imho spot on
<avsm>
it may return short if interrupted by a signal, but that's hopefully marginal
<avsm>
xen right now is just doing the self_init, for later improvement
<avsm>
we have all the right hooks to do it now i think
<noddy>
a can live with that and a small warning
<noddy>
btw what is the init sequence of a mirage unikernel?
<avsm>
yep, look at mirage-entropy and merge if ok?
<noddy>
and the teardown?
<avsm>
no teardown
<noddy>
init?
<noddy>
i want 'a Lwt.t to fire *before* start?
<avsm>
init is just call the start function in the Job
<avsm>
got to be done in a device connect
<noddy>
hmhmhm
<avsm>
what is it?
<noddy>
so for _correct_ use, we need to spin a timer ring and periodically read from the entropy, right?
<noddy>
TLS needs to, that is
<noddy>
so i was wondering how to avoid having the "init_now ()" in there
<noddy>
and just fire the ring in the background
<noddy>
without having a race condition too
djs551 has quit [Quit: Leaving.]
djs551 has joined #mirage
<noddy>
there needs to be at least one successful reseed before the stack starts
<noddy>
and it needs to continue in the background
<noddy>
how?
yastero_ has joined #mirage
<avsm>
the stack would just block in connect() until the reseed finishes, and background the thread
<avsm>
like the tcp stack does for its timers
tlockneyz has joined #mirage
<noddy>
i was afraid you'd say so :D
tlockneyz is now known as tlockney
djs55 has quit [*.net *.split]
IbnFirnas has quit [*.net *.split]
tlockney_away has quit [*.net *.split]
yastero has quit [*.net *.split]
<avsm>
that's ok isnt it?
<avsm>
it's the device model
djs551 is now known as djs55
<noddy>
actually not so bad since it's global
<noddy>
Tls.t does not have to sync
<noddy>
only connects need to wait for a lwt waiter
<miragebot>
[mirage] pqwy created new-entropy-sig (+1 new commit): http://git.io/sTCwEA
thomasga has joined #mirage
djs55 has joined #mirage
<noddy>
thomasga: ping
<thomasga>
noddy: pong
<noddy>
you are right, "Entropy" would name-clash
<noddy>
but
<hannes>
morning
<noddy>
i am not sure what having multiple would even mean
<noddy>
hannes: morning :)
<thomasga>
well, seems that you already have 2 implementations for unix
<noddy>
just one
<thomasga>
weak and strong ?
<noddy>
that got pulled out
<thomasga>
(just quickly read some of the patches)
<thomasga>
ha
<noddy>
check out half a day of discussion above. currently they are unique.
<thomasga>
well, I don't know then … but maybe you want to have a network stack with no entropy at all
<thomasga>
and one with some entropy
<thomasga>
and you want to test the two stacks in the same program
<thomasga>
but I haven't followed the discussion at all, so I trust your choices :p
<noddy>
i am actually slightly confused by the mirage.ml tool itself and how to provide those identifiers
<noddy>
there are two providers: `Entropy_xen` and `Entropy_unix (OS.Time)`
<noddy>
in the second case, there is a need to make an identifier for this application. this is where i get confused.. for every separate application you do in config.ml? a separate one for each of two providers?
<thomasga>
Name.of_key <something unique to the implementation> ~base:<prefix of the idenfier>
<noddy>
aha way just reading that module
<noddy>
was
<thomasga>
ideally you instanciate the functor once
<thomasga>
and you use the 'module_name' everywhere
<thomasga>
(and you add some 'module_name' = Functor(Arg1)(Arg2)(…)' in the configure function
<thomasga>
(as you've already done)
<noddy>
looking at Console, in mirage.ml
<thomasga>
you just need to make the 'module_name' a bit more unique
<noddy>
it uses its `t`-type to make a key
<noddy>
but who provides this `t` (the string)?
<thomasga>
ha, that's because there is only one Console implementation
<thomasga>
but you can have multiple consoles
<thomasga>
(with different connect parameters)
<thomasga>
the "t" string is provided by config.ml