The Quest for True Randomness

(I wrote a long post, and it turns out my login session had expired. On trying to save, I just lost the whole post. Just so that you know the hazards of leaving your browser tabs open for ages.)

How can a program running on a desktop/laptop get “true” random bits? From searching around a bit, the “good” answer seems to lie in the firmware hub RNG. (Also see this (pdf).) Since the raw bits produced by the circuitry are not unbiased, they use the von Neumann extractor: this is indeed sufficient, as long as the bits are i.i.d (independent and identically distributed) even if they are not uniform (i.e., as long as they are independent tosses using the same coin, even if the coin is not a fair coin).

Apparently you can access this RNG via /dev/hw_random. You may need root access for this. A word of caution: in the one Linux box that I have access to, cat’ing /dev/hw_random produces a rather questionable bit sequence of all 1’s. Maybe I don’t have the firmware hub in my box?

Anyway, it seems the more familiar source of randomness used by a program is /dev/random. It has a different philosophy to randomness:

The random number generator gathers environmental noise from device drivers and other sources into an entropy pool. The generator also keeps an estimate of the number of bits of noise in the entropy pool. From this entropy pool random numbers are created.

This sounds quite fine, as long as the raw randomness from this entropy pool is “refined” before use (or is already clean because it came from /dev/hw_random — well, not mine I hope). I don’t know what is there in the Linux kernel to refine raw randomness, but I’ll hazard a guess that it doesn’t probably have one of the shiny new randomness extractors (deterministic or seeded). (If anyone knows better, please do comment.) The randomness extractors are precisely the tools you want here: as long as there are some minimal guarantees of entropy and independence in the raw source, an extractor can be used to efficiently extract much of this randomness into a uniform source, ready to be used as a one-time pad or what not. It will be also nice to have a /dev/pseudorandom which can take such a seed from /dev/random and stretch it using a “provably secure” PRG.

Aside: Even the hardware RNGs have their share of trouble. See this for instance. Again, as long as the hardware can provide some partial guarantee in the face of such attacks, a randomness extractor will prove quite handy.