Bits, Nats, and Hartleys

An important concept in a recently-completed generals paper of mine was mutual information, a measure of how much information knowing the value of one random variable tells you about another.  Since it's a measure of information, you might expect that the units of mutual information are bits, and you'd be right, much of the time.  Bits are the most commonly used units nowadays, but they're not the only possible ones.

In the course of defining mutual information in his Transmission of Information: A Statistical Theory of Communications, Robert Fano discusses the units of information:

The amount of information provided by the occurrence of the event represented by y_i about the occurrence of the event represented by x_k is defined as

	I(xk;yi) ≡ log	P(xk|yi)	(2.16)
		P(xk)

The base of the logarithm used in this definition fixes the magnitude of the unit of information.  The base most commonly used is 2, in which case a unit of information is provided about x_k when its probability is increased by a factor of 2.  The natural base e is often used instead of 2, because of its mathematical convenience.  The corresponding unit of information is obtained when the probabilty is increased by a factor equal to e.  Clearly, an increase of probability by a factor of 10 yields the units associated with the use of base 10 logarithms.  The names "bit", "nat", and "Hartley" are commonly used to indicate these three units.  The name "bit" is a contraction of "binary digit", and the name "nat" is a contraction of "natural unit".  The decimal unit has been named in honor of R. V. L. Hartley, because of his pioneering work on communication theory.  (p. 27)

[Note, by the way, that the formula in 2.16 is only for single events x_k and y_i.  For a whole probablity distribution you use the summation formula given on the Wikipedia page; Fano calls this the average mutual information.]

Nats and Hartleys, huh?  I've never heard of those, so either they've fallen out of fashion in our modern binary world, or else I just don't know as much about information theory as I thought I did.  According to the apposite Wikipedia pages, nats used to be called nits, but that was changed to avoid confusion with the other unit called the nit.  What's more, Hartleys are also called bans, and a tenth of a Hartley is a deciban.  I think I'll stick with bits, but it's nice to know I have so many options.