Fabulous Adventures In Coding : Benford's Law

Floating Point And Benford's Law, Part Two

Eric Lippert — Thu, 13 Jan 2005 17:54:00 GMT

A number of readers asked for an explanation of my offhand comment that Benford's Law can be used to show that binary is the best base for doing floating point math.

One of the desired properties of a floating point system is that the "representation error" is as small as possible. For example, suppose we want to express "one third", in ten-place fixed point decimal notation. The closest we can get is "0.3333333333", which has a representation error of 0.0000000000333333…

A useful way to get a handle on representation error is to look at the granularity of the system. By granularity I mean the smallest difference we can make between two values. For ten-place fixed point decimal notation, the smallest nonzero difference between any two numbers is 10^-10. In a floating point system, the granularity changes as the exponent changes. Really large numbers have large granularity; the difference between two successive floats might be millions or billions. And really small numbers have tiny granularity. The maximum possible representation error is half the granularity. Since there is a clear relationship between granularity and representation error, I’m just going to talk about granularity from now on. Small granularity = small representation error = goodness.

Let's consider two similar systems, one which uses a binary mantissa and one which uses a hexadecimal mantissa. I’m going to continue my convention of showing binary numbers in blue, and we’ll do hex in green.

Suppose we've got on the one hand, 32 bit IEEE floats, aka "singles". That is, we've got one sign bit, nine bits of exponent biased by -255 (so exponents can be from -255 to 256), a 22 bit mantissa, and an implicit leading "1." So a number like (0, 100000011, 1100000000000000000000) would be +1.1100 x 2⁴ = 28

I’m getting sick of spelling out the exponents in non-biased binary. From now on, I’m just going to give them in decimal. And I’m going to give the sign bit by plus and minus, not zero and one.

We now want a hex system with roughly similar range that doesn't take up more storage. Suppose we've got one sign bit, seven bits of exponent biased by -63, and six hex digits for the mantissa. We don't get a leading "1." because not every hex number can be so expressed, so we'll have to use a leading 0. That system has roughly the same range, and if we were to "behind the scenes" express this thing as bits, we'd still be using 32 bits. One for the sign, seven for the exponent, and 24 for the mantissa.

In this system, (+, 2, A00000) would be +0.A00000 x 16² = 160

An immediate disadvantage of this system is that numbers have multiple representations. (+, 2, A00000) and (+, 3, 0A0000) are the same number. Let’s ignore that for now. We’ll ignore all hex mantissas with leading zeros. (And besides, the binary IEEE system wastes lots of cases for denormals and NaNs too, so it's not clear that this is any worse.)

What's the granularity of the hex system when, say, the exponent is 2? Well, consider a number like (+, 2, A00000) – the smallest possible number higher than this is (+, 2, A00001) which is 2^-16 larger. Clearly, 2^-16 is the granularity for all values with an exponent of 2. More generally, if the exponent is N then the granularity is 2^4N-24.

How would we represent (+, 2, A00000) in our binary system? That would be (+, 7, 0100000000000000000000) = +1.01 x 2⁷ = 160. The next largest number that can be represented in our system is (+, 7, 0100000000000000000001) which is 2^-15 larger. So the hex system has smaller granularity and hence smaller representation error, and is therefore the better system, right?

Not so fast. Let’s make a chart.

Decimal       Hex             Binary                  Binary Granularity Exponent

  16    (+, 2, 100000) (+, 4, 0000000000000000000000) -18
  32    (+, 2, 200000) (+, 5, 0000000000000000000000) -17
  48    (+, 2, 300000) (+, 5, 1000000000000000000000) -17
  64    (+, 2, 400000) (+, 6, 0000000000000000000000) -16
  80    (+, 2, 500000) (+, 6, 0100000000000000000000) -16
  96    (+, 2, 600000) (+, 6, 1000000000000000000000) -16
112    (+, 2, 700000) (+, 6, 1100000000000000000000) -16
128    (+, 2, 800000) (+, 7, 0000000000000000000000) -15
144    (+, 2, 900000) (+, 7, 0010000000000000000000) -15
160    (+, 2, A00000) (+, 7, 0100000000000000000000) -15
176    (+, 2, B00000) (+, 7, 0110000000000000000000) -15
192    (+, 2, C00000) (+, 7, 1000000000000000000000) -15
208    (+, 2, D00000) (+, 7, 1010000000000000000000) -15
224    (+, 2, E00000) (+, 7, 1100000000000000000000) -15
240    (+, 2, F00000) (+, 7, 1110000000000000000000) -15

The hex system always has a granularity exponent of -16. In 8/15ths of the cases, the binary system has worse granularity than the hex system. In 4/15ths of the cases, they have the same granularity, and in only 3/15ths of the cases, the binary system has better granularity. Therefore the hex system is clearly the same or better most of the time. We should use this system instead of binary floats.

Not so fast! We’ve forgotten Benford’s Law! That's true if any number is as likely as any other, but that's not realistic.

Suppose the numbers that we are manipulating obey Benford’s Law. In that case, we would expect that fully a quarter of them would begin with 1 when encoded in hex. We’d expect another quarter of them to begin with 2 or 3, another quarter to begin with 4, 5, 6 or 7, and the remaining quarter to begin with 8, 9, A, B, C, D, E and F. If we make that assumption then we must conclude that the binary system is better half the time, equal a quarter of the time, and worse a quarter of the time.

Clearly this isn't the case just for the exponent 2. For any hex exponent N, the hex system will have a granularity exponent of 4N-24. For numbers in the range expressible by the hex system with that exponent, a quarter of the time, the binary system will have a granularity exponent of 4N-26, a quarter of the time it'll be 4N-25, a quarter of the time it'll be 4N-24, and the remaining quarter it'll be 4N-23, so three-quarters of the time it'll be as good or better. On average, the binary system is considerably better if data are distributed according to Benford's Law.

This is just one example, not a proof. But we could generalize this example and show that for any system with a given number of bits, binary mantissas yield smaller representation errors on average than mantissas in any other base.

Benford's Law

Eric Lippert — Wed, 12 Jan 2005 20:18:00 GMT

While I was poking through my old numeric analysis textbooks to refresh my memory for this series on floating point arithmetic, I came across one of my favourite weird facts about math.

A nonzero base-ten integer starts with some digit other than zero. You might naively expect that given a bunch of "random" numbers, you'd see every digit from 1 to 9 about equally often. You'd see as many 2's as 9's. You'd see each digit as the leading digit about 11% of the time. For example, consider a random integer between 100000 and 999999. One ninth begin with 1, one ninth begin with 2, etc.

But in real-life datasets, that's not the case at all. If you just start grabbing thousands or millions of "random" numbers from newspapers and magazines and books, you soon see that about 30% of the numbers begin with 1, and it falls off rapidly from there. About 18% begin with 2, all the way down to less than 5% for 9.

This oddity was discovered by Newcomb in 1881, and then rediscovered by Frank Benford, a physicist, in 1937. As often is the case, the fact became associated with the second discoverer and is now known as Benford's Law.

Benford's Law has lots of practical applications. For instance, people who just make up numbers wholesale on their tax returns tend to pick "average seeming" numbers, and to humans, "average seeming" means "starts with a five". People think, I want something between $1000 and $10000, let's say, $5624. The IRS routinely scans tax returns to find unusually high percentages of leading 5's and examines those more carefully.

Benford's result was carefully studied by many statisticians and other mathematicians, and we now have a multi-base form of the law. Given a bunch of numbers in base B, we'd expect to see leading digit n approximately ln (1 + 1/n) / ln B of the time.

But what could possibly explain Benford's Law?

Multiplication. Most numbers we see every day are not random quantities in of themselves. They're usually computed qualities with some aspect of multiplication to them.

Consider, for example, any property which grows on a percentage basis. Like, say, the Dow Jones Industrial Average. It typically grows a few percent a year. Suppose, just to pick a rate, that on average the DJIA grows at 7% a year. At that rate, it doubles about every ten years. Suppose that the DJIA is 10000. After ten years of having 1 as the leading digit, it finally gets to 20000. Ten years go by again, but in that ten years, it doubles to 40000, not 30000. Therefore, those ten years were spent about half starting with 2, and about half starting with 3. Ten more years go by, and it doubles again to 80000. Now ten years have 4, 5, 6 and 7 as the leading digits in only ten years. Eventually we get up to 100000, and spend another ten years starting with 1. Pick a random date and you'd expect that the DJIA on that day would be twice as likely to start with 1 as 2, and four times as likely to start with 1 as 5.

We can easily write a program that demonstrates Benford's Law. As we multiply more and more numbers together, they tend to clump based on Benford's Law:

var counters = [
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0] ];

for (var multiplications = 0 ; multiplications <= 3; ++multiplications)
{
for (var trial = 0 ; trial < 10000 ; ++trial)
{
    var num = 1;
    for (var mult = 0 ; mult <= multiplications; ++mult)
      num = num * (Math.floor(Math.random() * 1000) + 1);
    var lead = num.toString().substr(0,1);
    counters[multiplications][lead] ++;
}
}
print(counters.join("\n"));

A typical run produces data from which we can draw this table:

                         Leading Digit
Mults       1    2    3    4   5   6   7   8   9
0       1102 1069 1085 1125 1167 1107 1083 1124 1138
1       2416 1752 1443 1162 1019 756 643 453 356
2       3046 1854 1265 979 778 632 551 468 427
3       3090 1814 1197 924 779 661 582 491 462
predicted 3010 1761 1249 969 792 669 580 511 458

As you can see, with no multiplications, the distribution is a flat 11% for each. But by the time we get up to two or three multiplications, we're almost exactly at the distribution predicted by Benford's Law.

What does this have to do with floating point math? Well, we could conceivably design chips that did decimal or hexadecimal floating-point arithmetic. Would such chips yield more accurate results? Well, recall that last time, we used the fact that you can stuff a leading 1 onto a bit field to define a number. Binary is the only system in which every number except 0 begins with a leading 1! You can make a statistical argument which shows that for bases other than binary, in which you cannot always assume a leading digit, have on average a larger representation error. The argument is somewhat subtle, so I'm not going to actually go through the details of it, but suffice to say that we can show that for typical uses, binary is the least error-producing system we can come up with given that we'll almost always be working with data which follow Benford's Law.