Benford's Law

re: Benford's Law

Alex Papadimoulis — Wed, 12 Jan 2005 20:55:00 GMT

Thanks, Eric. Great educational post!

re: Benford's Law

G. Man — Wed, 12 Jan 2005 21:44:00 GMT

Heh, you going to put this on the daily WTF? I was with Eric until the last paragraph and then I was going "WTF"!

re: Benford's Law

Jon Galloway — Wed, 12 Jan 2005 21:54:00 GMT

Thanks for taking the time to write this up. Really interesting!

re: Benford's Law

Marcus — Wed, 12 Jan 2005 22:12:00 GMT

How odd, I was just discussing Benford with a mate over a pizza the other day, and we sort of touched upon most of this.

Great post, as always though.

re: Benford's Law

Gabe Halsmer — Wed, 12 Jan 2005 22:21:00 GMT

Subtle indeed. What does the leading digit have to do with errors in representation?

re: Benford's Law

Adi Oltean — Wed, 12 Jan 2005 22:50:00 GMT

I remember that the guy who discovered this noticed that the a used book containing the logaritmic tables was worn more around the numbers starting with the "1" digit, and less and less going to higher digits...

re: Benford's Law

James — Thu, 13 Jan 2005 00:10:00 GMT

Interesting, but I'm sure if you gave a bunch of people in the financial industry the option of working with floating point in base ten, they would jump at the chance of an order-of-magnitude speed improvement. Their error bounds are defined in base 10 anyway.

re: Benford's Law

Norman Diamond — Thu, 13 Jan 2005 00:53:00 GMT

1/12/2005 2:50 PM Adi Oltean

> a used book containing the logaritmic tables
> was worn more around the numbers starting
> with the "1" digit,

Yes, I think Martin Gardner published that story (though was he first or second to do so?). Anyway, it made me think that the opposite book, containing exponential tables, would be uniformly worn. The contents of such a book would be uniformly distributed on the value of the argument (say x) but skewed on the value of the exponential (e**x), and most of the values found in it would be where the numeral for the e**x value starts with a relatively low digit.

1/12/2005 4:10 PM James

> Interesting, but I'm sure if you gave a
> bunch of people in the financial industry
> the option of working with floating point in
> base ten, they would jump at the chance of
> an order-of-magnitude speed improvement.

What speed improvement? Hardware to do base 10 arithmetic is grossly slower than hardware to do base 2 arithmetic, and making it floating instead of fixed would not lessen that slowness.

> Their error bounds are defined in base 10
> anyway.

Sure. Do you mean that financial programmers would find an order-of-magnitude speed improvement by doing programming using base 10 floating instead of either base 10 fixed or binary floating? I doubt that very much.

'Course, financial programmers will always have it easier than programmers for space ships or even cars. Accuracy equivalent to twenty decimal digits can be done by using twenty decimal digits, but that's nowhere near accurate enough for some kinds of physical operations.

re: Benford's Law

James — Thu, 13 Jan 2005 01:06:00 GMT

Norman Diamond, Mike Cowlishaw's presentation to the C committee from before Kona contains some interesting figures (I missed it, being C++):
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1037.pdf

re: Benford's Law

pete23 — Thu, 13 Jan 2005 01:46:00 GMT

top post. great exposition (i'd encountered this before, but hadn't realised it wasn't benford's)

re: Benford's Law

thomas woelfer — Thu, 13 Jan 2005 10:56:00 GMT

Eric,

thanks - that was fun.

WM_THX
thomas woelfer

re: Benford's Law

Marcus Tucker — Thu, 13 Jan 2005 14:25:00 GMT

Cheers for another fascinating post, Eric!

re: Benford's Law

Eric Wallace — Thu, 13 Jan 2005 18:10:00 GMT

Reminds me of a favorite Dilbert cartoon, in which he gets a tour of the accounting dungeon... The tour guide says, "Over here we have our Random Number Generator..." as they pass a troll spouting the words "NINE NINE NINE NINE NINE NINE NINE NINE". Dilbert asks, "Are you sure that's random?"

The response--"That's the problem with randomness: you can never be sure." Heh.

Oh, and thanks for the tax tips!

~ewall

re: Benford's Law

Todd — Thu, 13 Jan 2005 21:12:00 GMT

Awsome post.

Check Out The Big Brain on Eric

Stack Of Toast — Fri, 14 Jan 2005 23:19:00 GMT

re: Benford's Law

Dave Bacher — Tue, 18 Jan 2005 21:10:00 GMT

Just as a comment...

The advantage to decimal over the current format is that most readings taken "in the real world" by "real people" of anything of scientific or financial intrest aren't in base 2, and therefore anything that better represents things of interest to most people in the real world in the manner in which they are measured is likely to be better accepted.

It's slower, and optimizing hardware for it is more difficult, but if given a choice between being able to represent a .1 interval precisely as it is measured or not being able to measure it precisely as it is measured, I think most people would pick slow and precise over fast and inaccurate for their simulations, etc.

re: Benford's Law

Eric Lippert — Tue, 18 Jan 2005 21:28:00 GMT

Hence the "decimal" type in C#, VB6 and VB.NET. A decimal is a 96 bit integer scaled by a power of ten. The power can be any exponent from 0 to -28, which means that 0.1 can be represented exactly.

Similarly, the Currency type in VBScript is a 64 bit integer with a fixed exponent of 10^-4. That can also represent 0.1 exactly.

Both systems trade execution speed for decimal precision. I don't know why more people don't use them.

re: Benford's Law

Ken Warren — Sat, 07 May 2005 02:13:44 GMT

Multiplications explain the tendency for financial data to follow Benson's, but what about data from nature? River lengths show it for instance, irrespective of the units (miles, kilometres - even inches) used.

re: Benford's Law

Eric Lippert — Sat, 07 May 2005 15:03:00 GMT

Look at it this way: take a set of river lengths. Pick Measuring System #1, and determine the distribution of first digits. Suppose it's "miles"

Now pick a random number, say 12.3. Say a "frob" is 1/12.3 miles. Multiply every length by that number, so that you've now got a set of measurements in frobs, not miles.

Check out the new distribution.

Would you expect the distributions of first digits to be roughly the same whether in miles or frobs? Would you expect it to be the same given ANY choice of measuring unit?

Such sets are "scale invariant", and the only distribution which is scale invariant is the Benford's Law distribution.

re: Benford's Law

Jerry — Wed, 25 May 2005 19:24:18 GMT

Very Interesting. Now I'm wondering if as a Fibonacci Series. In Elliot Wave Theory, the stock market moves up and down based on Fibonacci series. And This series can be found ad infinitum in long term and short term charts. So my question is, does Benfords law apply if let's say the Stock market hit 1000. Could you drop the first digit and apply benfords law to the 2nd digit as if it were the 1st. And so forth. Am I just blabbering or is there patterns within patterns using Benfords Law

re: Benford's Law and the Grocery Store Receipt Law

RICHARD PALMER — Wed, 01 Jun 2005 20:55:37 GMT

This is a "MSD" law (Most Significant Digit). It is interesting to think about what an LSD law would look like. Consider what I will call a "Grocery Store Receipt" law. I am thinking about a receipt from a grocery store that shows quantities, prices, and total cost (Q*P). Assume quantities are uniformly distributed (not true in reality) & that prices are uniformly distributed. Then the distribution of the LSD of total costs is NOT uniformly distributed.

re: Benford's Law

Johnny Sprada — Sat, 04 Jun 2005 02:10:43 GMT

Jerry -
Check out the graphic chart at this link:
http://www.aicpa.org/pubs/jofa/may1999/nigrini.htm

I as per Mark Nigrini's web site, I made a list of the sizes of all the files on my C: drive (Window's machine) and applied Benford's Law to the list of numbers. It was quite amazing to see that the law applied to the file sizes with two anomalies: those beginning with the digits 5 and 7.. these happen to be font files and DLL's. Applying Benford's to not only the first digit, but the first and second combination, first second and third, etc... can reveal all types of anomalies.

re: Benford's Law

Mike Blakley — Sun, 14 Jan 2007 01:01:44 GMT

There are a wide variety of uses for the application of Benford's law - detection of tax fraud, voting (ballot box) fraud, "curb-stoning" in surveys, etc. I have also posed an Excel workbook at http://www.ezrstats.com/EZSXL/Test_Benfords_law.xls with all the formulae - first 1-3 digits, last 1-2 digits, second digit, etc.

re: Benford's Law

Stephen Druley — Mon, 22 Jan 2007 20:10:16 GMT

Eric,

Do you have a vba code example that will help me understand your example?

Could Benford's law be used to test the legitimacy of a gaussian time series?

re: Benford's Law

John Morrow — Wed, 01 Aug 2007 12:06:28 GMT

There is a free utility which will do Benford's law analysis on Comma Separated Value files here:

http://www.checkyourdata.com