David LeBlanc's Web Log : Vulns

CVE Count and Statistics

david_leblanc — Tue, 09 Dec 2008 01:04:00 GMT

Larry Seltzer had some interesting comments on my post about the rate of Office vulnerabilities at Vulnerabilities and Office Versions

There may be a little flaw in the analysis in that LeBlanc studied reports during the period from 9/18/2007 to 11/17/2008. By that time earlier Office versions had been around for a long time and many vulnerabilities had already been reported on them. But even so, it makes the numbers all the more impressive for the new versions; the older ones had already had the low-hanging fruit picked clean and yet they still had CVE numbers in excess of the new ones. It seems there is no low-hanging vulnerability fruit in new versions of Office.

Having had more grad school than I'd like to admit, I have a more than passing acquaintance with statistics. While there are certainly potential flaws in the numbers I posted, I don't think this is one of them. I'll argue that comparing vulnerability rates over the same time frame for two applications that are very similar, and which both have large market share, is better than comparisons of some number of days since release. If we have the same time frame, then the techniques used by the attackers are likely to be similar, and when we're looking at multiple versions of the same thing, we can get a good estimate of how resistant one version is to attacks that another version is susceptible to.

What will be a problem in my analysis is how small the overall sample size is, and the fact that updates tend to ship at most 3-4 times per year for most of these apps. For example, this month's set of bulletins are going to skew the results considerably, but the overall trend of substantial improvement will still show up. Once I get updated numbers, I'll work them up and post them here.

Improvements in Office Security

david_leblanc — Tue, 18 Nov 2008 07:17:00 GMT

We now have a pretty neat internal web site where I can easily search for CVE entries and bulletin counts by product. It shows some interesting trends that I hope will continue to hold. First, let me preface this by saying that CVE entry count is a better (though not perfect) way to measure how secure something is than bulletin count. We might sometimes package fixes for several CVE entries into one bulletin, and an older product might be vulnerable to all of them, but a newer product might only be vulnerable to around half.

We did a lot of work to make Office 2003 more secure in service pack 3 – one question I've had is just how much that's paid off? It has been about a year, and if I search from 9/18/2007 to 11/17/2008 (today), I get the following:

Product	CVE count
Office 2000 SP3	33
Office XP SP3	40
Office 2003 SP2	35
Office 2003 SP3	20
Office 2007 Gold	19
Office 2007 SP1	16

The trending here is pretty clear – while we did a lot of good work to try and make Office 2003 more secure than previous versions, against the attacks we're seeing in 2007, it wasn't any better than Office XP. Now if you factor in huge amounts of work (no magic, no silver bullet, just lots and lots of work) that we did fixing fuzz bugs in Office 2007 and Office 2003 SP3, it looks like we've cut the incoming vulnerability rate by approximately half. If we look at it app-by-app, I think PowerPoint is a clear winner – they've had 5 CVE entries for older versions and only 1 for PowerPoint 2007 since 1/1/2007! Word has also done very well, dropping from 11 and 12 CVE entries in prior versions to only 2 for Word 2007 over the same period.

We're continuing to do that level of work on anything that still has a service pack left – next SP will be SP2 for Office 2007. It will be interesting to see how much additional gain that gives us. I'd like to see us do even better over time – while we've clearly made some significant gains, we still have more work remaining. We are currently doing about as many fuzzing iterations per weekend as we're required to do to meet SDL requirements for the entire product cycle (to be fair, the requirement is for clean runs, and we're not there yet, and when we do get there, we use a different fuzzer). We've done twice as many fuzz iterations against Office 2007 SP2 as we did against Office 2007 during the entire product cycle, and 4x more against Office 14 than against Office 2007.

If there's anyone out there still on Office 2003 SP2, I hope I've given you some convincing data that shows an upgrade to SP3 or better yet Office 2007 is going to pay off in much better security.

Lies, Damn Lies, Information Leaks, and Statistics

david_leblanc — Tue, 01 Jul 2008 22:37:00 GMT

Robert Hensing posted some criticism of a study that purported to analyze how many users are at risk due to using out of date or unpatched browsers. Rob rightfully points out that you can actually be running a very old version of IE (depending on OS), and still be patched against current attacks.

A flaw that IE doesn't have is advertising to the server the exact minor version of the application. People often underestimate the value of information leaks – advertising the exact minor version is basically saying "Hello, you may attack me with these exploits, but I'm patched against those exploits." You can often figure this out with various fingerprinting techniques, but sometimes you can't. As it turns out, Safari, Firefox and Opera all have information disclosure flaws, and these were used to estimate the number of vulnerable browsers by examining Google's server logs. Because IE doesn't advertise this information to the server, they couldn't do a valid comparison, and dropped to a different data set.

Rob admits to not having much statistical training, but having spent far too much time in graduate school, I've had quite a bit. First thing to consider is the sample size. As it turns out, you don't need that many samples to be valid. Secunia's sample was around 500,000, which is more than adequate. The next thing to consider is whether we're really dealing with 2 different populations. To their credit, they do call this out:

Secunia [21] identified (for the month of May 2008) that 4.4% of IE7, 8.1% of Firefox, 14.3% of Safari (Windows only), and 15.2% of Opera users have not applied the most recent security patches available to them from the software vendor. In comparison, we discovered that 16.7% of Firefox, 34.7% of Safari (all OS), and 43.9% of Opera Web browser installations (using our Web server log-based measurements) had not applied the most recent security patches. We found that our Firefox, Safari, and Opera results were higher than those of Secunia's, differing by a factor of 2.1 (Firefox), 2.4 (Safari), and 2.9 (Opera), and attribute this difference to a probable bias for more security aware users to take advantage of Secunia's security scanner PSI than the average global community.

First of all, we can clearly establish that we're dealing with 2 distinctly different populations. The assertion that IE6 is insecure is invalid, as it is still in support and gets security patches just like IE7, and while IE 7 is doing a bit better on bulletin count, it isn't a huge difference, and as I've noticed with several other products, there is a grace period where attackers don't bother until there's enough adoption. While it is interesting that around twice as many Firefox users aren't fully patched as IE7 users, this might be an artifact of release timing. The authors of the study then attempt to deal with these different populations by comparing the Google results to the Secunia results, but there's a lot of variance between the browser types – if Firefox users going to Google are 2.1x less likely to be secure than Firefox users identified by the Secunia study, but Opera is different by a factor of 2.9x, then the difference between IE users overall vs. Secunia is really anyone's guess. Is it 2.9x? 4x? 1.5x? No one really knows.

It's an interesting thing to try and study, and the hypothesis that different patch delivery mechanisms might make a difference in how many users are at risk is also interesting, but data on IE users who are the majority of the population, and could behave differently as a group than users of other browsers, is really not available which makes the conclusions very questionable. Another factor that they appear not to have considered is that the number of browsers missing patches is going to be a function of how often you see patches. Something patched once a year is more likely to be patched than something patched 25 times a year.

Interesting paper – too bad their conclusions aren't supportable for the bulk of the users who are using IE.