CVE Count and Statistics

david_leblanc — Tue, 09 Dec 2008 01:04:00 GMT

Larry Seltzer had some interesting comments on my post about the rate of Office vulnerabilities at Vulnerabilities and Office Versions

There may be a little flaw in the analysis in that LeBlanc studied reports during the period from 9/18/2007 to 11/17/2008. By that time earlier Office versions had been around for a long time and many vulnerabilities had already been reported on them. But even so, it makes the numbers all the more impressive for the new versions; the older ones had already had the low-hanging fruit picked clean and yet they still had CVE numbers in excess of the new ones. It seems there is no low-hanging vulnerability fruit in new versions of Office.

Having had more grad school than I'd like to admit, I have a more than passing acquaintance with statistics. While there are certainly potential flaws in the numbers I posted, I don't think this is one of them. I'll argue that comparing vulnerability rates over the same time frame for two applications that are very similar, and which both have large market share, is better than comparisons of some number of days since release. If we have the same time frame, then the techniques used by the attackers are likely to be similar, and when we're looking at multiple versions of the same thing, we can get a good estimate of how resistant one version is to attacks that another version is susceptible to.

What will be a problem in my analysis is how small the overall sample size is, and the fact that updates tend to ship at most 3-4 times per year for most of these apps. For example, this month's set of bulletins are going to skew the results considerably, but the overall trend of substantial improvement will still show up. Once I get updated numbers, I'll work them up and post them here.

Improvements in Office Security

david_leblanc — Tue, 18 Nov 2008 07:17:00 GMT

We now have a pretty neat internal web site where I can easily search for CVE entries and bulletin counts by product. It shows some interesting trends that I hope will continue to hold. First, let me preface this by saying that CVE entry count is a better (though not perfect) way to measure how secure something is than bulletin count. We might sometimes package fixes for several CVE entries into one bulletin, and an older product might be vulnerable to all of them, but a newer product might only be vulnerable to around half.

We did a lot of work to make Office 2003 more secure in service pack 3 – one question I've had is just how much that's paid off? It has been about a year, and if I search from 9/18/2007 to 11/17/2008 (today), I get the following:

Product	CVE count
Office 2000 SP3	33
Office XP SP3	40
Office 2003 SP2	35
Office 2003 SP3	20
Office 2007 Gold	19
Office 2007 SP1	16

The trending here is pretty clear – while we did a lot of good work to try and make Office 2003 more secure than previous versions, against the attacks we're seeing in 2007, it wasn't any better than Office XP. Now if you factor in huge amounts of work (no magic, no silver bullet, just lots and lots of work) that we did fixing fuzz bugs in Office 2007 and Office 2003 SP3, it looks like we've cut the incoming vulnerability rate by approximately half. If we look at it app-by-app, I think PowerPoint is a clear winner – they've had 5 CVE entries for older versions and only 1 for PowerPoint 2007 since 1/1/2007! Word has also done very well, dropping from 11 and 12 CVE entries in prior versions to only 2 for Word 2007 over the same period.

We're continuing to do that level of work on anything that still has a service pack left – next SP will be SP2 for Office 2007. It will be interesting to see how much additional gain that gives us. I'd like to see us do even better over time – while we've clearly made some significant gains, we still have more work remaining. We are currently doing about as many fuzzing iterations per weekend as we're required to do to meet SDL requirements for the entire product cycle (to be fair, the requirement is for clean runs, and we're not there yet, and when we do get there, we use a different fuzzer). We've done twice as many fuzz iterations against Office 2007 SP2 as we did against Office 2007 during the entire product cycle, and 4x more against Office 14 than against Office 2007.

If there's anyone out there still on Office 2003 SP2, I hope I've given you some convincing data that shows an upgrade to SP3 or better yet Office 2007 is going to pay off in much better security.

David LeBlanc's Web Log : Security Misc

CVE Count and Statistics

Improvements in Office Security