Oh No! Security Metrics!

How Secure is Secure?

The Security Development Lifecycle — Thu, 08 May 2008 19:47:49 GMT

Hi folks, Eric Bidstrup here. As I touched on in my December posting on Common Criteria , and as Michael

re: Oh No! Security Metrics!

SDL Team — Mon, 28 Apr 2008 21:07:49 GMT

Bryan

We do measure attack surface, it's a critical part of a product's security and one facet of the SDL that has nothing to do with code security. In general, we've driven the attack surface down substantially from Win2000 and Windows XP. The good news is you can decide on your metrics and measure it for yourself :)

re: Oh No! Security Metrics!

bryansowen — Fri, 25 Apr 2008 14:23:28 GMT

Perhaps a review of relative attack surface quotient would offer insight about the effectiveness of early SDL stages.

Specifically include Windows 2008 and Server Core to the charts from 2003 "Measuring Relative Attack Surfaces" by Howard, Pincus, Wing.

re: Oh No! Security Metrics!

Patrick_Boyd — Mon, 21 Apr 2008 17:39:50 GMT

I really wouldn't worry about Mr. Lindstrom's criticism very much.

Are publicly disclosed vulnerability count a perfect metric of security? Of course not.

But is it the best metric that we currently have access to? I think so, and obviously Microsoft does too.

Until Mr. Lindstrom can suggest a better metric, or someone else can. I would stay the course and keep doing your best to secure the OS and tools that most of us run.

re: Oh No! Security Metrics!

asteingruebl — Fri, 18 Apr 2008 18:44:01 GMT

Michael,

To the outsider however its not clear where the defect reduction is coming from. What we'd love to have some insight into is whether you're measuring defects at multiple stages of the development lifecycle and seeing reductions throughout.

Part of this goes to the question of efficiency. You could for example simply test the hell out of things, hire better testers, etc. If you didn't do developer education, didn't do threat modeling, and still got a massive reduction in the end vulnerability count, you'd still have an effective process and it would show up in fewer patches.

If you don't have any metrics about the effectiveness of training, threat modeling, etc. and you can't track where defects are being created, and discovered/prevented, then its hard to know which parts of the process are working and which aren't.

This isn't to say that MS needs to share all of its internal metrics, tracking, etc. But, it does point to vuln counts as not at all indicating that the SDL is working, but maybe that some part of it is working.

I think you're positioned to pull together some very interesting metrics because of your diversity. Things like defect counts of a given type per programming language and/or dev environment. Details on the percentage of design vs. implementation defects, and when they are being discovered.

Do you have anyone sitting around doing nothing who wants to just work on publishing metrics and such for the rest of us to consume? :)

Oh No! Security Metrics!

Michael Howard's Web Log — Fri, 18 Apr 2008 18:35:02 GMT

I just posted an article over on the SDL blog about security metrics in reponse to an analysts criticisms