Security, Equality, Fraternity

What was meant as a fun little blog post over the weekend about the human element and excitement at ISO meetings spawned quite a reaction among the researcher crowd. I’d like to set a few things straight before Monday morning rolls around and even more people get the wrong idea and get upset when they could be coding or doing something else productive.

Myth: The ISO draft on Responsible Vulnerability Disclosure is some sort of plot by vendors to tell researchers what to do.

Fact: The ISO draft is misnamed. (I didn’t name it, and I can’t change it.) It should probably be called something like “Guide for Vendors to Implement a Vulnerability Handling Policy”.  It has a strict scope to apply to vendors’ actions only, not researchers’ actions. This was decided by the National Bodies participating for many reasons, not the least of which was the fact that researchers don’t usually follow ISO standards. :-)

Myth: Even if it were renamed, it still has “elements” in it that “tie researchers’ hands”. This will somehow still affect researchers!

Fact: I repeat -- There is nothing in this draft that tells researchers what to do.  Nothing.  Nada.  Nil.  No matter how hard one may try to make this about researchers, it simply isn’t about them.

The ISO draft is a guide for vendors to help them do three (simple, in theory) things:

0.       Receive vulnerability reports from researchers.

1.       Process vulnerabilities via some sort of investigation and potentially create some sort of remediation.

2.       Communicate the remediation (if any) to affected customers.

All of the guidance in the draft is very basic, and very agnostic to business particulars of how a vendor is to carry out these steps.  Since there is no prescriptive guidance in there for vendors, only high-level best practice guidance, there is nothing that will “affect researchers” except maybe, just maybe, make it easier for researchers to find the right contact to report security vulnerabilities at vendors who choose to comply with this ISO standard. 

That’s the sum total of the possible ramifications for researchers that could possibly come of this – it might make reporting vulnerabilities slightly less complicated.  I know, there’s nothing to get upset over with that, so I’m sure those who still want to yell on the Internet will ignore this entire post or pick it apart in 140 characters or less. ;-)

Besides, even if there were rules for researchers in the draft (which I repeat, there are NOT), researchers can do whatever they want. That has always been the case. Every researcher I’ve met has their own personal vulnerability disclosure (or non-disclosure) policy that they generally follow.  I say “generally” because this personal policy varies greatly from researcher to researcher, and can even have significant variation when a single researcher deals with one vendor or another, so much so that in most cases, it shouldn’t even be called a policy, but rather a “collection of exceptions”.  

Perhaps one vendor gives them more technical responses, so the researcher is inclined to give them more time. Perhaps another vendor gives them auto-generated emails every month, so the researcher gives up on ever speaking to a human and releases the vulnerability details after receiving a few robot-mails in a row.  The researcher may give a reason for changing their mind about giving a vendor more or less time, and they may not. 

The point is: nobody can tell researchers what to do now, so how or why on earth would ISO ever try? And if ISO did try to regulate researchers, who would follow the standard?  Not the researchers, that’s for sure.

Basically, researchers answer to no one.  They do what they want, when they want, and on their own timeline.    Nothing will ever change that, so why is there even a “disclosure debate” at all?

In my opinion, the debate only ever comes out when a researcher wants to be *thanked* by a vendor, either via vendor acknowledgement or via monetary compensation. That thanks (or no thanks) is up to the vendor and always will be. 

Now, let’s differentiate between gratitude and credit.  Credit, as opposed to gratitude, is not something that a vendor can give or withhold.  Credit is granted in the numerous vulnerability databases that will inevitably cite the researcher (if they made themselves known publicly), whether they dropped exploits on Full Disclosure ahead of a vendor-supplied fix or not.  A researcher who wants credit will always get it.  A researcher who wants gratitude will have to abide by the vendor’s rules for how the vendor chooses to express gratitude.

No standard will ever regulate the concepts of responsibility, whatever that means, (for researchers) or gratitude, whatever that means, (for vendors).

So there you have it, in all its futility: Researchers will continue to do whatever they want, and (shocking!) so will vendors. 

Only, when this ISO standard comes out, vendors may have a better idea on how to go about their vulnerability handling.  I’ve seen many faces of vendor response, and I can tell you that clues are rarer than you’d think, and that there is a place for this kind of basic level of guidance.

If you’re a vendor, this standard will not tell you how many security staff you need to hire to support a whole new vulnerability handling practice, or whether or not to thank a researcher.  That’s up to you.  If you are a researcher, ISO will not tell you how much time you have to give a vendor or whether or not you have the right to sell your work.  That’s up to you. 

Now you can all get back to work doing whatever you want, or continue screaming FUD-y murder.  It is, as it always has been, up to you.

When people ask me what I do at Microsoft, in the style of one of “the Bobs” in Office Space posing the question “What would you say ya do here?”, I point them to things like the SDL, the SDL Pro Network, which I manage, or MSVR, which I founded and is now managed by Adrian Stone over in MSRC.  Someday, in the next 2-3 years, I’ll also be able to point to an ISO standard. Never in my 9 lives would I have expected to say that at all, let alone with such passion and enthusiasm.

To my friends who have known me for many years, via hacker-spaces, hacker cons, soldering a Theremin in the desert, and marching with penguins, this may come as a shock. Even the fact that I came to work at Microsoft was less shocking than this.  Katie (AKA k8e), turned policy wonk?!  No way!! 

Way.

Working with ISO has been one of the most exciting (yes, you heard me) things I’ve done in my career.  Seriously.

One may think it’s boring, poring over standards drafts, looking up arcane directives about formatting terms and definitions (okay, that last part *is* a total snooze-fest), but the fact that I get to help pen part of what will be adopted as The Way to Do Things for many people around the world is exciting in and of itself.  Even that micro-slice of influence is nothing compared to the excitement of actually being there, in the twice-annual ISO meetings, hosted by rotating member nations around the world. This last meeting was hosted by the US and held at the Microsoft Conference Center, a convenient commute for me.  The previous two meetings took me to Beijing, China this past May and Limassol, Cyprus in October 2008 – the latter while 35 weeks pregnant. (Note to self: No more traveling in the form and shape of a manatee. I was thrice thrown whole raw fish, which was just a waste since pregnant women are not supposed to consume sushi, and once nearly captured and relocated by helicopter airlift to a cove in the Mediterranean to wallow in freakish misery with other large sea mammals.)

ISO meetings themselves are actually gripping microcosms of humanity.  Subtract the long sonorous stretches of Old Men Yelling at Clouds and you have yourself, as one friend put it, the UN of geeks.  It was beautiful and terrible, full of quiet riots, politics and poetry, kindness and cunning.  The real human drama that goes on has enough juice from which to squeeze a best-selling novel (working title: Bombasts and Bloviates - Policy Poetry and Pugnacious Punditry).  And yet there is a true heart to all of this; an unmistakable human element in what an outsider may otherwise assume is all sterile and businesslike in its formality of process.  We observed a somber and sincere minute of silence at the start of our working group plenary session to mark the passing of a dear friend and colleague who had been involved in SC27 since the beginning, an act clearly demonstrating how these people have really formed a community not only of experts, but of friends.

Over the past two years since joining Microsoft, I have been providing technical comments to an ISO draft standard on Responsible Vulnerability Disclosure.  I have made friends across nations and across companies in my service as a subject matter expert for this topic in the international meetings on behalf of the US National Body.  This past meeting, I was suddenly, unexpectedly asked to do much more.  The editor was indisposed due to illness and on Tuesday morning, day two of the ISO meeting and two hours before the ad hoc editing sessions began, I was asked to fill in as acting editor on the draft.

Suddenly being an acting editor for an ISO draft is like being thrown from a plane with a sufficient amount of silk thread to weave a parachute.  While success is a remote possibility, the sheer amount of work you have to do before you hit the ground (i.e. run out of time before the working group takes a vote at the end of the week) is astronomical.

We had roughly 245 comments to wade through, mostly technical, spanning nearly 80 pages on a standards draft that is a scant 12 pages including annexes.  New countries that had never provided comments on the two working drafts prior to this third draft had emerged as major players in the discussion. At its peak volume, around 15 experts from about 8 different national bodies attended the editing sessions – twice as many individuals as had ever participated in person before -- and they all had an opinion. 

Herding cats would have been easier.  We had to get consensus or at least majority agreement on everything from major concepts at the heart of disclosure, to the structure of the draft, to minute wording and definitions.  Somehow, likely because I was a merciless taskmaster who forced them all to show up early, stay late, and work through lunch, we got through all 245 comments in time for the working group vote. My heartfelt thanks go to all the participants, even those who argued the most.  Though we may not all be birds of a feather, we couldn’t have produced something as balanced and truly international without all of us there.

My work for this meeting done, I handed over the disposition of comments (the document that serves as the editor’s instructions on the changes that were agreed upon by all participating national bodies for the next iteration of the standard draft) to the editor, who was feeling much better.  I felt relief and pride that I had been able to wade through such a pungent mix of policy and personality to create something I can point to one day and say: I helped make that.  

Quite often in our industry, two (or five) people can look at the same problem from different angles, and see radically different things.  Rare is the situation that reads the same to everyone, forwards and backwards.  It’s all about perspective.
 

In my appearance on the ‘Partial Disclosure Dilemma’ Panel at SOURCEBoston this year, I found myself surrounded by great minds who most certainly do not think alike.  While there was some agreement and common ground between all parties on the dais, namely wanting to make the Internet safer and protecting people, there was little agreement on the best way to accomplish that goal.  The conversation between us friends and colleagues, both on stage and in the audience, wended its way down many tangential paths, most of which I will have to watch again on the video to fully understand how we got from Partial Disclosure to Dan Kaminsky saying “More people have died from windows crashing on them than from Windows^TM crashing.”  But I promised my redux of the panel, so I will guide you down the path I think was most interesting.  For a broader look at SOURCEBoston, check out Celene Temkin’s post.

The disclosure issues around Dan Kaminsky’s DNS vulnerability were one seed of the panel idea.  If you are reading this blog, then I will assume that you’ve heard of this vulnerability, else you must have been living under an Amish rock in a Luddite colony, high in the brisk, thin air of the Himalayas. 

As far as the disclosure route he chose and how that played out, he executed a plan he thought was best in order to get vendors to fix a serious issue (they did), *and* to get as many affected customers protected (some were) with the fix in place before broadly releasing the technical details.  He let a small number of people know the details, in the hopes that delivering those details to the right people and no one else would best protect the world’s critical infrastructure.  Hence, the term ‘partial disclosure’ was used to describe his approach.  Other notable researchers thought it was just hype, then took it back once they had spoken to Dan, some pretty much figured it out on their own, or chatted about it on DailyDave.  It was weaponized shortly thereafter, and a couple weeks after his initial announcement, some affected people had applied the update and some unfortunately hadn’t.  There were certainly more details I’m skipping here, but that’s the skinny.

Now that the panel stage was set, here’s one of the topics I thought was interesting.

In our introductions, we each counted ourselves among the security research community.  Some of us had also been or still are consultants, all of us had done the startup thing, and some of us had been in charge of running some kind of computing infrastructure. 
At the risk of sounding immodest, I believe I had a unique perspective on the topic of responsible disclosure as I was the only panel member  who has been, at various stages in my career, a vulnerability Finder, Coordinator, and a Vendor (for both open and closed source software).

Let my official punditry from this pugnacious pulpit begin.  ;-)

It was interesting to me that the panelist who most strongly endorsed “inflicting pain” in the form of exploit release in order to provide the necessary “wake up call” to vendors had never been responsible for maintaining any kind of infrastructure deployment.  We all know how much easier it is to break something than it is to build it or to fix it, yet there is a pervasive attitude among many security researchers that nothing should be more important than security, not even the business itself.

And that’s where our disagreement’s footing took a stronger hold on its rocky purchase.  Define pain, our moderator asked.  Who decides on the form the pain will take and how intense or widespread it is, I asked.

Sure, it took some pain to get the attention of software vendors to fix their products and build security in from the ground up.  But as security folks, aren’t we tired of having to use the same arguments of active, widespread exploitation to*prove* that something needs to be done?  Security people often complain that not enough has changed since the epoch began, but if that’s true, then why have we not looked at ways to stop beating our heads upon the supposed brick wall of vendors or deployers of technology, and instead tried something different to get the right eyes on the right issues at the right time to do the right thing?  Doesn’t executing the same behavior over and over again but expecting different results equal insanity?  When are we willing to stop the madness? 

At the end of the panel we were each asked to describe our security utopia.  My Shangri-La was this:  I would like to see more cross-over among those of us who say the sky is falling and those of us upon whom the sky will fall. 

Communication between two groups with different mindsets requires a lingua franca other than exploitation.  One might think that math is the language of the universe, and Proof of Concept serves as the mathematical proof needed for anyone and everyone to arrive at the logical conclusion of “drop everything NOW and create (if you’re a vendor) or apply (if you’re managing infrastructure) the update.”  Before I had ever been responsible for building anything or protecting anything, I might have agreed with that, since it made perfect logical sense to me at the time, in the context within which I worked. 

But it’s not doing the trick of convincing all vendors and all deployers by a long shot, so obviously, we need something to change.  PoC can and should be part of the conversation between responsible researchers and people to whom they are reporting the issues, but it must be framed appropriately for the listener.  PoC is not that simple for non-security types to immediately frame the same way we do as security people.  Even if they do grok the severity of the situation, they may not be *able* to move as quickly as a researcher feels they should. 

Consider this, researcher-types: If you’ve never managed infrastructure before, or been responsible for shipping and maintaining complex and widely deployed code, then you don’t have the context to understand why there are sometimes legitimate reasons to do things more carefully and therefore more slowly.  Once the talk recordings are posted, check out the very thorough treatise by our own MSRC on How Microsoft Fixes Security Vulnerabilities: Everything you wanted to know about the MSRC Security Update Engineering Process. Think about how you as a researcher and security expert would react if some CTO or IT person or developer who lacks your depth of security knowledge and subject matter expertise came and told you what to hack, how to hack, and at what pace to hack it?  That’s essentially what you’re doing when you say “you should be able to fix it and fix it now, and if you don’t do it on my timeline, then you obviously need to be made an example of so I’m going to release an exploit for it into the wild.” 

They don’t swim in your security research toilet :-), so why must you pee in their development or infrastructure pool? 

Okay, I couldn’t resist making the joke – and no, I don’t think security research is a cesspool, or I wouldn’t have founded two vulnerability research programs in my career.  What I am saying is that all of us should be striving for the delicious harmony of combining your chocolate with my peanut butter, your gin and my tonic, your milk and my shake, in order to make the whole greater than the sum of its parts.  As a researcher, one can choose to be the sabot and grind gears to a halt to prove a point, or one can be the grease that moves things along with less friction, earning the trust that will allow each subsequent notification pill to be swallowed more easily.  As a developer or deployer, one can choose to stuff up one’s ears until someone firmly inserts an icepick, or one can strive to fix things as quickly and safely as possible and learn from the experience to continually improve and speed up that process over time.

We need a better way to reach our common ground of protecting the computing environment on which we all rely.  Researchers need a means by which to communicate urgency that avoids descriptive hyperbole or causes damage, which erodes trust.  Developers and deployers need a better way to service existing code and infrastructure reliably, safely, and rapidly if necessary, to build trust among the researcher and customer communities that they are doing the best they can at any given time.  Around here, we’ve done serious work on making this a reality on the development front, with the dual-ninjas of SDL (proactive) and MSRC (reactive).  I’d like to see SDL someday brought up to a full double-D in the form of a Secure Development and Deployment Lifecycle, to build infrastructure design and servicing models that are resilient in the face of threats to deployments as well as software.  Perhaps I can begin to work on this here at Microsoft, if I can get some of my other work done first.  :-)

After we had each said our peace on what our security utopia looked like, that’s where we left things.  No agreement could be reached in the two hours or so we were on the stage, which is no surprise.  If the tape ran out before the end, then you won’t get to see us literally “hug it out” after all was said and done, disagreements notwithstanding.  I continue to have tremendous respect and share camaraderie with my fellow panelists and with researchers around the world.  It is my hope that the determination and vision of those on any side of the equation who can see across the role boundaries of researcher, vendor, and deployer will usher us into a new age. 

People often ask what more is there to say about disclosure that hasn’t already all been said.  I think the real conversation on how to get the results we all desire - to get things fixed *in spite of* our disagreement - has yet to truly begin.

I’m listening, as well as talking.  Are you?

Want to know more about the evolving vulnerability disclosure landscape?  Have a burning question or opinion about who should get to know, how much they get to know, and when they get to know, as it relates to vulnerability details?  Can't make it to SOURCEBoston to see me and a few security industry friends "hug it out" during the Partial Disclosure Dilemma Panel (Thursday 1:30 PM - 3:45PM)? 

Come back here in a week or so to read my redux on the discussion that is sure to be lively.  My fellow panelists include Dan Kaminsky (IOActive), Ivan Arce (CORE Security), Dino Dai Zovi (recently published co-author of The Mac Hacker's Handbook), and Alex Sotirov (Independent security researcher), moderated by Ryan Naraine (Kaspersky/ZDNet). 

Over two hours on the stage -- should be long enough to get a real dialogue going.  ;-)

-Katie Moussouris, Senior Security Strategist

Follow me on Twitter: http://twitter.com/k8em0