The Security Development Lifecycle : threat modeling

"The Threats to Our Products"

sdl — Fri, 28 Aug 2009 00:48:00 GMT

Adam here.

I’ve learned to love STRIDE as a framework for thinking about threats, but it makes a lousy classification system. That is, I can look at a system to find information disclosure threats, but once I have an attack that leaks, say, the location of a DLL in memory to a remote attacker, is that Information Disclosure or Elevation of Privilege? It’s likely to make the latter easy, and down the classification rat-hole we go. I’d like to suggest that calling it ‘the STRIDE framework is the most clear wording we can use.’

I found the 1999 internal Microsoft article written by (then) Microsoft employees Loren Kohnfelder and Praerit Garg, and was pleased to discover that they intended STRIDE “to help you identify potential vulnerabilities in your product during a security analysis.” They also make repeated reference to a “proactive security analysis process,” and shows some of the thinking that led to the SDL. I thought it was neat look into history, and so without further ado, it's attached. (59KB .docx)

Experiences Threat Modeling At Microsoft

sdl — Wed, 08 Oct 2008 21:12:00 GMT

Adam Shostack here. Last weekend, I was at a Security Modeling Workshop, where I presented a paper on “Experiences Threat Modeling at Microsoft,” which readers of this blog might enjoy. So please, enjoy!

And while I’m at it, I wanted to draw attention to some of the other presentations that I thought were very interesting, including one by Karine Peralta “Specifying Security Aspects in UML Models” and “Curriculum for Modelling Security: Experiences and Lessons Learned.”

What do you want to know about SDL threat modeling?

sdl — Fri, 01 Aug 2008 03:27:00 GMT

Adam Shostack here. I'm working on a paper about "Experiences Threat Modeling at Microsoft" for an academic workshop on security modeling. I have some content that I think is pretty good, but I realize that I don't know all the questions that readers might have.

So, what questions should I try to answer in such a paper? What would you like to know about? No promises that I'll have anything intelligent to say, but I'd love to know the questions you're asking. So please. Ask away!

Security Thoughts from TechEd 2008

sdl — Thu, 26 Jun 2008 18:07:00 GMT

Hi, this week is a post from Michael Howard and Laura Machado de Wright, who both attended and presented at TechEd 2008 in Orlando the week of June 2^nd.

First up is Laura.

I have been a Security Program Manager for the last 3 years, working as a security advisor for a variety of products across Microsoft and the last seven months as a member of the SDL policy team.

It's been a few years since I've been to TechEd, and this was my first time attending as a member of the security team. TechEd is now a two week conference, with one week dedicated to developers and the other to IT professionals. I think that breaking down the conference into a Developer week and an ITPro week was a good idea, and it allowed us to have good conversations with people who wanted more information about the SDL. I did two main things at TechEd:, I presented on threat modeling, and I spent a lot of time talking to customers at the SDL booth. At the SDL booth, we heard questions ranging from "What does the SDL stand for?" to "Our Web site was hacked; how do I stop it from happening again?" It was encouraging hearing people interested to hear more specifics about how we implement the SDL at Microsoft, and thinking through how they can apply it in their own companies. My understanding from other TechEd veterans in our booth is that interest in the SDL seemed higher, which is great.

During my Threat Modeling session, , most of the feedback and follow-up questions were similar to the ones in the booth: how to expand the threat modeling processes to their own companies, and how to get started.

My typical response to both questions is to start small and do what makes sense for your organization. At Microsoft, for example, when we introduce new SDL requirements, we usually start with a few teams so we can refine the requirement and supporting tools before expanding the requirements to a broader group. Similarly, while we have a core set of requirements that all teams have to meet, there are other requirements that are specific to a platform, scenario, or functionality. For example, there are some requirements that make sense for desktop-oriented products, but do not make sense for mobile devices. You may very likely have to make changes to our policies to make them relevant to your organization, your scenarios, and functionality.

Now over to Michael.

Hi, Michael here.

One of the joys of presenting at TechEd each year is hearing from real people about the issues they face using our products in the real world; rarely are the issues pure philosophical security geekness. This year I gave two talks and one "chalk talk." The talks were "Top Ten Strategies
To Secure Your Code" and "How To Review Your Code
and Test For Security Bugs", and the chalk talk, which was a lot of fun, was simply answering numerous developer questions.

It's interesting to gauge overall security awareness from our customers, and there is no doubt that over the years, the level of security knowledge and maturity has risen. I think it's possible to evaluate overall security maturity by the questions posed. Some years ago, security was never really a topic of discussion other than those that relate to security technologies, such as how to use and manage X.509 certificates. About four years ago the tide really changed and people started asking more questions about "secure" application deployment and management, and developers wanted to learn more about securing their code; especially C and C++ code. Even then there was still a reliance on exterior defenses like firewalls. All too often I would hear people claim that they don't need to focus on securing their apps because a firewall was in the way. Heck, David and I documented this excuse in the original version of Writing Secure Code (Appendix D, "Lame Excuses We've Heard, #6, ‘We're Secure-we use a Firewall'") way back in 2002.

Fast forward to 2008.

Things have obviously changed. I don't know if finally the security message is getting through because many people asked me highly specific questions about securing their apps and how best to use the defenses we offer in Windows Vista and Windows Server 2008.

I still hear the firewall excuse a little, but not too much!

Perhaps the most telling trend I saw this year was a great deal of interest in the SDL. Not cursory, "that looks interesting" interest, but, "how can I implement this in my company" interest. After answering specific questions, I pointed most folks to Jeremy's "Crawling Toward SDL" post on the subject.

In my opinion, getting to a point where you want to change your development process shows you really understand there's an issue that needs fixing.

And that's goodness.

SDL Threat Modeling: Past, Present and Future

sdl — Wed, 18 Jun 2008 00:59:50 GMT

Adam Shostack here.

I wanted to share my slides from the recent Layer One conference [link], where I talked about "SDL Threat Modeling: Past, Present and Future."

There are a few points that I wanted to emphasize. The first is that I'm talking about threat modeling from the perspective of the SDL. We have other threat modeling processes here at Microsoft, and we're working to bring you more clarity in how we speak about them. For my part, I'll try to clearly say "SDL threat modeling," or be explicit when I'm talking about threat modeling in broad terms.

Which brings me to my second point, and a slide I wanted to emphasize. (Shown here)

I no longer think of threat modeling as one thing. I see it as a label for a set of ways to address the question of "what could go wrong" with a design or set of requirements. The SDL has one process. The folks in ACE and Patterns and Practices each have another. All are customized to meet various needs. Much like we have lots of programming languages which address different problems, we're going to have lots of threat modeling processes.

Anyway, I hope you enjoy the slides.

SDL Threat Modeling @ ToorCon

sdl — Fri, 25 Apr 2008 02:30:00 GMT

Adam Shostack here. I spoke at Toorcon this past weekend on "SDL Threat Modeling: Past, Present and Future." I wanted to share my slides to help clarify a bit about where SDL threat modeling is and why, and a bit about where we're going.

(Click on the post title, and you'll see an attachment in the per-post page.)

Training People on Threat Modeling

sdl — Sat, 15 Mar 2008 02:11:12 GMT

Adam Shostack here. Blogger Ian Grigg has an interesting response to my threat modeling blog series, and I wanted to respond to it. In particular, Ian says “I then would prefer to see the threat - property matrix this way:”

I wanted to share an additional table from our training, and talk about repudiation a bit more.

Actually, I’d like to repudiate the term “repudiation.” It’s an awful name that most people never run into in day-to-day life. It doesn’t hit the simplification bar the way say, “denial,” would. Unfortunately, STDIDE (Spoofing, Tampering, Denial, Information Disclosure, Denial of Service, Elevation of Privilege) doesn’t make a very memorable acronym. Memorable is important when training people. Our reviewers have raised this as an issue, and ’d love to get feedback from our readers. How can we ensure that the software we build has the right level of logging and audit-ability? What evocative words can we use, and can you help us come up with a word or phrase that starts with R? Let us know!

And then, here’s the chart:

Threat	Property	Definition	Example
Spoofing	Authentication	Impersonating something or someone else.	Pretending to be any of billg, microsoft.com or ntdll.dll
Tampering	Integrity	Modifying data or code	Modifying a DLL on disk or DVD, or a packet as it traverses the LAN.
Repudiation	Non-repudiation	Claiming to have not performed an action.	“I didn’t send that email,” “I didn’t modify that file,” “I certainly didn’t visit that web site, dear!”
Information Disclosure	Confidentiality	Exposing information to someone not authorized to see it	Allowing someone to read the Windows source code; publishing a list of customers to a web site.
Denial of Service	Availability	Deny or degrade service to users	Crashing Windows or a web site, sending a packet and absorbing seconds of CPU time, or routing packets into a black hole.
Elevation of Privilege	Authorization	Gain capabilities without proper authorization	Allowing a remote internet user to run commands is the classic example, but going from a limited user to admin is also EoP.

(Ian’s post is here https://financialcryptography.com/mt/archives/001013.html . IE users will see a warning about certificate authorities when visiting this site. As I wrote this, Gunnar Peterson added commentary at "Threats, Mechanisms and Standards.")

Wrapping up Threat Modeling

sdl — Fri, 15 Feb 2008 01:51:35 GMT

One of the critiques of the threat modeling ~~blog posts~~ process is that it can seem interminable. And so, in this final post, I’d like to offer up some final thoughts on language, and cognitive load.

Specification versus Analysis

When Larry Osterman was writing about threat modeling, he casually tossed out:

A threat model is a specification, just like your functional specification (a Program Management spec that defines the functional requirements of your component), your design specification (a development spec that defines the architecture that is required to implement the functional specification), and your test plan (a test spec that defines how you plan on ensuring that the design as implemented meets the requirements of the functional specification).

Just like the functional, design and test specs, a threat model is a living document - as you change the design, you need to go back and update your threat model to see if any new threats have arisen since you started.

I found this pretty surprising. I think of threat modeling as an analysis technique, but hey, if test plans are specs, threat models aren’t that different. (It took some private back and forth with Larry to convince me.) This brings me to the topic of what words we use to describe things.

On language

[The English language] becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts. The point is that the process is reversible. Modern English, especially written English, is full of bad habits which spread by imitation and which can be avoided if one is willing to take the necessary trouble. If one gets rid of these habits one can think more clearly, and to think clearly is a necessary first step toward political regeneration: so that the fight against bad English is not frivolous and is not the exclusive concern of professional writers. (George Orwell, Politics and the English Language.)

We ask people to threat model (as part of the SDL) by threat modeling their application. To do that, they model the design, and then have a threat modeling meeting in which someone runs the threat modeling tool to produce a threat modeling document. In our particular case, I’m not sure it’s easily avoided.

“We ask people to analyze their designs (as part of the SDL). To do that, they diagram the design, then have a threat modeling meeting in which someone runs the codename tool to produce a threat modeling document.”

In fact, I’ve worked hard to reduce jargon in the process, but at one point, went too far. I tried to convince people to say “diagram” instead of DFD, and they revolted. Actually, they didn’t really revolt, but they did refuse to go along. Experienced threat modelers actually felt that the term DFD was more clear than saying ‘diagram,’ because DFD includes a specification of what kind of diagram, and they had already integrated the acronym into their thinking. Having integrated it, they could use it without thinking about it. Well, as Adam Barr points out, “you can pick your friends, and you can pick your nose, but you can't pick your online community's lexicon.”

It didn’t add to their cognitive load to say “DFD.”

Cognitive Load

People can only handle so much new information at one time. Learning to drive a stick shift is harder than learning on an automatic, because there’s a whole set of tasks you can ignore while the automatic transmission handles them for you. In my approach, this relates pretty closely to the concept of flow. If you’re so focused on rules and jargon, you can’t focus on what you should be building. Cool, well-designed features to help your customers.

In Conclusion

We started this series by looking at the trouble with threat modeling. How people had trouble getting started, relating the process to their other work, or validating what they’d done. We’ve looked at the new threat modeling process, and the steps involved. We talked about how to get into the flow with threat modeling. How clear goals, direct and immediate feedback, and balancing ability and challenges set people up to focus their attention and succeed. We’ve talked a bit about making threat modeling work better by adding structure to chaos, and how to use self-checks and rules of thumb to give people confidence they’re on the right trail. We’ve talked a very little bit about how to customize the process for your own needs, and where that customization can be dangerous.

All of this has come out of looking at our threat modeling activity as a human activity, and asking how we can best shape it to help people get to the results that we want. I hope that our readers have enjoyed the series, which we’ve converted into a downloadable document (here.)

Threat Modeling Self Checks and Rules of Thumb

sdl — Tue, 23 Oct 2007 00:04:01 GMT

Adam again. I hope you’re still enjoying this as we hit #5 in the threat modeling series.

In my last post, I talked about how almost everyone in software draws on whiteboards regularly, and this makes it an ideal first step. It’s an ideal first step because everyone can do it, see that they’ve done it, and feel like they’re making progress.

That wasn’t quite complete. Not only do we want people to see that they’ve done it, we want to give them a way to validate their work or other people’s work. So we ask them to tell a story. We’re not asking for Shakespeare here, we’re asking them to explain how their software will be used, and to make sure that their diagram supports that story, and that it relates to their actual software.

We also give them rules of thumb (lots of rules of thumb) about things we often see wrong in diagrams:

Don't have data sinks: you write the data for a reason. Show who uses it.
Data can’t move itself from one data store to another: show the process that moves it.

(Larry Osterman has some in his blog post, "Threat Modeling Rules of Thumb" I helped edit those, but want to suggest additional changes. In particular, “you need to be concerned” is not actionable. “Review this carefully,” or “Focus your attention here” are more actionable. People threat modeling are already concerned.)

Good “rules of thumb” encourage flow by empowering people to make a snap decision and move along.

Making Threat Modeling Work Better

sdl — Wed, 17 Oct 2007 03:23:53 GMT

Adam Shostack here, with part four of my threat modeling series. This post is a little less philosophical and a lot more prescriptive than the one about flow. It explains exactly how and why I changed a couple of elements of the process. The first is the brainstorming meeting, and the second is the way trust boundaries may be placed.

The brainstorming meeting is a mainstay of expert threat modeling. It’s pretty simple: you put your security experts in a room with system diagrams and a whiteboard. Usually, you put your system designers in there, and make them promise not to strangle your experts. Optionally, you can add beer or scotch. Sometime later, you get a list of threats. How long depends on how big the system is, how well its requirements are documented, and how well your experts work together.

We like having our developers threat model. There are a lot of reasons for this. Not only do they know the system better than anyone else, but getting people involved in a process helps ensure that they buy into it.

Now this desire is great, but it leads to some issues, first and foremost is that many of the people who are now involved aren’t security experts. This means that they lack both direct experience of the process and the background that informs it. This isn’t a slam on them. I lack experience in the database design process, and I don’t have years of experience to help orient me. So I’d make mistakes designing a database, and someone who isn’t a security expert may make mistakes in security. For example, someone might try to use encryption to mitigate tampering threats. (The SDL crypto requirements cover this, and I try to gently correct them to integrity mechanisms like MACs or signatures.) This is a reality that we have to account for at the process design level.

Adding Structure to Chaos

So how does this relate to the brainstorming meeting? It’s a dramatic increase in the need for structure. Where experts may think they do better threat modeling with scotch in hand, , it certainly doesn’t lead to beginners having a flow experience. So we need a structure, and we need to provide it.

We encourage people to get started by drawing a whiteboard diagram. Almost everyone in software draws on whiteboards regularly, and this makes it an ideal first step. It’s an ideal first step because everyone can do it, see that they’ve done it, and feel like they’re making progress.

The core mechanism we’ve used to provide it is the STRIDE/element chart. (I’ll talk a lot more about its origins and limits in a few posts, but for now, let’s pretend it’s gospel, and enumerates all possible threats.) Given this gospel, it becomes possible to step through the threat modeling diagram, “turn the crank,” and have threats come out. “Item 7 is a data flow? Let’s look for T,I and D.” (Tampering, Information disclosure, and Denial of service.)

Similarly, we have four ways of addressing threats – redesign, standard mitigations, new mitigations, and risk acceptance. We have training on mitigating threats, we have explanation of why and when to use each (and they’re presented in a preferred order).

Lastly, we provide advice about how to validate the threat model and it’s relation to reality.

Between these four steps and the hamster wheel which ties them together, we give people the structure in which they can take on the process. The other thing I wanted to address is how we respond to consistent “errors” that we see.

Where Trust Boundaries Show Up

We used to give people clear guidance that trust boundaries should only intersect with data flows. After all, you can’t really have a process that’s half-running as admin, and half as a normal user. Logically, you have two entities. And people kept drawing trust boundaries across processes and data stores. It drove me up the wall. It was wrong.

As people kept doing it, I decided to swallow my pride and accept it. I now tell people to put their trust boundaries wherever they believe one exists. And they’ve continued exactly as before, but I’m a lot happier, because I’ve found a way to help them draw more detailed diagrams where they need them. Which includes anywhere a trust boundary crosses a process or data store. They’re happier too. No one is telling them that they’re wrong.

I was going to title this post “Lord grant me the strength to change the things I can, the courage to accept what I can’t, and the wisdom to know the difference,” but, first, it’s too long, and second, if we started that way, it would be wrong to add beer or scotch.

Getting into the Flow With Threat Modeling

sdl — Fri, 12 Oct 2007 02:25:00 GMT

Adam Shostack again, with the third in our series on threat modeling. In this post, I want to explain one of the ‘lenses’ that seemed to help us focus threat modeling, and how I’ve applied it.

The concept of flow originated with Mihaly Csikszentmihalyi. It refers to a state where people are energetically involved with what they’re doing. Seeing this a few times during threat modeling sessions made it obvious when it was missing, and it was missing often. I set out to address some of the elements that seemed to make threat modeling harder. The Wikipedia article (currently) has a good list, so I’ll focus in on a few of them:

Clear goals
Direct and immediate feedback
Balance between ability and challenge
Focused attention

Let’s take these one at a time.

Clear Goals

Giving people clear goals is important because it helps take them from worrying about what your goals mean to worrying about how to achieve them. Without clear goals, it’s very challenging to get into the spirit of anything, whether playing a game or shipping an operating system. As goals go, “think like an attacker” and “brainstorm” aren’t up there with “A PC on every desktop.” The lack of a clear answer to “how do I know I’m done with my diagrams” or “how many threats should I have,” or “what is a good threat model” made it hard for people to do well, and just as importantly, even people who were doing well weren’t sure that they were doing well. Which brings us directly to the next point:

Direct and Immediate Feedback

When people have direct and immediate feedback, they can adjust their activities relatively easily. If you’re cooking from a well written recipe, at each stage, you have some indicators that things are done. “Are the onions brown?” “Is the water boiling?” “Does the pasta stick to the wall?” This is true of any skill we’re learning. With threat modeling, when there is no expert in the room, feedback can take weeks or months to come back. As we rolled threat modeling out at Microsoft, it was possible for an entire threat model to be cooked without any course correction. We’re doing better now - there are more experienced people around, better documentation, and more focus on how to self-check. But at one time we asked people to engage in really complex tasks from the get go, without trying to achieve a…

Balance Between Ability and Challenge

Almost by definition, when you’re new at something, you don’t have a lot of skill around it. If the new activity is something highly complex, like speaking a foreign language, you’re likely to be anxious. And once you are skilled, simple tasks, like talking to a beginner, are often boring. Which brings us to the chart, which I’ve redrawn from Csikszentmihalyi’s book, Flow.

The ideal learning experiences involve a balance of challenge and reward, each of which grows as you learn. For example, when you buy Halo, there’s a training mission that allows you to run around doing simple tasks, learning the controls. If we dropped people into the live online game without any practice, they’d get slaughtered, and that’s bad for everyone. The newb is frustrated, and the experienced player is bored. (In fact, there’s pretty explicit acknowledgement of this in the Wired article on Halo.)

So looking at the chart of skills versus challenge, there’s a channel in the middle—it’s the flow channel. We’d like to guide people there, by giving them challenges that are in line with their skills. Once people are there, we want to prod them from point A to point D, where there are larger challenges, and skills that allow people to address those challenges. Sometimes, we want them to leave their comfort zone and learn new things. To take on new challenges (going to B) and learn from them, going to D. If we don’t challenge people, they get bored, and it’s less likely that they’ll learn. (This obviously isn’t a complete model of learning, but it flows out of the discussion.)

We want to do that for Halo, and we want to do that for threat modeling. To date, this has been implicit. We’ve simplified the process, in part in the interest of getting people out of the anxiety zone, and into a place where they can feel comfortable and then learn. Over time, we’ll be challenging and encouraging people to do more complex tasks as part of SDL threat modeling.

This assumes that threat modeling is a frequent enough activity that people develop skills around it. I think that’s an ok assumption, as long as we design good refresher and update training and documentation.

Focused Attention

If you’re reading this blog post while talking on the phone and petting your dog, you’re going to get less out of it than if you do nothing else while reading it. Similarly, when you’re learning a new skill or performing a hard task, you need to focus your attention. Related to this is moving around. Getting a little blood flow helps you be energetic and engaged. This has been one of the hardest elements of flow to add to threat modeling. Many people here multi-task intensively, with laptops or blackberries always at hand. Challenging that habit seems like a challenge to the way people work, and I’ve shied away from issuing that challenge.

Why flow?

One final point I’d like to address is that flow is one of many frameworks we could use to address problems in threat modeling. My choice to use it was a driven by its striking absence. Flow isn’t the only framework for thinking about these things, and there are some criticisms associated with trying to shoehorn everything into flow. For threat modeling, it just seemed to well…flow.

The New Threat Modeling Process

sdl — Tue, 02 Oct 2007 04:15:35 GMT

Adam Shostack here, with the second post in my series on the evolved threat modeling process. To summarize, what I’ve tried to achieve in changing the process is to simplify, prescribe, and offer self-checks. I’ll talk in the next post about why those three elements are so important to me. For now, let me describe the process.

One of the largest changes that we’ve made is to a simplified process (and diagram). I like to say that this looks pretty much like every other software process diagram you see today. That’s intentional. There’s only so much we can expect people to take away from a class, and making this simple and familiar helps ensure there’s room for the other important parts.

First, the “process hamster wheel,” (with apologies to Yankee Group analyst Andy Jaquith):

Now that you’ve seen the wheel, I’ll briefly describe the steps:

Vision: Consider your security requirements, scenarios and use cases to help frame your threat modeling. What are the security aspects of your scenarios? What do your personas expect or hope doesn’t happen? What are the security goals of the system you’re building, and how do those interact with the system as it stands?
Model: The basic idea is to create a diagram of your software, showing all trust boundaries.

a. Draw a diagram of your software. We encourage use of the DFD formalisms, which Larry Osterman describes in this post.

Essentially, the elements are

External entities (anything outside your control)

Processes (running code)

Data stores (files, registry entries, shared memory, databases)

Data flows (which connect all the other elements)

b. Draw trust boundaries between components. You can do this on a whiteboard, in Visio, or in one of the specialized threat modeling tools we’ve built. (A trust boundary is anywhere that more than one principal can access an object, such as a file or process.)

c. If your trust boundary crosses something which isn’t a data flow, you need to break it into two logical elements, or draw a sub-diagram with more details. (This is different advice: we used to tell people trust boundaries could only cross data flows. People drew them anywhere that felt right. We decided to go with what people were doing—there was important information in what they were expressing.)

d. If you need more details to express where trust boundaries are, add an additional diagram.

e. When you don’t have any more trust boundaries to draw, you’re done.

f. If a diagram doesn’t have any trust boundaries, you may have drawn too many details.

3. Identify Threats using STRIDE/element

For each element in your diagram, consider threats of the types indicated in this chart. (We’ll come back to the chart’s origins in a later post.)

There’s an important mis-conception we often see, which is that STRIDE is appropriate for use as a classification system. It’s really hard to use STRIDE to describe attacks—the impacts blend together really quickly. The most valuable use of STRIDE is to help people think about how threats have impacted elements of a design in the past. That is, it’s a framework for finding threats, not for describing them. “What if someone spoofs this host?”

4. Mitigate

Here on the SDL strategy team, we love threat modeling. We know that not everyone feels that way, and we ask teams to threat model so that they can find and mitigate threats. A threat model document with great diagrams and lots of threats isn’t very useful if you don’t take the key step of addressing the issues you find. There are four ways to do that:

a. Redesign to eliminate threats.

b. Use standard mitigations, such as those provided by OS features, to mitigate threats.

c. Invent new mitigations, understanding that this is a subtle art.

d. Accept risk, when allowed by the SDL

5. Validate

There are two levels of validation. The first is within each stage, the second is a validation pass at the end of the process. That end of process validation entails:

a. Make sure that the diagrams are up-to-date and accurate

b. Ensure that you have STRIDE threats per data flow that crosses a trust boundary, and for each element that such a trust boundary connects to

c. Make sure you’re mitigating each threat

i. You have a bug filed per threat that you want to mitigate. The bug should be of the form “attacker can do X. Proposed fix: Y.” You might include tradeoffs you’re making, and possibly have test plans in the bug, if you include those.

ii. You have a valid reason for each non-mitigated threat not being mitigated.

iii. All threats are in class i or ii.

5.a. On change, re-validate

This hamster wheel has a very intentional brake on it: the word change, above validate. What that means is you want to go through the process again when you make changes that need to be on the diagram. Checking to see if your diagrams change is a relatively simple check that allows people to track changes against their threat model as their design iterates.

In the next post, I’ll talk about the reasoning behind the design, and offer up some process tools that anyone can use to make a process more user-friendly.

The Trouble with Threat Modeling

sdl — Wed, 26 Sep 2007 22:11:00 GMT

Adam Shostack here.

I said recently that I wanted to talk more about what I do. The core of what I do is help Microsoft’s product teams analyze the security of their designs by threat modeling. So I’m very concerned about how well we threat model, and how to help folks I work with do it better. I’d like to start that by talking about some of the things that make the design analysis process difficult, then what we’ve done to address those things. As each team starts a new product cycle, they have to decide how much time to spend on the tasks that are involved in security. There’s competition for the time and attention of various people within a product team. Human nature is that if a process is easy or rewarding, people will spend time on it. If it’s not, they’ll do as little of it as they can get away with. So the process evolves, because, unlike Dr No, we want to be aligned with what our product groups and customers want

There have been a lot of variants of things called “threat modeling processes” at Microsoft, and a lot more in the wide world. People sometimes want to argue because they think Microsoft uses the term “threat modeling” differently than the rest of the world. This is only a little accurate. There is a community which uses questions like “what’s your threat model” to mean “which attackers are you trying to stop?” Microsoft uses threat model to mean “which attacks are you trying to stop?” There are other communities whose use is more like ours. In this paragraph, I’m attempting to mitigate a denial of service threat, where prescriptivists try to drag us into a long discussion of how we’re using words.) The processes I’m critiquing here are the versions of threat modeling that are presented in Writing Secure Code, Threat Modeling, and The Security Development Lifecycle books.

In this first post of a series on threat modeling, I’m going to talk a lot about problems we had in the past. In the next posts, I’ll talk about what the process looks like today, and why we’ve made the changes we’ve made. I want to be really clear that I’m not critiquing the people who have been threat modeling, or their work. A lot of people have put a tremendous amount of work in, and gotten some good results. There are all sorts of issues that our customers will never experience because of that work. I am critiquing the processes, saying we can do better, in places we are doing better, and I intend to ensure we continue to do better.

We ask feature teams to participate in threat modeling, rather than having a central team of security experts develop threat models. There’s a large trade-off associated with this choice. The benefit is that everyone thinks about security early. The cost is that we have to be very prescriptive in how we advise people to approach the problem. Some people are great at “think like an attacker,” but others have trouble. Even for the people who are good at it, putting a process in place is great for coverage, assurance and reproducibility. But the experts don’t expose the cracks in a process in the same way as asking everyone to participate.

Getting Started

The first problem with ‘the threat modeling process’ is that there are a lot of processes. People, eager to threat model, had a number of TM processes to choose from, which led to confusion. If you’re a security expert, you might be able to select the right process. If you’re not, judging and analyzing the processes might be a lot like analyzing cancer treatments. Drugs? Radiation? Surgery? It’s scary, complex, and the wrong choice might lead to a lot of unnecessary pain. You want expert advice, and you want the experts to agree.

Most of the threat modeling processes previously taught at Microsoft were long and complex, having as many as 11 steps. That’s a lot of steps to remember. There are steps which are much easier if you’re an expert who understands the process. For example, ‘asset enumeration.’ Let’s say you’re threat modeling the GDI graphics library. What are the assets that GDI owns? A security expert might be able to answer the question, but anyone else will come to a screeching halt, and be unable to judge if they can skip this step and come back to it. (I’ll come back to the effects of this in a later post.)

I wasn’t around when the processes were created, and I don’t think there’s a lot of value in digging deeply into precisely how it got where it is. I believe the core issue is that people tried to bring proven techniques to a large audience, and didn’t catch some of the problems as the audience changed from experts to novices.

The final problem people ran into as they tried to get started was an overload of jargon, and terms imported from security. We toss around terms like repudiation as if everyone should know what it means, and sometimes implied they’re stupid if they don’t. (Repudiation is claiming that you didn’t do something. For example, “I didn’t write that email!,” “I don’t know what got into me last night!” You can repudiate something you really did, and you can repudiate something you didn’t do.) Using jargon sent several unfortunate messages:

This is a process for experts only
You’re not an expert
You can tune out now
We don't really expect you to do this well

Of course, that wasn’t the intent, but it often was the effect.

The Disconnected Process

Another set of problems is that threat modeling can feel disconnected from the development process. The extreme programming folks are fond of only doing what they need to do to ship, and Microsoft shipped code without threat models for a long time. The further something is from the process of building code, the less likely it is to be complete and up to date. That problem was made worse because there weren’t a lot of people who would say “let me see the threat model for that.” So there wasn’t a lot of pressure to keep threat models up to date, even if teams had done a good job up front with them. There may be more pressure with other specs which are used by a broader set of people during development.

Validation

Once a team had started threat modeling, they had trouble knowing if they were doing a good job. Had they done enough? Was their threat model a good representation of the work they had done, or were planning to do? When we asked people to draw diagrams, we didn’t tell them when they could stop, or what details didn’t matter. When we asked them to brainstorm about threats, we didn’t guide them as to how many they should find. When they found threats, what were they supposed to do about them? This was easier when there was an expert in the room to provide advice on how to mitigate the threat effectively. How should they track them? Threats aren’t quite bugs—you can never remove a threat, only mitigate it. So perhaps it didn’t make sense to track them like that, but that left threats in a limbo.

"Return on Investment"

The time invested often didn’t seem like it was paying off. Sometimes it really didn’t pay off. (David LeBlanc makes this point forcefully in “Threat Modeling the Bold Button is Boring”) Sometimes it just felt that way—Larry Osterman made that point, unintentionally in “Threat Modeling Again, Presenting the PlaySound Threat Model,” where he said “Let's look at a slightly more interesting case where threat modeling exposes an issue.” Youch! But as I wrote in a comment on that post, “What you've been doing here is walking through a lot of possibilities. Some of those turn out to be uninteresting, and we learn something. Others (as we've discussed in email) were pretty clearly uninteresting” It can be important to walk through those possibilities so we know they’re uninteresting. Of course, we’d like to reduce the time it takes to look at each uninteresting issue.

Other Problems

Larry Osterman lays out some other reasons threat modeling is hard in a blog post: http://blogs.msdn.com/larryosterman/archive/2007/08/30/threat-modeling-once-again.aspx

One thing that was realized very early on is that our early efforts at threat modeling were quite ad-hoc. We sat in a room and said "Hmm, what might the bad guys do to attack our product?" It turns out that this isn't actually a BAD way of going about threat modeling, and if that's all you do, you're way better off than you were if you'd done nothing.

Why doesn't it work? There are a couple of reasons:

It takes a special mindset to think like a bad guy. Not everyone can switch into that mindset. For instance, I can't think of the number of times I had to tell developers on my team "It doesn't matter that you've checked the value on the client, you still need to check it on the server because the client that's talking to your server might not be your code.".

Developers tend to think in terms of what a customer needs. But many times, the things that make things really cool for a customer provide a superhighway for the bad guy to attack your code.

It's ad-hoc. Microsoft asks every single developer and program manager to threat model (because they're the ones who know what the code is doing). Unfortunately that means that they're not experts on threat modeling. Providing structure helps avoid mistakes.

With all these problems, we still threat model, because it pays dividends. In the next posts, I’ll talk about what we’ve done to improve things, what the process looks like now, and perhaps a bit about what it might look like either in the future, or adopted by other organizations.