Software Engineering, Project Management, and Effectiveness
"All models are wrong, but some are useful." -- George Box
I often see confusion over reference models, reference architectures, and reference implementations. In this post, I’ll share my experience and learnings and observations:
I think the confusion is usually because the argument usually hinges on whether a reference architecture or reference implementation needs to be in code, when that’s really just a form factor decision. The distinction is actually the focus and concern, independent of how you share it, although code can help illuminate a reference implementation and a reference architecture. The other confusion is often around how big or small it needs to be to determine whether it’s a reference architecture or reference implementation, but that’ also a herring. Again, it goes back to what the focus of the example is. If it’s an exemplar of the architecture, it’s a reference architecture. If it’s an exemplar of the implementation, then it’s a reference implementation, and each serve different purposes, and require different levels of detail or abstraction.
Reference Model Examples Reference models give us a quick way to see and talk about a given problem space. Here is an example of a simple reference model that does nothing more than frame out some cornerstone concepts for cloud computing. In this case, it’s a visual model for cloud computing:
This is another of my favorite reference models. In this case, this is a continuum of moving up the stack with types of cloud services from IaaS to PaaS to SaaS :
I also like a few of the cloud reference models, especially how they can be used for overlaying security concerns.
Reference Architecture Examples The beauty of the reference architecture is that you can shape the application before you implement it. One of the simplest ways to achieve this is to whiteboard it. When you put it on the whiteboard, you can focus on key engineering decisions and avoid getting lost in the details. You can focus on a conceptual, physical, or logical view, … you can focus on the business perspective, user perspective, or technical perspective … and you can move up and down the stack to zoom in or zoom out … but the main point is to show how the bits and pieces should fit together. The power of a reference architecture is that it creates a shared mental model for the architecture and design and it can help you identify the key decisions and key risks early on. This helps you both shape a skeleton solution as well as identify what you need to prototype, especially in terms of cross-cutting concerns, or tracer-bullets. From an agile perspective, the Reference Architecture complements the system metaphor and it helps you identify potential areas to spike on. Here are a few examples …
One of my favorite reference architecture examples is the reference architecture from the ESB Toolkit.
Another of my favorite reference architectures is the reference architecture for the PetShop Sample Application:
One approach I like to use for sharing reference architectures is what I call Application Scenarios or Scenarios and Solutions. It’s a whiteboard sketch of the end-to-end solution. Here is an example:
Another of my favorite approaches is to use what I refer to as Application Patterns. It’s like the Application Scenarios, but I overlay patterns on top. Here is an example:
The real key of reference architectures is that they help you explore the thinking and design at a higher-level before getting lost in the weeds. Getting lost in the weeds is a common problem with reference implementations. That’s why it’s helpful when they include a reference to their corresponding reference architecture.
The best combinations I find for nailing a problem space include a reference model + reference architecture + reference implementation.
One of my new favorite books is, Enterprise Architecture as Strategy, by Jeanee W. Ross, Peter Weill, and David C. Robertson. My colleague, Danny Cohen recommended it as one of the best books on enterprise strategy and I agree.
A big focus of the book is on execution and how to build a strong foundation for execution. The key is an effective enterprise architecture. With an effective enterprise architecture, you can improve business agility, achieve higher profitability, reduce time to market, lower IT costs, improve access to shared customer data, lower the risk of mission-critical systems failures. Enterprise architecture as strategy means your enterprise architecture is addressing two key challenges: 1) integration and 2) standardization across the company.
Six Steps to Build a Foundation for Execution Here are six steps Ross, Weill, and Robertson identify for re-thinking a foundation for execution and creating and effective enterprise architecture:
Summary Table and Notes In this table, I summarized key notes along with each step:
“Life is like skiing. Just like skiing, the goal is not to get to the bottom of the hill. It’s to have a bunch of good runs before the sun sets.” – Seth Godin
It's been a good run. After more than 10 years in patterns & practices, I'm on to my next adventure here at Microsoft.
For this post, I wanted to take a stroll down memory lane. During my time at patterns & practices, I learned more about project management, application architecture, software development, processes, teamwork, leadership, product success, and making impact than I think I could have ever hoped to learn anywhere else. Perhaps the best part, is that I have 10+ years of deliberate practice in the art of “best practices” and I’ve had a chance to work with some of the most amazing people in the industry, lead SWAT teams around the world on fantastic voyages and epic adventures, and leave a legacy of several high-impact Microsoft Blue Books, all while mastering the art of prescriptive guidance.
There are a few ways I can share my journey in patterns & practices. One of my managers referred to me as “the abilities PM” because of my focus on quality attributes (security, performance, scalability, etc.) and that’s a pretty good place to start. My role as a PM (Program Manager) in patterns & practices, can largely be defined in terms of my impact on these three areas: security, performance, and application architecture. (Yes, there is more to the story, but those three areas, serve as a good enough lens for now.)
The Security Story It all started when I joined patterns & practices (PAG or the Prescriptive Architecture Guidance team at the time.)
I moved from Microsoft Developer Support with the expectation that I would share and scale broadly with customers what I learned from living in across the hall from folks like Scott Guthrie, Brad Abrams, and other .NET Framework leaders. Having performed more than 650 customer architecture and design reviews, I was in a unique position to share all the principles, patterns, practices, and anti-patterns that I had seen across a large number of customer projects, many of which were on the leading and bleeding edge of this space.
But life has a way of throwing curve balls. Around this time, security was Jim Allchin’s top priority, and he asked patterns & practices what we were going to do about security. In turn, my manager asked me, what I was going to do about security. Coming from a middleware background, security was something I was usually trying to work around, especially when I had to flow callers to the backend. My new challenge was to design security architectures for our application platform on .NET. Boy, was I in for a ride.
I knew security was the name of the game, but I had a lot to learn in terms of leading project teams around the world, brokering in all the right parts of the company, our community, and the industry. I also had a lot to learn in terms of how to create prescriptive guidance. The purpose was not to just put out a book. The purpose was to drive customer success on the platform. This included creating a durable, and evolvable security story that our customers could build on, and that we could use to “train the trainers.” This also meant creating a knowledge base that we could use as an anchor for driving product feedback and platform change. This was all in addition to learning how to think about security from an architecture and design standpoint, in a way that could help customers build more secure applications.
And thus the first Microsoft Blue Book, Building Secure ASP.NET Applications, was born. It was our answer to IBM Redbooks.
While I learned a lot from doing end-to-end security architectures and putting our variety of products and technologies together, the real learning came from driving Improving Web Application Security, or “Threats and Countermeasures”, as we affectionately called it. This journey was about learning how to bridge the gap between architecture and design, development, test, and operations. It was about learning how to think about security from a threats, attacks, vulnerabilities, and countermeasures perspective. It was about thinking about security in a more holistic way, and our little mantra was “secure the network, host, and app.”
We had the ultimate security dream team, on mission impossible, and we pulled it off. Our core engineering team included Alex Mackman, Michael Dunner, Srinath Vasireddy, Ray Escamilla and Anandha Murukan, and we had an extended team of industry champs including Joel Scambray of the Hacking Exposed series. (In fact, check out the extensive team list at the bottom of our Threats and Countermeasures Landing Page.)
How do I know we pulled it off? We had outstanding results in our platform security competitive study, and we were told that is was our prescriptive guidance that made the difference in a very big way. In addition, our guide was downloaded more than 800,000 times in the first six months, and it quickly established the mental models and language for how our growing customer base thought about security on the .NET platform. It was a powerful thing when customers would say to us back in our language, “We’re using the trusted subsystem model …” It was like poetry in motion.
The big thing we learned from the journey was the power of having end-to-end application scenarios, along with maps of threats and countermeasures, while baking security into the life cycle, and using proven practices, like threat modeling, to significantly change the game.
Here are some of the key security deliverables at a glance from the various security adventures over the years:
This doesn’t include the product feedback work we did, or the work we did to bake security into Visual Studio / MSF Agile, or the patents we filed, which were the icing on the cake. As a sidenote, behind the scenes we called our “threats and countermeasures” guide “WOMA.” It was short for weapon-of-mass adoption, because our field was continuously telling us stories of how they were winning accounts against the competition.
The Performance Story Performance was right up my alley, but it was a significant shift from security.
Ironically, I got a lot better at tackling security, by learning more about performance. The big lesson I learned was that you have to bound or constrain your efforts in some way. Just like performance can always be improved, so can security, so it’s all about prioritizing, knowing where to focus, and connecting back to the business objectives, aside from the technical perspective, and user experience.
Our first epic adventure in the performance space was the guide, Improving .NET Application Performance and Scalability (or, “Perf and Scale” as we affectionately called it.) It was where I first got to work with folks like Rico Mariani, Jan Gray, and Gregor Noriskin. It was mind blowing.
Working on performance and scalability was probably the most technically challenging exercise I’ve had at Microsoft. I remember spending countless days and nights walking through CLR internals, figuring out how to optimize collections, sorting through multi-threading patterns, and mastering how garbage collection really worked. Strangely enough, the “ah-ha” that I appreciated the most was figuring out that we could think of performance in terms of response time, throughput, and resource utilization (CPU, memory, network, and disk.) That little lens was the key to figuring out how to do effective performance modeling and solving nasty performance bottlenecks. It also helped us parse complicated performance scenarios down into bite-sized chunks.
Here are some of the key performance deliverables at a glance from the various performance adventures over the years:
While performance took me to the depth of things, it was application architecture that took me to the breadth of things …
Application Architecture Creating the Microsoft Application Architecture Guide was a chance to pull it all together.
It was a chance to finally showcase how to put our platform technologies together into common application archetypes. Rather than just a bunch of technologies, we could talk about our application platform very specifically in terms of application types: Web applications, RIA applications, desktop applications, Web services, and mobile applications. We could talk about scenarios and trade-offs. We could look at the shapes of applications in terms of architectural styles and patterns. We could look at cross-cutting concerns, such as caching, communication, concurrency, configuration management, data access, exception management, logging, state management, validation, and workflow. We could also walk the various quality attributes, like performance, security, reliability, manageability, usability, etc.
As you can imagine, this meant putting together a comprehensive map of our Microsoft application platform. The surprise for me was that by putting together the map and looking broadly at the industry, it was easy to see the forest from the trees, as well as better understand and anticipate where the industry was going in terms of what was growing and what was dying. It was a great way to get ahead of the curve. In fact, I’ve continued to map out our Microsoft application platform each year, as a way to quickly see the forest from the trees and to figure out where to spend my time.
During my adventures with application architecture, I got to learn a lot and work with amazing people. I also learned how to go across a lot of information faster and easier, and bring teams of people along the journey. The secret was to keep creating maps that helped everybody get on to the same page fast. This was an invaluable approach as our team was hunting and gathering all the pattern collections we could find. We basically built a large catalog and constellation of application patterns in the form of maps. While we didn’t include our maps in the guide, they helped our team and extended team ramp up in various spaces very quickly, as well as advance the practice of application architecture. Basically, we could browse patterns of solutions at a glance.
If you read nothing else, check out A Technique for Architecture and Design. Behind the scenes, we created this technique by synthesizing the expertise of more than 30 top solution architects, that had years of experience with structuring and designing end-to-end applications, dealing with security, performance, and reliability issues, and dealing with cross-cutting concerns. The idea was to put down on paper, a proven practice for rapidly modeling applications on a whiteboard and identify risks earlier vs. later.
What’s Next? I’ll have more to share as I go along. What you’ll most likely see is a shift to more focus on strategy, execution, and business impact. I’ll also share more information on the art and science of program management. I’ve been mentoring a lot of people and I think the PM role at Microsoft is a very special one. One of my mail goals is to broadly share the lessons I’ve learned from driving projects and leading teams and making impact as a PM on the patterns & practices team.
patterns & practices Guidance at a Glance Meanwhile, here is my catalog of patterns & practices guidance at a glance. Note that a lot of the prescriptive guidance I’ve worked on is out of date because of changes in technology. That said, you can still learn many of the key principles, patterns, and practices that the guidance is based on. In this respect, much of the guidance is “evergreen” in that it’s timeless and durable.
Developer Guidance Maps
Books / Guides
Practices at a Glance
Threats and Countermeasures
Questions and Answers
ASP.NET Security How Tos
WCF Security How Tos
Visual Studio Team System
My Related Posts
I often find myself sharing examples of evaluation criteria in the prescriptive guidance space. Here are some examples of criteria from one of my favorite platform assessments:
One way to drive more effective product design is to start with scenarios. One way to think of this is “Persona-based scenarios with goals.” You can use the scenarios as test cases. The scenarios can help you evaluate the design and they can help you evaluate implementation. Simply put, “Can the persona or user perform the particular scenario effectively?” You can then use criteria to evaluate.
It’s test-driven product design.
You can imagine how you can start with key themes and then break those down into more specific groups or frames. You end up with unit tests for experiences or scenarios and you can evaluate key things. For example, if you were doing a competitive assessment, you might evaluate against the following criteria:
The frames approach is a common approach for how platform competitive assessments are performed. That’s why we adopted it as an approach for how we do prescriptive guidance … it’s a proven practice for mapping out the problem domain within a given domain or space.
Example Scenarios Map This is an example of mapping out core scenarios, in this case, developer scenarios for source control:
Example of Sub-Scenarios This is an example of elaborating the core Branch/Label/Merge scenarios above:
There are many ways to gather, organize and use scenarios. In the end, what’s important is that you have the user scenarios validated with the actual users, and that you use them as a hub to bring together the customer, the developers, and the testers.
The beauty is that you can re-use the scenarios, whether it’s to explain your product, write documentation, or create prescriptive guidance.
What proven practices have you seen are vital behaviors that significantly contribute to customer success? …
That’s a common question I get, having spent the bulk of my time at Microsoft in patterns & practices. In fact, that’s a question the Windows Azure team asked me last Summer, as a way to boil down my lessons learned from patterns & practices and turn the insight into action.
Let me first set the stage that this information is primarily what contributes to a product’s success aimed at “developers.”
Banging on Products from Multiple Angles Helps You See the Good, the Bad, and the Ugly I’ve had a chance to see the good, the bad, and the ugly in terms of our products and how they land with customers. I’ve participated in many dog-fooding exercises, supportability reviews, platform evaluations, etc., so I‘ve learned what to look for. Because I’ve worked across so many products and technologies, I’ve had the chance to see what success patterns, as well as anti-patterns, show up time and again. In fact, one of the things I often do is cross-pollinate the best practices that I see across various product teams.
As an aside, before I joined patterns & practices, I worked in Microsoft Developer Support, which means I was on the receiving end of customer pain, and I learned how to walk a product from a usability and supportability standpoint, as well as learn many of the supportability (and “un-supportability”) patterns.
Some Products are Destined for Success To share what I’ve learned, I’ve created a simple frame that encapsulates some of the things that I’ve seen make a key difference. Creating prescriptive guidance has always been a combination of creating a “driver’s guide,” test driving the product against customer scenarios, finding effective solutions and workarounds, and improving the story around qualities, such as security or performance. During the process, I’ve found that some products are destined for success and have a strong customer adoption story out of the gate, while others face an uphill battle or never really succeed.
Sadly, I can easily see products that will fail, right up front too.
Product Success Frame Here is the frame I’ve been using for sharing some of the most important product success patterns.
It needs some iterations, but it’s been useful so far for putting some key success patterns into buckets. The table below shares some examples of what would fall under these buckets. After the table, I’ve provide some visual examples to help illustrate.
How to picture, think about, and talk about the problem space.
Sharing and scaling expertise, including driver’s guides and proven practices.
“Train-the-Trainer” material for wide-spread adoption and viral expertise.
Knowing your unique value and where you stand in the market or how you relate.
Building for the customer and with the customer to co-create a better future, as well as simplifying the user experience, and making it easy to find all the blocks.
Assume bad things happen to good people and making it easy to fall into the pit of success or get back to a successful state, or avoid falling off the cliff in the first place.
Know the touch points and the constellation of channels where users interact with your stuff, along with the user experiences for each of those channels.
Get the network on your side, from a tribe of raving fans, and closer followers, to a wider community base.
A Quick Tour of Examples … The following is a quick visual tour of some of the common things that help customers make the most of the product.
Application Scenarios and Solutions Example Application Scenarios and Solutions are a quick sketch of how to solve an end-to-end problem, along with key information on how to prototype the solution and test whether the solution will work for your scenario.
Canonical Application Architecture Example A canonical application architecture helps customer see how to put the building block technologies together into an ideal pattern.
CAT “Customer Advisory Team” Teams Customer Advisory Teams site between the product and the customer and they help address the patterns of customer problems through prescriptive guidance and hands-on support. They can also provide in-depth product feedback because they see the patterns and practices of large sets of customers, against specific scenarios and solutions.
For an example, see the SQL CAT Team.
Catalog of Solution Assets Example When you build out a catalog of solution assets, it’s a proven practice to have a set of content types. This helps users know what to expect in your catalog. This also helps you streamline your efforts for populating your catalog. The key is to focus on a small set of high ROI content types. For example, a common pattern is to focus on code samples, how tos, videos, and tutorials.
Customer Proof Points Customer Proof points are simply verbatums and scores on a slide from a customer to help show impact. The verbatums provide qualitative and the scores provide quantitative. The two key numbers to see are overall satisfaction and a change in confidence in the product, tool, or platform.
Deployment Patterns Deployment Patterns show how the common ways that the application is physically distributed. This can help you very quickly see the end in mind.
Feature Set Example Having your feature set at a glance, helps users quickly see all the dials, knobs, switches at a glance. Having the features at a glance is also an easy way to help your users learn the scope of your product. You can also use the features at a glance as one way for organizing, driving, and prioritizing your documentation. This, in conjunction with a scenarios map is a powerful combination.
Hot Spots for Architecture Hot Spots are basically a heat map of common pain points or key engineering decisions. They help you quickly see the forest from the trees in a given domain. For example, in a typical application, some of the cross-cutting concerns are auditing and logging, authentication, authorization, data access, exception management, etc. Knowing the Hot Spots helps inform you as an product designer or engineering, by helping inform you where to invest more energy.
Hot Spots for Security Example You can think of Hot Spots as a lens and you can overlay or focus on a particular topic, such as security, and highlight just those Hot Spots.
Hot Spots for Performance Example Here is an example of bubbling up common performance Hot Spots for a Web application.
How Tos How Tos provide step-by-step instructions for performing a task or solving a problem. When done well, they are among the most valued content because they directly map to user challenges, scenarios, and tasks.
See Security How Tos for an example of a How Tos collection.
Scenario Frames / Maps Example Scenario Maps are an easy way to gather and organize all the user scenarios in a meaningful way. Scenarios put requirements into context by showing what a particular user would try to accomplish. A collection of scenarios acts as a map that helps you see the landscape of your product, from both the engineering and the user view.
Scenario Scoreboard Example A Scenario Scoreboard is a simple way to evaluate the effectiveness of your product against the user scenarios. You simply add evaluation criteria and rate each scenario against the products, tools, and platform.
QuickStarts (Simple Starts for Common Tasks) QuickStarts are a way to share a quick map of common tasks, along with examples and step-by-step guidance.
See example at - http://quickstart.developerfusion.co.uk/QuickStart/howto/
"All projects experience changing requirements. Traditional projects view this as bad. Agile projects embrace it." -- Steven Thomas, BBC
Kelley Hunsberger writes about embracing scope creep, rather than fight it in her article “Change is Good”, in PM Network.
According to Hunsberger, scope creep can create three key opportunities:
In my experience, this tends to be true, especially for knowledge work or where the work is not well known. Change is a good thing, especially when it means acting on windows of opportunity, and delivering more value, in a timely way. Customers tend to embrace the change when they are involved throughout the process.
My favorite point in the article is about shaping scope as you go -- "With every sprint or iteration in agile, you are adding more clarity to the project's scope and reacting to it. But just because you are changing the scope of a project doesn't mean you are adding more scope to it."
Picking projects purely based on ROI is a narrow lens.
Sandra A. Swanson writes about using a broader approach for picking projects in her article, “All Things Considered”, in PM Network magazine.
Swanson quotes R. Max Wideman, author of A Management Framework for Project, Program, and Portfolio Integration, on his perspective: “Benefits realization is a much broader concept and encompasses not only financial gain, but all those intangibles such as improved working conditions, ease of use, work force satisfaction, and improvement in corporate morale."
My key take away from the article is that projects should be picked because of ROI and alignment to top strategic issues. When it's tough to show the ROI or strategic relevance, than consider the following factors: - meeting regulatory standards - improving quality - ensuring business continuity
Here is an example of a simple, but surprisingly effective way to test your product success. Around patterns & practices, a few of us affectionately called this a “consumer reports” view. Whether your product is prescriptive guidance or code, the concept is the same – you’re testing your product against user scenarios.
To create a Scenario Scoreboard, simply put together a matrix of scenarios. Next, add criteria. Next, test against the scenarios and rate the criteria. Note – There are two keys to making this successful: 1) have a useful collection of scenarios, and 2) get agreement on the criteria. You want agreement on the criteria so that people can stand behind the results. Tuning your criteria can help you tune your product success. You’re basically creating a feedback loop to measure the success of your product against user scenarios.
Example of a Scenario Scoreboard What’s important in the example is the simple frame: scenarios and criteria. The scenarios are grouped into buckets.
Scenarios / Tasks
Best Practice Compliance
Time to Implement
Arch / Design
Auditing and Logging
Monitoring / Health
Performance and Scalability
Best Practice Compliance Ratings
Implementation Complexity Ratings
Quality of Documentation and Sample Code Ratings
Developer/Administrator Competence Ratings
“Productivity is never an accident. It is always the result of a commitment to excellence, intelligent planning, and focused effort.” -- Paul J. Meyer
Anutthara Bharadwaj, a Program Manager on the Microsoft Visual Studio team shares her experience on adopting the Agile Results system in her post, Getting Results the Agile Way. In her story, Anu writes how she went from being overwhelmed to on top of her game by adopting three core practices:
I like her concrete examples. In her case, she uses One Note to implement the system. That’s the power of the Agile Results system. It’s a principle-based system so you can use whatever tools works best for you for adopting the practices. (I’ve used everything from pen and paper, whiteboards, Evernote, Outlook, One Note, simple text files, etc.)
Small changes in your day, add up to big changes in life. I’ve had the chance first-hand to watch Anu transform and it’s been an amazing journey.
Stop by and check out Anu’s post on Getting Results the Agile Way.
Early in my patterns & practices days, each time I built a new team, we had a hard time figuring out what level to cater our writing for because we had such a variety of audience, even among architects.
After a lot of pain, we finally adopted a three-level system that serves us very well. It helped us focus our writing and nail problems in an incremental way. You’ll never see this in our docs, but it shaped how we prioritize our docs. We used three levels …
Three Levels of Guidance Here is the behind-the-scenes look at how we talked about these three levels of guidance on the team:
Prioritizing Guidance As a rule of thumb, we decided that we would focus on first addressing Level 1 – “My Arm’s Broke, Fix Me.” This way, we could at least leave a trail of proven practices and pave a path of success. As a result, many of the guides I shipped from patterns & practices are heavy on “How Tos.” In fact, the guides are really “action guides.” The first half of the guide, sets the stage by sharing mental models, key concepts, and principles. This is optimized for reading in a sequential flow, but still modular so you can hop around. The second half of the guides is a focus on “action” and is a set of action modules (Cheat Sheets, Checklists, Guidelines, How Tos). It’s optimized for random access, and the individual modules link back to the related items.
This simple way to think about the majority of our guidance helped us significantly priorities the work we did for the following projects: