Software Engineering, Project Management, and Effectiveness
“Life is like skiing. Just like skiing, the goal is not to get to the bottom of the hill. It’s to have a bunch of good runs before the sun sets.” – Seth Godin
It's been a good run. After more than 10 years in patterns & practices, I'm on to my next adventure here at Microsoft.
For this post, I wanted to take a stroll down memory lane. During my time at patterns & practices, I learned more about project management, application architecture, software development, processes, teamwork, leadership, product success, and making impact than I think I could have ever hoped to learn anywhere else. Perhaps the best part, is that I have 10+ years of deliberate practice in the art of “best practices” and I’ve had a chance to work with some of the most amazing people in the industry, lead SWAT teams around the world on fantastic voyages and epic adventures, and leave a legacy of several high-impact Microsoft Blue Books, all while mastering the art of prescriptive guidance.
There are a few ways I can share my journey in patterns & practices. One of my managers referred to me as “the abilities PM” because of my focus on quality attributes (security, performance, scalability, etc.) and that’s a pretty good place to start. My role as a PM (Program Manager) in patterns & practices, can largely be defined in terms of my impact on these three areas: security, performance, and application architecture. (Yes, there is more to the story, but those three areas, serve as a good enough lens for now.)
The Security Story It all started when I joined patterns & practices (PAG or the Prescriptive Architecture Guidance team at the time.)
I moved from Microsoft Developer Support with the expectation that I would share and scale broadly with customers what I learned from living in across the hall from folks like Scott Guthrie, Brad Abrams, and other .NET Framework leaders. Having performed more than 650 customer architecture and design reviews, I was in a unique position to share all the principles, patterns, practices, and anti-patterns that I had seen across a large number of customer projects, many of which were on the leading and bleeding edge of this space.
But life has a way of throwing curve balls. Around this time, security was Jim Allchin’s top priority, and he asked patterns & practices what we were going to do about security. In turn, my manager asked me, what I was going to do about security. Coming from a middleware background, security was something I was usually trying to work around, especially when I had to flow callers to the backend. My new challenge was to design security architectures for our application platform on .NET. Boy, was I in for a ride.
I knew security was the name of the game, but I had a lot to learn in terms of leading project teams around the world, brokering in all the right parts of the company, our community, and the industry. I also had a lot to learn in terms of how to create prescriptive guidance. The purpose was not to just put out a book. The purpose was to drive customer success on the platform. This included creating a durable, and evolvable security story that our customers could build on, and that we could use to “train the trainers.” This also meant creating a knowledge base that we could use as an anchor for driving product feedback and platform change. This was all in addition to learning how to think about security from an architecture and design standpoint, in a way that could help customers build more secure applications.
And thus the first Microsoft Blue Book, Building Secure ASP.NET Applications, was born. It was our answer to IBM Redbooks.
While I learned a lot from doing end-to-end security architectures and putting our variety of products and technologies together, the real learning came from driving Improving Web Application Security, or “Threats and Countermeasures”, as we affectionately called it. This journey was about learning how to bridge the gap between architecture and design, development, test, and operations. It was about learning how to think about security from a threats, attacks, vulnerabilities, and countermeasures perspective. It was about thinking about security in a more holistic way, and our little mantra was “secure the network, host, and app.”
We had the ultimate security dream team, on mission impossible, and we pulled it off. Our core engineering team included Alex Mackman, Michael Dunner, Srinath Vasireddy, Ray Escamilla and Anandha Murukan, and we had an extended team of industry champs including Joel Scambray of the Hacking Exposed series. (In fact, check out the extensive team list at the bottom of our Threats and Countermeasures Landing Page.)
How do I know we pulled it off? We had outstanding results in our platform security competitive study, and we were told that is was our prescriptive guidance that made the difference in a very big way. In addition, our guide was downloaded more than 800,000 times in the first six months, and it quickly established the mental models and language for how our growing customer base thought about security on the .NET platform. It was a powerful thing when customers would say to us back in our language, “We’re using the trusted subsystem model …” It was like poetry in motion.
The big thing we learned from the journey was the power of having end-to-end application scenarios, along with maps of threats and countermeasures, while baking security into the life cycle, and using proven practices, like threat modeling, to significantly change the game.
Here are some of the key security deliverables at a glance from the various security adventures over the years:
This doesn’t include the product feedback work we did, or the work we did to bake security into Visual Studio / MSF Agile, or the patents we filed, which were the icing on the cake. As a sidenote, behind the scenes we called our “threats and countermeasures” guide “WOMA.” It was short for weapon-of-mass adoption, because our field was continuously telling us stories of how they were winning accounts against the competition.
The Performance Story Performance was right up my alley, but it was a significant shift from security.
Ironically, I got a lot better at tackling security, by learning more about performance. The big lesson I learned was that you have to bound or constrain your efforts in some way. Just like performance can always be improved, so can security, so it’s all about prioritizing, knowing where to focus, and connecting back to the business objectives, aside from the technical perspective, and user experience.
Our first epic adventure in the performance space was the guide, Improving .NET Application Performance and Scalability (or, “Perf and Scale” as we affectionately called it.) It was where I first got to work with folks like Rico Mariani, Jan Gray, and Gregor Noriskin. It was mind blowing.
Working on performance and scalability was probably the most technically challenging exercise I’ve had at Microsoft. I remember spending countless days and nights walking through CLR internals, figuring out how to optimize collections, sorting through multi-threading patterns, and mastering how garbage collection really worked. Strangely enough, the “ah-ha” that I appreciated the most was figuring out that we could think of performance in terms of response time, throughput, and resource utilization (CPU, memory, network, and disk.) That little lens was the key to figuring out how to do effective performance modeling and solving nasty performance bottlenecks. It also helped us parse complicated performance scenarios down into bite-sized chunks.
Here are some of the key performance deliverables at a glance from the various performance adventures over the years:
While performance took me to the depth of things, it was application architecture that took me to the breadth of things …
Application Architecture Creating the Microsoft Application Architecture Guide was a chance to pull it all together.
It was a chance to finally showcase how to put our platform technologies together into common application archetypes. Rather than just a bunch of technologies, we could talk about our application platform very specifically in terms of application types: Web applications, RIA applications, desktop applications, Web services, and mobile applications. We could talk about scenarios and trade-offs. We could look at the shapes of applications in terms of architectural styles and patterns. We could look at cross-cutting concerns, such as caching, communication, concurrency, configuration management, data access, exception management, logging, state management, validation, and workflow. We could also walk the various quality attributes, like performance, security, reliability, manageability, usability, etc.
As you can imagine, this meant putting together a comprehensive map of our Microsoft application platform. The surprise for me was that by putting together the map and looking broadly at the industry, it was easy to see the forest from the trees, as well as better understand and anticipate where the industry was going in terms of what was growing and what was dying. It was a great way to get ahead of the curve. In fact, I’ve continued to map out our Microsoft application platform each year, as a way to quickly see the forest from the trees and to figure out where to spend my time.
During my adventures with application architecture, I got to learn a lot and work with amazing people. I also learned how to go across a lot of information faster and easier, and bring teams of people along the journey. The secret was to keep creating maps that helped everybody get on to the same page fast. This was an invaluable approach as our team was hunting and gathering all the pattern collections we could find. We basically built a large catalog and constellation of application patterns in the form of maps. While we didn’t include our maps in the guide, they helped our team and extended team ramp up in various spaces very quickly, as well as advance the practice of application architecture. Basically, we could browse patterns of solutions at a glance.
If you read nothing else, check out A Technique for Architecture and Design. Behind the scenes, we created this technique by synthesizing the expertise of more than 30 top solution architects, that had years of experience with structuring and designing end-to-end applications, dealing with security, performance, and reliability issues, and dealing with cross-cutting concerns. The idea was to put down on paper, a proven practice for rapidly modeling applications on a whiteboard and identify risks earlier vs. later.
What’s Next? I’ll have more to share as I go along. What you’ll most likely see is a shift to more focus on strategy, execution, and business impact. I’ll also share more information on the art and science of program management. I’ve been mentoring a lot of people and I think the PM role at Microsoft is a very special one. One of my mail goals is to broadly share the lessons I’ve learned from driving projects and leading teams and making impact as a PM on the patterns & practices team.
patterns & practices Guidance at a Glance Meanwhile, here is my catalog of patterns & practices guidance at a glance. Note that a lot of the prescriptive guidance I’ve worked on is out of date because of changes in technology. That said, you can still learn many of the key principles, patterns, and practices that the guidance is based on. In this respect, much of the guidance is “evergreen” in that it’s timeless and durable.
Developer Guidance Maps
Books / Guides
Practices at a Glance
Threats and Countermeasures
Questions and Answers
ASP.NET Security How Tos
WCF Security How Tos
Visual Studio Team System
My Related Posts
You will indeed be missed! These are all fantastic references that I make use of every day, and I've pointed them out in my Windows Azure Learning Plan as well.
Good luck in your future "hills" - I look forward to hearing good things.
good luck in your new adventure, great work in the P&P
Its been one hell of a decade and you certainly have kicked some serious ***! We'll certainly miss JD.PnP but have high expectations for JD.ES ;)
Good luck in new adventure, JD.
I have read your post every week. Thanks so much.
I just wonder how I can get the post like yours.
An end of an era, what a ride its been just watching from the sidelines.
Your list of releases over the years always made me confident in the solutions I delivered in my daily work. Invaluable is the least I can say. Good luck in your future endeavors, I wonder what could top this, besides the change of scenery.
@ Buck --Thank you. I'll still be on the scene, but yet, conquering some new hills.
@ Eber -- Thank you.
@ Andy -- Enterprise Strategy looks to be a great ride.
@ Don -- There are more rides to come ;)
@ Dennis - Thank you. Sometimes the trick is to find the next wave, other times the trick is make one.
Thank you J.D. for all your contributions. I enjoyed reading your posts because they touched on much more than just technology and products.