Several years back, I did an exercise in mapping out families of application architectures and application types.  It was an extensive archeological expedition.

Key Goals / Outcomes
There were several goals of the exercise:

• Identify canonical application architectures and app types
• Figure out a useful way of sharing architectures
• Figure out whether it's better to focus on the business architectures or the technical architectures
• Find a simple way to overlay patterns work on top of technical architectures
• Find a simple way to share architectural styles
• Find a way to simplify and share visuals of end-to-end architectures

The exercise fed into a number of later works years later, including:

• Re-shaping the patterns & practices catalog to be organized using app types -- and this literally drove our product-line more formally toward app types (a pre-cursor to the patterns & practices "factories" effort in 2005)
• Identifying a canonical set of application types for the Microsoft platform as part of the Application Architecture Guide.
• Shaping the path of the home/hub pages on MSDN toward "deployment targets" and application types

Key Activities
Some of the key activities of this early exercise included:

• Creating an exhaustive catalog of application types grouped and sliced in various ways
• Interviewing many of our best application architects in the trenches
• Sanity checking with many of our top architects at head quarters, including Rico Mariani, Pat Helland, and Jim Gray
• Reverse engineering many available sample applications inside and outside of Microsoft
• Checking with industry experts on their lessons learned about the types and the shapes of all the applications they've seen and built over the years
• Creating an extensive inventory of application patterns from various sets of our  key customers that had a wide range of applications and platforms
• Evaluating competitive efforts to index and catalog application shapes and types, and making sense of various patterns efforts

Keep in mind, that going into this, I already had the benefit of doing more than 650 customer architecture and design reviews -- yet still, this was an overwhelming exercise.  It forced me to find new ways to deal with large bodies of data and information, and somehow turn them into  shared maps, browsable, reusable knowledge nuggets, and backdrops for deeper conversations and elaboration.

Lessons Learned from Architectural Exploration
Some of the surprises for me or things that I learned that I didn't expect include:

• Focusing on business architectures and verticals was less effective and less reusable than focusing on technical architectures as a baseline (from there, we could overlay business architectures)
• Knowing the deployment patterns was the fastest way to get a good sense of the application patterns (it's where the logical met the physical against the infrastructure)
• Looking through quality attributes as a lens made it easy to find and explore patterns for security, performance, reliability, manageability, etc.
• So many applications boiled down to simple CRUD (Create Read Update and Delete)
• The biggest variation among applications was the business logic and workflow
• IBMs simple model of user, process, and data was useful as one lens
• Views and viewpoints were a powerful way to change perspective and drill into another aspect of applications
• At a high-level, you could capture a customer's core system by capturing the data, workflows, and workloads (the workloads significantly shaped the design of the app)

In retrospect, the simplicity and common denominators of CRUD make a lot of sense.  It's all about interacting with information at a fundamental level.  People are just trying to get things done and there's only so many things you can do with information -- find it, browse, save it, share it, transform it, etc. as part of your workflow.

Early Map of App Types
I included one of the many early maps of the application types that helped us figure out what to throw out and what to keep as we moved forward.   One of the key distinctions that Ward Cunningham helped me figure out was to distinguish between the shape of the application architecture and design, and the actual “purpose” of the application.  Some purposes were more business-oriented, while some were more technically oriented, and this helped me find and distinguish different families of apps.

It's circa 2004, but the irony is how timeless the backdrop really has been.  It’s rough and raw, but like I said, it’s just one sampling of the many braindumps behind the scenes of mapping out app types.

Cloud security has been a hot topic with the introduction of the Microsoft offering of the Windows Azure platform.  One of the quickest ways to get your head around security is to cut to the chase and look at the threats, attacks, vulnerabilities and countermeasures.  This post is a look at threats and countermeasures from a technical perspective.

The thing to keep in mind with security is that it’s a matter of people, process, and technology.  However, focusing on a specific slice, in this case the technical slice, can help you get results.  The thing to keep in mind about security from a technical side is that you also need to think holistically in terms of the application, network, and host, as well as how you plug it into your product or development life cycle.  For information on plugging it into your life cycle, see the Security Development Lifecycle.

While many of the same security issues that apply to running applications on-premise also apply to the cloud, the context of running in the cloud does change some key things.  For example, it might mean taking a deeper look at claims for identity management and access control.  It might mean rethinking how you think about your storage.  It can mean thinking more about how you access and manage virtualized computing resources.  It can mean thinking about how you make calls to services or how you protect calls to your own services.

Here is a fast path through looking at security threats, attacks, vulnerabilities, and countermeasures for the cloud …

Objectives

• Learn a security frame that applies to the cloud
• Learn top threats/attacks, vulnerabilities and countermeasures for each area within the security frame
• Understand differences between threats, attacks, vulnerabilities and countermeasures

Overview
It is important to think like an attacker when designing and implementing an application. Putting yourself in the attacker’s mindset will make you more effective at designing mitigations for vulnerabilities and coding defensively.  Below is the cloud security frame. We use the cloud security frame to present threats, attacks, vulnerabilities and countermeasures to make them more actionable and meaningful.

You can also use the cloud security frame to effectively organize principles, practices, patterns, and anti-patterns in a more useful way.

Threats, Attacks, Vulnerabilities, and Countermeasures
These terms are defined as follows:

• Asset. A resource of value such as the data in a database, data on the file system, or a system resource.
• Threat. A potential occurrence – malicious or otherwise – that can harm an asset.
• Vulnerability. A weakness that makes a threat possible.
• Attack. An action taken to exploit vulnerability and realize a threat.
• Countermeasure. A safeguard that addresses a threat and mitigates risk.

Cloud Security Frame
The following key security concepts provide a frame for thinking about security when designing applications to run on the cloud, such as Windows Azure. Understanding these concepts helps you put key security considerations such as authentication, authorization, auditing, confidentiality, integrity, and availability into action.

Threats and Attacks

Vulnerabilities

Countermeasures

Threats and Attacks Explained

1.  Brute force attacks. Attacks that use the raw computer processing power to try different permutations of any variable that could expose a security hole. For example, if an attacker knew that access required an 8-character username and a 10-character password, the attacker could iterate through every possible (256 multiplied by itself 18 times) combination in order to attempt to gain access to a system. No intelligence is used to filter or shape for likely combinations.
2. Buffer overflows. The maximum size of a given variable (string or otherwise) is exceeded, forcing unintended program processing. In this case, the attacker uses this behavior to cause insertion and execution of code in such a way that the attacker gains control of the program in which the buffer overflow occurs. Depending on the program’s privileges, the seriousness of the security breach will vary.
3. Canonicalization attacks. There are multiple ways to access the same object and an attacker uses a method to bypass any security measures instituted on the primary intended methods of access. Often, the unintended methods of access can be less secure deprecated methods kept for backward compatibility.
4. Cookie manipulation. Through various methods, an attacker will alter the cookies stored in the browser. Attackers will then use the cookie to fraudulently authenticate themselves to a service or Web site.
5. Cookie replay attacks. Reusing a previously valid cookie to deceive the server into believing that a previously authenticated session is still in progress and valid.
6. Credential theft. Stealing the verification part of an authentication pair (identity + credentials = authentication). Passwords are a common credential.
7. Cross-Site Request Forgery (CSRF). Interacting with a web site on behalf of another user to perform malicious actions. A site that assumes all requests it receives are intentional is vulnerable to a forged request.
8. Cross-site scripting (XSS). An attacker is able to inject executable code (script) into a stream of data that will be rendered in a browser. The code will be executed in the context of the user’s current session and will gain privileges to the site and information that it would not otherwise have.
9. Connection pooling. The practice of creating and then reusing a connection resource as a performance optimization. In a security context, this can result in either the client or server using a connection previously used by a highly privileged user being used for a lower-privileged user or purpose. This can potentially expose vulnerability if the connection is not reauthorized when used by a new identity.
10. Data tampering. An attacker violates the integrity of data by modifying it in local memory, in a data-store, or on the network. Modification of this data could provide the attacker with access to a service through a number of the different methods listed in this document.
11. Denial of service. Denial of service (DoS) is the process of making a system or application unavailable. For example, a DoS attack might be accomplished by bombarding a server with requests to consume all available system resources, or by passing the server malformed input data that can crash an application process.
12. Dictionary attack. Use of a list of likely access methods (usernames, passwords, coding methods) to try and gain access to a system. This approach is more focused and intelligent than the “brute force” attack method, so as to increase the likelihood of success in a shorter amount of time.
13. Disclosure of sensitive/confidential data. Sensitive data is exposed in some unintended way to users who do not have the proper privileges to see it. This can often be done through parameterized error messages, where an attacker will force an error and the program will pass sensitive information up through the layers of the program without filtering it. This can be personally identifiable information (i.e., personal data) or system data.
14. Elevation of privilege. A user with limited privileges assumes the identity of a privileged user to gain privileged access to an application. For example, an attacker with limited privileges might elevate his or her privilege level to compromise and take control of a highly privileged and trusted process or account. More information about this attack in the context of Windows Azure can be found in the Security Best Practices for Developing Windows Azure Applications at http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0ff0c25f-dc54-4f56-aae7-481e67504df6
15. Encryption. The process of taking sensitive data and changing it in such a way that it is unrecognizable to anyone but those who know how to decode it. Different encryption methods have different strengths based on how easy it is for an attacker to obtain the original information through whatever methods are available.
16. Information disclosure. Unwanted exposure of private data. For example, a user views the contents of a table or file that he or she is not authorized to open, or monitors data passed in plaintext over a network. Some examples of information disclosure vulnerabilities include the use of hidden form fields, comments embedded in Web pages that contain database connection strings and connection details, and weak exception handling that can lead to internal system-level details being revealed to the client. Any of this information can be very useful to the attacker.
17. Luring attacks. An attacker lures a higher-privileged user into taking an action on his or her behalf. This is not an authorization failure but rather a failure of the system to properly inform the user.
18. Man-in-the-middle attacks. A person intercepts both the client and server communications and then acts as an intermediary between the two without each ever knowing. This gives the “middle man” the ability to read and potentially modify messages from either party in order to implement another type of attack listed here.
19. Network eavesdropping. Listening to network packets and reassembling the messages being sent back and forth between one or more parties on the network. While not an attack itself, network eavesdropping can easily intercept information for use in specific attacks listed in this document.
20. Open Redirects. Attacker provides a URL to a malicious site when allowed to input a URL used in a redirect. This allows the attacker to direct users to sites that perform phishing attacks or other malicious actions.
21. Password cracking. If the attacker cannot establish an anonymous connection with the server, he or she will try to establish an authenticated connection. For this, the attacker must know a valid username and password combination. If you use default account names, you are giving the attacker a head start. Then the attacker only has to crack the account’s password. The use of blank or weak passwords makes the attacker’s job even easier.
22. Repudiation. The ability of users (legitimate or otherwise) to deny that they performed specific actions or transactions. Without adequate auditing, repudiation attacks are difficult to prove.
23. Session hijacking. Also known as man-in-the-middle attacks, session hijacking deceives a server or a client into accepting the upstream host as the actual legitimate host. Instead, the upstream host is an attacker’s host that is manipulating the network so the attacker’s host appears to be the desired destination.
24. Session replay. An attacker steals messages off of the network and replays them in order to steal a user’s session.
25. Session fixation. An attacker sets (fixates) another person’s session identifier artificially. The attacker must know that a particular Web service accepts any session ID that is set externally; for example, the attacker sets up a URL such as http://unsecurewebservice.com/?sessionID=1234567. The attacker then sends this URL to a valid user, who clicks on it. At this point, a valid session with the ID 1234567 is created on the server. Because the attacker determines this ID, he or she can now hijack the session, which has been authenticated using the valid user’s credentials.
26. Spoofing. An attempt to gain access to a system by using a false identity. This can be accomplished by using stolen user credentials or a false IP address. After the attacker successfully gains access as a legitimate user or host, elevation of privileges or abuse using authorization can begin.
27. SQL injection. Failure to validate input in cases where the input is used to construct a SQL statement or will modify the construction of a SQL statement in some way. If the attacker can influence the creation of a SQL statement, he or she can gain access to the database with privileges otherwise unavailable and use this in order to steal or modify information or destroy data.
28. Throttling. The process of limiting resource usage to keep a particular process from bogging down and/or crashing a system. Relevant as a countermeasure in DoS attacks, where an attacker attempts to crash the system by overloading it with input.

Countermeasures Explained

1. Assume all input is malicious. Assuming all input is malicious means designing your application to validate all input. User input should never be accepted without being filtered and/or sanitized.
2. Audit and log activity through all of the application tiers. Log business critical and security sensitive events. This will help you track security issues down and make sense of security problems. Skilled attackers attempt to cover their tracks, so you’ll want to protect your logs.
3. Avoid storing secrets. Design around storing secrets. If necessary, sometimes they can be avoided by storing them after using a one-way hash algorithm.
4. Avoid storing sensitive data in the Web space. Anything exposed to the public Internet is considered “web space.” Sensitive data stored in a location that might be compromised by any member of the public places it at much higher risk.
5. Back up and regularly analyze log files. Some attacks can occur over time. Regular analysis of logs will allow you to recognize with sufficient time to address them. Performing regular backups lowers the risk of an attacker covering his tracks by deleting logging of his activities.
6. Be able to disable accounts. The ability to reactively defend an attack by shutting out a user should be supported through the ability to disable an account.
7. Be careful with canonicalization issues. Predictable naming of file resources is convenient for programming, but is also very convenient for malicious parties to attack. Application logic should not be exposed to users in this manner. Instead, you use file names derived from the original names or fed through a one-way hashing algorithm.
8. Catch exceptions. Unhandled exceptions are at risk of passing too much information to the client. Handle exceptions when possible.
9. Centralize your input and data validation. Input and data validation should be performed using a common set of code such as a validation library.
10. Consider a centralized exception management framework. Exception handling frameworks are available publically and provide an established and tested means for handling exceptions.
11. Consider authorization granularity. Every object needs to have an authorization control that authorizes access based on the identity of the authenticated party requesting access. Fine grained authorization will control access to each resource, while coarse grained authorization will control access to groups of resources or functional areas of the application.
12. Consider identity flow. Auditing should be traceable back to the authenticated party. Take note of identity transitions imposed by design decisions like impersonation.
13. Constrain input. Limit user input to expected ranges and formats.
14. Constrain, reject, and sanitize your input. Constrain, reject and sanitize should be primary techniques in handling input data.
15. Cycle your keys periodically. Expiring encryption keys lowers the risk of stolen keys.
16. Disable anonymous access and authenticate every principle. When possible, require all interactions to occur as an authenticated party as opposed to an anonymous one. This will help facilitate more effective auditing.
17. Do not develop your own cryptography. Custom cryptography is not difficult for experts to crack. Established cryptography is preferred because it is known to be safe.
18. Do not leak information to the client. Exception data can potentially contain sensitive data or information exposing program logic. Provide clients only with the error data they need for the UI.
19. Do not log private data such as passwords. Log files are an attack vector for malicious parties. Limit the risk of their being compromised by not logging sensitive data in the log.
20. Do not pass sensitive data using the HTTP-GET protocol. Data passed using HTTP GET is appended to the querystring. When users share links by copying and pasting them from the browser address bar, sensitive data may also be inadvertently passed. Pass sensitive data in the body of a POST to avoid this.
21. Do not rely on client-side validation. Any code delivered to a client is at risk of being compromised. Because of this, it should always be assumed that input validation on the client might have been bypassed.
22. Do not send passwords over the wire in plaintext. Authentication information communicated over the wire should always be encrypted. This may mean encrypting the values, or encrypting the entire channel with SSL.
23. Do not store credentials in plaintext. Credentials are sometimes stored in application configuration files, repositories, or sent over email. Always encrypt credentials before storing them.
24. Do not store database connections, passwords, or keys in plaintext. Configuration secrets should always be stored in encrypted form, external to the code.
25. Do not store passwords in user stores. In the event that the user store is compromised, an attack should never be able to access passwords. A derivative of a password should be stored instead. A common approach to this is to encrypt a version of the password using a one-way hash with a SALT. Upon authentication, the encrypted password can be re-generated with the SALT and the result can be compared to the original encrypted password.
26. Do not store secrets in code. Secrets such as configuration settings are convenient to store in code, but are more likely to be stolen. Instead, store them in a secure location such as a secret store.
27. Do not store sensitive data in persistent cookies. Persistent cookies are stored client-side and provide attackers with ample opportunity to steal sensitive data, be it through encryption cracking or any other means.
28. Do not trust fields that the client can manipulate (query strings, form fields, cookies, or HTTP headers). All information sent from a client should always be assumed to be malicious. All information from a client should always be validated and sanitized before it is used.
29. Do not trust HTTP header information. Http header manipulation is a threat that can be mitigated by building application logic that assumes HTTP headers are compromised and validates the HTTP headers before using them.
30. Encrypt communication channels to protect authentication tokens. Authentication tokens are often the target of eavesdropping, theft or replay type attacks. To reduce the risk in these types of attacks, it is useful to encrypt the channel the tokens are communicated over. Typically this means protecting a login page with SSL encryption.
31. Encrypt sensitive cookie state. Sensitive data contained within cookies should always be encrypted.
32. Encrypt the contents of the authentication cookies. In the case where cookies are compromised, they should not contain clear-text session data. Encrypt sensitive data within the session cookie.
33. Encrypt the data or secure the communication channel. Sensitive data should only be passed in encrypted form. This can be accomplished by encrypting the individual items that are sent over the wire, or encrypting the entire channel as with SSL.
34. Enforce separation of privileges. Avoid building generic roles with privileges to perform a wide range of actions. Roles should be designed for specific tasks and provided the minimum privileges required for those tasks.
35. Enforce unique transactions. Identify each transaction from a client uniquely to help prevent replay and forgery attacks.
36. Identify malicious behavior. Monitoring site interactions that fall outside of normal usage patterns, you can quickly identify malicious behavior. This is closely related to “Know what good traffic looks like.
37. Keep unencrypted data close to the algorithm. Use decrypted data as soon as it is decrypted, and then dispose of it promptly. Unencrypted data should not be held in memory in code.
38. Know what good traffic looks like. Active auditing and logging of a site will allow you know recognize what regular traffic and usage patterns are. This is a required step in order to be able to identify malicious behavior.
39. Limit session lifetime. Longer session lifetimes provide greater opportunity for Cross-Site Scripting or Cross-Site Request Forgery attacks to add activity onto an old session.
40. Log detailed error messages. Highly detailed error message logging can provide clues to attempted attacks.
41. Log key events. Profile your application and note key or sensitive operations and/or events, and log these events during application operation.
42. Maintain separate administration privileges. Consider granularity of authorization in the administrative interfaces as well. Avoid combining administrator roles with distinctly different roles such as development, test or deployment.
43. Make sure that users do not bypass your checks. Bypassing checks can be accomplished by canonicalization attacks, or bypassing client-side validation. Application design should avoid exposing application logic, and segregating application logic into flow that can be interrupted. For example, an ASPX page that performs only validations and then redirects. Instead, validation routines should be tightly bound to the data they are validating.
44. Pass Forms authentication cookies only over HTTPS connections. Cookies are at risk of theft and replay type attacks. Encrypting them with SSL helps reduce the risk of these types of attacks.
45. Protect authentication cookies. Cookies can be manipulated with Cross-Site Scripting attacks, encrypt sensitive data in cookies, and use browser features such as the HttpOnly cookie attribute.
46. Provide strong access controls on sensitive data stores. Access to secret stores should but authorized. Protect the secret store as you would other secure resources by requiring authentication and authorization as appropriate.
47. Reject known bad input. Rejecting known bad input involves screening input for values that are known to be problematic or malicious. NOTE: Rejecting should never be the primary means of screening bad input, it should always be used in conjunction with input sanitization.
48. Require strong passwords. Enforce password complexity requirement by requiring long passwords with a combination of upper case, lower case, numeric and special (for example punctuation) characters. This helps mitigate the threat posed by dictionary attacks. If possible, also enforce automatic password expiry.
49. Restrict user access to system-level resources. Users should not be touching system resources directly. This should be accomplished through an intermediary such as the application. System resources should be restricted to application access.
50. Retrieve sensitive data on demand. Sensitive data stored in application memory provides attackers another location they can attempt to access the data. Often this data is used in unencrypted form also. To minimize risk of sensitive data theft, sensitive data should be used immediately and then cleared from memory.
51. Sanitize input. Sanitizing input is the opposite of rejecting bad input. Sanitizing input is the process of filtering input data to only accept values that are known to be safe. Alternatively, input can be rendered innocuous by converting it to safe output through output encoding methods.
52. Secure access to log files. Log files should only be accessible to administrators, auditors, or administrative interfaces. An attacker with access to the logs might be able to glean sensitive data or program logic from logs.
53. Secure the communication channel for remote administration. Eavesdropping and replay attacks can target administration interfaces as well. If using a web based administration interface, use SSL.
54. Secure your configuration store. The configuration store should require authenticated access and should store sensitive settings or information in an encrypted format.
55. Secure your encryption keys. Encryption keys should be treated as secrets or sensitive data. They should be secured in a secret store or key repository.
56. Separate public and restricted areas. Applications that contain public front-ends as well as content that requires authentication to access should be partitioned in the same manner. Public facing pages should be hosted in a separate file structure, directory or domain from private content.
57. Store keys in a restricted location. Protect keys with authorization policies.
58. Support password expiration periods. User passwords and account credentials are commonly compromised. Expiration policies help mitigate attacks from stolen accounts, or disgruntled employees who have been terminated.
59. Use account lockout policies for end-user accounts. Account login attempts should have a cap on failed attempts. After the cap is exceeded the account should prevent further login attempts. Lockout helps prevent dictionary and brute force attacks.
60. Use application instrumentation to expose behavior that can be monitored: Application transactions that are more likely to be targeted by malicious interactions should be logged or monitored. Examples of this might be adding logging code to an exception handler, or logging individual API calls. By providing a means to watch these transactions you have a higher likelihood of being able to identify malicious behavior quickly.
61. Use authentication mechanisms that do not require clear text credentials to be passed over the network: A variety of authentication approaches exist for use with web based applications some involve the use of tokens while others will pass user credentials (user name/id and password) over the wire. When possible, it is safer to use an authentication mechanism that does not pass the credentials. If credentials must be passed, it is preferable to encrypt them, and/or send them over an encrypted channel such as SSL.
62. Use least privileged accounts. The privileges granted to the authenticated party should be the minimum required to perform all required tasks. Be careful of using existing roles that have permissions beyond what is required.
63. Use least privileged process and service accounts. Allocate accounts specifically for process and service accounts. Lock down the privileges of these accounts separately from other accounts.
64. Use multiple gatekeepers. Passing the authentication system should not provide a golden ticket to any/all functionality. System and/or application resources should have restricted levels of access depending on the authenticated party. Some design patterns might also enforce multiple authentications, sometimes distributed through application tiers.
65. Use SSL to protect session authentication cookies. Session authentication cookies contain data that can be used in a number of different attacks such as replay, Cross-Site Scripting or Cross-Site Request Forgery. Protecting these cookies helps mitigate these risks.
66. Use strong authentication and authorization on administration interfaces. Always require authenticated access to administrative interfaces. When applicable, also enforce separation of privileges within the administrative interfaces.
67. Use structured exception handling. A structured approach to exception handling lowers the risk of unexpected exceptions from going unhandled.
68. Use the correct algorithm and correct key length. Different encryption algorithms are preferred for varying data types and scenarios.
69. Use tried and tested platform features. Many cryptographic features are available through the .NET Framework. These are proven features and should be used in favor of custom methods.
70. Validate all values sent from the client. Similar to not relying on client-side validation, any input from a client should always be assumed to have been tampered with. This input should always be validated before it is used. This encompasses user input, cookie values, HTTP headers, and anything else that is sent over the wires from the client.
71. Validate data for type, length, format, and range. Data validation should encompass these primary tenets. Validate for data type, string lengths, string or numeric formats, and numeric ranges.

SDL Considerations
For more information on preferred encryption algorithms and key lengths, see the Security Development Lifecycle at http://www.microsoft.com/security/sdl/ .

What is the success pattern for leading a successful v-team?  I asked one of my mentors, a seasoned softie for his take on what are the keys to a successful v-team.  Here is what he had to say:

• you're building a coalition -- not just a vteam
• make sure everyone is committed ... not just involved (see Turning Chickens into Pigs)
• get commitments and support from their leadership
• do an "exec" vision scope and milestone reviews
• give the team members the credit
• show up as a leader who is worth following - defend them, work for them, help them succeed -- you are working with people that you may want to follow you one day

This is a walk down memory lane.  It’s an idea I was circulating circa 2005 so the words will seem stale now.  Periodically I like to flip through my old ideas to see how I’ve used them, what impact they made, and what I learned in the process.

When I originally pitched the idea, there were two basic reactions – either “this will never fly” or “we have to make this happen.”  The funny thing was, when Amazon launched its Mechanical Turk, several months later, some of the original naysayers, came back to me and said, “Oh, I guess you were on to something.”

I wasn’t passionate about the idea, so I didn’t pursuit it.  I was also busy with other game changers.  However, I did share the idea with some key folks, and then I wrote it up in a simple way so that whoever might need to see it, could quickly understand the scenarios and the value.  However, the thinking behind this did end up shaping some interesting efforts.  It also helped me articulate a principle that kept showing up – Human Shepherds and the Law of Relevancy. 

As an aside, the big thing I learned from having clusters of ideas at a time … (I filed several patents that year, created a few key software prototypes that internal/external customers built offerings around, and helped shape some key product-line paths) … is that you can actually get ahead of the innovation eight-ball, if you know how to combine key principles and patterns with the trends and the maturity cycles of markets.  The patterns and principles help you see things that might not otherwise be obvious.  There are even tricks to getting ahead of the mainstream market.   Someday I’ll share the patterns and practices for innovation that I’ve learned from the school of hard knocks (part of our group acted like applied research so we have the battle scars of making innovation stick against real-world scenarios.)

Here is the original idea of “Short-Burst Work”, in raw form.  Note that one of the other things I learned since then is that choosing a sticky name for an idea, makes all the difference in the world – alliteration and fun, helps a ton!

Idea: Short Burst Work

Elevator Pitch
”Connect short-burst human power to the relevant short-burst work opportunities ...”

Summary
Here's an idea that could be a game changer ...
What if you could connect a world-wide market of job seekers to short-burst opportunities through "adverstising" in the corp space? (consumer works too, but let's play w/corp scenarios for now)
This is nearly the "long tail" of job opportunities issue.   Imagine all the short-burst work opportunities in your day job.   What's interesting is this model is happening on SecondLife, with real people and real exchange of money.
Imagine the world of pooled resources, available 24x7.  Imagine the world of editors, coders, great thinkers …. etc.  Imagine how many former dot-Com'ers that would love to lend short-burst help.  Imagine all the great professors of the world you can suddenly involve when you can virtualize your work items and collaborate with a simple platform.

Situation

• Fine-grained work items (short-burst) can be tough to fill whether temp agency, Monster ...
• Find-grained opportunities (mentor me for 1/2 hr Wed, build me code to do X ... etc.) are a challenge to find the short-burst service/provider (yet the help exists ... it's just displaced or tough to make the match or there's workflow hurdles)
• Untapped  or displaced resource pools in the US
• Untapped resource pools world-wide
• Lots of challenges connecting a world-wide resource pool to short-burst, relevant work
• 24/7 availability of a distributed world-wide resource pool, waiting to be leveraged

Opportunity

• Connect services/providers to relevant short-burst work
• Improve the ability for consumers to find relevant labor pools

Recommendation
From a mental model, think of it as a highly relevant "match" service for short-burst work opportunities and resource pools. 

• Create a platform for a "Live Help/Services" model.
• Create a pluggable "service" model for providers (pluggable experts/skills:  mentoring, professional counseling, health, psychology, education,  software engineers ... anything)
• Simplify the ability for a service provider to plug in from anywhere in the world

Corporate Scenarios
Imagine live services to connect your short-burst work items to:

• Build me a strawman presentation for X
• Write me a doc on X that explains Y
• Write me a piece of code to do Z
• Show me how to build a solution for scenario N
• I need advice on today's big task
• Coach my on ....
• Tech edit my specs ...
• Help me create a demo to show ....
• Advise me on ....

Home Scenarios

• Write me a book report ...
• Help me with my math homework
• Help with my presentation ...
• Mentor my niece ...

Why is this feasible? 
It's happening on SecondLife.  Folks around their world are selling their services (scripting, image consulting, music dj …. You name it)  It's happening because the barrier to entry is low:

To sell your services/skills:

• You don't need to figure out how to exchange money; SecondLife figured out the exchange and abstracted (like PayPal on the net)
• You don't need to go through a bunch of hoops to advertise your services.

To consume the services/skills:

• You search/browse
• Can't find what you need?  Then you bring in Live Help and a concierge helps you find it.

There's reasons why existing services like this haven't taken off.  Connecting the consumer/provider to the "long tail" of the short-burst opportunities is a relevancy algorithm issue and a platform hurdle (services platform with good consumption/provider models).
Your engine can help folks find the short-burst services they need.  And when the relevancy is beyond organic search, you broker a "concierge" type service (Live Concierge) to connect a consumer to a provider … or a provider to a consumer.  It's a game changer.
How to make it happen?

Imagine you the user wants to connect to "live" help.  Think (Google (relevancy) + Monster (labor pool) + Temp Agencies (labor pool and short-burst work) + (plug in your favorite Web 2.0 model)

I use scenarios all the time for anything from designing a user experience to evaluating architecture. Scenario is an overloaded term though. There are lots of types of scenario tools. If you know the types of scenario tools, you can use the right one for the job. For example, exception scenarios are useful for assessing robustness.  Misuse cases are helpful for figuring out potential threats and attacks.

In the book Scenarios, Stories, Use Cases: Through the Systems Development Life-Cycle , Ian F. Alexander and Neil Maiden write about the types of scenarios and when to use them in software development.

• Alternative World, Situation, Snapshot
• Concept of Operations
• Exception(-handling) Scenario
• Negative Scenario, Misuse Case
• Roleplay, Playthrough (of an Acted Scene)
• Scenario Sequence, Script
• Scripted Sequence Diagram
• Simulation, animation
• Story
• Storyboard
• Swim lanes Diagram
• Test Case
• Use Case
• User Story

Here is a brief explanation of each:

• Alternative World, Situation, Snapshot - A scenario that describes a more or less static picture of an imagined future business situation
• Concept of Operations - A set of scenarios describing in more or less design-independent terms how a system is expected to be used in practice.
• Exception(-handling) Scenario - A scenario that describes how to handle an undesired event.  Useful for reliability analysis.
• Negative Scenario, Misuse Case - A scenario that is desired not to occur by the stakeholders responsible for a system. Useful for identify security threats or security issues.
• Roleplay, Playthrough (of an Acted Scene) - A scenario acted out to illustrate a desired behavior of or a possible problem with a future System.
• Scenario Sequence, Script - A simple list of steps ordered by time, possibly numbered, representing a Course of Events. Each step is an action taken by a named role (Actor).
• Scripted Sequence Diagram - Shows the steps of a user with the system.
• Simulation, animation - Narrates a scenario using images in action.
• Story - A text that narrates a Scenario with little or no formal structure.
• Storyboard - A sequence of diagrams or other images that narrates a Scenario.
• Swim lanes Diagram - A flowchart arranged with a “swimming pool lane” for each role, showing both the actions of each role and interaction between roles over time.
• Test Case - A sequence of steps to be run to verify that a system works as you intend it.
• Use Case - A UML scenario format that consists of an Actor, a Goal, and text that contain a set of related scenarios representing different combinations of sequences of the Use Case, depending on Events.
• User Story - A scenario format consisting of a brief, informal narrative text (a Story) that explains how a user can achieve a goal.

Workshops are one of the best ways to rapidly brainstorm ideas, capture scenarios, mine for patterns, etc.  The challenge can be designing an effective workshop. 

In the book Scenarios, Stories, Use Cases: Through the Systems Development Life-Cycle, Ian F. Alexander and Neil Maiden present The Six P’s Workshop Framework.  Here is a summary of the approach:

As you can see, the six P’s (Purpose, Participants, Products, Place, and Process) create the overall frame.   Each category within the frame has a driving question.  For example, “Purpose” contains the question, “Why are we doing the workshop?”  Each category within the frame also has a list of “Concerns.”  For example, “Purpose” contains the concerns, “goals", “need”, and “motivation.”

By addressing the question and the concerns, you can design and shape a more effective workshop.

While working with a customer at our Microsoft Executive Briefing Center (Microsoft EBC), one of the things we need to do was find a simple and effective way to partition SQL Azure.

The problem with SQL Azure is that depending on your partitioning strategy, you can end up allocating increments that you don’t need and still pay for it.  The distinction is that for Azure Storage, you pay for storage used, but with SQL Azure you pay for the allocated increments.  See Windows Azure Pricing.

After checking with our colleagues, one of the approaches we liked is to partition by customer ID ranges.  By adding this level of indirection, you can optimize or reduce unused increments because you already know the ranges and usage, and you can change the ranges and allocations if you need to.   Here is a visual of the approach:

Thanks to our colleague Danny Cohen for the insight, clarification, and suggestions.

What is a PM (Program Manager)?  While the Program Manager role seems unique to Microsoft, in general, when you map it to other companies, it’s a product manager, or a project manager, or a combination of the two.  At Microsoft, there are various flavors of PMs (“design” PM, “project” PM, “process” PM, etc.) and the PM discipline can be very different in different groups.  I’ve also seen the PM title used as a general job title, in the absence of something more specific.

At Microsoft it’s a role that means many things to many people.  In general though, when you meet a PM at Microsoft, you expect somebody who has vision, can drive a project to completion, can manage scope and resources, coordinate work across a team, bridge the customer, the business, and the technology, act as a customer champ, and influence without authority.  From a metaphor standpoint, they are often the hub to the spokes, they drive ideas to done, they take the ball and run with it, or find out who should run with the ball.  Some PMs are better at thought leadership, some are better at people leadership, and the best are great at both. 

Here is a roundup of some of my favorite points that elaborate, clarify, and distill what a PM is, what a PM does, and how to be one.

Attributes and Qualities of a PM
Here is a list of key attributes from Steven Sinofsky’s post -- PM at Microsoft:

• “Strong interest in what computing can do -- a great interest in technology and how to apply that to problems people have in life and work”
• “Great communication skills -- you do a lot of writing, presenting, and convincing”
• “Strong views on what is right, but very open to new ideas -- the best PMs are not necessarily those with the best ideas, but those that insure the best ideas get done”
• “Selfless -- PM is about making sure the best work gets done by the team, not that your ideas get done. “
• “Empathy -- As a PM you are the voice of the customer so you have to really understand their point of view and context “
• “Entrepreneur -- as a PM you need to get out there and convince others of your ideas, so being able to is a good skill”

Here is an example of PM qualities from Sean Lyndersay’s post -- Exchange team defines a Program Manager:

• You are passionate about working directly with customers, able to clearly articulate customers requirements and pains to executives, architects and developers.
• You have experience building strong teams and have a passion for mentoring.
• The ability to work with multiple teams to develop a plan to provide this value to the customers.
• You understand how software features will impact and/or modify the marketplace once they are shipped.
• You have a love of innovation and the ability to think through big, long term ideas.
• Demonstrated expertise at prioritization and project management.
• You are hands-on with software development, and passionate about user experience, both as an end-user and administrator.

Microsoft Careers site on Program Manager
Here is a description of a Program Manager from the Microsoft Careers site:
“As a Program Manager, you’ll drive the technical vision, design, and implementation of next-generation software solutions. You’ll transform the product vision into elegant designs that will ultimately turn into products used by Microsoft customers. Managing feature sets throughout the product lifecycle, you’ll have the chance to see your design through to completion. You’ll also work directly with other key team members including Software Development Engineers and Software Development Engineers in Test. Program Managers are advocates for end-users, so your passion for anticipating customer needs and creating outside-the-box solutions for them will really help you shine in this role. As a Program Manager you will have the ability to lead within a product’s life cycle using evangelism, empathy, and negotiation to define and deliver results. You will also be responsible for authoring technical specifications, including envisaged usage cases, customer scenarios, and prioritized requirements lists.” 

Chris Pratley on Program Manager 
Here are some points on Program Management from Chris Pratley’s post -- Program Manager:

• “One way to describe PMs is that they not only "pick up and run with the ball, they go find the ball". That really defines the difference between "knowing what to do and doing it", and "not knowing what to do, but using your own wits to decide what to do, then doing it". That means as a PM you are constantly strategizing and rethinking what is going on to find out if there is something you are missing, or the team is missing. You’re also constantly deciding what is important to do, and whether action needs to be taken.”
• "... In Office, there are "design" PMs who mainly work on designing the products, "process" PMs who mainly work on driving processes that make sure we get things done, localization PMs who are sort of like process PMs but also sort of like developers in that they sometimes produce code and make bits that ship in the product..."
• “This ‘jack of all trades’ or ‘renaissance man’ … acted as a hub of communication, and made the marketers job easier, since they had someone to talk to who could take the time to really understand the outside world, and the devs could talk to someone who spoke their language and was not ridiculous …”
• “You’re also constantly deciding what is important to do, and whether action needs to be taken. The number of such decisions is staggering. I might make 50 a day, sometimes more than 100 or even 200. Most of the time there is not nearly enough hard data to say for certain what to do, and in any case almost all these decisions could never have hard data anyway - you need to apply concentration and clear thinking.”

Here are the stages of your first year as a PM, according to Pratley:

1. Start off with excitement and enthusiasm for the new job.
2. About 4 weeks into the job, you start to feel strange. People keep asking you to decide things you don’t know anything about, as if you’re some kind of expert.
3. By month two, you're convinced you are the dumbest person on the team by far.
4. By month four, you have lived through a torture of feeling incompetent and a dead weight on your team.
5. By month six, you have a great moment.
6. By month 12, you have developed your network of contacts that pass information to you, you are a subject matter expert on your area, and people on the team are relying on you because you know lots of things they don't know. You have made it.

Joel Spolsky on Program Manager
Here are points from Joel’s post on How To Be a Program Manager:

• “Having a good program manager is one of the secret formulas to making really great software”
  “According to Joel, Charles Simonyi invented the job “Program Manager”, but Jabe Blumenthal, a programmer on the Mac Excel team in the late 80s, reinvented the job completely.”
• “What does a Program Manager do?  “Design UIs, write functional specs, coordinate teams, serve as the customer advocate.”
• “How To Be a Program Manger - Making Things Happen, by Scott Berkun, Don't Make Me Thin, by Steve Krug's, User Interface Design for Programmers, by Joel Spolsky, and How To Win Friends and Influence People by Dale Carnegie.“

Ray Schraff on Program Manager
Here are point from the comments on Chris Pratley’s post, Program Management:

• “Once they find the ball, PMs don't pick up every ball themselves... but they own the task of making sure that every ball is picked up and carried to the correct endzone by SOMEBODY. “
• “PMs translate technology to English “
• “PMs translate English to technology”

Sean Lyndersay on Program Manager

• “My favorite definition involves an analogy: A program manager is to a software project what an architect is to a building.”  See Reflections Program Management at Microsoft.
• “… the PM occupies a unique position in most software engineering structures – sort of the hub of a bumpy wheel (with dev, QA/test, design, usability, marketing, planning, customer support, etc. being the spokes).  See Someone has an Interesting View of PMs (at least at Yahoo!)”  See Someone has an Interesting View of PMs (at least at Yahoo)
• "You are the center of the hurricane, the eye of the product development storm. You have passion and more, you have vigor. You are fueled by pure energy and endless drive. You are reading this and wondering why it's not in bullet-points because that would have been more efficient. You are working on a game that merges poker with chess in your spare time because neither game uses your full capabilities and talents. You have no use for extraneous clutter or mundane activities and you wonder what is holding up the full-scale integration of robots into the home – its 2007 already and doing the dishes remains as mundane and inefficient as it ever was. You are thinking that this job description is taking too long to read, and you are right. So, here is the rest of it in bullet points."  See Exchange team defines a Program Manager.
• “With regard to where ideas come from, what we like to say is that the job of program management is not to have all the great ideas but to make sure all the great ideas are ultimately picked. The best program managers make sure the best ideas get done, no matter where they come from.”  See The Job of Program Management.

Steven Sinofsky on Program Manager
Here are points from Sinfoksy’s post on PM at Microsoft:

• “Learn, convince, spec, refine”
• “PM is one equal part of a whole system.”
• “PM is a role that has a lot of responsibility but you also define it--working in partnership with expert designers, clever developers, super smart testers, etc. you all work together to define the product AND the responsibilities each of you have”
• “… the PM role at Microsoft is both unique and one that has unlimited potential -- it is PMs that can bring together a team and rally them around a new idea and bring it to market (like OneNote, InfoPath).”
• “It is PMs that can tackle a business problem and work with marketing and sales to really solve a big customer issue with unique and innovative software (like SharePoint Portal Server).”
• “Where developers were focused on code, architecture, performance, and engineering, the PM would focus on the big picture of "what are we trying to do" and on the details of the user experience, the feature set, the way the product will get used.”
• “A key benefit of program management is that we are far more agile because we have program management.  That can be counter-intuitive (even for many developers at Microsoft who might be waiting for their PM to iron out the spec).  But the idea that you can just start writing code without having a clear view of the details and specification is a recipe for a poorly architected features.”
• “A great PM knows when the details are thought through enough to begin and a great developer knows when they can start coding even without the details for sure.  But like building a house--you don't start without the plans.  “
• “A good book that describes the uniqueness of PM at Microsoft is Michael Cussumano's book Microsoft Secrets or his new book, The Business of Software.”

My Related Posts

• 7 Habits of Highly Effective PMs at Microsoft
• Key Questions of a PM at Microsoft
• Patterns and Practices for New Hires

The Microsoft patterns & practices Windows Azure Architecture Guide - Part I is now available on MSDN. 

This guide focuses on a migration scenario.  It walks through migrating a custom expense tracking and reimbursement system to the cloud.

Chapters

• Introduction to the Windows Azure Platform
• The Adatum Scenario
• Phase 1: Getting to the Cloud
• How Much Will It Cost
• Phase 2: Automating Deployment and Using Windows Azure Storage
• Phase 3: Uploading Images and Adding a Worker Role
• Application Life Cycle Management for Windows Azure Applications
• Phase 4: Adding More Tasks and Tuning the Application

Key Topics

• Asynchronous processing, blobs, and shared access signatures
• Authentication and authorization
• Automating deployment and managing environments
• Background processing and performance considerations
• Data access
• Development life cycle
• Pricing and cost considerations
• Session management
• Terminology and components
• Using Windows Azure table storage

Key Links

• Windows Azure Architecture Guide - Part 1 (MSDN)
• Windows Azure Project Site (CodePlex)
• Download Windows Azure code samples

While working with a customer last week at our Microsoft Executive Briefing Center (Microsoft EBC), one of the things we need to do was find a simple way to whiteboard the architecture.  We walked through a lot of variations of application patterns, but the customer ultimately wanted to see one code base running on Azure that could support multiple clients (such as desktop, phone, browser, etc.)   We found a model that was pretty straightforward and allowed us to easily map the customer’s scenario onto the Azure platform.

Once we had a model, it was a lot easier to drill down on things such as why SQL Azure vs. Azure Storage or how to flow claims or how to partition the database.  It’s like a backdrop that you can overlay or drill down on security, performance, manageability, etc.  It helped us quickly explore paths without getting into weeds and keeping the big picture in mind.  It also helped us figure out where target some of the key architectural spikes.

Here are some of the models we walked through on the whiteboard:

Baseline

Azure Storage Example

SQL Azure Example