Chris Keyser's WebLog

Input on customization scenarios for the next release of SharePoint guidance

chriskeyser — Fri, 18 Sep 2009 17:13:00 GMT

We are rolling right into scoping the next release of patterns and practices SharePoint guidance for SharePoint 2010. Now this is a little different than how pattners and practices typically works since we working in advance of a product release. There will be many great features coming in 2010 that you'll hear about over the coming months, but the reality is that many of the fundamental business problems and required customizations remain the same. We are looking for input on what customization scenarios are most important to our customers as we plan for this release. Please give us feedback on new areas you'd like to see covered, or areas you'd like to see extended from our current release, Developing SharePoint Applications.

To provide your customization scenario feedback please go to our codeplex site where I have added a discussion forum.

Released Developing SharePoint Applications!

chriskeyser — Fri, 04 Sep 2009 22:44:00 GMT

We've just released Developing SharePoint Applications! Thanks to all of our advisors and a dedicated team for helping put together a packed release. The release contains some reusable bits, a new Partner Portal reference implementation, and a bunch of written content. While the referennce implementation represents a pretty complicated application that demonstrates advanced principles, we worked hard to make the content accessible incrementally, and to make sure you can get something out of the guidance without having to understand the reference implementation in depth. We also packed in a few reusable tidbits like a simple logging framework, a service locator, some classes to help you with buidling list based repositories, a configuration manager, and a small but helpful nugget for registering AJAX scripts in your web part without worrying about conflicting with other parts.

Please send on your feedback! We are also collecting scenarios for our next release, so I'd love to hear about any new areas you'd like to have us explore that are challenging as a developer.

Terminology and SharePoint ALM

chriskeyser — Mon, 09 Mar 2009 17:08:00 GMT

Thanks for the feedback on my last post. I figured I'd write another post since this is an area we've thought about a lot. Sorry for the delayed response, I was out watching my nephews wrestle in their final championships all weekend....both did great. One placed in New Jersey high school states (Robert Pletcher, he took 8th at 215 lbs) and one in college eastern regionals, EIWA (Matthew Pletcher, he took fourth at 165 lbs).

In terms of the use of assembling an application, I thought of both assembling and composing . Composing has a number of other meanings, so I chose assembling. I'm open to better suggestions, given the context, I thought it was clearly talking at a higher level than .NET assemblies but I'm open to other suggestions. We are frequently struggling with appropriate and precise language that doesn't conflict with other terms it seems with our SharePoint guidance.

I believe it makes sense to keep the activity I am calling assembly (constructing the application in the browser and with SharePoint Designer) as separate activity from developing code. At the end of the day both activities lead to a delivered business application, but how those applications are packaged, deployed, managed, maintained, as well as the scope of impact, depends upon whether the applications are developed using a coding methodology and delivered to production via a wsp, vs. put together in pieces that are retained in the content database. I am shying away from calling both activities development, since I think to many development implies writing code. I do think both activities are about building business applications. I'd like to hear other thoughts on this area.

We've strugged with the word application, and I differentiate here between something that is intended to be a complete solution (an application) from something that is intended to be incorporated (assembled) as a capability within an application. I use application and components to try and draw that out. By an application I mean that the entire project will be treated as a single monolithic conceptual grouping. A component is intended to be used in application development or assembled into an application. Honestly, my preferred term would be solution, but that then gets confused with a SharePoint Solution (wsp). We have been careful in guidance to not overload the terms feature and solution. We could use more feedback to help refine these terms.

Please keep feedback coming, we'd like to get this discussion to a point where it is easier and more concise to understand. Also I'd like to hear more about if there are obvious holes in the scenarios listed below - what are the cases that are missing, and where do we need more granularity?

SharePoint and ALM – application types/classification

chriskeyser — Fri, 06 Mar 2009 19:12:28 GMT

I’ve been doing some work on aspects of application lifecycle management with our version 2 SharePoint guidance currently in development. In version 1 of our published guidance, we looked at some ALM aspects with managing code developed for SharePoint, in particular we looked at team development, packaging, factoring, environment setup, and process considerations. We also covered upgrade, which has a set of challenges in SharePoint due to the power of the framework, in particular the ability to customize the application after developed assets are deployed.

We’re now starting to dig deeper into this arena with different types of SharePoint applications – taking a look beyond the development and management of a packaged application into what we are calling content driven applications. Really what we mean by this is moving into applications which are less structured from a development perspective, that mix developed assets, assembled applications, and generated content.

We are defining application assembly as the process of building an application by composing parts using the browser and SharePoint designer. SharePoint has a powerful model for assembling applications, a key reason for its huge popularity with information workers. But once we move into this arena we introduce new complexities into ALM:

How do I manage the lifecycle now of these assembled applications?
- How do I deploy the applications
- How do I maintain the applications
- How do I upgrade the applications
- How do I preserve customizations
Where do I build these sort of applications?
- In production?
- In authoring?
- In staging?
How do I test these applications
- When do I need to test the applications?
- Where do I test developed components vs. assembled components?
- How do I make sure that the dependences of my developed components are explicitly managed (not dependent upon authored definitions?)
- How do I manage differences in scope between developed components (farm or web application level) vs. assembled application logic (site collection/site level)?

I came up with this table to try and get some shape around the different types of scenarios to frame the different demands from an ALM viewpoint based upon the intent of the application. I’m looking for feedback – does this make sense, or is it hogwash? If it makes sense, does it seem complete enough, or are there missing aspects?

Title	Description
Unmanaged assembled application	Assembled using browser and SPD Built and maintained in production environment Single instance site Never packaged and reused
Non-upgradable assembled application (managed)	Assembled using browser and SPD Multiple site instances Does not support upgrade on change – new versions only apply to new sites May be assembled in a staging/authoring environment, or directly in production Packaged and reused with a site template
Upgradable assembled application (managed)	Assembled using browser and SPD Multiple site instances Supports upgrade on change – new versions can be applied to existing sites Assembled in a staging/authoring environment Packaged and reused with a site template
Upgradeable assembled published application (managed)	Assembled using browser and SPD One or more site instances Typically read only functionality for end users Supports upgrade on change from authoring to production. Production changes and data not preserved. May be assembled in a staging/authoring environment, or directly in production Packaged and deployed to sites using content deployment
Non-upgradeable developed application	An application that is developed and fully encapsulated in one or more WSPs. Supports multiple instances Does not upgrade deployed instances on change
Upgradeable developed application	An application that is developed and fully encapsulated in one or more WSPs. Supports multiple instances Upgrades deployed instances on change
Non-upgradable developed component	A component that is developed and fully encapsulated in one or more WSPs. A component is capability used within a constructed application New versions of the component will not upgrade existing instances.
Upgradeable developed component	A component that is developed and fully encapsulated in one or more WSPs. A component is capability used within a constructed application New versions of the component will upgrade existing instances.

I'm back...

chriskeyser — Fri, 12 Dec 2008 02:11:00 GMT

I think I must have set a record for the longest period of blog inactivity. Boy, could not believe its been since 2004, time sure does fly. I've been "meaning to" get back to blogging for a few years now, but you know what they say about good intentions...

I've moved over recently to patterns and practices after spending 18 months in Office as the Group Program Manager for Duet. I made the move for personal reasons as I'm going to eventually end back up on the east coast if the real estate market decides to cooperate. Fortunately patterns and practices doesn't mind distribute development, and we do it on a number of our projects - our dev manager, Ade Miller, even recently delivered a white paper on distributed agile practices in p&p. We are doing a lot of great stuff in Duet and I learned a lot in my role - pretty awe inspiring to see how the machinery of a huge development organization like Office works effectively. But now that I've moved to p&p, I've really enjoyed the change back to a small agile team and to start getting my hands dirty again. I had a chance to work as an IC on the first release of SharePoint guidance, and will be the program manager for the 2nd release.

I will blog regularly about our experiences and architecture issues as we move into the next release. We'll be kicking off development on that release in January. You can find the V1 release at http://www.microsoft.com/spg. We'll be focusing on three major areas: building enterprise class sharepoint applications (focusing on scale and supportability), content oriented SharePoint application (publishing, authoring, and composition), and integrating line of business data with SharePoint. Our codeplex project is at www.codeplex.com/spg, and we'll be doing bi-weekly drops for review and input once we get moving as is typical for a p&p project.

Talk to you soon!

Source code for SCT tokens in a Farm source code

chriskeyser — Wed, 01 Dec 2004 01:44:00 GMT

I had a question on the source code for this sample. The link is in the article tucked near the top. Here it is too:

http://download.microsoft.com/download/3/c/2/3c2c3aee-d3de-4b9f-94db-a1580cd51da8/Code-TokenCache.msi

This particular sample needs to be installed to the same directory as the WSE samples. There should be an updated version soon that removes this dependency, and a readme file describes the necessary setup steps that is installed with sample. Note that you will also need to install the sample WSE certs for this to run. Instructions can be found in the WSE documentation (default location is: Program Files\Microsoft WSE\v2.0\Samples\CS\QuickStart)

Article is live on MSDN

chriskeyser — Tue, 23 Nov 2004 23:01:00 GMT

I finally got an article on SCTs in a web farm live on MSDN. The sample code needs to be installed the the same directory as the WSE samples (default C:\Program Files\Microsoft WSE\v2.0\Samples\CS\QuickStart). There is an update that should be posted soon that removes this dependency. You can find the article here:

http://msdn.microsoft.com/webservices/default.aspx?pull=/library/en-us/dnwebsrv/html/sctinfarm.asp

Code for SCT implementation in a farm

chriskeyser — Wed, 29 Sep 2004 18:57:00 GMT

Sorry I've been dark for a few weeks - I've been in the process of trying to get my blogs and sample code on SCT's in a web farm packaged up and released to MSDN. I hope to have it out there in a couple of weeks. I will be getting back into my blog next week on something new.

SCT Options - Using a cookie approach with SCT's in a farm

chriskeyser — Mon, 06 Sep 2004 06:07:00 GMT

So far I've discussed using two techniques for using SCT's in a web farm, pinning and using a database as a caching mechanism. The third and final technique I'm going to cover is using a cookie-like approach of embedding the information in the extended area of the SCT.

Advantages of this technique:

Resilent to server failure
Does not require any state to be maintained on the farm - operationally simple
Cost - lower cost when compared to database cache as it doesn't require a redundant database to be in place to maintian state.

Disadvantages

Increased network bandwidth costs
Increased processing - due to processing extended data in payload (this needs to be parsed, even if it isn't decrypted/deserialized). I did not measure the cost of doing the database retrieval vs. the cost of processing the cookie for cases where the SCT needs to be re-hydrated if its not in cache - my gut feel is that its probably close to a wash..
Requires key management on the server side (key used to encrypt/decrypt the serialized infomration in the cookie)

One of the advantages of using an SCT is that it is singificantly lighter weight than other token types because it relies upon client and server maintaining state. The schema for the SCT can be viewed here: http://schemas.xmlsoap.org/ws/2004/04/sc/ where one can see that the contents of the SCT is simply an identifier, followed by any content. Using a cookie technique we put the necessary information to reconstitute the token here. This approach definitely compromises one of the advantages of an SCT over other token times since we bloat the SCT somewhat. However we can still achieve significantly better performance, and reduce most of the processing overhead (other than managing the network traffic and parsing the additional area in the DOM) to a one time hit that is still sigificantly less costly than processing the equivalent tokens. A big part of the cost of processing an XML token is xml signatures and encryption. In order to perform these operations, a process of cononicalization needs to occur (http://www.w3.org/TR/xml-exc-c14n/) that is very expensive to perform. We'll avoid this cost and be able to more compactly represent the token by using binary encryption and signing. While I won't do it here, compression can also be used to reduce this payload. Finally, while I've implemented a generic technique of getting the XML representation of the base token since that's the only generic approach currently available to serialize/deserialize a token, it would be much better to perform custom serialization to a more compact representation if you always know your base token type.

Serialization I've described in previous blogs and that remains the same process as when serializing the token for persistence to a database. I did re-factor my sample to move the serialization logic into its own class out of the token cache class so it was re-usable by the cookie logic.

I had to make a few additional changes to existing DistributedSCTTokenManager . First I added a mode to my with three enumerated values - database, cookie, and none. I then modified the PersistSecurityToken to add the cookie to the SCT if in cookie mode - this amounts to adding a single additional element to the extensibility section of the SCT, and putting the binary serialized, encrypted, and signed SCT information base64 encoded within that element.

  public void PersistSecurityToken(string identifier, SecurityContextToken token)
  {
     if(_cacheMode == CacheMode.database)
     {
         ((DistributedTokenCache)_tokenCache).PersistSecurityToken(identifier, token);
     }
     else if(_cacheMode == CacheMode.cookie)
     {
       AddCookieToSCT(token);
     }
  }

  private void AddCookieToSCT(SecurityContextToken token)
  {
     byte []data = _serializer.SerializeToBytes(token);
     byte []encryptedData = _encryptor.SignAndEncrypt(data);
     string serializedAndEncryptedData = Convert.ToBase64String(encryptedData);

XmlDocument doc = new XmlDocument();

     XmlElement extendedSCT = doc.CreateElement(TokenNames.Prefix,
            TokenNames.SCT.ExtensionElement, TokenNames.Namespace);
     extendedSCT.InnerText = serializedAndEncryptedData;
     token.AnyElements.Add(extendedSCT);
  }

Finally I overrode the LoadTokenFromXml method. In this method, I first just do the base method LoadTokenFromXml, try to retrieve the SCT from the local cache, and if I fail to find it, I then decrypt, validate, and load the SCT from the cookie information, and add the SCT to the local cache.

  public override SecurityToken LoadTokenFromXml(System.Xml.XmlElement element)
  {
     SecurityContextToken sct = base.LoadTokenFromXml(element) as SecurityContextToken;

     SecurityContextToken tokenToReturn = TokenCache[sct.Identifier] as SecurityContextToken;
     //do this in two stages for performance if token contains state- only load if not in cache
    if(tokenToReturn == null)
     {
         if(_cacheMode == CacheMode.cookie)
        {
            tokenToReturn = LoadSctFromCookie(element, sct.Identifier);
           TokenCache.Add(sct.Identifier, tokenToReturn);
       }
     }
     return tokenToReturn;
  }.

  private SecurityContextToken LoadSctFromCookie(System.Xml.XmlElement element, string identifier)
  {
     string serializedAndEncryptedData = null;
     SecurityContextToken sct = null;

     foreach(XmlElement childNode in element.ChildNodes)
     {
         if (childNode.LocalName == TokenNames.SCT.ExtensionElement &&
            childNode.NamespaceURI == TokenNames.Namespace)
         {
              serializedAndEncryptedData = childNode.InnerText;
              break;
         }
     }

     if(serializedAndEncryptedData != null)
     {
        byte []encryptedData = Convert.FromBase64String(serializedAndEncryptedData);

        byte []serializedData = _encryptor.DecryptAndValidate(encryptedData);
        sct = _serializer.DeserializeSCT(identifier, serializedData);
     }
     else
     {
        //Should not happenl.
        throw new System.Web.Services.Protocols.SoapException("Server does not support distributed cache for SCTs",
       System.Web.Services.Protocols.SoapException.ServerFaultCode);
     }
     return sct;
  }

This one I had tried before, so it came up and running pretty quickly. Well that's it for SCT's. I think I'll try something a little more abstract next blog...maybe service factoring, which is something I've struggled with over the last year.

SCT Options - Using a database to cache SCT's in a farm, final chapter

chriskeyser — Wed, 01 Sep 2004 05:06:00 GMT

OK, so here I am in my hotel room ready to rip off a really long blog entry to close out caching SCT’s in a database, and I can’t get to the internet. Go figure. I’ll have to wing where I left off from memory.

Last discussion I reviewed serializing a token, and left off with my first assumption proven wrong – that the token was complete when added to the cache, so I could just use this event to go ahead and also persist the token to the database. Unfortunately there is no event to hook that would key me into when the token is complete and can be persisted. An interchange with an excellent dev on the WSE team suggested overriding the security context token service to achieve the desired behavior. So while the SecurityContextTokenManager generally handles all token handling items of interest once a token is created, the SecurityContextTokenService is actually responsible for creating, populating, and issuing the SecurityContextToken. A simple override does the trick in this case:

public class DistributedCacheSCTService: SecurityContextTokenService
{
  protected override RequestSecurityTokenResponse IssueSecurityToken(SecurityTokenMessage request)
  {
     // invoke the base implementation
     RequestSecurityTokenResponse response = base.IssueSecurityToken(request);

     // now filled up the SCT's properties
     base.SetupIssuedToken(request, response);

     // now cache the fully-fledged SCT again
     ISecurityTokenManager mgr = SecurityTokenManager.GetSecurityTokenManagerByTokenType(WSTrust.TokenTypes.SecurityContextToken);
    SecurityContextToken token = response.RequestedSecurityToken.SecurityToken as SecurityContextToken;

    DistributedCacheSCTManager sctMgr = (DistributedCacheSCTManager)mgr;
    sctMgr.PersistSecurityToken(token.Identifier,
    response.RequestedSecurityToken.SecurityToken);

return response;
}
}

Essentially I’m doing the same functionality as the default implementation, only I’m also telling the manager I derived from SecurityContextTokenManager previously to persist the token once its been setup, rather then persisting when its been added to the cache. WSE also needs to be configured to use the DistributedCacheSCTService for issuing SCTs, which is achieved through configuration:

The good news is that now I did get a complete token serialized to the cache. Bad news is when I deserialized the token, I got a new security fault. As it turns out the base token was a UsernameToken, which has a nonce value. WSE tracks the nonce values and detects reuse of a nonce, which indicates a highly probable replay attack. While I am “rehydrating” the token with the LoadTokenFromXml method on the token manager, the manager actually thinks its loading the token from a message. Since the nonce has been used before, it throws a security fault. The problem here is that I was using an artificial means to test my caching logic by not putting the token in my local in-memory cache. Typically what would have occurred is that if the manager had seen it before, the token would already be cached in memory. Otherwise it would load it back from the database, and since the replay detection also relies upon an in-memory cache of nonces, it will not cause a problem on the second service instance in the farm. In any case, I was able to circumvent this issue by one more configuration setting, this one for the UsernameTokenManager

</securityTokenManager>

This did enable me to deserialize the SCT and base token. I figured I was out of the woods with my fully re-constituted token. Not so lucky. WSE ended up throwing one more exception indicating that it could not locate the security token. Now what? Well this one had me stumped, but I was fortunate enough to corner the same excellent WSE developer who helped me work through the best approach for getting around the incomplete token issue. As it ends up we discovered that there is a "feature" with the current SCT implementation that requires the token Id (which is different than the identifier) to be consistent across messages. Typically the token id should only need to be consistent within a particular message, while the SCT identifier is consistent across all messages. The simple work around was to add Id to the serialized SCT information, and to set the original value when the token is re-hydrated. Once I did this, the sample worked.

I was so excited about actually writing some code that I neglected to discuss the advantages and disadvantages of this approach. The advantages of the database cache approach are:

1) Minimizes network transport overhead (when compared to the ‘cookie’ method to be discussed next)

2) Resilient to service instance failures in the farm

3) Low performance overhead – slightly higher than using pinning, but only since local caching of tokens is still used, it’s only once per token per service instance maximum. This is a slight performance hit over the session pinning approach.

The disadvantages are:

1) Operational complexity is highest of all approaches. Database has to be maintained for caching. If a session database is already being maintained, then this could piggy back off of the existing session database.

2) Security – if the data written to the database is not encrypted, it could be compromised by someone who can sniff the network

3) Cost – database needs to be available for caching. For high availability systems, this would need to be a clustered database since an outage would take out all services that rely on SCT’s in your farm for security.

Next blog I’ll start covering the approach I prefer because it gives operational simplicity with resiliency at the cost of network and processing overhead. This uses a conceptual model near and dear to all web developers – using cookies as a means to manage session state rather than relying upon you backend services using stateful sessions. In this case though the state information is embedded within the extensibility area of the SCT itself.

SCT Options - Using a database to cache SCT's in a farm, Volume 2

chriskeyser — Mon, 30 Aug 2004 04:39:00 GMT

Last entry I discussed how to extend the SCT manager, and MRUSecurityTokenCache to use a database to persist tokens to share SCT's across service instances within a web farm. I left the details of token serialization not implemented however. This blog I'll dig into that with a generic method of serializing and deserializing tokens. I'd like to note that it would be more efficient generally if you have control over the base token to use a more efficient mechanism to serialize the token.

Tokens are unfortunately not serializable. There are a few reasons this design choice was made - the downer is that efficient token serialization is not as simple as it could be. There are two tokens we need to be concerned with - the first is the SCT itself, and the second is the "base token" of the SCT. I discussed the base token in a previous blog but realized that I wasn't particularly clear about where it came from. When estabilshing an SCT, a token is provided by the client to prove its authenticity and estabilish identity before the SCT will be issued. In the case of the secure conversation example, a UsernameToken is used for client credentials for the SCT request. If the credentials are accepted, then this token becomes the "base" token of the SCT. The SCT itself is generally very lightwieght, while the base token typically contains identity and perhaps authorization related information for the authenticated client.

So the bottom line is that we need to serialize both of these. The approach I took assumes that generally tokens that are not SCT's are stateless - i.e., they can be reconstituted based upon the information contained within the token xml, and perhaps a secret know on each service - for instance, the private key of a public cert installed on all servers in the farm, or a symmetric key shared in the configuraiton accross all servers in the farm. The method we are going to work on is one added to the DistributedTokenCache described in my last entry:

 provtected virtual void SaveToStore(SecurityToken token)       {
               … persist the token
       }
 
I'm going to describe how I implemented this for SCT's. One could generalize this for other tokens, but I haven't yet figured
out a valid use case for this need. For the base tokens, I'm going to use a method every token must implement to 
serialize itself to xml, GetXml, and to deserialize the token on reload using the appropriate token manager's 
LoadTokenFromXml method.
 
For the SCT, define a simple class to contain the relevant information from the SCT. For my purposes, I defined the
following as sufficient, altthough there are additional items you could push off - for instance, if you were using the
"AppliesTo" value for some of your validation, you would want to add that information to this class:
 
[Serializable] private class SCTData
{
 public byte[] KeyBytes;
 public string TokenIssuer;
 public DateTime Created;
 public DateTime Expires;
 public string BaseTokenType;
 public string SerializedBaseToken;
}
 
I set the relevant data values from the SCT being seriailzed, (Created and Expired come from Lifetime which is not
serializable). SerializedBaseToken is a string containing the XML retrieved from the base token using GetXml. I then
serialize this class into a byte stream and store it to the database along with the associated identifier (used to retrieve the
entry on deserialization). I also added a column where I put the current datetime so I can clean up the store later.
My first thought was to use Expires, but unfortunately Expires is a few thousand years in the future by default.
 
I created a somewhat artficial test case to see if this logic worked by skipping placing the token in the cache when first
added and expected to see everything run beautifully.. Well it didn't. I got a SecurityFault and a decryption error. I 
started digging, and I found out that the SCT wasn't in fact complete when it was added to the cache, so I was serializing 
an incomplete token to the cache database, which was then of course incomplete when de-serialized on the next
request. The base token wasn't set, nor the TokenIssuer (as well as a few other properties). Next entry I'll discuss 
how I fixed this particular issue (and two others) to get the cache running.

Using a database cache to share SCTs in a farm, volume 1

chriskeyser — Thu, 26 Aug 2004 17:08:00 GMT

Sorry I’ve gone dark for the last few days just as things were getting interesting and I had promised some code snippets. I decided to actually implement a solution using a distributed cache for SCT’s – I had implemented a proof-of-concept embedding state in the SCT already, so I assumed that doing a distributed cache approach would be even easier extrapolation of this work…well you know what the say about assumptions. In any case, it’s made this an interesting enough problem that I think I’ll write it up for MSDN in the future.

OK, well on to how to do a distributed approach. Last time I talked about the approach of using some sort of session affinity mechanism to ensure the client was always routed to the same service instance. This was the simplest approach, but had a number of drawbacks. The second approach we’ll examine is using a database cache.

Now to how we achieve this approach? My first theory was to override the token cache in the security token manager for the SCT to handle this situation, and write a new type of cache class for handling SCT's. In WSE, for each type of token handled there is an associated specialized security token manager that knows how to handle pretty much all aspects of managing that token type that is registered via the config file with the WSE runtime. The only requirement for a security token manager is that they implement the ISecurityTokenManager interface, however there is also a base SecurityTokenManager implementation that would more commonly be used if you are implementing your own custom tokens. In this case we don’t even need to do that – we really just need to derive from the SecurityContextTokenManager class, and override the TokenCache property. This part of the implementation is pretty trivial:

public class DistfibutedCacheSCTManager: SecurityContextTokenManager

{

DistributedTokenCache _cache;

public DistributedCacheSCTManager(): base()

{
_cache = new DistributedTokenCache();

}

protected override ISecurityTokenCache TokenCache

{

get

{

return _cache;

}

Then we define a distributed cache. In order to do this, I just derived from the WSE MRUSecurityTokenCache class (MRU == Most Recently Used). We still want to cache the token locally in addition to persisting it to a central database cache. This way we only have one database hit per SCT per service instance in the web farm maximum for each SCT that is established with a client (this assumes the token is never flushed from a local cache).

public class DistributedTokenCache: MRUSecurityTokenCache

{

public DistributedTokenCache(int capacity): base(capacity)

{

}

public override void Add(string identifier, SecurityToken token)

{

if(base.Contains(identifier) == false)

{

base.Add(identifier, token);

SaveToStore(token);

}

public override SecurityToken this[string identifier]

{

get

{

SecurityToken token = base[identifier];

If(token == null)

{

token = ReadFromStore(identifier);

if(token != null)

base.Add(token);

}

return token;

public override bool Contains(string identifier)

{

bool containsToken = false;

containsToken = base.Contains(identifier) ;

if(containsToken == false)

{

SecurityToken token = ReadFromStore(identifier);

if(token != null)

{

containsToken = true;

base.Add(identifier, token);

}

return containsToken;

provtected virtual void SaveToStore(SecurityToken token)

{

… persist the token

}

protected virtual SecurityToken ReadFromStore(identifier string)

{

… load the token

}

In order to use the derived security context token manager, you need to register the manager with WSE. This is a simple config file entry:

Next entry I’ll go into detail about saving and loading the token back from the database cache, and cover the first unanticipated issues I encountered with the implementation.

SCT Options - Pinning a client session to a service instance

chriskeyser — Fri, 20 Aug 2004 02:20:00 GMT

Most web developers are familiar with one way with maintaining state within a web farm from their ASP days...server affinity. This was really the only optional available before .NET other than writing your own distributed session management solution for web applications if you wanted to maintain server side session state. This approach can also be used in the case of SCTs. Since most web services today are typically accessed via http, this logic is already built into load balancing hardware, and often these balancers will setup affinity for a particular service instance based on such values as client IP address or a cookie value, such as session ID. A common term used to describe this is "pinning" the session to a server instance. This may be a perfectly adequate solution for your service, and this is certainly the easiest option to implement. There is no additional development work required to implement this approach - typically just flip a switch in your load balancer, and your off to the races. The advantages of this approach are:

Easiest to implement - no code required
The most performant solution (assuming that it doesn't introduce significant load imbalance issues that are possible when pinning occurs)
Simplest solution to support operationally
Supports all base token types efficiently (more on this below)

The disadvantages of this solution are:

Not resielent to server failures. Since the client is pinned to a particular instance, if the server goes down, the SCT is no longer valid. If the message subsequently gets rerouted to an available server, it will be rejected. This could be compensated for by logic in a smart client side proxy.
Can introduce imbalances in loading - this is a typical problem with pinning a session to a particular instance
Reliant upon some type of session affinity management. This is typical available with http load balancers, but may not be as likely of a feature for other load balancing solutions.
This is a point-to-point approach. Messages transiting multiple hops won't have an http connection directly between the client and service.

One of the advantages mentioned above is supports all types of base tokens efficiently. The limitiation with the other two approaches mentioned that I will cover in the future is that they require the base token information to be serialized. Since the WSE SecurityToken class is not serialable, no derived class can not be serializable. There is an out...you should typically at least be able to retrieve the XML representation of a token with GetXml and LoadXml, but this adds significant additional memory and processing costs over binary serialization for the cached token. A technique to mitigate this cost that will be covered later is to cache the token locally. Therefore the deserialization only needs to occur if the token is not in the local cache - which means only once per token per service instance (assuming that the token doesn't get flushed from a full cache).

For a high reliability system, this approach would not be the best choice. While choosing the simplest solution that meets your needs is almost always the best choice, I am bothered by the fact that this relies on out-of-band information for session affinity and requires point to point communications that seesms to violate the spirit of messaging within SOA. But then I think maybe I'm being too much of a purist...

Tomorrow I'll start covering the distributed cache approach, where I'll start digging into some code samples.

Why do we need to manage state when using a Security Context Token, and what are the implications?

chriskeyser — Wed, 18 Aug 2004 17:58:00 GMT

A security context token is a lightweight token that can be established for multiple message exchanges between two endpoints. It only makes sense to establish an SCT if it is anticipated that multiple messages will be exchanged. The specification for this feature can be found here and builds on both ws-security and ws-trust. You’ll notice that the defined payload of the token is very light – in fact, only one element mandatory element is defined, Identifier, for an SCT. While a third party approach to issuing SCT’s is not prohibited, the model that WSE supports out of the box and we anticipate will be predominantly used is based on a conversation between two endpoints. In fact, WSE has a really cool feature, auto issuing (configured with < autoIssueSecurityContextToken>) where the framework picks up an incoming SCT request issued directly to the service endpoint and handles it.

If you’ve seen what various security token looks like, you know that they can be pretty large, and the associated transport and parsing logic can add measurable overhead – for instance, a Kerberos based token can be well over 1K in size. SCT avoids this overhead by relying on the client and service retaining the key information as well as BaseToken used in originally establishing the secure context. An exchange has already occurred to authenticate and establish identity when an SCT is requested that has established the identity and credentials of the client. Once established, the client and service use the identifier in the SCT to pull the key and base token information back out of a cache for validating the message information and establishing the identity.

The implication here of course is that state is saved on both the client and service side that allows the parties to more efficiently manage security. However when your service is deployed to a web farm, this assumption on the server side needs to be addressed. Several ways to deal with this issue are:

1) Pin the client session to a particular server instance hosting the service in the farm. Most load balancers have an ability to route incoming requests to the same server instance based upon some form of session affinity.

2) Support a distributed cache on the server side for sharing the SCT information across the farm.

3) Put the state information for the SCT into the extensibility area of the SCT.

I will examine all three options over the next few blog entries.

Kickoff for a procrastinator - are you a top down, middle out, or bottoms up kind of person?

chriskeyser — Tue, 17 Aug 2004 01:09:00 GMT

I'm a software architect with the Archtecture Strategy group at Microsoft. I've been working up to starting a blog for some time now, and have finally run out of excuses to put it off any longer. Mind you I like blogs and gets lots of value out of what they have to say...just been waiting for that ultimate nugget of wisdom I could impart on everyone that would convince them that despite my lousy spelling, I really do have something worthwhile to say. Well I've given up on discovering that one and decided to just plow ahead.

I'm fairly new at Microsoft - I just crossed over my one year boundary. Its been a tremendous learning experience for me. I have a varied background - having worked in embedded systems, consulting and a string of startups in the last 15 years in software development. I spent five years out of college in the Navy as an officer in the nuclear power program. I'm always looking for new challenges, and working at Microsoft definitely fits into that category - I am most fortunate to be associated with a great team of non-egotistical really smart people. Boy, how often does that happen?

One of the challenges that I've encountered here in this group is thinking in the abstract. I always thought I was good at abstract thinking until I came here. . Now I think I'm still pretty good at drawing abstractions out of one or more concrete problems, refactoring those abstractions as the system evolves - that I've done all my career - but taking it to the next level just isn't the way my brain works. I need that concrete problem or set of problems to start with. If I can't draw the line from the abstraction to some vague implementation approaches in my mind, then it just doesn't work for me. I think it's kind of like the difference between an innovator and an inventor. Someone like Pat Hellend who can generalize in the abstract and come up with an elegant model of how it ought to be referring back to a very general problem space is amazing to work with - and it all makes sense to me when he presents it and applies it against specific instances - although elements of his thinking in some cases is beyond what technology can effectively deliver today, pointing to what we need to plan and strive to accomplish tomorrow (I'm sure anyone reading my lowly blog has perused Pat's long ago). After the last year of working within this group, if I hadn't had prior experience working with bottom's up person I would think that I must be one of those. But I have definitely been there working with a few bottoms up people. That's the person that when I arrive at some moderately elegant system architecture diagram with some pretty pictures in PowerPoint and Visio catches the few key leaps of faith I made when I doing my hand waving around my vague idea of implementation, and has saved my rear-end more than once. Now that I interface with lots of top down thinkers, I seem to be feeling that bottom-up thinker's pain as I struggle to draw the lines, but I'm definitely getting there as I'm being stretched. Thus I decided I must be a middle out kind of architect. If you ask me how I would realize my ideas, I'll have some semblence of how that may be translated to actual implementation. I'd like to think I've brought some value to the group as a result by being the pestering guy always asking the questions - kind of like my bottom's up compadre always did to me - and helping keep everyone honest. The most effective organizations have characters with each of these skills IMHO. Those that can traverse all of these levels effectively are few and far between. I had a chance to interact recently with Jack Greenfield, and I really like the thinking going on with domain specific langauages and software factories. This will give us a more formal and effective mechanisms for transcending these levels of abstractions and help reduce this friction.

Anyway, that was my soft, non-controversial touch-feely kind of introduction to convince you I'm a nice guy. From now on I'll try to get down to the really interesting (and more concrete) issues and avoid any further speculation about abstract issues ;-). To start with, in the tradition of being middle out, I'm going to do my first series of blogs starting this week based on managing Security Context Token's in a load balanced web farm with WSE2, discussing first the options within a farm for this problem, pluses and minuses of each approch, then dive into some code snippets. Since I hate to take credit for work someone else has done, I want to acknoweldge up front that the thought leadership for this work came out of a collaboration I had with Fred Chong, an excellent architect also in the Architecture Strategy group, and is based primarily on elements of a proposal that Fred generated. Stay tuned if you are interested...