Chaos Fosters Opportunity

For the last 6 or so years, I have been working almost exclusively on technologies related to data storage and retrieval.  This has included things like object persistence technology (Object Spaces), data warehousing for billion+ row databases, data analytics, and scalable, high-availability, database driven web sites (there maybe a buzzword bingo on that line alone).

I recently changed teams to help address performance and scalability challenges associated with a successful set of social features used across multiple Microsoft web sites, like MSDN and TechNet.  If your not familiar with these features, here are a few links to explore where some of these features can be found:

Forums

• Forums on Microsoft
• Forums on MSDN
• Forums on TechNet
• Forums on Expression

Search

• Expressions
• MSDN
• TechNet

Tags, Bookmarks and Profiles

• Social on Microsoft
• Social on MSDN
• Social on TechNet

 As you explore these social features ask yourself how big the data stores need to be to support something like this.  I can tell you it isn't small.  I can also tell you it isn't 'big' (like http://www.live.com big).  It is kind of in the middle.  Its big enough to require special attention to scaling out and performance, but no so big as to warrant vast amounts of specially written data storage and its associated technology.  

Over the next few posts, I will explain some of the things that we did to scale our system up and out, challenges we had along the way, ideas on where it will go in the future and some considerations you might make if you take the same trip.