Quick heads up. Eugenio has a blog posting on his experience migrating an app to SDS. The part about the posting that I like the most is his developing locally on SQL Express and just changing his connection string. Check it out here.
-Dave
Hey Everyone,
I wanted to give a quick update. Things on the team are cranking. There is a definite excitement in the air. We just started Sprint 8 and are getting ready for TechEd North America which is in LA and starts on May 11th. We are planning three sessions for TechEd this year. We will be presenting a What’s New in SDS session, a Roles and Responsibilities of managing a SDS database session, and my favorite, Developing apps with SDS and Windows Azure. I’ll post more details on the sessions in a couple weeks.
I am really looking forward to talking with people at TechEd. My thoughts are that by the time TechEd rolls around, people will have transitioned from the “Wow!!!” to the “How???” and should have some good questions and scenarios that we can help them enable. If there is anything specific you would like to see, please let me know.
Also, anyone planning on coming to TechEd, drop me an email @ david.robinson@microsoft.com. I’d love to chat.
-Dave
Hey All,
Its been 3 or 4 months since I blogged and I am getting super excited to start showing the community the forthcoming enhancements to SDS. Mix is only a handful of days away and that will be the first public, non super-duper NDA, demo of the new SDS features. I figured it would be good to take Live Writer off the shelf, wipe the dust off and get ready to start blogging...
I have to keep this short. My wife is out shopping and I am home with the kids and World War III is about to break out between the twins over a sippy cup, so I better intervene.
More info coming soon,
-Dave
Hey All,
Got back from PDC a few days ago and now I am sitting at SeaTac airport waiting for my flight to Barcelona for TechEd EMEA. I figured it was time to start sharing some of the stuff I showed at PDC...
Today we are going to talk about the performance gains you can see when you run you queries in parallel. The two points I am always trying to drive home are around partitioning and running your operations in parallel.
I have been talking about this for about 9 months and I figured for PDC it would be better to demonstrate what I have been talking about.
Inside a test Authority I created 26 containers - one for each letter in the alphabet. I then loaded up 200,000 fictitious users and partitioned the data based upon the first character of the user's last name. No big deal - what kind of developer doesn't have a fictitious name generator :)
I then wrote a quick winform app show below
The app lets you enter in a user's first name and the number of worker threads. When you click begin it will use the thread pool and queue up 26 queries (one for each letter in the alphabet) and when all queries have completed, it will show the number of records returned and the time it took...
Lets give it a spin...
Searching for the first name of Jacob across 26 containers returns 9495 entities in 7.72 seconds...Not too shabby. The reason that the thread lower bound is 4 is because the I ran this from my home machine which has 2 dual core processors and the lowest you can set the Max Worker Threads to is equal to the number of processors (the OS views each core as a processor). I intentionally ran this from home to insure I wasn't on the Microsoft network. I have cable Internet from Comcast.
Now lets try with 10 threads...
Not bad 9495 entities in 4.71 seconds...Let bump it up to 26 threads so all queries get processed concurrently...
9495 records in 2.94 seconds....WOW!!!
If we think about what just happened we launched 26 queries in parallel. Chances are each query got processed by a separate front-end server. Since I partitioned my data across 26 containers, chances are each container ended up on a separate back-end node. That means I most likely had 26 different front end servers querying 26 different back-end database nodes and returning the results (not just the counts, but the actual entities) to my test application. When all threads had completed the app aggregated all the results into a List<Person> and displayed the count. I keep saying chances are since I haven't looked to see on what nodes my containers ended up on. Similarly I don't know what front-ends processed my request.
Hopefully this little demonstration has conveyed how partitioning and multithreading your SDS calls can really provide some major performance benefits.
Let me know if you have any questions or comments,
Dave
Ok...So today is PDC2008 Pre-Con day. Just a quick heads up...
You will see some guys walking in and around the LA Convention Center with nice decked out video equiptment. Those are the Channel 9 guys.
You might also see some idiot walking around with a backpack and an average run of the mill HandyCam. That would be me :) What am I doing? Gathering data for my presentation and demo of course. What is the demo you ask? Well, I can't go into it right now. I wouldn't want to spoil the surprise. I can tell you its uber nerdy. I am married with 4 kids, no need to act cool anymore - EMBRACE YOUR INNER GEEK!!!
So, you want to see what it is? You have 2 options. Come to my talk on Tuesday @ 12:45, or wait until after my talk and I'll start posting details here.
So if you see me walking around, grab me and say Hi.
-Dave
Tommorow afternoon, I get on a plane and head down to LA for PDC2008. I have to be completely honest. We have some really super, pimped out, mack daddy, completely awesome stuff coming. Both in the teams presentations and in new features for Sprint 5. I just sent out the new features list to some internal softies for "Word Smithing" which you all should see in a few days, and the list is long and chock full o' goodies.
Thats all for now. If I keep typing I am bound to leak something and will wind up ruining the suprise :)
-Dave
PS - I promise much more posting after Monday.
Hola!!!
Just a quick note. Sorry for the lack of posts as of late. We have all been heads down preparing for PDC.
I wanted to give a heads up to all the SSDS users on the other side of the pond. Myself and Istvan Cseri will be presenting at TechEd EMEA in Barcelona. For more information click here.
-Dave
So…
I have gotten pinged by a bunch of people asking if we were planning any evening events for SSDS users. So, If there is sufficient interest in such an outing, please let me know by shooting me an email at david.robinson@microsoft.com
I am getting super excited about PDC. It is going to be so cool. So much stuff to show and talk about…plus I have to tell you, I love hanging out with our users. You folks are so cool.
See you all soon,
Dave
Hey All,
One of the great things about SSDS is that we belong to a rich family of products. One of those products is SQL Server Integration Services – everybody’s favorite ETL tool. Well, thanks to some hard work by Matt Masson and his team, we now have an SSDS Adaptor for SSIS.
You can now use SSIS to take data from virtually anywhere and pump it into or out of SSDS. This is one of the things I showed during my TechEd talk, but now you can try it for yourself.
Matt has posted to source and binaries over on CodePlex.
Enjoy!!!!
Hey all
During Tech-Ed North America 2008, Soumitra, Niraj and I did an interview for Tech-Ed online. Lots of good info.
The video can be found here http://www.microsoft.com/techedonline/
-Dave
Here is a question. Myself and other members of the SSDS team are always asking the “What do you want to see” question. I am going to ask a similar question…
- What kind of examples would be helpful?
- Would you rather see code snippets or a fully working application?
- Would you rather see a console app? WinForm? ASP.Net? WPF? Silverlight?
- What type of app?
- What are the challenging scenarios that you are facing that you would like a reference application to help you solve?
Feel free to email me at david.robinson AT microsoft.com
Hey All,
I have been really quiet lately. We had a ton of great stuff hit when we rolled out the Sprint 3. I had all these plans for samples and blog posts, but the last two weeks I have been completely out of pocket with a sick child.
The good news is that I have been back for a few days and I am almost done digging out of email. So I am ready to start rocking and rolling :)
Hope everyone has been having a tremendous summer…
Dave
Hey Everyone,
Just a quick post. I have been on vacation for the past 10 days in New York visiting family and relaxing with my wife and kids. I am getting ready to head to the airport to fly home and I just started digging out of email. Luckily for me I have a 6 hour flight back to Seattle. We are flying Jet Blue so hopefully my kids (ages 8, 6 and 3 (twins)) will enjoy the complementary TV and let me get some solid work done. It’s kind of weird travelling with kids. When I travel by myself, I am the guy that prays before I step foot on the flight that I am not sitting next to any younger kids or babies. Don't get me wrong, I absolutely love kids, but I look forward to the distraction free time on a plane to get work done. My kids are used to flying so they are usually well behaved...Usually...anyone who has kids knows that at the drop of a hat, a three year old can turn from a sweet little thing to a demon. I know they have the term terrible twos, but my experience has been that three has always been the age to watch out for...Needless to say, I am keeping my fingers crossed.
Also everyone should have received my email blast last week from Microsoft Connect on the new Sprint 3 features coming. The text from that can be seen here. Those features are getting rolled out this upcoming Thursday and should be completed by Friday. Our super - duper - magnificent operations team has been phenomenal at these rollouts and although the product team works really hard to crank out these new features, it is the unsung operations heroes that deserve some serious kudos.
Hope everyone didn't miss me too much :)
Dave
In the first best practice, we looked at how we can gain maximum performance out of SSDS by partitioning our data across multiple containers. In this post, we are going to look at how we can take advantage of the many SSDS front end servers to maximize our applications performance.
SSDS is an Internet based data storage and query processing utility service. Lets face it, SSDS can only be as fast as your Internet connection. With that being said, when we architect our applications, we need to keep that in mind and be sure that we take whatever steps necessary to ensure that our application performs well. There are a handful of access and caching patterns we can use to ensure that any latency doesn't impact our applications performance. In this post we are going to concentrate on concurrency.
If we once again look at how SSDS is architected, we will see that we have many, many front end servers that expose the service via our SOAP and REST interfaces. This is the area in the diagram below in blue. In addition, those front end servers can accept many concurrent connections.
Those front end servers look at the operation you are performing, and route the request to the back end server that your container is located on. The back end servers are in the orange area in the diagram above.
To gain maximum performance from SSDS, don't just perform your operations in a singleton fashion. Issue those operations concurrently!!! If you architect your application so that it can make multiple simultaneous asynchronous calls, you can be assured that you are best utilizing SSDS and that your application will perform well - at least from a data access perspective ;)
One thing to watch out for....SSDS only supports a simple concurrency model today. Cases where you need to perform multiple entity updates as part of an single operation, this batch of updates needs to be managed in your application (SSDS doesn't support batched updates today, but its a feature we have in our plans). These operations can be multithreaded, but you need to ensure that you save both the original version of your entity and the updated version of the entity within your application. This way, should one of the operations fail, you can have your application perform any remediation necessary.
So to recap...
- Multithread your SSDS calls when appropriate
- Use Asynchronous access patterns
- Helps combat latency
- Many front-end servers able to handle many requests
- Things to watch out for
- SSDS Supports Simple Concurrency Today
- Multiple entity updates need to be managed by your application
- No support for batching today, but its coming
If anything is not clear, or if you have specific questions around your scenarios, let me know. I'd be happy to help.
-Dave
Just a quick July 4th post. My MSDN article is up. You can read it for yourself here.
Happy July 4th to those fellow U.S. citizens out there.
-Dave
