Brian Harry's blog

New Windows Dev Center release

Brian Harry MS — Thu, 16 May 2013 14:51:08 GMT

Yesterday we launched a new Windows Dev Center. You might ask “Why are you telling us about that, Brian?” Well, the answer is that my team is responsible for MSDN and much of the infrastructure behind it. We partner very closely with the Windows team team produce an awesome Windows Dev Center. Yesterday was the culmination of months of work to really overhaul the dev center and make it serve our customers even better.

Here’s a list of some of the key improvements in the dev center…

Navigation aligned with the development lifecycle
Updates to key landing pages, with more visual appeal and interactivity
A “Design” section for Windows Store apps that is more comprehensive, easy to use, and discoverable
A new “Market” section for Windows Store apps that includes tips for marketing apps, evaluating telemetry data, and passing app certification.
New cross-site navigation
An experience that matches the personality of Windows and is aligned with Windows online

Check it out…

Brian

Team Foundation Service Update – May 13

Brian Harry MS — Tue, 14 May 2013 15:01:13 GMT

Yesterday, we deployed an update to Team Foundation Service. You can read the release notes to see the improvements. The improvements aren’t major but there’s a few nice things – Some updates to UI styling with a greater use of color, some navigation improvements that include the ability to see the task board for all sprints (rather than just the current), and support for multiple Git repos per Team Project.

Less prominent, but we’ve also made a bunch of changes to our authentication code. We’re working to better support people who use multiple Microsoft Accounts (Live IDs) and working towards ultimately supporting Active Directory Federation and O365 accounts. I’ve seen some hiccups today with people logging in – I expect they will be cleared up quickly.

This sprint’s deployment was delayed a week as we hit a number of issues in pre-production testing that had to be ironed out. Next sprint should resume on the normal schedule.

Brian

Some thoughts on a comment about VS 2012.3

Brian Harry MS — Wed, 08 May 2013 13:04:16 GMT

Yesterday, I got a thoughtful comment from Dean on my post about Update 3. I sat down to write a response this morning and it turned into a but of a novel (in fact, the blog says it’s too long to post as a comment ). So I’ve turned it into a post. Here was Dean’s comment:

Brian,

I looked at the list of fixes, you weren't kidding when you said small. Are those just the major ones (crashing)? There are a whole lot of things logged on Connect that when added together really stifle productivity (many involving the editor with intellisense, syntax highlighting, etc.). I get nervous at this point in a VS life-cycle because I fear that the bugs I care most about will once again get worked on for the next major version. We spend money on the product and the window of time for things getting fixed is unbelievably tiny...before MS turns off the lights/closes shop until the next version. The new update cycle is great, I just hope it is not merely SP1 in chunks...I hope you still give updates to the product all along the way up until the next version.

-Dean

And my reply…

@Dean, I try not to kid :) I'm not going to swear that's every single bug fix. Plus there will be a few more fixed between now and the final release - for instance, I just heard that we are fixing a few important Blend issues and those fixes didn't make the RC. It's not an incredibly small list - I count 59 bug fixes. I'm not sure whether to feel proud or ashamed that 34 of them are in my team :) I think the answer is proud, though I suspect this is an eye of the beholder issue. A story...

I was speaking last week with one of our test managers and he said to me...

People in shiproom don't know what to make of us. We bring so many more bugs to be fixed than any other team in VS that they just look at us sideways. But, my feeling is that if a customer reports a bug, we should just fix it. We're doing the update anyway and there's no point making the customer wait.

Aside - shiproom is a process we have of sharing the changes that each team is making as we get close to the end of any release that allows for peer review and feedback. The purpose is to help slow down the churn, reduce the risk of regression and warn other teams of any changes that might impact them.

It was a proud moment for me. I've spent the better part of 12 years arguing for a different way of thinking about this kind of thing. I grew up in startups where virtually every customer felt like the difference between going out of business or not. I spent 30% of my time on tech support and generally gave same day (or at most a couple of days) turn around on bug fixes if a customer had an issue. I've always believed that trying (look, I understand that you'll never fully succeed) to make each and every customer happy is important. And the connection between developers and the problems customers face is an important feedback loop that forces learning over time.

I don't want anyone to think this is a simple "motherhood and apple pie" vs ignorance issue. There's a legitimate debate to be had. First, every time we fix an issue we have a non-zero probability of introducing a regression that our testing misses. Customers tend to be super unhappy if they get a fix for one problem only to find a new problem they didn't have before. Further, we all know interruptions are bad - they sap productivity. Having to stop what you are doing and investigate a customer reported issue (which often turns out to be a configuration problem), produce a fix, test it and deliver it can really reduce the overall volume of value delivered. Further, fixing every issue any customer reports can cause you to spend an inordinate amount of time fixing issues that affect relatively few people while the bulk of your customers wait for value. It's not a simple trade-off and neither extreme is the right answer. It's a balance and it all comes down to the values to apply to weigh that balance.

As for the instability introduced by regressions, I argue that some regression rate is acceptable as long as your time to repair is sufficiently short. While a customer will be frustrated by getting a new bug along with their fix to the last, as long as it doesn't happen too often and you fix the new bug quickly, the net result of fixing people's issues more responsively is a win. Reasonable minds can disagree on what rate is considered acceptable. The lower you want to drive the rate though, the less churn you can tolerate and the less frequently you can release. Generally your tolerance also varies by the kind of component you are working on, the ease of deploying updates, and many other factors.

Now, again, before someone jumps to calling me a hypocrite, let's talk about Team Project Rename. There's an "issue" that has existed for a long time and we've done nothing about it (right Allen?). Could we? Of course, it's just software - anything can be done. So that must mean we have decided not to despite it being one of the top customer generated requests. Brian, you're a hypocrite. I like to think I'm not but you can judge that for yourself. As I said, it's a balance. This is probably one of the things I hang my head in shame about. We made some decisions many many years ago that, in retrospect, I would do differently. The ramification of those decisions are that Team Project rename is hard - it shouldn't be, but it is. We've costed it a couple of different times over the past several years and it always comes back as months and months of work with high regression probabilities. That doesn't mean we're ignoring it. I still very much want to do it but we're taking a longer road to get there. For a while we just decided to "live" with the problem. Now we have a plan but it calls for making changes to "undo" some of those decisions we made many years ago that makes it hard. Once we get that work done, we can do rename. All I can do is apologize for how long it is taking.

Back more to your point Dean. Yes, Update 3 is likely the last of the updates to the VS 2012 line. Of course, we'll still continue to fix any critical issues people find but we are winding it down and focusing on VS V.Next. I'd like to think many of the issues you refer to will get addressed there and, if not, I hope we'll get to them in the V.Next update train. Referring to the above - Update 3 is the update where we slow down the churn, address remaining high impact customer issues and any regressions introduced in Update 2.

When you say the window is "unbelievably tiny", I don't think I agree. Of course we tried to get as much customer value and feedback incorporated into 2012 RTM as we could and then followed with 3 updates over a period of 9-10 months in which we delivered, in the aggregate, a tremendous amount of stuff - much of which was directly driven by customer feedback (e.g. Desktop Express SKU, C++ XP targeting, Kanban support, Blue theme, and dozens of more significant improvements - and, hundreds of bug fixes). While I can understand it's frustrating that we didn't get everything - or even all the most popular ones, we did make a lot of progress - and we'll keep making progress in the context of V.Next.

I also don’t think it’s “SP1 in chunks”. The kinds of changes we’ve put into the updates go FAR beyond what we would have historically included in a Service Pack. Service Packs had an “aura” that they only contain bug fixes and while that was never strictly true – any time someone proposed a Service Pack change that didn’t smell like a bug fix, there was a lot of justification that had to be done. One of the fundamental mindset changes with the move from “Service Packs” to “Updates” has been that the primary value of Updates is new value – and sure we’ll fix a lot of bugs too, but that’s not the focus. Read my posts on the updates and you’ll that generally the bug fixes are a footnote. They are all about the cool new capabilities we are enabling.

I've exposed a lot of flank here - so I suspect it may generate a lively conversation. But, hopefully it sheds a little light on how I think about it (of course, only one person in a large company - so don't construe this as any kind of official policy). As always, I'm happy to engage in healthy debate and learn from customers and from mistakes.

Thanks as always for listening,

Brian

TFS 2012.2 Power Tools update available

Brian Harry MS — Wed, 08 May 2013 11:47:06 GMT

Yesterday we released our update to the TFS 2012 Power Tools to work with VS/TFS 2012.2 (Update 2).

Yes, I see the irony in announcing this the day after announcing the first “go-live” Update 3 CTP It took us longer than we had hoped to get the Power Tools ready. However, we’ve made a key change to mitigate this issue in the future.

Our Power Tools setup has traditionally had a block to prevent it from working with the next version of VS (which is why the Power Tools from Update 1 wouldn’t work with Update 2). This is a bit of a hold-over from before we started doing the regular update cadence. At that time, it was pretty likely that we’d change something in the 2 years between updates that would break the Power Tools and we’d work hard to have new releases of Power Tools ready before the final release of VS/TFS.

In the new world of frequent updates, it’s much less likely that we break something and trying to synchronize the 2 releases is much harder. So, we’ve removed the block and we fully expect that these Power Tools will continue to work fine with Update 3. I’ve just finished installing it on my VS 2012.3 RC and everything looks to be working well.

There really aren’t significant changes in the Power Tools release. Aside from removing the block, we fixed a hand full of bugs and removed the back/restore Power Tool – because the feature was moved into the “official product” in Update 2. So, the main reason to install this release is just to get it to work with VS 2012.2.

I apologize for the delay and the inconvenience it caused. I talked to a number of customers who decided to uninstall Update 2 rather than give up their Power Tools. We’re glad to hear that the Power Tools are valued so much, but at the same time, we’re sorry for forcing you to make a choice. You shouldn’t have to any more.

Thanks,

Brian

Visual Studio 2012.3 (Update 3) “go-live” CTP is now available

Brian Harry MS — Wed, 08 May 2013 00:45:00 GMT

I know it seems like 2012.2 just released (at least it does to me) but we’re already well into our schedule for 2012.3. Today, we released the updates for both Visual Studio and Team Foundation Server. This is the first “go-live” release in the 2012.3 line of CTPs – this means that we’ve tested it to the point that we think you can install it in your production environment (we have in ours) and get help with any significant issues you encounter. Our go-live process found 8 or 10 TFS bugs for Update 2. We were able to fix those for the final build and ensure everyone had a better experience. We really appreciate your help finding the last few issues.

As I mentioned in my Update 2 availability post – Update 3 is going to be very small compared to Updates 1 and 2. For the most part it just contains bug fixes that have either been reported by customers or found in our own testing. You can read the KB article to find a full list of fixes.

Installing it should be relatively straight forward – for both VS and TFS, just run the installer. There should be no compatibility breaks so you need not upgrade all of your components at once, though, over time, you should plan to get them all updated.

You can ask any questions on the Visual Studio Forums and log bugs on Connect

Thanks and let us know what you think,

Brian

Visual Studio 2012 Update 2 (VS 2012.2) ISOs available

Brian Harry MS — Mon, 06 May 2013 11:30:51 GMT

I’ve had a healthy discussion with some of you on this post and we’ve received quite a lot of feedback on Soma’s blog and other places. Based on this feedback we’ve produced ISOs for Visual Studio Update 2 and decided that for these "”larger” updates, we’ll continue to do so in the future.

You can read more about it on the Visual Studio blog here: http://blogs.msdn.com/b/visualstudio/archive/2013/05/03/announcing-availability-of-isos-for-visual-studio-updates.aspx

Brian

Team Explorer Everywhere 2012 Update 2

Brian Harry MS — Mon, 29 Apr 2013 13:52:20 GMT

I missed announcing the availability of Team Explorer Everywhere Update 2. We released the update the first week of April. You can get it either by:

Downloading it from our download page or
If you use our Eclipse Update site (http://dl.microsoft.com/eclipse/tfs), using Help –> Check for Updates in Eclipse

Mostly our Update 2 for Team Explorer Everywhere contains bug fixes and small feature improvements. Here’s a list of the improvements you will find…

Add delete options to builds and retention policy
Allow import of Java and Ant as Zip files into Version Control
Fix checkout for edit being disabled for items with merge,branch pending change
Remove deleted builds from the build definitions filter in builds page
Add error when attempting to delete a build that is marked as retain indefinitely
Update progress indicator during unshelve of large changes
Improve the way that tooltips are displayed on Ubuntu OOB color theme
Allow periods in WIT display names
Fix issue with setting execute bit on get when talking to Update 1 or Update 2 TFS 2012 server
Fix issue blocking some label, shelve and workspace operations when there are two users with the same display name on TFS2012
Fix issue canonicalizing local paths when setting working folder to a symlinked path
Give proper error message when check-ins rejected by an ISubscriber plug-in on the server
Language pack updates (fixes for French, German, Japanize and Brazilian Portuguese)
Hide the workspace selection page during connection, make it less prominent during share and import.
Allow use of alternative credentials when embedded browser is unavailable to Eclipse

Let us know if you have any feedback.

Thanks

Brian

Updated TFS 2010 MSSCCI Provider

Brian Harry MS — Mon, 29 Apr 2013 13:19:08 GMT

Last week we released an update to the TFS 2010 MSSCCI provider. The primary motivator for this release was to enable it to work with VS 2005. This now enables you to connect to TFS 2012 using VS 2005, particularly from Windows XP. The support for VS 2005 was the only enhancement to this update of the 2010 MSSCCI provider.

Thanks,

Brian

Visual Studio Tools for Git Update available

Brian Harry MS — Fri, 26 Apr 2013 12:18:25 GMT

Yesterday, we released a major update to our Visual Studio Tools for Git. While still nowhere near done, we feel like we’ve crossed a significant threshold of completeness and usability. If you’re already using them or have tried them and didn’t feel they were ready, I encourage you to give this update a go.

You can install the latest drop here: http://aka.ms/git4vs

It will require that you have Visual Studio 2012 Update 2.

The most compelling improvements include:

Push is dramatically faster
Larger repos now work a lot better
Merge and pull now allow you to have non-conflicting changes in your working copy

Additional improvements are:

Added support for integrated Windows authentication when receiving a 407 Proxy Authentication Required
Added support for pushing to Git servers which do not support Transfer-Encoding: Chunked
Added support for pushing to certain Git servers which require side-band-64k in order to report push status
The sync branch button on the Commits page is now functional. It does a fetch, merge, and push, in sequence
Merge and pull now properly prompt the user to save their work before starting
Merge now properly auto-reloads non-dirty solution items which changed in the working copy
The push command now sends fewer objects to the server, which is a substantial performance improvement
Merge and pull now use checkout instead of reset for fast-forward merges, which is a performance improvement
Fixed CRLF issue that was causing checkout to fail due to conflicts in certain scenarios
Improved performance of working directory status computation (much less hashing of workdir contents)
Fixed an issue where CRCRLF line endings would end up in the working copy in certain scenarios
The Add, Delete, and Ignore actions in the Untracked Items section of the Changes page are now working
Improved Solution Explorer glyph behavior with JavaScript and other CPS (Common Project System) projects
Undo merge option added to the Resolve Conflicts page
Fixed a bug where the user was not prompted for fresh credentials when stored credentials failed to authenticate
Submodule support from the libgit2 project has arrived in this release, but we haven't yet finished all the work in our VS plug-in to make them work smoothly.

As I never like talking about performance improvements without giving some numbers… Here’s a report that I got about a week ago. Not sure these are the final numbers but you can see the degree of improvement. These are differences between the Git client tooling between Sprint 44 (the previous public release) and Sprint 46 (this release), using the TFS Git Server. The numbers are in seconds.

We have also made some improvements to our Git server perf but these are just client improvements. Of course, we haven’t released an on-prem Git server yet so the only place you can see the server improvements is on http://tfs.visualstudio.com. The Git server perf improvements were deployed with our Sprint 46 service update yesterday.

Check it out and let us know what you think!

Thanks,

Brian

Sprint 45 service issues

Brian Harry MS — Tue, 23 Apr 2013 19:28:00 GMT

If you are a regular user of our Team Foundation Service - particularly during peak times, you've probably noticed that it hasn't been running as seamlessly as usual. Our recent issues started with our sprint 45 deployment that I wrote about here and foreshadowed here. In this post I want to give you some insights to what happened and what we've done about it. The team has been working long and hard since the deployment and I really appreciate the dedication and tenacity.

As our service has evolved, we've begun to expand the breadth of the services we are trying to offer. We are also working towards having support in data centers around the world so you can choose where your data is hosted. As part of this drive we've been gradually working to "decouple" components of Team Foundation Service to enable more independent evolotion and deployment. Sprint 45 was the culmination of one of the biggest peices of work we've done to date in that effort. Though you can't really see the effect (ok, other than some of the bugs we introduced), we refactored the core Team Foundation Service into two services - what we call SPS or Shared Platform Services and Team Foundation Service (the functionality you know and love).

SPS contains many things but the simplest way to think about it is the data that is outside a given tenant account - Subscriptions, location, configuration, identity, profile, etc. After many sprints of work, we finally deployed the new SPS service at the end of sprint 45 - in fact on March 22nd and 23rd. It involved a pretty massive data refactoring and the first upgrade with planned downtime in a long time (and I believe the last in a long time).

When you think about the change technologically, the biggest difference was that, for the first time, we introduced a service to service split with fairly heavy traffic. Before this, all of the calls between the "logical" services were either between DLLs in the same app domain or to the database. This split caused those calls to become service to service REST calls, crossing not only processes but even machines in the data center.

We did a ton of testing ahead of this deployment, however, it's clear we didn't catch everything we should have and the result has been a lower quality of service than we (and you) expect. The issues have mostly manifested themselves as slowdowns during peak hours (though in the first couple of days it was a little worse than that). The build service has been the most (and longest) affected service - mostly because it's very intensive: it does a full sync of all of your source to build it and that can put a lot of load on the system when lots of people are running builds at the same time.

I'm going to walk through a list of the issues that we hit. I'm going to try to group them a bit make it easier to digest.

REST stack

After we went live and started to see production load, we saw a great deal of time spent in our REST stack. I still don't fully understand why we didn't catch these issues in our pre-production load testing and the team is doing deep retrospectives over the next couple of weeks and this will be one of the key questions to answer.

ClientCertificationOption.Automatic - We have an object model for talking to TFS. With the new service to service architecture, we are now using portions of that object model on the service. A couple of years ago, we added multi-factor auth to our object model to support scenarios where stronger authentication is required. We discovered that this option was still turned on in our server context - where it was not needed and the result was scanning of the certificate store on every service to service request. We disabled the certificate lookup in this scenario and improved performance.

Thread pool threads - The new service to service architecture introduced a lot more outgoing web calls. The way the web stack works, under the covers, all of these calls are async and use the thread pool. We found that we were often running out of threadpool threads and waiting for the CLR to inject more into the pool. By default the CLR starts with 1 thread per core and then uses a "hill climbing" algorithm to adjust threads as needed. This algorithm tries to dampen changes in the threadpool size so it is not thrashing. However, we found that it could not handle our rapidly changing demands well. The result was long pauses of up to 20 seconds while requests queued up waiting for the thread pool to resize. Working with the CLR team, we decided to increase the minimum number of thread pool threads to 6 per core using (ThreadPool.SetMinThreads).

Wasteful CPU use - Many years ago, someone wrote some code to scan every request/response data stream for invalid characters. This was consuming a fair amount of CPU (I forget how much but more than 10%). No one, including the dev who did it, could remember why this code was put in there or think of any reason it was needed. We removed it.

Extra thread switching - We had a coding mistake that caused extra thread switches on every REST call. This is best explained in code. At the beginning of every REST call there's some lazy initialization logic:

if (a == null) 
{ 
 lock (sync) 
 { 
 if (a == null) 
 { 
 a = fetch some data; 
 } 
 } 
}

The issue is that "fetch some data" was a cross service call and the developer wanted to make it async. So they tried making the method async and doing and await fetch some data and discovered that you can't do an await from within a lock. Let's forget that this would completely have broken the synchronization. So they decided to do a Task.Run on the whole block of code. The problem is that a == null only the first time the code path is used, after which a is cached. The end result is that we did a thread switch just to run an if statement everytime there was a service to service call.

Too many cross service calls - As we started to investigate the cause of the slowdown, we quickly realized that we had way more "chattiness" between the services than we realized. This chattiness was exacerbated by the various performance issues above. Before the split most of these interactions were direct in-proc calls or database calls that were much lighter weight than the cross service calls. We were able to rework the code to reduce the number of calls across service boundaries substantially.

We fixed all of these REST stack problems within the first few days and it really helped restore the service nearly constant poor performance to intermittant poor performance and build service issues.

Other misc bugs

In addition to the set of bugs/issues we had with the transition to separate service, we had another set of bugs that were uncovered/exaggerated as part of the transition. Among them were:

Work item tracking identity sync - We have a process by which we sync identities from the identity system (now in the SPS service) into work item tracking (in the TFS service) for the purposes of enabling work item tracking rule enforcement based on groups and identities. This change uncovered/exacerbated two identity syncing bugs. One caused older accounts (the first 40K accounts) to start doing full identity syncs (consuming significant resources) frequently. The other was some thrashing in the identity service causing lots of activity but no progress in syncing.

Stored procedure optimization - We also found a few stored procedures with some significant query plan problems.

That pretty much summarizes our "bugs". There were clearly too many and they were too significant. I don't ever expect to get to the point where we can deploy a major change and not miss any bugs but it's clear to me we can do better. However, if it were just this, the overall impact would not have lasted nearly as long as it has. In addition to our own mistakes we got hit by a number of environment/operational issues. The cloud environment (Azure) that we operate in is constantly changing. Almost every day something, somewhere is being updated and we rely on enough of it that we can get wagged by many things. Here are some of the environmental things...

CLR/.NET

GC/Pinning - As I mentioned above in the section on Thread pool threads, we introduced a bunch more async patterns in our service with this change than we had ever had before. In fact, we now have about 10 async calls per second from each application tier machine. Async calls, in the CLR, ultimately involve some amount of "pinning" GC memory so that the OS can transfer data into buffers asynchronously without blocking the GC. The problem we have is that at that rate there's almost always an outstanding async call, and therefore pinned buffers. The CLR has some designs to help reduce the impact of this but we found they weren't working sufficiently well for us. We were using Large Azure roles (7GB of RAM) and finding that due to pinning effects on the GC, we were seing high CPU load from the GC and memory exhaustion that would ultimately result in machine recycling - causing intermittent availability. In the process of investigating this, we also uncovered the issue described below under "Windows Azure". We engaged the CLR team quickly and started investigating. They said they had seen this issue before in some high scale services but no service was able to work with them long enough to isolate and fix it - basically as soon as they found a mitigation their interest in it was over. I view part of our role as driving feedback and requirements into our platform to improve the overall quality. As such we're continuing to work with the CLR on this. We're expecting a longer term improvement from them but, in the shorter term, we've found that we could increased to an XL role (14GB) and that mitigated the pinning effects, eliminating the critical issues.

SQL Azure

Config DB provisioning - Each of our services have a single database that we call Config DB. It's the core database that the service needs to run. It's got critical configuration information like connection strings, manages cross app tier notifications, etc. Before now, we only had 1 Config DB for the entire service. That database was on a XL reserved SQL Azure instance to make sure we had a committed level of capacity behind this critical database. With the rollout of our new service and a second Config DB along with a new "accounts" DB, we needed to readjust how our config DBs were deployed so that we didn't have to pay for 3 XL reservations but were able, instead, to share the reserved resources between our Config DBs. In this process, there were manual operational procedures which caused all three database primaries to get landed on the same physical machine. The load balancer realized this was a bad idea and began trying to move two of the primaries onto another of our reserved machines. However, we hit a bug in the load balancer where it had difficulty balancing the resource consumption of the foreground work with the background database movement. The result was that it kept trying to move the databases, getting part way through and then giving up. This caused a lot of load on our SQL Azure databases that, coupled with our issues in the first section, compounded our performance problem. Ultimately this was addressed by explicit operational intervention to place the databases.

SQL Azure storm - As it turns out, that first week, SQL Azure (on the cluster we use), had a bit of a storm with some misbehaving tenants using an undo share of the resources and causing the performance of our databases to suffer. These storms happen from time to time in SQL Azure and they are working through throtting and quota mechanisms to contain them. They were MUCH worse a year or so ago - the SQL team has made a lot of progress but the problem is not yet completely gone.

So far, most of what I've talked about has been, at least, tangentially related to the "big" change we were actually rolling out. As it so happens we also got hit by some stuff that had absolutely nothing to do with the big changes we were rolling out. Over the first couple of weeks we had worked through all of the issues we could find with our change and the service was generally running reasonably well - with the exception of the build service. The build service was running poorly - during peak times builds were slow and frequently timing out/failing. Once we had ruled out everything that had anything to do with our changes we discovered some other things going on.

Windows Azure

Slow disk I/O - The slow and failed builds were continuing to plague us. Once we had stopped looking for causes related to the changes we had made, it became clear pretty quickly that the root problem had to do with the amount of time it was taking to download source out of version control. After much investigation we concluded it had something to do with changes in Windows Azure. We were seeing incredibly slow disk I/O on the local disk of our web role. We contacted the Azure team and learned they had just rolled out (just about the same day we rolled out our changes), a change to both host and guest VMs that changed the way write through caching works. We were seeing, under load, individual I/Os taking as much at 2 seconds - 2 orders of magnitude longer than it should. The source code we manage is in Azure blob store. However, to optimize delivery, we store a cache of source code on the application tier that we can use the OS TransmitFile API on. It is a VERY optimized API for transmitting files over the network that has very low overhead because it avoids copying buffers in and out of user mode, etc. The disk I/O slow down was causing this cache to behave very badly - causing build to take for ever, downloads to fail, builds to be aborted, etc. Further, it significantly exacerbated our issues with the GC because we'd see page fault in GC collection taking a second or more. It was a mess. To mitigate it, we have turned off our AT cache and are instead fetching source directly from Azure blob store. This is causing our file downloads to take about 1.6 times longer than it used to but it's still way better than where we were. We're continuing to work with the Azure team on a more permanent fix to restore the local disk performance.

Network failures - Unfortunately, with the slow disk I/O problem mitigate, we're still seeing an increased level of build failures. Upon further investigation, we've discovered that we're seeing a much higher level of network error in Azure than we are used to. At this time, I still don't know the cause of that - we believe it's related to the last Azure fabric rollout but we don't have a clear root cause yet. We are seeing hundreds of builds a day failing due to network failures starting about April 12th. Along the way of investigating many of these issues, we discovered that our build service didn't have appropriate retry logic to handle failed downloads. It's really not been an issue for the past year but all of the issues we've had in the last month has uncovered this hole. We'll soon be rolling out an update that will do retries (which quite honestly should be there in the first place because all cross service cloud calls should handle intermittent failures well) and it should significantly improve the build success rate. We'll still see some performance issues from the last two issues but at least builds should stop failing.

Debugging issues

Memory dump problems - To add insult to injury, our ability to debug the service has been seriously compromised. I haven't actually looked deeply at this one myself so my description may be a little sketchy. Debugging some of these problems are greatly facilitated taking memory dumps from production that we can diagnose off line. Historically, we have used a feature called Reflective dumps using procdump /r. This method reflects the memory state into a parallel process and then dumps from the parallel process so that there is almost no interruption of the production service. However, apparently, for a couple of years the Windows OS reflective dump mechanism has had a bug that caused it to miss fragments of the process memory. Apparently we've seen this in the form of "bad" memory dumps but it was infrequent enough that people just wrote it off to gremlins. Some change in .NET 4.5 (which we rolled out on the service a few months ago), has caused this reflective dump to fail every time - so we can't use reflective dumps any longer. Now we have to take a straight process dump in production and, unfortunately, this takes a minute or two, literally. This long of a pause causes cascades of timeouts and results in a significant service interruption. So every time we needed to take a dump to look at thread state, GC issues, or the myriad of other issues here, we had to basically reboot one of the app tiers, causing significant service interruptions, failed builds, etc. We're working with the Windows team on getting them to produce a hotfix that we can use and the rest of the world can use to reliably get .NET 4.5 reflected memory dumps.

As you can see there were quite a lot of issues that conspired to make it a bad month for us. We've been working incredibly hard and the service has gotten better day by day. We've also been working closely with the platform teams to ensure there are durable solutions to the problems we've hit there that work not just for us, but for everyone.

This is a long post and if you've read this far, I'm impressed. We're very sorry for the problems that we've caused. At this point we feel like we've worked through enough of our issues that we can get back on our regular deployment schedule. As such, we are deploying the delayed Sprint 46 build today and will deploy the Sprint 47 build a week from Monday. We'll keep working hard until everything is completely back to normal and if you see anything that looks squirelly, please let us know.

Thank you,

Brian