Connect Performance Issues
On February 22, Microsoft Connect was suddenly having problems for most of the users of the site. People trying to access the site were experiencing page timeouts, error messages, and generally unable to access the site. This lasted until about 6 PM Pacific Time.
Here's what happened: At about 11 AM Pacific Time, traffic on Connect spiked by fifteen- fold over the previous highest load we'd ever seen. This happened in less than 10 minutes, and was the highest traffic numbers we've ever seen. That day, a new build of Vista had been released, and some other programs had sent out large numbers of invitations to new users. In fact, the load was causing problems with Distribution Services and people were unable to use the FTM to download files, even if they were able to access Connect or had started a download before Connect started bogging down under load.
Everything- and I do mean everything- was put on hold while we figured out the problem and worked to solve it. We identified some issues with the way Connect was communicating with the database. Once we corrected these issues in a hotfix, the site became functional and responsive again. In fact, the next day, traffic was even higher and Connect withstood the load.
We've been analyzing the logs and doing post-mortems on what exactly occured and we're doing everything we can to make sure that 1) it doesn't happen again, and 2) we're able to handle even more load going forward. We've also revised our load-testing scenarios in order to make sure we're catching more of these types of problems before they become problems.
