VSTS Pioneer TFS2010 Dogfood Server – Hardware & Topology

Grant Holliday’s blog

Senior Service Engineer, Microsoft Visual Studio Team Foundation Service

VSTS Pioneer TFS2010 Dogfood Server – Hardware & Topology

  • Comments 4

(This blog post is part of a series of posts on the new VSTS Pioneer TFS dogfood server.)

While running the main TFS server for ~3,500 users in Developer Division we got a pretty good idea of the hardware required to support our enormous number of files, local versions & branches.

We knew that:

  • our SQL Server load patterns were very IO intensive
  • our SQL Server load patterns were very CPU intensive
  • our disk usage will grow by about 300% per year
  • the number of operations the server performs will increase

Servers

We wanted to purchase and setup hardware that was flexible enough and designed to scale as our needs do. For this reason we settled on two identical servers that are part of the Microsoft IT recommended systems for our needs:

According to Joe Chang who studies SQL Server performance, the “The X7460 (Dunnington) is a clear winner at 4-way for the high-call volume apps.” The 6-core CPUs also offer a pretty significant performance boost over the older quad-core CPUs:

  • “On TPC-C, the 7460 six core generated a 34% edge over the quad core AMD and 56% advantage over the quad core X3750”, and
  • “On TPC-E, the six core showed a 49% edge over the older quad core”

Virtual Machines

We are running 5 virtual machines at the moment with almost identical configurations. Since we’ll be constantly upgrading things like the .NET CLR that often don’t support build-to-build upgrades or clean uninstall, running the application tiers as VM’s makes perfect sense. It also improves our rollback story if we have a failed upgrade. (More about upgrade in a future post).

VM

Purpose

Configuration

AT01, AT02, AT03

TFS Application Tier

Windows Server 2008 R2

4 Virtual processors

4GB RAM

2 Virtual NICs

1 VHD for C$

1 VHD for version control cache

MIRROR

TFS-TFS Connector tool runs here

Windows Server 2008 RTM

4 Virtual processors

4GB RAM

1 Virtual NIC

1 VHD for C$

1 VHD for version control temporary workspace

MOSS

Microsoft Office SharePoint 2007 SP2 (Complete install)

Web Front End, Excel Services, Search Services & Shared Services Provider

Windows Server 2008 R2

4 Virtual processors

4GB RAM

1 Virtual NIC

1 VHD for C$

Databases reside on physical SQL server

Storage

Since these servers are hosted by the DevDiv IT guys, we went with their recommendation for storage. They already have a massive storage infrastructure for build & drop servers built on Xiotech technology, so it was easy for them to add additional ISE units to meet our size & performance requirements. In the end we ended up with the following configuration:

  • 5TB of “Balanced” storage
  • 5TB of “Performance” storage

The SQL Customer Advisory Team’s – Storage Top 10 Best Practices is a great resource when planning your own TFS deployment. In particular we specifically considered the following:

  • Isolate log from data at the physical disk level (e.g. O$ drive is on separate spindles)
  • Consider configuration of TEMPDB database (e.g. T$ drive is on separate spindles)
  • Make sure to give thought to the growth strategy up front (e.g. We can add additional ISE units at any time and grow any drive without having to migrate data)
  • Lining up the number of data files with CPU’s has scalability advantages for allocation intensive workloads. (e.g. we have 12 datafiles for TempDB because we have 12 CPU cores)

Here’s how the virtual disks (LUNs) are configured on the server:

SERVER1 (SQL, SSRS, SSAS)

Drive

Purpose

Storage Type

Size (Gb)

E$

SQL backup dump drive

Balanced

2741

G$

LocalVersion filegroup

Performance

1673

I$

OLAP data

Balanced

25

J$

All database data

Performance

1833

K$

Version filegroup

Performance

971

O$

Transaction Logs

Balanced

1500

T$

TempDB

Performance

500

SERVER2 (Hyper-V Host)

Drive

Purpose

Type

Size (Gb)

E$

Hyper-V VM's

Balanced

743

What’s cool about the Xiotech stuff is that it’s like Lego blocks – as you need more storage or more performance, you just buy more blocks and plug them in. What’s even cooler about it, is that you can control the SAN using web services – the build lab uses the SANMan tool they built to move virtual disks between build servers and automate the provisioning of disk space.

Topology

At just over 400 users, we probably don’t need three AT machines in a NLB configuration. However, NLB is an important new feature of TFS2010 and it’s a scenario that we need to dogfood for ourselves.

Here’s some quick facts about our topology:

  • We have a couple of “friendly name” DNS records for the NLB cluster IP, Reporting Services & Analysis Services – this allows us to change the underlying infrastructure without users having to connect to a new addresses.
  • We’re running Windows 2008 R2 RTM on our VM’s & soon on our physical servers
  • A single VM was created, then sysprep’d and copied. This means that we can spin up a new VM in a very short time period. (Copy VHD, Add to Hyper-V, Start, Join Domain, Apply Updates, Done.)
  • SERVER2 is running an SMTP server that relays mail on behalf of each of the other servers. At Microsoft, the corporate mail servers will only accept mail from authenticated users that have mailboxes. By running the SMTP server as an authenticated user, it allows us to run the application tiers as Network Service. This means when we have to change a password to a service account, it doesn’t cause a TFS service interruption – just a brief SMTP interruption.
  • Since all the SQL services are running on the same box, they don’t need to be run as a domain accounts either – no service interruption & less places that need a password changed when it expires.

clip_image001

That’s it for hardware and topology, next post is on performance and Dogfood statistics.

  • This is great info. Thanks Grant!

  • Thanks Grant.  We are planning on migrating to TFS 2010.  Is it possible to have the baseline Hyper V available for download?  This will help me a lot in my POC and hands on experimentation to understand the product.

  • Oops.  I meant baseline Hyper V VHD ...

  • Hi Grant,

    I read through your blog and it's a wonderful stuff!

    I'm a VSTS consultant and working out a hardware requirement for one of our key clients.

    Where the client in planning to start of with a user base of close 8,000 users and scale upto 25,000 users in the next 2 years. I'm unable to find any data for this massive deployment.

    Kindly assist us in achieving the same. Look forward for your help in this regard.

    Cheers,

    Srini

Page 1 of 1 (4 items)