granth's blog : VSTS Administering

TFS2010: What happens to subscriptions after an upgrade?

grantholliday — Fri, 11 Dec 2009 05:48:03 GMT

Both TFS2008 and TFS2010 allow users to create subscriptions to events. These can be Work Item events, CheckIn events and others. They are created via the ‘Project Alerts’ option in Team Explorer or Web Access, or the ‘Alerts’ feature in the Team Foundation Power Tools. An administrator can also run bissubscribe.exe on the Application Tier to setup web service/SOAP subscriptions.

A few people have asked me what happens to subscriptions when a server is upgraded from TFS2008 to TFS2010, so here’s how it works:

Upgrade from TFS2008 and Attaching a Team Project Collection to an existing instance

Subscriptions & alerts that users have configured get preserved as-is
- The only difference is that we’ll now store the subscriber by the internal TF identity GUID instead of the SID. This is transparent to the user.
For any subscription with a subscriber who is in the service accounts group, we replace the subscriber with the service group (e.g. ‘DOMAIN\tfsservice’ turns into ‘[CollectionFoo]\Service Accounts Group’)
We remove SOAP subscriptions which have endpoints that end in “/VersionControl/v1.0/Integration.asmx” or “/WorkItemTracking/v1.0/SyncEventsListener.asmx” or “/Warehouse/v1.0/warehousecontroller.asmx”. These are internal subscriptions used in previous versions to keep areas, iterations and identities in sync.
During TFS2008 upgrade or when you attach a new Team Project Collection (TPC) we replace the url prefix of SOAP subscription which ends in “/WorkItemTracking/v1.0/Integration.asmx” with a token that gets filled in at runtime with the actual public url to the TPC

TPC Move

This is the scenario that you detach a TPC from one instance and attach it to another instance

Subscriptions & alerts that users have configured get preserved as-is. They are stored in the collection’s database, so they move with the collection.

TPC Split

Subscriptions & alerts that users have configured get copied to the split collection.
When you split a TPC, you will want to make sure you clean out the subscriptions or at least warn users that they may receive duplicate alerts.

TFS2010: Object Model API changes after Beta 2

grantholliday — Tue, 08 Dec 2009 07:41:15 GMT

Based on feedback that we received both internally and externally we decided to make a rather significant change to the TFS client object model. This change centers around the TeamFoundationServer, TeamFoundationApplicationInstance and TeamFoundationServerBase classes and their supporting factories.

The History

First some background. In Orcas and Whidbey the TeamFoundationServer object was used as the starting point for working with the Team Foundation client object model. In TFS 2010 we now have Team Project Collections and the Team Foundation Configuration Server and unfortunately the TeamFoundationServer object is no longer specific enough to describe the end point that you are working with.

To combat this originally we decided to repurpose the TeamFoundationServer class for talking to Team Project Collections and introduce a sibling class called TeamFoundationApplicationInstance for talking to what was at the time called the Application Instance and has since been renamed to the Configuration Server. We also introduced a base class called TeamFoundationServerBase to contain all of the shared code between the TeamFoundationServer and TeamFoundationApplicationInstance classes.

Martin Woodward describes this old model in his first TFS 2010 API By Example post.

The Changes

Since we made that original change, we have gotten feedback both internally and more recently externally that these objects were very confusing. Because of that feedback we decided to make some changes to these objects to make them more intuitive. The main changes that were made are as follows:

Renamed the TeamFoundationServerBase class to TfsConnection.
Renamed the TeamFoundationApplicationInstance class to TfsConfigurationServer.
Introduced the TfsTeamProjectCollection class to replace the TeamFoundationServer class.
Obsoleted the TeamFoundationServer class.

The Impact

If you have started writing tools that use the TFS 2010 Beta 2 object model, you should anticipate these changes in our final release and allow yourself time to make the changes and test them.

Here are some of the things that have changed throughout this refactoring and some things that you should look for and validate if you are reviewing these changes:

The TeamFoundationServer class had many constructor overloads, several that took strings and several that took URIs. We have been wanting to get rid of the string overloads (that allowed you to pass a server name or a server url) as part of this effort so the new TfsTeamProjectCollection class does not have string constructor overloads. Instead you need to create this object with a URI just like you have to do with the TfsConfigurationServer object. There is a static helper on the TfsTeamProjectCollection class called GetFullyQualifiedUriForName that will return you the collection URI given a string that either contains the collection name or the collection url. If the collection url is passed in this is essentially a no-op.
Any public property or method on a public class that either returned a TeamFoundationServer object or took it in as a parameter was deprecated. Along with this, a mirroring property or method was added that simply replaced the TeamFoundationServer type with the TfsTeamProjectCollection type. If there was any doubt as to whether or not that property or method had shipped in Orcas or Whidbey we made the assumption that it had shipped and deprecated it instead of removing it. If we knew that it had been added in the TFS 2010 time frame, we simply removed the property or method that exposed the TeamFoundationServer class.
With the above refactoring of properties and methods we tended to only rename the property or method name if it was exposed to the public and very wrong. I.e. if we saw GetTeamFoundationServer(), we renamed it to GetTeamProjectCollection().
The TfsTeamProjectCollection class does have an internal property that exposes a related instance of the TeamFoundationServer class. The TeamFoundationServer object also exposes a TfsTeamProjectCollection instance. These properties are used to support public methods or functions that need a TeamFoundationServer in a backward compatibility scenario.

If you have an addin or tool built against the Beta2 object model and you don’t have the Beta2 assemblies installed, then you’ll get something like the following error:

Team Foundation Error

Could not load type 'Microsoft.TeamFoundation.Client.TeamFoundationServerBase' from assembly 'Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

System.TypeLoadException: Could not load type 'Microsoft.TeamFoundation.Client.TeamFoundationServerBase' from assembly 'Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

To fix this, you’ll need to contact the addin/tool owner to get them to release a compatible version with the changes mentioned in this post.

TFS2010: Older clients not able to connect

grantholliday — Tue, 08 Dec 2009 07:08:12 GMT

When the Forward Compatibility GDR for Visual Studio 2008 shipped, a change was made to the Team Foundation Server 2010 code that blocks incompatible clients. That means that after TFS2010 beta2, clients that don’t have a forward-compatible patch installed, won’t be able to connect.

For more information on what features will & won’t be available, see the TFS2010 Compatibility Matrix.

When a client connects to TFS, the UserAgent is checked against a list in the TFS registry and if it matches any of these, an error message specific to that client is returned to the user. For example:

Team Foundation services are not available from the server.

Technical information (for administrator):

TF253022: You must update your client with the Forward Compatibility Update in order to connect to the Team Foundation Server that you selected. To obtain this update, go to the Microsoft Web site: http://go.microsoft.com/fwlink/?LinkId=16648

The list of UserAgent’s that is checked is defined in the \Configuration\Application\DisabledUserAgents path in the TFS registry of the instance:

Team Foundation (*.exe, 8.0.50727.<4406) – Visual Studio 2005 without the forward-compat GDR (available closer to launch)
Team Foundation (*.exe, 9.0.21022.*) – Visual Studio 2008 RTM
Team Foundation (*.exe, 9.0.30729.<4172) – Visual Studio 2008 SP1 without the forward-compat GDR
Team Foundation (VSSConverter.exe, 8.*)
Team Foundation (VSSConverter.exe, 9.*)
Teamprise 1.*
Teamprise 2.*
Teamprise*/3.<3* – Teamprise 3.3 introduced basic support for TFS2010

Of course, if you know how to change the TFS registry, you can remove these bans. However, I wouldn’t recommend doing that, because using incompatible clients against the server can cause some strange things to happen. You’re much better off just patching the clients.

TFS2010: Troubleshooting SharePoint Dashboards data source problems

grantholliday — Thu, 03 Dec 2009 22:13:15 GMT

In the TFS 2010 release, if you have the Enterprise features of Microsoft Office SharePoint Server (MOSS) configured (i.e. Single Sign-On and Excel Services), then you can make TFS Excel Reports available to your users via a web site. This is very compelling since Excel makes PivotTables and PivotCharts easy to use and for users viewing the rendered charts via the web it is painless user experience.

For more information see the Excel Reports topic on MSDN. Here is an example of a working SharePoint Dashboard

However, there are a lot of moving parts required to get all this working. TFS 2010, SQL Analysis Services, MOSS (Single Sign-On Service, Excel Calculation Services). The purpose of this article is not to describe how to get it working, but where to look when it stops working. If you are starting from scratch, then you should refer to the “Checklist: Multiple-Server Installation” topic in the TFSInstall.chm guide for TFS 2010 and the Configure Settings for Dashboard Compatibility topic on MSDN. I’ve been through those steps more times than I can remember and my feedback went into it to help you be successful where I stumbled.

Error: Unable to retrieve external data

In my experience, this is the most common error:

Unable to retrieve external data for the following connections:
TfsOlapReport
The data sources may be unreachable, may not be responding, or may have denied you access.

It can mean a few different things:

Excel Services cannot connect to Analysis Services or SQL because of a connectivity issue (or it’s trying to connect to the wrong server)
The credentials returned from the SSO service are incorrect or they don’t have the right permission to connect to Analysis Services / SQL
The user browsing the website is not not a member of the group that the SSO credentials are allowed to be used by
The SSO service can’t retrieve the credentials from it’s store

The first thing to do is to check the connection.

Open one of the Excel reports from the Excel Reports document library in Excel (the desktop application)
Click on the Data tab in the ribbon
Click on the Connections button to display the Workbook Connections dialog box
Click on the TfsOlapReport data connection
Click on the Properties... button
Click on the Definition tab of this dialog box

This should show a connection string like the following:

Provider=MSOLAP.3;Integrated Security=SSPI;Persist Security Info=True;User ID="";Initial Catalog=Tfs_Analysis;Data Source=ASSERVER;Location=ASSERVER;MDX Compatibility=1;Safety Options=2;MDX Missing Member Mode=Error

If the connection string is incorrect, then you need to open the Team Foundation Administration Tool on your Application Tier and configure the reporting settings correctly.

The next thing to check is the Single Sign-On configuration.

Click on the Authentication Settings... button

The SSS radio button should be selected and an SSS ID specified

If an SSS ID (sometimes incorrectly referred to as an SSO ID) is not specified, then you need to open the Team Foundation Administration Tool on your MOSS server and configure an SSS ID.

Troubleshooting data source problems

The best place to start looking is the combined SharePoint and Excel Calculation Services trace logfiles.

Logon to the server where MOSS is installed
Open a command prompt as administrator
Type: explorer C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\LOGS
Open the workbook in a web browser – you should receive the error “Unable to retrieve external data” error message
At the command prompt type: net stop sptrace && net start sptrace
Select ‘Update > Refresh all data connections’ in the excel workbook (below)
Type: net stop sptrace && net start sptrace

Then if you sort the files in the LOGS directory by Newest to Oldest, the second most recent one should have the Excel Services & SSO tracing in it. That usually has a stacktrace or error message indicating where the failure is. You can also use ULSViewer to look at the log files, however I find Notepad is adequate.

Here’s an example of a successful workbook refresh.

Excel Services Web Front End      6k38   Medium        ExcelServerProxy.ExecuteWebMethod: sending request of type OpenWorkbook, to server=http://mossserver:56737/SharedServices1/ExcelCalculationServer/ExcelService.asmx, session=(null), state id=0

Excel Calculation Services        766u   Medium        ExcelService.LogRequest: starting request of type OpenWorkbook

Excel Services Session            5pbu   Medium        SessionManager.BeginRequest: There are 1 active users

Excel Services Session            1sxs   Medium        SessionManager.BeginRequest: Created Session id=23.485TDKRh5qu7zU3Ghay5tQ590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, caller=fe80::982:6438:527c:31f1%11

SSO                               0      Medium        RPC request to ncalrpc_AuthenticateClientBinding()

SSO                               0      Medium       RPC request to s_GetCredentials

SSO                               0      Medium        RPC request to GetCredentials()

SSO                               0      Medium        Cached entry is stale, so will not return credentials

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        User is a member of the group.

SSO                               0      Medium        Destroying CCommandEx

SSO                               0      Medium        Destroying CSessionEx

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        RPC request to ncalrpc_AuthenticateClientBinding()

SSO                               0      Medium        RPC request to s_GetApplication

SSO                               0      Medium        RPC request to GetApplication()

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        Destroying CCommandEx

SSO                               0      Medium        Destroying CSessionEx

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        RPC request to s_GetApplicationFields

SSO                               0      Medium        RPC request to GetApplicationsFields()

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        CSsoResourceManager succeeded in getting the connection string

SSO                               0      Medium        Provider=SQLOLEDB;Extended Properties='Server=SQLSERVER;Database=SSO;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15';

SSO                               0      Medium        Destroying CCommandEx

SSO                               0      Medium        Destroying CSessionEx

Excel Services External Data      13l9   Medium        ExternalSource.GetExternalKeyAndStateAndMarkUsed: Key exists in cache - waiting for it to be created, sessionId=23.485TDKRh5qu7zU3Ghay5tQ590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, externalsource=TfsOlapReport, key=[ts: 12/03/2009 13:04:14.192943, [serviceaccount, en-US, en-US, +0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, [Oledb, Stored, TFS, False, ], ()] BaseWB: "http://mossserver/sites/COLLECTION/Project/Reports/Bug Progress.xlsx" [0x409] [11/06/2009 19:14:04.000000] [BaseWB ID: 2]] 00000412-f5f3-0da1-0000-000050f7b00b

Excel Services External Data      13la   Medium        ExternalSource.GetExternalKeyAndStateAndMarkUsed: Key exists in cache - wait was successful, sessionId=23.485TDKRh5qu7zU3Ghay5tQ590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, externalsource=TfsOlapReport, key=[ts: 12/03/2009 13:04:14.192943, [serviceaccount, en-US, en-US, +0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, [Oledb, Stored, TFS, False, ], ()] BaseWB: "http://mossserver/sites/COLLECTION/Project/Reports/Bug Progress.xlsx" [0x409] [11/06/2009 19:14:04.000000] [BaseWB ID: 2]] 00000412-f5f3-0da1-0000-000050f7b00b

Excel Calculation Services        2m5e   Information   Session opened with workbook http://mossserver/sites/COLLECTION/PROJECT/Reports/Bug Progress.xlsx. [Session: 23.485TDKRh5qu7zU3Ghay5tQ590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060 User: REDMOND\granth]    00000412-f5f3-0da1-0000-000050f7b00b

Excel Calculation Services        5cag   Medium        ExcelService.PostProcessRequest: finished request of type OpenWorkbook    00000412-f5f3-0da1-0000-000050f7b00b

Excel Services Web Front End      6k39   Medium        ExcelServerProxy.ExecuteWebMethod: request of type OpenWorkbook succeeded, from server=http://mossserver:56737/SharedServices1/ExcelCalculationServer/ExcelService.asmx, session=23.485TDKRh5qu7zU3Ghay5tQ590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060, new state id=1

Excel Services Web Front End      6k30   Medium        ExcelServerProxy.SetReachable: server: http://mossserver:56737/SharedServices1/ExcelCalculationServer/ExcelService.asmx.

Excel Web Access                  5cbj   Medium        InternalEwr.RenderWebPart: Finished rendering internal EWR

And here’s the sorts of messages you’ll get if the workbook cannot be refreshed.

The workbook 'http://mossserver/sites/COLLECTION/PROJECT/Reports/Bug Progress2.xlsx' attempted to access external data using the unsupported provider 'Provider=MSOLAP.4;Integrated Security=SSPI;Persist Security Info=True;User ID="";Initial Catalog=Tfs_Analysis;Data Source=ASSERVER;Location=ASSERVER;MDX Compatibility=1;Safety Options=2;MDX Missing Member Mode=Error'. [Session: 23.3897Bz4UV5Lllg0FPFLZm8k90.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060 User: DOMAIN\user]                00000301-f5f3-0da1-0000-000050f7b00b

Refresh failed for 'TfsOlapReport' in the workbook 'http://mossserver/sites/COLLECTION/PROJECT/Reports/Bug Progress2.xlsx'. [Session: 23.3897Bz4UV5Lllg0FPFLZm8k90.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-0060 User: DOMAIN\user]

ExternalSource.ValidateConnection: Unable to get a connection: Microsoft.Office.Excel.Server.CalculationServer.Interop.ConnectionException: Exception of type 'Microsoft.Office.Excel.Server.CalculationServer.Interop.ConnectionException' was thrown.     at Microsoft.Office.Excel.Server.CalculationServer.ConnectionManager.GetConnection(ConnectionRequest connectionRequest, ExtendedConnectionInfo extendedConnInfo, Credentials credentials, Int64 privateConnectionId, Boolean auditConnection)     at Microsoft.Office.Excel.Server.CalculationServer.ExternalSource.TryGetValidatedConnection(Request request, Credentials credentials, ExtendedConnectionInfo extendedConnectionInfo, Boolean shouldReportFailure, Boolean auditConnection, Connection& connectionOut). sessionId=23.3897Bz4UV5Lllg0FPFLZm8k90.5.en...             00000301-f5f3-0da1-0000-000050f7b00b

Hopefully this is enough to lead you in the right direction, but for further help you should try the following:

Run the Team Foundation Server 2010 Best Practices Analyzer (BPA) tool that’s included in the Team Foundation Server Power Tools. It’s purpose is to help you find & fix configuration problems.
Ask a question on the Team Foundation Server - Reporting & Warehouse forums

VSTS Pioneer TFS2010: SQL Backups

grantholliday — Fri, 13 Nov 2009 22:29:00 GMT

(See this summary on the VSTS Pioneer dogfood server and all the other posts.)

Having an adequate backup strategy for your Visual Studio 2010 Team Foundation Server is a very important part of any deployment. I’ll share with you how we are running backups on our Pioneer Dogfood TFS2010 server, but you should refer to the MSDN guidance on How to: Back Up Team Foundation Server for definitive guidance.

To prevent accidental corruption of data, you must synchronize all backups of databases that Team Foundation Server uses, which include databases for SQL Server and SharePoint Products, to the same time stamp.

In the fine print a lot of people (myself included) don’t realize that you must restore all databases to the same point in time otherwise you may not have a functioning system. The stumbling block that we ran into on a recent upgrade dry-run was that the ‘Tfs_Configuration’ database was restored to a point in time that was 3 hours earlier than one of our collection databases. This caused identity information in the collection database to be out of sync with the configuration database and caused the cloned system to be unusable.

The reason this happened is because it takes us ~3 hours to backup our collection database, so the timeline went something like this:

12:00 Backup Tfs_Configuration (1 minute)
12:01 Backup Tfs_VSTSDF collection (3 hours)

The additional step we were missing in our dry-run steps was to restore the transaction logs for all the databases to a common point in time (a time after the collection database had finished backing up). See Restoring a database to a point within a backup and Performing a Complete Database Restore (Full Recovery model).

Our backup process

SQL backups are performed using stored procedures from Ola Hallengren. See SQL Server 2005 and 2008 - Backup, Integrity Check and Index Optimization

Essentially a stored procedure is called with a number of parameters from a SQL Agent Job which performs the backups.

The reason these scripts are used instead of standard SQL Maintenance Plans, is because they give us a more flexible and robust backup solution. For instance, the backup job log is written as the job executes & we have the flexibility to exclude certain databases while automatically including all new databases created on the server. See Comparison between SQL Maintenance Plans and Ola's Backup scripts

Job Name	Location	Schedule	Retention	Databases
DatabaseBackup - Weekly FULL	E:\MSSQL\Backup\Full	Occurs every week on Wednesday at 12:00:00 AM	Delete files older than 144 Hours (6 days)	SYSTEM_DATABASES, USER_DATABASES
DatabaseBackup - Nightly DIFF	E:\MSSQL\Backup\Diff	Occurs every week on Monday, Wednesday, Thursday, Friday, Saturday, Sunday at 10:00:00 PM	Delete files older than 23 hours	SYSTEM_DATABASES, USER_DATABASES * Automatically does a FULL backup if a DIFF cannot be performed.
DatabaseBackup - 15min LOG	F:\MSSQL\Backup\Log	Occurs every day every 15 minute(s) between 12:00:00 AM and 11:59:59 PM.	Delete files older than 24 hours	SYSTEM_DATABASES, USER_DATABASES

Then we have our IT group’s tape backup process come along every night and backup the SQL backup files to tape for on-site and off-site retention.

Backup Job Alerts

Email alerts are sent to a monitoring email list when each SQL backup job completes (regardless of success or failure). This is done using the SQL Mail functionality.

Compression

Additionally, if you are running SQL Server 2008 Enterprise Edition – you have SQL Backup Compression available to you. This can greatly reduce the size of your backups and the amount of I/O required, therefore decreasing the time it takes to perform backups & restores. The tradeoff is of course, additional CPU usage. See the following SQLCAT Tuning the Performance of Backup Compression in SQL Server 2008 article for more information on this.

VSTS Pioneer TFS2010 Dogfood Server: Statistics October 2009

grantholliday — Fri, 06 Nov 2009 09:21:36 GMT

(See this summary on the VSTS Pioneer dogfood server and all the other posts.)

As with the last few months, October was another busy month for dogfooding TFS. Since the Pioneer server has been such a success for Work Item Tracking and Source Control, we’ve really ramped up our efforts to dogfood every single feature as much as we can. The benefits of having everything online really lets us make sure our end-to-end integration is working as well as feel the pain before we share it with our customers. The most notable changes this month are:

Each of the different business units within VSTS have setup their own SharePoint MOSS Dashboards with bug trends, etc.
We’re now using Team Build 2010 with Gated Checkin on our part of the tree (we still have to follow the divisional processes of custom nightly build scripts at this stage)
We enabled Test Case Management features and are now dogfooding the Test Case Management (TCM) features for our major test passes
We setup and enabled the Lab Management functionality on our server and we’re starting to dogfood that across the organization as well

Here’s the latest stats:

Recent Users: 915

Version Control

Version Control Users: 622
Files: 11,053,314
Compressed File Sizes: 284,480
Uncompressed File Sizes: 870,228
Checkins: 29,275
Shelvesets: 7,249
Merge History: 10,897,164
Pending Changes: 13,344
Workspaces: 1,607
Local Copies: 382,464,960

Work Item Tracking

Users with Assigned Work Items: 2,607
Total Work Items: 284,813
Areas and Iterations: 2,910
Work Item Versions: 2,892,664
Work Item Attachments: 115,501
Work Item Queries: 5,728

Commands (last 2 weeks in October)

Work Item opens: 154,788
Work Item queries: 120,537
Work Item updates: 40,242
Checkins: 2,630
Uploads: 10,440
Shelves: 4,553
Downloads: 251,032
Gets: 41,615

DevDiv TFS Server

On top of enabling all these new scenarios on the Pioneer server, we also managed to get the main DevDiv TFS server upgraded from SQL 2005 to SQL 2008. This was the first step to getting the server to TFS2010. I can’t say it was all smooth sailing, but it looks like we’re through the worst of it now (man, what a fortnight…). The good news is that all of the extra performance optimizations we had to make to cope with our dataset sizes also made it into TFS2010 product.

Although no install guide will tell you, I highly recommend you put on SQL 2008 SP1 CU4 if you’re running a large, busy TFS2008 or TFS2010 server – in the processes of testing the upgrade and after the actual upgrade to SQL 2008, we hit these bugs which the SQL team quickly reacted and fixed for us:

KB967178 - When you restore a SQL Server 2005 backup file in SQL Server 2008, the operation takes much longer than when you restore the same backup file in SQL Server 2005 (fixed in SQL08 SP1 CU1)
KB970823 - CPU cost increases suspiciously when the number of threads working on the same index increases (fixed in SQL08 SP1 CU3)
KB970133 - Backup Log hangs when using compressed backups and the destination drive runs out of disk space. SQL Restart is required (fixed in SQL2008 SP1 CU4)

Now, onto the next Pioneer upgrade.

TFS2010: Diagnosing Email and SOAP subscription failures

grantholliday — Thu, 29 Oct 2009 01:33:04 GMT

Sometimes emails alerts such as "Work Item Changed" will fail to send. Sometimes SOAP (web service) subscriptions will not be called correctly.

To diagnose these, you can look in the tbl_JobHistory table in the Tfs_Configuration database.

If you run this SQL command, it will retrieve the last 10 failed notification jobs:

SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage

FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]

WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0

ORDER BY StartTime DESC

The JobId GUID is the well-known GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension”

The DATEADD() function converts the date from UTC to local Pacific time.

ResultMessage:

There were errors or warnings during notification delivery. 0/4 emails delivered. 0/6 soap notifications delivered. 1 errors. 0 warnings. Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings.

To get detailed information about why the job failed, you will need to set a flag in the TFS registry. This is different from the windows registry and probably warrants a blog post on it’s own.

Changing NotificationJobLogLevel setting in the TFS registry using PowerShell

If you have VS2010 & PowerShell installed on your local machine and you have the appropriate permissions on the server, then you can run these steps from there. Otherwise, you will need to logon to a machine that does, or an AT. If you logon to the AT, you will need to run PowerShell as an Administrator.

To change the registry setting, replace the server url & collection name and follow these steps:

Open PowerShell
Run the following commands:

# Load client OM assembly.
[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a");

$collectionBaseUrl = "http://YOURTFSSERVER:8080/tfs/YOURCOLLECTION/";

$tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($collectionBaseUrl);
$collectionHive = $tfs.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry]);

# Set the setting in the collection hive.
$collectionHive.SetValue("/Service/Integration/Settings/NotificationJobLogLevel", "2");

It’s a good idea to set this back to “0” once you’re done with it, to avoid filling up the job history logs unnecessarily.

Once you have set this setting, the next time the job runs and fails, it will log more detailed information including the exception and stack trace. For example:

There were errors or warnings during notification delivery. 0/0 emails delivered. 0/3 soap notifications delivered. 3 errors. 0 warnings. ------------------------------- Notification not delivered. Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://eventsserver/EventHandler.svc) Exception: Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: Team Foundation services are not available from server http://eventsserver/EventHandler.svc. Technical information (for administrator): HTTP code 415: Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'. ---> System.Net.WebException: The remote server returned an error: (415) Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'.. at System.Net.HttpWebRequest.GetResponse() at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.AsyncWebRequest.ExecRequest(Object obj) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader) at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response) at Microsoft.TeamFoundation.JobService.Extensions.Core.TeamFoundationNotificationClient.Notify(String eventXml, String tfsIdentityXml, Subscription subscription) at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification, TeamFoundationIdentityService identityService)

This particular error message indicates that the client was expecting a SOAP 1.1 message, and the server sent it a SOAP 1.2 message.

If your email alerts are failing to send, then you’ll get an SMTP error back that shows you where to start looking.

VSTS Pioneer TFS2010 Dogfood Server: Upgrading

grantholliday — Mon, 19 Oct 2009 08:58:00 GMT

(See this summary on the VSTS Pioneer dogfood server and all the other posts.)

One of the major goals of the Pioneer Team Foundation Server is to upgrade it to early builds so that we can get some “bake time” with them and feel confident with what we’re releasing to customers. Since we spun up the server, we’ve done two successful upgrades and we’re already planning the next one:

July 1st – Initial installation of Beta 1 build
August 29 – Upgrade to 8/22 build from the VSTS branch
September 17 – Upgrade to 9/17 build from the Beta 2 branch (yes, the build dropped at 5PM and we used it to upgrade at 6PM)
November – Upgrade to a post-Beta2 build

The process of testing the upgrades on a copy of our database is just as important as running the upgrade itself. With our interesting dogfood topology of a dedicated SQL server, NLB/multiple application tiers, a dedicated MOSS server, virtual machines and use of virtual DNS names, we find some interesting bugs that we’re able to fix before things are locked down for shipping.

There’s no whitepaper on “How to Dogfood” at Microsoft, so our process for deploying perpetual upgrades is something that just kind of happened. Here’s basically what we do:

Depending on where we are in the cycle and the amount of churn in the code base, we might add another iteration or skip one. What’s most interesting about this process, is that it’s driven by the end date. Based on the product schedule, we know when our windows of opportunity are for upgrading and we work back from then.

Tips for upgrading

Here are some tips to consider when planning your own upgrades:

Hardware – Get your production & pre-production servers in place early. Getting new hardware or re-configuring existing hardware usually has a long lead time. Make sure that you’ve also got enough storage space.
Start fresh – If at all possible, give yourself the best chance at success by starting with a fresh box. Sure, plenty of products support in-place upgrades & clean uninstalls – but in my experience, flattening the box and starting fresh lays a good foundation for the future.
Backups – Organize a copy of your production backups that you can test the upgrade with. Not only does it give you peace of mind that your backups work, it also helps you line up the DBA’s for when it’s time to do the real upgrade.
Start with the latest OS – If you start with Windows 2008 R2 now, it’s one less OS upgrade (and downtime) you’ll have to do in the future. It also means you’ll be on 64-bit, which is a very good thing.
Virtualization – We run our 3x Application Tiers as virtual machines with no problems whatsoever. This is great for when we’re installing pre-beta copies of the .NET framework that don’t have clean uninstalls yet. In our last upgrade we just turned off the old application tiers and spun up three fresh ones. You can treat AT’s as throwaways in a TFS2010 NLB topology, since there is no state stored on them.
Dry-Runs – Run through your plan and make sure everything is going to work. Review it with others and get their feedback.
Have a test plan – Once you’ve done the upgrade, you need to be sure that everything is back online and working as expected. Checkin, Checkout, Open work item, Save work item, Web Access, Sharepoint, Reports, Cube, Alerts, Monitoring, Builds, etc.
Do pre-requisites ahead of time – For example, to install SQL2008 you need to have .NET3.5 and Windows Installer 4.5 both installed first and both of these require a reboot. If you plan to do one on the first weekend and the other on the second weekend, it means that you don’t waste time during your more important SQL upgrade waiting for reboots.
Do one thing at a time – This is an extension of the last point. Let’s say you’re moving from a single-server TFS2008 SP1 on SQL2005 and Windows 2003 to a dual-server TFS2010 on SQL2008 SP1 and Windows 2008 R2. If it were me, I’d take it in steps: upgrade SQL, make sure everything still works, move to dual-server, check again, upgrade to TFS2010. The more steps you try and bundle together, the higher your risk of failure.

Upgrade logistics

I’ve been involved with at least 5 major upgrades this year and here are some things I’ve learnt from running them:

Book a conference room
- This is the single most valuable thing you can do. Try and get anybody who is actually doing any steps in the upgrade (DBA< IT, Helpdesk, etc) to come to the room.
- Have a spare computer or project your own desktop on the screen. This is useful for synchronizing with others in the room, running ‘ping –t’ when a server reboots and doing “group debugging” sessions when problems happen.
Setup a “party line” conference call. Invite anybody who wants to “listen in” to join. Invite your manager.

It’s also useful for anybody who you might need help from to have these details. If things go south, you don’t waste time trying to setup a conference call – it’s already there.

Setup a live meeting / desktop sharing session. Invite anybody who wants to be a “fly on the wall” to join.
- Sure, you can log a bug that says “when I clicked on X on screen Y, setup blew up” and people might believe you. But if you’re sharing your desktop and other people see it happen, then you have some people to back you up.
- (I’ve been meaning to do a screen recording of our upgrades, but various technology problems have prevented it each time – I think it would be useful for reviewing and improving the process for next time)
Setup an “on call / escalation” list. Find out who you need to call if you have problems with one of your dependent systems. The last thing you want to be doing is sitting on hold to a L1 helpdesk if your server doesn’t reboot cleanly or something.
Send at least 3 notifications to users of the downtime
- 1st Notice – Give as much notice as possible so people can plan milestones around the date. Even if it’s not set in stone yet, people would rather know about it and have it cancelled than to find out at the last minute.
- 2nd Notice – 7 days or on the Monday before a weekend upgrade is good for this.
- 3rd Notice – On the day of the upgrade (or 24 hours before if it’s a weekend), send another one.
- Guidelines: Make sure you describe what the impact to users is, any actions they need to take and where they can go to get future status updates and support.
Provide regular status to stakeholders. Start an email thread and trickle through progress as it happens. By keeping them informed, you buy their trust and their support when things don’t go as planned.

Planning your own upgrade

Brian Keller has a great post on how to Get ready to “go live” with Team Foundation Server 2010 beta 2! He includes two useful documents:

Overview presentation (.PPTX)

Detailed checklist (.DOCX)

I’m also going to share with you a copy of the generic deployment plan template that I’ve been using for the Pioneer upgrades.

Deployment Plan Template(.XLSX)

Planning, testing and upgrading to these new releases has been a fun experience and it’s a bonus when everything goes smoothly. It’s great to see our experiences influence the product and make it the best release yet.

VSTS Pioneer TFS2010 Dogfood Server: Statistics September 2009

grantholliday — Mon, 19 Oct 2009 07:13:18 GMT

(See this summary on the VSTS Pioneer dogfood server and all the other posts.)

September was a busy month for dogfooding TFS – all of the VSTS team were completely moved across and on 9/17 we upgraded to an almost final TFS2010 Beta 2 build. Here are the numbers from the end of September:

Recent Users: 887

Version Control

Files & Folders: 10,722,118
Compressed File Sizes (MB): 263,350
Uncompressed File Sizes (MB): 773,731
Checkins: 23,369
Pending Changes: 55,697
Local Copies: 303,331,190

Work Item Tracking

It looks like I don’t have the work item tracking stats for September – I’ll have them next month.

Commands (last 2 weeks in September)

Work Item opens: 113,128
Work Item queries: 101,203
Work Item updates: 23,653
Checkins: 1,855
Uploads: 6.622
Shelves: 2,402
Downloads: 216,452
Gets: 30,201

Table sizes

One of the interesting metrics that I’m also tracking is how our tables and indexes are growing on the disks. Here’s the top 10 for this month:

Table	Rows	Reserved (MB)	Data (MB)	Index (MB)	Unused (MB)
tbl_Content	5,915,547	277,260	275,129	4	2,127
tbl_LocalVersion	303,331,190	81,562	44,789	36,574	199
Attachments	63,783	11,928	11,908	13	7
tbl_Version	12,129,564	3,418	1,031	2,369	17
WorkItemLongTexts	2,728,618	3,314	2,533	777	5
tbl_VersionedItem	15,754,176	2,983	1,598	1,382	3
WorkItemsWere	1,239,831	2,482	1,617	863	2
tbl_PropertyValue	33,293,914	1,320	523	793	4
tbl_File	5,717,981	722	442	266	14

TFS2010: Installing pre-requisites from the command line

grantholliday — Mon, 19 Oct 2009 06:31:35 GMT

On the Team Foundation Server team, there is a new build of the server every night that comes out of our build lab. These vary in quality throughout the development cycle and the QA teams look at them and rate them as “Self Test”, “Self Test (with workarounds)”, or “Self Toast”. The QA & dev guys have lots of fancy automation for deploying nightly builds and testing them out, but I like to stay away from that stuff and use the builds as if I were a real customer. Once a build is declared “Self Test”, I’ll grab a copy and spin it up in a VM to see what’s changed.

The two preparation things I’ve done that have really sped up my installation turnaround times are:

Create a sysprep’d Windows Server 2008 R2 VHD
Create a SQL2008 SP1 slipstreamed installation point

Once I had those in place, I then just used these two long command lines below to install SQL & IIS. Of course, none of this is necessary for a “TFS Basic” install, since the installer will drop down IIS & SQL Express for you. See Brian’s blog on TFS 2010 for SourceSafe Users for more details on that.

Creating a sysprep’d Win2008 R2 VHD

Creating a sysprep’d VM is easy. Create a new VM and install a fresh copy of Windows 2008 R2. If you have a domain, join it to the domain and add yourself to the local administrators group (this will be retained after you sysprep & rejoin to the domain again). I don’t usually add the IIS role at this point, since I use these VHD’s for other things. Once you’re happy with it, run the following command:

C:\Windows\System32\Sysprep\Sysprep.exe /oobe /generalize /shutdown

Then take a copy of the VHD and name it something like ‘Win2008R2Sysprepped.vhd’. Whenever you need a fresh machine, just copy this VHD to a new file and away you go.

Creating a SQL2008 SP1 slipstreamed installation point

Follow the instructions for creating a slipstreamed drop on the SQL Server Setup team’s blog. Two things to note when doing this:

Paths matter – it’s best to pick a folder name and stick with it whenever you use it. e.g. C:\SQL2008SP1\
You DO have to download the ia64 & x86 SP1 packages & slipstream them as well, otherwise your setup.exe will fail

Once you’ve created the installation point, save it away somewhere – it’s really useful and it saves so much time by not having to do an RTM install followed by a SP1 install.

Installing TFS pre-requisites from the command line

Boot into your fresh 2008 R2 VM
Give it a new name, join it to the domain, apply any updates, reboot, etc
Install SQL2008 SP1 from your installation point:

Open a command prompt as administrator
Type: pushd \\fileserver\SQL2008SP1
Type the following command:

Setup.exe /q /ACTION=Install /FEATURES=SQL,FullText,AS,RS,Tools /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT="NT Authority\Network Service" /RSSVCACCOUNT="NT Authority\Network Service" /ASSVCACCOUNT="NT Authority\Network Service" /AGTSVCACCOUNT="NT Authority\Network Service" /SQLSVCStartuptype=2 /RSSVCStartupType=2 /ASSVCSTARTUPTYPE=2 /AGTSVCSTARTUPTYPE=2 /BROWSERSVCStartupType=2 /SQLSYSADMINACCOUNTS="Builtin\Administrators" /ASSYSADMINACCOUNTS="Builtin\Administrators" /SQMREPORTING=1 /ERRORREPORTING=1 /INDICATEPROGRESS

Install required IIS components:

Open a command prompt as administrator
Type the following command:

ServerManagerCmd -install WAS WAS-Process-Model WAS-NET-Environment WAS-Config-APIs Web-Server Web-WebServer Web-Common-Http Web-Static-Content Web-Default-Doc Web-Dir-Browsing Web-Http-Errors Web-Http-Redirect Web-App-Dev Web-Asp-Net Web-Net-Ext Web-ISAPI-Ext Web-ISAPI-Filter Web-Health Web-Http-Logging Web-Request-Monitor Web-Security Web-Windows-Auth Web-Filtering Web-Performance Web-Stat-Compression Web-Mgmt-Tools Web-Mgmt-Console Web-Mgmt-Compat Web-Metabase Web-WMI Web-Lgcy-Scripting Web-Lgcy-Mgmt-Console

At this point, you’re now ready to run setup.exe from the TFS media. You have a fully configured SQL server and the requires IIS pre-requisites installed.

Other useful SQL 2008 setup.exe command lines

For a TFS “Application Tier Only” install, IIS is one pre-requisite and the “SQL Client Tools Connectivity” is the other one. This is required so that the TFS Job Agent can talk to Analysis Services and tell it to process the warehouse. Here’s how to install just the client tools:

setup.exe /ACTION=Install /FEATURES=Conn /ERRORREPORTING=1 /INDICATEPROGRESS /SQMREPORTING /Q

If you just want to install SQL Server Management Studio on your developer machine:

setup.exe /ACTION=Install /FEATURES=ADV_SSMS /ERRORREPORTING=1 /INDICATEPROGRESS /SQMREPORTING /Q

For more information, see How to: Install SQL Server 2008 from the Command Prompt and available features.

Scripting Team Project Creation in TFS2010

grantholliday — Sat, 26 Sep 2009 00:17:46 GMT

In TFS2008 SP1 and the subsequent power tools release it was possible to create a team project from the command line. You can read Will-Peter’s detailed blog post on Creating a Team Project from the command line in TFS2008.

In TFS2010, it’s still not as good as we would like (it’s still client-based rather than server-based). But there’s one important difference – the feature has been rolled into the product and is no longer a power tool.

How to create a team project from the command line in TFS2010:

Create an XML file with the following contents and save it somewhere (e.g. C:\temp\pcw.xml). Substitute the highlighted as appropriate:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="ProjectCreationSettingsFileSchema.xsd">
       <TFSName>http://tfsserver:8080/tfs/DefaultCollection</TFSName>
       <LogFolder>c:\temp</LogFolder>
       <ProjectName>TestAgile2</ProjectName>
       <ProjectSiteEnabled>true</ProjectSiteEnabled>
       <ProjectSiteTitle>TestAgile2</ProjectSiteTitle>
       <ProjectSiteDescription>TestAgile2 project created from command line</ProjectSiteDescription>
       <SccCreateType>New</SccCreateType>
       <SccBranchFromPath></SccBranchFromPath>
       <ProcessTemplateName>MSF for Agile Software Development v5.0</ProcessTemplateName>
</Project>

Once you have created the file, you need to prepare Visual Studio to be automated and trigger the automation from the command line.

Open Visual Studio 2010
Go to View > Team Explorer and make sure that the Team Explorer tab is active
Close Visual Studio 2010
Open a command prompt
Type: CD /D %ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE
Type: devenv /command "File.BatchNewTeamProject C:\temp\pcw.xml"

At this point, Visual Studio will open and the Team Explorer tab will show “Connecting to server\collection…”. In the background it’s actually creating the team project. You can verify this by looking for a txt file with the name of your project in the path you specified for LogFolder. (e.g. c:\temp\TestAgile2.txt)

Once the project has been created, you can select ‘Add existing team project’ and select it from the dialog.

If you want to create a fully-automated project provisioning system there are a few other things that you’ll need to do on top of creating the team project. Some of these things are:

Add users to TFS groups (Project Administrators, Contributors, Readers)
Add permissions in sharepoint
Add permissions in reporting services

VSTS Pioneer TFS2010 Dogfood Server: Statistics August 2009

grantholliday — Thu, 27 Aug 2009 21:44:00 GMT

(This blog post is part of a series of posts on the new VSTS Pioneer TFS dogfood server.)

Everybody loves statistics, so here are the statistics for the first month on the VSTS Pioneer server.

Recent Users: 415

Work Item Tracking

Users with Assigned Work Items: 2,454
Work Items: 111,319
Areas & Iterations: 2,811
Work Item Versions: 1,056,416
Work Item Attachments: 50,833
Work Item Queries: 2,965

Version Control

Files & Folders: 8,561,565
Compressed File Sizes (MB): 325,080
Uncompressed File Sizes (MB): 856,859
Checkins: 18,867
Shelvesets: 1,982
Merge History: 1,789,333
Pending Changes: 48,155
Workspaces: 727
Local Copies: 184,231,208

These statistics are captured by using the TFS Best Practices Analyzer and a custom configuration file. The configuration tells the BPA tool to run a series of SQL queries and dump the results to an XML file. I have then created an XSLT that transforms the results into a more user-friendly HTML file. The BPA tool is then scheduled to run every night and generate a fresh set of statistics. I’m working with the TFS BPA team to get this configuration built into the next version of the BPA tool – I don’t have an ETA on when that will ship, but stay tuned.

This month I’ve also pulled the last 30 days of PerfMon counters to see how our hardware choices are holding up. If you’d like to setup something like this for yourself, have a look at my previous posts.

Disk Performance

Things are looking OK on the IO front. 95% of the time we’re pushing 45MB/sec (Read) and 33MB/sec (Write) or less to and from our storage. The response time is <40ms which is nothing to worry about for the configuration we have.

LogicalDisk(_Total)	50th Percentile	95th Percentile	99th Percentile
Avg. Disk sec/Read	0.004	0.039	0.371
Avg. Disk sec/Write	0.003	0.023	0.063
Disk Read Bytes/sec	2,142,548	44,926,021	94,273,782
Disk Write Bytes/sec	1,908,518	33,029,252	70,802,346
Avg. Disk Bytes/Transfer	42,867	133,887	436,920

Memory

The memory usage on our SQL server has remained completely constant. This is because SQL is set to use a fixed amount and Analysis Services & Reporting Services share the rest.

CPU Performance

(Avg: 32%). The majority of this usage is SQL Server, with a small portion going to Analysis Services & Reporting Services. It’s nice to see that we still have plenty of capacity left for the usage to grow.

That’s all for statistics, stay tuned for more next month.

VSTS Pioneer TFS2010 Dogfood Server – Hardware & Topology

grantholliday — Thu, 27 Aug 2009 18:23:00 GMT

(This blog post is part of a series of posts on the new VSTS Pioneer TFS dogfood server.)

While running the main TFS server for ~3,500 users in Developer Division we got a pretty good idea of the hardware required to support our enormous number of files, local versions & branches.

We knew that:

our SQL Server load patterns were very IO intensive
our SQL Server load patterns were very CPU intensive
our disk usage will grow by about 300% per year
the number of operations the server performs will increase

Servers

We wanted to purchase and setup hardware that was flexible enough and designed to scale as our needs do. For this reason we settled on two identical servers that are part of the Microsoft IT recommended systems for our needs:

HP ProLiant DL580G5 (room to add two more processors & upgradeable to 256GB RAM)
Windows 2008 x64 SP2
2x 6-core Intel Xeon X7460 2.66Ghz processors
32GB RAM
QLogic QLE2460 4Gbps Fiber Channel Host Bus Adapter (SAN connection)

According to Joe Chang who studies SQL Server performance, the “The X7460 (Dunnington) is a clear winner at 4-way for the high-call volume apps.” The 6-core CPUs also offer a pretty significant performance boost over the older quad-core CPUs:

“On TPC-C, the 7460 six core generated a 34% edge over the quad core AMD and 56% advantage over the quad core X3750”, and
“On TPC-E, the six core showed a 49% edge over the older quad core”

Virtual Machines

We are running 5 virtual machines at the moment with almost identical configurations. Since we’ll be constantly upgrading things like the .NET CLR that often don’t support build-to-build upgrades or clean uninstall, running the application tiers as VM’s makes perfect sense. It also improves our rollback story if we have a failed upgrade. (More about upgrade in a future post).

VM	Purpose	Configuration
AT01, AT02, AT03	TFS Application Tier	Windows Server 2008 R2 4 Virtual processors 4GB RAM 2 Virtual NICs 1 VHD for C$ 1 VHD for version control cache
MIRROR	TFS-TFS Connector tool runs here	Windows Server 2008 RTM 4 Virtual processors 4GB RAM 1 Virtual NIC 1 VHD for C$ 1 VHD for version control temporary workspace
MOSS	Microsoft Office SharePoint 2007 SP2 (Complete install) Web Front End, Excel Services, Search Services & Shared Services Provider	Windows Server 2008 R2 4 Virtual processors 4GB RAM 1 Virtual NIC 1 VHD for C$ Databases reside on physical SQL server

Storage

Since these servers are hosted by the DevDiv IT guys, we went with their recommendation for storage. They already have a massive storage infrastructure for build & drop servers built on Xiotech technology, so it was easy for them to add additional ISE units to meet our size & performance requirements. In the end we ended up with the following configuration:

5TB of “Balanced” storage
5TB of “Performance” storage

The SQL Customer Advisory Team’s – Storage Top 10 Best Practices is a great resource when planning your own TFS deployment. In particular we specifically considered the following:

Isolate log from data at the physical disk level (e.g. O$ drive is on separate spindles)
Consider configuration of TEMPDB database (e.g. T$ drive is on separate spindles)
Make sure to give thought to the growth strategy up front (e.g. We can add additional ISE units at any time and grow any drive without having to migrate data)
Lining up the number of data files with CPU’s has scalability advantages for allocation intensive workloads. (e.g. we have 12 datafiles for TempDB because we have 12 CPU cores)

Here’s how the virtual disks (LUNs) are configured on the server:

SERVER1 (SQL, SSRS, SSAS)

Drive	Purpose	Storage Type	Size (Gb)
E$	SQL backup dump drive	Balanced	2741
G$	LocalVersion filegroup	Performance	1673
I$	OLAP data	Balanced	25
J$	All database data	Performance	1833
K$	Version filegroup	Performance	971
O$	Transaction Logs	Balanced	1500
T$	TempDB	Performance	500

SERVER2 (Hyper-V Host)

Drive	Purpose	Type	Size (Gb)
E$	Hyper-V VM's	Balanced	743

What’s cool about the Xiotech stuff is that it’s like Lego blocks – as you need more storage or more performance, you just buy more blocks and plug them in. What’s even cooler about it, is that you can control the SAN using web services – the build lab uses the SANMan tool they built to move virtual disks between build servers and automate the provisioning of disk space.

Topology

At just over 400 users, we probably don’t need three AT machines in a NLB configuration. However, NLB is an important new feature of TFS2010 and it’s a scenario that we need to dogfood for ourselves.

Here’s some quick facts about our topology:

We have a couple of “friendly name” DNS records for the NLB cluster IP, Reporting Services & Analysis Services – this allows us to change the underlying infrastructure without users having to connect to a new addresses.
We’re running Windows 2008 R2 RTM on our VM’s & soon on our physical servers
A single VM was created, then sysprep’d and copied. This means that we can spin up a new VM in a very short time period. (Copy VHD, Add to Hyper-V, Start, Join Domain, Apply Updates, Done.)
SERVER2 is running an SMTP server that relays mail on behalf of each of the other servers. At Microsoft, the corporate mail servers will only accept mail from authenticated users that have mailboxes. By running the SMTP server as an authenticated user, it allows us to run the application tiers as Network Service. This means when we have to change a password to a service account, it doesn’t cause a TFS service interruption – just a brief SMTP interruption.
Since all the SQL services are running on the same box, they don’t need to be run as a domain accounts either – no service interruption & less places that need a password changed when it expires.

That’s it for hardware and topology, next post is on performance and Dogfood statistics.

Permissions required for Team Build Retention Policy in TFS2008

grantholliday — Thu, 25 Jun 2009 23:34:00 GMT

I got a question from one of our internal email lists today:

Our TFS build service is owned by a generic build account – domain\tfsbuild. We have retention policy set up to retain a fixed set of builds. The old builds were deleted when viewed via Team Explorer. However, on the drop server, the builds are not deleted. Apparently there’s an access issue when TFS attempted to delete the old builds.

Someone told me that the TFS service account will need admin rights. Is this true?

Aaron from the Team Build team did a good job of explaining what was happening:

When TFS deletes build drops it first tries to have the build agent do the deletion, and then falls back to the AT. Ideally, then, both the build service account and the AT service account would have full control to the root of the drop location. This does not mean that these accounts need to be administrators on the box, however – that should not be a requirement.

There should be errors in the event log on the AT whenever TFS tries to fully delete a drop – these should include the error that caused the deletion to fail. You might have to get your friendly TFS administrator to have a look on the server and see if they find any such errors for the builds whose drops didn’t get deleted.

Another is that the drop locations were in use when they got deleted – this is fairly common, typically resulting in everything on the drop but the handful of files that were in use getting deleted, along with whatever folder structure got preserved as a result. You might have a look at your undeleted drops to see if everything is there, or just a few files.

TFS Adoption at Microsoft – May 2009

grantholliday — Wed, 24 Jun 2009 01:26:00 GMT

As promised last week in my Dogfooding TFS show on the RadioTFS podcast, here is the latest on TFS within Microsoft that myself and Brian have posted about in the past.

May had the all-time peak in the total number of unique active users with 15,613. In addition, the total number of source code files stored surpassed 60 million. The total number of projects continues to increase and will end the fiscal year over 3,500, growing by over 1,300 in the last year. MSIT RXD continues to lead the way using a TFS2010 pre-beta1 build with an average of 100 active users per week.

Summary of growth indicators for the month:

May 2009	Growth	Totals
Unique Active Users	+29	15,613
Total Active Users	+24	17,680
Projects	+98	3,434
Work Items	+170,436	3,934,238
Source Code Files	+1,760,516	60,253,062

We call the internal TFS at Microsoft the “TFS Service Offering”. All the servers are setup and maintained by our partners in Microsoft IT.

Active Users - Service Offering:

Projects & Unique Active Users Across Service Offering:

The number of team projects was up by 98 this month. Added to the chart this month are labels for the releases since January 2007 to give some perspective on growth and when versions were released.

Team Builds:

Combined data from across the Service Offering.