- VSTS Pioneer TFS2010: SQL Backups
-
(See this summary on the VSTS Pioneer dogfood server and all the other posts.)
Having an adequate backup strategy for your Visual Studio 2010 Team Foundation Server is a very important part of any deployment. I’ll share with you how we are running backups on our Pioneer Dogfood TFS2010 server, but you should refer to the MSDN guidance on How to: Back Up Team Foundation Server for definitive guidance.
To prevent accidental corruption of data, you must synchronize all backups of databases that Team Foundation Server uses, which include databases for SQL Server and SharePoint Products, to the same time stamp.
In the fine print a lot of people (myself included) don’t realize that you must restore all databases to the same point in time otherwise you may not have a functioning system. The stumbling block that we ran into on a recent upgrade dry-run was that the ‘Tfs_Configuration’ database was restored to a point in time that was 3 hours earlier than one of our collection databases. This caused identity information in the collection database to be out of sync with the configuration database and caused the cloned system to be unusable.
The reason this happened is because it takes us ~3 hours to backup our collection database, so the timeline went something like this:
- 12:00 Backup Tfs_Configuration (1 minute)
- 12:01 Backup Tfs_VSTSDF collection (3 hours)
The additional step we were missing in our dry-run steps was to restore the transaction logs for all the databases to a common point in time (a time after the collection database had finished backing up). See Restoring a database to a point within a backup and Performing a Complete Database Restore (Full Recovery model).
Our backup process
SQL backups are performed using stored procedures from Ola Hallengren. See SQL Server 2005 and 2008 - Backup, Integrity Check and Index Optimization
Essentially a stored procedure is called with a number of parameters from a SQL Agent Job which performs the backups.
The reason these scripts are used instead of standard SQL Maintenance Plans, is because they give us a more flexible and robust backup solution. For instance, the backup job log is written as the job executes & we have the flexibility to exclude certain databases while automatically including all new databases created on the server. See Comparison between SQL Maintenance Plans and Ola's Backup scripts
|
Job Name |
Location |
Schedule |
Retention |
Databases |
|
DatabaseBackup - Weekly FULL |
E:\MSSQL\Backup\Full |
Occurs every week on Wednesday at 12:00:00 AM |
Delete files older than 144 Hours (6 days) |
SYSTEM_DATABASES, USER_DATABASES |
|
DatabaseBackup - Nightly DIFF |
E:\MSSQL\Backup\Diff |
Occurs every week on Monday, Wednesday, Thursday, Friday, Saturday, Sunday at 10:00:00 PM |
Delete files older than 23 hours |
SYSTEM_DATABASES, USER_DATABASES
* Automatically does a FULL backup if a DIFF cannot be performed. |
|
DatabaseBackup - 15min LOG |
F:\MSSQL\Backup\Log |
Occurs every day every 15 minute(s) between 12:00:00 AM and 11:59:59 PM. |
Delete files older than 24 hours |
SYSTEM_DATABASES, USER_DATABASES |
Then we have our IT group’s tape backup process come along every night and backup the SQL backup files to tape for on-site and off-site retention.
Backup Job Alerts
Email alerts are sent to a monitoring email list when each SQL backup job completes (regardless of success or failure). This is done using the SQL Mail functionality.
Compression
Additionally, if you are running SQL Server 2008 Enterprise Edition – you have SQL Backup Compression available to you. This can greatly reduce the size of your backups and the amount of I/O required, therefore decreasing the time it takes to perform backups & restores. The tradeoff is of course, additional CPU usage. See the following SQLCAT Tuning the Performance of Backup Compression in SQL Server 2008 article for more information on this.
- TFS2010: Public Workspaces
-
Public Workspaces is a feature that we built last year mainly for our internal build lab customers. Before TFS2010, if one user tries to use a workspace that belongs to another user on the same machine, they will receive an error message:
No workspace matching *;SecondUser on computer COMPUTER found in Team Foundation Server http://tfsserver:8080.
The Public Workspaces feature removes this limitation of one user per workspace and allows multiple users to use the same workspace mappings on a single machine. Unlocked workspaces are sometimes also referred to as "Shared workspaces", as they can be shared among multiple people.
Additionally, we enabled the ability to change the owner of a workspace, if you don’t want to share a workspace at all, but just reassign it to a different user.
Scenarios
Shared Machines
"I need to setup a shared ProductX enlistment machine; we’d like to use it to run tests, integrations and such."
"Our team has a rotating branch-owner role, so we have no choice but to use a service account."
"We find it very useful to have a shared machine that anyone on our team can use to investigate issues, checkin fixes, etc."
"Is there a way to make a TF enlistment user-neutral, so that anyone who logs into the build machine can open the enlistment, and use TF commands – and those commands will be run as the username of the loggedin user?"
Save disk space
"We need to use it to save disk space on shared machines."
Split personalities
"I have two admin accounts on the box. I use one of them to enlist in the branch and work with TFS. The problem is when I use another one (labaccount vs. personal account) I can login, but I cannot run any tf command because enlistment belongs to the first user only."
Hotdesking
"I'm taking over another dev's machine for a week while he works on another project and he has all the development tools already setup"
Workarounds for TFS2008
Before this feature existed, there were are few different workarounds that people were using:
| Workaround | Description |
| Shelvesets | The various developers pass shelvesets around |
| Run As | The owner starts a cmd shell / VS window with RunAs |
| Shared User Account | Create the workspace using a shared system account, then have all developers use that account on that machine. This has the downside that all changes are checked in as the shared account, rather than the actual user. You can overcome this by using the "/author" flag with the actual username, but only if the shared account has the CheckinOther permission. |
| SUBST path substitution | Use the windows SUBST.EXE "Substitution" utility to map a different drive letter to the same physical path. This tricks TFS into thinking that the workspace doesn't exist. |
Usage
The owner of the workspace should run the "tf workspace" command and select a workspace permission profile from the drop-down box. The default is "private."
- A Private workspaces works the same as a workspace in TFS2008.
- A Public (limited) workspace can be used by any valid user. Only the owner can check in or administer the workspace
- A Public workspace can be used by any valid user. Anybody can check in or administer the workspace as long as they have permissions to the underlying files. The checkins will still be attributed to the person checking them in.
Workspace permissions profiles
| | Private | Public-limited | Public |
| Read | Everyone | Everyone | Everyone |
| Use (most operations -- such as pend changes, resolve, etc.) | Owner only | Everyone | Everyone |
| Check in | Owner only | Owner only | Everyone |
| Administer (change owner, mappings, etc) | Owner only | Owner only | Everyone |
Once a workspace has been made public or public-limited, another user can log onto the machine with the workspace and use the workspace. The other user will need to refresh their local workspace cache manually by using the "tf workspaces /collection:http://server:8080/tfs/collection" command before they will be able to use the workspace (otherwise they will not know the workspace is there -- because they did not create the workspace, and the local workspace cache is per-user).
- VSTS Pioneer TFS2010 Dogfood Server: Statistics October 2009
-
(See this summary on the VSTS Pioneer dogfood server and all the other posts.)
As with the last few months, October was another busy month for dogfooding TFS. Since the Pioneer server has been such a success for Work Item Tracking and Source Control, we’ve really ramped up our efforts to dogfood every single feature as much as we can. The benefits of having everything online really lets us make sure our end-to-end integration is working as well as feel the pain before we share it with our customers. The most notable changes this month are:
- Each of the different business units within VSTS have setup their own SharePoint MOSS Dashboards with bug trends, etc.
- We’re now using Team Build 2010 with Gated Checkin on our part of the tree (we still have to follow the divisional processes of custom nightly build scripts at this stage)
- We enabled Test Case Management features and are now dogfooding the Test Case Management (TCM) features for our major test passes
- We setup and enabled the Lab Management functionality on our server and we’re starting to dogfood that across the organization as well
Here’s the latest stats:
Version Control
- Version Control Users: 622
- Files: 11,053,314
- Compressed File Sizes: 284,480
- Uncompressed File Sizes: 870,228
- Checkins: 29,275
- Shelvesets: 7,249
- Merge History: 10,897,164
- Pending Changes: 13,344
- Workspaces: 1,607
- Local Copies: 382,464,960
Work Item Tracking
- Users with Assigned Work Items: 2,607
- Total Work Items: 284,813
- Areas and Iterations: 2,910
- Work Item Versions: 2,892,664
- Work Item Attachments: 115,501
- Work Item Queries: 5,728
Commands (last 2 weeks in October)
- Work Item opens: 154,788
- Work Item queries: 120,537
- Work Item updates: 40,242
- Checkins: 2,630
- Uploads: 10,440
- Shelves: 4,553
- Downloads: 251,032
- Gets: 41,615
DevDiv TFS Server
On top of enabling all these new scenarios on the Pioneer server, we also managed to get the main DevDiv TFS server upgraded from SQL 2005 to SQL 2008. This was the first step to getting the server to TFS2010. I can’t say it was all smooth sailing, but it looks like we’re through the worst of it now (man, what a fortnight…). The good news is that all of the extra performance optimizations we had to make to cope with our dataset sizes also made it into TFS2010 product.
Although no install guide will tell you, I highly recommend you put on SQL 2008 SP1 CU4 if you’re running a large, busy TFS2008 or TFS2010 server – in the processes of testing the upgrade and after the actual upgrade to SQL 2008, we hit these bugs which the SQL team quickly reacted and fixed for us:
- KB967178 - When you restore a SQL Server 2005 backup file in SQL Server 2008, the operation takes much longer than when you restore the same backup file in SQL Server 2005 (fixed in SQL08 SP1 CU1)
- KB970823 - CPU cost increases suspiciously when the number of threads working on the same index increases (fixed in SQL08 SP1 CU3)
- KB970133 - Backup Log hangs when using compressed backups and the destination drive runs out of disk space. SQL Restart is required (fixed in SQL2008 SP1 CU4)
Now, onto the next Pioneer upgrade.
- TFS2010: Diagnosing Email and SOAP subscription failures
-
Sometimes emails alerts such as "Work Item Changed" will fail to send. Sometimes SOAP (web service) subscriptions will not be called correctly.
To diagnose these, you can look in the tbl_JobHistory table in the Tfs_Configuration database.
If you run this SQL command, it will retrieve the last 10 failed notification jobs:
SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0
ORDER BY StartTime DESC
The JobId GUID is the well-known GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension”
The DATEADD() function converts the date from UTC to local Pacific time.
ResultMessage:
There were errors or warnings during notification delivery. 0/4 emails delivered. 0/6 soap notifications delivered. 1 errors. 0 warnings. Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings.
To get detailed information about why the job failed, you will need to set a flag in the TFS registry. This is different from the windows registry and probably warrants a blog post on it’s own.
Changing NotificationJobLogLevel setting in the TFS registry using PowerShell
If you have VS2010 & PowerShell installed on your local machine and you have the appropriate permissions on the server, then you can run these steps from there. Otherwise, you will need to logon to a machine that does, or an AT. If you logon to the AT, you will need to run PowerShell as an Administrator.
To change the registry setting, replace the server url & collection name and follow these steps:
- Open PowerShell
- Run the following commands:
# Load client OM assembly.
[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a");
$collectionBaseUrl = "http://YOURTFSSERVER:8080/tfs/YOURCOLLECTION/";
$tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($collectionBaseUrl);
$collectionHive = $tfs.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry]);
# Set the setting in the collection hive.
$collectionHive.SetValue("/Service/Integration/Settings/NotificationJobLogLevel", "2");
It’s a good idea to set this back to “0” once you’re done with it, to avoid filling up the job history logs unnecessarily.
Once you have set this setting, the next time the job runs and fails, it will log more detailed information including the exception and stack trace. For example:
There were errors or warnings during notification delivery. 0/0 emails delivered. 0/3 soap notifications delivered. 3 errors. 0 warnings. ------------------------------- Notification not delivered. Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://eventsserver/EventHandler.svc) Exception: Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: Team Foundation services are not available from server http://eventsserver/EventHandler.svc. Technical information (for administrator): HTTP code 415: Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'. ---> System.Net.WebException: The remote server returned an error: (415) Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'.. at System.Net.HttpWebRequest.GetResponse() at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.AsyncWebRequest.ExecRequest(Object obj) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader) at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response) at Microsoft.TeamFoundation.JobService.Extensions.Core.TeamFoundationNotificationClient.Notify(String eventXml, String tfsIdentityXml, Subscription subscription) at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification, TeamFoundationIdentityService identityService)
This particular error message indicates that the client was expecting a SOAP 1.1 message, and the server sent it a SOAP 1.2 message.
If your email alerts are failing to send, then you’ll get an SMTP error back that shows you where to start looking.
- TFS2010: SQL Queries for TFS Statistics
-
Brian has been posting TFS statistics from the Developer Division dogfood TFS server for many years. I’ve updated the queries from TFS2008 to work with Team Foundation Server 2010 Beta 2. Here are the statistics for the TFS2010 Beta 2 server that we have been using internally since August 2009.
The queries
With the default permissions in SQL, mere mortals will not have enough access to run these queries. In fact, only TFS administrators should be able to even connect to SQL at all. To run these queries, you essentially need db_datareader rights on the Tfs_Configration and any Tfs_Collection* databases that you want statistics on.
In general, you do NOT want to query the TFS operational databases. The only supported API is the Microsoft.TeamFoundation.* Object Model. If you build applications that read the SQL store directly, you are likely to impact TFS performance and it is likely to break in the next version if the database schema changes. You’ve been warned! (Running these queries are OK periodically though :)
-- Change this to the name of your collection DB. You’ll need to run these queries for each of your collection DBs.
USE Tfs_DefaultCollection
GO
-- Recent Users
select count(distinct IdentityName) as [Recent Users] from tbl_Command with (nolock)
-- Users with Assigned Work Items
select count(distinct [System.AssignedTo]) AS [Users with Assigned Work Items] from WorkItemsAreUsed with (nolock)
-- Version Control Users
select COUNT(*) AS [Version Control Users] from [Tfs_Configuration].[dbo].tbl_security_identity_cache as ic JOIN tbl_Identity as i ON i.TeamFoundationId=ic.tf_id where ic.is_group = 0
-- Total Work Items
select count(*) AS [Total Work Items] from WorkItemsAreUsed with (nolock)
-- Areas and Iterations
select count(*) AS [Areas and Iterations] from tbl_nodes with (nolock)
-- Work Item Versions
select count(*) AS [Work Item Versions] from (select [System.Id] from WorkItemsAreUsed with (nolock) union all select [System.Id] from WorkItemsWereUsed with (nolock)) x
-- Work Item Attachments
select count(*) AS [Work Item Attachments] from WorkItemFiles with (nolock) where FldID = 50
-- Work Item Queries
select count(*) AS [Work Item Queries] from QueryItems with (nolock)
-- Files
select count(*) as [Files] from tbl_VersionedItem vi with (nolock) join tbl_Version v with (nolock) on v.ItemId = vi.ItemId where VersionTo = 2147483647
-- Compressed File Sizes
select (sum(convert(bigint,OffsetTo - OffsetFrom + 1)) / (1024 * 1024)) AS [Compressed File Sizes] from tbl_Content with (nolock)
-- Uncompressed File Sizes
select (sum(FileLength) / (1024 * 1024)) AS [Uncompressed File Sizes] from tbl_File with (nolock)
-- Checkins
select max(ChangeSetId) AS [Checkins] from tbl_ChangeSet with (nolock)
-- Shelvesets
select COUNT(*) AS [Shelvesets] from tbl_Workspace with (nolock) where type='1'
-- Merge History
select SUM(st.row_count) AS [Merge History] from sys.dm_db_partition_stats st WHERE object_name(object_id) = 'tbl_MergeHistory' AND (index_id < 2)
-- Pending Changes
select count(*) AS [Pending Changes] from tbl_PendingChange pc with (nolock) join tbl_Workspace w with (nolock) on pc.WorkspaceId = w.WorkspaceId where w.Type = 0
-- Workspaces
select COUNT(*) AS [Workspaces] from tbl_Workspace with (nolock) where type='0'
-- Local Copies
select SUM(st.row_count) AS [Local Copies] from sys.dm_db_partition_stats st WHERE object_name(object_id) = 'tbl_LocalVersion' AND (index_id < 2)
-- Command Counts
select Command, count(*) as [Execution Count] from tbl_Command with (nolock) WHERE Command IN ('QueryWorkitems', 'Update', 'GetWorkItem', 'Get', 'VCDownloadHandler', 'Checkin', 'Upload', 'Shelve') GROUP BY Command, Application ORDER BY [Application],[Command]
These are the core statistics to get you started. I need to add in some more on Team Build, Web Access, Excel & Project usage, etc. Let me know what other statistics you think would be useful.
If you’d like to share your statistics with me and/or the TFS team, send me a link or email them privately. We’d love to know what sort of usage patterns people are using the product in.
I’m also working on a plugin for the forthcoming TFS Best Practices Analyzer that will allow you to run these queries on a schedule and publish your own TFS statistics. By running it on a schedule, you will be able to track growth in size and usage over time and help you do some capacity planning. Stay tuned for that.
- VSTS Pioneer TFS2010 Dogfood Server: Upgrading
-
(See this summary on the VSTS Pioneer dogfood server and all the other posts.)
One of the major goals of the Pioneer Team Foundation Server is to upgrade it to early builds so that we can get some “bake time” with them and feel confident with what we’re releasing to customers. Since we spun up the server, we’ve done two successful upgrades and we’re already planning the next one:
- July 1st – Initial installation of Beta 1 build
- August 29 – Upgrade to 8/22 build from the VSTS branch
- September 17 – Upgrade to 9/17 build from the Beta 2 branch (yes, the build dropped at 5PM and we used it to upgrade at 6PM)
- November – Upgrade to a post-Beta2 build
The process of testing the upgrades on a copy of our database is just as important as running the upgrade itself. With our interesting dogfood topology of a dedicated SQL server, NLB/multiple application tiers, a dedicated MOSS server, virtual machines and use of virtual DNS names, we find some interesting bugs that we’re able to fix before things are locked down for shipping.
There’s no whitepaper on “How to Dogfood” at Microsoft, so our process for deploying perpetual upgrades is something that just kind of happened. Here’s basically what we do:
Depending on where we are in the cycle and the amount of churn in the code base, we might add another iteration or skip one. What’s most interesting about this process, is that it’s driven by the end date. Based on the product schedule, we know when our windows of opportunity are for upgrading and we work back from then.
Tips for upgrading
Here are some tips to consider when planning your own upgrades:
- Hardware – Get your production & pre-production servers in place early. Getting new hardware or re-configuring existing hardware usually has a long lead time. Make sure that you’ve also got enough storage space.
- Start fresh – If at all possible, give yourself the best chance at success by starting with a fresh box. Sure, plenty of products support in-place upgrades & clean uninstalls – but in my experience, flattening the box and starting fresh lays a good foundation for the future.
- Backups – Organize a copy of your production backups that you can test the upgrade with. Not only does it give you peace of mind that your backups work, it also helps you line up the DBA’s for when it’s time to do the real upgrade.
- Start with the latest OS – If you start with Windows 2008 R2 now, it’s one less OS upgrade (and downtime) you’ll have to do in the future. It also means you’ll be on 64-bit, which is a very good thing.
- Virtualization – We run our 3x Application Tiers as virtual machines with no problems whatsoever. This is great for when we’re installing pre-beta copies of the .NET framework that don’t have clean uninstalls yet. In our last upgrade we just turned off the old application tiers and spun up three fresh ones. You can treat AT’s as throwaways in a TFS2010 NLB topology, since there is no state stored on them.
- Dry-Runs – Run through your plan and make sure everything is going to work. Review it with others and get their feedback.
- Have a test plan – Once you’ve done the upgrade, you need to be sure that everything is back online and working as expected. Checkin, Checkout, Open work item, Save work item, Web Access, Sharepoint, Reports, Cube, Alerts, Monitoring, Builds, etc.
- Do pre-requisites ahead of time – For example, to install SQL2008 you need to have .NET3.5 and Windows Installer 4.5 both installed first and both of these require a reboot. If you plan to do one on the first weekend and the other on the second weekend, it means that you don’t waste time during your more important SQL upgrade waiting for reboots.
- Do one thing at a time – This is an extension of the last point. Let’s say you’re moving from a single-server TFS2008 SP1 on SQL2005 and Windows 2003 to a dual-server TFS2010 on SQL2008 SP1 and Windows 2008 R2. If it were me, I’d take it in steps: upgrade SQL, make sure everything still works, move to dual-server, check again, upgrade to TFS2010. The more steps you try and bundle together, the higher your risk of failure.
Upgrade logistics
I’ve been involved with at least 5 major upgrades this year and here are some things I’ve learnt from running them:
- Book a conference room
- This is the single most valuable thing you can do. Try and get anybody who is actually doing any steps in the upgrade (DBA< IT, Helpdesk, etc) to come to the room.
- Have a spare computer or project your own desktop on the screen. This is useful for synchronizing with others in the room, running ‘ping –t’ when a server reboots and doing “group debugging” sessions when problems happen.
- Setup a “party line” conference call. Invite anybody who wants to “listen in” to join. Invite your manager.
- It’s also useful for anybody who you might need help from to have these details. If things go south, you don’t waste time trying to setup a conference call – it’s already there.
- Setup a live meeting / desktop sharing session. Invite anybody who wants to be a “fly on the wall” to join.
- Sure, you can log a bug that says “when I clicked on X on screen Y, setup blew up” and people might believe you. But if you’re sharing your desktop and other people see it happen, then you have some people to back you up.
- (I’ve been meaning to do a screen recording of our upgrades, but various technology problems have prevented it each time – I think it would be useful for reviewing and improving the process for next time)
- Setup an “on call / escalation” list. Find out who you need to call if you have problems with one of your dependent systems. The last thing you want to be doing is sitting on hold to a L1 helpdesk if your server doesn’t reboot cleanly or something.
- Send at least 3 notifications to users of the downtime
- 1st Notice – Give as much notice as possible so people can plan milestones around the date. Even if it’s not set in stone yet, people would rather know about it and have it cancelled than to find out at the last minute.
- 2nd Notice – 7 days or on the Monday before a weekend upgrade is good for this.
- 3rd Notice – On the day of the upgrade (or 24 hours before if it’s a weekend), send another one.
- Guidelines: Make sure you describe what the impact to users is, any actions they need to take and where they can go to get future status updates and support.
- Provide regular status to stakeholders. Start an email thread and trickle through progress as it happens. By keeping them informed, you buy their trust and their support when things don’t go as planned.
Planning your own upgrade
Brian Keller has a great post on how to Get ready to “go live” with Team Foundation Server 2010 beta 2! He includes two useful documents:
- Overview presentation (.PPTX)
- Detailed checklist (.DOCX)
I’m also going to share with you a copy of the generic deployment plan template that I’ve been using for the Pioneer upgrades.
-
Deployment Plan Template(.XLSX)
Planning, testing and upgrading to these new releases has been a fun experience and it’s a bonus when everything goes smoothly. It’s great to see our experiences influence the product and make it the best release yet.
- VSTS Pioneer TFS2010 Dogfood Server: Statistics September 2009
-
(See this summary on the VSTS Pioneer dogfood server and all the other posts.)
September was a busy month for dogfooding TFS – all of the VSTS team were completely moved across and on 9/17 we upgraded to an almost final TFS2010 Beta 2 build. Here are the numbers from the end of September:
Version Control
- Files & Folders: 10,722,118
- Compressed File Sizes (MB): 263,350
- Uncompressed File Sizes (MB): 773,731
- Checkins: 23,369
- Pending Changes: 55,697
- Local Copies: 303,331,190
Work Item Tracking
It looks like I don’t have the work item tracking stats for September – I’ll have them next month.
Commands (last 2 weeks in September)
- Work Item opens: 113,128
- Work Item queries: 101,203
- Work Item updates: 23,653
- Checkins: 1,855
- Uploads: 6.622
- Shelves: 2,402
- Downloads: 216,452
- Gets: 30,201
Table sizes
One of the interesting metrics that I’m also tracking is how our tables and indexes are growing on the disks. Here’s the top 10 for this month:
| Table | Rows | Reserved (MB) | Data (MB) | Index (MB) | Unused (MB) |
| tbl_Content | 5,915,547 | 277,260 | 275,129 | 4 | 2,127 |
| tbl_LocalVersion | 303,331,190 | 81,562 | 44,789 | 36,574 | 199 |
| Attachments | 63,783 | 11,928 | 11,908 | 13 | 7 |
| tbl_Version | 12,129,564 | 3,418 | 1,031 | 2,369 | 17 |
| WorkItemLongTexts | 2,728,618 | 3,314 | 2,533 | 777 | 5 |
| tbl_VersionedItem | 15,754,176 | 2,983 | 1,598 | 1,382 | 3 |
| WorkItemsWere | 1,239,831 | 2,482 | 1,617 | 863 | 2 |
| tbl_PropertyValue | 33,293,914 | 1,320 | 523 | 793 | 4 |
| tbl_File | 5,717,981 | 722 | 442 | 266 | 14 |
- TFS2010: Installing pre-requisites from the command line
-
On the Team Foundation Server team, there is a new build of the server every night that comes out of our build lab. These vary in quality throughout the development cycle and the QA teams look at them and rate them as “Self Test”, “Self Test (with workarounds)”, or “Self Toast”. The QA & dev guys have lots of fancy automation for deploying nightly builds and testing them out, but I like to stay away from that stuff and use the builds as if I were a real customer. Once a build is declared “Self Test”, I’ll grab a copy and spin it up in a VM to see what’s changed.
The two preparation things I’ve done that have really sped up my installation turnaround times are:
- Create a sysprep’d Windows Server 2008 R2 VHD
- Create a SQL2008 SP1 slipstreamed installation point
Once I had those in place, I then just used these two long command lines below to install SQL & IIS. Of course, none of this is necessary for a “TFS Basic” install, since the installer will drop down IIS & SQL Express for you. See Brian’s blog on TFS 2010 for SourceSafe Users for more details on that.
Creating a sysprep’d Win2008 R2 VHD
Creating a sysprep’d VM is easy. Create a new VM and install a fresh copy of Windows 2008 R2. If you have a domain, join it to the domain and add yourself to the local administrators group (this will be retained after you sysprep & rejoin to the domain again). I don’t usually add the IIS role at this point, since I use these VHD’s for other things. Once you’re happy with it, run the following command:
C:\Windows\System32\Sysprep\Sysprep.exe /oobe /generalize /shutdown
Then take a copy of the VHD and name it something like ‘Win2008R2Sysprepped.vhd’. Whenever you need a fresh machine, just copy this VHD to a new file and away you go.
Creating a SQL2008 SP1 slipstreamed installation point
Follow the instructions for creating a slipstreamed drop on the SQL Server Setup team’s blog. Two things to note when doing this:
- Paths matter – it’s best to pick a folder name and stick with it whenever you use it. e.g. C:\SQL2008SP1\
- You DO have to download the ia64 & x86 SP1 packages & slipstream them as well, otherwise your setup.exe will fail
Once you’ve created the installation point, save it away somewhere – it’s really useful and it saves so much time by not having to do an RTM install followed by a SP1 install.
Installing TFS pre-requisites from the command line
- Boot into your fresh 2008 R2 VM
- Give it a new name, join it to the domain, apply any updates, reboot, etc
- Install SQL2008 SP1 from your installation point:
Setup.exe /q /ACTION=Install /FEATURES=SQL,FullText,AS,RS,Tools /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT="NT Authority\Network Service" /RSSVCACCOUNT="NT Authority\Network Service" /ASSVCACCOUNT="NT Authority\Network Service" /AGTSVCACCOUNT="NT Authority\Network Service" /SQLSVCStartuptype=2 /RSSVCStartupType=2 /ASSVCSTARTUPTYPE=2 /AGTSVCSTARTUPTYPE=2 /BROWSERSVCStartupType=2 /SQLSYSADMINACCOUNTS="Builtin\Administrators" /ASSYSADMINACCOUNTS="Builtin\Administrators" /SQMREPORTING=1 /ERRORREPORTING=1 /INDICATEPROGRESS
- Install required IIS components:
- Open a command prompt as administrator
- Type the following command:
ServerManagerCmd -install WAS WAS-Process-Model WAS-NET-Environment WAS-Config-APIs Web-Server Web-WebServer Web-Common-Http Web-Static-Content Web-Default-Doc Web-Dir-Browsing Web-Http-Errors Web-Http-Redirect Web-App-Dev Web-Asp-Net Web-Net-Ext Web-ISAPI-Ext Web-ISAPI-Filter Web-Health Web-Http-Logging Web-Request-Monitor Web-Security Web-Windows-Auth Web-Filtering Web-Performance Web-Stat-Compression Web-Mgmt-Tools Web-Mgmt-Console Web-Mgmt-Compat Web-Metabase Web-WMI Web-Lgcy-Scripting Web-Lgcy-Mgmt-Console
At this point, you’re now ready to run setup.exe from the TFS media. You have a fully configured SQL server and the requires IIS pre-requisites installed.
Other useful SQL 2008 setup.exe command lines
For a TFS “Application Tier Only” install, IIS is one pre-requisite and the “SQL Client Tools Connectivity” is the other one. This is required so that the TFS Job Agent can talk to Analysis Services and tell it to process the warehouse. Here’s how to install just the client tools:
setup.exe /ACTION=Install /FEATURES=Conn /ERRORREPORTING=1 /INDICATEPROGRESS /SQMREPORTING /Q
If you just want to install SQL Server Management Studio on your developer machine:
setup.exe /ACTION=Install /FEATURES=ADV_SSMS /ERRORREPORTING=1 /INDICATEPROGRESS /SQMREPORTING /Q
For more information, see How to: Install SQL Server 2008 from the Command Prompt and available features.
- Scripting Team Project Creation in TFS2010
-
In TFS2008 SP1 and the subsequent power tools release it was possible to create a team project from the command line. You can read Will-Peter’s detailed blog post on Creating a Team Project from the command line in TFS2008.
In TFS2010, it’s still not as good as we would like (it’s still client-based rather than server-based). But there’s one important difference – the feature has been rolled into the product and is no longer a power tool.
How to create a team project from the command line in TFS2010:
Create an XML file with the following contents and save it somewhere (e.g. C:\temp\pcw.xml). Substitute the highlighted as appropriate:
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="ProjectCreationSettingsFileSchema.xsd">
<TFSName>http://tfsserver:8080/tfs/DefaultCollection</TFSName>
<LogFolder>c:\temp</LogFolder>
<ProjectName>TestAgile2</ProjectName>
<ProjectSiteEnabled>true</ProjectSiteEnabled>
<ProjectSiteTitle>TestAgile2</ProjectSiteTitle>
<ProjectSiteDescription>TestAgile2 project created from command line</ProjectSiteDescription>
<SccCreateType>New</SccCreateType>
<SccBranchFromPath></SccBranchFromPath>
<ProcessTemplateName>MSF for Agile Software Development v5.0</ProcessTemplateName>
</Project>
Once you have created the file, you need to prepare Visual Studio to be automated and trigger the automation from the command line.
- Open Visual Studio 2010
- Go to View > Team Explorer and make sure that the Team Explorer tab is active
- Close Visual Studio 2010
- Open a command prompt
- Type: CD /D %ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE
- Type: devenv /command "File.BatchNewTeamProject C:\temp\pcw.xml"
At this point, Visual Studio will open and the Team Explorer tab will show “Connecting to server\collection…”. In the background it’s actually creating the team project. You can verify this by looking for a txt file with the name of your project in the path you specified for LogFolder. (e.g. c:\temp\TestAgile2.txt)
Once the project has been created, you can select ‘Add existing team project’ and select it from the dialog.
If you want to create a fully-automated project provisioning system there are a few other things that you’ll need to do on top of creating the team project. Some of these things are:
- Add users to TFS groups (Project Administrators, Contributors, Readers)
- Add permissions in sharepoint
- Add permissions in reporting services
- TFS2010: How to make Beta1 SharePoint sites work after upgrading to Beta2
-
Just like my last blog post on uploading the beta2 process templates to an upgraded beta1 server, I’ll admit that the audience for this blog post is also small. However, we hit this after upgrading our VSTS Pioneer Dogfood TFS server from Beta1 to an early Beta2 build last weekend.
I’m fortunate to share an office with another Australian (Hi Phil!) who happens to be on the ‘Process’ team that builds all the TFS SharePoint templates & dashboards, so we were able to diagnose & fix this issue quickly this morning.
Note: These steps will only allow you to get access to the content that existed when it was a Beta1 server – most of the dashboards will be broken because of warehouse schema changes between TFS Beta1 & Beta2.
Symptoms
After you upgrade a TFS2010 Beta1 server to a Beta2 build, you may not be able to access your old SharePoint sites. You’ll receive a SharePoint error page that says “File Not Found”. This can also happen if you take the SharePoint content database from a Beta1 server and attach it to a new Beta2 server. When debugging SharePoint issues, sometimes browsing to /<site>/_layouts/viewlsts.aspx will give you an idea if the site actually exists or if it’s completely broken. In this case, browsing to viewlsts.aspx shows all the document libraries but they all show ‘File Not Found’ when you browse through to them.
Cause
The reason this happens is because between TFS2010 Beta1 and Beta2, the dashboards team were working on SharePoint 2010 compatibility. One of the things they identified is that the existing approach of TFS having it’s own master page didn’t work too well (or look very pretty) when somebody used SharePoint 2010. So they made changes to the TFS templates to use the default SharePoint master page & style everything with a SharePoint theme (i.e. using CSS).
The unfortunate side-effect is that on a clean SharePoint server that only has the Beta2 version of the TFS SharePoint extensions installed, the master page from Beta1 doesn’t exist on disk – therefore SharePoint can’t display any pages that refer to that master page (i.e. most of them).
Resolution
I’m sure there are many other ways to solve this (write an app, write a PowerShell script, etc) but this is the one that we used.
The first thing you need to do is download the “SharePoint Manager 2007” application to edit the properties on each of the sites. http://www.codeplex.com/spm. This must be run on the SharePoint server and it must be run as an Administrator (SharePoint 2007 doesn’t have a separate installable object model and even if it did, connecting to a remote farm isn’t possible).
When you open SharePoint Manager, you’ll need to browse down into Content Service > Web Applications > Default Web Site > Site Collection > Your Beta1 Site Collection URL > Your Beta1 Site Collection Name > Webs > Select a web (see screenshot)
The MasterUrl is the property that you need to change to remove the ‘File Not Found’ error and get access to your site.
Beta1
/sites/<collection>/<site>/_catalogs/masterpage/dashboards.master
Beta2
/sites/<collection>/<site>/_catalogs/masterpage/default.master
There are two other things you need to do if you want your Beta1 sites to look like Beta2 sites.
- Clear the AlternateCssUrl property. (In Beta1 it was set to /_layouts/1033/STYLES/Microsoft.VSTS.Process.Dashboards/css/dashboards.css)
- Change the theme of the site to the Team Foundation Dashboards (TFSDASHBOARDS) theme. This theme is stored in: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\THEMES\TFSDASHBOARDS
Once again, there are multiple ways to add a new theme & apply it to the site, but this one worked for us. Before you can select the TFSDASHBOARDS theme, you need to add it to the list of available themes.
On a web frontend server, open the following file and add a new <Template> definition: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\LAYOUTS\1033\SPTHEMES.XML
<Templates>
<TemplateID>TFSDASHBOARDS</TemplateID>
<DisplayName>Team Foundation Dashboards</DisplayName>
<Description>A blue-grey theme for Team Foundation Server Dashboards</Description>
<Thumbnail>images/thwheat.gif</Thumbnail>
<Preview>images/thwheat.gif</Preview>
</Templates>
Then restart IIS. (Think these steps are crazy? That’s the official way from the SharePoint docs on MSDN, How to: Customize Themes).
Then you should be able to go to Site Action > Site Settings > Look and Feel > Site theme (/_layouts/themeweb.aspx) and select the ‘Team Foundation Dashboards’ theme.
Hope this helps
- TFS2010: Replacing Beta1 Process Templates with Beta2 versions
-
Update: It turns out that in the final release of Beta2 (instead of the pre-Beta2 build we’re using), we will rename the existing Beta1 templates to “Process Template X (Beta 1)” and upload new versions of the templates – so this won’t be as painful.
I’ll admit that the audience for this blog post is small, since TFS2010 Beta1 did not have a go-live license and there will be very few people who upgrade a Beta1 server to Beta2. However, we hit this after upgrading our VSTS Pioneer Dogfood TFS server from Beta1 to an early Beta2 build last weekend.
Problem #1 – Cannot create new team projects
After you upgrade a TFS Beta1 server to TFS Beta2, you may be unable to add new Team Projects to an existing Team Project Collection (TPC). When you run the Project Creation Wizard, it may fail with the following error message in the log:
2009-08-31T14:03:39 | Module: GSSStructureUploader | Thread: 17 | TF30181: Error occurred while adding access entry : action name: "ADMINISTER_TEST_ENVIRONMENTS" sid: "S-1-9-1551374245-678206409-4164841793-2695741645-3025924470-1-334192246-596365894-2376495270-2693603616" deny: "True"
Exception Message: TF50609: Unable to retrieve information for action ADMINISTER_TEST_ENVIRONMENTS, it does not exist.
The reason this happens, is because the old Beta1 Process Templates are still in the TPC – the upgrade does not replace them. (What would you think if we dropped any customizations you made to the process templates and uploaded new ones over the top?).
So, the solution is to delete these process templates and upload new ones.
Problem #2 – Cannot upload new process templates
When you try and upload the new process templates to the collection, you may receive another error. If you look in the log, it might look something like this:
2009-08-31T12:29:39 | Module: Work Item Tracking | Thread: 45 | Validating work item type from file 'C:\TEMP\ProcessTemplates\MSF for Agile Software Development v5.0\WorkItem Tracking\TypeDefinitions\TestCase.xml'...
Exception Type: Microsoft.TeamFoundation.WorkItemTracking.Client.Provision.ProvisionValidationException
Exception Message: TF237091: Actual reporting settings for the field Microsoft.VSTS.TCM.AutomationStatus are different from those specified in the XML. Changing these settings is prohibited.
The reason this happens, is because in the Beta1 process template, the ‘Reporting Type’ property is set to ‘detail’, whereas in the Beta2 process template it has changed to ‘dimension’.
Beta1:
C:\>witadmin listfields /collection:http://server:8080/tfs/collection /n:Microsoft.VSTS.TCM.AutomationStatus
Field: Microsoft.VSTS.TCM.AutomationStatus
Name: Automation status
Type: String
Use: 2Do (Test Case), CodeReviewer (Test Case), Intellisets (Test Case), TestAgile (Test Case), TestAgile1 (Test Case)
Indexed: False
Reportable As: detail
Beta2:
<FIELD reportable="dimension" type="String" name="Automation status" refname="Microsoft.VSTS.TCM.AutomationStatus">
Changing the reporting type of a field is not something that the process template upload service will do for you. Fortunately the solution is fairly simple, you just have to change the reporting type of the field before uploading the new process template.
Steps to replace your process templates with Beta2 versions
- Logon to the TFS Admin Console on the Application Tier
- Create a new Team Project Collection
- Using a computer with Team Explorer installed:
- Connect to the new collection
- Right-click on the server name, choose ‘Team Project Settings > Process Tempaltes’
- Download each of the new Process Templates to a local folder
- Open a Visual Studio command prompt and run the following command: witadmin changefield /collection:http://server:8080/tfs/collection /p:Project /n:Microsoft.VSTS.TCM.AutomationStatus /reportingtype:dimension
- Connect to the collection that was upgraded from Beta1
- Right-click on the server name, choose ‘Team Project Settings > Process Tempaltes’
- Delete existing Beta1 Process templates
- Upload each of the Beta2 process templates that you downloaded earlier
- Create a new team project with the new version of the process template
Hope this helps. Stay tuned for more upgrade stories.
- VSTS Pioneer TFS2010 Dogfood Server: Statistics August 2009
-
(This blog post is part of a series of posts on the new VSTS Pioneer TFS dogfood server.)
Everybody loves statistics, so here are the statistics for the first month on the VSTS Pioneer server.
Work Item Tracking
- Users with Assigned Work Items: 2,454
- Work Items: 111,319
- Areas & Iterations: 2,811
- Work Item Versions: 1,056,416
- Work Item Attachments: 50,833
- Work Item Queries: 2,965
Version Control
- Files & Folders: 8,561,565
- Compressed File Sizes (MB): 325,080
- Uncompressed File Sizes (MB): 856,859
- Checkins: 18,867
- Shelvesets: 1,982
- Merge History: 1,789,333
- Pending Changes: 48,155
- Workspaces: 727
- Local Copies: 184,231,208
These statistics are captured by using the TFS Best Practices Analyzer and a custom configuration file. The configuration tells the BPA tool to run a series of SQL queries and dump the results to an XML file. I have then created an XSLT that transforms the results into a more user-friendly HTML file. The BPA tool is then scheduled to run every night and generate a fresh set of statistics. I’m working with the TFS BPA team to get this configuration built into the next version of the BPA tool – I don’t have an ETA on when that will ship, but stay tuned.
This month I’ve also pulled the last 30 days of PerfMon counters to see how our hardware choices are holding up. If you’d like to setup something like this for yourself, have a look at my previous posts.
Disk Performance
Things are looking OK on the IO front. 95% of the time we’re pushing 45MB/sec (Read) and 33MB/sec (Write) or less to and from our storage. The response time is <40ms which is nothing to worry about for the configuration we have.
|
LogicalDisk(_Total) |
50th Percentile |
95th Percentile |
99th Percentile |
|
Avg. Disk sec/Read |
0.004 |
0.039 |
0.371 |
|
Avg. Disk sec/Write |
0.003 |
0.023 |
0.063 |
|
Disk Read Bytes/sec |
2,142,548 |
44,926,021 |
94,273,782 |
|
Disk Write Bytes/sec |
1,908,518 |
33,029,252 |
70,802,346 |
|
Avg. Disk Bytes/Transfer |
42,867 |
133,887 |
436,920 |
Memory
The memory usage on our SQL server has remained completely constant. This is because SQL is set to use a fixed amount and Analysis Services & Reporting Services share the rest.
CPU Performance
(Avg: 32%). The majority of this usage is SQL Server, with a small portion going to Analysis Services & Reporting Services. It’s nice to see that we still have plenty of capacity left for the usage to grow.
That’s all for statistics, stay tuned for more next month.
- VSTS Pioneer TFS2010 Dogfood Server – Hardware & Topology
-
(This blog post is part of a series of posts on the new VSTS Pioneer TFS dogfood server.)
While running the main TFS server for ~3,500 users in Developer Division we got a pretty good idea of the hardware required to support our enormous number of files, local versions & branches.
We knew that:
- our SQL Server load patterns were very IO intensive
- our SQL Server load patterns were very CPU intensive
- our disk usage will grow by about 300% per year
- the number of operations the server performs will increase
Servers
We wanted to purchase and setup hardware that was flexible enough and designed to scale as our needs do. For this reason we settled on two identical servers that are part of the Microsoft IT recommended systems for our needs:
According to Joe Chang who studies SQL Server performance, the “The X7460 (Dunnington) is a clear winner at 4-way for the high-call volume apps.” The 6-core CPUs also offer a pretty significant performance boost over the older quad-core CPUs:
- “On TPC-C, the 7460 six core generated a 34% edge over the quad core AMD and 56% advantage over the quad core X3750”, and
- “On TPC-E, the six core showed a 49% edge over the older quad core”
Virtual Machines
We are running 5 virtual machines at the moment with almost identical configurations. Since we’ll be constantly upgrading things like the .NET CLR that often don’t support build-to-build upgrades or clean uninstall, running the application tiers as VM’s makes perfect sense. It also improves our rollback story if we have a failed upgrade. (More about upgrade in a future post).
|
VM |
Purpose |
Configuration |
|
AT01, AT02, AT03 |
TFS Application Tier |
Windows Server 2008 R2
4 Virtual processors
4GB RAM
2 Virtual NICs
1 VHD for C$
1 VHD for version control cache |
|
MIRROR |
TFS-TFS Connector tool runs here |
Windows Server 2008 RTM
4 Virtual processors
4GB RAM
1 Virtual NIC
1 VHD for C$
1 VHD for version control temporary workspace |
|
MOSS |
Microsoft Office SharePoint 2007 SP2 (Complete install)
Web Front End, Excel Services, Search Services & Shared Services Provider |
Windows Server 2008 R2
4 Virtual processors
4GB RAM
1 Virtual NIC
1 VHD for C$
Databases reside on physical SQL server |
Storage
Since these servers are hosted by the DevDiv IT guys, we went with their recommendation for storage. They already have a massive storage infrastructure for build & drop servers built on Xiotech technology, so it was easy for them to add additional ISE units to meet our size & performance requirements. In the end we ended up with the following configuration:
- 5TB of “Balanced” storage
- 5TB of “Performance” storage
The SQL Customer Advisory Team’s – Storage Top 10 Best Practices is a great resource when planning your own TFS deployment. In particular we specifically considered the following:
- Isolate log from data at the physical disk level (e.g. O$ drive is on separate spindles)
- Consider configuration of TEMPDB database (e.g. T$ drive is on separate spindles)
- Make sure to give thought to the growth strategy up front (e.g. We can add additional ISE units at any time and grow any drive without having to migrate data)
- Lining up the number of data files with CPU’s has scalability advantages for allocation intensive workloads. (e.g. we have 12 datafiles for TempDB because we have 12 CPU cores)
Here’s how the virtual disks (LUNs) are configured on the server:
SERVER1 (SQL, SSRS, SSAS)
|
Drive |
Purpose |
Storage Type |
Size (Gb) |
|
E$ |
SQL backup dump drive |
Balanced |
2741 |
|
G$ |
LocalVersion filegroup |
Performance |
1673 |
|
I$ |
OLAP data |
Balanced |
25 |
|
J$ |
All database data |
Performance |
1833 |
|
K$ |
Version filegroup |
Performance |
971 |
|
O$ |
Transaction Logs |
Balanced |
1500 |
|
T$ |
TempDB |
Performance |
500 |
SERVER2 (Hyper-V Host)
|
Drive |
Purpose |
Type |
Size (Gb) |
|
E$ |
Hyper-V VM's |
Balanced |
743 |
What’s cool about the Xiotech stuff is that it’s like Lego blocks – as you need more storage or more performance, you just buy more blocks and plug them in. What’s even cooler about it, is that you can control the SAN using web services – the build lab uses the SANMan tool they built to move virtual disks between build servers and automate the provisioning of disk space.
Topology
At just over 400 users, we probably don’t need three AT machines in a NLB configuration. However, NLB is an important new feature of TFS2010 and it’s a scenario that we need to dogfood for ourselves.
Here’s some quick facts about our topology:
- We have a couple of “friendly name” DNS records for the NLB cluster IP, Reporting Services & Analysis Services – this allows us to change the underlying infrastructure without users having to connect to a new addresses.
- We’re running Windows 2008 R2 RTM on our VM’s & soon on our physical servers
- A single VM was created, then sysprep’d and copied. This means that we can spin up a new VM in a very short time period. (Copy VHD, Add to Hyper-V, Start, Join Domain, Apply Updates, Done.)
- SERVER2 is running an SMTP server that relays mail on behalf of each of the other servers. At Microsoft, the corporate mail servers will only accept mail from authenticated users that have mailboxes. By running the SMTP server as an authenticated user, it allows us to run the application tiers as Network Service. This means when we have to change a password to a service account, it doesn’t cause a TFS service interruption – just a brief SMTP interruption.
- Since all the SQL services are running on the same box, they don’t need to be run as a domain accounts either – no service interruption & less places that need a password changed when it expires.
That’s it for hardware and topology, next post is on performance and Dogfood statistics.
- VSTS Pioneer TFS2010 Dogfood Server
-
August 2009 marks an important milestone for TFS Dogfooding at Microsoft.
In 2004 there was a dogfood server for parts of the TFS team, then it expanded to all of Team System (VSTS). Then other teams in Developer Division (DevDiv) saw the value of having your source code and bugs in the same repository. The division committed to using TFS and eventually on-boarded all teams to the new server. It was constantly patched and had a few major upgrades and eventually became the TFS server that the whole of DevDiv uses today (~3,500 users).
But now that it is such an important server for the division, we can’t risk potential downtime during the development cycle caused by upgrades to new versions. As Brian has said in the past – “preventing almost 2,500 people from getting their work done is not a recipe for a long and healthy career”. Although it’s great that we’ve patched the server and got it to a point where we have >99% application availability, the TFS team has lost the ability to dogfood the latest releases of TFS and feel the pain (and fix it) before we inflict it upon our customers.
Introducing the VSTS Pioneer server:
This is a server that >400 people on the VSTS team are now using instead of the main DevDiv server. All work items and a subset of the source code from the main server is mirrored to this server using a mostly rewritten version of the TFS-TFS Connector built on the TFS Integration Platform (more about that later – see Willy-Peter’s blog for a preview). The server has been running TFS2010 Beta1 since mid-July with no major issues and will be upgraded to a pre-Beta2 build this weekend. It will continue to be upgraded on a regular basis (we’re thinking every 1-2 months) and will always be the first production server to run the latest TFS builds.
Coming up with the name for the server was interesting – the obvious choice is “VSTS Dogfood Server”, however that would mean we’d have two servers that people would call “the dogfood server”. This is a little ambiguous and awfully confusing. So I borrowed the term from the Exchange team who run their own mail server called “Pioneer” and this seems to have worked well.
Everybody loves to hear about how Microsoft uses their own products, so I’m going to try and answer all the obvious questions with a series of blog posts over the next few days:
Let me know any questions you have or what you’d like to hear more about and I’ll see what I can do.
- How do you update all Work Item Type Definitions on a server?
-
Today there was a question on one of the lists I’m on, it went something like this:
Has anyone written a tool that will batch-update WITD to all team projects on a server? Surely this must be a common scenario?
One of the built-in Windows commands that I use daily is the “FOR” command. Here’s how simple it is to upload a new Bug.xml for a list of projects.
TFS2008
for %a in ("Project 1", "Project 2", "Project 3") do witimport /f Bug.xml /t http://tfsserver:8080 /p %a
TFS2010
for %a in ("Project 1", "Project 2", "Project 3") do witadmin importwid /s http://tfsserver:8080/tfs/CollectionName /p %a /f Bug.xml
If you wanted to get fancy, you could even create a teamprojects.txt file with a team project name per line, and use the /F parameter to read them from a file. There are a few subtleties with that though, make sure you quote any parameters that need quoting:
for /F %a in (teamprojects.txt) do witimport /f Bug.xml /t http://tfsserver:8080 /p “%a”
And the last example for today is if you want to run the same command on every file in a directory. I use this all the time with relog.exe, when I want to concatenate a directory of perfmon logfiles (one for each hour) and change them to a different format. For example:
for /R %a in (*.blg) do relog %a –f CSV –o output.csv –a
Also remember, if you want to use the “FOR” command from within a batch file, you need to use %%a instead of just %a.
Update:
