All your TFServer are belong to us

Team Foundation Server Proxy 2008 Self-Help Troubleshooting Guide

2009-02-06T01:16:00Z

In addition to How TFS Proxy 2008 works and TFS Proxy 2008 FAQ, this post focuses on self-help troubleshooting TFS Proxy 2008 issues. The goal of this post is to guide the reader to troubleshoot simple but very common Proxy issues, or at least to give the reader a general idea what could be wrong. You may still need to seek for professional help if the issues with your TFS Proxy setup is beyond the scope of this self-help guide.

Make sure Proxy service account and its password are valid.

Make sure Proxy service account is not disabled.
Make sure Proxy service account's password is not expired.
Update Proxy service account password in Proxy AppPool with TfsAdminUtil.exe. You can find this in the Tools folder under TFS or TFS Proxy installation folder.
If your TFS setup is in a domain environment, you might want to consider running TFS Proxy 2008 with a network service account (It is a convenient trick but it is not officially supported by Microsoft.)

Make sure Proxy service account is allowed and not denied GENERIC_READ permission on TFServer. This is a bit complicated and will be explained in a paragraph below.
Make sure proxy.config contains the right TFServer address.

Make sure the Server entries in proxy.config use this format http://serverAddress:port/ instead of the old format used by TFS Proxy 2005.

Make sure TFServer and Proxy addesses can be correctly resolved and AppPool is running.

One easy way to test this is to try the following steps:

From a client machine, open http://proxyAddress:port/VersionControlProxy/v1.0/ProxyStatistics.asmx
From a client machine, open http://serverAddress:port/VersionControl/v1.0/ProxyStatistics.asmx
From Proxy server, open http://serverAddress:port/VersionControl/v1.0/ProxyStatistics.asmx

If everything is setting up correctly, for each trial you should see a web page with links such as "Service Description" and "QueryProxyStatistics".

The System Event Log (eventvwr) on the proxy and server machine is another place you can look for issues that block your TFS from working properly.

There is also a troubleshooting guide for TFS Proxy on MSDN: Troubleshooting Team Foundation Server Proxy http://msdn.microsoft.com/en-us/library/ms400681.aspx

The story behind "Make sure Proxy service account is allowed and not denied GENERIC_READ permission on TFServer."

First, three things to know:

TFS permissions can only be assigned to identities recognized by TFS.
A user/group inherits permission settings from its parent group.
Deny overrules Allow.

In order to make Proxy work, we need to make sure Proxy service account is recognized by TFServer and is allowed but not denied GENERIC_READ permission.

Preparation:

In a workgroup setup, you must have a local machine account on TFServer machine, which must have the same username and password as Proxy service account's.
Locate Tools\TfsSecurity.exe under your TFS installation folder.

The easy way:

Because GENERIC_READ is allowed on the "TF Valid Users" group, therefore we can create a server-level group and add Proxy service account as a member so that the server-level group inherits GENERIC_READ from "TF Valid Users" and Proxy service account inherits GENERIC_READ from the server-level group.

Create a server-level group. Let's call it "Proxy Service Accounts"

TfsSecurity /server:http://serverAddress:port/ /gcg "Proxy Service Accounts"

Add Proxy service account as a member

TfsSecurity /server:http://serverAddress:port/ /g+ "Proxy Service Accounts" domain\proxyServiceAccount

Done!

The hard way:

If the easy way did not work, then most likely GENERIC_READ is either explicitly and effectively denied for Proxy service account. To find out where GENERIC_READ is explicitly denied, do

TfsSecurity /server:http://serverAddress:port/ /acl $NAMESPACE

Keep in mind that a user/group inherits permission settings from its parent group. Check whether Proxy service account inherits "Deny GENERIC_READ" from its parent or grand-parent groups.

Update:

May 20, 2009: Corrected test links to server's proxy statistic page. Thanks to Len Ocin for pointing it out!

Team Foundation Server Proxy 2008 FAQ

2008-11-07T23:39:00Z

Since posting How TFS Proxy 2008 works, I have got a bunch of questions regarding TFS proxy. I would like to share them with you (with my answers, of course) :)

Q: How do I configure TFS Proxy?
A: http://msdn.microsoft.com/en-us/library/ms253156.aspx

Q: I got this error message: "source control proxy is not responding, so the request will be sent to the main server. Please verify your settings." -- what happens now?
A: As long as the main server is functioning and reachable, your command/operation will continue to execute and will deliver the same results. Version control proxy is a mean to speed things up by caching data locally on the proxy. If TFS Proxy cannot server client's download request (for any reason), client will automatically fall back and download from the main server.

Q: Can I install TFS AT and Proxy on the same machine?
A: Technically, yes, you can, but it is known that the proxy cache statistics may show incorrect numbers. Practically, there is no point in putting both AT and Proxy on the same machine.

Q: Can TFS Proxy cache work items?
A: No. For TFS 2005/2008, TFS Proxy caches version control data only.

Q: Can TFS Proxy act as a surrogate server or a fail-over backup server?
A: No. TFS Proxy merely caches version control data. Authentication and security checks are still handled by the server.

Q: Any recommendation for proxy hardware?
A: It is hard to say without knowing the specifics (the size of your organization and projects, number of people in your away teams, etc) However, the bottleneck will be disk space, disk I/O performance, and network bandwidth.

Update: Brian Harry's blog post on TFS hardware recommendations in general: http://blogs.msdn.com/bharry/archive/2007/10/18/tfs-2008-system-recommendations.aspx

Q: When do I need a TFS proxy?
A: TFS proxy is good for these scenarios

Cache version control data locally so it takes less time for your away team, who have a slow connection to the main server, to sync their workspace.
Reduce disk I/O load on the main server by serving download requests so your main server can concentrate on other things (processing branch merge requests, responding to work item queries, ... etc)

Q: Say I have two away teams A, B. They both have slow connections to the main server, but they have ok connection between them. Is it possible to have team A's proxy downloading files from the main server and have team B's proxy downloading files from team A's proxy?
A: No. TFS 2005/2008 Proxy can only download from the main server.

Q: Why is my TFS proxy just sitting there and not caching files?
A: TFS proxy does not prefetch files. It caches version control items after a client, which is configured to use that particular proxy, made a download request.

Q: How do I assess my TFS proxy utilization?
A: You can look at the cache hit rates ( http://msdn.microsoft.com/en-us/library/ms400683.aspx ). As a rule of thumb, the more people working in the same code branch, the bigger performance gain from TFS Proxy. You can also retrieve proxy statistics programmically with this class Microsoft.TeamFoundation.Client.ProxyStatistics.

Q: How do I find out who is/isn't using my TFS proxy?
A: You need to first enable TFS activity logging and query the logging for users who sends download requests directly to the main server -- I will try to find time to write another blog post to give a better detailed answer on this one.

Update: Grant Holliday has written a blog post on this subject: http://blogs.msdn.com/granth/archive/2008/09/16/how-many-users-are-not-using-a-tfs-proxy-server.aspx

Run Team Foundation Server Proxy 2008 with Network Service Account

2008-06-27T00:55:00Z

Although TFS Proxy is supposed to run as by a service user accont, however, there is a way to get around it if your setup is on a domain.

This is not officially supported by Microsoft, but it works.

Background

It is recommended that you read http://blogs.msdn.com/tsyang/archive/2008/03/24/how-team-foundation-server-proxy-2008-works.aspx before proceeding.

Instruction

Install proxy with a valid user account (for example, yours)
(Optional but recommended) Do not yet add your targeting TFServer address to proxy.config

This prevents users from using this proxy before you can change the service account. This will save you some headache (which I could write another short post to explain...)

Locate tfsadminutil.exe from where you install the proxy to
Run tfsadminutil changeaccount /proxy currentServiceAccount "NT Authority\Network Service"

currentServiceAccount refers to the account you entered at the beginning

Add your targeting TFServer address to proxy.config
iisreset the proxy server machine
Ask your TFServer admin to add this account "proxyMachineDomain\proxyMachineName$" to the TFS group where he/she manages proxy service accounts

"proxyMachineDomain\proxyMachineName$" is your proxy machine's machine account.

Done.

Update

Thanks to James Manning for pointing it out that this workaround only works in a domain environment.

How Team Foundation Server Proxy 2008 works

2008-03-25T05:21:00Z

Ever wondering how TFS Proxy works? While this MSDN article "Team Foundation Server Proxy and Source Control" ( http://msdn2.microsoft.com/en-us/library/ms252490.aspx ) provides a great overview, this blog post will add an end-to-end story for TFS users who likes to dig into technical details.

Table of Contents:

Workflow
Highlights
In other words

Workflow:

Let's see what happens among a user (CLIENT), TFServer (SERVER) and TFS Proxy (PROXY) when the user is trying to download a file from TFServer version control repository.

CLIENT authenticates with SERVER.

SERVER terminates connection if authentication failed. End of story.

CLIENT sends a file download request to SERVER.
SERVER checks CLIENT's read permission on the requested file.

SERVER reports "file does not exist" to CLIENT if CLIENT has no read permission. End of story.

SERVER sends a download ticket for the requested file to CLIENT.
CLIENT sends the download ticket to PROXY and wait for PROXY to return the requested file.

Update: If PROXY does not return the requested file within a certain amount of time for any reason, CLIENT will use the download ticket to download directly from SERVER. End of story.

PROXY checks whether the requested file is already cached.

PROXY returns the requested file to CLIENT if it is already cached. End of story.

PROXY service account authenticates with SERVER

SERVER terminates connection if authentication failed. PROXY reports error to CLIENT. CLIENT will download directly from SERVER. End of story.

PROXY asks SERVER for the location of VersionControl services.
SERVER checks whether PROXY service account has read permission on server-level information.

SERVER terminates connection if PROXY service account has no read permission on server information. PROXY reports error to CLIENT. CLIENT will download directly from SERVER. End of story.

SERVER tells PROXY where VersionControl services are.
PROXY uses CLIENT's download ticket to download the requested file from SERVER.
PROXY caches the requested file.
PROXY returns requested file to CLIENT. End of story.

Highlights:

SERVER always checks repository read permission against CLIENT, not PROXY service account.
SERVER always checks server-level information read permission against PROXY service account; and that is the only permission PROXY service account ever needs.
PROXY can save SERVER resources by serving CLIENT's downloading request when the requested file is already cached.

In other words:

PROXY and SERVER are binded at the server-level, not team project level.
PROXY does not act as a surrogate for SERVER; PROXY only does caching and all permission checking is done by SERVER.
PROXY service account can simply be placed in a server-level group, e.g. "[Server]\Proxy Service Accounts", without any extra security configuration. This effectively grants PROXY service account read permission on server-level information.

Adding PROXY service account to either TFServer Admin group, TFServer service account group, or any team project group will also grant PROXY service account read permission on server-level infromation; however, this practice is not recommended because it gives PROXY service account more permissions than it needs.