Recently I started seeing this error on our internal TFS SharePoint sites. These sites use the Excel Dashboards and if I opened more than a few sites at a time, I would start getting the following popup error:
Unexpected callback response! Error: 400 Bad Request Bad Request - Request Too Long HTTP Error 400. The size of the request headers is too long.
Unexpected callback response! Error: 400 Bad Request
Bad Request - Request Too Long HTTP Error 400. The size of the request headers is too long.
It didn’t seem to matter which browser I used or whether I ran in Private Mode (to get a clean session each time) – the result was always the same error.
The fix:
http://support.microsoft.com/kb/920862/en-us
Save this as a .reg file and run it.
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters] "MaxFieldLength"=dword:0000fffe "MaxRequestBytes"=dword:0007a120
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters] "MaxFieldLength"=dword:0000fffe "MaxRequestBytes"=dword:0007a120
After setting the settings, you will need to restart your server so that HTTP.sys can pickup the new parameters.
(Updated 7/16/2010)Since the Readify TFSNow! hosted TFS product that I helped to build is no longer in operation, people have asked if there is anybody else doing it.
There are a few partners that I know of that offer to run and maintain Team Foundation Server in the cloud for you:
I don’t have any experience with any of them, so please let me know what you think.
We've also started experimenting with what it would take to run TFS2010 in the Windows Azure environment. Brian talks more about the prototype on his blog.
This blog post describes how to interpret the Data Warehouse & Cube status reports included in the Administrative Report Pack for TFS2010.
If these tips and the help topic for the error message do not answer your question, see the Microsoft Technical Forums for Visual Studio Team Foundation (http://go.microsoft.com/fwlink/?LinkId=54490). You can search these forums to find information about a variety of troubleshooting topics. In addition, the forums are monitored to provide quick responses to your questions.
Yes. Some failures are part of typical processing. Three jobs require exclusive access to the same warehouse resources:
None of these jobs can run in parallel with any other job on the list. If one job is already in progress when another job starts, the second job will fail quickly with error TF276000, as shown in the following illustration of the Cube Processing Details view of the Processing Times report:
The following illustration shows a sample of typical processing failures:
The previous illustration shows the results of the following events:
The Cube processing job requires exclusive access to some of the warehouse resources that data synchronization jobs use. The Cube processing job will wait for the release of the resources (normally for an hour) before it gives up. If a data synchronization job does not release the resource in time, the Cube processing job will fail with the following error:
ERROR: TF221033: Job failed to acquire a lock using lock mode Exclusive, resource DataSync: [ServerInstance].[TfsWarehouse] and timeout 3600.
The following illustration shows failures that occur if the Cube processing job cannot access one or more of the warehouse resources that it requires:
To troubleshoot this issue, you must determine which Warehouse Data Sync job is preventing the Cube processing job from accessing the resource or resources that it needs. This report pack does not provide an easy way to determine which Warehouse Data Sync job is causing the problem, but you can determine that information by examining the Warehouse Job Status view. As the following illustration shows, the warehouse data for the problematic job will be much older than the warehouse data for other jobs for the same team project collection:
To troubleshoot issues with individual Warehouse Data Sync jobs, first unblock the overall Warehouse Sync process by disabling the offending job to allow re rest of the process proceed, and attempt to solve the issue with the individual job afterwards.
According to the process for synchronizing the warehouse, the cube should be processed incrementally throughout the day, and then a full synchronization should occur every day at 2:00 AM. Full synchronization jobs usually run longer and consume more system resources than Incremental jobs. However, the system will try to correct itself if an Incremental job failed. In that situation, the next Incremental job will be upgraded to a full synchronization. If multiple Incremental jobs are upgraded to Full, as the following illustration shows, you might first determine whether your network connectivity is reliable. You should inspect the error that the failing job returned and then address the issue.
If your network loses connectivity, the server-side execution of the Analysis processing job might finish but fail to report the job completion to the processing component for Team Foundation Server. Because of the same network failures, the resource lock might be released, but the Job Agent might not update the job’s state. The following illustration shows that a Full processing job started on June 24, 2010, at 2:00 AM and ran for more than 24 hours. Because it released the processing lock, the Incremental job was running in parallel with it.
The following illustration shows the worst case of the same problem. The Incremental job has run for more than nine hours, which means that no other jobs are scheduled and the cube is at least nine hours out of date. To mitigate this issue, you should use the AnalysisServicesProcessingTimeout setting for processing the cube for Team Foundation Server. This MSDN article describes how to Change a Process Control Setting for the Data Warehouse or Analysis Services Cube.
When a Team Project Collection gets blocked due to schema merge conflicts, the Warehouse Job Status table will show the conflict with a link to the sub report that displays details about the blocked fields. If you click the link that appears under the schema conflict error, a different report appears and shows the fields that are currently active and blocked for the blocked team project collection. See illustration below. For additional help on resolving the schema merge conflicts see Resolving Schema Conflicts That Are Occurring in the Data Warehouse.
One of the key components of TFS is the Data Warehouse, which is made up of a relational database and an Analysis Services cube. In general, people don’t have a problem with the performance or operation of our Data Warehouse. However there were two classes of problems that you’re likely to run into as your servers grow larger:
Internally at Microsoft we started noticing some problems after we upgraded TFS2008 servers and consolidated them onto larger TFS2010 servers as new Team Project Collections (TPCs). There is a constraint (or feature, depending how you look at it) in the system that you can only have a single data warehouse per instance of TFS2010. This feature enables you to do cross-project reporting for gathering data across all the projects in all collections on that server. The downside of this though, is that when a field’s reporting settings change in one team project in one collection on a server, it can impact the data warehouse experience for everybody else on that server.
The good news is that through our dogfooding efforts on the Pioneer server, we found some issues early enough and made changes to the product before it shipped to avoid some other issues. While we were investigating and fixing these issues, we developed some reports that visualize the information that TFS stores about the health of the data warehouse to track our improvements. Now we’re sharing those reports with you.
Please download them (below), install them, send us any feedback (comments on this blog are fine) and we’ll work on including them in the next official Power Tools.
These reports are useful to TFS administrators, operations/support teams, project administrators & end-users. The reports in this pack display the following kinds of information:
In the download, there is a FAQ document which includes screenshots of what different reports mean and common questions. The contents of the FAQ are also available at Monitoring the TFS Data Warehouse – FAQ.
Download and install this report pack to the Team Foundation Server Reporting Services Instance to monitor warehouse and cube processing status.
For information about how to install this report pack, see Installing Admin Report Pack for TFS.docx that is included in the download.
The pack includes:
Here’s an overview of how the reports look and what questions you can answer with them.
Use this report when you want to answer the following questions:
This table tells you whether the warehouse is currently being processed or when it will be processed next.
The Next Run Time is in local time, and Run time is the duration of the job that is currently running.
This table lists all team project collections and all Data Adapter Jobs and displays how long ago the Warehouse and Cube data were updated. In addition, the table will show the schema conflicts and other Data Adapter errors that caused data to be out of date:
If a conflict occurs when two schemas merge, the table will show the conflict with a link to the sub report that displays details about the blocked fields.
If you click the link that appears under the schema conflict error, a different report appears and shows the fields that are currently active and blocked for the blocked team project collection.
The following table shows blocked fields that have conflicts over all team project collections.
Blocked fields appear before the fields with which they are conflicting. The conflicting areas appear in bold.
The following table shows field changes that are queued behind the blocked changes in the previous illustration.
After you resolve the blocked changes, the queued changes will then be applied to the warehouse.
This report shows all reportable fields in the deployment of Team Foundation Server. Administrators of team projects should review this report before they add a reportable field or change the properties of an existing field. The report helps prevent potential schema-merge conflicts. It lists fields across all collections, including any fields that are blocked.
Found 2 fields that match ‘Found In’
This report shows list of recently completed Cube processing jobs.
I hope that you find these reports useful. Please comment on this post if you have questions or other feedback.
To prepare for the DevDiv TFS2010 upgrade we had to copy 8TB of SQL backups about 100 miles across a WAN link so that we could restore it on our test system. The link speed was reasonably good and the latency fairly low (5ms), but when you’re dealing with files this big then the odds are against you and using sneakernet can be a good option. In our case it wasn’t an option and we had to find the next best solution. In the end we were able to copy all 8TB over 7 days without having to resume or restart once.
The 8TB backups were spanned across 32 files of 250GB each which makes them a little easier to deal with. The first problem that you’ll encounter when using a normal Windows file copy, XCopy, RoboCopy or TeraCopy to copy these large files is that your available memory on the source server will start to drop and eventually run out. The next problem you’ll encounter is the connection will break for some reason and you’ll have to restart or resume the transfer.
Fortunately the EPS Windows Server Performance Team have a blog post on the issue and a great recommendation: Ask the Performance Team : Slow Large File Copy Issues
The problem lies in the way in which the copy is performed - specifically Buffered vs. Unbuffered Input/Output (I/O). Buffered I/O describes the process by which the file system will buffer reads and writes to and from the disk in the file system cache. Buffered I/O is intended to speed up future reads and writes to the same file but it has an associated overhead cost. It is effective for speeding up access to files that may change periodically or get accessed frequently. There are two buffered I/O functions commonly used in Windows Applications such as Explorer, Copy, Robocopy or XCopy: CopyFile() - Copies an existing file to a new file CopyFileEx() - This also copies an existing file to a new file, but it can also call a specified callback function each time a portion of the copy operation is completed, thus notifying the application of its progress via the callback function. Additionally, CopyFileEx can be canceled during the copy operation. So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data. Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data. CreateFile() - The CreateFile function creates or opens a file, file stream, directory, physical disk, volume, console buffer, tape drive, communications resource, mailslot, or named pipe. The function returns a handle that can be used to access an object. ReadFile() - The ReadFile function reads data from a file, and starts at the position that the file pointer indicates. You can use this function for both synchronous and asynchronous operations. WriteFile() - The WriteFile function writes data to a file at the position specified by the file pointer. This function is designed for both synchronous and asynchronous operation.
The problem lies in the way in which the copy is performed - specifically Buffered vs. Unbuffered Input/Output (I/O).
Buffered I/O describes the process by which the file system will buffer reads and writes to and from the disk in the file system cache. Buffered I/O is intended to speed up future reads and writes to the same file but it has an associated overhead cost. It is effective for speeding up access to files that may change periodically or get accessed frequently. There are two buffered I/O functions commonly used in Windows Applications such as Explorer, Copy, Robocopy or XCopy:
So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data. Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data.
Which Tool? ESEUTIL
Yes, the tool has some limitations – but in my experience it’s well worth the time investment to get running. See How to Run Eseutil /Y (Copy File)
To get the utility, you need access to an Exchange server or to install Exchange in Administrator-only mode. When you install Exchange in Administrator-only mode, the appropriate binaries are copied to your computer and you can then copy these three files off and use them on another computer:
ese.dll eseutil.exe exchmem.dll
It does not accept wildcard characters (such as *.* to copy all files), so you must have to specify a file name and copy one file at a time. Or use a command like: FOR %f IN (d:\backups\*.BAK) DO ESEUTIL /Y "%f"
COPY FILE: DESCRIPTION: Copies a database or log file. SYNTAX: D:\BIN\ESEUTIL /y <source file> [options] PARAMETERS: <source file> - name of file to copy OPTIONS: zero or more of the following switches, separated by a space: /d<file> - destination file (default: copy source file to current directory) /o - suppress logo NOTES: 1) If performed on arbitrary files, this operation may fail at the end of the file if its size is not sector-aligned.
Example
D:\>d:\bin\eseutil /y c:\Backups\Backup1.bak /d \\destination\c$\Backups\Backup1.bak Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.01 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating COPY FILE mode... Source File: c:\Backups\Backup1.bak Destination File: \\destination\c$\Backups\Backup1.bak Copy Progress (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Operation completed successfully in 7.67 seconds.
D:\>d:\bin\eseutil /y c:\Backups\Backup1.bak /d \\destination\c$\Backups\Backup1.bak
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.01 Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating COPY FILE mode... Source File: c:\Backups\Backup1.bak Destination File: \\destination\c$\Backups\Backup1.bak
Copy Progress (% complete)
0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ...................................................
Operation completed successfully in 7.67 seconds.
Other Tools
If you read the comments of the performance team’s blog post, you’ll see that XCopy has a /J option in Windows 7 and Windows 2008 R2 that does unbuffered I/O. However that’s not an option when you haven’t upgraded to R2 yet.
/J Copies using unbuffered I/O. Recommended for very large files.
Which Direction?
Through trial and error, we determined that it was much more reliable to run eseutil.exe on the SOURCE server and push the files to the remote share. This seemed to absorb any network blips and required no manual interruption over the 7 days it took us to copy the files.
Verifying hashes
The third problem you want to avoid is getting the files copied and then finding out that they match in size but the contents are corrupt. You can check for this by generating hashes on both the source and target systems and comparing them after the copy.
You can download the Microsoft File Checksum Integrity Verifier fciv.exe tool from Microsoft Downloads.
Then run it like this on each system:
fciv.exe C:\Backups -type *.bak -r -wp -xml hashes.xml