TFS2010: Diagnosing Email and SOAP subscription failures
Sometimes emails alerts such as "Work Item Changed" will fail to send. Sometimes SOAP (web service) subscriptions will not be called correctly.
To diagnose these, you can look in the tbl_JobHistory table in the Tfs_Configuration database.
If you run this SQL command, it will retrieve the last 10 failed notification jobs:
SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0
ORDER BY StartTime DESC
The JobId GUID is the well-known GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension”
The DATEADD() function converts the date from UTC to local Pacific time.
ResultMessage:
There were errors or warnings during notification delivery. 0/4 emails delivered. 0/6 soap notifications delivered. 1 errors. 0 warnings. Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings.
To get detailed information about why the job failed, you will need to set a flag in the TFS registry. This is different from the windows registry and probably warrants a blog post on it’s own.
Changing NotificationJobLogLevel setting in the TFS registry using PowerShell
If you have VS2010 & PowerShell installed on your local machine and you have the appropriate permissions on the server, then you can run these steps from there. Otherwise, you will need to logon to a machine that does, or an AT. If you logon to the AT, you will need to run PowerShell as an Administrator.
To change the registry setting, replace the server url & collection name and follow these steps:
- Open PowerShell
- Run the following commands:
# Load client OM assembly.
[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a");
$collectionBaseUrl = "http://YOURTFSSERVER:8080/tfs/YOURCOLLECTION/";
$tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($collectionBaseUrl);
$collectionHive = $tfs.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry]);
# Set the setting in the collection hive.
$collectionHive.SetValue("/Service/Integration/Settings/NotificationJobLogLevel", "2");
It’s a good idea to set this back to “0” once you’re done with it, to avoid filling up the job history logs unnecessarily.
Once you have set this setting, the next time the job runs and fails, it will log more detailed information including the exception and stack trace. For example:
There were errors or warnings during notification delivery. 0/0 emails delivered. 0/3 soap notifications delivered. 3 errors. 0 warnings. ------------------------------- Notification not delivered. Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://eventsserver/EventHandler.svc) Exception: Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: Team Foundation services are not available from server http://eventsserver/EventHandler.svc. Technical information (for administrator): HTTP code 415: Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'. ---> System.Net.WebException: The remote server returned an error: (415) Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'.. at System.Net.HttpWebRequest.GetResponse() at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.AsyncWebRequest.ExecRequest(Object obj) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader) at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response) at Microsoft.TeamFoundation.JobService.Extensions.Core.TeamFoundationNotificationClient.Notify(String eventXml, String tfsIdentityXml, Subscription subscription) at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification, TeamFoundationIdentityService identityService)
This particular error message indicates that the client was expecting a SOAP 1.1 message, and the server sent it a SOAP 1.2 message.
If your email alerts are failing to send, then you’ll get an SMTP error back that shows you where to start looking.
