TFS2010: Diagnosing Email and SOAP subscription failures

Grant Holliday’s blog

Senior Service Engineer, Microsoft Visual Studio Team Foundation Service

TFS2010: Diagnosing Email and SOAP subscription failures

Rate This
  • Comments 7

Sometimes emails alerts such as "Work Item Changed" will fail to send.  Sometimes SOAP (web service) subscriptions will not be called correctly.

See Chris Sidi's blog for an overview of the TFS Job Agent service 

To diagnose these, you can look in the tbl_JobHistory table in the Tfs_Configuration database.

If you run this SQL command, it will retrieve the last 10 failed notification jobs:

SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage

FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]

WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0

ORDER BY StartTime DESC

The JobId GUID is the well-known GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension”

The DATEADD() function converts the date from UTC to local Pacific time.

ResultMessage:

There were errors or warnings during notification delivery.    0/4 emails delivered.  0/6 soap notifications delivered.   1 errors.  0 warnings.    Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings.   

To get detailed information about why the job failed, you will need to set a flag in the TFS registry. This is different from the windows registry and probably warrants a blog post on it’s own.

Changing NotificationJobLogLevel setting in the TFS registry using PowerShell

If you have VS2010 & PowerShell installed on your local machine and you have the appropriate permissions on the server, then you can run these steps from there. Otherwise, you will need to logon to a machine that does, or an AT.  If you logon to the AT, you will need to run PowerShell as an Administrator.

To change the registry setting, replace the server url & collection name and follow these steps:

  1. Open PowerShell
  2. Run the following commands:

# Load client OM assembly.
[Reflection.Assembly]::Load("Microsoft.TeamFoundation.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a");

$collectionBaseUrl = "http://YOURTFSSERVER:8080/tfs/YOURCOLLECTION/";

$tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($collectionBaseUrl);
$collectionHive = $tfs.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry]);

# Set the setting in the collection hive.
$collectionHive.SetValue("/Service/Integration/Settings/NotificationJobLogLevel", "2");

It’s a good idea to set this back to “0” once you’re done with it, to avoid filling up the job history logs unnecessarily.

Once you have set this setting, the next time the job runs and fails, it will log more detailed information including the exception and stack trace. For example:

There were errors or warnings during notification delivery.    0/0 emails delivered.  0/3 soap notifications delivered.   3 errors.  0 warnings.    -------------------------------  Notification not delivered.    Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://eventsserver/EventHandler.svc)    Exception: Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: Team Foundation services are not available from server http://eventsserver/EventHandler.svc.  Technical information (for administrator):    HTTP code 415: Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'. ---> System.Net.WebException: The remote server returned an error: (415) Cannot process the message because the content type 'application/soap+xml; charset=utf-8' was not the expected type 'text/xml; charset=utf-8'..     at System.Net.HttpWebRequest.GetResponse()     at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.AsyncWebRequest.ExecRequest(Object obj)     --- End of inner exception stack trace ---     at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader)     at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response)     at Microsoft.TeamFoundation.JobService.Extensions.Core.TeamFoundationNotificationClient.Notify(String eventXml, String tfsIdentityXml, Subscription subscription)     at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification, TeamFoundationIdentityService identityService) 

This particular error message indicates that the client was expecting a SOAP 1.1 message, and the server sent it a SOAP 1.2 message.

If your email alerts are failing to send, then you’ll get an SMTP error back that shows you where to start looking.

  • Grant,

       How does this new functionality in 2010 address the issue you outlined here: http://blogs.msdn.com/granth/archive/2008/09/15/does-tfs-guarantee-event-subscription-delivery.aspx

    We are worried about the Ithe event being permanently lost.

    We are looking to use FDD and TFS 2010. see the issues outlined here: http://www.featuredrivendevelopment.com/node/1103

  • Any idea if the issue you outine before is fixed in 2010?

  • It's by design and that design hasn't been changed in TFS2010. Event delivery is still "best effort", which means ~6 retries.  After that, the failed event delivery gets logged in the job history in TFS.

    If you are developing any tool that uses the events, the recommended pattern is to have a mechanism that runs periodically to "catch up" any missed events by querying work items / changesets, etc.

    If you want to guarantee your actions happen on Checkin (or any server event), then there is a new extensibility model in 2010 - Checkout the \Plugins directory under \Web Services for examples.

  • Hi Grant

    I'm trying to create an extension for TFS to control work item changes. But my service does not run on TFS 2010 Beta2.

    Could you help me out?

    I've posted my question on MSDN without any luck:

    http://social.msdn.microsoft.com/Forums/en/tfsprerelease/thread/029e824e-daa0-4f0e-80eb-47d997ff77bb

    Thanks for your help!

    Nicolas

  • Any comments about the reason of this exception?

    Notification not delivered.    Notification: WorkItemChangedEvent (DeliveryType: Soap; Address: http://MyPC-w7/tfseventlistener/WorkItemSubscriber.svc)    Exception: System.Web.Services.Protocols.SoapException: The message with Action 'schemas.microsoft.com/.../Notify& cannot be processed at the receiver, due to a ContractFilter mismatch at the EndpointDispatcher. This may be because of either a contract mismatch (mismatched Actions between sender and receiver) or a binding/security mismatch between the sender and the receiver.  Check that sender and receiver have the same contract and the same binding (including security requirements, e.g. Message, Transport, None).     at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ProcessHttpResponse(HttpWebResponse response, Stream responseStream, WebException webException, XmlReader& xmlResponseReader)     at Microsoft.TeamFoundation.Client.TeamFoundationClientProxyBase.ExecWebServiceRequest(HttpWebRequest request, XmlWriter requestXml, String methodName, HttpWebResponse& response)     at Microsoft.TeamFoundation.JobService.Extensions.Core.TeamFoundationNotificationClient.Notify(String eventXml, String tfsIdentityXml, Subscription subscription)     at Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension.SendSoapNotification(TeamFoundationRequestContext requestContext, TeamFoundationNotification notification, TeamFoundationIdentityService identityService)

  • what priveleges are required to register an event (ie. Project Administrator, Collection Administrator or Server Administrator)

  • Hi,

    We are facing a typical problem of huge number of notifications coming as e-mails even when the workitems are not changed.

    This particular problem is when the MPP is published to TFS. The MPP has a large number of workitems which are refreshed from TFS. For all these workitems in MPP, the Publish and Refresh option is set to No but enabled to Yes for only the workitems that are required to be published to TFS. In TFS the project alerts are enabled to send a notification when the workitems assigned to me are changed. When I publish this MPP into TFS, I get the notifications for all the workitems including the one's that are not changed and that are not actually published from MPP to TFS thereby flooding my mailbox with a large number of e-mails. Can someone try to explain the logic behind this notifications and suggest how I can avoid getting the notifications for the workitems which are actually not changed.

    The support is highly appreciated.

Page 1 of 1 (7 items)