I have always noticed many of my customers complaining about a very common scenario in CRM where the system jobs are always in “waiting” status. Though there can be many reasons for this behavior I thought of writing about the most common scenario I have ever noticed. It is pretty easy to approach this problem if we know how system jobs are processed in CRM. In CRM 4.0 we have got a pretty neat windows service called “Microsoft CRM Asynchronous Processing Service”. One of the many functions of this service is to process the system jobs. So always ensure that this service is started.
The service will build its own URL to hit the CRM web service to process the jobs. In any case if it cannot hit the web service properly after building the URL the job will go to waiting.
The reason is simple as asynchronous service is building a wrong CRM web service URL.
There are 3 key values which determine the CRM web service URL built by Asynchronous service: ADRootDomainScheme:
This will tell the asynchronous service about the protocol (HTTP or HTTPS) to be used while hitting the web service. This value is mentioned in the “DeploymentProperties” table in the MSCRM_CONFIG database under column “NVarCharColumn” corresponding to ColumnName “ADRootDomainScheme”
This will tell the asynchronous service the hostname / IP address to be used while hitting the web service. This value is mentioned in the “DeploymentProperties” table in MSCRM_CONFIG database under column “NVarCharColumn” corresponding to ColumnName “AsyncSdkRootDomain”. On a default CRM installation this value will be empty and Asynchronous service interprets an empty value as loopback address - 127.0.0.1
This determines the port to be used while hitting the web service. This value is mentioned as a registry key under “HKLM\Software\Microsoft\MSCRM” in the server where asynchronous service is running (This will be the CRM application server unless you have a split server role installation where you might have a dedicated server running asynchronous service). In a typical CRM installation if you select “Default website” in IIS for installing CRM, then this value will be 80 and if you select a new website the value will be 5555 unless you change it manually.
ADRootDomainScheme – HTTP AsyncSdkRootDomain – empty LocalSdkPort – 5555 URL built: http://127.0.0.1:5555/MSCrmServices/2007/CrmService.asmx
ADRootDomainScheme – HTTP AsyncSdkRootDomain – empty LocalSdkPort – 80 URL built: http://127.0.0.1/MSCrmServices/2007/CrmService.asmx
ADRootDomainScheme – HTTPS AsyncSdkRootDomain – empty LocalSdkPort – 443 URL built: https://127.0.0.1/MSCrmServices/2007/CrmService.asmx In Scenario3 you might have configured the CRM website to accept requests only on HTTPS by installing an SSL certificate and configuring IIS to accept only HTTPS requests. In that case you need to change the ADRootDomainScheme and LocalSdkPort values accordingly so that Asynchronous service can hit the correct web service URL. Please note that the asynchronous service can parse the SSL certificate warnings so it is not required to populate AsyncSdkRootDomain with the SSL cert name. But it is recommended.
ADRootDomainScheme – HTTP AsyncSdkRootDomain – IPaddress (of CRMServer) LocalSdkPort – 5555 URL built: http://IPaddress:5555/MSCrmServices/2007/CrmService.asmx In Scenario4 you might have configured the CRM website under a dedicated IP address. You need to change ADRootDomainScheme and LocalSdkPort accordingly if the same website is locked with an SSL certificate.
ADRootDomainScheme – HTTP AsyncSdkRootDomain – host header name LocalSdkPort – 5555 URL built: http://hostheader:5555/MSCrmServices/2007/CrmService.asmx
In Scenario5 you might have configured the CRM website to work with an alias host name by adding a host header to CRM website and removing the default unassigned binding in IIS. You need to change ADRootDomainScheme and LocalSdkPort accordingly if the same website is locked with an SSL certificate.
In a load balanced environment it is recommended to populate AsyncSdkRootDomain with the load balanced URL / alias name
I always prefer to check if all the system jobs including workflow jobs are in waiting or if it happens only to specific system jobs. If it happens to only specific system jobs then the reason can be different as those jobs in succeeded status clearly tell us that the asynchronous service is building and hitting the web service URL in a correct fashion. It is a good to confirm this by creating a sample workflow like one which will create an activity on creation of an account / contact. Another placeholder to check is the IIS logs for the CRM website. If we check the IIS logs we are likely to see entries as:
#Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2010-01-27 00:02:44 #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 401 2 2148074254 2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 401 1 0 2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 REPLICA\CRMREPLICA$ 127.0.0.1 Mozilla/4.0+ (compatible; +MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 200 0 0
In the above case the AsyncSdkRootDomain value is set as empty (default) hence the source IP is loopback and CRM website is on port 5555 with an “all unassigned” binding in IIS. Note the sc-status / sc-substatus codes. 401.2 is expected as CRM website is not configured for anonymous authentication followed by 401.1 which is an NTLM challenge followed by 200 (Status OK/authenticated) code shows us that the request made to CRM web service is a success.
I also prefer to do an advance find on those system jobs (http://blogs.msdn.com/benlec/archive/2009/02/10/how-to-troubleshoot-system-jobs-failures-using-advanced-find.aspx) on waiting status after customizing my advance find view to add columns “Error Code” & “Message”. As always my last option would be a CRM platform trace.