Working on some cases related to the Asynchronous service in CRM, I have come across couple of questions from users. Some of them are not documented or discussed on any blogs as it is related to how the asynchronous service works.
I will be discussing the Asynchronous service behavior in an FAQ model as it will be more easier to understand it.
So lets move to the first query,
1. I have two servers which hosts CRM Asynchronous services, Server A and Server B which is on a Load balancer. When I look at the Asynchronous jobs on both servers, it never appears as same. At some instance I see one server has 15 jobs and other sitting Idle some times vice versa. At the end of the day when I look at the jobs it is not similar.
So what is the reason for the difference in jobs on both Asynchronous servers, Though it is on a load balancer?
CRM Asynchronous service processes the jobs independently i.e. if you have multiple Async servers it will not sync with each other to really balance the jobs.
Each servers polls the CRM web service and in turn the "AsyncoperationBase" table for Jobs at specific intervals and they may find some work to do.
If one server picks up the available Jobs, another server may starve for jobs i.e. if one server is able to grab and process the available jobs and keep selecting jobs with none left for the other server, other server will starve for jobs.
So you cannot really load balance the Asynchronous service. All you can really do is have overflow. The servers will be competing with each other for work rather than sharing the work.
To better understand I will explain the behavior giving two scenarios,
Before Jumping on to the scenarios it would better to explain below terms,
AsyncItemsInMemoryHigh – Max number of async operations the service will store in memory. Upon selection interval, if the number of items in memory falls below AsyncItemsInMemoryLow, the service will pick enough to reach up to AsyncItemsInMemoryHigh again.
AsyncItemsInMemoryLow – Minimum number of async operation the service needs to have in memory before loading new jobs into memory. Upon selection interval, if the number of items in memory falls below this value, the service will pick up enough to reach AsyncItemsInMemoryHigh again.
• AsyncItemsInMemoryHigh is 2000 (This is the default value)
• AsyncItemsInMemoryLow is 1000 (This is the default value)
• AsyncSelectInterval is 5 (seconds) (This is the default value)
• Current jobs ready to be processed in CRM is 5000
• Two Asynchronous Services (AS1(Server A), AS2(Server B))
• Services both servers starts at the same time
AS1 and AS2 will start selecting works 5 seconds after service initialization is complete in chunks of AsyncItemsInMemoryHigh – Outstanding Operations. Outstanding Operations are operations currently loaded into memory from previous selects.
Only 1 service should be able to select at a time to avoid both services selecting the same work.
Since both have empty outstanding operation queues they will select the max of 2000 rows each.
Let’s say all the work AS2 selected is long running work and so after 5 seconds it still has 2000 outstanding operations. It will skip its selection of work.
Let’s say all the work AS1 selected is short running work and so after 5 seconds it has processed all of its work. It will select another AsyncItemsInMemoryHigh – Outstanding Operations, in other words 2000 items. There is only 1000 items in the queue so it selects 1000.
As you can see from this example AS1 did 60% of the work while AS2 did 40%. This ratio can continue to slide one way or the other. i.e. if one service continues to be unlucky and select work that takes longer, or when it queries the AsyncoperationBase table there is nothing left you will the Big difference on both servers.
• AsyncItemsInMemoryHigh is 2000
• AsyncItemsInMemoryLow is 1000
• AsyncSelectInterval is 10 (seconds) (Modified in DB)
• Current jobs ready to be processed in CRM is 5, with 5 new work items introduced every 20 seconds
• Two Async Services (AS1, AS2)
• Services both starts at same time.
After 10 seconds AS1 will select all 5 jobs
AS2 will have to wait for AS1 to select its work because AS1 got to the work just a little bit faster, and so AS2 will have no work to complete, and wait another 10 seconds
After 10 seconds 5 new jobs show up.
After 10 seconds AS1 will select all 5 new jobs
Again AS2 will have no work to complete, and wait another 10 seconds
In this case AS1 will do 100% of the work.
2. How CRM Asynchronous service process the job? Is it in batch OR one by one?
In your CRM deployment there are two Asynchronous servers and each server will poll the ‘AsyncOperationBase’ table looking for the work to perform. It will select work in batches up to the limit (‘AsyncItemsInMemoryHigh’ this value is stored in the DB) It wants it’s in memory queue to be as full as possible. This can result in 1 server performing more work than the other.
3. From where Asynchronous service pulls the jobs? Is it queries the DB or the CRM Web service?
CRM Asynchronous service polls CRM Web service every 5 seconds. This is by default and stores this value as ‘AsyncSelectInterval’ in the database.
Now let’s move to some CRM tweaks to improve the performance.
Being said that 2 Asynchronous servers polls the web service for jobs, we may come in to a situation where two servers actively looking for jobs to process and will certainly at same point will try to process the same jobs causing a deadlock.
You can Configure the following registry key on the Asynchronous servers in order to improve performance and avoid dead locks.
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRMName: AsyncDBAppLock Type: DWORDValue: 1
For more information on the above registry key refer to the following KB:http://support.microsoft.com/kb/2249156
Refer to below article which discuss about CRM 4.0 SQL Dead locks.
If Microsoft CRM Async Service show signs of high CPU usage and high memory usage when a high transaction load and a large number of workflows are in use with Microsoft CRM. You can follow below KB article which discuss this in detail,
Gireesh T G
Dynamics CRM Support Engineer.