Working on some cases related to the Asynchronous service in CRM, I have come
across couple of questions  from users.  Some of them are not documented or
discussed on any blogs as it is related to how the asynchronous service works.

I will be discussing the Asynchronous service behavior in an FAQ model as it
will be more easier to understand it.

So lets move to the first query,

1. I have two servers which hosts CRM Asynchronous services, Server A and
Server B which is on a Load balancer. When I look at the Asynchronous jobs on
both servers, it never appears as same. At some instance I see one server has 15
jobs and other sitting Idle some times vice versa. At the end of the day when I
look at the jobs it is not similar.

So what is the reason for the difference in jobs
on both Asynchronous servers, Though it is on a load
balancer?

CRM Asynchronous service  processes the jobs independently i.e. if you have
multiple Async servers it will not sync with each other to really balance the
jobs.

Each servers polls the CRM web service and in turn the "AsyncoperationBase"
table for Jobs at specific intervals and they may find some work to do.

If one server picks up the available Jobs, another server may starve for jobs
i.e.  if one server is able to grab and process the available jobs and keep
selecting jobs with none left for the other server, other server will starve for
jobs.

So you cannot really load balance the Asynchronous service. All you can
really do is have overflow. The servers will be competing with each other for
work rather than sharing the work. 

To better understand I will explain the behavior giving two scenarios,

Before Jumping on to the scenarios it would better to explain below
terms,

AsyncItemsInMemoryHigh – Max number of async operations the
service will store in memory. Upon selection interval, if the number of items in
memory falls below AsyncItemsInMemoryLow, the service will pick enough to reach
up to AsyncItemsInMemoryHigh again.

AsyncItemsInMemoryLow – Minimum number of async
operation the service needs to have in memory before loading new jobs into
memory. Upon selection interval, if the number of items in memory falls below
this value, the service will pick up enough to reach AsyncItemsInMemoryHigh
again.

 Scenario 1

•  AsyncItemsInMemoryHigh is 2000 (This is the default value)

•  AsyncItemsInMemoryLow is 1000 (This is the default value)

•  AsyncSelectInterval is 5 (seconds) (This is the default value)

• Current jobs ready to be processed in CRM is 5000

• Two Asynchronous Services (AS1(Server A), AS2(Server B))

• Services both servers starts at the same time

AS1 and AS2 will start selecting works 5 seconds after service initialization
is complete in chunks of AsyncItemsInMemoryHigh – Outstanding Operations.
Outstanding Operations are operations currently loaded into memory from previous
selects.

Only 1 service should be able to select at a time to avoid both services
selecting the same work.

Since both have empty outstanding operation queues they will select the max
of 2000 rows each.

Let’s say all the work AS2 selected is long running work and so after 5
seconds it still has 2000 outstanding operations. It will skip its selection of
work.

Let’s say all the work AS1 selected is short running work and so after 5
seconds it has processed all of its work. It will select another
AsyncItemsInMemoryHigh – Outstanding Operations, in other words 2000 items.
There is only 1000 items in the queue so it selects 1000.

As you can see from this example AS1 did 60% of the work while AS2 did 40%.
This ratio can continue to slide one way or the other. i.e. if one service
continues to be unlucky and select work that takes longer, or when it queries
the AsyncoperationBase table there is nothing left you will the Big difference
on both servers.

Scenario 2

 

• AsyncItemsInMemoryHigh is 2000

• AsyncItemsInMemoryLow is 1000

• AsyncSelectInterval is 10 (seconds) (Modified in DB)

• Current jobs ready to be processed in CRM is 5, with 5 new work items
introduced every 20 seconds

• Two Async Services (AS1, AS2)

• Services both starts at same time.

After 10 seconds AS1 will select all 5 jobs

AS2 will have to wait for AS1 to select its work because AS1 got to the work
just a little bit faster, and so AS2 will have no work to complete, and wait
another 10 seconds

After 10 seconds 5 new jobs show up.

After 10 seconds AS1 will select all 5 new jobs

Again AS2 will have no work to complete, and wait another 10 seconds

In this case AS1 will do 100% of the work.

2. How CRM Asynchronous service process the job? Is it in
batch OR one by one?

In your CRM deployment there are two Asynchronous servers and each server
will poll the ‘AsyncOperationBase’ table looking for the work to perform. It
will select work in batches up to the limit (‘AsyncItemsInMemoryHigh’ this value
is stored in the DB) It wants it’s in memory queue to be as full as possible.
This can result in 1 server performing more work than the other.

3. From where Asynchronous service pulls the jobs? Is it
queries the DB or the CRM Web service?

CRM Asynchronous service polls CRM Web service every 5  seconds. This is by
default and stores this value as ‘AsyncSelectInterval’ in the database.

Now let’s move to some CRM tweaks to improve the performance.

Being said that 2 Asynchronous servers polls the web service for jobs, we may come
in to a situation where two servers actively looking for jobs to process and
will certainly at same point will try to process the same jobs causing a
deadlock.

You can Configure the following registry key on the Asynchronous servers in
order to improve performance and avoid dead locks.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM
Name: AsyncDBAppLock          

Type: DWORD
Value: 1

For more information on the above registry key refer to the following
KB:
http://support.microsoft.com/kb/2249156  

Refer to below article which discuss about CRM 4.0 SQL Dead locks.

http://social.technet.microsoft.com/wiki/contents/articles/6755.dynamics-crm-4-0-sql-dead-locks.aspx

If Microsoft CRM Async Service show signs of high CPU usage and high memory
usage when a high transaction load and a large number of workflows are in use
with Microsoft CRM. You can follow below KB article which discuss this in
detail,

http://support.microsoft.com/kb/2489162

 

Best Regards,

 

Gireesh T G

Dynamics CRM Support Engineer.