You might got into this situation before... you are working in the Dynamics AX Client as usual and all of a sudden you see the following warning message:
After you click OK the Dynamics AX Client is closed (terminated).
OK, what has happened now? While the error message is technically correct, it's a bit misleading. The AOS would like to tell the Client that it is running out of memory and is trying to free used resources to keep "surviving". Usually when one user is getting this message other users are starting seeing this message too - until the AOS is restarted. It can also happen, and usually does, that the AOS service is crashing soon after the first users are seeing this message.
This message is often an indicator of either:
What is a Memory Leak or a High Memory Situation?
A Memory Leak is a situation where for an object (or more general a resource) memory was once requested, that is however not freed when the object (resource) is no longer needed and the last reference to the object was removed.
So if the application developer decides it is a good idea to hold GB of data in memory (e. g. for caching) this is not a Memory Leak – although looking at the memory consumption of the process it might look like. Unfortunately this example happens too often in reality and so some badly written X++ code can easily cause lots of trouble. We call this "not clever" usage of memory not a Memory Leak but instead a High Memory Situation.
So the difference is mainly if there is any chance to release the object (resource) and it's allocated memory at a later time or not.
Memory Leaks in Dynamics AX are seldom and usually located in the Dynamics AX Kernel itself. This does not mean they don't exists, in Dynamics AX 4.0 the issues KB948185, KB958213, KB973633 and KB978110 are some examples of a real Memory Leak.
But saying that, most of the time when you think you are running into a Memory Leak in reality you are facing a High Memory Situation caused by custom X++ code or custom modifications of standard code.
Why is a Memory Leak or a High Memory Situation bad?
Each user mode process in Windows can allocate memory – Virtual Memory. The amount of Virtual Memory depends on the version / configuration of Windows and the processor architecture (x86 or x64). It does not directly depend on the amount of Physical Memory (actually it could if the application requires but for Dynamics AX it does not). For more information on Virtual Memory see the Blog post Memory Management 101.
In a 32bit (x86) version of Windows in total 4 GB of Virtual Memory can be addressed in total however only a part of the address space is available for the user mode process, by default 2 GB. There are situation where 3 GB and even 4 GB are available for the 32bit user mode process, however I don't want to get into detail here. For more information see the KB article 888732.
The important fact to remember is: We have a limited amount of addressable Virtual Memory and if this is used up we are running into an out of memory situation. And unfortunately the Dynamics AX Kernel doesn't handle out of memory situations gracefully. When we are running out of memory the process most of the time simply terminates (crashes).
Tracking down the root cause
Finding out the cause of a Memory Leak or a High Memory Situation is a harder task. In the following sections I'm describing steps that can help to narrow down the cause. But still a lot of manual work is required to identify the piece of code bringing Dynamics AX into trouble.
Finding patterns in the memory consumption
As a start we should find out how the memory consumption behaves. Does the once allocated memory keep allocated or is it released when users are logging off or a lengthy running task is completed?
The Windows Task Manager can give us a first hint if the memory consumption changes quickly. On the Processes tab the column Virtual Memory Size (<= Win2003) or Memory – Working Set (>= Vista) is the interesting one to look at.
If the memory consumption is changing slower over a longer period of time the Performance Monitor is the better tool of choice. In Performance Monitor the following counters are usually helpful:
The Private Bytes and Virtual Bytes are the important ones, the other three just complete the picture.
Once the Performance Monitor log was recorded it needs to be analyzed and checked for common patterns:
Checking these steps can already give you a first idea where to look at.
Separating users on different AOS instances
If enough AOS instances are available or enough licenses for new AOS servers exist is has proved to be a good idea to separate three groups of users to their own AOS instances to see what group of users is involved in the issue:
Seeing which user groups AOS instance is suffering from the memory consumption gives another ideas where to look at.
And by the way, looking at customization is always a good idea to start with.
The root cause is the load
Lets imagine you have a High Memory Situation caused by your custom X++ code. But looking at your X++ code you see it is optimal and using memory and resources wisely. Simply too many users at the same time are executing the code.
If you can't limit the number of users executing your code what can you do next?
Over the time we have seen some common issues that cause effects identified as Memory Leaks or High Memory Situation: