Before talking about how to troubleshoot memory issue of StreamInsight application, in this post, I list out several aspects or terminologies related to memory you should be aware of for the context of my memory performance blogs.

The Scenario

 

Assume your StreamInsight application is the only process running on the machine; the total memory can be allocated by your StreamInsight application is subject to “commit limit imposed by hardware and operating system the process is running at”. “Commit limit” is the sum of physical memory and sizes of page files.   In reality, not all of physical memory counts toward the commit limit since the operating system reserves part of physical memory for its own use. The amount of committed virtual memory for all the active processes, called the Current Commit Charge, cannot exceed the system commit limit. When the commit limit is reached, virtual allocations will fail.

 The current commit charge and commit limit can be viewed using Process Explorer in its System Information window.    

The following graph gives you a memory picture where total committed virtual memory in your application reaches the commit limit.

                 Graph 0 – Memory snapshot when application is close to running OOM

 The Current commit Charge counts for Physical Memory plus paging file setting. Due to page file setting it is hard to find exact amount the process uses through performance counter. So when you troubleshoot memory consumption issue of StreamInsight application, you can look up “Process<youapplicationname>\Private Bytes” performance counter to see whether it reaches the current commit limit.

StreamInsight Memory Consumption

 

The following diagnostic views are provided by StreamInsight to help you trouble shoot memory related issues. For information about all StreamInsight diagnostic views, please check out here.

All Events Memory [cep:/server/EventManager]

All Events Memory

This diagnostic gives you total memory allocation on incoming and output events

StreamInsight engine allocates and de-allocates memory for input, intermediate, and output events on the native heap during query processing.

Index Memory

OperatorIndexMemory

Cumulative memory (in bytes) occupied by the index structures in the operators, which stores state of operator, such as event being kept by the operator for incremental computation, or output correctness.

Stream Memory

StreamMemoryIncludingEvents

Sum of memory allocated for streams and the memory for events that constitute the stream event count (in bytes).

This is the total memory allocated for internal stream queue and the events in the queue, it has some overlap with the [Event Memory] in the event part, but relatively small, there are few events in the stream queue in most cases.

Managed Heap

Not only StreamInsight query manager but the input/output adapters allocate memory in the managed heap. You may find the amount of heap memory your process uses by looking at performance counter: [.NET CLR Memory\#Bytes in all heaps].

Case Study: My StreamInsight application throws OOM, what’s going on?

Now you have known different ways to gather memory usage for your process, let’s look at a specific scenario.

Assumption:  only StreamInsight application is running on the machine, and only one query is running.

Your process’s total memory consumption can be viewed by looking at performance counter [Process\Private Bytes], this include all memories that this process owns that cannot be shared with other processes. The memory includes all code, managed data and unmanaged data.

If you want to where all the memory goes, you could look at the following specific items:

o    Native memory consumption – this can be viewed using StreamInsight diagnostic views:

§  A = OperatorIndexMemory

§  Retrieve the value through query diagnostic view  [“cep:/Server/Application/<appname>/Query/<queryname>”]

§  B = AllEventsMemory

§  Retrieve the value through URI [“cep:/Server/EventManager”]

§  C= StreamMemoryIncludingEvents 

§  Retrieve the value through stream diagnostic view [cep:/Server/Application/<appname>/Query/<queryname>”]

o    Managed memory consumption

§  D = All managed memory consumption

§  Retrieve the value through performance counter [.NET CLR Memory \#Bytes in all Heaps]. Look up the current peak value through Perfmon, pick your application instance.

Usually the [Process\Private Bytes] will give you a slightly higher value than the sum of all above values, this is because there are other overhead in your application or some code that is not sharable with other processes.

Following memory snapshots demonstrate the above formula,  23GB(A) + 13GB(B) + 23MB (C) + 27GB(D) ~= 66GB (F peak private bytes)

Below Graph 1-4 demonstrate  the most critical statistics (A), (B), (D) and (F) you should look into when you troubleshoot memory problem in StreamInsight application; (C) is usually very small so can be ignored in most cases.

             

               

 

Graph 1, the index memory(A) allocation trend during query lifetime, the peak is 23GB

 

Graph 2, the Event memory (B) allocation trend during query lifetime, the peak is 13 GB.

 

Graph 3, #Bytes in all heaps(C) trend during query lifetime, the peak is 27GB

 

           

Graph 4, Process\Private Bytes (F) trend during query life time, the peak is 67 GB