CLR, architectures, and stuff

Limiting resource consumption on servers.

Patrick Dussud — Tue, 08 May 2007 02:33:00 GMT

In my last blog entry I described the problem of servers over committing resources. There are several solution to this problem. The simplest and most common is to make sure that the critical resource is easily and smoothly controlled by the host OS: CPU time. If the server load is CPU bound, the other resources won’t be a problem. IT operation groups know how to monitor and load balance servers based on CPU utilization.

A much more sophisticated but general solution is based on controlling the inbound processing rate of requests. It requires to monitor *all* of the resources which can become scarce (memory, database connections, etc..) and start throttling the processing of incoming requests when any of these resources become close to their maximum. This can be accomplished by placing incoming requests into a staging queue where they will be picked up and processed. The staging queue itself can be maintained using very few resources from the server - some memory, very little CPU, and pretty much nothing else because the request is left into an un-processed state when it is queued up. The staging queue length needs to be regulated: Ideally, it should be very small when the server is running under optimal load and it should be constant under steady state. It should be allowed to grow when resources get scarce because of an over commitment. The idea is that this over commitment is going to be resolved when the few “abnormal” requests have been fully processed.

This approach works pretty well in the situations where I have seen it deployed, as long as the server process we are talking about is not competing for the resources with another process on the same host. I have seen cases where a server is doing great and suddenly the available physical memory drops on the machine because of a competing program. The server goes into throttling mode but it isn’t effective enough to prevent paging because the amount of memory it has committed cannot decrease as fast as the demand from the competing process. When the server starts paging, things get into a standstill because most of the memory pages are hot, containing data for the processing of the requests. I have seen people taking drastic measures to make up for this: They either abandon the semi computed state for some requests and put them back on the staging queue, or they serialize the semi processed state onto disk and put the request onto another queue, to be processed once the scarce situation is over. Serializing the semi processed state is difficult because it requires the developer to know exactly for each request what needs to be serialized. There could be an issue with shared data between requests. Another problem with serialization is that it uses disk resources which maybe scarce already and it may compete with the OS trying to page memory out to disk.

I recommend trying out or simulating the basic staging queue approach before going to more sophisticated approaches. The pop3 server which I described in my previous arcticle did fine after they implemented the staging queue approach: It normally ran with 100MB of physical memory load; when the physical memory went above 200MB, it stopped processing incoming requests, which queued up on the staging queue. The staging queue had a maximum size of 4000 entries. It is interesting to note that the measured throughput varies more because it decreases when the staging queue is growing but it does not crash anymore!

Modeling servers for resource consumption

Patrick Dussud — Tue, 05 Dec 2006 03:57:00 GMT

In the past few years, I have helped various groups deal with server problems related to resource usage. A typical scenario goes like this “we have a server we are about to release. It does great on our internal stress tests – generally composed of a server under an artificial load of multiple client machines – and when we put it online, it runs great for a while and then it crashes or stops working because it runs out of resource X – memory, database connection, whatever finite resource you can imagine –“ In general these folks have been very good at making their server CPU efficient: at the maximum load specified the CPU usage on the machine under say 80%. Indeed, when things start to go bad, the CPU is far from pegged.

Let’s take the specific example of a CLR based POP3 mail server. The team did a great job to make it efficient and simple. On a 4 CPU machine, all CPUs showed the same 75% load at the specified maximum artificial load. The memory usage was about 100MB (the machine had 512MB of memory and no swap file). When deployed online this server would last anywhere between a few days and 2 weeks before crashing with an out of memory exception. Looking at the full memory dump collected at the time of the exception, we could see that the heap usage was very high, with a lot of very large byte arrays. We traced them back to the actual contents of the mail messages retrieved from the email store and on their way to the user’s mail client. These messages were a lot larger than the average mail payload (less than 20KB) and they were a surprising amount of them, enough to drive the server process out of memory. Actual request measurement told us that very few mails will be large, yet when the server crashed, we found a lot of them. This was the clue to understand the problem: small mail messages were sent quickly and all of the memory allocated by the request was quickly reclaimed by the garbage collector. In house stress tests showed this worked well. The requests for large pieces of mail were the problem, a few of them got sent quickly but some of them were sent to dialup users so the actual time for the whole request to be over was very large compared to the average requests. Soon these lingering requests accumulated and the memory got exhausted. I asked the server team to show me their load specifications: They counted an average request size (this is for all of the objects in use during a request) of 20KB and an average of 4000 requests in flight. The heap size should therefore be around 80MB. They told me that the maximum payload size was 4MB (I think I remember that it was limited by the server, which wouldn’t accept single email pieces larger than that). A quick calculation shows that if the server was unlucky enough to serve only the largest email pieces, the heap size would have to be 16GB to accommodate 4000 request in flight. Clearly that’s not possible on the CLR platform for Win32. Therefore if it was possible to crash the server by requesting enough large pieces of mail. By accepting and starting to process all of the incoming requests, the server have overcommitted and died. This actual workload is hard to replicate in house because generally, labs are not equipped with slow communication lines such as dial ups.

Writing a CLR based simulator for servers.

It turns out we can model these resource usage variations and simulate them fairly simply:

At heart, a server takes requests, create some memory and ties up some resources, sends a response back and flushes the request. The lifetime of the request, the resource usage and the incoming rate are what makes the server go into an overcommitted situation.

We can model this simply by allocating and storing some object simulating the requests into a queue at every “tick” of the simulator. This simulates the incoming requests. The size of the queue represents the number of requests in flight in the server. Now the requests need to be processed and a response needs to be written back. To simulate the processing , you allocate some unrooted (temporary ) objects and some objects rooted in the request object. They represent the memory consumption of the request. You can randomly vary their sizes according to the profile of the server load you are simulating. To start simply, you can do this allocation in the same step as when you queue up the request. You can also allocate whatever other resource you need to model (database connection, etc..). A more sophisticated processing would entail assigning steps or states to the request objects and have them go through a series of steps or state transitions – each with some processing and resource consumption- to reach the final state: the response has been written out. To simulate the final state, you can remove the requests in semi random order. To simulate the POP3 server, the requests with the smallest simulated response would be removed first and the ones with the largest responses would stay in the queue for more ticks. This super simple model will reproduce the instability of servers without a throttling mechanism. It is helpful in evaluating real throttling mechanisms in a very simple environment. In another post I will talk about some server throttling mechanisms.

Aggregated response to comments

Patrick Dussud — Fri, 01 Dec 2006 21:47:00 GMT

I am aggregating the responses to the comments about my first post:

About Common Lisp and Lisp in general:

I have worked on Lisp for many years, and I think it has strength that sets it apart from other dynamic languages. My favorite is the fact that you can extend the Lisp language to create your own special purpose language in a very natural way using macros. Lisp has a very simple syntax (prefix notation). As long as your extensions adhere to that syntax, it is possible to make them seamless. Arguably, CLOS is a very large extension to Common Lisp, even though you don't need *any* compiler changes to do a pretty good job implementing it on top of regular Common Lisp.

Last year, I gave a talk at the International Lisp Conference about the need for Lisp to integrate with the CLR (and / or Java). See the following URL

http://www.international-lisp-conference.org/2005/speakers.html#patrick_dussud

About JVM limitations:

Well, the JVM has technical limitations that were hard for us to solve: the JVM is designed to execute type safe languages only and we wanted to executed unsafe C++ as well. Solving this would have required the support of the entire Java community and this didn't make sense for the community at large.

Other limitation were contractual. We discovered that the compliance test suite didn't allow us to extend the existing Java classes so it was hard for us to reflect the Windows platform with more fidelity.

We also discovered that it was very hard to negociate for a different / better Java to Native code interop layer. This was both a technical and political issue: the interop layer is heavily archicture specific - it is hard to make it universal aka Run anywhere- and it is quite visible to programmers unlike an extension library on the side.

About the rationale for our design decisions:

First and foremost, we wanted to support safe execution of type safe languages . We designed a security model to separate trusted code from untrusted code and app domains to isolate code execution contexts from each other. We also wanted to be able to recompile an existing C++ application with an extra /CLR flag and run this application under the CLR. These high level goals lead us to where we are now. We briefly thought about a small GCed runtime for C++ but because C++ isn't type safe, it wasn't that interesting to us. Also you are severely limited in the ability to manage memory automatically for a language which can "hide" references in other datatypes or even by adding extra bits to the reference and such clever tricks.

How It All Started…AKA the Birth of the CLR

Patrick Dussud — Wed, 22 Nov 2006 00:58:00 GMT

Hello everyone, I am the Lead Architect of the CLR/UIFX group and the Chief Architect of the .NET Frameworks. I was thinking about what I should put in the first entry of my blog… as one of the founders of the CLR I thought it would be interesting to give you some history of how CLR was born in my blog’s intro entry.

Before the CLR I was working on JVM. When Mike Toutonghi, Peter Kukol, Anders Hejlsberg, and I realized that the JVM would not give us the flexibility to innovate in the places we cared about - support for other languages, deep interop with COM and unmanaged code, extensive frameworks to expose MS technology - we decided to join force with the COM team and form a new group, tentatively named COM 2.0.

We soon disagreed about object management, specifically about explicit management like COM (AddRef/Release) versus automatic management (Garbage Collection), we decided to form 2 groups: COM+ and CLR. This marked the birth of the CLR group.

Early on, we were joined by a meta data group who had been prototyping some meta data schema for the C++ compiler. Peter Kukol and the C++ group collaborated on the IL instruction set. I designed the architecture of the runtime and wrote the Garbage Collector (and yes the GC prototype was was written in Common Lisp first and I wrote a translator to convert it to C++).

Soon we had all the fundamental components of a runtime, some runtime helpers for the base classes (String and Object), and a new cool language: C#. We were in the business to build the full fledge CLR and the Framework.

So there you go, for those curious minds this is how CLR was started.

In the next blog entry I will be writing about server modeling. Also feel free to let me know if there are specific topics you want to see me write about.