Speaking of which... : OutOfMemory

The KB article every one should know about

JohanS — Wed, 23 May 2007 18:23:46 GMT

Have you read http://support.microsoft.com/?kbid=307340? If not, I suggest you do so. If you need convincing or simply want to know why this problem occurs I suggest you keep reading.

The problem described in the article above can cause your application to spike in CPU time, Memory usage or both. This will lead to your application hanging or possibly even crashing. The cause of the problem is one of the most basic operations, and is something that we do very frequently.

The problem

The problem in a nutshell is that string concatenation is bad. I don't mean bad as in "You shouldn't eat that extra piece of chocolate". I mean it as in "You really shouldn't put your head in wet concrete". If you've been reading my previous post on memory management you might be able to figure out why string concatenation could be such a potential hazard. Consider the following scenario:

You're getting ready to go grocery shopping. You take a sheet of paper and a pen and begin writing down what you need to buy. The first item that crosses your mind is milk, so you write it down. Now you remember that you're out of bread as well, so you throw away the first sheet of paper, pick out a new one and write down milk and bread. You then throw away this paper too and write down milk, bread and apples on a third piece of paper. You throw away the third piece of paper and repeat until finally the list is done. If you have 20 items on your list you now have 19 discarded papers lying on the floor waiting for the "garbage collector".

This is exactly how regular string concatenation works in .NET. Every time you concatenate a string the framework needs to allocate a new memory segment, large enough to hold the result of the concatenation and store the string in this new segment. The old segment is flagged as ready for garbage collection and as we all know garbage collection costs CPU and pending garbage collection costs memory.

A real example

So if you thought that 19 sheets of paper were a waste consider the following piece of code:

private String buildTable()
{
   String sReturn;
   int i, j;
   sReturn = "<table>"
   for (i = 0; i < 35; i++) // Rows
   {
      sReturn += "<tr>"
      for (j = 0; j < 15; j++) // Columns
      {
         sReturn += "<td>"
         sReturn += i.ToString() + " : " + j.ToString();
         sReturn += "</td>"
      }
      sReturn += "</tr>"
   }
   sReturn += "</table>"
   return sReturn;
}

All in all we have 2697 concatenations which means we have 2696 unnecessary allocations. The allocations will be gradually bigger and bigger, so the last ones will be almost as big as the final table. Needless to say this will consume a lot of unnecessary memory. Also, if this table was built for a "real" application it would probably contain a lot more data for each cell, not to mention custom styles and classes. Imagine what would happen when the string goes past the magic 85000 bytes barrier that defines what goes on the large object heap and what doesn't. In a larger scale this would have a severe impact on performance.

This is a really weird behavior, how can it be?!

It's not that odd really. If we extend the grocery shopping analogy a little we can see why there's a logical reason for this: You know that you want to write something down. You have a limited amount of paper, so what do you do? You cut out a piece of paper just large enough for you to write what you need. When you find that you need to add another item to the list you'll have to discard the old piece and cut out a new one. Not very practical, agreed, but you're trying to minimize your paper usage.

As the clever beings that we are we normally make a rough estimate of how big a piece of paper we need and then start writing. The framework, however, has no way of estimating how big a string is going to be. It doesn't know if you're going to concatenate it zero or ten thousand time, so that's something you'll have to estimate yourself.

So what should I do instead?

One word: StringBuilder

The StringBuilder class is made especially for this type of operations. When using the StringBuilder class our sample above would look something like this

private String buildTableSB()
{
   StringBuilder sReturn = new StringBuilder();
   sReturn.Capacity = 8000; // This is optional, but may be very efficient
   int i, j;
   sReturn.Append("<table>");
   for (i = 0; i < 35; i++) // Rows
   {
      sReturn.Append("<tr>");
      for (j = 0; j < 15; j++) // Columns
      {
         sReturn.Append("<td>");
         sReturn.Append(i.ToString());
         sReturn.Append(" : ");
         sReturn.Append(j.ToString());
         sReturn.Append("</td>");
      }
      sReturn.Append("</tr>");
   }
   sReturn.Append("</table>");
   return sReturn.ToString;
}

The difference in performance for a larger operation is enormous. I highly recommend trying out these samples. Copy the functions to a winforms/webforms application, increase the number of rows to 500 and run both versions. The StringBuilder version will finish within milliseconds, and the "classic" version will not... For additional tidbits, take a look at the GC counter in performance monitor and look at how many GCs you have to make to complete the operation.

Bye for now / Johan

Memory management in the .NET Framework

JohanS — Fri, 20 Apr 2007 16:07:29 GMT

This is a subject that has been covered before and I have no intention of writing the ultimate post on the subject. Still I think this is something that every good developer should know.

Why do I need to know this?

My colleagues and I are quite often asked about the necessity of knowing how the Garbage Collector (GC) works. After all, it isn't really necessary to know what's going on behind the scenes, right..? A developer shouldn't have to bother about how the framework works..? Or should he..?

Well you may be a pretty good bus driver even if you know nothing about engines. Basic stuff like not stepping on the break and the gas at the same time is probably enough. If you want to do F1 racing, on the other hand, you will want to know as much as you can about your car. So, if you're content doing small scale web applications or not so efficient winforms applications, then you probably don't have to bother. If you want to do front-line work, then you've got to get your hands dirty.

The different generations

In order to be effective the managed heap is trying to figure out which objects it can disregard and only check up on occasionally. Some objects may span the entire lifecycle of the application and others may be very short lived, so if an object stays alive for a longer period of time the GC will check up on it less frequently. This is accomplished by dividing the managed heap into three generations. Generations 0, 1 and 2. An object will begin in generation 0 and if it is in use for a long period of time it will travel through generations 0 to 2. A possible metaphor for this scenario would the following:

You're asked to call a customer. You're pretty good with numbers so you keep the phone number in your head. (Generation 0)
There is no reply and so you need to remember the phone number a little longer. Meanwhile you're given additional numbers to call.
You know from past experience that your limit for keeping numbers in your head is 10, so once you reach that critical point you decide to write down the numbers you still need on post-it notes (Generation 1)
As the day goes on you keep transferring numbers from memory to post-its.
Your desktop has now reached maximum capacity. It is cluttered with post-it notes, so you sort through them, determining which notes you still need. You throw away the ones that are obsolete and write down the remaining to your address book. (Generation 2)

Like I stated above; when an object is created it is put in Generation 0. The initial size limit of Generation 0 is determined by the size of the processor cache. This is dynamically changed depending on the allocation rate of the application. Once Generation 0 reaches its limit it will go through all the items in Generation 0, tag the obsolete objects for collection and remove them. This is called the Mark and Sweep phase. Everything that survives this sweep will be compacted and moved to Generation 1. The size limit of Generation 1 is also determined by the allocation rate of the application, and once it is reached a Generation 1 collection will occur. This will first mark and sweep all items in Generation 1, moving the surviving items to Generation 2, and then mark and sweep the items in Generation 0.

The healthy ratio between GC's in the different generations is approximately 100 - 10 - 1, so for 100 Generation 0 GC's you normally have 10 GC's of Generation 1 and 1 of Generation 2. A normal Generation 0 GC usually takes a few milliseconds. Performing a GC of Generation 1 rarely takes more than 30 milliseconds, but a Generation 2 collection can take quite some time depending on the application.

The Large Object Heap

To top it off we also have what is called "The Large Object Heap" (LOH). This is where anything larger than 85.000 bytes will be stored. The LOH is collected each time a new segment needs to be reserved (see below), but it is not compacted like generations 0, 1 and 2. When you perform a collection on the LOH you will also perform a GC on the other generations.

Now, if the limit is at 85.000 bytes, won't most of my objects end up on the LOH? - Well not necessarily. For example: A dataset may contain lots and lots of data, but the dataset object only contains references to other objects. The data in the columns are each stored in individual strings and as long as they're not larger than 85 KB you're clear.

Memory segments

The managed heap will reserve memory in segments. The size of the segments depend on the configuration. If <gcServer enabled="true"/> you will reserve memory in 64 MB segments, otherwise you'll be doing it in 32 MB segments. LOH are reserved in 16 MB segments. Only Generation 2 and the LOH will span several segments.

What happens during GC?

Let's say we're performing a full GC, including the LOH. This is what happens:

The objects on the LOH are Marked. Each item in the LOH is checked for references. If none are found it's ready to be collected.
The LOH is Swept. All marked objects are released from memory.
The LOH is not compacted.
Generation 2 is Marked.
Generation 2 is Swept.
Generation 2 is compacted. (Imagine removing a few books from your bookcase, and then pushing the remaining books together freeing up continuous space at the end.)
Generation 1 is Marked.
Generation 1 is Swept.
Everything that survived the sweep is compacted.
The pointer for where Generation 2 ends is updated. Everything that survived the sweep is now Generation 2.
Generation 0 is Marked.
Generation 0 is Swept.
Everything that survived the sweep is compacted.
The pointer for where Generation 1 ends is updated. Everything that survived the sweep is now Generation 1.

A few quick tips

This topic could off course be a lot bigger, but here are some quick suggestions.

Try to stay out of Generation 1

Well off course Generation 1 is better than Generation 2, but you should aim for keeping only a select few objects in Generation 2. Those should be variables that are defined at the beginning of the application lifecycle and released at the end. In an ideal world all other variables should be of the hit-and-run variety and never leave Generation 0.

Don't call GC.Collect()

This has been said before. It will be said again, and again, and again. You should almost never call GC.Collect() manually. And by almost I mean once in a lifetime, not once per application, and certainly not once per function call. I occasionally call GC.Collect() for testing purposes just to see if memory has been released. Normally you would never call it. The GC is self-balancing and by calling GC.Collect() you are disrupting that balance. Think of it as tampering with the eco-system, pouring sugar in the gasoline or whatever metaphor you prefer. :-)

Avoid large objects

If you can stay below the 85.000 byte limit, then do so. If not, consider reusing the object. When it comes to large objects it's better to use one for a long time than to use many for short periods of time.

Don't use finalizers

When your object has a finalizer the finalize method will be called when the object is no longer alive. So far so good. Unfortunately your object will be passed into the next generation, since it's not yet ready to be collected. This means that all objects with a finalizer will at least end up in Generation 1. Most likely in Generation 2.

Well, as I said: There is a lot more to cover on this. There are books to be written and songs to be sung, but you've got to draw the line somewhere. I will most certainly cover more of this in future posts.

/ Johan

Why adding more memory won't resolve OutOfMemoryExceptions

JohanS — Thu, 19 Apr 2007 12:43:59 GMT

Quite often I am met with the incorrect assumption that out of memory exceptions can be resolved by adding more memory. I can understand why you'd think that, but actually it won't matter at all. Additional RAM may increase performance, but an additional 8GB of memory won't increase the available amount of memory for the .NET application.

Here's what you need to know:

If you have a 32 bit machine then you have 2 GB of memory available.

That's the way it is. Never mind if you have 512 MB or 8 GB of RAM. A 32-bit system can address 4 GB of virtual memory. 2 GB of those are reserved for the operating system, leaving 2 GB for each user mode process. If there is no RAM available we will use the page file and save infrequently used memory to disk instead. So additional RAM will cause fewer page faults, but the amount of memory available for the process is still only 2 GB.

(Actually you can tune this in boot.ini. There is a switch that will allow you to steal 1 GB from the OS and increase the limit for the user process from 2 GB to 3 GB. It still requires some additional tweaking though. I suggest looking at http://support.microsoft.com/kb/820108/en-us for more information on this.)

Okay, so now that we know this it's time for the next vital piece of information:

Your process will most likely be out of memory when it reaches ~800 MB

- What!? How is that possible!? Didn't I just say that we have 2 GB available? I did, but there is another thing you really need to know about memory allocations:

Memory allocations need to be continuous.

If you want to allocate a 100 MB string, then you need 100 MB of continuous space. Memory allocations don't work the way the file system does. If we were talking about files we could save part of the file in one place, another part in another place, etc. but memory != file system. Open up the disk defragmenter and take a look at your hard drive. You may have 100 GB available on your 250 GB hard drive, but how much of that is continuous? The available memory for your process will eventually become just as fragmented as your hard drive. And as your memory becomes more and more fragmented it will be harder and harder to squeeze in those large allocations.

Let's say you and your party of four walk into a restaurant. The restaurant is packed with four seat tables so finding a seat shouldn't be a problem. Unfortunately there is 1-3 persons sitting at each table. So even if only half the seats are taken you are still unable to get your own table. The usher gets an OutOfSeatsException as he tries to seat you. :-)

The rule of thumb is that you'll begin seeing OutOfMemoryExceptions at around 800 MB. Off course this isn't set in stone. It depends entirely on the average size of the objects you allocate and how fragmented your memory becomes. You might make it up to 1.2 GB or you might see the exceptions already at 550 MB.

So what can be done?

Well if you're in the planning stage of a WebService that will load 4 GB images, process them somehow and then return them to the client you'll want to either:

Work with streams and make sure you never load the full image into memory
Set aside the money to invest in a 64 bit system
Rethink

I'm currently writing a bigger post on how the managed heap and garbage collector work, so expect more to come on the subject.

/ Johan

I am getting OutOfMemoryExceptions. How can I troubleshoot this?

JohanS — Thu, 11 Jan 2007 19:46:47 GMT

Problem:

You've written an ASP.NET application that is getting OutOfMemoryExceptions.

Cause:

Let's find out...

Resolution:

Use Windbg to take a look at the heap.

Is it a leak?

Take a look at the memory usage of you application using perfmon. If memory is slowly increasing and never released, then you have a leak. If it is going up and down like a rollercoaster, then you are most likely using huge amounts of memory for certain operations which is later garbage collected.

How do I troubleshoot this?

Below I'll try to give you a step by step description of how to troubleshoot this scenario

1. Get a memory dump

This is done using Windbg and Adplus. If you don't have Windbg installed, check out my previous post.

2. Open up the dump and load SOS

Open the dump in Windbg and load the SOS extension. You'll find it in the framework directory so if you're debugging an application for Framework 2.0 look in "C:\Windows\Microsoft.NET\Framework\v2.0.50727". Anyway, load SOS by typing:

.load [path]\sos

3. Run dumpheap

Execute the following command:

!dumpheap -stat

It will show statistics for the objects on the heap in a nice little summary divided in tho four columns.

The method table of the object
The number of objects of this type on the heap
The total size of these objects in bytes
The name of the object type

Be careful not to omit the -stat parameter. If you do then Windbg will dump the address of each object in the entire heap to your screen, which will be a lot of information to say the least.

Analyzing the information provided by !dumpheap

Here's a sample heap from one of my cases...

0:000> !dumpheap -stat
Statistics:
MT             Count       TotalSize Class Name
7a787cc4           1              12 System.IO.FileSystemWatcher+FSWAsyncResult
7a75904c           1              12 System.Diagnostics.OrdinalCaseInsensitiveComparer
7a7575cc           1              12 System.CodeDom.Compiler.CodeDomConfigurationHandler
7a7571a8           1              12 System.Net.DefaultCertPolicy
7a75661c           1              12 System.Diagnostics.TraceListenerCollection
7a755834           1            12 System.Diagnostics.PerformanceCounterCategoryType
........CONTINUED.........
68a66a88     227,559      12,743,304 System.Web.UI.WebControls.Literal
68a2f7fc     399,272      14,373,792 System.Web.UI.ControlCollection
68a92e2c     768,731      33,824,164 System.Web.UI.Control+OccasionalFields
68a884a0     641,952      38,517,120 System.Web.UI.LiteralControl
79124228     879,515      43,394,976 System.Object[]
790fa3e0   1,431,594     122,806,484 System.String

Total 10,389,625 objects, Total size: 463,313,540

So in this particular dump I have 1,431,594 strings whose total size is 122 MBytes, 879,515 Objects with a total size of 43 MBytes, etc.

Objects in heap may be larger than they appear

The TotalSize column isn't 100% true. Look at the LiteralControls that are on third place. They only use a total of 38 MBytes, or do they? The TotalSize refers to the object structure, but the member variables like strings, integers and other child objects are not included. It kind of makes sense, since otherwise the Total size would be completely off the scale. The LiteralControl objects contain a number of child objects and three of those are strings. Their respective size is listed with the System.String object.

Using !dumpobj

Let's take a closer look at one of the System.Web.UI.LiteralControls. We list all of the controls with the following command !dumpheap -type System.Web.UI.LiteralControl and quickly press Ctrl+Break before the screen fills with too many lines:

0:000> !dumpheap -type System.Web.UI.LiteralControl
Address       MT Size Gen
023ea0a8 68a884a0   60   2 System.Web.UI.LiteralControl
023ea0e4 68a884a0   60   2 System.Web.UI.LiteralControl
023ea374 68a884a0   60   2 System.Web.UI.LiteralControl
023ea460 68a884a0   60   2 System.Web.UI.LiteralControl
023ea510 68a884a0   60   2 System.Web.UI.LiteralControl
023eab3c 68a884a0   60   2 System.Web.UI.LiteralControl
........CONTINUED........
023fe31c 68a884a0   60   2 System.Web.UI.LiteralControl
023fe414 68a884a0   60   2 System.Web.UI.LiteralControl
023fe4c4 68a884a0   60   2 System.Web.UI.LiteralControl
023fe500 68a884a0   60   2 System.Web.UI.LiteralControl

As you can see each LiteralControl is 60 bytes in size. Like I said before that's the size of the object structure alone, not it's referenced objects and properties. We now pick the address of one of the LiteralControls and execute the !dumpobj command (!do for short). This gives us the following result:

0:000> !do 023ea0a8
Name: System.Web.UI.LiteralControl
MethodTable: 68a884a0
EEClass: 68a88428
Size: 60(0x3c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field Offset                   Type   VT     Attr    Value Name
790fa3e0 4001fe0      4          System.String    0 instance 00000000 _id
790fa3e0 4001fe1     8          System.String    0 instance 00000000 _cachedUniqueID
68a2af44 4001fe2     c   ...em.Web.UI.Control    0 instance 023e8864 _parent
68a91070 4001fe3     2c           System.Int32    0 instance        0 _controlState
68a85ea0 4001fe4     10   ...m.Web.UI.StateBag    0 instance 00000000 _viewState
68a2af44 4001fe5     14   ...em.Web.UI.Control    0 instance 023e8864 _namingContainer
68a273d0 4001fe6     18     System.Web.UI.Page    0 instance 01df4514 _page
68a92e2c 4001fe7     1c   ...+OccasionalFields    0 instance 00000000 _occasionalFields
68a2b378 4001fe8     20   ...I.TemplateControl    0 instance 00000000 _templateControl
68a14528 4001fe9     24   ...m.Web.VirtualPath    0 instance 00000000 _templateSourceVirtualDirectory
68a8bb48 4001fea    28   ...rs.ControlAdapter    0 instance 00000000 _adapter
68a3a8f8 4001feb    30   ...SimpleBitVector32    1 instance 023ea0d8 flags
790f9c18 4001fda    c70          System.Object    0   shared   static EventDataBinding
>> Domain:Value 000f0d00:NotInit 0011a720:01df0028 <<
790f9c18 4001fdb    c74          System.Object    0   shared   static EventInit
>> Domain:Value 000f0d00:NotInit 0011a720:01df0034 <<
790f9c18 4001fdc    c78          System.Object    0   shared   static EventLoad
>> Domain:Value 000f0d00:NotInit 0011a720:01df0040 <<
790f9c18 4001fdd    c7c          System.Object    0   shared   static EventUnload
>> Domain:Value 000f0d00:NotInit 0011a720:01df004c <<
790f9c18 4001fde    c80          System.Object    0   shared   static EventPreRender
>> Domain:Value 000f0d00:NotInit 0011a720:01df0058 <<
790f9c18 4001fdf    c84          System.Object    0   shared   static EventDisposed
>> Domain:Value 000f0d00:NotInit 0011a720:01df0064 <<
79124228 4001fec    c88        System.Object[]    0   shared   static automaticIDs
>> Domain:Value 000f0d00:NotInit 0011a720:01df0070 <<
790fa3e0 4002211     34          System.String    0 instance 02238664 _text

Cool, now we can take a look at the specifics. For example the value of the text-property, which is located in the string down at the bottom with address 02238664. To get it's value, simply perform a !do on the address:

0:000> !do 02238664
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 158(0x9e) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: </td>
</tr>
</table>


Fields:
MT Field Offset Type VT Attr Value Name
790fed1c 4000096 4 System.Int32 0 instance 71 m_arrayLength
790fed1c 4000097 8 System.Int32 0 instance 70 m_stringLength
790fbefc 4000098 c System.Char 0 instance 3c m_firstChar
790fa3e0 4000099 10 System.String 0 shared static Empty
>> Domain:Value 000f0d00:790d6584 0011a720:790d6584 <<
79124670 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 000f0d00:01d413b8 0011a720:01d44f80 <<

Okay, so we can see that the string contains some data to close off a table. We can also take a look at the other properties for the object and examine them if we wish. But there is another command that is really useful...

Using !objsize

Is there a way to get the total size of a specific System.Web.UI.LiteralControl? - Simple! We use the !objsize -command. !objsize looks at all the pointers within an object and calculate their total size. Below is the built in documentation for the !objsize -command:

0:000> !help objsize
-------------------------------------------------------------------------------
!ObjSize [<Object address>]

With no parameters, !ObjSize lists the size of all objects found on managed
threads. It also enumerates all GCHandles in the process, and totals the size
of any objects pointed to by those handles. In calculating object size,
!ObjSize includes the size of all child objects in addition to the parent.

For example, !DumpObj lists a size of 20 bytes for this Customer object:

0:000> !do a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(C:\pub\unittest.exe)
Fields:
MT         Field Offset Type              Attr    Value  Name
009038ec 4000008      4 CLASS          instance 00a79ce4  name
009038ec 4000009      8 CLASS          instance 00a79d2c bank
009038ec 400000a      c System.Boolean instance        1 valid

but !ObjSize lists 152 bytes:
0:000> !ObjSize a79d40
sizeof(00a79d40) = 152 ( 0x98) bytes (Customer)

This is because a Customer points to a Bank, has a name, and the Bank points to
an Address string. You can use !ObjSize to identify any particularly large
objects, such as a managed cache in a web server.

Objects in heap may also be smaller than they appear

So what do we get if we run !objsize on our LiteralControl? This is really interesting, because what happens is; the debugger gets really busy for quite some time and eventually we get this:

0:000> !objsize 023ea0a8
sizeof(023ea0a8) = 456918136 ( 0x1b3c0478) bytes (System.Web.UI.LiteralControl)

456 MBytes! How is that possible? Well if you scroll up to where we ran the !do command on the LiteralControl, you'll see that the control holds a reference to the page. The page in turn has a reference to the cache, and before long we'll have referenced almost the entire heap.

Summary

Hopefully this is enough to give you a quick glimpse of what is possible with three relatively simple commands from the sos extension. The commands were:

!dumpheap
!dumpobj
!objsize

Over and out

/ Johan

Are you getting OutOfMemoryExceptions when uploading large files?

JohanS — Wed, 15 Nov 2006 12:13:00 GMT

Problem:

Using the WebClient.Upload method for posting large files will eventually leave you stranded with OutOfMemoryExceptions.

Cause:

WebClient.Upload reads the entire file to memory by default.

Resolution:

Build your own uploader.

Scenario:

One of my customers was using WebClient.Upload in a Winforms application to transfer files to a webserver. The idea in itself was fine, but when they transferred a couple of large files they'd get OutOfMemoryExceptions. When uploading a 500 MB file the application would need approximately 520 MB of memory and if you uploaded a few large files after each other you quickly hit the roof. Running GC.Collect(); after each transfer didn't help. Judging from the number of hits on the internet for this scenario they weren't the only ones with this problem.

This is the code they were using:

WebClient oWeb = new WebClient();
oWeb.UploadFile("http://localhost/test.aspx", "c:\\bigfile.cab");

Okay, so why was this happening?
Well, first of all I wouldn't recommend running GC.Collect(); in any application. A lot has been written on this allready, but if you're interested in why I suggest you look at Rico Mariani's post on the subject. Anyway, for testing purposes we ran the following instead:

GC.Collect(3);

GC.WaitForPendingFinalizers();

GC.Collect(3);

And this cleared the memory. So why isn't this a valid solution? Well, like I said, I wouldn't recommend using GC.Collect(); in any application, and why read the entire file to memory when you can stream it? I looked up the UploadFile-method and it seems like it does read the entire file to a byte array before posting. This is great for smaller files, but in this particular scenario it wasn't too good. So what I did was to write my own uploader:

public static string MyUploader(string strFileToUpload, string strUrl)

{

string strFileFormName = "file";

Uri oUri = new Uri(strUrl);

string strBoundary = "----------" + DateTime.Now.Ticks.ToString("x");

// The trailing boundary string

byte[] boundaryBytes = Encoding.ASCII.GetBytes("\r\n--" + strBoundary + "\r\n");

// The post message header

StringBuilder sb = new StringBuilder();

sb.Append("--");

sb.Append(strBoundary);

sb.Append("\r\n");

sb.Append("Content-Disposition: form-data; name=\"");

sb.Append(strFileFormName);

sb.Append("\"; filename=\"");

sb.Append(Path.GetFileName(strFileToUpload));

sb.Append("\"");

sb.Append("\r\n");

sb.Append("Content-Type: ");

sb.Append("application/octet-stream");

sb.Append("\r\n");

string strPostHeader = sb.ToString();

byte[] postHeaderBytes = Encoding.UTF8.GetBytes(strPostHeader);

// The WebRequest

HttpWebRequest oWebrequest = (HttpWebRequest)WebRequest.Create(oUri);

oWebrequest.ContentType = "multipart/form-data; boundary=" + strBoundary;

oWebrequest.Method = "POST";

// This is important, otherwise the whole file will be read to memory anyway...

oWebrequest.AllowWriteStreamBuffering = false;

// Get a FileStream and set the final properties of the WebRequest

FileStream oFileStream = new FileStream(strFileToUpload, FileMode.Open, FileAccess.Read);

long length = postHeaderBytes.Length + oFileStream.Length + boundaryBytes.Length;

oWebrequest.ContentLength = length;

Stream oRequestStream = oWebrequest.GetRequestStream();

// Write the post header

oRequestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);

// Stream the file contents in small pieces (4096 bytes, max).

byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))];

int bytesRead = 0;

while ((bytesRead = oFileStream.Read(buffer, 0, buffer.Length)) != 0)

oRequestStream.Write(buffer, 0, bytesRead);

oFileStream.Close();

// Add the trailing boundary

oRequestStream.Write(boundaryBytes, 0, boundaryBytes.Length);

WebResponse oWResponse = oWebrequest.GetResponse();

Stream s = oWResponse.GetResponseStream();

StreamReader sr = new StreamReader(s);

String sReturnString = sr.ReadToEnd();

// Clean up

oFileStream.Close();

oRequestStream.Close();

s.Close();

sr.Close();

return sReturnString;

}

One of the things worth noting is that you need to set oWebrequest.AllowWriteStreamBuffering = false; Otherwise you will read the entire file to memory anyway. This is because the default behavior of the WebRequest is to buffer the entire request in case it needs to re-send it due to authentication, connectivity problems, etc. Again, this is a default behavior that normally is a performance boost, but in this case is a performance killer.

So what was the end result?

During my first test runs the application needed as much memory as the file I was trying to upload, and then some. So in order to upload a 500 MB .cab-file the application needed at least 520 MB. The application using the custom uploader never went above 23 MB.

End of transmission

/ Johan