“Ensure that the debug="false" on the <compilation> element in the web.config file of each and every ASP.NET application on the server. The default during development is "true" and it is a common mistake to allow this development time setting to find its way onto production servers during deployment. You don't need it set to true in production and it often leads to memory overhead and inefficiencies.”
What problems does leaving debug=true cause?
There are three main differences between debug=true and debug=false:
When debug is set to true, asp.net requests will not time out. This is to allow you to debug with visual studio at your own pace without having to worry about the requests suddenly disappearing. Of course in a production environment timeouts are crucial to avoid for requests to be stuck indefinitely, so this is reason #1 to make sure debug is set to false when the application is deployed into production.
In short, when debug=true, we don’t batch compile, when debug=false we do…
What does this mean?
When an aspx, asax, or ascx page is first requested it gets compiled into an assembly. This assembly has a name like 3ks0rnwz.dll or similar (8 characters) and stores the class for the actual ascx, asax, or aspx page (not the code behind). The assembly goes into a folder in the C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files with the same name as the application.
The code behind class gets compiled into the main dll for the assembly, and it along with all the other dlls in the applications bin directory get shadow copied to the Temporary ASP.NET files.
Back to the 3ks0rnwz.dll… If we have debug=true, we create one dll per aspx, asax, or ascx page and this dll is compiled in debug mode, so if you have 100 web pages, you will have 100 assemblies, and they are generated as the pages are requested.
If we instead have debug=false, we batch compile, which means that the first time you request any page in your application, we compile the whole batch into one big assembly. This is a truth with some modification. The user controls (ascx pages) are compiled into a separate assembly from the aspx pages and the aspx pages are compiled in groups based on what other files (read usercontrols) they include. The global.asax is also compiled separately. And batch compilation occurs on a directory bases, meaning that if your application has subdirectories, the subdirectories are compiled separately to avoid for example name clashes, as it is valid to have two aspx pages with the same name in different directories. But all in all, instead of 100 dlls, you might end up with 3 or 4.
Ok, big deal? It’s the same code so the size of the combined assemblies shouldn’t much differ from the size of the individual assemblies right? Truth is, there probably isn’t an enormous difference. But… and this is a big but… there is overhead for each dll, and if the dll is compiled in debug mode there is overhead for items needed for debugging, and … last but not least (in fact probably most important), the assemblies won’t be laid exactly side by side, so with a large number of assemblies you start fragmenting the virtual address space making it harder and harder to find large enough spaces to store the managed heaps, potentially causing out of memory exceptions.
One caution even if you have debug=false, is that if you go in and change something in one of your aspx pages, this page will have to be recompiled, but this doesn’t cause an appdomain reload so the whole application is not batch compiled again. This has the effect that the page will now get recompiled separately and get its own dll, so don’t change your aspx pages on a live server too often.
There is a setting in machine.config determining how many recompiles are allowed before the app domain restarts, by default it is set to 15, so after 15 recompilations the app domain will restart, just as it would if you touched the web.config or touched the bin directory.
In order to be able to step through code line by line the JITter can’t really optimize the code which means that your debug dlls will be less performant than if they were compiled in release mode.
So as you can probably figure, there is a large benefit to having debug=false in production…
How can you identify it in a memory dump?
To find out if any of the applications on your server run with debug=true you can run a nifty command in sos.dll called !finddebugtrue which will list out all applications where debug=true in the web.config, now how easy is thatJ
Debug set to true for Runtime: 61b48dc, AppDomain: /MyDebugApplication
Debug set to true for Runtime: 1f50e6d8, AppDomain: /MemoryIssues
Total 16 HttpRuntime objects
And to find out if you forgot to compile some of your assemblies in release mode run !finddebugmodules
Loading all modules.
Searching for modules built in debug mode...
MyDebugApplication.dll built debug
MemoryIssues.dll built debug
fl4sq-9i.dll built debug
wepr1t3k.dll built debug
r9ocxl4m.dll built debug
zmaozzfb.dll built debug
The dlls above with weird 8 character names are the dlls generate when JITing the aspx pages, so they will go away when debug=false.
Oh, before I forget… when you change from debug=true to debug=false it is a good idea to clean out your Temporary ASP.NET files for this application so you don’t have some old junk in there causing it to still not batch compile.
In ASP.NET 2.0 there is a switch that can be turned on in machine.config that turns off all debug=true, so in 2.0 applications you can do this directly without worrying about finding out which applications do and don’t have it.
If you want some more goodies about debug=true, read ScottGu’s blog post about it http://weblogs.asp.net/scottgu/archive/2006/04/11/442448.aspx
Do you have any general benchmarks on how much smaller the memory footprint can be if debug mode = false? Are we talking 50% more memory, twice as much memory, or more?
I understand that the answer is "It depends on your app", but do you have any rules of thumb. We've got a pretty complex app with hundreds of thousands of lines of code.
We would prefer to keep the debug info in there because we're concerned about the effort to retest the app and side effects it would have on our support team, who are used to using the debug info to resolve customer cases.
On the other hand, our app has a huge memory footprint. If the compile in release mode is going to reduce it by 50% or more, then we may need to look into it.
Thanks in advance
There is really no good way to say. Picture this... if you have 20 pages = 20 assemblies, and then your app allocates 1 GB worth of memory for various datasets etc. then the overhead for the debug data for the 20 assemblies would be negligeable. If on the other hand you have 1000 assemblies/pages and your app is otherwise frugal with memory usage then the overhead is not so negligeable so it is really impossible to say. Even on a specific page/assembly you can't give a percentage, and it also depends on if the page is changed during the app lifetime. The best thing is probably to test with debug=true or debug=false and see. But more importantly, with debug=true you will not take advantage of timeouts or other code optimization and that is really a bigger issue than the memory usage.
What type of debug info does your support team use that would not be available if you have debug=false? debug.write output for example would still be present...
Yes, it applies to web services as well. Web services (asmx) is basically a subset of asp.net
We've run into this issue in our ASP.net 3.5 code where the machine.congif has the Deployment retail=true and our code managed to get updated in Prod with Debug=true in the web.config.
However since the machine.config is set it should have been fine, yet it seems this is not working. Do you know if this is something that works in 3.5? A look at my memory dumps during an issue led me to the setting and then mass debate occured between systems, architects/devs and our team over how this could not happen..the machine.config settings are there.
Any experience on this?
APS.net 3.5.20729.1 (.Net 3.5 SP1) on Windows 2003 64 Bit using 64bit framework.
First I thought that debug=true would override this but after looking at this in more detail both in dumps and in documentation I found that retail=true disables the debug=true setting, so if retail=true then debug=true should not be in effect.
Thanks Tess for the response....we have an issue where the machine.config has retail=true yet when debug=true is set on the web applications our performance goes out the window.
If we set it to false (which why wouldn't we on production anyway?) it goes normal. Our issue has only really been that it slips up there in dpeloyments some times and the last time it got us we put in a critical support ticket with MS who turned around and said you have debugging enabled in production and we see in your dumps that is eating a lot of your resources.
Which started the debate on retail=true and debug=true. It seems to me there are a few gotchas still out there and possibly some further digging to what really changes when both these settings are true versus retail=true and debug=false.
If debug=true you will NOT get time-outs no matter what the setting is for retail in machine.config. That is because the ASK.NET Deadlock detection mechanism literally looks in web.config and if denug=true it theoretically runs for an infinite amount of time so that you have more than enough time to attach a debugger.
Thank you for your posts. I would not have solved my application's memory issues without this particular post and all the debugging tips throughout your blog. I've really learned a lot!
how do i run a a nifty command in sos.dll ?
Our application uses one dll called ISDBLayer.dll. This is one of the class library projects in a DLL Solution. When I compile it the solution in Debug mode, and deploy it to production its size is abot 48 kb and the app works fine. Whereas, when I compile it in Release mode (where the size becomes 44 kb), and then deploy it to prod, the application gives an error. This is baffling to me. Any ideas why would such a thing happen just because of a change in the debug/release option.
This article is a great read. We have a web application project and use custom http handlers. It has a File import feature to import data from various sources. My question is does this affect custom http handlers? Are the custom http handlers compiled at runtime?