Internet applications using ASP.Net recently had an update (MS11-100) pushed out of band back in December that set a limit of 1000 items to be accepted by a web form. While this is not the only limit imposed, it is the one that some applications are hitting. If you exceed this value, an exception is thrown that looks like the one below:
System.Web.HttpException:The URL-encoded form data is not valid. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object. at System.Web.HttpValueCollection.ThrowIfMaxHttpCollectionKeysExceeded() at System.Web.HttpValueCollection.FillFromEncodedBytes(Byte bytes, Encoding encoding) at System.Web.HttpRequest.FillInFormCollection() --- End of inner exception stack trace --- at System.Web.HttpRequest.FillInFormCollection() at System.Web.HttpRequest.get_Form()
The key here is ThrowIfMaxHttpCollectionKeysExceeded. If that is in your stack trace, you know that you have exceeded the value the patch imposes.
To keep this exception from being thrown, you need to change the value of aspnet:MaxHttpCollectionKeys in the web.config of your application to a value that is as high as the highest allowed count of keys in your hash table.
To help you figure out what value you should use in production, I have created a HTTP Module that counts the number of keys in that collection for each request and stores the high water mark in the event log once the module is unloaded. I have included a sample that shows how this works.
For you to test this in your application, you need to build the MS11-1000Helper.DLL and place it in your web application’s Bin directory. Once you have done this, you need to add the following to your web.config file to enable the HTTP Module:
<add type="MS11_100_Helper.CounterModule, MS11-100-Helper" name="CounterModule" />
Additionally, you need to add the following to your appSettings:
This is a super high number to use as a starting point in TESTING your application.
You should never have a number like this in your PRODUCTION web.confg.
Instead, you should use this HTTP module (MS11-100-Helper.dll) to help you determine
the appropriate number to use for your setting in PRODUCTION.
<add key="aspnet:MaxHttpCollectionKeys" value="9999" />
Then you should TEST your application. If you are dynamically generating Forms items in your application, you need to test with the highest amount of generated UI as you expect to ever see while your application is being run.
After you have tested the above scenario, you need to close the browser and terminate the web server (stop the development server or issue an IISRESET). Once you have done this, the high water mark should be written to the event log “Applications and Services Logs/CounterSourceLog” and you should have an Information entry that looks like this:
Largest Value Seen = X
Where X is some number.
You can then use this number to help you set the value of aspnet:MaxHttpCollectionKeys in production.
I would personally set the number to be a little higher than the value here, but that depends on the application. If you don’t have a lot of dynamically generated forms controls, then the number is probably pretty close to what you can use for your deployed application. You might want to add just a few more to the value just in case you have a last minute tweak that puts you over the limit.
If you have pages that can dynamically generate a ton of controls without limit (indicating a bad design), then this isn't going to help you much.
Just be sure to remove the module and 9999 limit from your production code!!!
There are a couple of additional caveats with this as far as being the end all to this issue.
First, realize that MS11-100 is a workaround to help limit a DOS attack. The final fix will be to randomize the hash tables used in the ASP.Net will be in a change that comes later. Once that change is available, this workaround may not be needed.
Second, this helper only counts the keys in the Form collection. If you are running into an issue where you are seeing an exception thrown for Json, then this code will not help you.
Third, if the high water mark is less than 1000, I wouldn't recommend changing the default app setting of aspnet:MMaxHttpCollectionKeys. A DOS attacker will probably be hitting numbers much higher than this to cause the issue to occur, which is why we choose the number to begin with.
Fourth, anytime you make an application change, you should reuse this tool in TEST to help you figure out the value of the setting in production.
Special thanks to Wyn Lewis-Bevan for the idea on developing this helper!
So 1000 form fields is below any type of DoS threashold.
What it too large a value?
Is 3000 form fields "safe"?
The value limits the impact a DoS attack would have on your website, it does not eliminate it. Too large of a value would be some arbitraty number like X thousand that you just picked out of the air. That is why I created this HTTP module; to give people a better idea of what value they should place in their configuration.
Proper load testing can give you some idea on the amount of CPU generated from high value settings and is dependant on your hardware and farm configurations.
Paul, thanks for the reply. I'm not picking a value out of the air - I know the value.
I'm trying to decide whether to write code or edit the web.config :)
Here is why I am asking the question.
MS says 1000 form fields is the default limit which implies it is "safe". But you've given us a way to increase that limit -- how do we know what is "safe" to increase that limit too?
In other words, when do we move from "safe" to "unsafe" in respect to values for aspnet:MaxHttpCollectionKey?
btw - maybe MS is fixing the hashing function and we won't have to worry about this "work around"?
Sam, Safe and unsafe can only be determined by testing the application. 1000 was choosen because a high percentage of internet facing sites (those that would be subject to this attack) have values below this limit and the limit could be imposed by the update. Some sites have interfaces that require a value greater than this for their applications to run correctly. This tool helps those sites determine what the correct number should be for their site.
Prior to the patch, there were no limits and a specifically crafted response with many keys in it could cause the CPU to spike to 100% for long periods of time causing a DoS. There are no implied or intended terms for safe or unsafe. We will be working on fixing the hashing funciton in the future.
Thanks a lot. It works fine :)
Paul, you may want to point out what web.config files need to be changed for this to work in a CRM environment. We found that changing the config in ...\ReportManager on the application server and in ...\CRMWeb on the front end server did the trick.
The real pain here wasn't having to add that key to the web.config, but rather it was determining what was going on in the first place. The error messages were pretty useless and we ended up having to use log files and Fiddler to get us pointed in the right direction. It was a good week of our time to finally determine and resolve the true problem.
Thanks Kevin. You're correct in that this can happen for applications like CRM as well. Any ASP.Net application that takes this update may have this exception and the module can be used to assist in changing the setting on those systems as well.
Is the key " aspnet:MaxHttpCollectionKeys" suitable for application run on .Net 1.1?
The article states that MS11-100 fixes .Net 1.1 SP1, so that setting should be available, but I have not tested it on that version. See technet.microsoft.com/.../MS11-100
But how to build the MS11-1000Helper.DLL
SuKe - using the source code above, you can build the DLL using Visual C# Express 2010 available for free from here: www.microsoft.com/.../visual-csharp-express
Hi,Paul,Thank you very much for your reply.But I hve two problems: first,It always has a problem about config file when I built the source code above with VS2005;second ,where is the 'Applications and Services Logs/CounterSourceLog'?So can you give me a suggest what i should do next
Hi Suke. The source code was built with VS 2010, I have no idea what will occur when you try and use VS 2005 to build this solution as I did not target that platform. Please us the compiler from the link above. The 'Applications and Services Logs/CounterSourceLog' is a available in the Event Viewer. Expand the 'Applications and Services' folder under the Event Viewer and select the CounterSourceLog to see your results. This was tested under the latest releases of Windows Server 2008 R2 and Windows 7. You can always modify the source code to change the behavior to write to a text file if you want, you do not need to log to the event log.
in IIS 7 when this value at server level...it doesn't propagate it to individual applications, even though IIS 7 claims on individual site/app that the setting is inherited.
ND - I haven't tried this, but my guess is that you will need to add the MS11-1000Helper.DLL to the GAC instead of placing it in your web application’s Bin directory.