The "Server too busy" error is a common one that causes a lot of confusion when related to WCF. It is possible for WCF to respond with a server too busy fault when a quota is exceeded in the security layer. However you wouldn't get this when you hit a WCF throttle limit. Server too busy typically comes from ASP.Net.
As a case in point, I recently looked into an issue where a user was running a web-hosted WCF service in a performance test. The test was to push 10,000 simultaneous requests onto the server. They hit the server too busy problem at around 5,000 requests. To diagnose the issue, I asked for a dump file. When examining the dump file, the first thing I checked was the WCF throttles. For information on how to do that, check out this post: http://blogs.msdn.com/b/dmetzgar/archive/2011/02/01/checking-a-dump-file-for-wcf-throttles.aspx
All the counts I saw were 0 so we didn't have anything laying around in the WCF layer. The next command I used was !threads to see how many threads were in the process. It turned out there were only 147 threads and only a handful were IO completion threads (the kind WCF uses). This would generally point to ASP.Net being the culprit. Just to be sure I opened a log file like this: .logopen ThreadStacks.log and then ran ~*k to dump the stacks for every thread. I then closed the log with .logclose. Using notepad, I opened the log file and searched for ServiceModel. If there are any stacks with ServiceModel in them, then WCF is involved. However, the find did not turn up a match.
Usually one of the things I revert to is to look for anything with Queue in the name in the heap. So I ran !dumpheap -type Queue -stat and found an interesting object called System.Web.RequestQueue. There was only one instance of this class and it looks like this:
0:018> !do 02954c74
Size: 64(0x40) bytes
MT Field Offset Type VT Attr Value Name
6de72978 4001082 14 System.Int32 1 instance 352 _minExternFreeThreads
6de72978 4001083 18 System.Int32 1 instance 304 _minLocalFreeThreads
6de72978 4001084 1c System.Int32 1 instance 5000 _queueLimit
6de6aedc 4001085 2c System.TimeSpan 1 instance 02954ca0 _clientConnectedTime
6de7662c 4001086 28 System.Boolean 1 instance 0 _iis6
6de69f24 4001087 4 ...Collections.Queue 0 instance 02954cb4 _localQueue
6de69f24 4001088 8 ...Collections.Queue 0 instance 02954cd8 _externQueue
6de72978 4001089 20 System.Int32 1 instance 5000 _count
6de627b8 400108a c ...ding.WaitCallback 0 instance 02954d8c _workItemCallback
6de72978 400108b 24 System.Int32 1 instance 0 _workItemCount
6de7662c 400108c 29 System.Boolean 1 instance 0 _draining
6de6aedc 400108d 34 System.TimeSpan 1 instance 02954ca8 _timerPeriod
6de6ab08 400108e 10 ...m.Threading.Timer 0 instance 02954dcc _timer
As you can see, there is a _queueLimit of 5000 and the current _count is 5000. That would make sense why we're getting the server too busy exception. Going on this, I found this TechNet article: http://technet.microsoft.com/en-us/library/dd425294(office.13).aspx
The request queue limit is therefore something that can be controlled by the <processModel> tag in the machine.config. Note that when you change something in processModel, you need to turn autoConfig off. The customer indicated that in addition to changing the requestQueueLimit they also had to run the following:
appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:10000
Note that this is for IIS7. IIS6 is set up differently and you should be able to get more information from the TechNet article linked above.