If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

Case Study: ASP.NET Deadlock calling WebServices

Case Study: ASP.NET Deadlock calling WebServices

Rate This
  • Comments 15

Lately, no matter where you turn there are LINQ presentations and Silverlight demos.  A couple of years back the hot new stuff was webservices and XML.  

All these technologies are really cool and serve their purpose very well for the right applications, but as with anything you always have to weigh the pros against the cons of using a technology for a specific purpose, otherwise you will end up with an app like "The Beast" featured on one of the best geek humor sites around "The Daily WTF"

This post is about some very bad things that can happen if you over-use web services (which is a pretty common issue that we see in support).  Web services are great if you need to be able to access functionality remotely, from multiple platforms.  If you have the option to access the functionality locally however, you are way better off writing a component/class with the functionality.  And finally if you need to access it both remotely and locally, my recommendation would be to write it as a component for local use and then wrap it in WebMethods for remote use. 

Rather than speaking about a specific case here, because all of these types of cases look pretty much the same, I will talk about a common architecture decision and show how bad it can get with a super-simplified demo.

Architecture and thoughts:

I think must of us have been indoctrinated since college with the multi-tier approach.  One tier for the presentation layer, one for the business layer and one for the data layer.   On larger sites we would have a web farm with multiple servers serving the same content and loadbalancing the work in between them.   So far so good (even though I personally don't really fancy multi-tiering for the sake of multi-tiering, but that's a different topic all togheter).

Now, naturally in a web scenario, the perfect choice for communicating between the tiers would be a webservice, so we would end up with something like this...

                  

Furthermore these three layers exist in three different web applications load balanced over the servers on the web farm so at any given time Default.aspx may be calling Busiesslayer\service.asmx on any of the servers on the webfarm.  This will in turn call Datalayer\service.asmx which may also at any given time be served from any of the servers on the webfarm.    

Problem description

When the load is high the requests to the website take a very long time to execute, sometimes they don't return at all and an IISReset is necessary to bring the servers back on-line again.

Reproducing the issue

In order to vastly reduce complexity i have only one machine with 3 web applications, in the same application pool, and the web services do nothing but pass a string "hello world" back and forth.

Presentation\Default.aspx simply calls Service.HelloWorld from Businesslayer and this in turn calls Service.HelloWorld from Datalayer.  HelloWorld in the datalayer is just the standard web method that returns "hello world", and this is then passed back all the way back to the presentation layer.

I then use a tool from the IIS Resource Kit called tinyget.  Tinyget is a simple stress tool that allows you to make http requests to a given URI. Just like with webservices there is a place and a time for Tinyget.  It is great for simple stress testing, but i wouldn't use it to do proper stress testing of any major site:)

The command line i use looks like this

tinyget -srv:localhost -uri:/Presentation/Default.aspx -threads:30 -loop:50  

 

This means that tinyget will use 30 concurrent threads to make 50 GET requests to http:\\localhost\Presentation\Default.aspx

Btw, tinyget has a lot more interesting features like scripts etc. that may be interesting to check out.

 

Troubleshooting/Debugging the issue:

I have written before about blocking webservice requests in 1.1 where we by default only have 2 outgoing connections so we can only make two webservice requests at once.  As I mentioned back then the defaults have changed in 2.0 to avoid this type of blocking and the defaults are now 12 connections / app domain / URI, and 12 worker threads available for ASP.NET requests per logical CPU (total = maxworker*CPU-minFreeThreads) . 

The interesting piece here is that in this particular case, the web layer, the middle tier and the data tier live on the same machine, which in theory should make no difference, it should just make things faster since we don't have to cross machine boundaries, and we have an unlimited amount of connections for in-process requests.   

After tinyget had been running for a while I took a dump with adplus -hang -pn w3wp.exe and then I ran my little automated .net hang analysis tool. 

The following threads are waiting in a WaitOne: 
	21 24 27 28 36 40 41 42 43 46 58 64 68 69 

...

The following threads are waiting in a Socket.Receive: 
	14 22 25 26 29 30 31 32 33 34 35 37 38 39 45 47 48 49 50 51 53 54 55 56 57 59 60 61 62 63 65 66 67 70 

So I have 34 threads waiting in a Socket.Receive and 14 threads waiting in a WaitOne.

If I run my DumpRequests script I find that I have 38 requests executing /Presentation/Default.aspx and 10 requests executing /Businesslayer/Service.asmx but no requests for DataLayer/Service.asmx, so for some reason the calls never make it through to the DataLayer web service.

As a sidenote,  we can also see that the Timeout for Default.aspx is extremely high which is an indication of that debug is set to true in the Presentation application, but for this particular demo that is of minor importance.

HttpContext	StartTime		TimeOut (sec)	HttpResponse	Completed  	ReturnCode	HttpRequest	RequestType	URL+QueryString
===================================================================================================================================================================
031bb1e4	Dec 17 14:26:35.670	30000000		031bb34c		No		200		031bb2a0	GET		/Presentation/Default.aspx
031e5014	Dec 17 14:26:35.857	30000000		031e517c		No		200		031e50d0	GET		/Presentation/Default.aspx
031f04b8	Dec 17 14:26:36.857	30000000		031f0620		No		200		031f0574	GET		/Presentation/Default.aspx
031fc498	Dec 17 14:26:37.357	30000000		031fc600		No		200		031fc554	GET		/Presentation/Default.aspx
03208498	Dec 17 14:26:37.857	30000000		03208600		No		200		03208554	GET		/Presentation/Default.aspx
03214498	Dec 17 14:26:38.857	30000000		03214600		No		200		03214554	GET		/Presentation/Default.aspx
03220498	Dec 17 14:26:42.857	30000000		03220600		No		200		03220554	GET		/Presentation/Default.aspx
0322c498	Dec 17 14:26:43.857	30000000		0322c600		No		200		0322c554	GET		/Presentation/Default.aspx
03238498	Dec 17 14:27:21.639	30000000		03238600		No		200		03238554	GET		/Presentation/Default.aspx
0711ca78	Dec 17 14:26:35.670	30000000		0711cbe0		No		200		0711cb34	GET		/Presentation/Default.aspx
0714a4b8	Dec 17 14:26:36.357	30000000		0714a620		No		200		0714a574	GET		/Presentation/Default.aspx
07156498	Dec 17 14:26:38.357	30000000		07156600		No		200		07156554	GET		/Presentation/Default.aspx
07162498	Dec 17 14:26:39.857	30000000		07162600		No		200		07162554	GET		/Presentation/Default.aspx
0716e498	Dec 17 14:26:40.857	30000000		0716e600		No		200		0716e554	GET		/Presentation/Default.aspx
0717a498	Dec 17 14:26:41.857	30000000		0717a600		No		200		0717a554	GET		/Presentation/Default.aspx
07186498	Dec 17 14:27:20.639	30000000		07186600		No		200		07186554	GET		/Presentation/Default.aspx
0719163c	Dec 17 14:27:22.639	110		071917a4		No		200		071916f8	POST		/Businesslayer/service.asmx
071a0640	Dec 17 14:27:23.639	110		071a07a8		No		200		071a06fc	POST		/Businesslayer/service.asmx
...

We have 3 interesting types of stacks

24 requests are waiting on a socket call stemming from Default.aspx so these are waiting for results from BusinessLayer/Service.asmx

  70  Id: 1694.270 Suspend: 1 Teb: ffe7c000 Unfrozen
ChildEBP RetAddr  Args to Child              
110ee2d0 7db40563 00000e84 00000001 110ee2f8 ntdll!ZwWaitForSingleObject+0x15
110ee30c 7db51016 00000e84 00000f74 00000000 mswsock!SockWaitForSingleObject+0x19d
110ee384 71c02fee 00000f74 110ee3bc 00000001 mswsock!WSPRecv+0x203
110ee3cc 025da1c3 00000f74 0745218c 00001000 WS2_32!recv+0x83
110ee3f4 7a5ff523 02c33a88 07452184 00000000 CLRStub[StubLinkStub]@25da1c3
074532c8 7a5ff401 00000000 00000000 00000000 System_ni!System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)+0xd3
110ee48c 7a5d3798 00000000 00000000 00000000 System_ni!System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)+0x21
110ee48c 7a5ab17a 00000000 00000000 00000000 System_ni!System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)+0x78
110ee4f8 7a5b01d3 00000000 00000000 00000000 System_ni!System.Net.PooledStream.Read(Byte[], Int32, Int32)+0x1a
110ee4f8 7a5b0089 00000000 00000000 00000000 System_ni!System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean)+0x133
00000001 7a5b52fb 00000000 00000000 00000000 System_ni!System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean)+0x79
110ee550 7a582278 00000000 00000000 00000000 System_ni!System.Net.ConnectStream.PollAndRead(Boolean)+0x1b
110ee550 7a580be5 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.EndWriteHeaders(Boolean)+0x128
110ee5b0 7a5b52bc 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus, System.Net.ConnectStream, Boolean)+0x15
110ee5b0 7a5820d2 00000000 00000000 00000000 System_ni!System.Net.ConnectStream.WriteHeaders(Boolean)+0x2dc
110ee5dc 7a57f61b 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.EndSubmitRequest()+0xa2
110ee62c 7a57f842 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.CheckDeferredCallDone(System.Net.ConnectStream)+0x4b
110ee62c 65cfc18e 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.GetResponse()+0x212
110ee660 65cfcbd5 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest)+0xfe
110ee6a4 65d0b591 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest)+0x5
110ee6a4 0f0109dd 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])+0xad
00000000 0f0104fa 00000000 00000000 00000000 Presentation!Presentation.prather.Service.BusinessLayerHelloWorld()+0x35 
110ee914 66f12980 00000000 00000000 00000000 Presentation!Presentation._Default.Page_Load(System.Object, System.EventArgs)+0x42 110ee914 6628efd2 00000000 00000000 00000000 System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x10
110ee914 6613cb04 00000000 00000000 00000000 System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x22
110ee914 6613cb50 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x64
110ee914 6614e12d 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x30
110ee914 6614d8c3 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x59d
110ee94c 6614d80f 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0x67
110ee988 6614d72f 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57
110ee9a4 6614d6c2 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13
110ee9a4 0f0101b6 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32
110ee9e4 65fe6bfb 00000000 00000000 00000000 App_Web_ssvdcbij!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x1e

10 requests are in BusinessLayer/Service.asmx waiting for results back from DataLayer/Service.asmx

  14  Id: 1694.1674 Suspend: 1 Teb: fff88000 Unfrozen
ChildEBP RetAddr  Args to Child              
0252e934 7db40563 00000524 00000001 0252e95c ntdll!ZwWaitForSingleObject+0x15
0252e970 7db51016 00000524 00000e00 00000000 mswsock!SockWaitForSingleObject+0x19d
0252e9e8 71c02fee 00000e00 0252ea20 00000001 mswsock!WSPRecv+0x203
0252ea30 025da1c3 00000e00 031d4b2c 00001000 WS2_32!recv+0x83
0252ea58 7a5ff523 0f3638f0 031d4b24 00000000 CLRStub[StubLinkStub]@25da1c3
031d5c68 7a5ff401 00000000 00000000 00000000 System_ni!System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)+0xd3
0252eaf0 7a5d3798 00000000 00000000 00000000 System_ni!System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)+0x21
0252eaf0 7a5ab17a 00000000 00000000 00000000 System_ni!System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)+0x78
0252eb5c 7a5b01d3 00000000 00000000 00000000 System_ni!System.Net.PooledStream.Read(Byte[], Int32, Int32)+0x1a
0252eb5c 7a5b0089 00000000 00000000 00000000 System_ni!System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean)+0x133
00000001 7a5b52fb 00000000 00000000 00000000 System_ni!System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean)+0x79
0252ebb4 7a582278 00000000 00000000 00000000 System_ni!System.Net.ConnectStream.PollAndRead(Boolean)+0x1b
0252ebb4 7a580be5 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.EndWriteHeaders(Boolean)+0x128
0252ec14 7a5b52bc 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus, System.Net.ConnectStream, Boolean)+0x15
0252ec14 7a5820d2 00000000 00000000 00000000 System_ni!System.Net.ConnectStream.WriteHeaders(Boolean)+0x2dc
0252ec40 7a57f61b 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.EndSubmitRequest()+0xa2
0252ec90 7a57f842 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.CheckDeferredCallDone(System.Net.ConnectStream)+0x4b
0252ec90 65cfc18e 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.GetResponse()+0x212
0252ecc4 65cfcbd5 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest)+0xfe
0252ed08 65d0b591 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest)+0x5
0252ed08 0f520578 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])+0xad
0252ed30 0f5203fd 00000000 00000000 00000000 App_WebReferences_wy0uskl5!prather.Service.DataLayerHelloWorld()+0x20
0252ed30 79e7c74b 00000000 00000000 00000000 App_Code_lwzoeb2p!Service.BusinessLayerHelloWorld()+0x1d
0252ed30 79e7c6cc 0252ee00 00000000 0252edd0 mscorwks!CallDescrWorker+0x33
0252edb0 79e7c8e1 0252ee00 00000000 0252edd0 mscorwks!CallDescrWorkerWithHandler+0xa3
0252eeec 79e7c783 0f36c148 0252f044 0252ef40 mscorwks!MethodDesc::CallDescr+0x19c
0252ef08 79e7c90d 0f36c148 0252f044 0252ef40 mscorwks!MethodDesc::CallTargetWorker+0x1f
0252ef1c 79f321b6 0252ef40 0252f0fc 79f91478 mscorwks!MethodDescCallSite::Call_RetArgSlot+0x18
0252f108 79f31eaf 0f3674f0 031d3cc8 00000000 mscorwks!InvokeImpl+0x559
0252f1c8 793a44bd 0f36750c 00000086 0252f1e0 mscorwks!RuntimeMethodHandle::InvokeMethodFast+0xbd

and 14 requests are waiting to start off a request to BusinessLayer/Service.asmx

  69  Id: 1694.16a8 Suspend: 1 Teb: ffe7f000 Unfrozen
ChildEBP RetAddr  Args to Child              
1106e29c 7d4e286c 00000001 1106e2e8 00000000 ntdll!NtWaitForMultipleObjects+0x15
1106e344 79ed98fd 00000001 1106e584 00000001 kernel32!WaitForMultipleObjectsEx+0x11a
1106e3ac 79ed9889 00000001 1106e584 00000001 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
1106e3cc 79ed9808 00000001 1106e584 00000001 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
1106e450 79ed96c4 00000001 1106e584 00000001 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
1106e4a0 79fcfc58 00000001 1106e584 00000001 mscorwks!Thread::DoAppropriateWait+0x40
1106e5a4 793b03fe 00000000 00000000 ffffffff mscorwks!WaitHandleNative::CorWaitOneNative+0x156
1106e608 793b0c1b 00000000 00000000 00000000 mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int64, Boolean)+0x2e
1106e608 7a569d4c 00000000 00000000 00000000 mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int32, Boolean)+0x23
1106e608 7a5ad0cf 00000000 00000000 00000000 System_ni!System.Net.LazyAsyncResult.WaitForCompletion(Boolean)+0x5c
1106e64c 7a58c5f2 00000000 00000000 00000000 System_ni!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)+0x2df
1106e680 7a581a51 00000000 00000000 00000000 System_ni!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)+0x82
1106e6b4 7a57ec30 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)+0x141
1106e6e4 65d0b54c 00000000 00000000 00000000 System_ni!System.Net.HttpWebRequest.GetRequestStream()+0x1d0
1106e724 0f0109dd 00000000 00000000 00000000 System_Web_Services_ni!System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])+0x68
00000000 0f0104fa 00000000 00000000 00000000 Presentation!Presentation.prather.Service.BusinessLayerHelloWorld()+0x35 
1106e994 66f12980 00000000 00000000 00000000 Presentation!Presentation._Default.Page_Load(System.Object, System.EventArgs)+0x42 
1106e994 6628efd2 00000000 00000000 00000000 System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x10
1106e994 6613cb04 00000000 00000000 00000000 System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x22
1106e994 6613cb50 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x64
1106e994 6614e12d 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x30
1106e994 6614d8c3 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x59d
1106e9cc 6614d80f 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0x67
1106ea08 6614d72f 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x57
1106ea24 6614d6c2 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)+0x13
1106ea24 0f0101b6 00000000 00000000 00000000 System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0x32
1106ea64 65fe6bfb 00000000 00000000 00000000 App_Web_ssvdcbij!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x1e

 

Ok, so why do the requests never make it through to the Datalayer/service.asmx?  And why are so many threads waiting to initiate a request to BusinessLayer/Service.asmx? That seems to be crucial for the application to respond...  

If we take a look at the ThreadPool at the time of the issue we will find the answer...  

0:059> !threadpool
Work Request in Queue: 0
--------------------------------------
Number of Timers: 19
--------------------------------------
CPU utilization 12%
--------------------------------------
Worker Thread: Total: 48 Running: 48 Idle: 0 MaxLimit: 400 MinLimit: 4
Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 200 MinLimit: 2

The default settings for the threadpool in 2.0 for ASP.NET is that MaxWorkerThreads = 100*#logical CPU = 400 max and the MinFreeThreads is set to 88 per CPU leaving us with  12 threads per CPU to make ASP.NET/HTTP Requests on.  This is a process wide setting so it means that at one single time we can make 48 ASP.NET requests on a dual core HT machine. The reason it is set up like this is so that we won't overload the process with requests, and although 12 is an arbitrary number it usually works out pretty well...

The problem for us is that the requests came in in big bursts so we managed to start off 38 requests for Default.aspx leaving 10 threads for other requests. 

10 of these Default.aspx pages then started up the calls to BusinessLayer/Service.asmx and then we were out of threads so the calls from the other 28 Default.aspx to BusinessLayer/Service.asmx and the calls from the executing BusinessLayer/Service.asmx to DataLayer/Service.asmx will never finish since there are no threads left to service these requests.

In other words we are in a deadlock situation or rather a really long wait that will only end when the BusinessLayer/Service.asmx calls start timing out.

Now, a crafty ASP.NET developer will probably immediately go and up the number of threads available for ASP.NET request, but really that is only a patch since the issue may still occurr given enough concurrent requests.  And if you up it enough to cover the largest burst you can think of, it may result in poor performance because of context switching.

Summary

In summary, calling webservices inside the same application pool is bad practice since it can lead to deadlocks and even if you change the threading parameters so that you will not enter this deadlock it is still bad for performance.

Consider if my simple demo, instead of just returning a string, returned a dataset instead (which would be kind of appropriate, especially comming from the datalayer:)). In that case we would have something like this happening in the process.

 

1. Request for Presentation/Default.aspx calls BusinessLayer/Service.asmx passing in a dataset with items purchased

2. Serialization of the dataset in the Presentation domain

3. Deserialization of the dataset in the BusinessLayer domain

4. Processing of data in BusinessLayer/Service.asmx - BusinessLayer/Service.asmx calls DataLayer/asmx passing in a revised dataset

5. Serialization of the new dataset in the BusinessLayer domain

6. Deserialization of the dataset in the Datalayer domain

7. Processing of the data and gathering of results from the database

8. Datalayer/service.asmx returns a revised dataset to the businesslayer

9. Serialization of dataset in the Datalayer domain

10. Deserialization of the dataset in the BusinessLayer domain

11.  BusinessLayer/service.asmx does some modification and returns the dataset to the presentation layer

12. Serialization of the dataset in the Businesslayer domain

13. Deserialization of the dataset in the Presentation domain

14. Presentation layer presents the data

 

All the items shown in bold are only necessary because we are making cross-domain webservice calls.  If we would use a component/class in the Presentation domain instead to perform the tasks done in the business layer and data layer we would have none of this, and in many cases the overhead for this can greatly overshadow the processing power used for performing the actual tasks.

 

So... after a long rant, I repeat the point I was making in the first statement... if you have the option of processing the data in the original application domain then do so.  In other words, replace all calls to local webservices with component calls...  and wrap them up in webmethods if they need to be used remotely.

 

Anyways, I'm off for holidays now...

So happy new year everyone,

Tess





  • Happy holidays!

    I don't disagree with your ultimate point that you shouldn't do unnecessary inter-process calls, especially with web services.

    But I'm not happy with the "don't do it that way" answer to the fundamental problem of a system locking up under load because the thread pool is out of threads.  I'd like to see a better solution to prevent the lockup in the first place, even when there's a multi-layer web services architecture, because sometimes that kind of architecture is necessary.

  • Hi Alan,

    Well, I'm not really sure what could be done about it.  You as the developer or web admin decide how many threads can take incomming http requests and the service won't really know what calls are going to be made from your http request until the time comes to do them, and at that point all the threads are already used up.  

    It can't kick a request out that is already running on a thread and it can't create new threads cause you told it not to through the configuration, so its a catch-22.

    Do you have any suggestions as to what could be done? I would be interested to hear the ideas...

    If you do have a situation like this where this kind of architecture is neccessary, my suggestion would be to run the different web services in different application pools.  

  • Wouldn't making the web service calls asynchronously take care of things? With asynchronous calls you would end up serializing across the minFreeThreads for the callbacks, but would never be blocking an ASP.NET worker thread waiting for a web service response.

  • .NET: IntroducingNHaml-AnASP.NETMVCViewEngine 8SimpleRulesforDesigning...

  • Hey Tess,

    This was a very interesting read.

    So on a dual core HT machine, a .net process can only handle 48 concurrent requests especially if for every incoming request, the process has to make a http call to some backend service?

    Is this limit only imposed if we use .Net ThreadPool?

  • Alan/Tess: Wouldn't asynchrony go a long way in helping avoid lockups in this scenario? If the HTTP WS handlers were IHttpAsyncHandlers (or the ASP pages had @Async enabled) the worker threads would be returned to the pool immediately, using a completion port thread only when necessary to perform the actual data leveraging from the remote system.

    Tess: "the defaults have changed in 2.0 to avoid this type of blocking and the defaults are now 12 connections / app domain / URI" - I thought it was still 2 (assuming we're talking about the maxConnection in machine.config). What documentation should I look at here?

  • Tess,

    As always, a *fantastic* read, sparking my interest, and making me hurry off to learn more about the topic(because I'm sure we have issues that could benefit from understanding these constraints).   Please keep these great entries coming!

    Happy New Year to you!  Hope you have a great 2008!

  • Thank you much, a happy new year to all of you...

    I'm working on some new posts but things are crazy busy at work after new years so it might be a few days.

    Thanks

    Tess

  • This sounds like a design defect of System.Net.* and System.Web.* to me...

  • Hi Tess,

    I'm working with an application (Winforms 1.1) that uses the HttpWebRequest in order to call one HttpHander (ASP.Net 1.1).

    In some scenario, the application executes "7" requests concurrently, using differents threads.

    I've configured the "maxconnections" attribute and I've checked that the application has used "7" different connections (one per each concurrent request).

    My problem is that some of these "parallel" requests spends much more time than the usual.

    There is some type of contention at the connection. If I execute the requests serialized, the time of some of them is minor than if I execute them in parallel.

    I've checked the HttpHandler's times and the ASP.NET Application Queue and they are ok. There isn't requests queued and the HttpHandler executes very quickly.

    The time is lost trying to get the response and reading the network stream.

    Do you have any idea?

    Thanks in advance

    Paul

  • Did you also change the thread configuration settings to match so that you have enough free threads for callbacks from the webservice?

  • Hi Tess,

    I'm a big fan of yours.

    We're facing an issue that very relevant to this post.

    We have high end application that cannot establish more than ~70 outbound concurrent connections. (we're using httpwebrequest directly).

    Hardware is no limit, CPU and memory running in very low level.

    It deployed on qaud core machine with 4Gb memory, clr 3.5.

    I tried to tweak the machine.config maxconnection, io \ worker thread pool size and also the recommendations regarding http.sys and tcpip.sys with no help.

    We also tried to run similar app written in c++ which use winhttp, wininet directly and we got the same results.

    Does windows limit the connections elsewhere?

    Thanks in advance,

    Ronen.

  • I dont think there are any limits around 70,  try getting a hang dump and see what it is waiting for  

  • It's maybe not a hard limit, but when I increase the simultaneous connections, I can see that the httpwebrequest response time increase dramatically.

    Does it make sense?

    Thanks,

    Ronen.

Page 1 of 1 (15 items)
Leave a Comment
  • Please add 2 and 3 and type the answer here:
  • Post