AsiaTech: Microsoft APGC Internet Developer Support Team

We focus on various troubleshooting plan and solution on IIS web platform and distributed applications

NTLM authenticated web site load slowness over Internet when browsing via Proxy

NTLM authenticated web site load slowness over Internet when browsing via Proxy

  • Comments 1

 

Browsing public web site which require NTLM authentication in IE, it loads quickly when access directly, but suffering dramatically slow when access from corp intranet domain.  We want to find out why the slowness only happens in corp intranet domain.

 

Environment

==========

There are upstreams and downstreams proxies setup in Corp domain.

 

Symptom

==========

When we access from external, authentication credential pops up in one second, while it takes at least 10 seconds in company’s intranet domain.

Also, page loading period are much longer than direct browsing.

When IE load web page, there is no un-response (hang).

 

Troubleshooting

==========

Based on network monitor trace, it shows that there are delay gaps between HTTP requests. Moreover, there are several gaps during page loading.

 

12293         8:42:47 AM 6/22/2012                   273.3723386                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Ok, URL: http://ntlm.slow.com/_layouts/images/slow.SPS2010.Internet/lmenu/lmen                 {HTTP:260, TCP:255, IPv4:66}

12294         8:42:47 AM 6/22/2012                   273.3723386                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Ok, URL: http://ntlm.slow.com/_layouts/images/slow.SPS2010.Internet/lmenu/lmen                 {HTTP:68, TCP:67, IPv4:66}

12327         8:42:47 AM 6/22/2012                   273.4192136                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Ok, URL: http://ntlm.slow.com/_layouts/images/slow.SPS2010.Internet/lmenu/lmen                 {HTTP:261, TCP:256, IPv4:66}

12342         8:42:47 AM 6/22/2012                   273.4660886                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Ok, URL: http://ntlm.slow.com/_layouts/images/slow.SPS2010.Internet/lmenu/lmen                 {HTTP:258, TCP:253, IPv4:66}

12365         8:42:47 AM 6/22/2012                   274.1692136                                         CLIENTMACHINE          internalDNSServer     DNS                        DNS:QueryId = 0xC1C6, QUERY (Standard query), Query  for ntlm.slow.com of type Host Addr on class Internet                        {DNS:483, UDP:482, IPv4:72}

12393         8:42:48 AM 6/22/2012                   275.1692136                                         CLIENTMACHINE          internalDNSServer     DNS                        DNS:QueryId = 0xC1C6, QUERY (Standard query), Query  for ntlm.slow.com of type Host Addr on class Internet                        {DNS:483, UDP:482, IPv4:72}

12428         8:42:50 AM 6/22/2012                   277.1692136                                         CLIENTMACHINE          internalDNSServer     DNS                        DNS:QueryId = 0xC1C6, QUERY (Standard query), Query  for ntlm.slow.com of type Host Addr on class Internet                        {DNS:483, UDP:482, IPv4:72}

… …

12572         8:42:58 AM 6/22/2012                   285.0754636                                         CLIENTMACHINE          internalDNSServer     DNS                        DNS:QueryId = 0xBA96, QUERY (Standard query), Query  for ntlm.slow.com.casinostg.internal of type Host Addr on class Internet         {DNS:494, UDP:493, IPv4:72}

12573         8:42:58 AM 6/22/2012                   285.0754636                                         internalDNSServer     CLIENTMACHINE          DNS                        DNS:QueryId = 0xBA96, QUERY (Standard query), Response - Name Error                         {DNS:494, UDP:493, IPv4:72}

… …

13285         8:43:18 AM 6/22/2012                   304.4045646                                         internalDNSServer     CLIENTMACHINE          DNS                        DNS:QueryId = 0xB882, QUERY (Standard query), Response - Server failure                    {DNS:514, UDP:513, IPv4:72}

13356         8:43:20 AM 6/22/2012                   307.0764396                                         internalDNSServer     CLIENTMACHINE          DNS                        DNS:QueryId = 0xB882, QUERY (Standard query), Response - Server failure                    {DNS:516, UDP:515, IPv4:69}

… …

13617         8:43:31 AM 6/22/2012                   318.0764396                                         CLIENTMACHINE          internalDNSServer     DNS                        DNS:QueryId = 0xDF5F, QUERY (Standard query), Query  for ntlm.slow.com.casinotest.internal of type Host Addr on class Internet         {DNS:547, UDP:546, IPv4:72}

13618         8:43:31 AM 6/22/2012                   318.0764396                                         internalDNSServer     CLIENTMACHINE          DNS                        DNS:QueryId = 0xDF5F, QUERY (Standard query), Response - Name Error                         {DNS:547, UDP:546, IPv4:72}

13619         8:43:31 AM 6/22/2012                   318.0764396                 iexplore.exe                   CLIENTMACHINE          proxyserver                        HTTP             HTTP:Request, GET http://ntlm.slow.com/Lists/BoxNavigationLinksEntertainment/Box-Navigation_JUP04894_, Using NTLM Authorization   {HTTP:481, TCP:479, IPv4:66}

13620         8:43:31 AM 6/22/2012                   318.0764396                 iexplore.exe                   CLIENTMACHINE          proxyserver                        HTTP             HTTP:Request, GET http://ntlm.slow.com/Lists/FlashMediaLibrary/Config/FlashBanner-Entertainment-Level, Query:id=d85ec795-5d6e-40f4-a5ca-b468d0c98e5c        {HTTP:260, TCP:255, IPv4:66}

13621         8:43:31 AM 6/22/2012                   318.0764396                 iexplore.exe                   CLIENTMACHINE          proxyserver                        HTTP             HTTP:Request, GET http://ntlm.slow.com/Lists/FlashMediaLibrary/Config/FlashIndex-Sub.xml, Query:id=47be0411-0304-46e1-af5a-9ee32e2109fa, Using NTLM Authorization         {HTTP:259, TCP:254, IPv4:66}

13625         8:43:31 AM 6/22/2012                   318.1858146                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Unauthorized, URL: http://ntlm.slow.com/Lists/FlashMediaLibrary/Config/FlashIndex-Sub.xml                       {HTTP:259, TCP:254, IPv4:66}

13626         8:43:31 AM 6/22/2012                   318.1858146                 iexplore.exe                   CLIENTMACHINE          proxyserver                        HTTP             HTTP:Request, GET http://ntlm.slow.com/Lists/FlashMediaLibrary/Config/FlashIndex-Sub.xml, Query:id=47be0411-0304-46e1-af5a-9ee32e2109fa, Using NTLM Authorization         {HTTP:259, TCP:254, IPv4:66}

13627         8:43:31 AM 6/22/2012                   318.2014396                 iexplore.exe                   proxyserver                     CLIENTMACHINE                          HTTP             HTTP:Response, HTTP/1.1, Status: Ok, URL: http://ntlm.slow.com/Lists/FlashMediaLibrary/Config/FlashBanner-Entertainment-Level              {HTTP:260, TCP:255, IPv4:66}

 

Between these gaps, DNS query with response Name Error and Server Failure logged.

Such list of DNS query should be the reason why HTTP request delay happens. But:

 

1. Why IE will do kind of DNS query?

2. If DNS query response are Name Error and Sever Failure how web page can be displayed?

 

For 1st question, those additional DNS requests because of NTLM reflection. This as a security enhancement and has first been implemented in MS09-054. For security consideration, if the website request NTLM authentication automatically, we will use DNS check to see if it’s a fake server.

 

For 2nd question, we got answer from proxy vendor: the external DNS resolution is done by proxy server on behalf of the client.

Normally, if web sites are used in Intranet or trusted domain, such security feature doesn’t cause any problem. But if we put it on Internet which need proxy access, it might have the problem.

 Such delay problem doesn't usually happens because the DNS query will normally return a "non-exist domain" in a short time. In this case, this is a "Server Failure" with long time delay from DNS. The error “Server Failure” means it’s can’t connect to the Up-Level DNS server due to network error. Internal DNS may send the request to up-level DNS and waited for a long time, finally, the result is negative. 

In this case, there is no forwarder to external DNS servers on internal DNS server.    

 Solution

========== 

a.    Create a hosts file that contains dummy entries as required for any sites that display this issue

This would result in fast DNS resolution and minimize network traffic – in so far as no external DNS queries/responses would be generated. However, this may not be a manageable solution – any change would require a new HOSTS file to be propagated across all clients

b.    Add similar dummy entries to accessible internal DNS servers that you manage

This would result in DNS resolution, with a small network overhead for the queries and responses. This may be a more manageable solution – any changes would simply require an update to the appropriate DNS server infrastructure

c.    Possible solution in DNS: Remove all global forwarders and root hints on internal DNS server.

Risk of Action: Make sure that we do not need external name resolution from internal DNS server. There will be no impact at all if we confirm that. Also, we can add the root hints back when we need to.

d.    Proxy server solution: Point Internal DNS to an upstream proxy to do external DNS resolution. 

It needs to have the DNS service on the upstream proxy to be intercepted, for more details, consult proxy server vendor.

 

More Information

===========

We verified the delay reason from memory dump captured during http requests gap.

It was blocked at thread 23. This problem is related to a lock when using gethostbyname after calling wininet!AUTHCTX::GenerateFQDNAndSPN

Client machine are unable to resolve the host ntlm.slow.com.

  23  Id: 954.d9c Suspend: 0 Teb: 7ffdd000 Unfrozen

ChildEBP RetAddr 

0741efdc 7c90daea ntdll!KiFastSystemCallRet

0741efe0 77e7c672 ntdll!NtRequestWaitReplyPort+0xc

0741f02c 77e7a80e rpcrt4!LRPC_CCALL::SendReceive+0x228

0741f038 77e7a83f rpcrt4!I_RpcSendReceive+0x24

0741f04c 77ef5675 rpcrt4!NdrSendReceive+0x2b

0741f430 76f23627 rpcrt4!NdrClientCall2+0x222

0741f444 76f235bb dnsapi!R_ResolverQuery+0x1b

0741f4a0 71a526c6 dnsapi!DnsQuery_W+0x14f

0741f4d4 71a5266f mswsock!HostentBlob_Query+0x29

0741f500 71a51b0a mswsock!Rnr_DoDnsLookup+0x7d

0741f948 71ab32b0 mswsock!NSPLookupServiceNext+0x533

0741f960 71ab3290 ws2_32!NSPROVIDER::NSPLookupServiceNext+0x17

0741f97c 71ab325a ws2_32!NSPROVIDERSTATE::LookupServiceNext+0x1c

0741f9a8 71ab31f8 ws2_32!NSQUERY::LookupServiceNext+0xae

0741f9c8 71ab5af0 ws2_32!WSALookupServiceNextW+0x78

0741f9ec 71ab55e8 ws2_32!WSALookupServiceNextA+0x63

0741fa18 71ab53e7 ws2_32!getxyDataEnt+0xa1

0741fc54 3d9b0493 ws2_32!gethostbyname(char * name = 0x07ad0690 "ntlm.slow.com")+0xb4

0741fcbc 3d940814 wininet!AUTHCTX::GenerateFQDNAndSPN(char * lpszHostName = 0x07ad0690 "ntlm.slow.com", unsigned short destPortNumber = 0x50)+0xb4

0741fd18 3d941d6c wininet!PLUG_CTX::PostAuthUser+0x90

0741fd3c 3d93ecec wininet!ProcessResponseHeaders+0x151

0741fe64 3d94e270 wininet!AuthOnResponse+0x317

0741fe84 3d94567b wininet!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x540

0741fe98 3d9455ff wininet!CFsm_HttpSendRequest::RunSM+0xa0

0741feb0 3d945763 wininet!CFsm::Run+0x39

0741fee0 77f69598 wininet!CFsm::RunWorkItem+0x79

0741fef8 7c92796d shlwapi!ExecuteWorkItem+0x1d

0741ff40 7c9279ab ntdll!RtlpWorkerCallout+0x70

0741ff60 7c927a6d ntdll!RtlpExecuteWorkerRequest+0x1a

0741ff74 7c927a44 ntdll!RtlpApcCallout+0x11

0741ffb4 7c80b729 ntdll!RtlpWorkerThread+0x87

0741ffec 00000000 kernel32!BaseThreadStart+0x37

 

 

Enjoy

Anik from APGC DSI Team

Leave a Comment
  • Please add 4 and 2 and type the answer here:
  • Post
  • Can we disable de dns query?

Page 1 of 1 (1 items)