Internet Explorer Support Tools and Debugging

Improved version of STRACE and HTTPREPLAY

New version of STRACE and HTTPREPLAY have been posted on Microsoft's download center :

• http://www.microsoft.com/downloads/details.aspx?familyid=d25ba362-c17b-4d80-a677-1faff862e629&displaylang=en
• http://www.microsoft.com/downloads/details.aspx?familyid=F5EC767F-27F2-4FB3-90A5-4BF0D5F4810A&displaylang=en

In addition to a couple of fixes, the following features have been added to HTTPREPLAY :

• Fiddler (http://www.fiddlertool.com/fiddler/) traces can be replayed
• both HTTP requests and responses are extracted from a STRACE log (/XTRACT switch)
• the SENDFILE utility provided can be used to replay a request (like, for example, a large POST request)
• new flag (/D:port /H:host ) can be used to route unresolvable requests to a specific host/post

When used with latest version of STRACE, HTTPREPLAY can also detect transport related issue (ex : TCP RESET upon send/receive). Such issues will be highlighted in the HTTPREPLAY report :

 Socket errors :

Using STRACE and HTTPREPLAY with Firefox

Some people have asked me if the STRACE and HTTPREPLAY tools can be used with other browsers than Internet Explorer like Mozilla Firefox. The answer to this question is "Yes". You can generate a STRACE log with Firefox by creating the following batch file (straceff.cmd) file in the c:\program files\strace directory :

@echo off
set FFPATH="%programfiles%\Mozilla Firefox\firefox.exe"
set STRACEDLL=STRACE.DLL_IE6
runelevate withdll /d:%STRACEDLL% %FFPATH% about:blank

Running the above batch will create a STRACE log that you can subsequently use with HTTPREPLAY. This allows to do funny things like generating a STRACE log using Firefox and replay it with Internet Explorer (or the opposite)... Note that, since FireFox relies on his own SSL layer (it doesn't use the EncryptMessage/DecryptMessage APIs), STRACE logs generated using Firefox only contain clear text HTTP.

SOCKTRC tool

The SOCKTRC command line utility which is part of the HTTPREPLAY tool is a "socket based proxy". Some people also call such utility a "socket listener". I wrote this tool long time ago mainly to troubleshoot HTTP issues but the tool can also be used to trace any TCP based protocol such as SMTP, POP3..etc.  In addition to being a socket listener, the tool implements the following features :

- display & filtering capabilities
- change data on the fly
- simulate slow speed connection
- extensible architecture through simple extension DLL

Let's take a deeper look at the tool...

Using SOCKTRC

For HTTP troubleshooting, the tool can be used using 2 different methods :

Method #1 - Use SOCKTRC to remote a web server (or any TCP based server like SMTP,POP...etc)

From the command prompt, run "SOCKTRC /S:80 /D:80 /OS /OD /V /H:mywebserver". Congratulations! You now have a new web site available on your machine. To dump the traffic between IE and mywebserver, run Internet Explorer and connect to "http://localhost". Note that if you already have a local web server, you'll need to use another port than 80 (ex. : run "SOCKTRC /S:82 /D:80 /OS /OD /V /H:mywebserver" and use "http://localhost:82" as the URL). Of course this method will only work as long as the hyperlinks used are relatives since clicking on a URL like http://www.microsoft.com will completely "bypass" the tool...

Method #2 - SOCKTRC as HTTP proxy server

Assuming TCP port 80 is not used, run "SOCKTRC /S:80 /D:80 /OS /OD /V /H:myhttpproxy". You can then run Internet Explorer and set localhost:80 as your HTTP proxy server (ensure IE option "Bypass proxy for local addresses" is *not* checked). Once this has been done, all traffic will be dumped by SOCKTRC regardless of the URLs accessed

If you want to trace non HTTP protocols like SMTP, POP..etc only method 1 can be used.

Customizing trace output

By default, SOCKTRC just displays a summary of the data sent/received. For example, running "SOCKTRC /S:80 /D:80 /H:MYWEBSERVER" and connecting with IE to "http://localhost" will result in the following display :

SOCKTRC /S:80 /D:80 /H:MYWEBSERVER

11:41:34:057 ==================================================================
11:41:34:057 Protocol : TCP
11:41:34:057 Source port : 80
11:41:34:057 Bind on adapter : INADDR_ANY
11:41:34:057 Destination port : 80
11:41:34:057 Destination host : MYWEBSERVER
11:41:34:057 ==================================================================
Press to exit.
11:41:37:159 #0 - New connection accepted (127.0.0.1:3387)
11:41:37:174 #0 - 127.0.0.1:3387 -> :80 (377 bytes / total : 377 bytes) :3388 -> MYWEBSERVER:80
11:41:42:516 #0 - 127.0.0.1:3387 <- :80 (4096 bytes / total : 4096 bytes) :3388 <- MYWEBSERVER:80
11:41:42:516 #0 - 127.0.0.1:3387 <- :80 (593 bytes / total : 4689 bytes) :3388 <- MYWEBSERVER:80
11:41:42:516 #0 - 127.0.0.1:3387 -> :80 (463 bytes / total : 840 bytes) :3388 -> MYWEBSERVER:80
11:41:43:064 #0 - 127.0.0.1:3387 <- :80 (4096 bytes / total : 8785 bytes) :3388 <- MYWEBSERVER:80
11:41:43:064 #0 - 127.0.0.1:3387 <- :80 (504 bytes / total : 9289 bytes) :3388 <- MYWEBSERVER:80
11:41:43:064 #0 - 127.0.0.1:3387 -> :80 (615 bytes / total : 1455 bytes) :3388 -> MYWEBSERVER:80
11:41:43:299 #0 - 127.0.0.1:3387 <- :80 (359 bytes / total : 9648 bytes) :3388 <- MYWEBSERVER:80
11:41:43:330 #0 - 127.0.0.1:3387 -> :80 (493 bytes / total : 1948 bytes) :3388 -> MYWEBSERVER:80
11:41:43:518 #0 - 127.0.0.1:3387 <- :80 (4096 bytes / total : 13744 bytes) :3388 <- MYWEBSERVER:80
...

To get all the data displayed use "/OS" (dump data from client) "/OD" (dump data from server) and "/V" to display all data in hex format :

SOCKTRC /S:80 /D:80 /H:MYWEBSERVER /OS /OD /V
...
12:26:31:401 #0 - New connection accepted (127.0.0.1:3784)
12:26:31:401 #0 - 127.0.0.1:3784 -> :80 (269 bytes / total : 269 bytes) :3785 -> MYWEBSERVER:80

=> ==================================================================
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 0123456789abcdef

0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 41 63 63 Accept: */*..Acc
0020: 65 70 74 2d 4c 61 6e 67 75 61 67 65 3a 20 65 6e ept-Language: en
0030: 2d 75 73 0d 0a 41 63 63 65 70 74 2d 45 6e 63 6f -us..Accept-Enco
0040: 64 69 6e 67 3a 20 67 7a 69 70 2c 20 64 65 66 6c ding: gzip, defl
0050: 61 74 65 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a ate..User-Agent:
0060: 20 4d 6f 7a 69 6c 6c 61 2f 34 2e 30 20 28 63 6f Mozilla/4.0 (co
0070: 6d 70 61 74 69 62 6c 65 3b 20 4d 53 49 45 20 36 mpatible; MSIE 6
0080: 2e 30 3b 20 57 69 6e 64 6f 77 73 20 4e 54 20 35 .0; Windows NT 5
0090: 2e 31 3b 20 2e 4e 45 54 20 43 4c 52 20 31 2e 30 .1; .NET CLR 1.0
00a0: 2e 33 37 30 35 29 0d 0a 48 6f 73 74 3a 20 6c 6f .3705)..Host: lo
00b0: 63 61 6c 68 6f 73 74 3a 38 32 0d 0a 43 6f 6e 6e calhost:82..Conn
00c0: 65 63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 ection: Keep-Ali
00d0: 76 65 0d 0a 43 6f 6f 6b 69 65 3a 20 41 53 50 53 ve..Cookie: ASPS
00e0: 45 53 53 49 4f 4e 49 44 47 47 51 51 47 4b 57 47 ESSIONIDGGQQGKWG
00f0: 3d 4f 43 4b 4e 45 4e 48 43 49 42 44 46 4b 47 42 =OCKNENHCIBDFKGB
0100: 50 4b 45 4d 46 46 4a 43 43 0d 0a 0d 0a PKEMFFJCC....
==================================================================
12:26:31:401 #0 - 127.0.0.1:3784 <- :82 (1544 bytes / total : 1544 bytes) :3785 <- MYWEBSERVER:80

=> ==================================================================
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 0123456789abcdef

0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a 53 65 72 76 65 72 3a 20 4d 69 63 72 6f 73 6f .Server: Microso
0020: 66 74 2d 49 49 53 2f 35 2e 31 0d 0a 44 61 74 65 ft-IIS/5.1..Date
0030: 3a 20 57 65 64 2c 20 30 34 20 53 65 70 20 32 30 : Wed, 04 Sep 20
0040: 30 32 20 31 30 3a 32 36 3a 33 31 20 47 4d 54 0d 02 10:26:31 GMT.
0050: 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0060: 20 31 33 39 30 0d 0a 43 6f 6e 74 65 6e 74 2d 54 1390..Content-T
0070: 79 70 65 3a 20 74 65 78 74 2f 68 74 6d 6c 0d 0a ype: text/html..
0080: 43 61 63 68 65 2d 63 6f 6e 74 72 6f 6c 3a 20 70 Cache-control: p
0090: 72 69 76 61 74 65 0d 0a 0d 0a 3c 68 74 6d 6c 3e rivate....
00a0: 20 0d 0a 20 20 0d 0a 3c 68 65 61 64 3e 0d 0a 3c .. ....<
00b0: 6c 69 6e 6b 20 74 79 70 65 3d 27 74 65 78 74 2f link type='text/
00c0: 78 6d 6c 27 20 72 65 6c 3d 27 61 6c 74 65 72 6e xml' rel='altern
00d0: 61 74 65 27 20 68 72 65 66 3d 27 2f 44 65 66 61 ate' href='/Defa
...

To display only the first 100 bytes of client data, use "/OS:100" :

SOCKTRC /S:80 /D:80 /H:MYWEBSERVER /OS:100
...
12:33:58:693 #0 - New connection accepted (127.0.0.1:3819)
12:33:58:693 #0 - 127.0.0.1:3819 -> :82 (269 bytes / total : 269 bytes) :3820 -> MYWEBSERVER:80

=>GET / HTTP/1.1..Accept: */*..Accept-Language: en-us..Accept-Encoding: gzip, deflate..User-Agent: Moz
12:33:58:708 #0 - 127.0.0.1:3819 <- :82 (1544 bytes / total : 1544 bytes) :3820 <- MYWEBSERVER:80
12:33:58:708 #0 - 127.0.0.1:3819 -> :82 (413 bytes / total : 682 bytes) :3820 -> MYWEBSERVER:80

=>GET /_themes/artsy/arts1011.css HTTP/1.1..Accept: */*..Referer: http://localhost..Accept-Language
12:33:58:708 #0 - 127.0.0.1:3819 <- :82 (141 bytes / total : 1685 bytes) :3820 <- MYWEBSERVER:80
12:33:58:740 #0 - 127.0.0.1:3819 -> :82 (441 bytes / total : 1123 bytes) :3820 -> MYWEBSERVER:80

=>GET /default.asp HTTP/1.1..Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application
12:33:58:755 #0 - 127.0.0.1:3819 <- :82 (2291 bytes / total : 3976 bytes) :3820 <- MYWEBSERVER:80
12:33:58:755 #1 - New connection accepted (127.0.0.1:3821)
12:33:58:755 #1 - 127.0.0.1:3821 -> :82 (439 bytes / total : 439 bytes) :3822 -> MYWEBSERVER:80

=>GET /navbar.asp HTTP/1.1..Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/v
12:33:58:755 #1 - 127.0.0.1:3821 <- :82 (1444 bytes / total : 1444 bytes) :3822 <- MYWEBSERVER:80
12:33:58:755 #0 - 127.0.0.1:3819 -> :82 (438 bytes / total : 1561 bytes) :3820 -> MYWEBSERVER:80

To display only the 100 bytes sent by the server, just use "/OD:100" (this can be combined with /OS:100).

It is also possible to filter the data displayed based on its contents. For example, you can display only "401" responses from server using "/HSS:401". Same filtering can be done on the client. For example, to only display POST request, use "/CSS:POST".

Other tips & tricks

• SOCKTRC uses the ETC/SERVICES file so, assuming your SERVICES file is correctly populated, you can type something like :

"SOCKTRC /s:HTTP /d:HTTP /os /od /v". This is usefull if you do not know all port numbers...

• SOCKTRC binds by default on INADDR_ANY. You can bind SOCKTRC on a specific network adapted using the "/B" option
• You can change data on the fly using the "/CSR:" (data from client) and "/HSR:" (data from server) flags. The following command changes "Cache-Control" to "Xache-Control" in all content sent by MYWEBSERVER : 

"SOCKTRC /s:80 /d:80 /os /od /v /HSS:Cache-Control /HSR:Xache-Control /H:MYWEBSERVER".

This feature can be usefull for testing purposes (for example, to adjust an invalid content-length or remove HTTP no-cache header). Note that in many cases, you have to make sure that searched/replaced strings have the same length (to maintain valid content-length or chunck size).

• When you search or replace data, you can specify binary data using hex format HxHH (example : "/CSS:0x610x620x63" is equivalent to "/CSS:abc")
• The "/F" flag can be used to display only the data coming from a specific client. (this is usefull if multiple machines are connecting to SOCKTRC source port as you may only be interested to do traces for one specific machine).

FAQ 

Here is a small FAQ regarding the tool and issues you may get and hopefully resolve!

SOCKTRC arguments

SOCKTRC </S:port> </D:port> </B:adapter> [/H:host] [/F:client] [/OS:n] [/OD:n] [/U] [/R] [/L:speed] [/V]
/S:sourceport	source port to use
/D:destport	destination port to use (-1 to drop incoming data)
/B:adapter	bind on adapter (IP/hostname)
/H:host	destination IP/hostname
/F:client	only output data to/from client
/CSS:csstring	only output data from client containing csstring
/CSR:csrstring	replace csstring by csrtring
/HSS:hsstring	only output data from destination containg hsstring
/HSR:hsrstring	replace hsstring by hsrstring
/OS[:n]	output [n] characters sent by source
/OD[:n]	output [n] characters sent by destination
/U	use UDP (by default, TCP is used)
/R	resolve client IP address
/L:speed	simulate a speed bps connection
/PC:sleepdelay	sleep x ms before sending to source
/PH:speeddelay	sleep x ms before sending to destination
/MC:buffersize	use buffersize for recv from client
/MH:buffersize	use buffersize for recv from destination
/N:string	decode BASE64 data located after string
/V	verbose (hexadecimal) output
/T	silent mode (minimum output)
/I	install as a service
/Z	remove service
/X	only allow incoming connection from local machine
/E:DLL	load DLL extension
Examples :
socktrc /s:http /d:http /h:www.ibm.com /os:40 /v
socktrc /s:dns /d:dns /h:mydnsserver /os /od /v /u /r
socktrc /s:82 /d:80 /h:myproxy /os /od /v /r /n:NTLM0x20

When I run SOCKTRC, I get the following error : "ListeningThread : error on bind() : 10048"

This occurs because the source port (/S:portnumber) specified is already bound by another application (for example IIS). You can solve the issue by either using a different source port for SOCKTRC or change the port used by the other application (IIS).

What is the purpose of the /X option ?

The /X option has been introduced to prevent remote incoming connections. This option is usefull if you want to avoid to proxy malware data for example. Proxying malware traffic could make think network administrators that your machine is infected by malware even if it isn't. By using the /X option, you only allow local application to connect to SOCKTRC.

How do I use SOCKTRC as a service ?

First, ensure that the SOCKTRC arguments that you plan to use are valid. If your plan is to use SOCKTRC to silently proxy HTTP traffic, then you may want to use arguments like : "SOCKTRC /S:80 /D:80 /H:someweb /T". Once you have tested and confirmed that the SOCKTRC command line is valid, all tou have to do is to type the desired command line and add the "/I" flag to install the SOCKTRC service. You can then "NET START SOCKTRC"... When SOCKTRC is executed as a service, the output is sent using OutputDebugString (you can use a tool like DBMON to display it).

Introduction to STRACE/HTTPREPLAY support tools

I'm Emmanuel Boersma and I'm Escalation Engineer on Internet Explorer for EMEA (I'm located in Paris). As a support engineer, I use and occasionally write troubleshooting tools. Today, I'm going to give some details on the following tools recently posted on Microsoft's download center:

• STRACE - http://www.microsoft.com/downloads/details.aspx?FamilyID=f5ec767f-27f2-4fb3-90a5-4bf0d5f4810a&displaylang=en
• HTTPREPLAY - http://www.microsoft.com/downloads/details.aspx?FamilyID=d25ba362-c17b-4d80-a677-1faff862e629&DisplayLang=en

Good and bad things about WININET logs

If you are dealing with HTTP issues in IE or if you write applications on top of WININET, you probably know WININET.DLL and WININET logging in order to generate HTTP traces. WININET logging is described in the following article : http://support.microsoft.com/kb/884931. The log produced contains details regarding calls to WININET API, data sent & received including clear text & encrypted data if you are using SSL.

Here's an abstract of a WININET.LOG for a connection to www.microsoft.com :

18:00:38.369 00000b60:<app> 001 InternetConnectA(0xcc0004, "www.microsoft.com", 80, "", "", INTERNET_SERVICE_HTTP (3), 0x00000000, 0x203668)
18:00:38.370 00000b60:<app> 001 HttpOpenRequestA(0xcc0008, "GET", "/", "", "", 0x137518, 0x00400000, 0x00203668)
18:00:38.372 00000b60:<app> 001 HttpSendRequestA(0xcc000c, "Accept-Language: en-us\r\nAccept-Encoding: gzip, deflate", -1, 0x0, 0)
18:00:38.679 000007a4:<---> 000 sending data:
18:00:38.679 000007a4:<---> 000 658 (0x292) bytes @ 0x217518
18:00:38.679 000007a4:<---> 000 00217518  47 45 54 20 68 74 74 70-3a 2f 2f 77 77 77 2e 6d   GET http://www.m
18:00:38.679 000007a4:<---> 000 00217528  69 63 72 6f 73 6f 66 74-2e 63 6f 6d 2f 20 48 54   icrosoft.com/ HT
18:00:38.679 000007a4:<---> 000 00217538  54 50 2f 31 2e 31 0d 0a-41 63 63 65 70 74 3a 20   TP/1.1..Accept:
18:00:38.679 000007a4:<---> 000 00217548  69 6d 61 67 65 2f 67 69-66 2c 20 69 6d 61 67 65   image/gif, image
18:00:38.679 000007a4:<---> 000 00217558  2f 78 2d 78 62 69 74 6d-61 70 2c 20 69 6d 61 67   /x-xbitmap, imag
18:00:38.679 000007a4:<---> 000 00217568  65 2f 6a 70 65 67 2c 20-69 6d 61 67 65 2f 70 6a   e/jpeg, image/pj
18:00:38.679 000007a4:<---> 000 00217578  70 65 67 2c 20 61 70 70-6c 69 63 61 74 69 6f 6e   peg, application
18:00:38.679 000007a4:<---> 000 00217588  2f 78 2d 73 68 6f 63 6b-77 61 76 65 2d 66 6c 61   /x-shockwave-fla
18:00:38.679 000007a4:<---> 000 00217598  73 68 2c 20 61 70 70 6c-69 63 61 74 69 6f 6e 2f   sh, application/
18:00:38.679 000007a4:<---> 000 002175a8  76 6e 64 2e 6d 73 2d 65-78 63 65 6c 2c 20 61 70   vnd.ms-excel, ap
18:00:38.875 000007a4:<---> 000 received data:
18:00:38.875 000007a4:<---> 000 1024 (0x400) bytes @ 0x217518
18:00:38.875 000007a4:<---> 000 00217518  48 54 54 50 2f 31 2e 31-20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.
18:00:38.875 000007a4:<---> 000 00217528  0a 50 72 6f 78 79 2d 43-6f 6e 6e 65 63 74 69 6f   .Proxy-Connectio
18:00:38.875 000007a4:<---> 000 00217538  6e 3a 20 4b 65 65 70 2d-41 6c 69 76 65 0d 0a 43   n: Keep-Alive..C
18:00:38.875 000007a4:<---> 000 00217548  6f 6e 6e 65 63 74 69 6f-6e 3a 20 4b 65 65 70 2d   onnection: Keep-
18:00:38.875 000007a4:<---> 000 00217558  41 6c 69 76 65 0d 0a 43-6f 6e 74 65 6e 74 2d 4c   Alive..Content-L
18:00:38.875 000007a4:<---> 000 00217568  65 6e 67 74 68 3a 20 33-32 31 30 31 0d 0a 56 69   ength: 32101..Vi

In addition to the data sent & received on connections, WININET log provides output regarding calls to WININET API which can be useful if you want to troubleshoot WININET application. The painful things regarding WININET are the following :

• you need to get the appropriate debug build of WININET.DLL for your OS/configuration. You cannot use a debug build of WININET.DLL for XP SP2/IE7 and use it on Windows 2003 or Vista. Since WININET is also included in IE security fixes, it can be a real pain to find the appropriate debug version of WININET matching your configuration
• the log produced may not contain any socket information indicating on which connection the data is sent/received
• you cannot generate a WININET log for a process that has already been started without WININET logging enabled.
  

STRACE as an alternative to  WININET logging

The main goal of STRACE is to produce similar logging to WININET log without the above constraints. In addition STRACE is not limited to IE/HTTP. It can be used to trace any socket based application like, for example, Outlook Express. If the application uses SSL, STRACE will dump clear text data before encryption & after decryption. All you need to do is install the tool and double click on it : this will run a new Internet Explorer instance and generate a trace on the desktop for this IE instance. Here's the STRACE equivalent for above WININET log:

03/28/2007 13:33:48:758 - socket 0x000002cc created
03/28/2007 13:33:48:758 - connect socket 0x000002cc (65.53.196.57:80)
=====================================================
03/28/2007 13:33:49:305 - 719 byte(s) sent on socket 0x000002cc
=====================================================
      00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f   0123456789abcdef
0000: 47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 6d   GET http://www.m
0010: 69 63 72 6f 73 6f 66 74 2e 63 6f 6d 2f 20 48 54   icrosoft.com/ HT
0020: 54 50 2f 31 2e 31 0d 0a 41 63 63 65 70 74 3a 20   TP/1.1..Accept:
...
=====================================================
03/28/2007 13:33:49:540 - 583 byte(s) received on socket 0x000002cc
=====================================================
      00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f   0123456789abcdef
0000: 48 54 54 50 2f 31 2e 31 20 33 30 32 20 46 6f 75   HTTP/1.1 302 Fou
0010: 6e 64 0d 0a 50 72 6f 78 79 2d 43 6f 6e 6e 65 63   nd..Proxy-Connec
...
=====================================================
03/28/2007 13:33:49:540 - 737 byte(s) sent on socket 0x000002cc
=====================================================

If you want to generate a LOG for a process other than IE7, Outlook Express for example, you'll just need to change the running process in STRACE.CMD by MSIMN.EXE  :

withdll /d:STRACE.dll_IE6 "%programfiles%\outlook express\msimn.exe"

The above command allows to trace SMTP/POP3/IMAP activity for Outlook Express on XP SP2.

Since STRACE is based on detours library (http://research.microsoft.com/sn/detours/), you can also "inject" the STRACE tracing DLL in an existing process using the "injlib" tool (see STRACE page for details).

HTTPREPLAY or how to parse and replay HTTP from a log file

Now, there is still one problem with either WININET or STRACE logging : you need to use your favorite editor and walk through the log …. Since a WININET or STRACE log can quickly take more than 100 MB, you may spend some time to find exactly what you are looking for.  This is where the HTTPREPLAY tool can help you. HTTPREPLAY provides the following services :

• HTTPREPLAY parses STRACE or WININET log and generate a report
• HTTPREPLAY can then be used to replay a live scenario based on the log

Once you have install the tool, you can browse to any STRACE and WININET log and open it using right-click -> Open With -> navigate to “c:\program files\httpreplay\httpreplay.cmd”. After the log is parsed, you’ll get a report similar to the following :

The HTTPREPLAY report provides :

• summary of HTTP requests / responses
• details for every request/reponse (click on GET request or HTTP status code to get details)
• statistic allowing to measure number of request/response time, bytes sent & received…etc
• logic to point common HTTP issues like : slow response from the server, failed request, bad content-length

Replaying HTTP from a log 

Now that the LOG has been loaded & parsed, HTTPREPLAY can be used to replay any response for the URLs listed in the log. The replay can be done in “proxy mode” or “direct mode”. “Proxy mode” just consists to set the replay tool as your proxy (by default : localhost:81). Direct mode requires that you edit the hosts file and enter all hosts referenced in the trace. I personally recommend to use proxy mode since this ensures every HTTP request will hit the tool. When you request specific content (by clicking on a URL in the report), you’ll see the requests/responses in the HTTPREPLAY command window :

17:53:14:623 #0 - GET http://www.microsoft.com/ [FOUND]
17:53:14:623 #0 - 127.0.0.1:2142 <- :81 (583 bytes / total : 583 bytes)
17:53:14:639 #0 - 127.0.0.1:2142 -> :81 (731 bytes / total : 1444 bytes)
17:53:14:639 #0 - GET http://www.microsoft.com/en/us/default.aspx [FOUND]
17:53:14:639 #0 - 127.0.0.1:2142 <- :81 (1024 bytes / total : 1607 bytes)
17:53:14:639 #0 - 127.0.0.1:2142 <- :81 (3356 bytes / total : 4963 bytes)
17:53:14:639 #0 - 127.0.0.1:2142 <- :81 (1460 bytes / total : 6423 bytes)
17:53:14:639 #0 - 127.0.0.1:2142 <- :81 (1536 bytes / total : 7959 bytes)
17:53:14:654 #0 - 127.0.0.1:2142 <- :81 (1460 bytes / total : 9419 bytes)
17:53:14:654 #0 - 127.0.0.1:2142 <- :81 (6072 bytes / total : 15491 bytes)
17:53:14:654 #0 - 127.0.0.1:2142 <- :81 (1460 bytes / total : 16951 bytes)
...

If the HTTPREPLAY command window doesn’t show anything then you are either not using the tool as your proxy or forgot to add a host in you hosts file.

Changing replayed content

Since we know how to generate a trace and replay it, wouldn’t it be interesting to change the way the content is “replayed” ? For example, what should I do to add an alert() in a js files ? Well, the 1st step consists to extract the responses in individual files. Running the following command will do the job :

C:\Program Files\HTTPREPLAY>httpreplay STRACE_IEXPLORE_PID_3932.LOG /xtract
...
18:10:44:393 50 files extracted (use /USEFILES switch to replay from files)

C:\Program Files\HTTPREPLAY>dir *.replay

Directory of C:\Program Files\HTTPREPLAY

31/05/2007  18:10           7 763 index.replay
31/05/2007  18:10              583 request0001.replay
31/05/2007  18:10          40 389 request0002.replay
...
31/05/2007  18:10              735 request0050.replay
 

The index.replay lists all URLs & response files :

C:\Program Files\HTTPREPLAY>type index.replay
http://www.microsoft.com/@request0001.replay
http://www.microsoft.com/en/us/default.aspx@request0002.replay
http://js.microsoft.com/shared/core/1/js/library.js@request0003.replay
…

You can “replay” from the response files using the following command: httpreplay /USEFILES. If you want to change the response’s content, you’ll just need to edit the required “*.replay” file.  Since HTTPREPLAY can use both STRACE log and index.replay, consider the following approach if you want to add an alert to a single JS file :

• edit index.replay and only keep responses that you plan to change (example : http://js.microsoft.com/shared/core/1/js/library.js@request0003.replay)
• edit the response file (request0003.replay), add alert(‘test’) in the 1st script block and save it (don’t forget to remove the content-length header as detailed in the tool’s documentation)
• clear the IE cache and run the tool to replay from both log file and index : httpreplay strace_www-microsoft-com.log /USEFILES

The above approach can be useful for support engineers to reproduce problems, implement & test workarounds. It can also help web site developpers to write and test modifications offline.

I hope you’ll find the above tools useful and will get back to you with more blogs/info in the near future….

 - Emmanuel