Looking at temporal data can be quite informative. For example, I have a table of about 16000 photographs. Each record has a datetime stamp. Looking at the temporal distribution one can clearly see events, such as the birth of a child, the visit of relatives or the purchase of a camera.
Internet Information Server (IIS) is the web server that ships with Windows XP Professional. If you have XP Pro, you can install it from the Control Panel->Add/Remove programs/ Add/remove Windows Components. Once installed, you can hit your web site just by typing “localhost” in the address bar of internet explorer on the same machine. From a different machine on the network you can hit that web site by typing in the machine name. To hit that machine on the internet, you can get a static IP address from your online provider and attach a domain name to it. For example my IP is http://188.8.131.52 and my domain name is www.calvinhsia.com.
When installed, IIS defaults to logging every web hit to logfiles. Choose Start->Control Panel->Administrative tools->Internet Information Services, then drill down to local computer/Web sites/default web sites. Right click and choose Properties. The Web Site page shows an “Enable logging” checkbox, logging options and properties. The default is to log to “C:\WINDOWS\System32\LogFiles” with Log file name: W2SVC1\exyymmdd.log
Go ahead and hit your own web site via this http://localhost then look at the generated log file. The datetime logged by default is Universal Coordinated Time, which for most people is not local time. This might make your logfile generated name be for tomorrow (or yesterday) depending on your time zone.
(In addition to the IIS logging, you can do your own custom logging. For example, my web site logs every web hit into a fox table.)
The 120 line program below parses the IIS logs into a cursor and graphs the results. It took only a few seconds to scan through more than 1000 daily log files resulting in over half a million web hits. The graph was displayed in about one second.
This SQL Select statement is handy when creating and scaling the data for the graph:
SELECT INT((dtime-this.nMin)/this.nBucketSize) as bucket,COUNT(*) as cnt FROM webhits GROUP BY 1 INTO CURSOR buckets
When you click on a point on the graph, a SQL Select statement for the corresponding bucket is executed to get a summary of the web hits for that day, which is displayed on a grid on another form.
*ERASE webhits.dbf && erase the file if you want to reparse
PUBLIC oGraph as Form
DEFINE CLASS GraphForm as Form
SELECT MIN(dtime) as min, MAX(dtime) as max FROM webhits INTO CURSOR minmax
this.nBucketSize=(this.nMax-this.nMin) / this.nBuckets
SELECT MAX(cnt) as maxheight FROM buckets INTO CURSOR maxheight
SCAN && scan through the cursor, plotting each bucket in a graph
PROCEDURE Mousedown(nButton, nShift, nXCoord, nYCoord)
SELECT * FROM webhits WHERE INT((dtime-this.nMin)/this.nBucketSize)=nXCoord INTO CURSOR results
SELECT uri,count(*) from results GROUP BY 1 ORDER BY 2 descending INTO CURSOR daily
IF VARTYPE(this.oGridForm)!= 'O'
WITH .grid1 as Grid
thisform.oGridForm.caption = DTOC(TTOD(results.dtime))
DEFINE CLASS ParseLogs as custom
CREATE TABLE webhits (dtime t,ip c(15),method c(20),uri c(50))
FOR i = 1 TO n && for each log file (Log file name: W2SVC1\exyymmdd.log)
PROCEDURE ProcFiles(cFile as String)
DO WHILE !FEOF(nh)
IF ASC(cstr)=0 && null line is empty log
IF cStr="#" && if it's part of the log header
dtime=CTOT(cDate+" "+cTime) && time
ip=LEFT(cStr,nAt-1) && IP address
method=LEFT(cStr,nAt-1) && method
uri=LEFT(cStr,nAt-1) && URI
INSERT INTO webhits FROM memvar