Usage on My Blog and SharePoint Usage Stats

I was doing some analysis on the usage on this blog and came across some interesting info, that I thought you as well might find interesting.  Some of you might be suprised what you can find out about your traffic by analyzing your logs or running a web analyzer product against your logs.

• 11% of visits are from Windows Servers 2003 another 2% from Windows 2000.  What does this tell me? There are a large number of IT Pros that don't use the IE hardening.  I personally had mixed emotions.  It bothered me, not that I would surf from my SharePoint servers, but in WSS 2.0/SPS 2003 there were server side scenarios where the server would make requests on behalf of the user.  Remember troubleshooting smigrate and finding out that IE hardening was blocking it?  That was a strange one.  I haven't necessarily come across any reasons you'd need to turn it off with WSS 3.0/MOSS 2007, unless you are supporting the OWC or remote XML/RSS feeds. (User Agent String)
• 18% of visits are from Vista.  Way to go.  There are some tech savy users out there.  Vista's only been on the consumer market for a couple of days.  Not only that but 30% of visits at 1280 x 1024 resolution and 18% higher than that means there are some nice monitors out there. (User Agent String and client variables?)
• At 1% combined MAC and Linux users don't frequent my blog. Let me speculate and say that SharePoint Admins don't use MACs or Linux for their primary PCs. (User Agent String)
• With 56% of users on Cable or DSL, and 25% on Corporate LANs, and less than 7% on dialup.  Again based on connection speeds I've got quite the savvy crowd. (Bytes transfered, time to last byte with some math)
• I've got loyalty... 33% of visitors return for more with an average of nearly 2 page requests per visit.
• These loyal SharePoint land fans from a page perspective either have favorites or know the URL well.  22% of traffic is directly entered while the largest portion comes from various search traffic. (Referrer)
• With over 90 Countries and 30 languages represented and more than half of the traffic coming from outside of the U.S. I think I have justified my need to travel around the globe.  Maybe I'll see you at the SharePoint Conference in Sydney still TBD in May.  At 8% I've got a decent following in the UK.  I'm sure there's some thanks Steve. (Client IP Reverse DNS lookups and client variables)
• As a group most traffic does come from inside Microsoft Network 5% (client IP reverse DNS lookup)

Since I'm talking about Usage maybe I should share some of my SharePoint usage observations.

WSS 3.0 usage reports are extremely similar to WSS 2.0.  They are disabled by default, and have to be enabled.  They can be viewed at the Site collection level and they are primarily text based with info on page requests.  If you use SharePoint Designer, you can get at some more interesting stats like browser/OS, usage over more time.  Don't forget to enable these in the Central Admin operations.

MOSS 2007 usage reports are much more verbose.  Again disabled by default and enabled in the SSP Usage Reports.  You'll want both WSS and MOSS Usage enabled.  One of my favorite new reports is query/search requests.  Imagine being able to get this type of data without having to grovel the logs looking for populated query strings (SPS 2003).  These nice pie charts are very handy to understand what's going on from both a design and service perspective.  In addition to usage logs, I am a fan of the audit logs, even in Pivot table format it's handy to look at security changes on the site, or view document deletions to find the culprit.  Obviously these need to be enabled as well.  By the way, if you're concerned about web farms, not to worry.  Server farms usage analysis is handled via timer jobs and usage data is populated in the content database. 

I've seen info on getting at more than the first 30 days in the previous versions of the product.  Serge has some excellent tips across usage and using IIS log parser (one of my favorite free tools in this space from the IIS 6 Resource kit).  I keep waiting to hear more about WebTrends next version to hear what they are doing in this space.  In my search, I came across an old press release, but having a partner that provides usage with both the SharePoint logs and IIS logs is bound to happen soon.

So as I mentioned above, the usage logs are site collection based.  So using tools like IIS Log parser a powerful command line tool, or a freeware tool like Funnelweb.  For as simple to use tool as funnelweb, I've been impressed how well it handled logs that were larger than 1GB.  I've consolidated a few different WFE logs onto a single desktop with decent RAM 1-2GB and although it consumes all resources it did finish a few of logs parsing millions of requests.  WebTrends or most others could give you the majority of what you are looking for in a farms logs, the top pages isn't what I'd want, but I think we're getting closer... it's a matter of time before the partners do the work to parse them both.  I do see a few SharePoint hosters providing logs via WebTrends 7 and 8.

Snippet from http://www.logparser.com/Repository.htm 

From Tadd E. Dawson:

 Here is a little LogParser SQL that I found useful in determining what users were searching for on our Intranet use SPS 2003 Search.

I call it via "C:\Program Files\IIS Resources\Log Parser\LogParser" -o:csv file:search.sql

search.sql:

SELECT DISTINCT 
TO_UPPERCASE(EXTRACT_VALUE(cs-uri-query, 'k')) AS SearchString, 
EXTRACT_VALUE(cs-uri-query, 's') AS Scope, COUNT(*) AS HowMany
FROM *.log
TO search.csv
WHERE cs-uri-stem = '/search.aspx'AND cs-uri-query NOT LIKE 
'%[Microsoft+Office+SharePoint+Portal+Server+2003+LOG]%' AND SearchString IS 
NOT NULL
GROUP BY SearchString, Scope
ORDER BY HowMany DESC

If you have any problems with usage on your farms you may find these tips useful.