Rahul Soni's blog

Never assume the obvious is true!

How to find out details using LOG PARSER from huge text files without opening them in Notepad, etc.

How to find out details using LOG PARSER from huge text files without opening them in Notepad, etc.

  • Comments 6

Quite often, we receive log files for analysis which are simply HUGE!!! We try opening that in Notepad and it hangs. After n minutes, we kill the notepad, try MS Excel and that hangs as well. Sometimes, MS Excel shows "File not loaded completely". Painful, isn't it??

OKAY, what's the point?? We are wasting time in trying to open the files in the first place!!!

So, I have a Filemon Log of 500 MB and I want to search the file for the lines which contain "Access Denied". Why am I trying to open that file to find out just a few lines containing the string I am interested in? May be because we are used to CTRL+F. Is there a way out??? Yes, there is... and in comes Log Parser. Download it from http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en and run the setup.

Now, we will see how to use Log Parser to parse the file without opening it. The filename is Filemon.log and it is located in C:\. All I am interested in, is to find the lines which contain the string "Access Denied" WITHOUT opening the Filemon.log, because none of the software is responding in a timely manner (due to the size factor). You need to start the Log Parser and you will see a command line interface. Type the following and hit enter...

LOGPARSER "Select Text from C:\Filemon.log where Text like '%Access Denied%'" -i:TEXTLINE -q:Off

You will see an output in a similar format as follows...

Text
--------------------------------------------------------------
7447    1:49:24 PM      explorer.exe:1200       DIRECTORY       C:\      Access Denied

Statistics:
-----------
Elements processed: 640444
Elements output:    1
Execution time:     12.75 seconds

Not bad at all...

By the way, there are tonnes of native log files like IIS Log files, CSV, TSV, URLSCAN, REG(istry), FS (Filesystem), XML, etc which the Logparser can parse for you in a more robust fashion. Go through the documentation that comes along with the log parser. This is one of the tools which you will definitely like to master and keep it in your arsenal of tools for troubleshooting various kind of issues.

Cheers!
-Rahul Soni

  • I am rather new to asp.net 2.0 and I have a problem here. i give the configsections as below

    <configSections>
       <sectionGroup name="Path">
    <section name="PathInfo" type=".."/>
       </sectionGroup>
    </configSections>

    <Path>
      <PathInfo>
    <add key="UploadPath" value="C:\Aspiren\Documents\Uploaded.txt"/>
      </PathInfo>
    </Path>

    The problem is when i access this though the code behind(as in 1.1), it asks for some alias.

    NameValueCollection nvcGeneral = (NameValueCollection)ConfigurationSettings.GetConfig("Path/PathInfo");

    Is it that i am missing any namespaces or is ther any other way to access it?
  • Actually, System.Configuration.GetConfig(string) is obsolete now. Try using System.Configuration.ConfigurationManager.GetSection and you should be good to go!

    Hope that helps!
    Rahul
  • I want to parse whole Vb.Net file. Any body how to parse the same.
    Regards
    Allahbaksh
  • Hi Allahbaksh,

    You can try reading the .vb files line by line and do the needful.

    http://www.builderau.com.au/program/windows/soa/Reading_and_writing_text_files_with_VB_NET/0,39024644,20267367,00.htm

    There are many similar links which might help you out.

    Thanks,
    Rahul
  • Hi Rahul,

    We have developed an application on our Desktops to create a file with data of our clients: The Desktop has Windows XP (latest SP) with VS 2003 Enterprise running Framework 1.1.  The IIS Version is 5.1.  When we execute our web application locally it takes only 2 minutes to generate the file.

    When we take the application over to the server  IBM x3850 16 Gig of Ram, 4 processor  (Windows 2003, IIS 6.0), The program takes over 10 hours to produce the file.  

    The program makes one call to the database to retrieve all the data, then writes each clients info to a file.  For both the server and desktop we are using the same database server.  The problem takes place after the call to the database so that should not be an issue.

    We tried this application on another server and the same result happen.

    Any suggestion?

    is there any any configuration on servers that limits the bandwidth of writing to the file etc…  

    thanks for your& regards

    arshad

  • Hi Arshad,

    The difference is huge!! 10 hrs to 2 mins, no comparisons at all. I suspect something fishy is going on.

    I would suggest taking a trace.axd on the server and see what is causing that much of delay. BTW, was it really 10 hrs or 10 mins??

    Basically, I would try browsing on the server with trace enabled and start from there!

    HTH,

    Rahul

Page 1 of 1 (6 items)
Leave a Comment
  • Please add 4 and 3 and type the answer here:
  • Post