Often times, either due to a misconfiguration/bug/solar eclipse or otherwise, customers call into Microsoft Product Support Services complaining that their Exchange server is churning out transaction logfiles at an alarming rate. For every instance of this symptom, there are at least a dozen reasons why this is happening. Regardless, there's never been a good way to parse the transaction logs and extract any useful patterns. In lieu of rolling up my sleeves and actually writing code to accomplish such a task, I've slapped together a bunch of utilities that will do the job. Ugly? Sure. Useful? You bet. Having used it against many customer issues, I can attest that this actually works, and works quite well.
1. Download the "Unix for Win32" utilities from http://downloads.sourceforge.net/unxutils/UnxUtils.zip?modtime=1172730504&big_mirror=0
2. Extract all files from the UnxUtils\usr\local\wbin subsirectory to C:\UNIX
3. Download strings.exe from http://live.sysinternals.com/strings.exe, and place strings.exe into C:\UNIX
4. Make a C:\TMP directory (Unix tools need a Win32 equivalent of /tmp)
5. Make a directory for all your transaction log files (i.e. D:\customers\test), and place all the logs in this dir
6. From a cmd prompt, navigate to your C:\UNIX dir
7. Run the following command:
strings -q -n 16 D:\customers\test\*.log | cut -f3 -d: | sort | uniq -c | sort | tee c:\log-output.wri
What this is doing:
· Identifies all strings in the logs greater than 16 chars
· Removes the D:\customers\test\E00xxxx.log: from the output
· Sorts the output
· Finds all duplicate records, and retains a count
· Sorts the final output (ending with the largest # of occurrences)
· Writes all the output to c:\log-output.wri (use WordPad / write.exe to open; notepad.exe mangles the output)
If you're running this on Windows 7 or above, you'll have to modify the output directory as follows (as it won't let you write directly to the root of the C: drive) ...
strings -q -n 16 D:\customers\test\*.log | cut -f3 -d: | sort | uniq -c | sort | tee c:\users\yourname\log-output.wri
The output will be sorted from the least number of repeating occurences to greatest, so crack open that log-output.wri file, scroll to the bottom, and commence spelunking!
We initially tried using Powershell, but the performance was unacceptably slow.
When I use strings -q -n 8 I got last 2 lines to:
Why is the last line empty, or is it a special character like CR or LF?
This is from 1 minute sample that had 90 MB of logs.
Could it be that some device or client causes lot of CR or LF in the logs?
Your observation is accurate; seems like a client is passing in a blank line (of at least 8 chars long) in addition to a likely CRLF. You may find support.microsoft.com/.../972705 helpful in your troubleshooting as well.
really wish this had helped, all i am seeing in output file is bunch of pages with DDDDDDDDDDDD and OOOOOOO and then the last page with a user account name and then DocumentSummaryInformation
what does this mean?? please help.
Thanks for this helpful post, I get ESE super ECCXORChecksum on the top of the list, should I disable the online maintenance for testing purposes?
also there are some user names in this format FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=
can you please advise
I wouldn't recommend disabling OLM here; ECCXORChecksum is simply what's being written to & read off of disk by ESE (the storage engine). As far as the user name format listed above, that's referred to as the LegacyExchangeDN. If you have one user that is recurring more than others (regardless if it's an SMTP address, LegacyExchangeDN, or otherwise), I'd investigate that user activity more in-depth.
While my approach is admittedly hacky (at best), also check out our Exchange Team's blog post on this subject:
Strings.exe download link is broken.
Correct Link: technet.microsoft.com/.../bb897439.aspx
@Pavan - thanks for the feedback! I've updated the link to point directly to our sysinternals repository.
I'm also struggling with the problem of a massive transaction log growth since a few days.
After a lot of research I found your article and let the command run over aprox. 6000 log files.
The last lines are these, what does this mean? Do you have any idea?
50416 W. Europe Standard Time
I don't know how to interpret this.
Many many thanks in advance!
Hi. Thanks for this, however, just like Bastian, my top "users" are;
W. Europe Standard Time
as well as
0000000000 65535 f
However, the bulk of data is "DDDDDDDD" or "OOOOOOOO" along with some "RRRRRR". Those occupy about 95% of all pages, so I guess whatever those are, they are the cause for my log growth. But what does it mean?
Hi Emil and Bastian,
Sorry about the delayed reply. Inferring from the output ("W. Europe Standard Time") and the stEvt string, it sounds like both issues are due to some sort of "Recurring Meeting Gone Wild." My advice is to check out some of the tips & tricks on how to nail down this type of issue in the two following blog posts:
Hope this helps,
This is a very delayed comment based on the post creation date, but still valuable :) I'm a PFE and worked on a case where the customer experienced a huge log growth and confirmed, using this post as a reference, that the 2nd on the string rank was "IPM.Note.EnterpriseVault.PendingArchive". Moreover, the 1st one was a mailbox displayname. With this in mind, customer and Symantec found some issues on EV archive settings.
Sure, the threads here are old and grey, but i too was seeing extended strings of 'D' 'O' and 'H' in the output from this article. I noticed several of you had the same answer and thought this may be helpful:
Specifically, check out the fill pattern character for page zeroing. I suspect these strings are maintenance operations.