183 : Comp::*

Is solid state up to an ESE challenge?

BrettSh — Mon, 05 Mar 2007 19:07:00 GMT

~Eric and I were at Fry's the other day, when we stopped by flash drives ... we were debating how long until we see ESE (well Exchange specifically, though I'm more concered about laptops) on such drives. With the advent of larger and large capacity SSDs, and even commercial level drives like these, becoming ever more imminent, such a possibility looms nearer and nearer. And with a few ESE consumers on the client O.S. I expect we will be seeing our databases on SSDs on laptops within the year (maybe two).

We were discussing the various rumors about how fast they are at read vs. write, how low latency are the reads/writes, +how reliable+ (if you remember there was always this issue with earlier drives, SSD can only be written to so many times), how good at random IO, and other properties that would make them useful as Exchange storage, or at least whether they will be causing us problems on laptops. So curious was I, that of my own accord (and with my own money I'll note) I went ahead and bought about $100 worth of flash drives for running through our little JET engine, see if they survive ...

Here is a list of the drives I acquired (except the last one, being a control of sorts that I already owned):

Manufacturer	Model	Cap (GB)	Cost[1]	Warranty	MBs/$
SanDisk	Cruzer Micro Drive	1.00	$20	2 years	50.0
Kingston	DataTraveler	1.00	$20	5 years	50.0
Sony	MicroVault	0.25[2]	$18	1 year	13.8
PQI	U320 / Traveling Disk	0.50	$15	unknown[3]	33.3
Patriot	XPorter	4.00	$24[4]	5 years	166.7
AComdata	USB Hard Drive	320.00	$300	?3-5 years?	1066.7

[1] For all units, the cost was rounded up by 1 cent to non-retardo-consumer values. Tax was not added, but it applies here in Washington, 8-something% or so.
[2] I really wanted a larger Sony, but the 2 GB was $70 (too much), and I couldn't find a 512 MB or 1 GB unit anywhere on the shelf. Sigh. :P
[3] Documentation claimed it was on web site (www.pqimemory.com), but couldn't find it in a generous 40 seconds of searching.
[4] This one was an incredible value, it was actually $70 in the store, but until 3/5/2007(so only today, yeah sorry for the late notice) you get a $45 rebate if you buy it at Fry's.

All such drives claim USB 2.0 compliance on the packaging, except perhaps the USB Hard Drive, not sure about that one.

I don't want to rate these drives based on arbitrary tests[5], such as whether it comes with a wrist strap, retractable USB plug, etc ... we want our analysis to be on serious benchmarks that tell us if these have the capability to back a real ESE database load / support Exchange, and so we start our tests ...

Test 0; Part A: Getting the packaging off...

I've definately no love of the ridiculously tough plastic modlded packaging that these things come in ... and so I decided as the preliminary test in our evaluation we would test the difficulty of removing the packaging and installing the drive.

(lower is better)
4.0    Drive A (SanDisk Cruzer) had unusually tough to cut plastic, and an asymetrical shape as well (making getting the scissors to purchase hard) as very large borders making the entire thing very tough to cut open.
2.0    Drive B (Kingston DataTraveler) had somewhat better packaging, thin-ish borders, and slightly easier to cut, but 4 sided seals
2.8    Drive C (Sony MicroVault) Pretty easy to cut, reasonable sized shoulder, and only sealed on 3 sides (e.g. where the plastic is like heat fused).
3.4    Drive D (PQI U230) hardish to cut, but only sealed on 3 sides, last side is bent over, so easy to open after 3 sides.
4.4    Drive E (Patriot XPorter) had very tough packaging, the fused seal of plastic was right up against the part where the plastic turns vertical, so there was very little "shoulder" to cut along.
1.5    Drive F (AComdata USB Hard Drive) this was (IIRC) "saran wrapped" cardboard box, which is the ultimate in usable packaging.

Now, you might say this ranks up there with the wrist straps, and claim that my testing is not enough... no, no, if you have got to build an Exchange drive off 10s or 100s of these, you're going to start to care how hard it is to get them out of thier plastic, this would be a serious TCO issue.

Part B: Installing the drive ...
Installing the drive consisted of plugging in the drive, change to the drive ("g:" or "h:" usually),"mkdir \SSDBashData", "cd SSDBashData", and running my test script (I'll publish this soon) to make sure everything is working and it can at least store a results file (no verification that the file written is the file stored). Sort of a pre-race inspection.

3.0    SanDisk gets a base 2.5 for automatically installing (or maybe just running) software. And another .5 doc'd because of how it brought up both a E:\ (CD-ROM RO drive) and an H:\ (USB Mass Storage RW drive), that's just weird.
1.0    Kingston gets as good a score as I will give in this test, it came up promptly, didn't install any software, AND didn't even have any "cool programs" on the root of the drive. Kingston understands its just producing a stick of storage, and I appreciate that.
1.5    Sony gets .5 for having data in the root, but it didn't install / run anything by default that I can tell so that's all they get.
1.0    PQI like Kingston gets the a plain storage device. Good job.
2.0    Patriot gets a .5 for bringing up two drives like the SanDisk. And .5 because one of the drives had software in it. Interestingly both were RW, but the drive with software was only 24 MBs big, the other was 4 GBs.
1.0    AComdata didn't install anything, came right up, removed properly.

I wanted to complete this step also to verify I could use the drives w/o accepting any EULAs and checking the documentation that it didn't have any DeWitt clauses (the term for the clause in EULAs of products like SQL Server, Oracle, etc that say you can't publish benchmark / performance numbers of the product). I do not believe ESE has any such limitation by the way (but one would have to read the Windows OS or Exchange EULAs to be sure).

And as a plus to make sure that terrible spyware they are all bound to install is going to be universally installed when I get to performance tests, surprisingly only one seemed to (not that I looked very hard).

Conclusion:

Total Scores (lower is better):
SanDisk = 7.0
Kingston = 3.0
Sony       = 4.3
PQI   = 4.4
Patriot = 6.4
HD      = 2.5

Since the rest of the tests are likely to be based off metrics, copy times, IOPS, etc, these subjective and arbitrary test results will be used to break any ties, to allow the vendor feels maximally cheated in its ratings. ;) Next test up serial copy performance ...

Cheers,
BrettSh

[5] Yes I do. Seriously, the varying sizes of the drives may throw some of the tests way off, or be difficult to make any claims of scale ... if I were doing real testing, I would be making Microsoft pay for the hardware, getting those larger 32, 64, and 160 GB models mentioned in some of the above articles, and probably wouldn't be blogging about it. ;-) Sooo ... I don't actually intend to answer the question posed in the title, just sort of a check in to see where these cheap flash drives are ... and they all[3] have warranties so I can always recoup my investment if ESE makes them catch on fire.

I decided to check out the size of Eric's DIT ...

BrettSh — Mon, 12 Jun 2006 19:49:00 GMT

... take some time, measuring the exact dimensions of Eric's DIT ... and I must say, I've seen a fair amount of DITs in my time, and I can say with a fair amount of certainty, Eric's DIT is the biggest I've ever seen! I can't believe what a massively huge a DIT Eric has.

Note: while this is about an Active Directory database, Exchange is based on the same database technology, so it would (and does) have similar space hierarchy.

Table of ESE Space usage:

The black in this table is just the output from 3 of the columns of esentutl /ms adamntds.dit (original report), the blue are columns/rows I've added to break out the space usage in a clearer way:

Name	Friendly name	Owned	Available	Owned(GB)	Avail(GB)	b-tree lvls	% of DB	% of Total Idxs
	<calc: DB real>	268179344	6088	0.000	0.046
F:\DNT\adamntds.dit		268179344	218	2046.046	0.002

	<calc: datatable real>	268178751	5689	2046.041	0.043
datatable		268178751	2260	2046.041	0.017
	<calc: Row Data>	186707905	2260	1424.468	0.017	5	69.62%
<Long Values>		42	18	0.000	0.000	2	0.00%
	<sum: Idx Totals>	81470804	3411	621.573	0.026	0	30.38%	100.00%
PDNT_index	Int: PDNT + Name	11482791	7	87.607	0.000	5	4.28%	14.09%
nc_guid_Index	Int: NC + objGuid	10870892	5	82.938	0.000	4	4.05%	13.34%
INDEX_00090002	Att: objectGuid	10791866	3	82.335	0.000	4	4.02%	13.25%
INDEX_00000003	Att: cn	9658638	51	73.690	0.000	4	3.60%	11.86%
INDEX_00090001	Att: name	8999729	83	68.662	0.001	4	3.36%	11.05%
Ancestors_index	Int: Ancestry	7917036	10	60.402	0.000	4	2.95%	9.72%
DRA_USN_index	Int: Repl USN	7083583	251	54.043	0.002	4	2.64%	8.69%
INDEX_0009030E	Att: objectCategory	5274627	21	40.242	0.000	4	1.97%	6.47%
DRA_USN_CREATED_index	Int: Repl Created USN	4479144	34	34.173	0.000	4	1.67%	5.50%
INDEX_00020078	Att: uSNChanged	4279711	0	32.652	0.000	4	1.60%	5.25%
deltime_index	Int: deltime	371267	15	2.833	0.000	4	0.14%	0.46%
INDEX_00020030	Att: isDeleted	261493	2929	1.995	0.022	4	0.10%	0.32%

... deleted about a dozen small indices ...

I'll discuss the permutations I performed on the esentutl /ms output, in the hopes it will be clear ...

First I sum up the owned space for all indices in the datatable, this comes out to 81470804. Note the #'s above may not add up exactly because I deleted a dozen or so super small indices. I summed up all the indices because it makes the next calculation easier, and also so we can get the "% of Total Idxs" column as well.

So first understand that ESE's "owned" space is hierarchical, so the "datatable" owns all the space owned by each of the indices and the LV B-Tree in the datatable. But the primary B-Tree for the datatable also contains (and thus owns) the normal row data. So the real data that is in the regular row data for the datatable is 268178751 (datatable) - 42 (datatable's LV B-Tree owned) - 81470804 (owned by sum of all datatable indices) = 186707905 (i.e. the "<calc: Row Data>" line).

I then created a couple columns to turn this page counts into a usable unit (GBs), i.e. <# of pages> * 8 / 1024 / 1024.

Finally I added a friendly name column, so you'd know roughly what the index was indexing.

Some analysis:

From the above table we can easily see the row data 1,424 GBs and all the indices combined is 621 GBs. This breaks out like so:

Based on the table above this is showing us a full 30% of this database is indices!!! That's a huge amount. This isn't a common space breakdown for most AD objects, as the objects making up Eric's DIT are very very small / light weight. He was just creating containers w/ minimal attributes (see Eric's initial post), and so just the base set of indices on a basic object lead up to a significant portion of the objects overall "footprint" in the DB.

As for the breakup of the individual index usages, it looks something like this:

Of the secondary indices on the datatable, 10 are always updated! And another 2 (the very slender ones) are only updated on delete. Since there are over 2 billion objects in this database, that means we inserted about 22 billion B-Tree entries, kind of neat.

One last, somewhat technical thing that I think a few of you might find interesting, is that even the largest 1,424 GB primary B-Tree is only 5 levels deep. This means that to locate a specific row (by DNT) will only take 5 disk seeks in the worst case (cold cache). B-Trees have this very nice high fan out, that keeps disk seeks minimal.

Interestingly, I dumped the root page, and it only has 3 nodes (TAG 0 doesn't count), what this means is that we could add about 100x more data to this b-tree and there would be no increase in the # of disk seeks to fetch a row from this table.

Anyway that seems like enough for now ...

Writing a Unicode file via perl ...

BrettSh — Wed, 07 Jun 2006 20:25:00 GMT

Several months ago someone filed bugs across Windows Vista to make sure all performance monitoring .ini files were Unicode, so the files could be properly localized ("translated") to various languages (so we could have Korean, or Hindi descriptions). A noble goal to be sure.

For most people this was as easy as checking out the file for editing, opening it in notepad, doing a "save as", picking Unicode. ESE however has ALOT of perf counters (esp. when you Squeaky Lobster a machine - more on that later) so we use a perl script to generate several parts of the performance monitoring files, the .ini, the .hxx, and some fairly repatitive .cxx code that gets compiled into ESE binaries ... I know some of you are saying, "You can use perl on Windows?" ...

Anyway, I looked all over the internet, and couldn't even find help when I scoped to Mr. Unicode's blog ... then I posted to an internal alias on perl at Microsoft, and someone came to my rescue, since he said he didn't mind and I couldn't find it on the internet (at least at the time), I'd figured I'd post his comments ...

His comments, I wholesale included in our perl code (I had to read it twice to really grok how the ":raw" type parts were like piping through converting text commands, but in reverse so you read them right to left):

#
# Some notes from someone smarter than me about Perl and Unicode ...
# ----
#
# Which encoding do you want to use? UTF16-LE is the standard on Windows (nearly
# all characters are encoded as 2 bytes), UTF8 is the standard everywhere else 
# (characters are variable length and all ASCII characters are a single byte).
#
# Here's what I've figured out after lots of experimentation. To get UTF16-LE 
# output you need to play a few games with perl...
#
#   open my $FH, ">:raw:encoding(UTF16-LE):crlf:utf8", "e:\\test.txt";
#   print $FH "\x{FEFF}";
#   print $FH "hello unicode world!\nThis is a test.\n";
#   close $FH;
# Reading the IO layers from right to left (the order that they will be applied 
# as they pass from perl to the file) ...
#
# Apply the :utf8 layer first. This doesn't do much except tell perl that we're 
# going to pass "characters" to this file handle instead of bytes so that it 
# doesn't give us "Wide character in print ..." warnings.
#
# Next, apply the :crlf layer as text goes from perl out to the file. This 
# transforms \n (0x0A) into \r\n (0x0D 0x0A) giving you DOS line endings. Perl 
# normally applies this by default on Windows but it would do it at the wrong 
# stage of the pipeline so we removed it (see below), this is where it ought to 
# be.
#
# Next apply the UTF16-LE (little endian) encoding. This takes the characters 
# and transforms them to that encoding. So 0x0A turns into 0x0A 0x00. Note that 
# if you just say 'UTF16' the default endianness is big endian which is 
# backwards from how Windows likes it. However, because we're explicitly 
# specifiying the endianness perl will not write a BOM (byte order mark) at the 
# beginning of the file. We have to make up for that later.
#
# Finally, the :raw psuedo layer just removes the default (on Windows) :crlf 
# layer that transforms \n into \r\n for DOS style line endings. This is 
# necessary because otherwise it would be applied at the wrong place in the 
# pipeline. Without this the encoding layer would turn 0x0A into 0x0A 0x00 and 
# then the crlf layer would turn that into 0x0D 0x0A 0x0A and that's just goofy.
#
# Now that we've got the file opened with the right IO layers in place we can 
# almost write to it. First we need to manually write the BOM that will tell 
# readers of this file what endianness it is in. That's what the 
# print $FH "\x{FEFF}" does.
#
# Finally we can just print text out.
#
# If you want UTF8, I'm pretty sure it's a lot easier. Also, this is also a lot 
# easier on unix, the CRLF ordering problem is definitely a bug but the default 
# to big endian (and ensuing games to get the BOM to output without a warning) 
# are by design. I'm pretty sure that none of the core perl maintainers use perl 
# on Windows (even though at least one keeps perl on VMS working...).
#

#
# Until Exchange decides it wants a Unicode eseperf.ini, we're going to generate
# the old ASCII one.  Also if Exchange wants one, it will have to update it's
# version of Perl to understand the open modes we're using below.  Currently we
# get this error:
#	1>Unknown open() mode '>:raw:encoding(UTF16-LE):crlf:utf8' at .\perfdata.pl line 325,  line 6189.
#


if ( $ESENT ){ #ifdef ESENT 

	open( INIFILE, ">:raw:encoding(UTF16-LE):crlf:utf8", "$INIFILE" ) || die "Cannot open $INIFILE: ";
	print INIFILE "\x{FEFF}";  # print BOM (Byte Order Mark) for the unicode file

} else { #else

	open( INIFILE, ">$INIFILE" ) || die "Cannot open $INIFILE: ";

} #endif

The code worked like a charm, yeah Unicode esentprf.ini. Well, until I sync'd the code to Exchange then it broke, that is the source of the "if ( $ESENT )" which is only defined when we build the ESE sources for Windows. I should mention in closing that I know this code works for perl 5.8.7, and I know it does not work for perl 5.6.1. I've heard the perl support got much better in 5.8 or so...

Oh I guess that's code, so I'm required to say something like:
Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm (I'm having a hard time imagining how such a small snipit could be subject to that, but whatever).

Oh here is what the BOM is, and more on the BOM.

Update 2006/08/20: Turned out Exchange wanted a Unicode eseperf.ini after all, and has updated thier version of perl, so good news the NT - Ex code bases grow that much more similar.

thread degraded mode ... the sequel.

BrettSh — Sun, 19 Mar 2006 12:08:00 GMT

I've run in thread degraded mode for 2 or 3 weeks without a hitch before, this time I wasn't so lucky. After about 6 hours explorer spun up taking 50% CPU (given it's a HT machine, that usually means 1 thread spinning endlessly). Heuve! That just won't do. But we can merely degrade this thread too ... once we find it ...

So I ctrl-C in the debugger window that got started from part 1 ...

Alternatively: If you don't have debugger attached up to the process yet, you can do so, by going to the C:\debuggers directory (expained in part 1), typing "tlist" to find the Process ID (PID) of the explorer.exe process, and then run "ntsd -p <PID>".

At the debug prompt we type "!runaway", which gives you something like this:
    0:076> !runaway
     User Mode Time
    Thread       Time
    73:1acc      0 days 6:36:51.328
       1:ec0       0 days 1:39:27.203
    15:1ac       0 days 0:06:01.640
    12:f94       0 days 0:05:44.281
    ... deleted the other ~80 threads ...
The time column is cumulative CPU time the thread has used.

At this point you 'g' the debugger, wait for a short timed interval, then hit ctrl-C again to re-break into the debugger. I waited 30 seconds myself, and then re-run the !runaway command ...
    0:082> !runaway
     User Mode Time
    Thread       Time
    73:1acc      0 days 6:37:21.359
       1:ec0       0 days 1:39:28.406
    15:1ac       0 days 0:06:01.671
    12:f94       0 days 0:05:44.281
    ...
You can see from the blue that thread 73 is our culprit, as it's cumulative CPU time went up by nearly exactly 30 seconds. Note the culprit thread isn't guaranteed to be the top thread, but it was in my case.

So the "~f" command from the first blog affects the "current" thread in the debugger, which you can see above is thread 82, not exactly, well ... not at all what is desired. We want thread 73 to be frozen though, so here is how you freeze a specific thread:
    0:082> ~73f
    0:082> g
Now I'm back to normal, CPU settles down, go back to work.

Sometime later the next thing is I AV'd when I typed a search in the MSN Desktop Search box on the task bar see the bottom of my start bar ... since MSN DS was the source of the original issue, not sure why I'd have expected that to work (I can be really stupid sometimes, I'll blog more about that), one more thread to freeze and then 'g' the process.

The last thing that AV'd is <window>-E. Don't know why that AV'd, but don't care I can live without file explorer.

Though I am not sure I can live without desktop search ... I sense a reboot is in my future ...

Try explorer's thread degraded mode ...

BrettSh — Fri, 17 Mar 2006 04:29:00 GMT

On any given day that is past say 7 days of uptime, I have 100 - 300 windows open, not kidding, here is a shot of my current task bar ... there shouldn't be anything msft confidential there, at least that you'd actually be able to read more than 4 or 5 letters of ... I know by heart I have 41 rows, so that's 204 windows open there ... I turn off that "group similar taskbar buttons" "feature", and the buttons show up in order, so a given "job" usually has task buttons around each other (in fact the last 6 buttons there are for this blog post) ... so what does this have to do with explorer ...

The basic upshot of this computing lifestyle choice is that my heart skips a beat and then visceral pain sets in whenever explorer AVs (Access Violation) ... explorer is what controls the start bar, and when it restarts the task buttons will be in a random order ... for the *nix types, this is like your window manager core dumping ... it's awefulness.

Right so getting to thread degraded mode ... my own term, for when you simply freeze the AV'd thread in a process, and allow the process to continue on its merry way. You can do this because maybe the thread may not be doing something particularly useful, ergo it is "not a very serious AV", or maybe call it a "slight AV". The process often (sometimes?) continues to function.

How to use thread degraded mode:

First, you will have to prepare your machine for initiating thread degraded mode ...

You will need to get a user mode debugger (there may already be a ntsd.exe in your system32, which should work, but no one uses that anchient one) go get a good version, which for an x86 box installs from this exe (i think). Install it to C:\debuggers, everyone else around here seems to.

Navigate to this registry key (read more about it):
"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug"
Create registry string (REG_SZ) value with a name of "Debugger" and value of
"C:\debuggers\ntsd -p %ld -g -G -e %ld"
If it already has a value you may want to save it.

You only have to do those steps once, and now you are ready to run in thread degraded mode, if the need should arise.

The next time explorer (or any application) crashes / AVs on you, you will get an option to debug the process ... select "Debug" or "Yes" or whatever ... this will open a debugger attached to explorer ( you'll probably have to alt-tab to find this new debugger window, because the task bar will be frozen/unresponsive while you debug it ;).

The debugger will open with the AV'ing thread as the current thread, so use "~f<enter>" (that is a tilde), to freeze this thread. Then "g<enter>" will let the task bar come back to you (maybe). At this point you should be praying that the thread you froze isn't holding any crucial critical sections or locks, and that things will return to "normal" ... your mileage may vary ... greatly.

It will look like this:
    0:008> ~f
    0:008> g

After you 'g' it, it will start printing this kind of thing in the debugger ...
    System 0: 1 of 84 threads are frozen
    System 0: 1 of 84 threads were frozen
    System 0: 1 of 84 threads are frozen
That's just explorer letting you know it loves you for not letting go, and putting it on life support.

Oh the crash was in MSN Desktop Search, but I don't fault (intended ;) them because I'm running the first beta of the software released in Dec 2004, I've heard they've had an update since then.

Anyway, as of approximately 8:20 AM (PST) yesterday (wed) morning, I've been running in thread degraded mode ... as I finish this post I've got 4 frozen threads ... there were a few more threads with "issues" but I don't have time to blog about them right yet ...