183

Is solid state up to an ESE challenge?

~Eric and I were at Fry's the other day, when we stopped by flash drives ... we were debating how long until we see ESE (well Exchange specifically, though I'm more concered about laptops) on such drives.  With the advent of larger and large capacity SSDs, and even commercial level drives like these, becoming ever more imminent, such a possibility looms nearer and nearer.  And with a few ESE consumers on the client O.S. I expect we will be seeing our databases on SSDs on laptops within the year (maybe two).

We were discussing the various rumors about how fast they are at read vs. write, how low latency are the reads/writes, +how reliable+ (if you remember there was always this issue with earlier drives, SSD can only be written to so many times), how good at random IO, and other properties that would make them useful as Exchange storage, or at least whether they will be causing us problems on laptops.  So curious was I, that of my own accord (and with my own money I'll note) I went ahead and bought about $100 worth of flash drives for running through our little JET engine, see if they survive ...

Here is a list of the drives I acquired (except the last one, being a control of sorts that I already owned):

 [1] For all units, the cost was rounded up by 1 cent to non-retardo-consumer values.  Tax was not added, but it applies here in Washington, 8-something% or so.
 [2] I really wanted a larger Sony, but the 2 GB was $70 (too much), and I couldn't find a 512 MB or 1 GB unit anywhere on the shelf.  Sigh.  :P
 [3] Documentation claimed it was on web site (www.pqimemory.com), but couldn't find it in a generous 40 seconds of searching.
 [4] This one was an incredible value, it was actually $70 in the store, but until 3/5/2007(so only today, yeah sorry for the late notice) you get a $45 rebate if you buy it at Fry's.

All such drives claim USB 2.0 compliance on the packaging, except perhaps the USB Hard Drive, not sure about that one.

I don't want to rate these drives based on arbitrary tests[5], such as whether it comes with a wrist strap, retractable USB plug, etc ... we want our analysis to be on serious benchmarks that tell us if these have the capability to back a real ESE database load / support Exchange, and so we start our tests ...

Test 0; Part A: Getting the packaging off...

I've definately no love of the ridiculously tough plastic modlded packaging that these things come in ... and so I decided as the preliminary test in our evaluation we would test the difficulty of removing the packaging and installing the drive.

 (lower is better)
4.0    Drive A (SanDisk Cruzer) had unusually tough to cut plastic, and an asymetrical shape as well (making getting the scissors to purchase hard) as very large borders making the entire thing very tough to cut open.
2.0    Drive B (Kingston DataTraveler) had somewhat better packaging, thin-ish borders, and slightly easier to cut, but 4 sided seals
2.8    Drive C (Sony MicroVault) Pretty easy to cut, reasonable sized shoulder, and only sealed on 3 sides (e.g. where the plastic is like heat fused).
3.4    Drive D (PQI U230) hardish to cut, but only sealed on 3 sides, last side is bent over, so easy to open after 3 sides.
4.4    Drive E (Patriot XPorter) had very tough packaging, the fused seal of plastic was right up against the part where the plastic turns vertical, so there was very little "shoulder" to cut along.
1.5    Drive F (AComdata USB Hard Drive) this was (IIRC) "saran wrapped" cardboard box, which is the ultimate in usable packaging.

Now, you might say this ranks up there with the wrist straps, and claim that my testing is not enough... no, no, if you have got to build an Exchange drive off 10s or 100s of these, you're going to start to care how hard it is to get them out of thier plastic, this would be a serious TCO issue.

Part B: Installing the drive ...
Installing the drive consisted of plugging in the drive, change to the drive ("g:" or "h:" usually),"mkdir \SSDBashData", "cd SSDBashData", and running my test script (I'll publish this soon) to make sure everything is working and it can at least store a results file (no verification that the file written is the file stored).  Sort of a pre-race inspection.

3.0    SanDisk gets a base 2.5 for automatically installing (or maybe just running) software.  And another .5 doc'd because of how it brought up both a E:\ (CD-ROM RO drive) and an H:\ (USB Mass Storage RW drive), that's just weird.
1.0    Kingston gets as good a score as I will give in this test, it came up promptly, didn't install any software, AND didn't even have any "cool programs" on the root of the drive.  Kingston understands its just producing a stick of storage, and I appreciate that.
1.5    Sony gets .5 for having data in the root, but it didn't install / run anything by default that I can tell so that's all they get.
1.0    PQI like Kingston gets the a plain storage device.  Good job.
2.0    Patriot gets a .5 for bringing up two drives like the SanDisk.  And .5 because one of the drives had software in it.  Interestingly both were RW, but the drive with software was only 24 MBs big, the other was 4 GBs.
1.0    AComdata didn't install anything, came right up, removed properly.

I wanted to complete this step also to verify I could use the drives w/o accepting any EULAs and checking the documentation that it didn't have any DeWitt clauses (the term for the clause in EULAs of products like SQL Server, Oracle, etc that say you can't publish benchmark / performance numbers of the product).  I do not believe ESE has any such limitation by the way (but one would have to read the Windows OS or Exchange EULAs to be sure).

And as a plus to make sure that terrible spyware they are all bound to install is going to be universally installed when I get to performance tests, surprisingly only one seemed to (not that I looked very hard).

Conclusion:

Total Scores (lower is better):
  SanDisk  = 7.0
  Kingston  = 3.0
  Sony       = 4.3
  PQI         = 4.4
  Patriot     = 6.4
  HD          = 2.5

Since the rest of the tests are likely to be based off metrics, copy times, IOPS, etc, these subjective and arbitrary test results will be used to break any ties, to allow the vendor feels maximally cheated in its ratings. ;)  Next test up serial copy performance ...

Cheers,
BrettSh

[5] Yes I do.  Seriously, the varying sizes of the drives may throw some of the tests way off, or be difficult to make any claims of scale ... if I were doing real testing, I would be making Microsoft pay for the hardware, getting those larger 32, 64, and 160 GB models mentioned in some of the above articles, and probably wouldn't be blogging about it. ;-)  Sooo ... I don't actually intend to answer the question posed in the title, just sort of a check in to see where these cheap flash drives are ... and they all[3] have warranties so I can always recoup my investment if ESE makes them catch on fire.

Massive power outage == Adventure Friday (posthumously blogged 12/16 when my server is back up) ...

As we woke up Friday morning, our bedrooms had that certain briskness in the air that indicates only one thing ... power outage.  Nika (a friend/roommate) and his kid went to school to see if there is any school, power out there too.  We got a hold of another friend, Mary & Tony (Tony also works at Microsoft), power out at thier house and Microsoft.  As we were listening to the news and driving around we got to get the idea that pretty much everything was down, and it turned from "huh, I wonder if work is going to come up" day into Adventure Day lets see where power is, and find a warm place for Nika and Mary and the kids ...

Some data:

• As of Friday morning, reportedly 1.3 M customers (is that homes/businesses or people I could never tell) were without power.
• We heard there were multiple hospitals without grid power, and at this point you realize how far your work (even if it is Microsoft) is like down the priority list.
• News also said 80% of Bellevue was without power.
• PSE specifically had 700k down (the area that services my house) of ~1M customers.
• Some 90 of 150 substations were down.
• And 80 transmission line (over 1/2 of such lines) were damaged.
• Apparently they've flown/flying in 100+ repair crews from other states.
  
• We saw literally dozens of fallen trees close to our house ...
• some of the streets were practically green with blown off tree branches. 
  
• We ended up driving over the tops of 4 or so trees fallen across the road.
• A house about 6 doors down, had a tree through its roof.
• Along one road going by Microsoft we saw 7 down trees within a few blocks (I'm sure there is a [un]fair joke there at Microsoft's expense ... ;-)
• Gusts to 60 - 80 mph in the general area during the wind storm on Thur night / Fri AM.

I love stats, but I don't think any of these put it in as much perspective as the traffic flow maps (which I figured out were a GREAT way to figure out if an area likely has power!), here is the seattle area map @~noon today.  Yesterday however was an even more severe story:

Since you can differentiate from No Equipment and No Data, we guessed No Data meant, No Power.  Nearly everything was out for I dunno, maybe 50 miles (including all of Microsoft's main campus that we heard about) of solid residential setting.  We drove for about 20 miles (which took 1.5 hours) without seeing a single spot of power that morning as we maneuvered North around the lake to get to friends w/ power/heat.  That's how massive it was ...

The funniest stories of the day ...

Eric went for a walk, came by my house (ironically while we were knocking on his door to see if he wanted to go North to heat with us :P) and Tony's house. Anyway, while he was at Tony's house he saw a crew clearing a tree from the road.  Now Tony's house was fine, but clearly one of the wires to his house was slack and low hanging over the road, pulled down by I believe some blown off branch in the middle of the night.  After the tree was cleared from the road, Eric saw a Comcast Cable truck try to sneak under it, and caught it on their ladder or roof rack, and nearly rip it off the pole. The two guys got out of the van, looked up at the pole / wire, and this was the approximate gist of the conversation:
    Guy 1: "Awww crap ... its cable."
    Guy 2:  "We should probably fix that."
Tony later came home and found them fixing his cable ... my they can be expedient ... of course they couldn't check it worked, no power.

We later got to validate that indeed the traffic maps were the best foreteller of power, because we heard on the news that Bellevue Square Mall was out, but Lincoln Center across the street had power ... however according to the downtown Bellevue traffic flow map, power was out at Lincoln Center and came back 1 block to the East ... later we went there to see a movie, and indeed Lincoln Center had only some sort of emergency lighting on, but not real power, such as to see a movie, but enough for Nika to play thier Grand Piano, which then security got VERY PISSY about, I think he thought we were young holigans or something, but Nika actually knows how to play pleasant Nordstrom style / elevator back ground music ... luckily the Galleria (which is coincidentally one block to the East ;) had power, so we were not thwarted in our movie quest!  We saw, "Bobby", it was a bit slow but ok.

It was rumored (though don't know it was actually true) said there were only like 3 working gas stations in downtown Bellevue (that's the green near the yellow camera and bellevue sign in the map), and we heard police had to be deployed to keep order at one point, and direct traffic ... they at one point had sectioned off the street for several blocks to the south, just to create a line into the gas station, and the wait was apparently ridiculous.  This wasn't a problem for us because we decided to wait until after our movie (@ midnight Sat) before visiting, so we ended up waiting maybe 10 minutes.

The # of customers (or people?) dropped to 900k without power by the end of Friday, according to the news.  Lastly when we came home that night, we checked our mailbox, and indeed some US Postal mail was there, hows the motto go, "neither rain, nor sleet, nor snow..." ... well they should add wind if it isn't there already.  Apparently our Govenor declared a state of emergency, I think today / saturday.

Cheers,

BrettSh (msft) 

[1] Not that I do or don't believe in God, I'm just saying ...TEN TREES!?

Update: MartinC showed me that wsdot has historical data, so I replaced my own previously doctored version with the actual map at 11:20 AM on Friday, it is even more effective at showing what it looks like to have 1.3 M customers without power.

I decided to check out the size of Eric's DIT ...

(i.e. the "<calc: Row Data>" line).

I then created a couple columns to turn this page counts into a usable unit (GBs), i.e. <# of pages> * 8 / 1024 / 1024.

Finally I added a friendly name column, so you'd know roughly what the index was indexing.

Some analysis:

From the above table we can easily see the row data 1,424 GBs and all the indices combined is 621 GBs.  This breaks out like so:

Based on the table above this is showing us a full 30% of this database is indices!!!  That's a huge amount.  This isn't a common space breakdown for most AD objects, as the objects making up Eric's DIT are very very small / light weight.  He was just creating containers w/ minimal attributes (see Eric's initial post), and so just the base set of indices on a basic object lead up to a significant portion of the objects overall "footprint" in the DB.

As for the breakup of the individual index usages, it looks something like this:

Writing a Unicode file via perl ...

#
# Some notes from someone smarter than me about Perl and Unicode ...
# ----
#
# Which encoding do you want to use? UTF16-LE is the standard on Windows (nearly
# all characters are encoded as 2 bytes), UTF8 is the standard everywhere else 
# (characters are variable length and all ASCII characters are a single byte).
#
# Here's what I've figured out after lots of experimentation. To get UTF16-LE 
# output you need to play a few games with perl...
#
#   open my $FH, ">:raw:encoding(UTF16-LE):crlf:utf8", "e:\\test.txt";
#   print $FH "\x{FEFF}";
#   print $FH "hello unicode world!\nThis is a test.\n";
#   close $FH;
# Reading the IO layers from right to left (the order that they will be applied 
# as they pass from perl to the file) ...
#
# Apply the :utf8 layer first. This doesn't do much except tell perl that we're 
# going to pass "characters" to this file handle instead of bytes so that it 
# doesn't give us "Wide character in print ..." warnings.
#
# Next, apply the :crlf layer as text goes from perl out to the file. This 
# transforms \n (0x0A) into \r\n (0x0D 0x0A) giving you DOS line endings. Perl 
# normally applies this by default on Windows but it would do it at the wrong 
# stage of the pipeline so we removed it (see below), this is where it ought to 
# be.
#
# Next apply the UTF16-LE (little endian) encoding. This takes the characters 
# and transforms them to that encoding. So 0x0A turns into 0x0A 0x00. Note that 
# if you just say 'UTF16' the default endianness is big endian which is 
# backwards from how Windows likes it. However, because we're explicitly 
# specifiying the endianness perl will not write a BOM (byte order mark) at the 
# beginning of the file. We have to make up for that later.
#
# Finally, the :raw psuedo layer just removes the default (on Windows) :crlf 
# layer that transforms \n into \r\n for DOS style line endings. This is 
# necessary because otherwise it would be applied at the wrong place in the 
# pipeline. Without this the encoding layer would turn 0x0A into 0x0A 0x00 and 
# then the crlf layer would turn that into 0x0D 0x0A 0x0A and that's just goofy.
#
# Now that we've got the file opened with the right IO layers in place we can 
# almost write to it. First we need to manually write the BOM that will tell 
# readers of this file what endianness it is in. That's what the 
# print $FH "\x{FEFF}" does.
#
# Finally we can just print text out.
#
# If you want UTF8, I'm pretty sure it's a lot easier. Also, this is also a lot 
# easier on unix, the CRLF ordering problem is definitely a bug but the default 
# to big endian (and ensuing games to get the BOM to output without a warning) 
# are by design. I'm pretty sure that none of the core perl maintainers use perl 
# on Windows (even though at least one keeps perl on VMS working...).
#

#
# Until Exchange decides it wants a Unicode eseperf.ini, we're going to generate
# the old ASCII one.  Also if Exchange wants one, it will have to update it's
# version of Perl to understand the open modes we're using below.  Currently we
# get this error:
#	1>Unknown open() mode '>:raw:encoding(UTF16-LE):crlf:utf8' at .\perfdata.pl line 325,  line 6189.
#


if ( $ESENT ){ #ifdef ESENT 

	open( INIFILE, ">:raw:encoding(UTF16-LE):crlf:utf8", "$INIFILE" ) || die "Cannot open $INIFILE: ";
	print INIFILE "\x{FEFF}";  # print BOM (Byte Order Mark) for the unicode file

} else { #else

	open( INIFILE, ">$INIFILE" ) || die "Cannot open $INIFILE: ";

} #endif

Ready to feel old ... USSR was soooo last century ...

My mom teaches high school physics.
She said that most of her students haven't even heard of the Soviet Union. (The exceptions are the ones who happened to take history.)

I realize that school education is far from balanced, but I'm a bit surprised that popular culture doesn't mention it more (e.g. movies like Hunt for Red October).

-martin

thread degraded mode ... the sequel.

Alternatively: If you don't have debugger attached up to the process yet, you can do so, by going to the C:\debuggers directory (expained in part 1), typing "tlist" to find the Process ID (PID) of the explorer.exe process, and then run "ntsd -p <PID>".

Try explorer's thread degraded mode ...

Show me the backups (win2k3 sp1)...

C:\bin\rel\win2k3\sp1\x86fre>repadmin.exe /showbackup mycorp-dc-02

Loc.USN                          Originating DC   Org.USN  Org.Time/Date          Ver Attribute
=======                          =============== ========= =============          === =========
DC=DomainDnsZones,DC=MyCorp,DC=com
329205835     084f51ed-d53e-4bad-83db-28694870fdb9 127958011 2006-02-08 02:51:22  197 dSASignature
DC=ForestDnsZones,DC=MyCorp,DC=com
329258203     084f51ed-d53e-4bad-83db-28694870fdb9 127958010 2006-02-08 02:51:22  202 dSASignature
CN=Schema,CN=Configuration,DC=MyCorp,DC=com
330447680     e0cc9580-1546-4da9-af2b-0929c37a378a  68598018 2006-02-09 02:14:56  897 dSASignature
CN=Configuration,DC=MyCorp,DC=com
330447359     e0cc9580-1546-4da9-af2b-0929c37a378a  68598017 2006-02-09 02:14:56  898 dSASignature
DC=MyCorp,DC=com
329205750     084f51ed-d53e-4bad-83db-28694870fdb9 127958006 2006-02-08 02:51:20  205 dSASignature
DC=UnLovedOlderChild,DC=MyCorp,DC=com
DC=UnLovedYoungerChild,DC=MyCorp,DC=com

• The above command should've resolved the Orig DC invocation IDs into DC names, so you could know where that backup was taken.  That's just fricken sloppy, sorry about that.  I really piss me off sometimes.
• In retrospect it would have been better to add another partition test to dcdiag.  That would've been way sweeter, fails if over timestamp latency, and /v would print out how old the last backup is, and what DC it was taken on.
• Would've been cool to add to ntdsutil the ability to control the backup latency event's sensitity on a per partition basis.  The feature has this ability today it is just not exposed, instead you've got to use a reg key (which I don't recommend).
  
• This was actually from a corp DC, but I changed the names ... but it makes me wonder if that's right our child domains aren't being backed up, or there is a bug in the tool / mechanism?  Those are partial replicas though, it might not being working on partial replicas ... that's an excercise for the reader ... let me know.