In Part 1, I discussed a bit of the history and function of SMIs.  How does this make them EEEEVIL, is the question?

Essentially, SMIs are the final word in what happens on a CPU, outside of removing power.  They cannot be interrupted, even by a Non-Maskable Interrupt (NMI).  Also, since they are not assertable from within software, it's impossible to use them or detect when they happen.  Essentially, the BIOS has control over everything that happens when it takes over.  Since it is it's own execution mode, the assumptions and mechanisms of the previous ones are ignored.  Specifically, this means any hardware breakpoints you may have set in your debugger will not fire based on anything that is happening in SMM. 

Now, when SMM was originally used only to implement power savings via Advanced Power Management (APM), this wasn't a huge problem.  When it became a problem was when BIOS makers and their OEM's started using this ability to implement other functionality via SMM trickery.  The most common application is implementing a USB keyboard handler for real-mode operation.  This also happens to be one of the most frustrating issue we see, as it can cause any variety of problems with the system's normal operation.

To understand why, think of the implications of an undetectable Hypervisor mode that has full access to the system.  Necessarily, to implement a keyboard handler like that, it needs to touch the hardware.  This means meddling with registers on devices, and even physical memory.  Now if you implement the perfect SMM handler for this kind of work, fine.  If you have a bug however, havoc can ensue.  You can be running along in a critical, essentially non-preempt-able code path, and from one instruction to the next have a section of memory or a hardware register changed out from under you.  This can result in all kinds of strange issue, from a crashed application, to a bluescreen, to a hang.

I'll cover some of the more common problems and symptoms in another article.

Want to know why I started posting again just now? 

Adi Oltean posted a great entry about his favorite hardware bug.  This prompted Larry Osterman to post his favorite, and I started feeling left out.  I have a ton to choose from, after all I deal with new bleeding edge hardware on a daily basis. I'm hard pressed to call them my favorites, because they generally cause me to suffer, but it's my job so I can't complain too much.   They can be a ton of fun to work on too, much more abstract and asynchronous than just working on dry code. 

Unfortunately, I can't really share the best ones since they're still pretty fresh on the market.  All the stories would start like this:  There was this one time a few <CENSORED> ago, when our OEM, <CENSORED>, was about to ship a new <CENSORED>, and it had this really strange behavior when you <CENSORED>...and end like this:  In the end, we fixed it.  OR: In the end, they hoped not many people would see it and just shipped anyway. 

Riveting stuff, eh?  I know I could be more generic, but since I work with these vendors on a daily basis I don't want to chance it.  I can tell you what is the bane of any operating system developer: System Management Interrupts (SMIs)  BIOSes that make heavy use of SMIs can wreak havoc with an OS, and there's very little in the way of figuring it out, and no way to stop it except a new BIOS.  I'll go into the pain and misery of SMIs in my next post.

I know this is a slightly more esoteric topic, even for me, but I want to address cc:NUMA platforms, and how they matter to Windows and Windows applications.  What is NUMA you ask?  NUMA stands for Non-Uniform Memory Architecture.  (The cc: stands for Cache Coherent, by the way, because there is non-cache coherent NUMA as well, but I won't address that here since there are no Windows support platforms that are non-cache coherent.)

To understand why NUMA exists, we need to look at Symmetric Multiprocessing (SMP).  SMP has a few core principles it is built around, and one is that every CPU in the system has an identical view of the system. Memory, I/O subsystem, and other CPU's can all be treated the same by software.  The problem comes when this assumption is no longer true.  As you scale up the size of a system, it becomes harder and harder to keep everything close together, literally.  The more switches and busses your data flows through, the longer it takes.

This fact means that in order to squeeze the maximum amount of performance out of the system, it behooves the OS as well as the programmer to try and keep data as close to the place where it's needed as possible.  By keeping track of which pages of memory and CPU's have the best locality to each other, decisions can be made when threads are scheduled and memory allocated that will squeeze that extra little bit out of the system.

Until only a few years ago, this was exclusively the realm of large mainframe style computers, not the PC world.  But with the introduction of the Unisys ES7000 in 2000, the PC suddenly had something to benefit by being NUMA aware.  Even then, this was something that mostly concerned large scale-up server implementations, not the average user or programmer.  That is, until AMD announced their unique implementation of their new Opteron and Athlon64 processors.  Suddenly, any system that has more than one of those CPUs could potentially benefit from NUMA optimizations.  I'll go into why in the next entry.

Ok, my arm is warm now.  Time to start tossing some theory bombs out there, and hope none get picked off.  They said Italians couldn’t quarterback, but look at Vinny Testaverde!  (Err…no, don’t.)

The reason treating support processes like a manufacturing endeavor fails is because it doesn’t take into account the sheer mass of uncontrolled variables that go into fixing software, IMO.  In an ideal world, you could have an unlimited number of truly gifted people performing every job function.  In the real world, you have to balance the needs for customer service with technical savvy, troubleshooting with good communication, prompt response with cost, etc.  People aren’t machines.

Taking a deterministic approach to creating your workflows means “sticking your head in the sand” to the variables out of your control, and overloading the system should a bottleneck occur.  Example:  If you require all cases go to your next tier of support at N days, but you neglect to factor that all of Europe takes August off, your small, specialized upper tier will be overloaded with issues from people who aren’t answering phones, (and when they do it will all be on the same day.)

Going in the opposite direction is fraught with peril as well.  As anyone who has worked in a helpdesk or support environment knows (at least from the armchair quarterback standpoint), the only consistently effective way to improve the quality of support is to have more people available to handle your volume, many of whom are aces at what they do.  As anyone who has been involved in managing a project like that knows, doing so is so expensive that you’d trash any profit your product might make.  So the question is how to strike the right balance.

The problem is, I don’t think there’s a good answer, at least not in terms of traditional support.  The real answer lies with software.  I’ll go into that next.

This is something that most people in the mainframe business have taken fom granted for decades now.  To the PC world, it’s relatively new…and to the PC OS world, even newer.

Starting with the Pentium and Pentium Pro, Intel introduced the Machine Check Architecture (MCA), which was a way for the CPU and other components of the system to report internal inconsistencies to software, so that the operating system can make decisions about how best to protect the user and data and/or report the problem.  For full information on how this works, see the IA-32 Architecture Software Developer’s Manual Volume 3: System Programming Guide, Chapter 14.

Now, that’s all well and good, but unfortunately Windows didn’t support anything but the most basic level of reporting until Server 2003.  Before that release, we would stop the system is a fatal error occurred, but not much else.  With Server 2003 however, the reporting mechanism became more sophisticated. 

If your processor and platform support it, we can read and log events into the event log to tell you more clearly what happened.  This might seem redundant, but not all Machine Check Exceptions (MCE) are fatal.  Some are just informative.  For example, you could have one particular region of memory that keeps returning corrected parity errors.  Corrected is great, no problems with your data.  The fact that they keep happening?  Usually bad news.  Go get it replaced! 

The worst-case scenario, of course, is an unrecoverable error.  Those are reported with a STOP 0x0000009C.  If you encounter one of these, it’s best to contact your OEM instead of Microsoft.  There’s really nothing we can do.  This is a hardware problem, always.  We might be able to help interpret, but it’s not likely.  If the system is critical, get to hardware swapping.

A comment from the earlier memory management entry posed a good question.  How does PAE factor into the new No Execute (NX) mechanism enabled by the Opteron, Athlon64, and new Prescott-based Xeon?

In Windows XP SP2 and Server 2003 SP1, the two are inexorably linked.  The two level address translation scheme used by the non-PAE kernel does not have enough room to accommodate any further descriptive information about individual pages of memory.  The three-level scheme that PAE necessitates allows the new NX attribute to be used.  (It is simply a bit in the Page Table Entry (PTE) that indicates the memory in this location is not allowed to be referenced by the instruction pointer.  “Do Not Run Under Penalty of Death”.)

When you use the /NoExecute switch on these OS’s, ntldr now loads the PAE kernel, but in a special mode.  You don’t get access to over 4GB of RAM, and more importantly, your drivers don’t get physical addresses over the 4GB mark either.  This is important because we’ve found that many devices and device drivers, especially in the consumer space, happily assume they’ll never have to address memory at an address over the 4GB boundary.

While you may only get to use a total of 4GB RAM in XP, that doesn’t mean that some of it can’t have a physical address above the 4GB boundary.  The BIOS or devices may re-map memory up there with the assumption that it won’t be seen or used.  When the PAE kernel starts handing out addresses to those pages of memory, things can get ugly.  The easiest way to ensure everything works like it did in the past, while allowing the new feature, is to make sure we don’t hand out any addresses over that boundary. 

If you add the /PAE switch, you get the normal PAE behavior, and all bets are off.  Of course, this is exactly what you want in the server space; after all, you got that extra RAM for a reason, right?  Also note that the properties aren’t transitive.  While both /PAE and /NoExecute use the same kernel file (ntkrnlpa.exe or ntkrpamp.exe) and address translation mechanism, you need both switches in place to enable both features.

[Added 8/6/2004 @ 2:56PM, thanks to Adam]

Note that the above information applies only to NX on processors in x86 mode.  The IA-64 and the x64 platforms natively support NX, which is being enabled for the first time with the release of Server 2003 and XP for 64bit.  Those platforms both already use a 3-level address translation scheme, but they're not related to PAE in any way.  They address 64bits natively, and the memory structures have room to include the NX information.  We just needed to add the support to the OS.

IMO, it's not what anyone else might think.  SQL, Exchange, and Web Services get all the hype, but I think Terminal Services will get the most immediate benefit from the backwards-compatible nature of the x64 architecture.  Let's look at some of the benefits that the platform provides over x86 and IA-64:

• Huge VA memory space for the whole system.  The current limits for x86 TS implementations is limted kernel memory space.  Once you run out, that's it...no more users.  What surprises many people is that it often happens well before 4GB of RAM gets exhausted.  Forget about /PAE, that just makes the problem worse.
• Full-speed execution of 32bit x86 code, with full Win32 support.  This means no one has to port business apps to see the benefit of the new platform.  You can move to it on your own schedule.  This is the way most Enterprises want to work.
• Shared pages for 32bit code.  Besides the slow emulation solution for current IA-64 platforms, there's also a problem in that there are no shared code pages, due to the page size difference from x86 to IA-64 (4kb for x86 vs. 8kb for IA-64).  That means every copy of Outlook needs 150MB, whereas on a shared memory system, many of those pages could be used by different processes for hundreds of users.
• No Execute (NX) means that there's an added level of security that wasn't available previously.  This might cause heartache for some in-house apps, but thanks to the configurable nature of the NX support, most companies can use this feature to their benefit, without having to purchase all new workstations.

I don't have numbers, but I expect that you would get FAR more users onto a 4-proc (Opteron) 4GB or 8GB machine running 64bit Server 2003, than the same machine running 32bit Server 2003.

This posting is provided "AS IS" with no warranties, and confers no rights.

On my way into work today, I realized that Seattle traffic is one of those human systems that visibly conform to the second law of thermodynamics… OK, that’s a gross oversimplification, but it’s still amazing to watch people intentionally move from a high energy state to a low one, with no good explanation. 

Example:  On the exit I take to get to work every day, there are two lanes that turn left.  Invariably, there is a line of 20+ cars in one lane, and 1 or 2 in the other.  Why no one else uses the other lane is beyond me.  I might understand better if it was always the same lane, but it often changes. 

Another Example:  If someone is doing 60Mp/h in their lane, in a line of cars, with good spacing, and a spot opens up in another lane (left or right)…Someone will invariably dive into that spot, but not change speeds at all, thus blocking that lane for faster traffic. 

On a related note, if you ever need to calculate the noise from a segment of road you’re planning, the National Physical Laboratory in the UK has the calculations for you here.