Valorie’s first day back at work was yesterday, and the principal at her school led a discussion about the following parable:
Once upon a time there was a town that was built just beyond the bend of a large river. One day some of the children from the town were playing beside the river when they noticed three bodies floating in the water. They ran for help and the townsfolk quickly pulled the bodies out of the river.
One body was dead so they buried it. One was alive, but quite ill, so they put that person into the hospital. The third turned out to be a healthy child, who then they placed with a family who cared for it and who took it to school.
From that day on, every day a number of bodies came floating down the river and, every day, the good people of the town would pull them out and tend to them – taking the sick to hospitals, placing the children with families, and burying those who were dead.
This went on for years; each day brought its quota of bodies, and the townsfolk not only came to expect a number of bodies each day but also worked at developing more elaborate systems for picking them out of the river and tending to them. Some of the townsfolk became quite generous in tending to these bodies and a few extraordinary ones even gave up their jobs so that they could tend to this concern full-time. And the town itself felt a certain healthy pride in its generosity.
However, during all these years and despite all that generosity and effort, nobody thought to go up the river, beyond the bend that hid from their sight what was above them, and find out why, daily, those bodies came floating down the river.
- Dennis Connor, Gonzaga University. He’s unaware of its origin…
This story led to a rousing discussion among the teachers, and I realized that it’s true outside its intended circle (lawyers in training). It’s just as relevant to software engineering as it is to any other field of endeavor.
As a simple example, think of computer security. Issuing patches is equivalent to tending the bodies coming from the river. Eventually you get pretty good at it, and can be rightly proud of your work. But issuing patches ignores the root causes of the problem; it just deals with the symptoms. Sometimes you’ve got to go beyond the bend in the river to find the source of problems. In many ways, XP SP2 is an example of going beyond the bend – it’s an attempt to preemptively thwart the bodies from coming down the river.
And this goes way beyond that. How many times have you spent fixing bug after bug after bug in a particular component without spending the time to look at the root cause of those bugs? Maybe taking a step back could let you eliminate entire classes of bugs. Maybe there’s an underlying class that’s too hard to use and instead of applying band-aid after band-aid, the right thing to do is to throw away the old code and replace it?
The old suggestion box had its comment period expire, so I've recreated it. Feel free to add to it at will.
Well, the day was pretty much a wash, work-wise. I spent most of the day swapping war stories with co-workers. On the other hand, the 20 dozen bouncy balls I brought into work were a huge hit.
Bouncy balls have long participated in Microsoft events, Hans Spiller, one of the original compiler guys has a great write-up of the significance of the bouncy ball on his site.
Oh, and nobody trashed my office :) I think I'm happy. But I did get at least one decent picture out of the day (thanks Steve!)
That's what was left of the bouncy balls at about 3PM. The flowers (20 roses) are from Valorie.
Other highlights you can see in the picture: Some of my collection of lego models. Unfortunately, the Imperial Star Destroyer's out of frame, as is the Statue of Liberty. On the bottom shelf of the bookcase, you can see my collection of service awards (3, 5, 10, 15, and now 20). On the shelf above it is my rather small collection of patent cubes. When you submit a patent at Microsoft, they give you a little cube with the title of the patent and your name. When the patent is granted, they give you a wooden plaque with the front page of the patent on it. I know people who don't have enough wall space to hold all their patent plaques, instead they're kept in a pile in their office.
On my desk is a small part of my collection of manipulative puzzles, I've not yet gotten them all out after my last office move (back in January).
Nope, Sgt Pepper didn’t teach the band to play.
20 years ago today, a kid fresh out of Carnegie-Mellon University showed up at the door of the 10700 Northup Way, ready to start his first day at a real job.
What a long strange trip it’s been.
Over the past 20 years, I’ve:
· Worked on two different versions of MS-DOS (4.0, 4.1).
· Worked on three different versions of Lan Manager (1.0, 1.5, 2.0)
· Worked on five different releases of Windows NT (3.1, 3.5, XP (SP2), W2K3 (SP1), Longhorn)
· Worked on four different releases of Exchange (4.0, 5.0, 5.5, and 2000)
I’ve watched my co-workers move on to become senior VP’s. I’ve watched my co-workers leave the company.
I’ve seen the children of my co-workers grow up, go to college, marry, and have kids.
I’ve watched the younger brother of my kids babysitter who I met at 12 years of age grow up, go to college and come to work at Microsoft in the office around the corner from mine (that one is REALLY weird btw).
I’ve seen strategy’s come and go (Lan Manager as an OEM product, then retail, then integrated with the OS).
I’ve watched three different paradigm shifts occur in the software industry, and most of a fourth. The first one was the shift of real computing to “personal” computers. The second was the GUI revolution, the third was the internet, and now we’re seeing a shift to smaller devices. We’re still not done with that one.
I’ve watched Microsoft change from a “small software startup in Seattle” to the 800 pound gorilla everyone hates.
I’ve watched Microsoft grow from 650ish people to well over 50,000.
I’ve watched our stock grow and shrink. I’ve watched co-workers fortunes rise and fall.
I’ve watched governments sue Microsoft. I’ve watched Governments settle with Microsoft. I’ve seen Microsoft win court battles. I’ve seen Microsoft lose court battles.
I’ve watched the internet bubble start, blossom, and explode.
I’ve watched cellular phones go from brick-sized lumps to something close to the size of matchbooks.
I’ve seen the computer on my desktop go from a 4.77MHz 8088 with 512K of RAM and a 10M hard disk to a 3.2GHz hyper-threaded Pentium 4 with 1G of RAM and an 80G hard disk.
I’ve watched the idea of multimedia on the PC go from squeaky beeps from the speaker to 8-channel surround sound that would rival audiophile quality products.
I’ve watched video on the PC go from 640x350 Black&White to 32bit color rendered in full 3d with millions of polygons.
When I started at Microsoft, the computer that they gave me was a 4.77MHz PC/XT, with a 10 megabyte hard disk, and 512K of RAM. I also had a Microsoft Softcard that increased the RAM to 640K, and it added a clock to the computer, too (they didn’t come with one by default)! Last month, I bought a new computer for my home (my old one was getting painfully slow). The new computer is a 3.6GHz Pentium 4, with 2 GIGABYTES(!) of RAM, and a 400 GIGABYTE hard disk. My new computer cost significantly less than the first one did. If you index for inflation, the new computer is at least an order of magnitude cheaper.
I still have the letter that Microsoft sent me confirming my job offer. It’s dated January 16th, 1984. It’s formatted in Courier, and the salary and stock option information is written in ink. It’s signed (in ink) by Steve Ballmer. The offer letter also specifies the other benefits; it’s not important what they are. I also have Steve’s business card – his job title? VP, Corporate Staffs. Yup, he was head of HR back then (he did lots of other things, but that’s what his title was). I also have the employee list they gave out for the new hires, as I said before; there are only about 600 people on it. Of those 600 people, 48 of them are still with Microsoft. Their job titles range from Executive Assistant, to UK Project Troubleshooter, to Architect, to Director.
The only person who I interviewed with when I started is still at Microsoft, Mark Zbikowski. Mark also has the most seniority of anyone still at Microsoft (except for Steve Ballmer and Bill Gates).
When I started in the Lan Manager group, Brian Valentine was a new hire. He was a test lead in the Lan Manager group, having just joined the company from Intel. He (and Paul Maritz) used to tell us war stories about their time at Intel (I particularly remember the ones about the clean desk patrol).
In the past twenty years, I’ve had 16 different managers: Alan Whitney (MS-DOS 4.0); Anthony Short (MS-DOS 4.0); Eric Evans (MS-DOS 4.0, MS-DOS 4.1); Barry Shaw (Lan Manager 1.0); Ken Masden (Lan Manager 1.5, Lan Manager 2.0); Dave Thompson (Lan Manager 2.0, Windows NT 3.1); Chuck Lenzmeier (Windows NT 3.5); Mike Beckerman (Tiger); Rick Rashid (Tiger); Max Benson (Exchange 4.0, 5.0, 5.5); Soner Terek (Exchange 5.5, Exchange 2000); Jon Avner (Exchange 2000); Harry Pyle (SCP); Frank Yerrace (Longhorn); Annette Crowley (Longhorn) and Noel Cross (Longhorn).
I’ve moved my office 18 different times (the shortest time I’ve spent in an office: 3 weeks). I’ve lived through countless re-orgs. On the other hand, I’ve never had a reorg that affected my day-to-day job.
There have been so many memorable co-workers I’ve known over the years. I can’t name them all (and I know that I’ve missed some really, really important ones), but I’ll try to hit some highlights. If you think you should be on the list but aren’t, blame my poor memory, I apologize, and drop me a line!
Gordon Letwin – Gordon was the OS guru at Microsoft when I started, he was the author of the original H19 terminal ROM before coming to Microsoft. In many ways, Gordon was my mentor during my early years at Microsoft.
Ross Garmoe – Ross was the person who truly taught me how to be a software engineer. His dedication to quality continues to inspire me. Ross also ran the “Lost Lambs” Thanksgiving Dinner – all of us in Seattle without families were welcome at Ross’s house where his wife Rose and their gaggle of kids always made us feel like we were home. Ross, if you’re reading this, drop me a line :)
Danny Glasser – Danny had the office across the hall from me when I was working on DOS Lan Manager. He’s the guy who gave me the nickname of “DOS Vader”.
Dave Cutler – Another inspiration. He has forgotten more about operating systems than I’ll ever know.
David Thompson – Dave was the singularly most effective manager I’ve ever had. He was also my least favorite. He pushed me harder than I’ve ever been pushed before, and taught me more about how to work on large projects than anyone had done before. Valorie was very happy when I stopped working for him.
David Weise – David came to Microsoft from Dynamical Systems Research, which I believe was Microsoft’s third acquisition. He owned the memory management infrastructure for Windows 3.0.
Aaron Reynolds – Author of the MS-NET redirector, one of the principal DOS developers.
Ralph Lipe –Ralph designed most (if not all) of the VxD architecture that continued through Win9x.
David, Aaron, and Ralph formed the core of the Windows 3.0 team; it wouldn’t have been successful without them. Collectively they’re the three people that I believe are most responsible for the unbelievable success of Windows 3.0. Aaron retired a couple of years ago; David and Ralph are still here. I remember David showing me around building 3 showing off the stuff in Windows 3.0. The only thing that was going through my mind was “SteveB’s going to freak when he sees this stuff – this will blow OS/2 completely out of the water”.
Paul Butzi – Paul took me for my lunch interview when I interviewed at Microsoft. He also was in the office next to mine when I started (ok, I was in a lounge, he was in an office). When I showed up in a suit, he looked at me and started gagging – “You’re wearing a ne-ne-ne-neckt….” He never did get the word out.
Speaking of Paul. There was also the rest of the Xenix team: Paul Butzi, Dave Perlin, Lee Smith, Eric Chin, Wayne Chapeski, David Byrne, Mark Bebie (RIP), Neil Friedman and many others. Xenix 386 was the first operating system for the Intel 386 computer (made by Compaq!). Paul had a prototype in his office, he had a desk fan blowing on it constantly, and kept a can of canned air handy in case it overheated.
Ken Masden – the man who brought unicycle juggling to Microsoft.
All of the “core 12”: Dave Cutler (KE), Lou Perazzoli (MM), Mark Lucovsky (Win32), Steve Wood (Win32, OB), Darryl Havens (IO), Chuck Lenzmeier (Net), John Balciunas (Bizdev), Rob Short (Hardware), Gary Kimura (FS), Tom Miller (FS), Ted Kummert (Hardware), Jim Kelly (SE), Helen Custers (Inside Windows NT), and others. These folks came to Microsoft from Digital Equipment with a vision to create something brand new. As Tom Miller put it, it was likely to be the last operating system ever built from scratch (and no, Linux doesn’t count – NT was 100% new code (ok, the command interpreter came from OS/2), the Linux kernel is 100% new, but the rest of the system isn’t). And these guys delivered. It took longer than anyone had originally planned, but they delivered. And these guys collectively taught Microsoft a lesson in how to write a REAL operation system, not a toy operating system like we’d been working on before. Some day I’ll write about Gary Kimura’s coding style.
Brian Valentine – Brian is without a doubt the most inspirational leader at Microsoft. His ability to motivate teams through dark times is legendary. I joined the Exchange team in 1994, the team was the laughing stock at Microsoft for our inability to ship product (Exchange had been in development for almost six years at that point), and we still had another year to go. Brian led the team throughout this period with his unflagging optimism and in-your-face, just do it attitude. For those reading this on the NT team: The Weekly World News was the official newspaper of the Exchange team LONG before it was the official newspaper of the Windows team.
Max Benson – Max was my first manager in Exchange. He took a wild chance on a potentially burned out engineer (my time in Research was rough) and together we made it work.
Jaya Matthew – Jaya was the second person I ever had report to me; her pragmatism and talent were wonderful to work with. She’s also a very good friend.
Jim Lane, Greg Cox, and Ardis Jakubaitis – Jim, Greg, Ardis, Valorie and I used to play Runequest together weekly. When I started, they were the old hands at Microsoft, and their perspectives on the internals of the company were invaluable. They were also very good friends.
And my list of co-workers would not be complete without one other: Valorie Holden. Yes, Valorie was a co-worker. She started at Microsoft in 1985 as a summer intern working on testing Word and Windows 1.0. While she was out here, she accepted my marriage proposal, and we set a date in 1987. She went back to school, finished her degree, and we got married. After coming out here, she started back working at Microsoft, first as the bug coordinator for OS/2, then as Nathan Myhrvold’s administrative assistant, then as a test lead in the Windows Printing Division, eventually as a program manager over in WPD. Valorie has stood by my side through my 20 years at Microsoft; I’d never have made it without her unflagging support and advice (ok, the threats to my managers didn’t hurt either).
There’ve been good times: Getting the first connection from the NT redirector to the NT server; Shipping Exchange 4.0; Shipping Exchange 2000 RC2 (the ship party in the rain). Business trips to England. Getting a set of cap guns from Brian Valentine in recognition of the time I spent in Austin for Lan Manager 2.0 (I spent 6 months spending Sunday-Wednesday in Austin, Thursday-Saturday in Redmond).
There’ve been bad times: Reorgs that never seemed to end. Spending four years in ship mode (we were going to ship “6 months from now” during that time) for NT 3.1 (Read Showstopper! for more details). The browser checker (it took almost ten years to get over that one). A job decision decided over a coin toss. Working in Research (I’m NOT cut out to work in research).
But you know, the good times have far outweighed the bad, it’s been a blast. My only question is: What’s in store for the next twenty years?
Edit: Forgot some managers in the list :)
Edit2: Missed Wayne from the Xenix team, there are probably others I also forgot.
Edit3: Got some more Xenix developers :)
So the newswires and forums are buzzing about this reported security flaw in XP SP2. Essentially they are complaining that the security center in SP2 uses WMI to store its metadata and an administrator can modify the metadata to convince the user that they’re protected when they’re not.
In the original eWeek article, Microsoft’s response is quoted as:
In SP2, we added functionality to reduce the likelihood of unknown/devious applications running on a user's system, including turning Windows Firewall on by default, data execution prevention, attachment execution services to name a few. To spoof the Windows Security Center WMI would require system-level access to a PC. If the user downloads and runs an application that would allow for spoofing of Windows Security Center, they have already opened the door for the hacker to do what they want. In addition, if malware is already on the system, it does not need to monitor WSC to determine a vulnerable point of attack, it can simply shut down any firewall or AV service then attack – no WSC is necessary."
"Windows Security Center, found in the Windows XP Control panel, provides customers the ability and makes it easier to check the status of these essential security functionalities such as firewalls, automatic updates and antivirus. Windows Security Center will inform users whether key security capabilities are turned on and up to date and will notify users if it appears that updates need to be made or if additional action steps may need to be taken to help them get more secure."
In other words, if you’re running as an administrator, you can run an application that can mess up your computer. Yup, but if you’re running as an admin and you’re running untrusted code then IMHO, spoofing the security center is the LEAST of your problems – the application that spoofed the security center could also have installed a rootkit on your machine, and at that point, the bad guys own your computer.
Mike Dimmick also has an excellent rebuttal to the original eWeek article.
Back in 1995, I was working over in the Northup building. Valorie had joined me from school, and was working as a tester on Microsoft Word (she tested the Apricot version of Word and the AT&T 3b5 port, and the IBM OfficeWriter converter) and Microsoft Windows 1.0.
I was working on MS-DOS 4.0. The entire team was ramping up for an October demo with IBM - we were going to pitch MS-DOS 4.0 to them as the next major release of DOS.
It turns out that they liked what they saw. But not enough to actually buy into the project. Instead they wanted something that would help them sell more of their newest PC, the PC/AT, which was a 286 based processor. While the technologies in MS-DOS 4.0 were great, it wouldn't help sell more AT's. So instead of taking our MS-DOS code, they eventually decided on a much more ambitious program.
They would start phasing out support for real-mode 8088 processors and concentrate on protected mode 286 applications. Because they knew they couldn't do it all in a single release cycle, they staged it as two releases. The first release was text-mode only, the second release was GUI based.
Of course, I'm talking about OS/2.
I never did get to work on OS/2, I ended up spending my time working on DOS variants until I moved to the Lan Manager group.
It was 1986. I was working for Eric Evans at the time, working with Alan Whitney on finishing up MS-DOS 4.0 (for Goupil) and starting to get ramped up on MS-DOS 4.1.
I was also getting settled into my new office in building 2 in the new campus. The move was "interesting" - Microsoft canceled the President's Day holiday for the move - instead of giving us all a holiday, they asked us to work through the holiday and to take the real holiday on moving day. This prompted David Byrne's famous "Pope Gregory" memo where he compared BillG to Pope Gregory.
On the other hand, I spent moving day in Paris. We were almost done shipping DOS 4.0, but Goupil was having some difficulties with MS-DOS 4.0 and they asked Alan and I to come to Paris to debug the system. From my end, there was a problem on one of their machines with 4 floppy drives - my code worked fine on machines with two floppies but with 4, drive not ready errors were failing
So Alan and I flew from Seattle to LA, then back over Seattle to Paris - my first business class trip :)
So I've seen Paris, but only from the back seat of a Citroen 2CV and only a single whirlwind tour.
I was finishing up on the MS-DOS 4.1 project and started working in the networking group, working on MS-NET 1.10, which was a precursor to the first Lan Manager product.
I was working for Barry Shaw, with two other developers, who were working as vendors from Apricot in the UK. Humorously enough, they both ended up working at Microsoft, one just recently retired, the other is still working here :). Barry retired long ago.
DOS Lan Manager was another fascinating project. We basically took the Lan Manager protocols (later known as CIFS), which were designed for OS/2 and got them working on MS-DOS. We also added support for the messenger service - my very first T&SR application - it would pop up a message on top of the console when someone did you a "Net Send".
Going back a little earlier in the year (in January), Valorie and I got married. And within 36 hours of returning to Microsoft from my honeymoon, I was on a plane to England to deal with a crisis in MS-DOS 4.1. I still kick myself for not insisting that Valorie be allowed to accompany me on the trip, it would have been cool to get a trip to England out of it.
This weekend, Daniel and I went to Dash Point State Park for a lovely two days of working out in the rain with a dozen other members of our dojo (karate school).
We arrived on Saturday morning at 8:30AM and left at 5:00 on Sunday, and we worked out a LOT. There's something about doing Nifunchun Shodon for 45 minutes in the rain, non-stop. Or standing still, with your knees bent, and tailbone tucked under for an hour. Lots of fun, but my quads are SO sore right now :).
There's no question that for me, the highlight of the weekend came at the end. The three candidates for black belt got up and demonstrated the "Core Six" Katas (forms) for the rest of the group.
Here's Daniel doing Bassai Dai:
Later on in Bassai Dai:
And, when it was all done,
Daniel finally got his black belt! He's been working hard for this for more than half his lifetime - he started training when he was five and a half years old, and he's been training for the belt now for seven years.
Needless to say, I'm increadibly proud of him, I know how much he's worked on getting to this point. You ca't really see the goofy grin on his face in this picture, but it was there, take my word for it.
One of the comments in my “interviews” post reminded me about something that happened when I was interviewing.
The interview process at Microsoft has the interviewee being shepherded from one office to the next, and the interviewee gets asked questions in the employee’s office.
Well, in my case, for at least one interview it didn’t happen. I think it was because my first comment when I looked into the interviewee’s office was “Hey, that’s a cute looking computer, what kind is it”? I was quickly shuttled away from the office and into a lounge.
Well, my interview was in early January 1984. The SuperBowl was two weeks away. Nobody had ever heard of that little white box, but after that SuperBowl, everyone had heard of it.
Of course the computer in question was the Macintosh.
Edit: Found the link :)
So there was a discussion on an internal mailing list yesterday, and Raymond popped in with the following quote:
The problem is that some people draw stacks growing downward (so "top of stack" is lowest) and others growing upward (so "top of stack" is highest). The documentation here wants "top of stack" to be highest. In other words, "higher" = "closer to the point the exception is raised". The second interpretation is probably more common.
People also talk about a calling function being "higher" than a called function. This is directly in conflict the second interpretation above. Yet this terminology is also very common.
He brings up an interesting point. Which way DO stacks grow?
On Intel processors, the PUSH EAX instruction is equivalent to:
SUB ESP, 4MOV [ESP], EAX
So on Intel processors, stacks grow down. Whenever I’m writing calls stacks that’s the way I draw them – the caller is on top, the callee is on the bottom. But other machines have stacks that grow UP, not DOWN. For example, the Decsystem-20’s stack grows up (I first learned assembly language programming on the Dec-20 – it’s still my favorite instruction set). So the fact that my stacks grow down is clearly not related to the language I first learned. On the other hand, I spent many years writing exclusively in x86 assembly (from 1984 to 1989, more or less), so it may be that that’s what I’m familiar with.
Interestingly enough, it can be argued (pretty strongly) that stacks that grow down are a cause of security holes (or rather stacks that grow in a direction opposite of array accesses). Since array accesses typically go up in memory, and stacks grow down, the area where they cross has huge amounts of potential. If stacks grew up and arrays grew up, then it would be much harder (not impossible, mind you, but much harder) for easy coding mistakes to result in buffer overruns (wcscpy(stackbuffer, inputstring) isn’t as much a security hole if the return address is BELOW the stackbuffer instead of above the stackbuffer). Unfortunately, it’s WAY too late for this behavior to change; to change this behavior would require a wholesale move away from x86 compatible platforms onto another platform, every existing application would break, etc. And I’m sure that the Intel engineers who designed the 8080 had a good reason for making their stacks grow down – that was a decision made almost 30 years ago, back in a different era.
Oh, and btw. There’s a related question to the stack question: How do your trees grow? My trees (binary, n-ary, etc) are rooted at the top of the whiteboard and grow down. Other people’s trees are rooted at the bottom of the whiteboard and grow up. I don’t know if the two are related, but…
Edit: Raymond pointed out that the SUB comes first on the PUSH instruction :)
This morning, Dmitry asked what the heck was the audio service for anyway.
That's actually a really good question.
For Windows XP, the most common use for the audiosrv service is that if the audiosrv service didn't exist, applications that linked with winmm.dll would also get setupapi.dll in their address space. This is a bad thing, since setupapi is relatively large, and for 99% of the apps that use winmm.dll (usually to call PlaySound), they don't need it until they actually start playing sounds (which is often never).
As a part of this, audiosrv monitors for plug and play notifications (again, so the app doesn't have to) and allows the application to respond to plug and play changes without having to burn a thread (and a window pump) just to detect when the user plugs in their USB speakers. All that work's done in audiosrv.
There's a bunch of other stuff, related to global audio digital signal processing that audiosrv manages, and some stuff to manage user audio preferences, but offloading the PnP functionality is the "big one". Before Windows XP, this functionality was actually a part of csrss.exe (the windows client/server runtime subsystem), but in Windows XP it was broken out into its own service.
For Longhorn, Audiosrv will be doing a lot more, but unfortunately, I can't talk about that :( Sorry.
I really do want to be able to talk about the stuff we're doing, but unfortunately none of it's been announced yet, and since none of its been announced yet...
Edit: Corrected title. Also added a little more about longhorn.
In yesterday’s post, I mentioned that SP2’s installation didn’t update a Longhorn version of the Windows Audio service, and that caused customer difficulties.
I left unanswered the basic question: Why on earth didn’t the SP2 upgrade modify the existing version? My answer in the post was that since the version number of the file on the disk was higher than the SP2 version, it didn’t update it. And that’s the case. But the date on the file on the disk was back in October of 2003, while the SP2 version was August of 2004. Why on earth did the XP SP2 setup take an older file instead of the newer one?
Well, it’s because the version number trumps file times. This is actually a GOOD thing. Microsoft often provides hotfixes to components and it uses the version number of the file to determine if the file on the local machine is “newer” than the one being installed. If the service pack installation didn’t check for the version number, then it could potentially break something (service packs don’t always include all the updates for components).
It turns out that if you do a full OS install using a slipstream version of XP (a full install version of Windows XP with SP2 installed), then the full install WILL blast all the files, regardless of version, this is because a full OS reinstall is considered a “repair” install, and in that case, the user NEEDS to ensure that his OS is as close to the RTM bits as possible.
The better news is that this kind of a problem almost never hits real customers, since they don’t typically have access to incorrectly versioned versions of Microsoft DLL’s – our release team is quite good about ensuring that mistakes like that don’t happen.
And even if they do, it’s highly unlikely that they can get a bad version of the binary to stay on their machine due to system file protection (unless they’ve done things to their system that would cause SFP to be bypassed, which was effectively the case here (it’s too complicated to explain, but it’s related to the test root certificate)). Btw, Raymond mentioned how to verify the signatures of your system DLLs here.
A number of people have asked for me to write up my experiences debugging a problem. The thing is that it’s hard to do that explicitly without disclosing internals of functions that probably shouldn’t be disclosed (because they relate to features that haven’t been announced, etc). However, I debugged a problem the other day that fits the bill perfectly.
We had an internal helpdesk request come in that a user had lost audio shortly after installing XP SP2. It turns out that while I didn’t have a huge impact on SP2 (mostly doing code reviews), there were a couple of things I added to the system. The biggest feature I added was the ability to stop the Windows Audio service.
And it turns out that this could have caused the problems the user was seeing.
So I asked for (and was granted) RDP (remote desktop) access to the machine. Looking at the machine, there were no MME (MultiMedia Extensions, our term for the waveXxx, mixerXxx, midiXxx APIs) devices enumerated. Well, it looked like I needed to break out the debugger.
Since the windows audio service runs in the same process as the networking services, and since the XP SP2 symbols are available over the network, the first thing I needed to do was to split the windows audio service into its own process. I made the necessary registry modifications to make it run in its own process (no, I’m not going to document them, nobody needs to know them), and then I stopped the windows audio service.
C:\>net stop “windows audio”The requested pause or stop is not valid for this service.More help is available by typing NET HELPMSG 2191.
Huh? Wait a second. Why isn’t windows audio stoppable?
Time to pull the service debuggers bag-o-tricks. One of the utilities bundled with XP is a diagnostic tool known as sc.exe, it’s a general purpose service control API utility. To do this, I need to use the short name for the windows audio service, audiosrv.
C:\>sc query audiosrvSERVICE_NAME: audiosrv TYPE : 20 WIN32_SHARE_PROCESS STATE : 4 RUNNING (NOT_STOPPABLE,NOT_PAUSABLE,IGNORES_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0
Hold on. I mentioned above that the biggest change I made for XP SP2 was to make the windows audio service stoppable. What’s going on here?
I know that windows audio is stoppable in SP2 installations, I’ve verified that. The thing is that a service tells the service controller its capabilities when it first calls SetServiceStatus. Well, windows audio’s call capabilities are hard coded, the only way that it would not be marked as stoppable is if something went wrong with the registration.
I wonder if somehow the DLL that holds the windows audio service didn’t get updated with the SP2 installation. So I start explorer up on c:\windows\system32 and look at the windows audio DLL. It’s got a file version of 6.0.4017. That’s wrong; it should have a version of 5.1.2600.2180. 6.0.4017 is a Longhorn version number.
So I asked the person having the problem if he’d done anything that might have caused a longhorn version of audiosrv to be put on his machine. It turns out that he’d run an internal install script that copied over an interim Longhorn build of this DLL onto his machine.
And, since SP2’s install didn’t replace the file (because the file version on the file on his machine was newer than the SP2 version of the file), he was running a Longhorn version of the windows audio service on his XP SP2 machine.
We deleted the DLL, SFP copied back the right version and his machine had audio again!
Edit: Fixed screwy text.
In yesterday’s post I talked about the AARD code. One of the questions that perennially comes up is “Why on earth didn’t the Windows guys just remove that code”?
Well, the answer is that it would have likely broken far more code to remove the AARD code from the product than to jump around it. The Windows guys could render the code ineffective by simply modifying a single JMP instruction to avoid the AARD code, while removing the code would have meant that the windows executable would change – every routine would be at a different offset, because the AARD code wasn’t there any more.
What’s the big deal with the Windows executable changing? Well Chris Pratley explained it really well in his “More on Quality” post:
A last anecdote to leave you with. Even re-linking your code (not even recompile) can introduce a crashing bug. A few years ago, we were working on the release candidate for an Asian-language version of Word97. We thought we were done, and ran our last optimization on the build. We have some technology at Microsoft that profiles code usage and arranges the code modules so that they are in the executable in the optimal order to produce the best possible boot speed. After running this process which involves mainly the linker and is considered very safe, we put the code in escrow while the testers tried to break it. And they did - they found that it would crash on some of the machines they had when a certain feature was used. But the unoptimized build did not crash with the same steps and machine.
So we ran the "debug" build (a version of the build that has the same code as the "ship" build you all use, but includes extra information to allow debugging) and it also did not crash. We then tried to debug the ship build - but just running the ship build in the debugger caused the crash to go away. Some developers love this sort of mystery. One of the future inventors of Watson stopped by to get involved. No matter what they did, as soon as they tried to find out what was causing the crash, it went away. We went to get an ICE - a hardware debugger that might give us some clues.
Then we noticed that there was a pattern. The only machines that showed this crash had Pentium processors of 150MHz or less, and some of the 150MHz machines did not show the problem. We had a hunch, and searched the Intel web site for "errata" (their word for bugs - we prefer "issues"). Sure enough, there was a flaw in the Pentium chip which under certain obscure circumstances could cause a fault - there needed to be a JMP instruction followed exactly 33 bytes later by a branch, and the JMP instruction had to be aligned with a "page" (4096 byte block) boundary in memory. Talk about specific. The flaw had been fixed in later versions of the 150MHz generation, and all chips produced later.
Now, there are actually flaws in chips quite often (one or two are even famous). Those chips include software after all. But as they are discovered, chip manufacturers tell the people who write compilers about the bugs, and the compiler people modify their compilers to produce code that cannot hit the bug. So the flaws seem to just disappear (thank goodness for software!) It turned out we were using a slightly older version of the compiler which did not know about this flaw. An Intel rep at Microsoft confirmed the details of the problem, and rather than relinking or taking any chances whatsoever, we ran a check for this byte sequence, and manually moved three bytes in our 5MB executable to make sure those instructions were 34 bytes apart. Problem solved. So now when someone tells me a fix is "safe", I can tell them that no fix is truly safe - you really can never know.
I can personally attest to this problem. We hit exactly this bug in Exchange 5.5 – we had a build which consistently crashed on one test machine (and only on that test machine). We spent days trying to figure out the problem, no amount of instrumentation would allow us to find the cause of the problem. Every thing we changed made the problem go away, but it was absolutely reproducible. Eventually we stumbled on the errata Chris mentioned, and it turns out that our code was hitting that errata. In our case, we were far enough from shipping that we fixed the problem by globally enabling the compiler flag that fixed this problem.
Anyway, given this history, it’s not at all surprising that the Windows team chose to disable the AARD check by simply jumping around it – they KNEW that Windows worked with the AARD check code in place, they’d been testing it for weeks (or months). When you’re close to shipping a product, you don’t want to invalidate your entire test pass (which can literally take months to complete) by taking a fix that potentially could invalidate it. Instead you find the lowest-impact fix that works around the problem.
I just discovered this web page (via /.).
The wizards at Microsoft research have figured out how to turn real life movies into animated movies. It takes tweaking to make it work, but...
Check out the WMV file too. Wow.
I just ran into this on Microsoft's "What to Know Before You Download and Install Windows XP Service Pack 2" site:
Memory: You need at least 2 gigabytes (GB) of free memory space on your hard disk. To check your free memory space, click Start, click My Computer, and right-click Local Disk (C:) or the name of the drive you are installing SP2 on, and then click Properties.
Memory: You need at least 2 gigabytes (GB) of free memory space on your hard disk.
To check your free memory space, click Start, click My Computer, and right-click Local Disk (C:) or the name of the drive you are installing SP2 on, and then click Properties.
Does anyone ELSE find it disturbing that Microsoft's recommending that people have 2G of memory to install the service pack? Or that Microsoft's recommending that people have memory space on their hard disk? How do I get memory space on my hard disk? Should I increase my paging file to 2G to make that work?
Sigh. If you treat people like they don't understand computers, how on EARTH can you expect them to learn about computers?
If there was ever a question that I’m a glutton for punishment, this post should prove it.
We were having an email discussion the other day, and someone asked:
Isn't there a similar story about how DOS would crash when used with [some non-MS thing] and only worked with [some MS thing]? I don't remember what the "thing" was though =)
Well, the only case I could think of where that was the case was the old AARD code in Windows. Andrew Schulman wrote a great article on it back in the early 1990’s, which dissected the code pretty thoroughly.
The AARD code in Windows was code to detect when Windows was running on a cloned version of MS-DOS, and to disable Windows on that cloned operating system. By the time that Windows 3.1 shipped, it had been pulled from Windows, but the vestiges of the code were left behind. As Andrew points out, the code was obfuscated, and had debugger-hiding logic, but it could be reverse engineered, and Andrew did a great job of doing it.
I can’t speak as to why the AARD code was obfuscated, I have no explanation for that, it seems totally stupid to me. But I’ve got to say that I totally agree with the basic concept of Windows checking for an alternative version of MS-DOS and refusing to run on it.
The thing is that the Windows team had a problem to solve, and they didn’t care how they solved it. Windows decided that it owned every part of the system, including the internal data structures of the operating system. It knew where those structures were located, it knew what the size of those data structures was, and it had no compunction against replacing those internal structures with its own version. Needless to say, from a DOS developer’s standpoint, keeping Windows working was an absolute nightmare.
As a simple example, when Windows started up, it increased the size of MS-DOS’s internal file table (the SFT, that’s the table that was created by the FILES= line in config.sys). It did that to allow more than 20 files to be opened on the windows system (a highly desirable goal for a multi-tasking operating system). But it did that by using an undocumented API call, which returned a pointer to a set of “interesting” pointers in MS-DOS. It then indexed a known offset relative to that pointer, and replaced the value of the master SFT table with its own version of the SFT. When I was working on MS-DOS 4.0, we needed to support Windows. Well, it was relatively easy to guarantee that our SFT was at the location that Windows was expecting. But the problem was that the MS-DOS 4.0 SFT was 2 bytes larger than the MS-DOS 3.1 SFT. In order to get Windows to work, I had to change the DOS loader to detect when win.com was being loaded, and if it was being loaded, I looked at the code at an offset relative to the base code segment, and if it was a “MOV” instruction, and the amount being moved was the old size of the SFT, I patched the instruction in memory to reflect the new size of the SFT! Yup, MS-DOS 4.0 patched the running windows binary to make sure Windows would still continue to work.
Now then, considering how sleazy Windows was about MS-DOS, think about what would happen if Windows ran on a clone of MS-DOS. It’s already groveling internal MS-DOS data structures. It’s making assumptions about how our internal functions work, when it’s safe to call them (and which ones are reentrant and which are not). It’s assuming all SORTS of things about the way that MS-DOS’s code works.
And now we’re going to run it on a clone operating system. Which is different code. It’s a totally unrelated code base.
If the clone operating system isn’t a PERFECT clone of MS-DOS (not a good clone, a perfect clone), then Windows is going to fail in mysterious and magical ways. Your app might lose data. Windows might corrupt the hard disk.
Given the degree with which Windows performed extreme brain surgery on the innards of MS-DOS, it’s not unreasonable for Windows to check that it was operating on the correct patient.
Edit: Given that most people aren't going to click on the link to the Schulman article, it makes sense to describe what the AARD check was :)
Edit: Fixed typo, thanks KC
Bruce Wells asked (in the comments of my “opening firewalls” ‘post) where you should go to get netfw.h. Well, after a bit of searching, I found it, it was surprisingly difficult (don’t ask).
It turns out that netfw.h isn’t actually shipped separately. Instead, as Tony Goodhew describes it, the netfw.idl file is shipped in the platform SDK instead, and you need to run the MIDL compiler on it to generate the .h file.
The platform SDK update that contains netfw.idl can be found here.
Funny coincidences. I was thinking about writing up my experiences at the one time I was invited to Bill’s house (about 5 years ago, it was a party for 15+ year veterans), and I ran into this post from Jeff Maurone.
Jeff’s experiences pretty much mirror my own. The house is amazing, the security is insane (he didn’t mention the discreet video cameras that cover the entire grounds). I’m not sure he noticed the two story tall projection TV screen that’s hidden behind the tapestry in the living room (the wall of which is constructed of braided pieces of cedar). And the Hollywood grade projection room with the plush seats and individual remote controls. And the swimming pool and exercise room.
The public areas of the house are really more like a convention center than a living area though – yes, there are pictures of Bill and his family but…
Btw, it’s kinda funny – the most memorable part of my 15th year celebration was actually the same kind of event that Jeff mentioned – Bill’s son and his nanny (the son’s nanny, not Bill’s nanny) came down to play on the play structure before bedtime.
And Bill broke away from his guests to go over and hang with his son for about half an hour – just talking, pushing him on the swing, hanging out. Here he had over 100 guests, and he totally ignored them while he went to play with his son.
It totally made my day, and made me realize just how much he had changed from the early days of Microsoft. The old Bill (the Bill of “Is this thing REALLY 8 F*ing BASIC’s”) wouldn’t have snuck away from the party to spend time with his family.
I was cleaning up the remaining bits in the Simple Control Protocol, and wondering what I'd be doing next.
SCP was a cool project to work on - it was a low bandwidth networking technology intended for home automation. On the project, I was responsible for many of the DDK components, and a bridge that alllowed you to control SCP devices using UPnP
Last week (or so), Joe Wilcox of MicrosoftMonitor posted this article describing the Joe’s experiences in a hotel.
He is SO on. A couple of weeks ago, I spent the afternoon at an espresso stand on Queen Anne Hill which advertised free WIFI access (I’m not going to mention their name for reasons which will become obvious).
They DID have free WIFI. Unprotected WIFI (not the end of the world, my laptop’s locked down reasonably tightly). But just for grins, I connected my browser to http://192.168.0.1.
And there was the admin interface to their WAP in all its glory. Yup, they hadn’t bothered to set an admin password on the WAP. I had an easy dump of the IP addresses of all 6 machines connected to the network. And the names of the computers too, if I was so inclined.
I was astonished that they were so lax in their security. I mentioned it to the barista and her only response was effectively “Huh?”
We can’t expect our systems to remain secure unless everyone who offers access to the net takes at least the simple steps to secure the network.
A bit of context for those of you who aren’t in the Seattle area.
Recently, a local church in Bothell decided to host a homeless shelter known as “Tent City 4”. This decision caused a great deal of consternation on the part of the people living around the church, and the city of Bothell sued to force the church to close the tent city. There’s been a bunch of going around about it, and eventually another church, the Northshore United Church of Christ agreed to host the homeless shelter.
It turns out that the Northshore United Church of Christ is located across the street from a local Junior High School, and down the street from the elementary school my daughter Sharron attends.
Last night, they had a public meeting about the church, which was attended by about 200 people. One of the attendees was John Gronquist, who later on wrote the following email message in response to an “Oooh, they’re going to stick homeless people in OUR neighborhood, we’ve got to stop them!” email message.
Normally I can't stand people who "reply all", but since you've taken the liberty of emailing everyone with your views, it's only fair that at least one counter view is allowed a chance to speak. This email was forwarded to me by my wife's account so there's a chance I've missed any discussion which was spurred from TinyWeeHobbit's mail. Sorry in advance for all those who don't want to be involved..
I live off of 168th, and have two young children. I've also visited Tent City in Bothell and seen it with my own eyes. I've also looked up the stats of the abuses in Bothell, as well as the 2003 crime statistics for the city of Woodinville as a whole.. Rather surprising those were. Did you know that there were 83 counts of domestic violence in Woodinville in 2003? That there were 6 forcible rapes? 21 residential Burglaries? 67 Vehicle thefts? 6 arsons?? MY GOD! Someone should do something! Someone should keep all of Woodinville away from Woodinville!!
Compared to the average small town Tent City's crime stats are really low. It's more actively patrolled than our neighborhood currently is, has more active social programs visiting it to check up on problems, and unlike most neighbors they have a REASON to try and behave themselves, because believe it or not the vast majority of them really do want to show that they can make something of themselves and not be simply 'drains on society'.
As a landowning member of the neighborhood and a father I support the Tent City. I'm not going to say that there's no risk at all, that'd be foolish. But I will say that the risk is low, actively managed, and ultimately worth it.
Every night I drive down 168th and see the signs, as do my children.. In fact, another neighbor's children were holding signs last night and cursed me out when I failed to honk in support of their protest. Signs in support of the tent city have strangely been removed and/or defaced. Odd that. Interesting that one of the key arguments against the Tent City is that it'd cause 'litter, vandalism, and profanity' in the neighborhood. So far the only litter, vandalism, and profanity I've encountered, has been from the protesters against the tent city. Clearly there's some key difference I'm missing..
In any case, one sign in particular seems to be the 'big deal'. That there was a "Level 2 Sex Offender found in Bothell Tent City!!" Well, there's a level 2 sex offender living on 165th right now. RIGHT NOW! You can find her name doing a search in the King Country Sheriff's Sex Offender search by zip code. Not sure how long she's been there, but then, it is a free nation supposedly, and people who've served their punishment normally are considered a chance to start their lives over, and not suffer the rest of their lives for their crimes.. That's normally the case, unless, it appears, you've had the misfortune of not being able to get a job after leaving jail, due, I'm sure, in no part to prejudice about hiring previous offenders.
By the way, you can also find the names of every homeless or transient sex offender registered in the state, which is the same list that the guards working at the tent city will be looking up whenever a new person arrives. Can you say the same of the road on which you live? Do you demand to see the names of each new neighbor and look them up in the Sex Offender database when they move in? Great way to be a popular neighbor, I'd wager.. "Oh, sorry, we can't talk to you until we've looked up every bad thing you've ever done, even if you've served your time for it, and parade it before everyone in the neighborhood so they can shun you as well. Give you a chance? Sorry, can't chance it."
By the way, did you know that one of our neighbor's illegally burns his garbage right across the street from Leota Jr. high, sending toxic clouds of smoke from burning plastics and god knows what into the air every single week we've all lived there? He does it at night, and on weekends, so at least his only is poisoning the after hours programs.. I won't name names, but the police have been repeatedly called, and still there's no sign of a garbage can in front of the house on Fridays, and the lovely smell of toxic fumes wafting through my yard and house and into my children's lungs on a continual basis. This neighbor owns a home though, so I guess illegally burning garbage on regular basis is okay. Have to say though, the night he burned lightbulbs and they went off sounding like 6 gunshots during a dinner party we were having was especially disturbing.
Sorry.. ..I just figured that since we're involved in a little game of shunning people for the no legitimate reason, I'd get my digs in as well..
As far as the 'worth it' argument, I've only this to add..
I won't live in fear. Not of neighbors with 'histories', not of neighbors with irrational fears of the unknown, nor of terrorists, nor of polluting whackjobs who won't spend $16 a month to put the garbage out on the street instead of into my kid's lungs..
We almost lost our youngest child to E. Coli when he was 1 year old from a hamburger at a state fair. THAT'S fear. That's horror the likes of which most of you will never know. Did anyone do anything to clean up our meat industry? NO. It was up to us to protect ourselves, as it always is for any family from any threat from anywhere they live.
I won't live in fear, and I won't teach it to my children. Life is to short and precious and full of wonders to live like that, and I've nothing but honest concern for those who do.
We need to be a part of solving the problems of this nation, instead of pushing them off for future generations or other neighborhoods to deal with. If you're really worried about the security of the Tent City, volunteer for night guard duty. Maybe I'll see you there?
In any case, I apologize for those to whom this mail is mere spam.
John is absolutely right. The decision to object to the hosting of the tent city in Woodinville is backed by nothing but FUD. The real dangers associated with the tent city are nowhere NEAR as bad as the dangers associated with just living in Woodinville (not exactly a high-crime area). Having homeless people in your neighborhood makes people uncomfortable. It reminds them that there are people in our society that AREN’T middle class with nice homes and nice cars. The danger represented by the homeless is an excuse to justify the classist (not racist, but classist) discrimination.
I find it SO frustrating that people can’t bring themselves to actually understand that just because someone is homeless doesn’t mean that their morals and values are any different from anyone else. Just because someone is homeless doesn’t mean that they are evil and must be avoided. We should be trying to HELP these people get back on their feet and not hurting them. Places like Tent City 4 give people dignity. It gives them an address and a phone number that they can use as a reference. It gives them a place they can shower before they go to their interview. If you’re homeless you can’t even get a job at McDonalds because you can’t meet the cleanliness requirements of the restaurant. We don’t have public toilets and showers here on the eastside. We don’t have enough shelter space for the homeless right now, and shutting down things like tent city only increase this problem.
At a minimum, the shelter is only going to be open for six weeks. Is the presence of the shelter SO intolerable that you can’t even abide its presence for 6 weeks?
I’m ecstatic that the United Church of Christ stepped up to the plate to host Tent City 4; I only wish others would understand.
Addendum: The City of Woodinville graciously stepped up and donated the use of several acres of unused city property for the shelter, thus rendering the issue of locating the tent city in a residential area moot.
Edit: Removed comments because the post was becoming a forum for the anti-tent city people. Once this post exceeds the comment time limit (sometime next month), I'll re-enable comments so people can see the other comments that have been made on the article.
Over most of this week, I’ve discussed how ignoring the underlying network architecture can radically hurt an application. Now it’s time for a war story about how things can go awry if you don’t notice these things.
One of the basic limitations of networking is that you really shouldn’t send a server more data than it is expecting. At a minimum, it’s likely that your connection will be flow-controlled. This is especially important when you’re dealing with NetBIOS semantics. Unlike stream-based sockets (like TCP), NetBIOS requires message-based semantics. This means that if the transmitter sends a buffer that is larger than the buffer that the receiver is prepared to accept, the send will fail. As a result, the SMB protocol has the concept of a “negotiated buffer size”. Typically this buffer size is about 4K.
Lan Manager 1.0 had a bunch of really cool new enhancements to the basic SMB protocol. One of the neatest ones (which was used for its IPC mechanism) was the “transaction” SMB. The idea behind the transaction SMB was to enable application-driven large (up to 64K :) ) transaction. The protocol flow for the transaction SMB went roughly like this:
Client: Request Transaction, sending <n> bytes, receiving <m> bytes Server: Ok, buffers allocated to receive <n> bytes, go ahead Client: Sending 4K block 1 Client: Sending 4K block 2 Client: Sending 4K block <n> <The server does it’s thing and responds> Server: Response 4K block 1 Server: Response 4K block 2 Server: Response 4K block 3 Server: Response 4K block <n>
The idea was that the client would “shotgun” the sends asynchronously, and as quickly as possible to the server, and the server would do the same. The thinking was that if the transmitter had lots of outstanding sends, then the transport would deliver the data as quickly as possible.
It looks good at this level. But if you were following the discussion on the earlier posts, you should have some red flags raised by now. Now lets consider what happens to the code at the network layer:
Client: Request Transaction, sending <n> bytes, receiving <m> bytes Server: ACK Request Server: Ok, buffers allocated to receive <n> bytes, go ahead Client: ACK Request Client: Sending 4K block 1, frame 1 Client: Sending 4K block 1, frame 2 Client: Sending 4K block 1, frame 3 Server: ACK Request Client: Sending 4K block 2, frame 1 Client: Sending 4K block 2, frame 2 Client: Sending 4K block 2, frame 3 Server: ACK Request Client: Sending 4K block <n>, frame 1 Client: Sending 4K block <n>, frame 2 Client: Sending 4K block <n>, frame 3 Server: ACK Request <The server does it’s thing and responds> Server: Sending 4K block 1, frame 1 Server: Sending 4K block 1, frame 2 Server: Sending 4K block 1, frame 3 Client: ACK Request Server: Sending 4K block 2, frame 1 Server: Sending 4K block 2, frame 2 Server: Sending 4K block 2, frame 3 Client: ACK Request Server: Sending 4K block 3, frame 1 Server: Sending 4K block 3, frame 2 Server: Sending 4K block 3, frame 3 Client: ACK Request Server: Sending 4K block <n>, frame 1 Server: Sending 4K block <n>, frame 2 Server: Sending 4K block <n>, frame 3 Server: ACK Request
Well, that’s a lot more traffic, but nothing outrageous, on the other hand, that idea about multiple async sends being able to fill the pipeline clearly went away – the second send doesn’t start until the first is acknowledged. In addition, the sliding window never gets greater than 3K but that’s not the end of the world…
But see what happens when we add in delayed (or piggybacked) acks to the picture… Remember, CIFS uses NetBIOS semantics. That means that every byte of every send must be acknowledged before the next block can be sent.
Client: Request Transaction, sending <n> bytes, receiving <m> bytes Server: ACK Request Server: Ok, buffers allocated to receive <n> bytes, go ahead Client: ACK Request Client: Sending 4K block 1, frame 1 Client: Sending 4K block 1, frame 2 Client: Sending 4K block 1, frame 3 Server: wait 200ms for server response and ACK Request Client: Sending 4K block 2, frame 1 Client: Sending 4K block 2, frame 2 Client: Sending 4K block 2, frame 3 Server: wait 200ms for server response and ACK Request Client: Sending 4K block <n>, frame 1 Client: Sending 4K block <n>, frame 2 Client: Sending 4K block <n>, frame 3 Server: wait 200ms for server response and ACK Request <The server does it’s thing and responds> Server: Sending 4K block 1, frame 1 Server: Sending 4K block 1, frame 2 Server: Sending 4K block 1, frame 3 Client: wait 200ms for client request and ACK Request Server: Sending 4K block 2, frame 1 Server: Sending 4K block 2, frame 2 Server: Sending 4K block 2, frame 3 Client: wait 200ms for client request and ACK Request Server: Sending 4K block 3, frame 1 Server: Sending 4K block 3, frame 2 Server: Sending 4K block 3, frame 3 Client: wait 200ms for client request and ACK Request Server: Sending 4K block <n>, frame 1 Server: Sending 4K block <n>, frame 2 Server: Sending 4K block <n>, frame 3 Server: wait 200ms for client request and ACK Request
All of a sudden, an operation that looked really good at the high level protocol overview turned into an absolute nightmare on the wire. It would take over a second just to send and receive 28K of data!
This is the consequence of not understanding how the lower levels behave when you design higher level protocols. If you don’t know what’s going to happen on the wire, design decisions that look good at a high level turn out to be terrible when put into practice. In many ways, this is another example of Joel Spolsky’s Law of Leaky Abstractions – the network layer abstraction leaked all the way up to the application layer.
My solution to this problem when we first encountered it (back before NT 3.1 shipped) was to add the TDI_SEND_NO_RESPONSE_EXPECTED flag to the TdiBuildSend API that would instruct the transport that no response was expected for the request. The transport would then disable delayed acks for the request (if it was possible). Now for some transports it’s not possible to disable piggyback acks, but for those that can, this is a huge optimization.
Yesterday’s post was a simple implementation of TransmitFile. First things first: I need to apologize. I was being a bit sneaky in this case. The code in the example was written to function correctly, and that mislead a lot of people who were trying to find a bug in the code, when there was no bug in the code. There was something wrong with the code, but it was a performance issue, not a correctness issue (although arguably a performance issue is a correctness issue).
The problem was that the implementation didn’t consider the interaction between sending data and the Nagle Algorithm/delayed acks. These algorithms exist to solve two halves of a basic problem with networking protocols.
Whenever you send data from one machine to another, the sender must wait until the receiver acknowledges the request before sending the next buffer (this is a rough approximation, ignoring sliding windows etc). So when you look at the data flow for a send, on the wire, it appears like this:
Client: Send frame 1 to Server Server: Send acknowledgement for frame 1 to client Client: Send frame 2 to Server Server: Send acknowledgement for frame 2 to client etc.
There is a common misconception that the amount of time it takes to send a message from one machine to another is related to the size of the packet. On some networks this is true (dialup comes to mind immediately). But one of the fundamental properties of an Ethernet (or token ring) network is that sending one byte of data takes essentially the same amount of time as sending 1568 bytes of data (the max Ethernet packet size). The critical factor is the number of packets transmitted, not the size of each packet.
As a result, if the client sends small buffers of data, the server has to acknowledge each of the buffers before the client can send the next buffer. The Nagle Algorithm as represented in RFC896 coalesces those small writes from the client into a larger buffer, so that the acknowledgement traffic isn’t as daunting. After 100-200ms of inactivity on the part of the client, the TCP implementation will flush the data out. As a result, if you’re doing small writes to a socket, if nagling is enabled, then you’ll see that your writes don’t appear on the wire until 100-200 milliseconds after the write. That wasn’t what was happening in this case, however.
Now consider what happens with a normal request/response type protocol (like POP3):
Client: Send “USER” to Server Server: Send Ack for “USER” to Client Server: Send “+OK” to Client Client: Send Ack for “+OK” to Server Client: Send “PASS” to Server Server: Send Ack for “PASS” to Client Server: Send “+OK” to Client Client: Send Ack for “+OK” to Server Client: Send “UIDL” to Server Server: Send Ack for “UIDL” Server: Send “+OK” to client etc.
You’ll notice that there’s twice the number of frames being sent on the wire. And that each ack is immediately followed by a response from the server. Now remember that the acks are just as expensive as the data sends. It turns out that this behavior is known as the “Silly Window Syndrome”, or SWS. RFC1122 (the TCP specification) specifies a solution to SWS:
A TCP SHOULD implement a delayed ACK, but an ACK should not be excessively delayed; in particular, the delay MUST be less than 0.5 seconds, and in a stream of full-sized segments there SHOULD be an ACK for at least every second segment.
So the receiver can (and should) delay acknowledging the send before it sends the ACK so it can piggyback the ack on the response. This behavior is also codified in section 4.2 of RFC2581, TCP Congestion Control. This “piggyback-ack” behavior isn’t unique to TCP/IP by the way; the Netbeui protocol has also had it, for about as long as TCP has.
Now let’s consider the buggy TransmitFile implementation. In that case, the client’s issuing multiple sends, but there’s no response. And since the sends are synchronous, each send can’t begin until after the previous send has completed. And again, because they’re synchronous sends, TCP can’t buffer the data to coalesce the writes. It has no choice but to wait for the acknowledgement before the send is completed.
Compounding this problem is the size of the transmit buffer: 4096 bytes. It turns out that the NT SWS avoidance algorithm delays an acknowledgement every other frame (details are here, on page 34). Once you remove the TCP overhead from the 1568 byte Ethernet packet size, the maximum payload size for TCP on Ethernet is 1460 bytes per frame. That means that a 4096 byte buffer takes 3 frames to transmit. That’s an odd number of frames, so, the receiving system defers the response for 200 milliseconds in the hope that the receiver will respond to the sender.
And that’s the crux of the bug. The buffer size used for writing the file data is will take an odd number of frames, which means that each write takes 200 milliseconds. Simply increasing the buffer size to 8192 bytes will speed it up by several orders of magnitude, because 8192 byte buffers take up an even number of frames (6). This is a hack solution though, a more correct solution uses overlapped I/O to keep the network adapter saturated.
I’ve found that whenever ANYONE says that their network I/O’s are taking too long, the problem is almost always related to one of these two problems (either nagling or delayed acks). And this isn’t a theoretical problem either. Over the last weekend, I decided to write up this series of articles, and when I came in on Monday morning, I was told that there was a performance problem with the Windows Media Connect HTTP server. My first thought was that it was problem with the layer reading data under the server, after that was ruled out, my next thought was that the problem was deferred acks. And after about 15 seconds of looking at an ethereal trace, I realized quickly that that was the problem. Fortunately, the fix was relatively simple, but the problem is VERY real.
This was not an isolated case – about every six months or so, someone posts a message on the internal performance analysis alias asking for help with a networking performance problem, the answer is almost always related to either nagling or delayed acks.
Kudos: This turned out to be harder than was expected. Nicholas Allen came up with the right answer, and Simon Cooke expanded on it quite nicely.
And for the unanticipated bugs:
Simon Cooke pointed out that the algorithm sends 0 byte writes if the file size is exactly a multiple of the buffer size.
Anon pointed out that there’s no check for null buffers. This is critical if there’s no header or trailer specified, the code would crash if the user didn’t specify them.
And, because sockets are opened for overlapped I/O, the synchronous WriteFile call to write to the socket only works if there are no other threads interacting with the socket. So the article should have stipulated that the app was single threaded.
During my research for this article, I ran into a rather nice write-up on SWS and this problem in this article on speeding up web servers.