January, 2007

Larry Osterman's WebLog

Confessions of an Old Fogey
  • Larry Osterman's WebLog

    What is AUDIODG.EXE?

    • 41 Comments

    One of the new audio components in Vista is a new process named audiodg.exe.

    If you look at it in taskmgr, the description shows "Windows Audio Device Graph Isolation", but that's not really particularly helpful when it comes to figuring out what it does.

    The short answer is that audiodg.exe hosts the audio engine for Vista.  All the DSP and other audio processing is done in audiodg.exe.  There are two reason it runs outside of the windows audio service.

    The first is that there's 3rd party code that gets loaded into audiodg.exe.  Audio hardware vendors have the ability to install custom DSPs (called Audio Processing Objects or APOs) into the audio pipeline.  For a number of reasons (reliability, serviceability, others) we're not allowed to load 3rd party code into svchost processes (svchost.exe is a generic host process for services that's used inside Windows). So we need to move all the code that interacts with these 3rd party APOs outside the audio service (that way if an APO crashes, it won't take out some other critical part of the system with it).

    The second reason for using a separate process for the audio engine is DRM.  The DRM system in Vista requires that the audio samples be processed in a protected process, and (for a number of technical reasons that are too obscure to go into) it's not possible for a svchost hosted service to run in a protected process.

     

    So why audiodg?

    As I mentioned in my post "Audio in Vista, The Big Picture", the route that audio samples take through the audio engine can be considered a directed graph.  Internally, we refer to this graph as the "Audio Device Graph" (ok, strictly speaking we call the part to the left of the mixer as the local graph, and the part to the right of the mixer the device graph, but when we consider the big picture, we just call it the audio device graph).

    So why AudioDG?

    Originally we called the process DeviceGraph.Exe.  For a number of reasons that are no longer relevant (they're related to the INF based installer technology that was used before Vista), we thought that we needed to limit our binary names to 8.3 (it's a long story - in reality we didn't, but we thought we did).  So the nice long names we had chosen (AudioEngine.Dll, AudioKSEndpoint.Dll, and DeviceGraph.Exe) had to be truncated to 8.3.

    I felt it was critically important that all the audio components had to have the word "Audio" in the beginning to make it clear that they had to do with audio functionality.  Since we thought we were limited to 8.3 names, that meant we had 3 letters to play with in the name.  For AudioEngine.Dll, it was relatively simple - it shortened to AUDIOENG.DLL.  Similarly for AudioKSEndpoint.Dll, it shortened to AUDIOKSE.DLL.

    But DeviceGraph was somewhat more complicated.  I originally went with AudioADG.EXE (audio+ADG for Audio Device Graph), but people thought it was redundant (It expanded to audio audio device graph). 

    Eventually we settled on "AUDIODG.EXE".

    So why the funky description?  Well because it accurately reflects what audiodg is - it's a part of Windows, so you get "Windows", it hosts the "Audio Device Graph", and isolates it from the Windows Audio Service.

  • Larry Osterman's WebLog

    Vista Ship Gifts

    • 19 Comments

    Our ship gifts for Vista came the other day.  We got a fleece pullover and the final DVD for our DVD cube.

    Fully updated, the cube is a collection of 4 DVDs, containing the bits for Vista Beta1, Vista Beta2, Vista RC1, and Vista RTM.

    There's also a booklet: 

    If you can't read the text, it says "You are holding 3.5 Gigabytes of passion, dedication, and pure geek goodness"  "RTM NOVEMBER 8th 2006"

     

    Within is a timeline of pictures of computers running Win 1.0, Win 3.1, Win95, and Windows XP with the text "time to make history..."

    And then, the best part (IMHO).  A picture of Vista and the text "Handcrafted by:"

     

    Of course, I'm there:

    Btw, I'm using a coaster we were given when Microsoft moved into it's current campus 20 years ago as a paperweight.  I somehow thought it was appropriate.

     

    At the back is a bunch of pictures of the ship party and the text: "Wow.  What an incredible journey.  Thank you."

      

    Very cool.

  • Larry Osterman's WebLog

    Nostalgia Waxes

    • 13 Comments

    So apparently there's something going on today in New York City (I don't know what it is, since I'm not invited :)).

     

    However, a couple of weeks ago, someone dug up a piece of Microsoft nostalgia and posted it on Google Videos.  I saw the first couple of seconds and I shouted out "Hey, it's W-W-W-Windows!"

     

    Wow, that brought back HUGE memories. 

    The video was filmed in Building 2, where I worked - while they were filming it, Valorie and I were standing off camera watching the proceedings (they took over the lobby of the building and we couldn't walk through it).  My office at the time was just behind the receptionists desk.

    The "Nice Guy who sold her Windows/386" was Adrian King,who was the head of the OS division at the time.  He later left Microsoft with Rob Glaser to found Real Networks. Silly trivia: when you installed Real Player, back in the day, it played a short audio clip ("Welcome to Real", I believe).  It was Adrian's voice on the clip.

    "T. Bone Perkins" was really Frank Gaudette.  Frank was the CFO for Microsoft at the time.  Frank was known internally as the soul of Microsoft.  He used to give the most remarkable company meeting financial announcements.  He passed in 1993, but his legacy lives on in the Village Theatre's primary venue, the Francis Gaudette theater.

    Victoria Carver, the actress who played "Linda" later on did a series of ads for a local eyeglass company.  Every time Valorie and I saw her in the ads, we'd shout out "Hey, it's the W-W-W-Windows lady".

    Some other things I noticed...

    Did you notice the big brown Compaq 386 box?  IIRC it was the only one on the market at the time.  Also the prominant mention of OS/2.

    The clock at the very beginning was a Microsoft time-in-service award - they gave a clock away for 5 years in service, that was the 1st generation clock (I have the cheesy 2nd generation clock (my name's stuck on the battery cover - how lame can you get)).

    The reason that the video made SUCH a big deal about "Printing" is that OS/2 couldn't really print (or rather, that out of the box it only supported printing to about 5 different models of printers).

    Also, several of the extras standing around in the final shot were HR people.  The guy who was 2nd from the left at the final shot was John Jenkins from OEM sales.

     

    Edit: s/Susan/Linda/ ; Thanks Juan.

    Edit: Added more Frank comments.

     Edit: Added clock.

     

  • Larry Osterman's WebLog

    Software Contracts, Part 9: Annotations Outside the Compiler - More Runtime Enforced Annotations

    • 9 Comments

    Ok, if it's not become crystal clear that I'm writing this on-the-cuff, this post will finally put a nail in the coffin.

    My last post discussed runtime enforced annotations and I used the RPC runtime library as an example of a runtime which enforces contractual annotations.

    Of course I totally ignored the single most common example of a runtime enforced contractual annotation in favor of a relatively obscure (but near to my heart) example.

     

    Of course, the most common example of a runtime enforced annotation is our old friend ASSERT (and it's brethren).  Assertions are typically used to ensure that invariants are in fact maintained, and a function's contract is always invariant.  Thus assertions can (and IMHO should) be used to enforce contracts. 

    Remember my "what's wrong with this code" from yesterday?

    :
    HRESULT DoSomething(handle_t BindingHandle, const wchar_t *StringParam, int *ReturnedValue)
    {
       if (StringParam == NULL || ReturnedValue == NULL)
       {
          return E_POINTER;
       }
       <DoSomething>
    }
    :

    I asserted that the error check was incorrect because the RPC contract ensured that the StringParam and ReturnedValue parameters wouldn't be null.  Norman Diamond called me on it because it was possible that the function would be called incorrectly from inside the module.  Ultimately, he's right, there SHOULD be a check.  But it doesn't deserve to be a full-on check which returns an error, so the check ends up being just dead code.

    Dead code is bad for two reasons:

    First off, dead code can conceivably be used as an attack vehicle - because it's never been executed, it's never been tested, and a attacker might be able to find a way to use that dead code to exploit some vulnerability.

    But more importantly, dead code increases the number of code paths through your system.  One way of measuring the effectiveness of your test strategy is to measure the number of code paths that are executed by your test suites - in theory, the more code paths executed by the tests, the higher the quality of the tests.  But if you have dead code (which thus cannot be executed by the tests), the dead code reduces the code coverage numbers for your tests, leading you to believe that (on this metric) your tests are worse than they actually are.

    It's far better to enforce the contract for a situation like this with an assertion.

    :

    HRESULT DoSomething(handle_t BindingHandle, const wchar_t *StringParam, int *ReturnedValue)
    {
       _ASSERT(StringParam != NULL && ReturnedValue != NULL);
       <DoSomething>
    }
    :

    This is a far better implementation - it uses a runtime assertion to enforce the contract, but it doesn't increase the number of code paths.

    One of the interesting aspects of assertions as a mechanism for runtime contract enforcement is that they normally aren't checked in all cases.  Instead the assertion is usually present only on debug builds.  Thus assertions don't affect the performance of (and number of code branches in) retail builds.

     

    Next: Other kinds of annotations used to allow for contract enforcement.

  • Larry Osterman's WebLog

    Software Contracts, Part 8: Annotations outside the compiler - runtime enforced annotations.

    • 11 Comments

    Ok, it's taken 7 other posts, but we've finally gotten close to where I wanted to be when I started this series.

    Remember my definition of an annotation: An Annotation is an addition to the source code for a program that allows an external translator to enforce the program's contract.

     

    My first examples of annotations are annotations that are enforced by the compiler - utilizing a language's type system.  These annotations are highly useful because source code must be passed through the compiler before it's executed, thus allowing the compiler to correct the contract violation before it gets executed.

    Sometimes the annotation (and thus the contract) is enforced at runtime.  My favorite example of this occurs with the MIDL compiler.

    Here's a quick "what's wrong with this code":

    foo.idl:

    [
        uuid(<uuid>),
        version(1.1),
        pointer_default(unique),
    ]
    interface Foo
    {
        HRESULT DoSomething([in] handle_t BindingHandle, [in, string]const wchar_t *StringParam, [out] int *ReturnedValue);
    }

    foo.c:

    :

    HRESULT DoSomething(handle_t BindingHandle, const wchar_t *StringParam, int *ReturnedValue)
    {
       if (StringParam == NULL || ReturnedValue == NULL)
       {
          return E_POINTER;
       }
       <DoSomething>
    }
    :

    The error is that the code in DoSomething is checking for the StringParam and ReturnedValue being NULL.  The contract for the DoSomething function as expressed in the IDL file above specifies that the StringParam and ReturnedValue are not optional[1].

    For basic RPC, the RPC runtime library will enforce the functions contract as expressed in the IDL file (I'll talk about COM in a bit).    That means that on the client side, if you attempt to call DoSomething with an invalid parameter, the function will fail (it will raise an RPC exception code).  On the server side, the runtime library guarantees that StringParam points to a null terminated string and it will allocate the storage to hold the ReturnedValue parameter.  If you use the /robust flag (and if at all possible, you should always do this), the RPC runtime library will also protect your call from clients who bypass the client side runtime library - it will filter out all callers who don't provide input that matches the signature[2].

    In this example, since the input and output parameters don't have the "unique" or "ptr" attribute, by default they're "ref" pointers[3].  That means that they're always passed by reference, and reference parameters may not be null.  As a result, checking for a null value is pointless, since t can never happen.

     

    For COM, it's important to realize that this only happens when the RPC runtime library operates on the parameters.  The thing is, COM doesn't always get its hands on the function parameters.  In general, the RPC runtime library will only see the parameters for the function when the call has to cross a boundary (either an apartment, process, GIT or IDispatch boundary).  If you make a call to a method on an interface within your apartment, then for performance reasons, the RPC runtime library doesn't enter the picture.

    Next: Other forms of runtime enforced contracts.

     

    [1] For this example, I'm assuming that DoSomething is only called by the RPC runtime library - if it can be called from somewhere else, the checks may be appropriate.

    [2] Please note: The client can still provide a StringParam that points to a 500K long string in an attempt to overflow a local buffer - the only thing that RPC ensures is that the input matches the  contract as expressed in the IDL file.  There are other ways of ensuring that the string passed in is "reasonable".

    [3] Before I start seeing "Stupid Larry: See there in the interface definition - it says that the pointer default is "unique" - why are you saying that the pointers are "ref"?" in the comments:  It seems stupid, but that's the way it works - as MSDN documents, the pointer_default only applies to indirect pointers.  Top level pointers in parameter lists always default to "ref" pointers.

  • Larry Osterman's WebLog

    Software Contracts, Part 7: Contracts as annotations - language features used to express contracts

    • 14 Comments

    My last post on contracts introduced the idea that a languages type system can be used as a mechanism to expose the contract of a function.  It turns out that language designers started recognising this fact and they started adding annotations to types that allow the contract for functions to be expressed.

    As is to be expected, originally these annotations were relatively limited in scope - for instance, the C language added a concept of "type qualifier" when it was standardized (K&R had no such concept).  They only added a couple of simple qualifiers - const and volatile.  When a parameter to a function was declared to be "const", it meant that the function's contract included a guarantee not to modify the parameter [1]. 

    Of course, sometimes it was rather hard to decypher the difference between "const char *" and "char *const" (the first is a pointer to a constant character, the second is a constant pointer to a character), but the intent was there.  The C language specification treated the type qualifier mostly as a hint - it didn't do a particularly good job of enforcing them.

     

    For C++, the language designers went further than in C. They tightened up the semantics for "const" adding stricter type checks.  And they added new parameter type, a "reference" parameter.  When a function took a reference to a variable as a parameter, it essentially says "the contract for this function includes the ability to modify the contents of this variable".

     

     

    Next:  Contract Annotations enforced outside the compiler.

    [1] Yeah, I know about CreateProcess.

    Edit: Ok, I screwed up the definitions of const char * and char * const.

  • Larry Osterman's WebLog

    How the Magic of Windows Vista saved 38G of my data

    • 14 Comments

    I always love it when the operating system I run finds new ways to absolutely delight me.

    Yesterday, due to a stupid pilot error, I accidentally deleted all the music and pictures from my machine at work (8G of pictures and 30G of music).  This wasn't actually the end of the world, since all the data's backed up at home (thrice, see yesterday's post about single points of failure).  But my data's gone from my machine at work, which means I didn't get to annoy my neighbors by playing Avenue Q at full volume.

    Well, I was poking around, trying to figure out what had gone wrong with my machine, and I noticed the "Restore Previous Versions" tab when I navigated to the users\larryo directory.

    Just for grins, I clicked on it and guess what showed up?  Yup, the previous version of my music and photos directory!

    Wow, that's cool - so I selected the appropriate version of the folder and selected "Restore", it asked me if I wanted to restore the pictures, and like Magic, the system restored my files from my volume shadow store.

    Of course, the magic that made all this work is the Volume Shadow Copy Service which has been in Windows since XP.  In Vista, the VSS guys enabled it for most user data (by default only the system drive is managed).  This Technet article has some information about how to enable it and demo it.

    Ultimately, I don't care - my data's back and all is well and I didn't need to bring in one of my backup drives.  And Vista's a bit more magic to me.

     

    I love it when that happens.

  • Larry Osterman's WebLog

    Single Points of Failure

    • 14 Comments

    Over the weekend, my family had an annoying lesson in one of the basic principles of software engineering, the "Single Point of Failure". 

    You see, when we built our house, we decided to go with a "hai-tekku" heating solution, and we're paying the price for it.

    Normally, when you build a house you get two units installed - a hot water heater and a furnace.  For a house heated with natural gas, that means that you need to have two chimneys - one for the water heater and another one for the furnace.  That means you (a) need to have two holes punched in your roof, and (b) you send a ton of heat up the chimney of your furnace (because air is a lousy thermal conductor).  Hot water heaters are WAY more efficient than furnaces (because water is a pretty good thermal conductor, and thus retains heat better).

    Instead of relying on a more traditional hot water header/forced air heater combination, we chose to go with a Polaris hot water heater, which allowed us eliminate the furnace - instead, the hot water from the hot water heater is pumped into a heat exchanger and the residential air is warmed by the heat exchanger.

    This system works great - we get a huge amount of hot water (as a result of the large capacity tank and it's relatively quick charge time) and it usually "just works".

     

    Until Saturday afternoon.  Valorie came back from riding and discovered, much to her chagrin that we didn't have any hot water.  I went down and checked the tank and sure enough, the "Igniter" light was out - the igniter had died on the hot water heater.

    Well, having no hot water's not THAT big a deal - it's somewhat annoying, but it's no big deal.  We called the repair people and they came out today to repair the system.

     

    Remember above where I talked about our not having a furnace and instead using a heat exchanger to heat our house?  Well when the hot water heater went out, so did the heat inside our house.  When I left for work this morning, it was 52 degrees Fahrenheit inside the house - a bit nippy.

     

    Our hot water tank acted as a single point of failure for our homes environmental systems - not only did its failure take out the hot water, it also took out our heating system as well.

    Fortunately, we had power and so we were able to heat water, and throwing on a couple of sweatshirts fixed things.  It was also fortunate that Microsoft has lockers with showers in them so we were all able to get showered before we went to theater on Sunday :).

     

    For our next house, we need to consider if we need a backup heating solution or not.

     

    Edit: Fixed awful transliteration, thanks Norman.

     

  • Larry Osterman's WebLog

    Software Contracts, Part 6: Annotations

    • 13 Comments

    In short, an "annotation" is an addition to the source code for a program that allows an external translator to enforce the program's contract.

    Way back in the beginning, the ONLY way to discover a function's contract was by it's documentation.  A function's contract looked like (from the MS-DOS 2.0 reference manual):

    Get Time (Function 2CH)
        Call:
        AH = 2CH

        Return:
        CH
            Hour (0-23)
        CL
            Minutes (0-59)
        DH
            Seconds (0-59)
        DL
            Hundredths (0-99)

    Function 2CH returns the current time set in the operating system as binary numbers in CX and DX [...] Depending on how your hardware keeps time, some of these fields may be irrelevant.  As an example, many CMOS clock chips do not resolve more than seconds.  In such a case the volume in DL will probably always be 0.

    There was no other mechanism to let you know what a function expect (ok, your program would crash if you passed in invalid input, but that isn't particularly useful).

    That's the problem with documentation-only contracts - they're extraordinarily fragile.  As a result, starting way back then (in the 1950s and 1960s) language designers started adding annotations to computer languages to help ensure that the contract for functions was met.

    You've all seen these annotations - for instance, the first non trivial program in my copy of "A Practical Introduction to Pascal" published in 1978 has:

    PROCEDURE DRAWALINE(LENGTH : INTEGER)
       VAR I : INTEGER;
       BEGIN
       FOR I := 1 TO LENGTH DO
          WRITE('-');
       WRITELN
       END;

    The annotation is, of course the declaration of "length" as an integer.  When languages added type information to function prototypes, the type information functioned as annotations which allowed a software translator (the compiler) to enforce the contract.  It turns out that the compiler was in an ideal position to enforce the contract - by simply refusing to convert your high level program to machine code, it was quite simple for the compiler to ensure that you didn't violate the functions contract (too much).

    Now the enforcement of the "type" contract varied from language to language.  For example the Pascal compiler was quite strict about its enforcement of annotations - if the parameters provided to a function didn't strictly match those of the function it refused to compile the code.  The "C" language compiler, on the other hand, was rather lax about enforcing parameters - for instance, the caller of a function didn't actually HAVE to match the number of parameters in the declaration of the function.  Unfortunately this rather lackadaisical attitude meant that it was easy to write some rather awkward code.  In addition, it turns out that it was somewhat difficult to translate C's semantics to some computer architectures (RISC machines, in particular).  As a result, in the 1990's as a part of the standardization of the C language, it also was changed to be strongly typed.

    Type safety isn't a requirement - there are many languages that are essentially type-neutral (most scripting languages appear to be typeless (or have relatively weak type systems), for example), but that just means that it is harder to enforce strong contracts.

    Next: Language Annotations: Beyond Simple Types

  • Larry Osterman's WebLog

    20 years and going strong.

    • 18 Comments

    It was a day very similar to today.  The sun was shining (at least during the day), and there was a light dusting of snow on the ground.

    The weather forecaster was warning that snow was threatening, and Valorie and I were running around like crazy running last second errands.  We had to get a wine glass, and of course we had to get our hair done (obviously a bigger deal for Valorie than I).  When we were in the Grand Union shopping for stuff, we noticed the headline of the one of the tabloids: "Wedding Cake Bomb Kills Cheating Bride" (astonishingly I was able to find a reference for it).  We still have that newspaper, of course we HAD to get it - it was the perfect metaphor for the day.

    And then the big day came.  The florist had done a remarkable job, as had Valorie's hairstylist, it was perfect.  The Chuppah was a smidge droopy (we'd been told that the synagogue had poles to hold the tallit up, we didn't realize that they meant bamboo garden stakes in cement filled coffee cans until the day of the ceremony), but we managed.

    My sister Laura (who was 5ish at the time) and Brother Andrew (who was 18 months old) started the wedding procession, followed by the rest of the wedding party (Miriam Gullotta, Martine (her last name is escaping me right now) and Sue Woolfe on the bride's side, Jim Lane, David Byrne and Jeff Osterman on the grooms).

     

    Even though it's been 20 years, I still remember that evening vividly.  After the ceremony, the entire wedding party formed a receiving line.  Unfortunately the guests to the party seemed to believe that this was an invitation for them to take photos, not to congratulate the bride and groom :).

    The rest of the evening passed without event, and a great time was had by all (a major highlight was the guys performing a kick line). 

    By the next morning, the threat of snow had turned into a reality and everyone scrambled like crazy to get out of town before the snow really hit.

     

    Today, Valorie and I celebrate our 20th wedding anniversary. 

     

    It's been a long and wonderful journey.

     

    I love you.

     

    Edit To Add: At lunch today, my group was surprised by the Sweet Adelines Quartet "Happy Hour" who serenaded me with a selection of songs.  Thanks guys, I it was wonderful.

  • Larry Osterman's WebLog

    A Midsummer Night's Dream (The Musical) at KIDSTAGE TeenSelect!

    • 0 Comments

    Somehow I managed to forget (except for a passing reference) that Daniel's in THe Village Theatre's KIDSTAGE production of A Midsummer Night's Dream up in Everett, WA.

    He's got the role of Demetrious, one of the "fools these mortals be", in a Rock and Roll adaptation (by Arne Zaslove) of the classic.

    I'm pretty excited, this is Daniel's first KidStage production, I'm really looking forward to it.

     

     

    The show runs from February 9th to the 18th at the Everett Performing Arts Center, you can find more information about the production here.

  • Larry Osterman's WebLog

    Software Contracts, Part 5: Hold on a second, why do we care about this stuff anyway?

    • 15 Comments

    I'm more discombobulated than usual on this series, I totally missed the third article in the series when I should have gotten to it (this, btw, is why Raymond writes his stuff 8 months in advance - it lets him fix stuff like this).

    So consider this post the 2nd in the series (the first is "Software Contracts", the second is "there are two sides to every contract", this is the 3rd, the 4th is "Sometimes contracts are subtle", etc...).

    Why do we care about software contracts?

     

    Well, for the exact same reason we care about real-world contracts.  Contracts define the expectations between two parties.  Without fully understanding the contract for a function, you don't know how to correctly call it.

    Take the ReadFile example I mentioned the other day.  The ReadFile contract tells you which parameters to the function MUST be provided (hFile, lpBuffer, nNumberOfBytesRead), which MAY be provided (lpNumberOfBytesRead and lpOverlapped).  It also includes how you determne if the function succeded or not (or if the function has success/failure semantics).

    We're so used to interpreting software contracts that they become ingrained.  Normally, we don't even bother think about them, and for the most part, you don't need to wonder about them (just like in real world contracts).

    However the instant you step outside the simplest case, understanding the contract for an API becomes critical.

    As a simple example, consider one small aspect of the contract for the standard C++ library (from "Thread Safety in the Standard C++ Library"):

    A single object is thread safe for reading from multiple threads. For example, given an object A, it is safe to read A from thread 1 and from thread 2 simultaneously.

    If a single object is being written to by one thread, then all reads and writes to that object on the same or other threads must be protected. For example, given an object A, if thread 1 is writing to A, then thread 2 must be prevented from reading from or writing to A.

    It is safe to read and write to one instance of a type even if another thread is reading or writing to a different instance of the same type. For example, given objects A and B of the same type, it is safe if A is being written in thread 1 and B is being read in thread 2.

    These rules concisely lay out the threading guarantees for C++ library functions (to be honest, I really like this version of the text, usually I just hear it written as "An object is thread safe for reading from multiple threads or writing from a single thread").

    You know from this part of the contract (which applies to the Microsoft implementation of the container classes (I don't know if it's in the standard, since I don't have a copy of the standard)) that you can have multiple readers of an object, but the instant you have a single writer, you need to add some kind of a lock to isolate the readers from the writers.  On the other hand, you don't need a lock if all you're doing is reading the data.

    Without this text being a part of the contract, you MUST assume that it's not possible to call the container classes in the C++ library from multiple threads (because the contract doesn't say that you can).

     

    A failure to appreciate software contracts can result in a myriad of different bugs, including various and sundry security bugs.  In my experience, most subtle, hard-to-diagnose bugs ultimately turn out to be caused by a misunderstanding about the contract associated with a function.

    For example, it's long been known that strcpy is a haven for security bugs.  One of the naive suggestions for fixing strcpy bugs is to simply replace the calls to strcpy with calls to strncpy.  Unfortunately in many ways the strncpy API is just as bad as strcpy because it fills the destination string with null characters up until the length provided and doesn't ensure that the destination string is properly formed. But most people looking for a "safe" replacement for strcpy will ignore that part of the contract and thus introduce different security bugs while trying to fix existing problems (according to Michael Howard, this mistake has happened more than once in the wild). 

    If the people recommending replacing strcpy with strncpy fully understood the contract for strncpy, it's likely that they wouldn't have make that mistake (or would have added more caveats).

  • Larry Osterman's WebLog

    Software Contracts, Part 4: Documentation

    • 11 Comments

     

    The other day, I started talking about software contracts.  Today I'd like to start talking about how contracts are embodied.

    The primary place people go to look for a function's explicit contract is the documentation for that function, either in the published documentation or in the header for the function (that's why the routine header I mentioned way back here is so important), and why it's so critical to make sure that the header accurately reflects the function.

    So how are software contracts realized?  They can take several forms - contracts are either explict or implicit.  Explicit contracts are spelled out in the function's declaration or documentation.

    So let's take a look at an example of an embodiment of a software contract - the documentation for the Win32 ReadFile API.  According to the documentation, the function reads from a file starting at the current file pointer.  It also indicates that the function has synchronous and asynchronous modes of operation.

    There are 6 parameters to ReadFile: hFile, lpBuffer, nNumberOfBytesRead, lpNumberOfBytesRead, and lpOverlapped.  I'm doing to discuss the contracts for the three of them.

    The documentation for hFile says (in part):

    [in] A handle to the file to be read.

    The file handle must be created with the GENERIC_READ access right. For more information, see File Security and Access Rights.

    For asynchronous read operations, hFile can be any handle that is opened with the FILE_FLAG_OVERLAPPED flag by the CreateFile function, or a socket handle returned by the socket or accept function.

    The documentation for lpBuffer says:

    [out] A pointer to the buffer that receives the data read from a file

    The documentation for lpNumberOfBytesRead says:

    [out] A pointer to the variable that receives the number of bytes read.

    ReadFile sets this value to 0 (zero) before doing any work or error checking. If this parameter is 0 (zero) when ReadFile returns TRUE on a named pipe, the other end of the message mode pipe calls the WriteFile function with nNumberOfBytesToWrite set to 0 (zero).

    If lpOverlapped is NULL, lpNumberOfBytesRead cannot be NULL.

    If lpOverlapped is not NULL, lpNumberOfBytesRead can be NULL.

    If this is an overlapped read operation, you can get the number of bytes read by calling GetOverlappedResult.

    If hFile is associated with an I/O completion port, you can get the number of bytes read by calling GetQueuedCompletionStatus.

    If I/O completion ports are used and you are using a callback routine to free the memory that is allocated to the OVERLAPPED structure pointed to by the lpOverlapped parameter, specify NULL as the value of this parameter to avoid a memory corruption problem during the deallocation. This memory corruption problem causes an invalid number of bytes to be returned in this parameter.

    So let's look at what the contract for ReadFile says about these three parameters.

    First the explicit statements:

    1. The caller is responsible for providing a HANDLE input value that points to a opened Win32 handle.  The object to which the handle points has to have been opened for GENERIC_READ[1].
    2. The caller is responsible for providing a pointer to valid memory for the lpBuffer parameter.  That buffer MUST be of length at least nNumberOfBytesToRead in length.
    3. The caller is responsible for providing EITHER a valid lpOverlapped parameter (which points to valid memory of length sizeof(OVERLAPPED)) or a a valid lpNumberOfBytes (which points to valid memory of length sizeof(DWORD)) or both (that comes from the 2nd "If lpOverlapped..." clause.
    4. ReadFile guarantees that the value of lpNumberOfBytesRead is either 0 (on failure) or the number of bytes read (on success).  Under all circumstances, the lpNumberOfBytesRead parameter will contain the number of bytes read.

    Now the implicit parts of the contract:

    1. The HANDLE, lpBuffer, lpNumberOfBytesRead (and lpOverlapped) values need to remain valid for the lifetime of the API call (more on that in a smidge).
    2. The ReadFile API will write at most nNumberOfBytesToRead to lpBuffer.  In other words, lpBuffer[nNumberOfBytesToRead] will remain unchanged by the API.

     

    What things can you infer about the implementation as represented in the ReadFile contract:

    1. It's probably a bad idea to specify both an lpOverlapped and a lpNumberOfBytesRead parameter if you're using I/O completion ports - this comes from the 6th comment which discusses I/O completion ports and indicates that there's some mechanism that could cause memory corruption after completion.  There must be something that happens when I/O associated with a completion port runs.

     

    What is NOT included in the ReadFile contract?

    • Performance guarantees - there is no indication in the contract for ReadFile about how long the operation should take, programs can make no assumptions about this.
    • If the function fails, there are no guarantee about any side effects. For ReadFile, that's not a big deal, but for WriteFile it's more relevant - for WriteFile, the API doesn't include any guarantees of the state of the region of the file being written.  In addition, if ReadFile completes with an error, nothing can be assumed about the contents of the memory - it might have been updated with some data from the file, it might not[2].
    • There is no indication of the state of the contents of the buffer while the call to ReadFile is in progress - the only guarantee is that at the time the API call completes successfully, the contents of the buffer will contain the relevant data from the file.
    • What happens when multiple threads call the synchronous file APIs simultaneously.  This was explicitly left out of the contract because the results are undefined when you do it.

     

    One interesting discussion that came up w.r.t. the PlaySound API is that it's often difficult to know the lifetime of the API call.  ReadFile has the same issues - there's no way of knowing how long the call to ReadFile might take (it could literally be hours).  If the call to ReadFile is async, it means that the memory pointed to by the lpBuffer parameter must remain valid until the read completes.

    The good news is that for ReadFile it IS possible to know if the API has completed - for instance, you can use the HasOverlappedIoCompleted API to determine if an async read has completed (there are multiple other mechanisms that are well documented).  However it's critical that the memory pointed to by the lpOverlapped, lpNumberOfBytesRead and lpBuffer  parameters all remain valid until the time that the API call completes.

    Next: Annotations as a form of software contract.

     

    [1] Note: this is actually wrong.  ReadFile requires the file handle be open for FILE_READ_DATA for disk files, GENERIC_READ grants more rights than is necessarily required.

    [2] Note: For some kinds of errors, there actually IS a guarantee explicitly included in the contract - when reading from a named pipe in message mode, the ReadFile API can return a distinguished error (ERROR_MORE_DATA) which indicates that some data in the buffer may be valid.

     

    Edit: Added comment about simultaneous operations not being in the contract.  Also fixed a really stupid typo.

     

  • Larry Osterman's WebLog

    A Snowy Day...

    • 9 Comments

    Well, yesterday was "fun".  I left work at 5PM and got home at 10PM (to be fair, we stopped for dinner for an hour or so in the middle).

     

    ON THE other hand, this is what I see outside our front door this morning:

    What I see from my window right now:

     

    Across the street:

     

    I love snow days :) 

  • Larry Osterman's WebLog

    A couple of thoughts on the iPhone...

    • 27 Comments

    It's weird, but I just don't get it.

    At MacWorld SF yesterday, Apple unveiled their new iPhone.  I've got to say, it's drop-dead gorgeous.  But I'm trying to understand the hype behind it, and I'm somehow failing miserably.

     

    Yeah, I work for Microsoft, so of course I'm going to be perceived as biased, but actually I honestly like Apple's products.  Some of the things about them mystify me (like how you keep track of the software you've installed on your Mac), but in general they do an unbelievable job of producing insanely beautiful devices and experiences.  There's nobody in the world that does as good job of dealing with the whole package - they do an great job of sweating the details and making sure they get them right.

    And the iPhone is no different - as I said, the iPhone is flat-out gorgeous.  I'm REALLY impressed with what Apple's done, and the UX looks to be as seamless as every other thing they've done.  I saw the demos with my tongue hanging out of my mouth with a bit of drool dripping down my chin (ok, that was TMI, sorry).

     

    But I just don't see why everyone seems to believe that it's the next great thing.

    For one thing, has anyone seen someone holding and using one of these?  It's supposed to be available in 6 months, why aren't there samples on the floor?

    Also: Why hasn't Apple shown us videos of people using this thing?  They've got a video up of people interacting with the address book, but it's just a director demo - the videos of the device aren't videos showing the real use of it.

    Oh, and the phone videos don't show people dialing the phone - they show address book navigation, but that's it.  No pictures of users dialing the phone or interacting with an on screen keyboard or anything like that.  But for a device that's apparently being targeted as a smartphone-type device, the interaction with the keyboard is going to be critical.

     

    Other thoughts...

    I'm a bit concerned about battery life, especially given that you can't change the battery.  Historically it seems that iPod batteries have about a 18 to 24 month lifetime, after which the battery life seems to degrade.  For a portable media device that costs $250 that's not outrageous.  But for a cell phone where I'm locked into a 2 year contract, having the battery die on me mid-contract seems problematic.

    The phone has bluetooth (good), but there's been no discussion about hands free operation.  It seems to me that to be successful, it's going to have to have a good hands free story.

    Without buttons, I suspect that the usability will be somewhat limited - I suspect I'd go nuts without the tactile feedback associated with clicking a button.  I suspect they can fix that with audio feedback, but I'm not sure.

    Oh, and that really nice shiny surface is going to be scratched to a fare-the-well in your pocket (which is where most phones go).  I don't want to THINK about what my car keys would do to it.  So they need to have some kind of a bag/case for the phone.  That's ok, as long as they provide a hands free option (see above).  However if they think that the hands free mode is likely, then why all the cool UX.  Ultimately it boils down to forcing the users to make a choice: They can use the cool UX for the phone and have a horribly scratched phone (but clearly be one of the kool kids with your white wires and happy dancing silhouette), or you can put the phone in a bag and use hands free (in which case the cool UX isn't that interesting).

     

    Please note: I'm a bit of a technology Luddite.  Our TV at home is a 12 year old 27" Samsung TV (we've got a 100+ inch projection TV upstairs but we hardly ever use it (because it's inconvenient and noisy)).  We didn't have Tivo in our house until last year sometime.  Heck, I've only had a digital cell phone for a year now (before that I used the analog phone in my car).  So I'm not in the target demographic for the iPhone.  But I still just don't get the hype.

    On the other hand, it IS very pretty.

     

    Edit to add (before the deluge hits): What was I thinking of posting this?  Now all the mac folks are going to descend on my poor little blog.  I'm also going to re-iterate the disclaimer on the side: This posting is provided "AS IS" with no warranties, and confers no rights.   In addition, obviously this is MY opinion, not my employers.

     

     

  • Larry Osterman's WebLog

    How long is a WAV file?

    • 15 Comments

    One question that kept on coming up during my earlier post was "How long is it going to take to play a .WAV file?".

    It turns out that this isn't actually a hard question to answer.  The answer is embedded in the .WAV file if you know where to look.  Just for grins, I spent a few minutes and whipped up a function that will parse a WAV file and return the length of the function.

    Remember that a .WAV file is a RIFF file which contains a "WAVE" chunk, the "WAVE" chunk in turn contains two chunks called "fmt " and "data".  The "fmt " chunk contains a WAVEFORMATEX structure that describes the file.  It's roughly based on the "ReversePlay" sample (but I didn't learn about that sample until after I'd written this code :)).

    I'm using the built-in multimedia I/O functions, which have the added benefit of being able to parse RIFF files without my having to come up with a ton of code.

    #define FOURCC_WAVE mmioFOURCC('W', 'A', 'V', 'E')
    #define FOURCC_FMT mmioFOURCC('f', 'm', 't', ' ')
    #define FOURCC_DATA mmioFOURCC('d', 'a', 't', 'a')

    DWORD CalculateWaveLength(LPTSTR FileName)
    {
        MMIOINFO mmioinfo = {0};
        MMCKINFO mmckinfoRIFF = {0};
        MMCKINFO mmckinfoFMT = {0};
        MMCKINFO mmckinfoDATA = {0};
        MMRESULT mmr;
        WAVEFORMATEXTENSIBLE waveFormat = {0};
        HMMIO mmh = mmioOpen(FileName, &mmioinfo, MMIO_DENYNONE | MMIO_READ);
        if (mmh == NULL)
        {
            printf("Unable to open %s: %x\n", FileName, mmioinfo.wErrorRet);
            exit(1);
        }

        mmr = mmioDescend(mmh, &mmckinfoRIFF, NULL, 0);
        if (mmr != MMSYSERR_NOERROR && mmckinfoRIFF.ckid != FOURCC_RIFF)
        {
            printf("Unable to find RIFF section in .WAV file, possible file format error: %x\n", mmr);
            exit(1);
        }
        if (mmckinfoRIFF.fccType != FOURCC_WAVE)
        {
            printf("RIFF file %s is not a WAVE file, possible file format error\n", FileName);
            exit(1);
        }

        // It's a wave file, read the format tag.
        mmckinfoFMT.ckid = FOURCC_FMT;
        mmr = mmioDescend(mmh, &mmckinfoFMT, &mmckinfoRIFF, MMIO_FINDCHUNK);
        if (mmr != MMSYSERR_NOERROR)
        {
            printf("Unable to find FMT section in RIFF file, possible file format error: %x\n", mmr);
            exit(1);
        }
        // The format tag fits into a WAVEFORMAT, so read it in.
        if (mmckinfoFMT.cksize >= sizeof( WAVEFORMAT ))
        {
            // Read the requested size (limit the read to the existing buffer though).
            LONG readLength = mmckinfoFMT.cksize;
            if (mmckinfoFMT.cksize >= sizeof(waveFormat))
            {
                readLength = sizeof(waveFormat);
            }
            if (readLength != mmioRead(mmh, (char *)&waveFormat, readLength))
            {
                printf("Read error reading WAVE format from file\n");
                exit(1);
            }
        }
        if (waveFormat.Format.wFormatTag != WAVE_FORMAT_PCM)
        {
            printf("WAVE file %s is not a PCM format file, it's a %d format file\n", FileName, waveFormat.Format.wFormatTag);
            exit(1);
        }
        // Pop back up a level
        mmr = mmioAscend(mmh, &mmckinfoFMT, 0);
        if (mmr != MMSYSERR_NOERROR)
        {
            printf("Unable to pop up in RIFF file, possible file format error: %x\n", mmr);
            exit(1);
        }

        // Now read the data section.
        mmckinfoDATA.ckid = FOURCC_DATA;
        mmr = mmioDescend(mmh, &mmckinfoDATA, &mmckinfoRIFF, MMIO_FINDCHUNK);
        if (mmr != MMSYSERR_NOERROR)
        {
            printf("Unable to find FMT section in RIFF file, possible file format error: %x\n", mmr);
            exit(1);
        }
        // Close the handle, we're done.
        mmr = mmioClose(mmh, 0);
        //
        // We now have all the info we need to calculate the file size. Use 64bit math
        // to avoid potential rounding issues.
        //
        LONGLONG fileLengthinMS= mmckinfoDATA.cksize * 1000;
        fileLengthinMS /= waveFormat.Format.nAvgBytesPerSec;
        return fileLengthinMS;
    }

    Essentially this function opens the WAV file specified, finds the RIFF chunk at the beginning, locates the WAVE chunk, then descends into the WAVE chunk.  It locates the "fmt " chunk within the WAVE chunk, reads it into a structure on the stack (making sure that it doesn't overflow the buffer).  It then pops up a level and finds the "data" chunk.  It doesn't bother to read the data chunk, the only thing needed from that is the length of the chunk which is then used to calculate the number of bytes that are occupied by the samples in the WAV file.

    Once we have the format of the data, and the number of bytes in the data chunk, it's trivial to figure out how long the sample will take to play.

    Btw, please note that this only looks for WAVE_FORMAT_PCM samples - there are other constant bitrate formats that could be supported but I wanted to hard code this to just PCM samples (it IS just a sample program). 

     

    To verify that my calculation is correct, I took my function and dropped it into a tiny test harness:

    int _tmain(int argc, _TCHAR* argv[])
    {
        if (argc != 2)
        {
            printf("Usage: WaveLength <.WAV file name>\n");
            exit(1);
        }
        DWORD waveLengthInMilliseconds = CalculateWaveLength(argv[1]);
        printf("File %S is %d milliseconds long\n", argv[1], waveLengthInMilliseconds);
        DWORD soundStartTime = GetTickCount();
        PlaySound(argv[1], NULL, SND_SYNC);
        DWORD soundStopTime = GetTickCount();
        printf("Playing %S took %d milliseconds actually\n", argv[1], soundStopTime - soundStartTime);
        return 0;
    }

     If I run this on some of the Vista sounds, I get:

    C:\Users\larryo\Documents\Visual Studio 2005\Projects\WaveLength>debug\WaveLength.exe "c:\Windows\Media\Windows Exclamation.wav"
    File c:\Windows\Media\Windows Exclamation.wav is 2020 milliseconds long
    Playing c:\Windows\Media\Windows Exclamation.wav took 2281 milliseconds actually

    The difference between the actual time and the calculated time is the overhead of the PlaySound API itself.  You can see this by trying it on other .WAV files - there appears to be about 200ms of overhead (on my dev machine) associated with building the audio graph and tearing it down.

  • Larry Osterman's WebLog

    Sometimes you find the answers to questions you didn't even know you were asking...

    • 5 Comments

    Yesterday's post by Raymond struck a chord when I read it.  He discussed how the property sheets wrote to resource sections and how the kernel would patch up a resource section by making it read/write when the resource was written to.

     

    It turns out that this bit of functionality in the kernel (actually in ntdll.dll, fwiw) was removed in Vista.  How do I know this?  Well, it turns out that the PlaySound API depended on this functionality to play sounds from resources on 64bit machines. 

    And how did I know THAT?  Well, the Windows startup tone that's played during boot is a resource that's embedded in one of the system dll's, and is played with the SND_RESOURCE flag.  You see, on 64bit machines, if the wave format structure within the .WAV file is aligned incorrectly (on a 16 bit boundary, for example), the PlaySound API will move the wave format structure around until it is properly aligned.  This safe to do, because it's normally operating on an allocated in-memory copy of the WAV file (when you specify a filename).  But when you're playing from a resource, it copies the data around inside the resource - again, not a problem, because there's enough padding within the resource so that this is ok.

    However, when they removed the code to automatically mark touched resource pages as read/write, this code that moved the memory around started access violating.  We learned about it when every single 64bit machine in the core OS division started crashing on boot if you had a kernel debugger installed.

    Fortunately we were able to track down the change relatively quickly and the guys who made the change fixed it (they moved the logic that fixed up resources into our code).  While they were at it, they also made the same change to the property sheet logic, because it turns out that problem that Raymond mentioned is still a real issue - there are still apps with broken property sheets that require modifying resources on the fly.  Sigh.

  • Larry Osterman's WebLog

    Software Contracts, Part 3 - Sometimes implicit contracts are subtle

    • 32 Comments

    I was planning on discussing this later on in the series, but "Dave" asked a question that really should be answered in a complete post (I did say I was doing this ad-hoc, it shows).

     

    Let's go back to the PlaySound API, and let's ask two different questions that can be answered by looking at the APIs software contract (the first one is Dave's question):

    I am happy to fulfill my contractual obligations but I need to know what they are. If you don't tell them, how is the caller to know that you need their memory until the sound finishes playing?

    If I call PlaySound with the SND_ASYNC flag set, how can I know if the sound's been played.

    As I implied, both of these questions can be answered by carefully reading the APIs contract (and by doing a bit of thinking about the implications of the contract).

    Let's take question 2 first.

    The explicit contract for the PlaySound API states that it returns TRUE if successful and FALSE otherwise.  If you specify the SND_ASYNC, what does that TRUE/FALSE return mean though?  Well, that's not a part of the explicit contract, it must be a part of the impicit contract.

    Remember that the PlaySound API only has three parameters (the sound name, a module handle and a set of flags).  All of these parameters are INPUT parameters - there's no way to return the final status in the async case.  Since there's no way for the AP to return whether or not the sound successfully played, the only way that the return from the API contained an indication of the success/failure of playing the sound implies that the SND_ASYNC flag didn't actually do anything.  And that violates the principle of least surprise - if the SND_ASYNC flag was a NOP, it would be a surprise.

    And in fact all the call to PlaySound does is to queue the request to a worker thread and return - the success/failure code refers to whether or not the request was successfully queued to the worker thread, not to whether or not the sound actually played.

     

    No for Dave's question...

    First off: One critical part of interpreting software contracts is:  If you have a question about whether or not a function behaves in a specific manner, if it's not specified in the explicit contract, assume the answer is 'no' unless otherwise specified.

    Since the contract for PlaySound is currently silent about the use of memory in combination with the SND_ASYNC flag, you should always make the most conservative assumptions about the behavior of PlaySound.  Since the API documentation doesn't say explicitly that the memory can be freed while the sound is playing, you should assume that it shouldn't.  And that means that the memory handed to the PlaySound call must remain valid until the call to PlaySound has completed playing the sound.

     

    But even without that, with a bit of digging, you can come to the same answer.

    Here's how my logic works. Both of the givens below are either explicit or implicit in the contract.

    1. You own the memory handed to PlaySound - you are responsible for allocating and freeing it. You know this because PlaySound is mute about what is done with the memory, thus it has no expectations about what happens to the memory it uses (this is an implicit part of the contract).
    2. The default behavior for PlaySound is synchronous (you know this because the documentation states that the SND_SYNC flag is the default behavior) (this is an explicit part of the contract).

     

    You can also assume that the SND_ASYNC flag is implemented by dispatching some parts of the call PlaySound to a background thread.  This is pretty obvious given the fact that something has to execute the code to open the file, load it into memory, and play it.  You can verify this trivially by using your favorite debugger and looking at the threads after calling PlaySound with the SND_ASYNC flag.  In addition, there are no asynchronous playback calls in Windows, so again, it's highly unlikely the playback is done using some kind of interrupt time processing (it's possible, but highly unlikely - remember that PlaySound was written for Windows 3.1).  I actually went back to the Windows 3.1 source code for PlaySound and checked how it did it's work (there were no threads in Windows 3.1) - on Windows 3.1, if you specified the SND_ASYNC flag, it created a hidden window and played the sound from that windows wndproc.

    But even given this, we're not done.  After all, it's possible that the PlaySound code makes a private copy of the memory passed into PlaySound before returning from the original call.  So the decision about whether or not the memory passed into the PlaySound API can be freed when specifying SND_ASYNC really boils down to this: If PlaySound makes a private copy of the memory, then the memory can be freed immediately on return, if it doesn't, you can't.

    This is where you need to step back and make some assumptions.  Up until now, pretty much everything that's been discussed has been a direct consequence of how the API must work - SND_ASYNC MUST be implemented on a background thread, you DO own the memory for the API, etc.

    So let's consider the kind of data that appears in the memory for which the PlaySound API is called.

    Remember that most WAV files shipped with Windows (before Vista) were authored as 22kHz, 16 bit sample, mono files (for Vista, the samples are all stereo).  That means that each second of audio takes up 44K of RAM.  That means that all non trivial WAV files are likely to be more than 64K in size (this is important).  Again, consider that the PlaySound API was written for Windows 3.1 where memory was at a premium, especially huge blocks of memory (any block larger than 64K of RAM had to be kept in "huge" memory allowing the blocks to be contiguous. 

    If Windows were to take a copy of the memory, it would require allocating another block the size of the original block.  And on a resource constrained OS like Windows 3.1 (or Windows 95) that would be a big deal.

    Also remember my 2nd point above - the defaut behavior for PlaySound is synchronous.  That means that the PlaySound call assumes that it's going to be called synchronously. 

    Given the fact that PlaySound was originally written for Windows 3.1 and given that the default for PlaySound is synchronous, and given the size of the WAV files involved, it thus makes sense that the PlaySound API would not allocate a new copy of the memory for the .WAV file and instead would use the samples that were already in memory - why take the time to allocate a new block and copy its contents over when it was already available.

    Now this is a big assumption to make - it might not even be right.  But it's likely to be a reasonable assumption.

    So you should assume that PlaySound doesn't take a copy of the memory being rendered, and thus you need to ensure that the memory is valid across the life of the call.

     

    Btw, I just was told by the doc writers that they're planning on making this part of the contract explicit at some point in the future.

     

    Tomorrow: Let's look at some explicit contracts.

  • Larry Osterman's WebLog

    Software Contracts, Part 2 - There are two sides to every contract.

    • 11 Comments

    One thing that's critically important to understand when thinking about software contracts is that just like in real life, every contract has at least two sides to it. 

    For real world contracts (that is agreements between two or more parties), the contract embodies responsibilities for each party involved in the contract.  For example, if I enter into a contract to buy your house, my responsibility is to pay you the money for the house, your responsibility is to transfer ownership of the house to me.

    Just like real world contracts, software contracts typically two sides (they can have more than two sides, but those situations are relatively rare).  Typically each software contract includes responsibilities for both the caller of a function and the function itself.

    And it's critically important to understand your responsibilities w.r.t. the contract for a function.  Consider the contract for the GetFileAttributesEx function.  This function takes three parameters: The first is a path to the filename, the second is an infolevel, the third is an lpvoid pointer.

    From the point of view of the caller, the GetFileAttributesEx API will retrieve a WIN32_FILE_ATTRIBUTE_DATA structure for the file corresponding to the filename parameter.  The caller knows that the GetFileAttributesEx API will either return a non zero value, in which case the memory pointed to by the lpvoid pointer will be filled in with the WIN32_FILE_ATTRIBUTE_DATA structure (assuming that the infolevel is GetFileExInfoStandard, the only documented infolevel).

    However it's important to consider the other side of the contract.  From the point of view of the GetFileAttributesEx API, the caller is required to provide:

    • A valid pointer to a null terminated string that contains the name of the file to check in the filename parameter.
    • The value of GetFileExInfoStandard for the infolevel parameter.
    • A valid pointer to a block of memory that is at least sizeof(WIN32_FILE_ATTRIBUTE_DATA) bytes in length in the lpvoid pointer parameter.

    If the caller does not provide all of those pieces of information, then the caller has violated their half of the contract, and the GetFileAttributesEx API is under no obligation to honor it's half of the contract.

    For an example like this, both halves of the contract are blindingly obvious, but there are other examples that aren't as obvious.

     

    A great example of a situation where a caller unknowingly violated the contract for an API was discovered during testing for Windows Vista.  During one of our test passes of Vista, the unit test application for the PlaySound API started crashing deep inside the bowels of the PlaySound API.  Of course I was asked to investigate (since I own the PlaySound API).

    I was mystified by the failure - we were crashing while trying to access the samples for a sound being played.  In addition, the test didn't always fail, which was really quite strange.  I dug a bit deeper into the test application and realized that the test was effectively doing:

    memory = LoadResource(resourceId);
    PlaySound(memory, SND_MEMORY | SND_ASYNC);
    FreeResource(memory);

    The problem here is that when the test case called FreeResource memory, they violated a part of the PlaySound contract.  You see, when you call PlaySound with the SND_ASYNC flag and the SND_MEMORY flag together, the PlaySound contract requires that the memory remain valid until the sound has completed playing.

    It turns out that this part of the PlaySound contract is NOT explicit - nowhere in the current PlaySound documentation does it say that the memory must remain valid while the sound is being played (as of this writing - I've filed a bug report on it).

    But even though this behavior is not explicit, it's still a part of the contract, and must be honored by the caller.

  • Larry Osterman's WebLog

    Contractual obligations

    • 8 Comments

    Once upon a time, I watched the TV show The Paper Chase.  As I recall it, the feared and respected professor Kingman tought contract law, which could be thought as the organic chemistry of the legal profession (organic chem was the class that the chemical engineers at Carnegie-Mellon used to dread, because it was so difficult). 

    Software contracts are almost as complicated as legal contracts.  And we use them every single day without thinking about them.

    Every function that's ever been written specifies a contract.  It doesn't matter if the function is public or private, every single function has a contract.

    And just like in contract law, there's a HUGE body of work that exists to help specify the contract for a function.  This is a big topic, and I suspect it will take several posts to cover it (and I'm sure I can't do it justice).  Much like the "programming with style" series, I'm mostly going to be making this up as I go along - I have a huge number of thoughts about software contracts, and it may take a drunkards walk before I can finally get all them out.

    What is a software contract?

    "Let's start at the very beginning, a very good place to start".

    A function's software contract, simply put, defines the behavior of a function.  It allows the caller of a function to know what the function does, how it fails, and (in general) what happens when the function executes.

    Contracts can be both explicit and implicit - an explicit contract is expressed directly,

    A function's explicit contract is embodied in many forms, and is expressed in different places.  It's embodied in the documentation of the function (either in official published documentation or just in the comments in the function header).  It can be embodied in annotations applied to the function definition (SAL annotations are a great example of this).  It can be embodied in the names of the parameters to a function.  And some contracts are expressed by some well known conventions.

    Implicit contracts, on the other hand are never expressed.  Implicit contracts are embodied in something that is called "The principle of least surprise".  These are the unwritten parts of a function's contract that fall into the general area of "common sense".

    The principle of least surprise.

    Simply stated, the principle of least surprise is: "Whatever happens when a function executes, it should not behave in a manner that is unexpected".

    So CreateFile shouldn't reformat the hard disk if it fails.  Or, to use a somewhat less drastic version: If CreateFile fails, the filesystem is left in the state it was before the CreateFile call was made.  In other words, if you did a DIR of the filesystem before the failed call to CreateFile, and did the same DIR afterwards, you would get the same results.  This isn't to say that under the covers changes were made to the filesystem - that's possible. But the externally visible behaviors should remain the same.

    Similarly, an API that retrieves some data shouldn't modify that data.  Or an API that takes an input parameter shouldn't modify that parameter without notification.  As I've mentioned before, CreateProcess fails this.  If your function DOES violate the principle of least surprise, you need to make that violation explicit in the contract for your function (usually in the documentation, as was done in the CreateProcess example).

    Contract Enforcement

    The enforcement of a function's contract varies.  For example, when the RPC runtime library marshals a function from one process/machine/thread to another process/machine/thread, it enforces the contract specified in the MIDL annotations on the function.  So when you're writing the server stub for a function with the function prototype: "HRESULT Function([out] int * returnedInt);" you don't have to check for the "returnedInt" parameter being null - the RPC runtime library guarantees that the server side of the function will always have valid storage to hold that value.  Other times, the code of a function enforces the contract - it checks for non optional parameters being non null, for example.  And sometimes there are external tools that check for contract enforcement (the SAL annotations mentioned above are perfect examples of that - the compiler (with the /analyze switch) ensures that the contract specified in the annotation is met.

     

    Why do we care about software contracts? 

    We care about software contracts because they help us write better code.  In order to interact with a function, you MUST know its contract.  In addition, knowing a function's contract allows you to make certain assumptions, and allows you to avoid pitfalls.  For example, the problem in this example becomes blindingly obvious once you knew that CoUnitialize's contract destroyed the apartment on the last call to CoUnitialize, and that destroying an apartment destroyed all the objects created within that apartment (btw, I couldn't find any documentation that spelled this out - bad Microsoft :().

  • Larry Osterman's WebLog

    Where do you go to get answers to your technical questions.

    • 46 Comments

    One of the things I'm currently working on is analyzing our community efforts, so I'd like to turn the blog around and ask:

    When you have a technical question about a product, where do you go to look for answers?

    Places I know about (in no particular order):

    • My blog :).
    • Other Microsoft people's blogs :).
    • The MSDN Support forums.
    • The Microsoft Newsgroups.
    • Google/Live Search/Yahoo/pick your favorite search engine.
    • Paid Support.
    • Mailing lists (wdmauddev is a great example of this)

    I'm not just looking for programming questions - even questions like "where do I get a driver for my <whatever> card" or "how do I do <blah>" count.

     

    Any and all answers would be appreciated - I'm just trying  to understand the landscape right now.

    Edit to add: Btw, for those of you proposing "Google" as the generic answer, what happens when the answer isn't on the search engines? 

     

  • Larry Osterman's WebLog

    Where's Larry been?

    • 10 Comments

    Ok, 5+ weeks and no posts, what gives?

     

    Basically, after Vista shipped, I took a much needed vacation.  I was looking at losing 2 weeks of vacation at the end of the year and I decided to spend some of the time I would have carried over and just take off until the new year. 

    It's funny.  I used to ask my friends who retired early what they did with all that time, and they invariably answered: "It's not a problem at all - there's tons of stuff to do".  Ya know, they were right - there IS tons of stuff to do.

    I mostly spent my time off wrapping Christmas presents (my side of the family's Christmas was 20 people this year so there are a lot of presents to wrap (it'll be 24 people next year since my brother's family joins us on odd years), adding in Valorie's family and friends here in Seattle means we do presents for something like 40 people).  I also spent time schlepping the kids to various events and shopping and generally just puttering around.  It was really quite relaxing (and quite the change to how my life was before we shipped).

     

    And then we had the windstorm.  On the night of the storm, Valorie and I sat on our front porch and watched the trees across the street bend back and forth.  It was truly freaky - there was essentially no wind at street level, while the trees across the street were bending at least 20 degrees.  I was convinced they were going to snap.

    We lost power at about 6PM on Thursday the 14th.  What we didn't realize until later was that we lived at essentially ground zero for the windstorm powerline damage - while there was no damage at all in our neighborhood, the same couldn't be said about the power lines around our house.  On Friday morning, we left the house to get supplies (always after the fact) and drove to Woodinville-Duvall road which was simply devastated - trees across the road held up only by power lines, power poles snapped in half, traffic and street lights lying twisted in the middle of the road.

    While power was out, we never bothered with the shelter thing, we just ate out at restaurants and drove around taking care of last minute things (silly stuff like dealing with the load of laundry that was running in the washing machine when power went out).  Fortunately power came back at Microsoft on Saturday afternoon, so we were able to go and get showered in the locker rooms (thanks Lisa for the towels).  On Sunday night there were a half a dozen families scattered around my building who had decided to camp out for the duration.

    We left for the east coast on Monday the 18th still without power, the power didn't come back at home until sometime on the 22nd (I called into home daily and when the answering machine picked up, I knew we had power). 

     

    Christmas itself was very fun, it was great seeing everyone in our family again - we spent the first week with my mom in NYC (we saw Spamalot and the Big Apple Circus (a holiday tradition, this year's show was particularly impressive)).

    We next drove up to Boston and celebrated Christmas with the rest of the family, which was again great - lots of great food, conversations, etc.  Gift highlights for Christmas were: Sharron's getting her ears pierced, and Daniel receiving the most remarkable pair of Chuck Taylor Converse All Stars I've ever seen (I didn't even know they MADE shoes in metallic gold).  Here's a link to the shoes.

    And of course getting to see all the "littles" again was really cool - my cousins have 5 kids all under 6 between them so there are tons of really cute kids running around all the time. 

    We then returned to Albany for a couple of days with my immediate family - always a lot of fun, including a birthday dinner hosted by my sister Edie (we had chicken puff (an old family recipe ) and a Carvel cake (we don't get them on the West Coast)) :).

     

    And after all that, we then returned to Seattle and started dealing with the aftermath of the storm - everything in our freezers is toast, so we're restocking the house. 

    Oh, and now that I'm back at work, I'm dealing with six weeks of email backlog.  That's also "exciting", but in a different way.

    Coming up?  Well, Daniel's been cast as Demetrious in the Village Theatre's Kidstage production of "A Midsummer Night's Dream", so he's going to be in rehearsals essentially non-stop for the next 5 weeks, then he is in the ensemble for the Overlake School's production of "The Robber Bridegroom".  So we're gonna be doing a ton of schlepping :).

Page 1 of 1 (22 items)