April, 2005

Larry Osterman's WebLog

Confessions of an Old Fogey
  • Larry Osterman's WebLog

    So what's wrong with DRM in the platform anyway?

    • 53 Comments

    As I said yesterday, it's going to take a bit of time to get the next article in the "cdrom playback" series working, so I thought I'd turn the blog around and ask the people who read it a question.

    I was reading Channel9 the other day, and someone turned a discussion of longhorn into a rant against the fact that Longhorn's going to be all about DRM (it's not, there will be DRM support in Longhorn, just like there has been DRM support in just about every version of Windows that's distributed windows media format).

    But I was curious.  Why is it so evil that a platform contain DRM support?

    My personal opinion is that DRM is a tool for content producers.  Content Producers are customers, just like everyone else that uses our product is a customer.  They want a platform that provides content protection.  You can debate whether or not that is a reasonable decision, but it's moot - the content producers today want it.

    So Microsoft, as a platform vendor provides DRM for the content producers.  If we didn't, they wouldn't use our media formats, they'd find some other media format that DOES have DRM support for their content.

    The decision to use (or not use) DRM is up to the content producer.  It's their content, they can decide how to distribute it.  You can author and distribute WMA/WMV files without content protection - all my ripped CDs are ripped without content protection (because I don't share them).  I have a bunch of WMV files shot on the camcorder that aren't DRM'ed - they're family photos, there's no point in using rights management.

    There are professional content producers out there that aren't using DRM for their content (Thermal and a Quarter is a easy example I have on the tip of my tongue (as I write this, they've run out of bandwidth :( but...)).  And there are content producers that are using DRM.

    But why is it evil to put the ability to use DRM into the product?

  • Larry Osterman's WebLog

    Little Lost APIs

    • 32 Comments
    When you have an API set as large as the Win32 API set, sometimes APIs get "lost".  Either by forgetfulness, or by the evolution of the hardware platform.

    We've got one such set of APIs here in multimedia-land, they're the "aux" APIs.

    The "aux" APIs (auxGetNumDevs, auxGetDevCaps, auxGetVolume, auxSetVolume, and auxOutMessage) are intended to control the volume of the "aux" port on your audio adapter.

    It's a measure of how little used these are that when I asked around my group what the aux APIs did, the general consensus was "I don't know" (this isn't exactly true, but it's close).  We certainly don't know of any applications that actually uses these APIs.

    And that's not really surprising since the AUX APIs are used to control the volume of either the AUX input jack on your sound card or the output volume from a CDROM drive (if connected via the analog cable).

    What's that you say? Your sound card doesn't have an "AUX" jack?  That's not surprising, I'm not sure that ANY sound card has been manufactured in the past 10 years with an AUX input jack (they typically have a "LINE-IN" jack and a "MIC" jack).  And for at least the past 5 years, hardware manufacturers haven't been connecting the analog CD cable to the sound card (it enables them to save on manufacturing costs).

    Since almost every PC system shipped in the past many years (at least 5) has used digital audio extraction to retrieve the CD audio, the analog cable's simply not needed on most systems (there are some exceptions such as laptop machines, which use the analog connector to save battery life when playing back CD audio).  And even if a sound card were to add an AUX input, the "mixer" APIs provide a more flexable mechanism for managing those APIs anyway.

    So with the "aux" APIs, you have a set of APIs that were designed to support a series of technologies that are at this point essentially obsolete.  And even if your hardware used them, there's an alternate, more reliable set of APIs that provide the same functionality - the mixer APIs.  In fact, if you launch sndvol32.exe (the volume control applet), you can see a bunch of sliders to the right of the volume control - they're labeled things like "wave", "sw synth", "Line in", etc.  If your audio card has an "AUX" line, then you'll see an "Aux" volume control - that's the same control that the auxSetVolume and auxGetVolume API controls.  Similarly, there's likely to be a "CD Player" volume control - that's the volume for the CD-ROM control (and it works for both digital and analog CD audio).  So all the "aux" API functionality is available from the "mixer" APIs, but the mixer version works in more situations.

    But even so, the "aux" APIs still exist in the system in the event that someone might still be calling them...  Even if there's no hardware on the system which would be controlled by these APIs, they still exist.

    These APIs are one of the few examples of APIs where it's actually possible that we might be able to end-of-life the APIs - they'll never be removed from the system, but a time might come in the future where the APIs simply stop working (auxGetNumDevs will return 0 in that case indicating that there are no AUX devices on the system).

    Edit: Clarified mixer and aux API relationship a bit to explain how older systems would continue to work.

  • Larry Osterman's WebLog

    What's wrong with this code, part 11: Launching notepad.

    • 25 Comments

    Anyway, I ran into this problem while I was writing some stuff at work.  I've restructured it to remove the "work-ness" of the code, but the problem still exists.

    It's short, but sweet:

    #include <stdio.h>
    #include <windows.h>

    #define PROCESS_NAME L"C:\\WINDOWS\\NOTEPAD.EXE"
    int _tmain(int argc, _TCHAR* argv[])
    {
        PROCESS_INFORMATION processInformation = {0};
        STARTUPINFO startupInfo = {0};
        DWORD status;

        if (GetFileAttributesW(PROCESS_NAME) == INVALID_FILE_ATTRIBUTES)
        {
            wprintf(L"Can't find " PROCESS_NAME L"; error %d, aborting\n", GetLastError());
            return(1);
        }

        startupInfo.cb = sizeof(startupInfo);

        if (!CreateProcessW(NULL, PROCESS_NAME, NULL, NULL, FALSE, 0, NULL, NULL, &startupInfo, &processInformation))
        {
            status = GetLastError();
        }

        wprintf(L"Status launching " PROCESS_NAME L" is %d\n", status);

        return 0;
    }

    Some things that are NOT wrong with this code:

    1. The code is unicode-only.  It is not intended to work in non unicode environments.
    2. The code expects that the system directory is C:\WINDOWS.  This is not a bug, it's by design.  The real code was more complicated this simplified version, but both show the same bug.

    It's possible that you won't see the bug if you take it and compile it.  But it is still there, and if you compile it in the NT build environment, you'll see the problem (you may also see the problem with current versions of Visual Studio, I was able to reproduce the problem quite easily.

  • Larry Osterman's WebLog

    Compressible Encryption

    • 23 Comments

    Time to spread a smidge of dirt on Microsoft :).

    One of my favorite dialog boxes is found in Outlook.  If you dig deep enough into your email accounts, you'll find the following dialog box:

      Outlook Offline File Settings Dialog

    The reason I like this dialog box is the default setting "Compressible Encryption".  Why?  Because if you select it, you're not encrypting ANYTHING.  "Compressible Encryption" is really "compressed".  When this option is selected, the data in the OST in specified is compressed (I'm not sure of the algorithm).

    Calling a compressed OST file "encrypted" is sort of like saying that a ZIP file is an encrypted version of the file.  After all, if you look at the contents of the ZIP file, you'll not find any the information directly represents the original file (ok, the filenames might be in the archive uncompressed but that's about it).  But of course it's not encrypted.

    If you specify "High Encryption" then you get a truly encrypted OST file.  I'm not sure of the algorithms they use, but it IS really encrypted.

    So why on earth do they call it compressible encryption?  Well, I'm not 100% sure, but I suspect that the answer is that some executive decided to type their PST file (or OST file) and noticed that their email was found in clear text within the file.

    They also noticed that if they used compression on the PST file, then they weren't able to see the contents of the file.  So they equated compression with encryption (hey, they couldn't see the data, could they?).  And thus "compressible encryption" was born.

    It's really just a silly affectation - they should never have called it "encryption" because someone might think that the data's actually hidden, but...  If the dialog was being designed today (the actual dialogs over 10 years old), the term "encryption" would never be used but nowadays it's sort-of a historical oddity.

    If you do a search for "compressible encryption", the first Google and MSN search hit is Leo Notenboom's article on compressable encryption, here's the official KB article on compressible encryption.

    There are other examples where similar obfuscation has occurred, and I'm sure that other vendors have done similar things (or worse).  For example, the Exchange MAPI client-to-Exchange Server protocol is obfuscated because an executive noticed that if he took a network sniff of the traffic between his client and the Exchange server he could see his email messages going by.  So he asked the team to obfuscated the stream - we knew that it did nothing, and so did the executive, but as he pointed out, it's enough to protect from casual attackers.  If you really want encrypted communications, then if you specify the "Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server" option in the Security tab of the Microsoft Exchange Server dialog, then that specifies RPC_C_AUTHN_LEVEL_PKT_PRIVACY, which uses a encryption mechanism to protect the data (I believe it's DES-56 but I'm not 100% sure).  I believe that this option is the default in all current versions of Outlook, but I'm not 100% sure.

  • Larry Osterman's WebLog

    Where's Larry?

    • 20 Comments
    Wow, what a weekend.

    Sorry about going dark without notice, I had this really great post (much funnier and more insightful than my usual drivel) ready to go last Wednesday when I got a phone call from my step-mom.

    "Your Dad's collapsed in court today, he's in the hospital, and we don't know what his prognosis is".

    Oh crud. 

    So as of about 12:30 on Wednesday, I went into total panic mode.  I called Valorie and she slipped into "hyper-competent super-mom" mode and by the time I'd gotten home at 1PMish, she'd already looked into flights, arranged for child care for the kids.  I was left to grab clothing and pack (and actually book the flights and arrange the rental car).

    Thank god for Valorie, I'm not sure I'd have been able to pull it together.  I'd also made some more phone calls and figured a bit more about what had happened.

    Dad had shown up for the last day of a trial he's been working on for the past 9 months.  After initial motions, he asked for a 15 minute recess to catch his breath.  The judge took one look at him and told my father that he was calling the EMTs.  Ten minutes later, he was in the White Plains hospital emergency room, where he fell unconscious (he had no blood pressure for a short while, and his blood electrolytes were totally messed up).  Norma (my step-mom) was called, and she started down to White Plains, and started calling the family together.

    By 3:30pm or so our time, I'd gotten some more updates, my father was looking much more stable, although he was still unconscious (on a respirator, etc).  Wednesday was the last day of Daniel's "Singing on Stage" class at SCT, so we went downtown to attend the final performance (from 7PM to 8PM).  At 8PM, the sitter showed up to pick up the kids, and Valorie and I headed down to the airport.  At this point, my father was stable, although still in the ICU.

    We flew a red-eye into LaGuardia airport, and rented a car to White Plains.  We got to the hospital about 11:30AM (more-or-less).  Dad was still in the ICU, but he'd been getting steadily better all night.  He was still wired up, but it wasn't as bad as I'd expected (I've watched too many episodes of Trauma: Life in the ER, I think).

    We were the last of the family to get there, my youngest brother from Colorado (where he's attending school), my youngest sister from DC (where she works), my younger brother and his wife from Scarsdale and my sister from Albany.  With my step-mom and Valorie, we totally filled up the small ICU waiting room.

    Over the course of the day, my father gradually regained consciousness, and by about 6 or so in the evening, they'd removed the respirator and feeding tube. 

    We all went out to dinner (except for Dad, of course) at Legal Seafoods, (I'd never been there before, but it was extraordinary).  All during dinner, my right eye was tearing, and it was getting more and more annoying.

    We all split up, Valorie and I to the hotel, the rest of the family to my brothers house.

    I had a really rough night - I kept waking up with a sore throat and my right eye was hurting more and more.  It was starting to get gunk in it too.  I eventually woke Valorie up at about 8AM, she took one look at my eye and announced that it was our turn to go to the ER.  It turns out that over the night, my watery eye turned into a full blown edema - it was swollen to the size of a golf ball, and the eyeball was bright red (with really gross green stringy bits across it). 

    So it was off to the ER for another Osterman male.  The doctor took one look at it and prescribed some antibiotics for it, and I was released to go to the ICU with the rest of my family.  Of course, the ICU nurse freaked and threw me out of the ICU the instant she saw my eye (Valorie kept referring to it as the "Eye of Sauron"), so I spent the day in the waiting room.  Dad improved over the course of the day, and by about mid-day, he was out of the ICU and had been brought up to a private room.

    While I was waiting in the ICU waiting room, I learned that my brother Jeff (who'd been running a 102 degree fever for the past couple of days) was also in his doctor's office for bronchitis.

    Not a good day for the Ostermen :).

    But in general things improved over Friday, and have continued to improve over the weekend.  Dad's had some minor setbacks (like some issues with his catheter), but most of the IV's are gone at this point, he's getting most of his meds orally, and he's hoping to get out of the hospital tomorrow or Wednesday.

    Valorie and I flew back to Seattle yesterday afternoon, arriving at about 10:00 or so in the evening - exhausted but much happier.  And the Eye of Sauron's finally gone, so I won't be frightening small children any more than usual.

    So that's what I did on my weekend :)

  • Larry Osterman's WebLog

    Stress reduction

    • 18 Comments

    Someone at work passed this around, it's a relaxing moment for our hectic days.

    It made me laugh, so...

     

     

    Edit: Since Sanjay's still getting hits from this one, a bit more info's involved.

    It turns out that Sanjay wasn't posting that to post an ironic commentary on the "scare people"  meme, instead he posted it to subtly announce his new venture, http://www.sanjaytandon.com.

     

  • Larry Osterman's WebLog

    Real world security by obscurity

    • 16 Comments

    I first heard about this issue on Car Talk the other day, and recently ran into this article on Snopes about it...

    It turns out that on VW cars (and other manufacturers), the pattern for the door key is based on the VIN for the car.

    What that effectively means is that the security of your VW (or other) car is based solely on the difficulty of cutting the keys for the lock - all the information needed to generate the keys is publicly available (and externally visible on the car).

    Imagine if the same were true of an operating system - what would we say about the security of a system where the recovery password for the system was etched onto the outside of the case, and if any helpdesk technician could come by, write down the recovery password and, using that recovery password bypass all the system protections?  Of course, the technician would have to own a $5,000 hardware decoding unit that would know how to convert the etched recovery password into the real password, but once they owned that decoding unit, they could bypass all the protections on your computer.

    Would you buy such a computer system?

     

  • Larry Osterman's WebLog

    Alignment (part 1)

    • 15 Comments

    I got an email the other day from someone (who will remain nameless) complaining about the fact that some of the NT structures had to be declared with #pragma pack:

    Consider file WinBase.h from the Platform SDK, containing the following
    declaration:
    > typedef struct _WIN32_FIND_DATAA {
    > DWORD dwFileAttributes;
    > FILETIME ftCreationTime;
    [...] 

    By this person's logic, since the ftCreationTime field in the WIN32_FIND_DATAA structure was an 8 byte FILETIME structure, the ftCreationTime should be at offset 8 from the start of the structure, but he discovered that it was at offset 4 from the start.

    And he was convinced that either there was something incorrect in the documentation for structure packing in Windows or that there had to be a hidden #pragma pack directive (which changes the default structure packing) in the Windows headers.

    I realized after reading this article that many people have forgotten the lessons learned from the early days of MS-DOS.  This kind of stuff was known to every developer who worked in C and coded to MS-DOS.  You see, back in the MS-DOS days, memory was king.  So DOS packed all of its data structures as tightly as possible to save memory.

    And if you attempted to make MS-DOS system calls from C, you invariably ran into packing issues.  For example, the MS-DOS get country data (which returned internationalization information) returned a structure that contained:

    Offset Length Description
    0x00 2 Date format
    0x02 5 Currency Symbol (ASCIZ string)
    0x07 2 Thousands separator (ASCIZ string)
    0x09 2 Decimal separator (ASCIZ string)
    0x0b 2 Date separator (ASCIZ string)
    0x0d 2 Time separator (ASCIZ string)
    0x0f 1 Bit Field
    0x10 1 Currency places
    0x11 1 Time format
    0x12 4 Case-map call address (DWORD)
    0x16 2 Data-list separator (ASCIZ string)
    0x18 10 Reserved

    If I was to represent this as a C structure, the naive representation would be:

     struct INTL_DATA
    {
      WORD _DateFormat;
      CHAR _CurrencySymbol[5];
      CHAR _ThousandsSeparator[2];
      CHAR _DecimalSeparator[2];
      CHAR _DateSeparator[2];
      CHAR _TimeSeparator[2];
      BYTE _Padding;
      BYTE _CurrencyPlaces;
      BYTE _TimeFormat;
      LPVOID _CaseMapCallAddress;
      BYTE _DataListSeparator[2];
      BYTE _Reserved[10];
    };

    The problem is that this structure definition wouldn't work.

    You see, the compiler has a fairly straightforward set of rules defining how structures are aligned, and this structure violates them.

    The compilers rules are:  In general, data is aligned on it's "natural" boundary.

    What's that mean?  "Natural" boundary?  What on earth is that thing?

    Typically, the "natural" alignment of data is based on its size.  A 1 byte field can be located at any address in memory.  A 2 byte field (a short) should only appear at even addresses in memory.  A 4 byte field (a long) should only appear at multiples of 4 bytes in memory.  And an 8 byte field (a longlong) should only appear at multiples of 8 bytes in memory.

    A simple example goes a long way towards explaining this:

    struct A
    {
       int _FieldA1;
       char _FieldA2;
       short _FieldA3;
       char _FieldA4;
       long _FieldA5;
       void *_FieldA6;
    };

    Consider what the compiler's going to do with a variable of type struct A.  The first thing it does is to lay the structure out in "memory":

    Field Index Field Name Field Size
    0 _FieldA1 4
    1 _FieldA2 1
    2 _FieldA3 2
    3 _FieldA4 1
    4 _FieldA5 4
    5 _FieldA6 4 (8 on 64 bit)

    The next thing it does it to assign an offset for each field:

    Field Index Field Name Field Size Field Offset
    0 _FieldA1 4 0
    1 _FieldA2 1 4
    2 _FieldA3 2 6
    3 _FieldA4 1 8
    4 _FieldA5 4 12
    5 _FieldA6 4 (8 on 64 bit) 16

    This is where the "natural" alignment comes to play - The natural alignment for _FieldA3 is 2 bytes (it's a word), so it gets put at offset 6 - there's some empty space left in the structure between _FieldA2 and _FieldA3 (and between _FieldA4 and _FieldA5).  The overall "sizeof" struct A is 20 bytes (24 on 64 bit platforms).

    For a given structure, the alignment of the structure is determined by the worst case field within the structure.  So in struct A's case, the alignment of the structure is either 4 bytes (on 32bit platforms) or 8 bytes (on 64bit platforms).

    When the compiler lays out nested structures, it just follows its rules recursively.  Consider this structure:

    struct B
    {
       short _FieldB1;
       struct A _FieldB2;
       int _FieldB3;
       struct A _FieldB4;
    }

    Field Index Field Name Field Size
    0 _FieldB1 2
    1 _FieldB2.FieldA1 4
    2 _FieldB2.FieldA2 1
    3 _FieldB2.FIeldA3 2
    4 _FieldB2.FieldA4 1
    5 _FieldB2.FieldA5 4
    6 _FieldB2.FieldA6 4 (8 on 64bit)
    7 _FieldB3 4
    8 _FieldB4.FieldA1 4
    9 _FieldB4.FieldA2 1
    10 _FieldB4.FIeldA3 2
    11 _FieldB4.FieldA4 1
    12 _FieldB4.FieldA5 4
    13 _FieldB4.FieldA6 4 (8 on 64bit)

    And again, the compiler then assigns offsets to the fields:

    Field Index Field Name Field Size Field Offset
    0 _FieldB1 2 0
    1 _FieldB2._FieldA1 4 4+0=4 (8+0=8 on 64 bit)
    2 _FieldB2._FieldA2 1 4+4=8 (8+4=12 on 64 bit)
    3 _FieldB2._FIeldA3 2 4+6=10 (8+6=14 on 64 bit)
    4 _FieldB2._FieldA4 1 4+8=12 (8+8=16 on 64 bit)
    5 _FieldB2._FieldA5 4 4+12=16 (8+12=20 on 64 bit)
    6 _FieldB2._FieldA6 4 (8 on 64bit) 4+16=20 (8+16=24 on 64 bit)
    7 _FieldB3 4 24 (32 on 64 bit)
    8 _FieldB4._FieldA1 4 28 (40 on 64 bit)
    9 _FieldB4._FieldA2 1 28+4=32 (40+4=44 on 64 bit)
    10 _FieldB4._FIeldA3 2 28+6=34 (40+6=46 on 64 bit)
    11 _FieldB4._FieldA4 1 28+8=36 (40+8=48 on 64 bit)
    12 _FieldB4._FieldA5 4 28+12=40 (40+12=52 on 64 bit)
    13 _FieldB4._FieldA6 4 (8 on 64bit) 28+16=44 (40+16=56 on 64 bit)

    Note that _FieldB2._FieldA1 has different offsets depending on whether it's 32bit or 64bit - this is because of the rule I mentioned earlier - _FieldB2 is aligned to the natural alignment of struct A, which is 4 bytes on 32 bit platforms and 8 bytes on 64 bit platforms.

    It's also interesting to see what happens with a slight rearrangement of the fields in struct A:

    struct A
    {
       int _FieldA1;
       char _FieldA2;
       char _FieldA4;
       short _FieldA3;
       long _FieldA5;
       void *_FieldA6;
    };

    All I did was to move _FieldA4 up next to _FieldA2.  But this change dramatically changed what happens when the structure is laid out in memory:

    Field Index Field Name Field Size Field Offset
    0 _FieldA1 4 0
    1 _FieldA2 1 4
    3 _FieldA4 1 5
    2 _FieldA3 2 6
    4 _FieldA5 4 8
    5 _FieldA6 4 (8 on 64 bit) 12 (16 on 64 bit)

    So on 32bit platforms, by just moving one field, the structure's memory footprint shrunk by 4 bytes!  Note that the size of the data contained in the structure didn't change at all - all that was changed was the packing of the data in the structure.  Also note that the total size of the structure didn't change on 64 bit platforms - this is because _FieldA3 pushes the alignment off on _FieldA6. 

    So let's go back to the original question about WIN32_FIND_DATAA.  The structure in winbase.h is:

    typedef struct _WIN32_FIND_DATAA {
    DWORD dwFileAttributes;
    FILETIME ftCreationTime;
       :
    How does this get laid out in memory?

    Lets run through the exercise above.

    A FILETIME is:

    typedef struct _FILETIME {
        DWORD dwLowDateTime;
        DWORD dwHighDateTime;
    } FILETIME, *PFILETIME, *LPFILETIME;

    So, continuing as before:

    Field Index Field Name Field Size Field Offset
    0 dwFileAttributes 4 0
    1 ftCreationTime.dwLowDateTime 4 4
    2 ftCreationTime.dwHighDateTime 4 8
    :     : : :

    And the mystery of my reader is solved - ftCreationTime is aligned at offset 4 because it only has offset 4 member variables.

    And finally, lets consider the INTL_DATA structure mentioned above - why did I say that it didn't work?

    Field Index Field Name Field Size Field Offset
    0 _DateFormat 2 0x0
    1 _CurrencySymbol 5 0x2
    2 _ThousandsSeparator 2 0x7
    3 _DecimalSeparator 2 0x9
    4 _DateSeparator 2 11
    5 _TimeSeparator 2 13
    6 _Padding 1 15
    7 _CurrencyPlaces 1 16
    8 _TimeFormat 1 17
    9 _CaseMapCallAddress 4 20
    10 _DataListSeparator 2 24
    11 _Reserved 10 26

    We're just fine up until we get to the _CaseMapCallAddress field.  That one's supposed to be at offset 18 according to the MS-DOS documentation, but it's at offset 20 in the C structure!  This is because the natural alignment of a far pointer to a function was 32bits even on Win16.

    Tomorrow, I'll write about how this was resolved for MS-DOS clients (and refine the packing algorithm mentioned above)

    Most of the content in this post was already been posted in GrantRi's blog post Alignment from last summer, it's an excellent reference to the topic.  In addition, Raymond Chen wrote about the FILETIME issue here.

  • Larry Osterman's WebLog

    Alignment (part 2): Packing

    • 14 Comments

    Yesterday, I wrote a bit about how the C compiler determines the alignment of structures.  I left with an example of an MS-DOS structure that didn't follow the alignment rules.

    So how do you deal with this?

    The first question that comes to mind is "What's the language standard say about this behavior?".  Well, this morning, I chased down what appears to be an online version of the C standard.  In section 6.5.2.1, you find:

    [#11] Each non-bit-field member  of  a  structure  or  union
           object   is  aligned  in  an  implementation-defined  manner
           appropriate to its type.

    Hmm. The standard says that the structure alignment is implementation defined, that's not much of a help.

    It turns out that Microsoft's C compiler defines a series of #pragma's that allow the developer to specify the structure alignment, as I implied yesterday, they're called #pragma pack.  Essentially the #pragma pack allows the caller to override the compilers default rules for structure alignment.  If you say #pragma pack(4) you're saying "if the natural alignment of this structure is greater than 4, treat it as 4 instead".  Similarly, #pragma pack(1) says "Treat all the members of this structure as having a natural alignment of 1 byte".  And #pragma pack(16) says "Treat all members as having a natural alignment of 16 - essentially it's a NOP on current architectures".  You can specify the packing for an entire source file with the -Zp compiler command line switch, but that's usually hideous overkill.  Instead, most people just use #pragma pack(n) around their structure definitions.

    So lets look at the examples from yesterday and see what #pragma pack does to the structure.

    First, here's struct A:

    struct A
    {
       int _FieldA1;
       char _FieldA2;
       short _FieldA3;
       char _FieldA4;
       long _FieldA5;
       void *_FieldA6;
    };

    Lets consider what happens with struct A when compiled with #pragma pack(1), #pragma pack(4) and #pragma pack(16):

    Field Name Field Size Field Offset (default) Field Offset (#pragma pack(1) Field Offset (#pragma pack(4) Field Offset (#pragma pack(16)
    _FieldA1 4 0 0 0 0
    _FieldA2 1 4 4 4 4
    _FieldA3 2 6 5 6 8
    _FieldA4 1 8 7 8 12
    _FieldA5 4 12 8 12 16
    _FieldA6 4 (8 on 64 bit) 16 12 16 20

    So if you specify #pragma pack(1), you get the tightest possible packing for data.

    Now why on earth might you want to specify #pragma pack?  It turns out that for the vast majority of applications it doesn't matter.  For example, most of the Win32 API set is defined without packing (there are some exceptions like some of the messages for common controls).  And more importantly, you shouldn't ever have to specify a structure packing before including any of the Windows header files - the windows header files are supposed to be built to work regardless of any external compiler switches or flags.  This is why you see all the "#include <pshpack2.h>" etc in the header files - this ensures that the structure packing rules are locked in regardless of the -Zp switch.

    But there IS one situation where this becomes important.  As I mentioned yesterday, some operating systems (like MS-DOS) have their data structures built with #pragma pack(1).  And some networking protocols (such as the CIFS protocol) are defined with packed structures.  So if you define structures to interact with those protocols, then you need to use #pragma pack(1) to ensure that your structure definition lines up with the alignment of the networked protocol.

    One other thing to keep in mind. The C language contract for allocators (malloc, free, etc) is:

    The  pointer  returned  if  the allocation  succeeds  is  suitably aligned so that it may be assigned to a pointer to any type of object and then used to access  such  an object  or an array of such objects in the space allocated (until the  space  is  explicitly  freed  or reallocated). 

    What this means is that a C/C++ allocator cannot return an address whose address isn't at least a multiple of 8 (the largest native allocation alignment). In reality, most allocators return memory aligned on a 16 or 32 byte boundary to take advantage of cache line effects.

    Edit: Corrected rules for #pragma pack(16)

  • Larry Osterman's WebLog

    Hacking the Decsystem 20

    • 14 Comments
    A bit of truly ancient history today, and it's not even Microsoft related.

    Back when I was in college, we did all our class work on Decsystem 20's running TOPS-20.  I've got to say that the dec20 is STILL my favorite processor architecture, even if it did have its quirks (like a 36 bit word size).  In particular, I LOVED the COMND% JSYS (Jump to SYStem, or system call) call - it integrated a command line parser and help system in a single system call.

    Two of my friends in college, Alex Kluge (my roommate at the time) and Vince Fuller (there were several others, but Vince and Alex immediately come to mind) spent a fair amount of their time trying to hack the campus systems (hey, we were young and some of the more nuanced issues of the morality of that kind of action escaped us).  I was never particularly competent at it (lacking the right mental attitude), but Alex and Vince were rather good at it.  I distinctly remember the head of the computer center commenting that both Vince and Alex (mostly Vince) were responsible for finding most of the bugs in their system.  IIRC, after graduation, he ended up hiring Vince :)

    There were two ways that Vince and Alex used to mess with the systems.  The first was to find a bug that would crash the mainframe and exploit it.

    Nowadays, in this world of personal computers, we forget what a big deal this was.  The CompCenter Dec20's typically had hundreds of users logged into them at any time, most of us trying desperately to get work done for the assignment that was due the next day.  So when the computers crashed, it was a big deal - hundreds of people would lose their work.

    On the other hand, immediately after the machines crashed, it would take several minutes for those people to log back on.  And some of them would give up in frustration and go back to their dorms until fewer people were on the systems.  So if, somehow, you could predict when the machines would crash, you could have a few minutes where the system was less loaded (and thus get more work done).

    So knowing how to exploit a bug that would crash the mainframe was pretty handy piece of information.  And these guys tried every way they could to figure out how to crash the machine.  I vaguely remember that one of them exploited a bug in the addressing model of the processor - they set the "memory indirect" bit on the first word of every page in memory and then executed an instruction that then forced every page in the 18 bit data address space to be faulted into memory simultaneously.

    The other thing that they'd do was to try to patch the kernel so that they'd be able to gain supervisor (root) access to the system.  But it wasn't enough to patch the running kernel - whatever patch they came up with had to be able to survive a reboot.  In todays terms, this is sort-of the equivalent of installing a rootkit on the machine.

    So they took to patching obscure system calls that would never be used.  Their favorite call was the LIGHTS% JSYS, which was used to manipulate the non-existant console lights on the Dec-20.  As such, there was no legitimate code that would ever use the LIGHTS JSYS, so they would patch the LIGHTS JSYS code in the kernel image to enable superuser privilege, and then be on their merry way.  Eventually (fairly quickly actually) the system administrators figured out what they were doing and installed checks to ensure the integrity of the system binaries, which raised the bar for them.

    I guess the bottom line is that some of the actions of hackers today aren't new at all - they've been going on for at least 25 years now, if not longer.

    Btw, I in no way condone what these guys did, and I never participated in those activities.  They knew the risks when they took them.  I know of at least one person who ended up getting a degree in music composition from a rather prestigious university instead of getting a CS degree because he participated in some of this kind of activity when he was in college, so the risks of hacking systems are very real :)

  • Larry Osterman's WebLog

    More surrealism

    • 14 Comments

    So I'm in a training course this afternoon. And the guy in front of me is surfing the web on his tablet.

    And I noticed what was on his screen...

    http://blogs.msdn.com/larryosterman.

     

    Very, very wierd.  I doubt he even realized that I was behind him.

     

  • Larry Osterman's WebLog

    Surreal drive time

    • 13 Comments

    Every morning, I take Daniel to school.  He goes to school in Edmonds, which is about 20 minutes north of our home (and about 40 minutes from Microsoft).  It's an long-ish commute (7:30 to somewhere between 9:00 and 9:30) but it's worth it :).

    So this morning, I'm driving up I-405 and I notice something moving near the windshield wipers in my car.  This isn't unusual, leaves sometimes get caught there.

    We got to our exit and pulled off the highway onto city streets.

    And then I saw the head pop up from under the top of my hood.  A little pointed nose sniffed the air and the mouse huddled down against the wind.

    I pointed it out to Daniel and he indicated disbelief, until I pulled into his school and we popped the hood.

    There he was, a field mouse sitting on top of the engine of my car.  I guess he realized it was warm last night and decided to bunk down for the night.

    I went to get a cup to remove him, but by the time I got back to the car, he had disappeared.  I'm hoping he took the opportunity to run away, I'm sure I would.

    Very wierd.  This is the first time this has ever happened to me :)

     

  • Larry Osterman's WebLog

    Playing Audio CDs

    • 12 Comments

    One of the cool things about working in multimedia is that you sometimes have opportunities to play with cool bits of hardware.

    This article isn't about one of them :).

    Instead, it's about something really quite mundane.  Playing a CD programmatically.

    It turns out that there are at least three different ways of playing a CD programmatically on Windows, ranging from easy to hard.  I'll start with the easy version and move to progressively harder versions.  The three mechanisms are:
    1. Letting the Windows Media Player play the CD.
    2. Playing the CD with the MCI API set
    3. Playing the CD with Digital Audio Extraction.

    Let me start with a bit of details about an audio CD.

    An audio CD consists of a set of audio tracks which can be read from the CD.  The track database on the CD contains the start offset within the CD of each of the tracks on the CD, that's what a player uses to determine where to seek on the CD to play.

    Clearly, however you are going to play a track from the CD, you need to start by retrieving the track list.  So that's where we'll start.  I started with a console application built in visual studio 2003, adding ATL support (to hide the COM management goo).

    #import "wmp.dll" // Needed to import the WMP interface definitions.
    CWinApp theApp;
    using namespace std;
    int _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
    {
        int nRetCode = 0;
        CComPtr<WMPLib::IWMPCdromCollection> cdRomCollection;
        HRESULT hr;

        CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);

        // initialize MFC and print and error on failure
        if (!AfxWinInit(::GetModuleHandle(NULL), NULL, ::GetCommandLine(), 0))
        {
            _tprintf(_T("Fatal Error: MFC initialization failed\n"));
            nRetCode = 1;
        }
        else
        {
            hr = cdRomCollection.CoCreateInstance(__uuidof(WMPLib::WindowsMediaPlayer));
            if (hr != S_OK)
            {
                printf("Couldn't instantiate CDRom player: %x\n", hr);
                return 1;
            }
            long driveCount;
            hr = cdRomCollection->get_count(&driveCount);
            if (hr != S_OK)
            {
                printf("Couldn't get drive count: %x\n", hr);
                return 1;
            }
            if (driveCount == 0)
            {
                printf("Machine has no CDROM drives\n");
                return 1;
            }

            CComPtr<WMPLib::IWMPCdrom> cdromDrive;
            cdromDrive = cdRomCollection->Item(0);

            CComPtr<WMPLib::IWMPPlaylist> playlist;
            hr = cdromDrive->get_Playlist(&playlist);
            if (hr != S_OK)
            {
                printf("Couldn't get playlist for CDRom drive: %x\n", hr);
                return 1;
            }
            for (int i = 0 ; i < playlist->count ; i += 1)
            {
                CComPtr<WMPLib::IWMPMedia> item;
                hr = playlist->get_Item(i, &item);
                if (hr != S_OK)
                {
                    printf("Couldn't get playlist item %d: %x\n", i, hr);
                    return 1;
                }
                printf(_T("Track %d (%S)\n"), i, item->name.GetBSTR()); // leaks :)
            }
        }
        CoUninitialize();
        return nRetCode;
    }

    So what's going on here?  First off, we import the type library from wmp.dll.  This is because VS 2003 doesn't include the wmp.h header file that includes the type information, so instead we need to import it from the DLL directly.  Note that I'm NOT using the wrapper classes that #import gives you (except for using item->name in the printf above).  This is simply because the wrapper functions throw exceptions instead of returning errors and I prefer my error handling to be more explicit (it's a personal preference thingy).

    Everything starts with a WMP CDRom collection, which is a list of CDRom drives available on the computer.  We enumerate the collection to ensure that there's at least one drive available, then access the first drive in the collection (usually the D drive).  Assuming that there's an audio CD in the drive, this will return an IWMPCdrom object.  On the CDRom object, there's a playlist, which is the collection of audio tracks.  To enumerate the tracks on the CD, you can simply enumerate the items in the playlist collection, and print that information out.

    One of the cool things about letting Windows Media Player do the heavy listing is that it looks up the track info in the CD database.  Which means that you get a friendly name for the track.  That can be pretty darned cool.  It's also slow.  Another downside is that this example is written to the Windows Media Player 10 SDK, if you don't have WMP 10 on the target machine, it's very likely that it won't work.

    Tomorrow, I'll add the code to actually play audio.

  • Larry Osterman's WebLog

    Playing Audio CDs, part 3 - MCI

    • 8 Comments
    Today, I want to talk about one of the most weird and wonderful APIs in all of Windows.  It's also one of the oldest - the creation date on the source file is April 25, 1990.

    This API is the MCI command set.  It's an example of defining two separate APIs that do exactly the same thing -   There are really only two different MCI APIs, mciSendString and mciSendCommand.  They are utterly identical, the only difference between the two of them is that mciSendString sends its commands using strings, and mciSendCommand uses buffers.  There's a parser in winmm.dll that converts the parameters to mciSendString into the buffers specified by mciSendCommand.

    The MCI commands can be used for most multimedia functions - you can play audio CDs with the APIs, you can play videos (not wmv videos, but AVI files), you can play wave files.  It's really quite an extraordinary API set.

    So here's the CD player specialization for MCI strings:

    CMCIStringCDPlayer::CMCIStringCDPlayer(void)
    {
    }

    CMCIStringCDPlayer::~CMCIStringCDPlayer(void)
    {
    }

    HRESULT CMCIStringCDPlayer::Initialize(void)
    {
        MCIERROR mciError;
        TCHAR mciReturnBuffer[512];

        mciError = mciSendString("status cdaudio media present", mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x determining CD media status\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        if (stricmp(mciReturnBuffer, "true") != 0)
        {
            printf("No media in CDRom drive\n");
            return E_FAIL;
        }
        return S_OK;
    }

    HRESULT CMCIStringCDPlayer::DumpTrackList()
    {
        MCIERROR mciError;
        TCHAR mciReturnBuffer[512];
        mciError = mciSendString("status cdaudio number of tracks", mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x determining CD media track count\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        sscanf(mciReturnBuffer, "%d", &_TrackCount);

        for (int i = 0 ; i < _TrackCount ; i += 1)
        {
            TCHAR mciCommandBuffer[512];
            DWORD trackHours, trackMinutes, trackSeconds;
            sprintf(mciCommandBuffer, "status cdaudio length track %d", i+1);
            mciError = mciSendString(mciCommandBuffer, mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
            if (mciError != 0)
            {
                printf("MCI Error %x determining track length\n",mciError);
                return HRESULT_FROM_WIN32(mciError);
            }
            printf("Track %d: Length %s\n", i, mciReturnBuffer);
        }
        return S_OK;
    }

    HRESULT CMCIStringCDPlayer::PlayTrack(int TrackNumber)
    {
        printf("Next article\n");
        return E_FAIL;
    }

    Following the pattern of the WMP articles, this one only dumps the contents of the CD tracks.  The first thing you'll notice is that the MCI string APIs use strings for their parameters (no duh, that's why they're called string APIs).  But the strings you specify are pretty verbose - for instance, to retrieve the number of tracks, you say "status cdaudio number of tracks".  The first word in the command string is the verb - MCI supports verbs like open , set, cut, paste, play, spin, etc.  You can even tell the MCI commands to open or close the CDRom door (with "set cdaudio door open").

    The other thing to notice is that audio tracks are )ORIGIN 1, not )ORIGIN 0.  The other thing to notice is that the length of a track is returned in m:s:f format, or minutes:seconds:frames.

    Tomorrow, I'll add the playback functions.

  • Larry Osterman's WebLog

    Playing Audio CDs, part 4 - MCI Playback.

    • 8 Comments
    The other day, I wrote about dumping the track database on an audio using the MCI string command set.

    Today, I'll include the piece I left out of the last article, actually playing a track from the CD.

    HRESULT CMCIStringCDPlayer::PlayTrack(int TrackNumber)
    {
        MCIERROR mciError;
        TCHAR mciReturnBuffer[512];
        TCHAR mciCommandBuffer[512];
        if (TrackNumber > _TrackCount)
        {
            printf("Track out of range\n");
            return E_FAIL;
        }
        // Set the MCI time format to track/minute/second/frame.
        mciError = mciSendString("open cdaudio", mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x opening CDRom\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        // Set the MCI time format to track/minute/second/frame.
        mciError = mciSendString("set cdaudio time format tmsf", mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x setting time format\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        sprintf(mciCommandBuffer, "play cdaudio from %d to %d", TrackNumber, TrackNumber+1);
        mciError = mciSendString(mciCommandBuffer, mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x playing audio\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }

        Sleep(_TrackTimes[TrackNumber]);

        // Close the CD 
        mciError = mciSendString("close cdaudio", mciReturnBuffer, sizeof(mciReturnBuffer), NULL);
        if (mciError != 0)
        {
            printf("MCI Error %x closing CDRom\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        return S_OK;
    }

    There are a couple of things to notice about the code.  The first thing that's done is to open the CDAudio device.  We do that to be able to set the audio time format to tmsf (track/minute/second/frame).  The default audio time format appears to be frame based, but it's not easy to tell from the documentation. 

    Once I set the cdrom track, it was trivial to have it play the selected track - you just say "play".

    This ease of use is why the MCICDA commands are so widely used - playing a track is literally as easy as hitting the buttons on the player.

    Next time, a different version of the MCI command set.

    Edit: Fixed typo in comment.

  • Larry Osterman's WebLog

    Playing Audio CDs, part 2

    • 7 Comments
    Yesterday, I looked a a simple application for dumping the track information on a CD.

    Since I'm going to be working on this concept (playing a CD) for a while, I figured I'd restructure the app to make it somewhat more generic.

    First off, lets define a pure virtual base class for the player...

    class CCDPlayer
    {
    public:
        virtual ~CCDPlayer(void) {};
        virtual HRESULT Initialize() = 0;
        virtual HRESULT DumpTrackList() = 0;
        virtual HRESULT PlayTrack(int TrackNumber) = 0;
    };
     

    It's a really trivial class, it defines an Initialization step, dumping the tracks and playing a track. 

    Now, lets take the core application and rewrite it using this base class.


    int _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
    {
        int nRetCode = 0;
        CWMPCDPlayer wmpCdPlayer;
        CCDPlayer *player = &wmpCdPlayer;
        HRESULT hr;

        // initialize MFC and print and error on failure
        if (!AfxWinInit(::GetModuleHandle(NULL), NULL, ::GetCommandLine(), 0))
        {
            // TODO: change error code to suit your needs
            _tprintf(_T("Fatal Error: MFC initialization failed\n"));
            nRetCode = 1;
        }
        else
        {
            hr = player->Initialize();
            if (hr != S_OK)
            {
                printf("Couldn't initialize CD player: %x\n", hr);
                return 1;
            }
            hr = player->DumpTrackList();
            if (hr != S_OK)
            {
                printf("Couldn't dump track database: %x\n", hr);
                return 1;
            }
            int trackNumber;
            printf("Pick a track: ");
            scanf("%d", &trackNumber);
            hr = player->PlayTrack(trackNumber);
        }

        return nRetCode;
    }

    Much better (simpler).  Now we can concentrate on the meat of the problem, the actual player implementation.

    Here's a complete implementation of the CWMPCDPlayer class:

    CWMPCDPlayer::CWMPCDPlayer(void)
    {
    }
    CWMPCDPlayer::~CWMPCDPlayer(void)
    {
        CoUninitialize();
    }
    HRESULT CWMPCDPlayer::Initialize()
    {
        HRESULT hr;
        hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
        if (hr != S_OK)
        {
            printf("Couldn't initialize COM: %x\n", hr);
            return hr;
        }
        hr = _CdRomCollection.CoCreateInstance(__uuidof(WMPLib::WindowsMediaPlayer));
        if (hr != S_OK)
        {
            printf("Couldn't instantiate CDRom player: %x\n", hr);
            return hr;
        }

        long driveCount;
        hr = _CdRomCollection->get_count(&driveCount);
        if (hr != S_OK)
        {
            printf("Couldn't get drive count: %x\n", hr);
            return hr;
        }
        if (driveCount == 0)
        {
            printf("Machine has no CDROM drives\n");
            return E_FAIL;
        }
        return S_OK;
    }
    HRESULT CWMPCDPlayer::DumpTrackList()
    {
        HRESULT hr;
        CComPtr<WMPLib::IWMPCdrom> cdromDrive;
        cdromDrive = _CdRomCollection->Item(0);

        hr = cdromDrive->get_Playlist(&_Playlist);
        if (hr != S_OK)
        {   
            printf("Couldn't get playlist for CDRom drive: %x\n", hr);
            return hr;
        }
        for (int i = 0 ; i < _Playlist->count ; i += 1)
        {
            CComPtr<WMPLib::IWMPMedia> item;
            hr = _Playlist->get_Item(i, &item);
            if (hr != S_OK)
            {
                printf("Couldn't get playlist item %d: %x\n", i, hr);
                return hr;
            }
            printf(_T("Track %d (%S)\n"), i, item->name.GetBSTR());
        }
        return S_OK;
    }

    HRESULT CWMPCDPlayer::PlayTrack(int TrackNumber)
    {
        HRESULT hr;
        CComPtr<WMPLib::IWMPPlayer> player;
        CComPtr<WMPLib::IWMPControls> controls;
        CComPtr<WMPLib::IWMPMedia> media;
        hr = player.CoCreateInstance(__uuidof(WMPLib::WindowsMediaPlayer));
        if (hr != S_OK)
        {
            printf(_T("Couldn't create player instance: %x\n"), hr);
            return hr;
        }
        hr = player->put_currentPlaylist(_Playlist);
        if (hr != S_OK)
        {
            printf(_T("Couldn't set player playlist: %x\n"), hr);
            return hr;
        }
        hr = player->get_controls(&controls);
        if (hr != S_OK)
        {
            printf(_T("Couldn't get player controls: %x\n"), hr);
            return hr;
        }

        hr = _Playlist ->get_Item(TrackNumber, &media);
        if (hr != S_OK)
        {
            printf(_T("Couldn't get playlist item: %x\n"), hr);
            return hr;
        }

        hr = controls->put_currentItem(media);
        if (hr != S_OK)
        {
            printf(_T("Couldn't set player control item: %x\n"), hr);
            return hr;
        }
        hr = controls->play();
        if (hr != S_OK)
        {
            printf(_T("Couldn't get playlist item: %x\n"), hr);
            return hr;
        }
        double duration;
        hr = media->get_duration(&duration);
        if (hr != S_OK)
        {
            printf(_T("Couldn't get track duration: %x\n"), hr);
            return hr;
        }
        //
        // Wait for the track to stop playing.
        //
        MSG msg;
        DWORD tickCountStart = ::GetTickCount();
        while (GetMessage(&msg, NULL, 0, 0) && GetTickCount() < tickCountStart+(duration*1000))
        {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
        return S_OK;
    }

    That's it! I left out the WMP Cdplayer class definition, it consists of a derivation of the base class, and the addition of two member variables:

        CComPtr<WMPLib::IWMPCdromCollection> _CdRomCollection;
        CComPtr<WMPLib::IWMPPlaylist> _Playlist;

    These exist to carry information across the various abstract methods.

    Most of the code above is a repackaging of the code from yesterdays example, the new bits are the PlayTrack method.  There looks like a lot of code in the playback code, but it's actually pretty straightforward.  You allocate a player object to actually perform the playback, and set the current playlist on the player.  You get access to the controls of the player (that's the controls object), tell the control what track to play, and then ask the control to start playing.

    I then wait until the track completes before returning to the caller.  Please note that the wait loop pumps windows messages.  This is because we're an STA application and when you're in an STA application, you need to pump messages instead of blocking, because STA COM objects use windows messages to communicate (and the WMP class is an STA object).

    Tomorrow, I'll talk about the next method for doing CD playback, the MCI command set.

  • Larry Osterman's WebLog

    What's actually on an audio CD?

    • 6 Comments
    Before I can talk about reading audio CDs using DAE (Digital Audio Extraction), I need to talk a bit about what data's actually on the audio CD, since we're going to be reading the raw audio data from the CD.

    An audio CD contains audio samples sampled at 44.1kHz, stereo.   Each sample is 16 bits of data long.  So one second of data takes up 176,400 bytes.  This will become important during playback, since we'll need to tell the audio adapter what its rendering.  There's a lot of math and physics that went into figuring out those numbers, essentially, 44.1kHz/16bits is "good enough" to be considered full fidelity to most people. 

    The data on the audio CD is laid out in "sectors", each of which contains 1/75th of a second worth of data.   So each sector is 2352 bytes in length.

    That number will be critical to the reading process later, since we'll be reading the data from the CD one sector at a time.

    Note that the sector size for a CDROM drive is 2048 bytes - that's used to make life easier for applications.

    When the CDROM filesystem sees an audio CD, it reads the track database off the CD (I'll talk about that when I'm showing the code that reads the database) and synthesizes a tiny RIFF file for CD audio.  The file contains enough information for Windows to hand to the media player to have it play the media back.

     

    How Stuff Works has a pretty good tutorial on how the actual data is laid out on the CD here.

    Oh, and they have a much better series of examples of how digital audio is sampled here.

    Prof. Kelin Kuhn has an awesome breakdown of the details in the course materials for UW's EE498 course

  • Larry Osterman's WebLog

    What's wrong with this code, part 11 - the answers

    • 5 Comments
    Well, clearly the shorter the example, the quicker people pick up the problem.

    The problem was:

        if (!CreateProcessW(NULL, PROCESS_NAME, NULL, NULL, FALSE, 0, NULL, NULL, &startupInfo, &processInformation))

    The issue here is that you can't pass a constant string to CreateProcessW.  The documentation for CreateProcessW states:

    The Unicode version of this function, CreateProcessW, will fail if this parameter is a const string.

    But it doesn't explain why it fails.  Skywing pointed out what's going on - the developer who wrote CreateProcessW temporarily modified the input string to separate the executable name from the arguments.  There are two issues here.

    The first is the obvious one: You can't use constant strings.  And there's a second one: You can't call CreateProcess with the same buffer on multiple threads.  While the call to CreateProcess restores the contents of the lpCommandLine string before it returns, there is still a period of time when the contents of lpCommandLine are not what was passed in originally.

    So if you have two threads calling CreateFile with the same buffer, some of the calls to CreateProcess will have no command line arguments, and some will have the expected command line arguments (Hmmm.  Maybe I should have used this as the example instead of the one I did - to show a concurrency issue in the base API set.  Oh well...).

    As I said, I got burned by this one, the reason for this is that the current compiler puts constant strings in a read-only data section.  And thus the PROCESS_NAME variable is read-only, and it causes the crash.

    During the comments, I asked "But why did I say that it was possible that you won't see the bug?".  Well, it's because the bug depends on the compiler switches.  The default switches exhibit the bug.  But if you were using an older compiler (circa vc6, if I recall correctly) or if your compiler switches were set differently, the behavior would be different.

    The relevent switch is the /Gf switch.  The /GF switch enables a compiler feature called string pooling - essentially constant strings are collapsed together if they're the same string.  If your code was compiled with the /Gf switch (enable string pooling as read/write strings), then the constant strings used for PROCESS_NAME would not be kept in a read-only section.

    The reason that you see this the bug with the default compilation options is that the /ZI command line switch (which enables symbolic debugging in a PDB file) forces the /GF switch to be enabled.

    The history of string pooling in NT is long and varied.  When NT 3.1 was under development, the person responsible for the builds changed the state of the /Gf flag several times during the lifetime of the project - there was code in the system that had depended on constant strings being read/write, and other code that depended on the lack of string pooling (you'd have two strings with the same text and modify one of the two strings).  I believe that NT 3.1 eventually shipped without the string pooling, and that it was added after we shipped (probably in the NT 3.51 timeframe, when we were concentrating on performance).

     

    Kudos:

    ParadoxLost nailed the base problem in the very first comment.  Mike Dunn nailed the constant string issue, and Purplet nailed the /Gf /GS switch issue.

    Mea culpas:

    Joel Spolsky and eruprahgp pointed out the possibility of a handle leak with the handles returned in the processInformation structure.  That's a fair criticism, but since the process exits immediately afterwards, the handles will be closed.  But in production systems this is a real issue.

    gregg, Bradly Grainger, and Universalis pointed out that status is only assigned if CreateProcess fails -it's uninitialized otherwise.

    Setfan Kuhr totally nailed a problem I'd not seen: The startupinfo doesn't have the wShowWindow and dwFlags fields - so notepad might not be visible when it's launched (I don't know if this is a problem or not - the process I was launching was a console process so it was irrelevant).

    Major kudos to brantgurga for thinking to run this silly little program under appverifier.  That pointed out a couple of other issues: The code shouldn't have lpApplicationName as null and lpCommandLine as non null because of the spaces in file names issues.

    And Adrian totally blew me away when he pointed out that on versions of NT before NT4 SP5, the GetFileAttributes API did not have a stable return value.  Wow, that's cool, I'd never known that.

  • Larry Osterman's WebLog

    Sharron utterly ROCKs

    • 5 Comments
    The family (Mom, Dad, Daniel, Sharron and Grandma) spent this weekend up at the Whidbey Equestrian Center for the WEC's April Fools Dressage Show.  Sharron (and, of course Oliver the wonder pony) was riding the USDF Training Level 1 and 2 tests (twice - once each on Saturday and once each on Sunday).  Accompanying us was Sharron's instructor, Pam Pentz with her horses Whidbey and her stallion Rodioso.

    Since this was Sharron's first ever "big" competition (i.e. a real competition where she'd be judged according to strict national standards, not a training show (where the standards are somewhat looser)), we were all a smidge nervous.  It was also Sharron and Oliver's first two day show, which added a bunch to the stress - instead of just having to be on for one day, they'd have to be able to hold it together for two.

    Valorie barely slept on Friday night - a storm moved into Whidbey that evening and she knew that Oliver was going batty in his stall (Oliver really doesn't like rain on the stall roof).

    Saturday started off in typical northwest fashion - it was still pouring rain.  And Valorie was right - Oliver had tried to dig a hole out of the bottom of his stall (fortunately he failed).  And as in all things, the rain didn't stop the competition, Sharron's first ride was at 8:50AM, in the downpour - we were all pretty wet, and Oliver and Sharron were utterly drenched (they didn't have the benefit of rain gear).  But she looked pretty darned good in the ring, and her scores were totally respectable - she got a 59% on her first test (good enough for a blue ribbon).

    The next rider to go was Pam on Rodioso, performing the level 4-1 test, she got 64%, which was top score in her class.  Rodioso is an utterly amazing horse, personally, I think he might even be Olympic quality.

    Sharron's second test was even better than her first test (it had stopped raining and that helped), she got a 61% on that test (which was good for a 3rd place ribbon) .  She and Oliver looked absolutely amazing together.  She was 1 point away from the 2nd place ribbon on that test.

    Pam next rode on Whidbey, the FEI 6-year old test, a grueling test that's only given to 6 year olds, and as usual, Whidbey shone.  She later rode Whidbey in the USDF 3-3 test, and again he was great (two blue ribbons).

    On Sunday, the weather was far better, the rain had blown away and the sun was shining (it was still bitterly cold but better).

    Sharron rode at 8:09 AM (moved up from 7:30 originally), and she and Oliver were awesome, they got a 59% even though another horse caused Oliver to spook when he was entering the canter (the judge called it an "abrupt entry" :)).  If he hadn't spooked, she'd have gotten another 60%+.  That performance got her a 2nd place ribbon.

    At 10:10 or so, Sharron rode again, this time she had a somewhat sloppier performance (she and Oliver were really tired at this point) and she "only" got a 55%, but that was good enough for another blue ribbon.  We bummed around in the afternoon waiting for Pam's last ride, and we packed up the horses (ok, Pam packed up the horses - we just helped) and went home.

    We all celebrated at the end of the day at Claim Jumper, after dinner we staggered home and collapsed after a really long weekend.

    The total haul from the weekend: Two Blue ribbons, a yellow ribbon and a red ribbon.  All in all, a wonderful experience for all concerned.  Sharron really did rock - she and Oliver worked their tails off and it shows.

    The pictures are still on my computer at home, I'll try to get some up in the near future.

    Edit: ROCK->ROCKs

  • Larry Osterman's WebLog

    It's the hardware, stupid!

    • 5 Comments

    Riffing on Raymond, once again :)

    Raymond's post today reminded me of an email message sent out (company wide) by one of the very senior developers on the Windows 1.0 team about 6 months before they shipped.

    In his email, the developer announced that he wasn't going to be looking into any more crashes in Windows because they were all caused by bad hardware.

    It turns out that he actually almost had a point.  This particular email went out after he'd spent several weeks debugging a series of problems on a number of people's machines (dozens).  In every case, the problem was caused by bad 3rd party RAM.

    You see, back in those days, machines typically came with 512K of RAM, to get the extra 128K of RAM to get to 640K (or more), people bought 3rd party memory extenders.  And often, those 3rd party memory extenders used sub-standard memory chips (either chips without parity, or chips that were just flat-out bad).  And this developer happened to run across a large number of busted cards.

    So sometimes, it IS the hardware.

     

  • Larry Osterman's WebLog

    Alignment (part 3): Why bother?

    • 4 Comments
    I mentioned in the last post that the C standard is agnostic w.r.t. alignment.

    So why on earth did Microsoft chose the default alignment to be naturally aligned?

    After all, the compiler would be completely within the standard if it chose to pack its data tightly.  And, as I said, on MS-DOS platforms, memory size was everything.

    Well, I lied :). It turns out that on all x86 architecture based platforms, memory size isn't everything.  Memory is critically important, but so is memory alignment.

    You can see this really clearly if you hook an 8088, 80186, 80286, or 80386 machine to an In Circuit Emulator (ICE) (the behavior's not as obvious with newer processors because of the L1 cache).  For those that have never had the opportunity to use an ICE, an ICE is essentially a processor add-on that lets you analyze what's happening on the processor at the external pin level.  For the 8088-80286, it required a special version of the processor (called a bond-out chip) that added some additional trace information on the external bus.  An ICE would let you look at all the memory accesses done by the machine, and would even allow you to look back in time - if your system would crash mysteriously (and on the 8088, where there was no illegal instruction trap, this happened often), you could look at the history of instructions fetched and rendered and see what happened.

    For Intel machines before the 486, there was a really simple rule for performance: If your code or data was small, then your program ran faster.  This is because the processor didn't have a particularly effective cache and memory was significantly slower than the processor (this is still the case).  So the single thing you need to ensure is that you don't have to go to memory too often.

    If you look at an instruction like:

        mov ax, ds:[20] ; Load ax with the value at 20 bytes into the DS segment

    Under an ICE, you'll see something like (assuming that DS points to 0x300):

        mov ax, ds:[20] ; Load ax with the value at 20 bytes into the DS segment
                    << FETCH: WORD @003020, Value 0x3525

    But if you look at a fetch from 21, you see:

        mov ax, ds:[21] ; Load ax with the value at 21 bytes into the DS segment
                    << FETCH: BYTE @003021, Value 25
                    << FETCH: BYTE @003022, Value 35

    So an unaligned memory access is broken up by the processor and turned into two aligned fetches.  This is because the memory bus can't perform unaligned word accesses to memory.  On some processors (like most RISC machines), they don't even bother to break the memory access up, they just crash (generate an exception which is usually handled by the operating system, taking tens of thousands of instructions to execute).

    So unaligned access to memory is a huge source of performance problems.  One solution to this was the __unaligned attribute on data - it acts as a flag to the compiler that instructs it that the data referenced by this pointer might be unaligned, and thus it should generate code to access the data one byte at a time - even though that involves many fetches of memory, it's STILL faster than handling the exception that might be generated.

    But even with the unaligned attribute, it's important to realize that there are concurrency issues associated with unaligned memory access.  In general, aligned reads and writes are atomic - if you write a value of 0x12345678 and then write 0x87654321 to an aligned block of memory, when you read it back, you'll get either 0x12345678 or 0x87654321.  On the other hand, if the memory is unaligned, you might get 0x12344321 when you read it back.  Now, given that modern CPUs can (and do) reorder writes, you'd properly need to put a memory barrier between the two writes, and I believe that the memory barrier would ensure that this wouldn't happen, but in the absense of a memory barrier, this is a very real problem (I've seen it happen in production code).

    There's a great write-up on alignment on MSDN that I found here.

    For more current processor architectures, the alignment issues aren't as significant - the L2 cache essentially renders many (most) of the issues moot, because it only accesses main memory a cache line at a time (32bytes).  If the unaligned access crosses a cache line boundary however, all bets are off - your memory access will be indescribably slow (and historically, unaligned data access across cache line (and page) boundaries has been a source of many x86 CPU bugs (yes, CPUs have bugs too)).

    Edit: Added comment about breaking atomicity, thanks Dmitry.

  • Larry Osterman's WebLog

    RIP: Andre Norton

    • 2 Comments

    Over the weekend, I got my most recent copy of Locus (the only magazine I read cover-to-cover :)).  In it, the lede was that Andre Norton died on the 17th of March, 2005.

    I'm not really surprised, she'd been in poor health for quite some time, but...

    Andre wrote the very first novel I ever checked out of a library (possibly the first novel I ever read), when I was 6 years old.  I went to the Briarcliff Manor public library (inside the Briarcliff Manor train station - it's still there) and checked out my first book with my sparkling new library card.

    It was Star Rangers (1953), and the adventures of Kartr and his band of rangers utterly enthralled me.

    I was hooked, utterly and irrevocably.

    I have many of her books in the library at home, some of which are worn to the point that they're almost unreadable.  Its been so wonderful seeing Baen and other publishers re-issue her stories, she was truly one of the great ones.

    Farewell Alice. 

  • Larry Osterman's WebLog

    Purdue's entry in the Rube Goldberg contest

    • 2 Comments

    This year's Rube Goldberg contest requires that the team build a device to remove the old batteries and insert new batteries in 20 or more steps.

    The winning team, from Purdue took 125 steps.

    A Quicktime video of their machine can be found here.  The /. article for this didn't have the full movie, but Valorie forwarded it from the Destination Imagination mailing list.

    Edit: Fixed spelling

  • Larry Osterman's WebLog

    Playing audio CDs, part 6 - MCI Command playback

    • 2 Comments
    Today, we play the actual audio track on the CD using the MCI commands.

    HRESULT CMCICommandCDPlayer::PlayTrack(int TrackNumber)
    {
        MCIERROR mciError;
        MCI_SET_PARMS setParms = {0};
        MCI_PLAY_PARMS playParms = {0};
        //
        // The track number from the host app is )ORIGIN 0, MCI is )ORIGIN 1, so adjust it.
        //
        TrackNumber += 1;
        if (TrackNumber > (int)_TrackCount)
        {
            printf("Track out of range\n");
            return E_FAIL;
        }

        // Set the MCI time format to track/minute/second/frame.
        setParms.dwTimeFormat = MCI_FORMAT_TMSF;
        mciError = mciSendCommand(_MciHandle, MCI_SET, MCI_SET_TIME_FORMAT, (DWORD_PTR)&setParms);
        if (mciError != 0)
        {
            printf("MCI Error %x determining CD media status\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }

        playParms.dwFrom = MCI_MAKE_TMSF(TrackNumber, 0, 0, 0);
        if (TrackNumber < _TrackCount)
        {
            playParms.dwTo = MCI_MAKE_TMSF(TrackNumber+1, 0, 0, 0);
        }
        mciError = mciSendCommand(_MciHandle, MCI_PLAY, MCI_FROM | (TrackNumber <_TrackCount ? MCI_TO : 0) | MCI_WAIT, (DWORD_PTR)&playParms);
        if (mciError != 0)
        {
            printf("MCI Error %x playing media\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }

        return S_OK;
    }

    Not much different from the previous versions, in fact it's a pretty straightforward transcription of the previous programs.  One minor change is that I'm using the MCI_WAIT flag to ask the MCI commands

    That allows me to avoid the Sleep() at the end.  I could have done the same thing in the string based version by appending the " wait" command to the play command string.

    I'm going to skip a couple of days on this series, it'll take a while to get the Digital Audio Extraction version of this problem ready for publication, DAE's sufficiently compilicated that it'll take several articles to fully describe what I'm doing.

  • Larry Osterman's WebLog

    Playing a CD, part 5 - MCI commands.

    • 1 Comments
    I'm almost done with the easy part of the "playing a CD" series.  Today, we'll look at the other half of the MCI APIs, the MCI command set.

    HRESULT CMCICommandCDPlayer::Initialize(void)
    {
        MCIERROR mciError;
        MCI_STATUS_PARMS statusParms = {0};
        MCI_OPEN_PARMS openParms = {0};

        openParms.lpstrDeviceType = "cdaudio";
        mciError = mciSendCommand(NULL, MCI_OPEN, MCI_OPEN_TYPE, (DWORD_PTR)&openParms);
        if (mciError != 0)
        {
            printf("MCI Error %x determining CD media status\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        _MciHandle = openParms.wDeviceID;

        statusParms.dwItem = MCI_STATUS_MEDIA_PRESENT;
        mciError = mciSendCommand(_MciHandle, MCI_STATUS, MCI_STATUS_ITEM, (DWORD_PTR) &statusParms);
        if (mciError != 0)
        {
            printf("MCI Error %x determining CD media status\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        if (statusParms.dwReturn == 0)
        {
            printf("No media in CDRom drive\n");
            return E_FAIL;
        }
        return S_OK;
    }

    HRESULT CMCICommandCDPlayer::DumpTrackList()
    {
        MCIERROR mciError;
        MCI_STATUS_PARMS statusParms = {0};

        statusParms.dwItem = MCI_STATUS_NUMBER_OF_TRACKS;
        mciError = mciSendCommand(_MciHandle, MCI_STATUS, MCI_STATUS_ITEM, (DWORD_PTR)&statusParms);
        if (mciError != 0)
        {
            printf("MCI Error %x determining number of tracks\n",mciError);
            return HRESULT_FROM_WIN32(mciError);
        }
        if (statusParms.dwReturn == 0)
        {
            printf("No media in CDRom drive\n");
            return E_FAIL;
        }
        _TrackCount = statusParms.dwReturn;

        for (DWORD_PTR i = 0 ; i < _TrackCount ; i += 1)
        {
            statusParms.dwTrack = (DWORD)i+1;
            statusParms.dwItem = MCI_STATUS_LENGTH;
            mciError = mciSendCommand(_MciHandle, MCI_STATUS, MCI_STATUS_ITEM | MCI_TRACK, (DWORD_PTR)&statusParms);
            if (mciError != 0)
            {
                printf("MCI Error %x determining track length\n",mciError);
                return HRESULT_FROM_WIN32(mciError);
            }
            DWORD trackLength = (DWORD)statusParms.dwReturn;
            printf("Track %d: Length %02d:%02d:%02d\n", i, MCI_MSF_MINUTE(trackLength), MCI_MSF_SECOND(trackLength), MCI_MSF_FRAME(trackLength));
            _TrackLengths.Add(trackLength);
        }
        return S_OK;
    }

    Pretty simple, and not surprisingly, the code bears a close resemblance to the command based code. 

    Things to note: When using the MCI command functions you need to open the device ahead of time.  The string functions will happily redirect to the appropriate device when parsing, but the command functions aren't as clever.

    The other rather unique thing is the behavior of the fdwCommand parameter to mciSendCommand.  The fdwCommand parameter basically describes what kind of structure lives in the dwParam parameter to mciSendCommand.  But not always.  As you can see from the call to determine the length of the track, it also is used to describe the semantics of the fields within the track. 

    Tomorrow, playing the CD using the CD commands.  Then we get to the hard part, playing CDs using digital audio extraction (AKA going straight to the metal).

Page 1 of 2 (26 items) 12