Larry Osterman's WebLog : Audio

Elliot Omiya, Frank Yerrace and I make a video…

LarryOsterman — Wed, 12 Aug 2009 00:39:12 GMT

Charles just let me know that he’s posted a video that Elliot, Frank and I did talking about the audio features added to Win7 and some of the architectural decisions that went into it.

Enjoy!

A quick run through the new Windows 7 multimedia audio SDK samples

LarryOsterman — Fri, 07 Aug 2009 22:17:27 GMT

As I mentioned yesterday, the Windows SDK is now live. For the Windows SDK, there are 9 new samples (and one changed samples).

Two of the SDK samples demonstrate the WIndows 7 “Ducking” feature – they’re actually based on the code I wrote for my PDC talk last year, but tweaked to show some new scenarios and clean up the code (the original PDC code was not ready for prime time, it was strictly demo-ware).

The DuckingMediaPlayer application shows both a really simple DirectShow based media player but also shows how to integrate the players volume control with the Windows per-application volume mixer. So if you’re wondering how to integrate the volume control for your application with the built-in per-application volume controls, this sample will show you how to do it.
The DuckingCaptureSample application is a trivial application that uses the wave APIs to capture audio data (and throw it away). This shows how to trigger the “ducking” feature in WIndows 7.

The other 7 samples reproduce functionality in the old WinAudio SDK sample – instead of providing a single monolithic audio sample, we crafted different samples each of which shows one aspect of audio rendering. All of these samples are simple console applications which read their parameters from the command line. They’re intentionally extremely simple to reduce the potential for confusion in the samples.

The EndpointVolume sample shows how to use the IAudioEndpointVolume APIs. It demonstrates using the VolumeStepUp, VolumeStepDown, SetMute and SetMasterVolumeLevelScalar APIs.
The CaptureSharedEventDriven sample shows how to use WASAPI in “event driven” mode to capture data in shared mode from the microphone.
The CaptureSharedTimerDriven sample shows how to use WASAPI in “timer driven” mode to capture data in shared mode from the microphone.
The RenderSharedEventDriven sample shows how to use WASAPI in “event driven” mode to play a simple sine wave in shared mode from the speakers.
The RenderSharedTimerDriven sample shows how to use WASAPI in “timer driven” mode to play a simple sine wave in shared mode from the speakers.
The RenderExclusiveEventDriven sample shows how to use WASAPI in “event driven” mode to play a simple sine wave in exclusive mode from the speakers. This also shows the “exclusive mode swizzle” that is required to align the buffer size with the hardware buffer size (which is required to work with many HDAudio solutions).
The RenderExclusiveTimerDriven sample shows how to use WASAPI in “timer driven” mode to play a simple sine wave in exclusive mode from the speakers. This also shows the “exclusive mode swizzle” that is required to align the buffer size with the hardware buffer size (which is required to work with many HDAudio solutions).

The reason we don’t have capture exclusive samples is that we felt that users of the SDK could derive the exclusive capture samples from the render samples if it was important to them.

All the shared mode samples also show how to implement stream switching.

One of the things I’m quite proud about the samples is their structure. Each sample has the following basic file layout:

 Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0\Samples\multimedia\audio\RenderExclusiveEventDriven

08/07/2009  10:08 AM    <DIR>          .
08/07/2009  10:08 AM    <DIR>          ..
07/14/2009  06:54 PM             7,079 CmdLine.cpp
07/14/2009  06:54 PM               760 CmdLine.h
07/14/2009  06:54 PM             2,084 ReadMe.txt
07/14/2009  06:54 PM               533 stdafx.cpp
07/14/2009  06:54 PM               935 stdafx.h
07/14/2009  06:54 PM             1,067 targetver.h
07/14/2009  06:54 PM             1,925 ToneGen.h
07/14/2009  06:54 PM            18,376 WASAPIRenderer.cpp
07/14/2009  06:54 PM             2,560 WASAPIRenderer.h
07/14/2009  06:54 PM            14,754 WASAPIRenderExclusiveEventDriven.cpp
07/14/2009  06:54 PM             1,283 WASAPIRenderExclusiveEventDriven.sln
07/14/2009  06:54 PM             8,403 WASAPIRenderExclusiveEventDriven.vcproj
              12 File(s)         59,759 bytes
               2 Dir(s)  62,822,105,088 bytes free

Each sample has the same set of common files:

CmdLine.cpp/CmdLine.h: A very simple command line parsing function
stdafx.cpp/stdafx.h: Common header definitions
targetver.h: Defines the target platform for these samples (Windows Vista in this case).
ToneGen.h: A very simple sine wave generating function (not present for the capture samples).
WASAPIRenderer.cpp/WASAPIRenderer.h (WASAPICapture for capture samples): The code which does all the WASAPI rendering and/or capturing
<SampleName>: Scaffolding for the sample – this defines the command line parameters and instantiates the actual render/capture object and asks it to do the rendering/capturing.

Each of these samples is essentially identical, in fact they’re sufficiently similar that you can use your favorite file comparison tool to see what code has to change to go from one mode to another. So to see what changes are required to go from exclusive timer driven to exclusive event driven, you can windiff the RenderExclusiveEventDriven\WASAPIRenderer.cpp and RenderExclusiveTimerDriven\WASAPIRenderer.cpp files and see what changes are required to implement the different model.

The Windows 7 SDK is live!

LarryOsterman — Fri, 07 Aug 2009 02:56:05 GMT

I just received email that the new Windows 7 SDK is now live! Apparently it’s not on the Windows SDK download site yet, but if you click on the first link, you can download an ISO which contains the SDK.

For the Win7 SDK, I wrote about 8 new samples in the Multimedia\Audio category, they’re described here (check the Multimedia\Audio samples. These samples replace the old WinAudio SDK sample – basically I took the old WinAudio sample and blew it up into 8 different samples which demonstrate different aspects of rendering audio using WASAPI.

I’ll write some more about the samples tomorrow – there are some cool things in the way the samples are structured that I hope will help developers who want to understand the difference between the various WASAPI render and capture modes.

A few of my favorite Win7 Sound features – Stream Switching

LarryOsterman — Thu, 06 Aug 2009 00:45:57 GMT

Way back when I was in college, I learned Lisp using a derivative of Lisp called MACLISP (named for MIT’s project MAC, not for anything that came from a fruity company in Cupertino). One of the coolest features that MACLISP offered was the (DWIM) command – basically if you had a typo when entering an s-list into MACLISP, you could type (DWIM) and the MACLISP interpreter would fix your code for you (and yeah, it usually got it somewhat wrong :)).

Stream Switching is a DWIM feature in the audio stack. If an application is rendering to a default audio device and the device is removed, the audio stream will automatically switch to the new device. The same happens if the default device changes for other reasons (if the user changes the default device for example) or if the sample rate on the device changes (this can happen with certain types of audio hardware allow external controls to change the sample rate for the device).

We were able to figure out how to implement the stream switching logic in a fashion that causes it to work without requiring changes from 3rd party applications, which is really cool because it allows us to enable new scenarios without breaking appcompat – as long as the application is rendering to a default endpoint, we’ll stream switch the application without the app being notified.

If an application is rendering to a specific endpoint, we’re not going to stream switch when the endpoint is removed – we don’t know the reasons for the application choosing that particular endpoint so we don’t attempt to second guess the applications intent (maybe the user has asked that their music player only play through the headphones and not through the speakers because playing through the speakers would disturb the baby).

We also don’t support stream switching if the application is using WASAPI (the low level audio rendering APIs) to render audio. That’s for a number of reasons, but mostly it’s because the presumption is that any application that is using WASAPI is using a low level rendering API and thus doesn’t want this kind of behavior.

The stream switching logic is really cool in action, especially if you’ve got a machine which supports dynamic jack detection – when you’re watching a DVD in windows media player and you plug in a pair of headphones, poof – the audio gets redirected to the headphones just like you’d expect it to.

A few of my favorite Win7 Sound features – Capture Monitor (aka Listen to…)

LarryOsterman — Tue, 04 Aug 2009 20:30:21 GMT

The capture monitor is a feature that allows you to listen to a portable media player (or any other microphone input) through your PC speakers.

First a bit of history. Way back in the dark ages (Windows XP timeframe), audio solution manufacturers used to include an analog circuitry that connected the line in to the speaker jack on the PC. They routed this through an analog volume control and exposed this through the audio device topology. People used this functionality to connect their portable media players to their PCs. While this feature was popular with customers, the cost of providing the circuitry was too much for some IHVs and they started removing this functionality from their products starting some time before Vista shipped.

Not surprisingly, customers complained about this and we decided to implement equivalent functionality in the audio subsystem in Windows. Because we’re doing this in the audio subsystem instead of in hardware, it allows you to configure the capture to run between any two devices.

To configure the capture monitor, you go to the properties page for the input device and select the “Listen” tab:

Once you select the “Listen to this device” checkbox (and hit apply), the system will start capturing data from the input and rendering it to the output device. I’ve been using it to listen to my media player for months :).

I’ve been really gratified to see that both Ed Bott and Lifehacker (who picked this up from Ed) have noticed this feature, it was a huge amount of fun to write. It was also my first experience using TDD (or rather a variant of TDD – instead of writing the test first then the code, I wrote the code and the test for the code at the exact same time) - based on my experiences, I’m totally sold on it as a development paradigm.

A few of my favorite Win7 Sound features – UI refinements

LarryOsterman — Tue, 04 Aug 2009 02:08:22 GMT

Well, we shipped Windows 7, and now I’d like to talk about a few of my favorite features that were added by the Sound team. Most of them fit in the “make it just work the way it’s supposed to”, but a few are just cool.

I also want to call out some stuff that people probably are going to miss in the various Windows 7 reviews.

One of the areas I want to call out is the volume UI. There’s actually been a ton of work done on the volume UI in Windows 7, although most of it exists under the covers. For instance, the simple volume control (the one you get to with a single click from the volume notification area) uses what we call “flat buttons”.

Windows 7 Simple Volume UI:	Windows Vista Simple Volume UI:

Both the mute control and the device button are “flat buttons” – when you mouse over the buttons, the button surfaces:

By using the “flat buttons”, the UI continues to have the old functionality, but it visually appears cleaner. There have been a number of other changes to the simple volume UI. First off, we will now show more than one slider if you have more than one audio solution on your machine and you’re using both of them at the same time. This behavior is controlled by the new volume control options dialog:

As I mentioned above, the device icon is also a “flat button” – this enables one click access to the hardware properties for you audio solution.

The volume mixer has also changed slightly. You’ll notice the flat buttons for the device and mute immediately. We also added a flat button for the System Sounds which launches the system sounds applet.

Another subtle change to the volume mixer is that there are now meters for individual applications as well as for the master volume:

And finally, the volume mixer no longer flickers when resizing (yay!). Fixing the flicker was a problem that took a ton of effort (and I needed to ask the User team for help figuring out the problem) – the solution turned out to be simple but it took some serious digging to figure it out.

Windows 7 fixes the PlaySound(XXX, SND_MEMORY|SND_ASYNC) anti-pattern

LarryOsterman — Thu, 25 Jun 2009 00:54:11 GMT

A number of times in the past, I’ve mentioned that the PlaySound(xxx, xxx, SND_MEMORY|SND_ASYNC) pattern is almost always a bad idea. After the last wave of crash dumps were received for this problem, our team decided to do something about it. Starting with Windows 7, if you call PlaySound with SND_MEMORY|SND_ASYNC, instead of relying on the memory passed in by the application, we allocate our own buffer for the sound file on the heap and copy the file into that buffer. We’ll only do it for WAV files that are smaller than 2M in size, and if the allocation of the buffer fails, we fall back on the original code path, but it should dramatically reduce the number of apps that crash while using this pattern.

It’s a little thing, but it should make life much easier for those applications.

PlaySound(xxx, SND_MEMORY | SND_ASYNC) is almost always a bad idea.

LarryOsterman — Fri, 20 Feb 2009 01:31:57 GMT

Whenever you submit a crash report to OCA, a bug gets filed in the relevant product database and gets automatically assigned to the developer responsible for the code. I had a crashing bug in the PlaySound API assigned to me.

In this case, the call was crashing deep inside of the waveOutOpen API and it was crashing because the input WAVEFORMATEX structure was bogus. The strange thing is that the PlaySound API does some fairly thorough validation of the input WAVEFORMATEX read from the .WAV file and that validation had to have passed to get to the call to waveOutOpen.

I looked a bit deeper and came to the realization that every single one of the crashes (in maybe a dozen different applications) had specified SND_MEMORY | SND_ASYNC in their call to PlaySound.

I’ve talked about that particular combination before in my blog, but I wanted to call it out in a top level post in the hopes that people will stop making this common mistake.

When you call PlaySound with the SND_MEMORY flag, it tells the PlaySound API that instead of reading the audio data from a file, you’re passing in a pointer to memory which holds the wave contents for you. That’s not controversial, and can be quite handy if (for instance) you want to build a .WAV file in memory instead of calling the wave APIs directly.

When you call PlaySound with the SND_ASYNC flag, that flag tells PlaySound that instead of blocking until the sound has finished playing, the API should return immediately instead of blocking while the sound is played.

Neither of these flags is controversial and neither of them is particularly dangerous until you combine the two together.

The problem is that there’s really no way of knowing when the sound has finished playing and thus when the application frees the memory, it’s entirely possible that the PlaySound API is still using it. That means that if you ever call PlaySound with both of these flags, you stand a very high chance of crashing due to the combination of these behaviors.

The unfortunate thing is that this behavior has existed since the SND_MEMORY flag was added back in Windows 3.1. The only safe way of dealing with this that works on all current Windows operating systems is to call PlaySound(NULL, 0, 0) before freeing the memory – the call to PlaySound(NULL, 0, 0) will block until the currently playing sound has completed playing (or abort the playsound if it hasn’t started yet).

When you do UX work, sometimes you have to worry about the strangest things…

LarryOsterman — Tue, 13 Jan 2009 20:13:40 GMT

I recently got a bug reported to me about the visuals in the sound control panel applet not being aligned properly (this is from the UI for a new Windows 7 feature):

The problem as reported was that the microphone was aligned incorrectly w.r.t. the down arrow. – the microphone was too far to the right.

But if you look carefully, you’ll see that that isn’t the case – drawing a box around the controls makes it clearer:

Nitpickers Corner: For those of you that love to count pixels, it’s entirely possible that the arrow might be off by a couple of pixels but fixing it wouldn’t fix the problem, because then the arrow would be off-center with respect to the speakers. The real problem is that the microphone icon is visually weighted to the right – the actual icon resource was lined up with the arrow, but because the visual weight was to the right, it displayed poorly.

It turns out that there’s really no good way of fixing this – if we were to adjust the location of the icons, it wouldn’t help, because a different device would have a different visual center (as the speaker icon does)…

Instead, we looked at the visuals and realized that there was an alternative solution: Adjust the layout for the dialog and the problem more-or-less goes away:

The problem still exists at some level because the arrow is centered with the icons but some icons (like the stalk microphone above) are bottom heavy. But for whatever reason, the visuals aren’t as disconcerting when laid out horizontally.

As I said in the title – sometimes you need to worry about the strangest things.

Why do people think that a server SKU works well as a general purpose operating system?

LarryOsterman — Thu, 08 Jan 2009 21:17:04 GMT

Sometimes the expectations of our customers mystify me.

One of the senior developers at Microsoft recently complained that the audio quality on his machine (running Windows Server 2008) was poor.

To me, it’s not surprising. Server SKUs are tuned for high performance in server scenarios, they’re not configured for desktop scenarios. That’s the entire POINT of having a server SKU – one of the major differences between server SKUs and client SKUs is that the client SKUs are tuned to balance the OS in favor of foreground responsiveness and the server SKUs are tuned in favor of background responsiveness (after all, its a server, there’s usually nobody sitting at the console, so there’s no point in optimizing for the console).

In this particular case, the documentation for the MMCSS service describes a large part of the root cause for the problem: The MMCSS service (which is the service that provides glitch resilient services for Windows multimedia applications) is essentially disabled on server SKUs. It’s just one of probably hundreds of other settings that are tweaked in favor of server responsiveness on server SKUs.

Apparently we’ve got a bunch of support requests coming in from customers who are running server SKUs on their desktop and are upset that audio quality is poor. And this mystifies me. It’s a server operating system – if you want client operating system performance, use a client operating system.

PS: To change the MMCSS tuning options, you should follow the suggestions from the MSDN article I linked to above:

The MMCSS settings are stored in the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile

This key contains a REG_DWORD value named SystemResponsiveness that determines the percentage of CPU resources that should be guaranteed to low-priority tasks. For example, if this value is 20, then 20% of CPU resources are reserved for low-priority tasks. Note that values that are not evenly divisible by 10 are rounded up to the nearest multiple of 10. A value of 0 is also treated as 10.

For Vista, this value is set to 20, for Server 2008 the value is set to 100 (which disables MMCSS).

Why specify SND_NODEFAULT when calling PlaySound?

LarryOsterman — Fri, 26 Sep 2008 00:46:34 GMT

Because the alternative is often much worse.

Several months ago, I got a bug report that if you launched mmsys.cpl then set the “Select” sound to a value, then cleared the sound, the reporters application would ding whenever you moved around their tree control.

I dug into the problem a bit and discovered that the problem was actually in the Windows common controls. Under some circumstances the common controls would call PlaySound specifying a sound alias and the SND_ASYNC and SND_ALIAS flags only.

The problem with this is that if you specify SND_ALIAS without also specifying SND_NODEFAULT, the PlaySound API decides that you really want the sound to be played and it will play the default “ding” sound instead.

From the PlaySound API’s point of view this makes sense. After all, you asked the API to play a sound, it doesn’t know that you meant “only play this sound when the sound file represented by the alias exists”.

In fact, that’s the entire reason for the SND_NODEFAULT flag – it lets the PlaySound API that you only want to play a sound when the sound has been defined.

They just announced my PDC talk, so I guess I can mention at least the title of the talk.

LarryOsterman — Wed, 24 Sep 2008 21:21:59 GMT

The PDC folks just announced a host of Windows 7 related PDC talks, one of which is mine.

The title of the talk is “Windows 7: Building Great Communications Applications”. You can find it under the Windows 7 track on the Microsoft PDC site.

The primary target for my talk is developers who are building an application that in any way communicates between users (voice mail, instant messaging, voice over ip, etc). In addition, if you’re a games developer or a media player developer, you should also attend, there’s stuff in the talk for you too.

There are also some other cool talks included in the list that I’m absolutely planning on attending.

See you in LA!

Why call PlaySound(NULL, NULL, SND_NODEFAULT)?

LarryOsterman — Tue, 16 Sep 2008 03:18:17 GMT

Someone just wandered over to my office and he had noticed the following pattern in his code:

PlaySound(NULL, NULL, SND_NODEFAULT);
PlaySound(".Default", NULL, SND_SYSTEM | SND_ASYNC | SND_NODEFAULT);

He was wondering why on earth the code would do that call to PlaySound(NULL).

As I explained it to him, the reason is because you almost always want to cancel the current sound playing before you queue up the next sound.

The problem is that the current implementation[1] of PlaySound(…, SND_ASYNC) simply queues the request to a worker thread which blocks waiting on the currently playing sound to complete. So if you have a situation where you call PlaySound(…, SND_ASYNC) many times in succession, you’ll find that all the calls to PlaySound pile up behind each other, which means that you might end up playing sounds long after the action associated with the sound has completed.

Of course you might want this behavior – it’s certainly possible to string lots of system sounds together to produce any number of interesting effects.

But most of the time you just want to stop the current sound before you start playing the next.

[1] There are obviously no guarantees that the implementation of PlaySound won’t change – I’m just describing what the current code does. Even if the implementation is changed, it won’t change the underlying behavior.

Why doesn’t Windows support amplification of audio samples?

LarryOsterman — Mon, 11 Aug 2008 19:36:21 GMT

Nils Arthur asked in another post:

While we are talking volume controls. Could you explain why it's only possible to lower the volume in Windows (i.e. setting a volume between 0% and 100%) and not raise it (i.e setting it higher than 100%)?

Before I get into the the answer, let me define some terms: Attenuation means reducing the amplitude of a signal from a baseline - so if the signal is a full range sine wave going from 1.0 to -1.0, if you attenuate it to 50%, you get a sine wave from 0.5 to -0.5. I wrote about it (with pictures :)) in this post.

The answer to Nils' question is both simple and complicated.

The simple part of the answer: Because most PC audio hardware only supports attenuation and not amplification.

Now for the complicated parts of the answer: We only support what the hardware allows for master volume. And most hardware only supports attenuation. There are a lot of reasons for that, but the primary one is that it's dramatically cheaper (and uses less power) to attenuate signals than it is to amplify them.

The other issue w.r.t. amplification/attenuation is signal quality. As I mentioned in the post on volume above, you can attenuate a sample in the digital domain without loss of fidelity. However when you attempt to amplify a signal in the digital domain, it clips. That means that amplification MUST be done in the analog domain. Again, this goes to hardware costs - because amplification needs to be done in the analog domain, it means that the audio hardware needs to have an amplifier that can be digitally controlled, which is (again) more expensive. The audio hardware doesn't even have to support hardware volume - if Windows doesn't find a hardware volume control, it simply inserts a master volume into the audio pipeline.

Some audio hardware DOES support amplification, but the audio volume controls map the volume control from low to high into a range from 0..100 because it's dramatically simpler to represent that to the user.

That means that if an audio solution presents a hardware volume from -96.0dB to +3dB (there are a number of them that do that), we'll map that 99dB range into a 0.0 to 1.0 range that maps nicely into a slider. We've thought about differentiating between attenuation and amplification in the volume UI, but the reality is that the net effect is the same whether we represent amplification or not.

You can see if your audio hardware supports amplification by going to the multimedia control panel. Select the audio endpoint you want to check, go to the "Properties" dialog. On that dialog, check the "Levels" tab, the hardware master volume control is present there. You can right click on the text box and change the units from linear to dB, you can then move the slider around to see the dB range. Or you could write some code and call the IAudioEndpointVolume::GetVolumeRange API, which will return the information directly.

Whatever happened to Wave Out Mix?

LarryOsterman — Sat, 12 Jul 2008 00:54:19 GMT

The Intertubes are all atwitter with reports that Dell and other OEMs colluded with the RIAA to disable the Wave Out Mix option on new laptops.

Wow, what a tempest in a teapot. I just LOVE watching conspiracy theories as the echo chamber does it’s magic.

And of course it’s almost certainly hogwash (I don’t know for sure, but I do know that some of the rumors are totally stupid).

First off, what is Wave Out Mix? It’s an option that some audio manufacturers added to their audio hardware (Creative calls it “What U Hear”). Typically the Wave Out Mix is implemented by connecting the analog output from the DAC (Digital-to-Audio Converter) to a specific input on the ADC (Analog-to-Digital Converter) which is labeled as “Wave Out Mix”.

If you record on the Wave Out Mix input, you will capture the samples that are being played via Wave Out.

In Windows Vista, by default we only enable microphone, line in and digital inputs to the audio hardware (the theory being that users typically only want to be able to listen to those inputs). If the audio solution offers other inputs, they’re still there but we bury them somewhat.

You can find those additional inputs in mmsys if you start the sound control panel and go to the “Recording” tab. If you right click and select “Show Disabled Devices” you can enable those alternate inputs.

In addition, these days many OEMs don’t bother adding the Wave Out Mix support. It costs slightly more to order chips with Wave Out Mix support than it does to order chips without the functionality, and OEMs are incredibly cost conscious. The other reason is that for those OEMs that implemented the Wave Out Mix with an analog tap, you can achieve almost the same results with a $2.50 analog cable run between the output and the line in input of the machine.

Part of the reason that I know that this is just a conspiracy theory running rampant is that Windows Vista built the support for the Wave Out Mix input directly into the operating system. If you pass the AUDCLNT_STREAMFLAGS_LOOPBACK flag to the IAudioClient::Initialize method, then the audio system will initialize the engine in loopback mode. You can start capturing data off that IAudioClient object and you’ll get the post-mix output for the endpoint.

The loopback support was designed primarily for use by AEC functionality (which needs to be able to know what samples are being played), but it also allows you to perform essentially the same functionality as the Wave Out Mix hardware used to do.

If you want to play with the loopback functionality, the WinAudio SDK sample application allows you to capture using the loopback functionality.