Cooking with Code

What’s “Dynamic” about this SoundEffectInstance? (Part 2)

ashtat — Sat, 02 Oct 2010 14:09:19 GMT

I just finished writing an article on this topic for the Creators Club….ummm…sorry, I mean the App Hub! Check out the article, it has lots of pretty pictures and everything!

What’s “Dynamic” about this SoundEffectInstance? (Part 1)

ashtat — Wed, 08 Sep 2010 21:57:06 GMT

Or in other words, what’s up with the new “DynamicSoundEffectInstance” type in XNA Game Studio 4.0? If you’re familiar with the SoundEffect API in XNA Game Studio 3.1, you’ll know that to play sound effects you need to -

Add your audio files to the content project in your solution and set SoundEffectProcessor options.
Build your project which generates all the appropriate XNB files.
Load these XNB files in your game at runtime to get back SoundEffect objects.
Use “Fire and Forget” or “Create and Configure” patterns to playback multiple sound instances.

This means that basically all the sounds in your game are “pre-authored” that is your game is essentially limited to using these “static” sounds that cannot be changed at runtime. Of course you can tweak the playback of these sounds at runtime, change the volume or pitch, spatially position them in your 3D world using Apply3D, control the loops . But what if you wanted to procedurally generate some sounds – I mean doesn’t everyone want to write a modular synth? Too bad, can’t do it. Can you do some pre-processing on your content before playing it back? Nope. How about sample accurately synchronizing multiple sound instances? Out of luck. Not so in Game Studio 4.0 - the “Dynamic Audio” features now remove these limitations.

Dynamic is anti-static!

In a nutshell, dynamic audio allows working with raw audio data that the game can control at runtime. There are couple different ways work with “dynamic” audio -

Create a SoundEffect from the raw buffer: Once created this SoundEffect is in no way different from a SoundEffect hydrated from an XNB file (using Content.Load). All the playing SoundEffectInstance objects share the audio buffer owned by this SoundEffect. It’s not possible to change the audio data after creation.
DynamicSoundEffectInstance: This is the shiny new class we introduced in XNA 4.0 that enables streamed playback of audio buffers. It inherits from SoundEffectInstance and provides all the basic semantics (with some caveats) of a SoundEffectInstance with additional ability to dynamically control the audio data being rendered.

Why would you choose one over the other? That depends on what you’re trying to do. In general, DynamicSoundEffectInstance is useful for scenarios where the playback changes over time whereas SoundEffect is useful when you have a sound that is shared by multiple instances that doesn’t fundamentally change with time. For example – if you’re going to procedurally generate an explosion, you’d probably want to use SoundEffect so that all explosions in your game can share the same audio buffer and optimize on memory usage. On the other hand DynamicSoundEffectInstance is probably more appropriate if you’re trying to create a virtual instrument from a synthesized audio source where the player has parametric control over the synthesis. Another example is streaming from a large audio files where DynamicSoundEffectInstance is probably a better choice in terms of the memory usage.

Next up – all the juicy details on how to use DynamicSoundEffectInstance…

SoundEffect Creation in XNA Game Studio 4.0

ashtat — Thu, 03 Jun 2010 02:23:00 GMT

There are a couple new ways to create a SoundEffect in XNA 4.0 - a new factory method that loads wave files and a couple new constructors that take raw PCM audio data.

SoundEffect.FromStream

public static SoundEffect FromStream(Stream stream)

This method takes a plain vanilla Stream to a standard .wav file with PCM audio data. Yep, wav files can contain audio in a lot of different format – PCM, ADPCM, MP3, WMA, whatever. Most wav files in the wild are usually already in PCM format but you can always fire up your favorite audio editor to check/convert the format. Audacity or Wavosaur are both easy to use and free editors.

Here are some useful details -

Format constraints - PCM, 8 or 16-bit, 8KHz to 48KHz, mono or stereo.
Reads the loop region from the file. If there isn’t one the whole file will loop – only if you play it looped of course.
We don’t mess with the seek position of the stream so make sure it’s set to the right place when you pass it in (and reset it when this method returns if you’re going to use it elsewhere in your code).
If you’re adding wave files to your content project make sure you set “Build Action” to “Content” and “Copy to Output Directory” to “Copy if Newer”. This will ensure that it doesn’t get compiled to an xnb file and that the file gets deployed with your project.

SoundEffect Constructors

public SoundEffect(byte[] buffer, int sampleRate, AudioChannels channels)

public SoundEffect (byte[] buffer, int offset, int count, int sampleRate, AudioChannels channels, int loopStart, int loopLength)

These constructors are handy for things like creating procedural sound effects, reading from your custom wave bank etc.

Format for the audio data is restricted to PCM, 16-bit, 8KHz, 48KHz, mono or stereo.
Offset and count are in bytes – offset is where the audio data starts within the buffer and count is the length of audio data.
The length of the audio data for playback must be block aligned for the format. This means that -
- For the first simpler constructor the size of the entire buffer must be block aligned.
- For the second meatier overload the offset and count values must be block aligned.
Loop start and length are in samples. If you don’t specify any (that is pass in 0 for both) we’ll loop the entire buffer.

The most common questions we get at this point are -

1. If the format is restricted to 16-bit, why do these methods take byte[] and not short[]?

When we first designed this API our first instinct was to do exactly that – any method that took raw audio buffers took short[] and not byte[]. When we looked a little deeper though we realized that this really wouldn’t address some important scenarios. For example, things got really clumsy whenever we wanted these APIs to play nicely with streams. Also when we thought about extensibility especially around formats that we may want to support in the future the special implicit meaning given to the short[] (“array of 16-bit samples”) seemed odd and anti .Net. So after considering all the options we settled on byte[] – for one it plays nice with existing .Net APIs that take raw buffers. It also follows an established and familiar pattern. It also gives us better flexibility for supporting other formats in the future. Within the audio rendering stack everything eventually turns into arrays of bytes anyway. Yeah, there are scenarios where this forces you to do some extra work to do conversions to byte[] (especially true when writing any DSP code) but this is something we could easily optimize in a future release. And taking short[] would not have really solved this anyway.

2. What exactly is block alignment and how do I calculate it?

In a single sentence, block alignment value is the number of bytes in an atomic unit (aka block) of audio for a particular format. For PCM formats the formula is extremely simple: “Block Alignment = Bytes per Sample * Number of Channels”. For example, block alignment value for mono 16-bit PCM format is 2 and for stereo 16-bit PCM format it’s 4. We have a couple handy helpers that can help calculate the block aligned values – GetSampleSizeInBytes and GetSampleDuration convert from time units to block aligned byte value and back.

3. Why are the loop arguments expressed in samples? Why not in bytes like the offset and count?

Most (all?) audio editors that allow setting loops in audio files do so in samples. In most cases when someone wants to loop a sub-region within a sound they use an audio editor to define it in the content itself. Even compressed formats like XMA on Xbox read the loop values in samples. Ultimately these formats that support sample accurate looping (i.e. smooth loops that play back exactly as authored and don’t pop or glitch) decode to PCM and use these loop sample values during playback (the loop values for non-PCM formats can have their own alignment restrictions but that’s a discussion for a different post). In any case for PCM formats converting from byte value to sample value is incredibly trivial - “Sample Offset = Byte Offset / Block Alignment”. So we optimized for the most common case and loop arguments ended up being in sample units.

Pseudocode is worth a thousand words

Example 1 – Playback a wave file from the content folder. In this case “Sample.wav” is a sound file that’s included in the content project as “content”.

SoundEffect soundFromFile = SoundEffect.FromStream(TitleContainer.OpenStream(@"Content\Sample.wav"));

soundFromFile.Play();

Example 2 – Create and playback a second long 48KHz stereo sound.

// Allocate a buffer to hold 1 second worth of stereo data

byte[] buffer = new byte[SoundEffect.GetSampleSizeInBytes(TimeSpan.FromSeconds(1), 48000, AudioChannels.Stereo)];

// Method fills the passed buffer with audio data – left as an exercise to the reader (I’ve always wanted to write that!) :)

FillBuffer(buffer);

// Create the SoundEffect from the buffer

SoundEffect sound = new SoundEffect(buffer, 48000, AudioChannels.Stereo);

// Play a "fire and forget" instance.

sound.Play();

Example 3 – Plays back 500ms from 200ms offset within the buffer.

int bufferLength = SoundEffect.GetSampleSizeInBytes(TimeSpan.FromSeconds(1), 48000, AudioChannels.Stereo);

int offset = SoundEffect.GetSampleSizeInBytes(TimeSpan.FromMilliseconds(200), 48000, AudioChannels.Stereo);

int count = SoundEffect.GetSampleSizeInBytes(TimeSpan.FromMilliseconds(500), 48000, AudioChannels.Stereo);

byte[] buffer = new byte[bufferLength];

FillBuffer(buffer);

SoundEffect sound = new SoundEffect(buffer, offset, count, 48000, AudioChannels.Stereo, 0, 0);

sound.Play();

Ahemm…is this thing on?

ashtat — Wed, 05 May 2010 22:23:00 GMT

As the great James Anchower once said, “it’s been a long time since I rapped at ya, but there’s been all sorts of mess going on in my life”. For instance I got super-ultra-Windows Phone 7-series++ busy with XNA GameStudio 4.0. And then my wife and I shipped our “Baby 1.0” this year (thanks!). So here I am, a proud father of beautiful girl and a proud dev of some shiny new XNA framework features.

So what have I got to show for all this time? A bunch of cool stuff.

Dynamic Audio - Ask long enough and ye shall receive! You can now playback raw audio buffers so go forth and build your modular synthesizer, sequencer, procedural sound effects and whatever else that floats your proverbial boat.
Microphone Support – Find all the connected microphones and use them in your game.
Imaging support – Create and save textures from/to JPEG/PNG streams. A small update to media API on the phone lets you even save the picture to the media library.
Silverlight support – We refactored our assemblies so a lot of the XNA framework features (like audio and media) can be used in Silverlight applications. We also added a couple new ways to create SoundEffect and Song types that don’t depend on the ContentManager.

Get your XNA GameStudio 4.0 CTP here. As always the devil’s in the details…so stay tuned!

Video Playback in XNA Game Studio 3.1

ashtat — Sat, 20 Jun 2009 02:01:00 GMT

Michael did a post some time back to give an overview of the new video playback feature. Here’re some additional tips that might help -

Adding video to your project

In a nutshell - make sure your videos are encoded with Windows Media Video v9 codecs, are using the CBR format and that they have an audio track (also encoded with CBR). If you’re using Windows Media Encoder, select “File download (computer playback)” for your encoding destination format. This will give you the options to select CBR encoding for both video and audio streams.

The video clip must have an audio track otherwise the build will fail with - “Please make sure that the video is not DRM protected and is a valid single-pass CBR encoded video file” (yeah, I know, we could done better and added some info about the missing audio track here). This is an unfortunate limitation of the native technology we depend on for playback but you can always mute the playback – “VideoPlayer.IsMuted = true”.
I’ve discovered that attempting to add a “silent” or “blank” audio track in Windows Media Encoder can be troublesome (sometimes the video will get “stuck” on the first frame on Windows). I’ve successfully used the encoder to do screen captures and format conversions (e.g AVI to WMV). If my source content didn’t have an audio track I used Windows Movie Maker to add one there.
- Adding an audio track to your movie in Windows Movie Maker takes little more skill than knowing how to drag and drop.
- When publishing the movie from Windows Movie Maker, select “DVD Quality” encoding which should give you the encoded video in CBR format.

Video playback

You shouldn’t have to call Video.GetTexture more than once per frame. The video stream decoder runs on a separate thread and handles synchronization with the audio track. Calling GetTexture always returns the last fully decoded frame. If a new frame is not quite ready it returns without doing anything but you still incur that managed to native call penalty.
If you want to show the same video on multiple screens in your game, just get the current video frame once and use it multiple times in your drawing code. This will ensure that all the screens show perfectly synchronized playback. If you call GetTexture multiple times there’s a chance that the frame can update in between two calls and the screens will look out of sync.
Playback of multiple videos is supported but playing back multiple high-res videos may not be such a great idea. Decoding video streams is expensive and the cost goes up quickly with the resolution of the source content.

SoundEffect API Changes in Game Studio 3.1

ashtat — Mon, 15 Jun 2009 20:13:00 GMT

Shawn beat me to the punch and did this great post on the changes to the SoundEffect API in XNA GS 3.1. Here’re some tips if you’re trying to migrate your 3.0 code to the new release.

“Fire and Forget” Playback

SoundEffect.Play methods are now exclusively for “Fire and Forget” sounds. Instead of returning a SoundEffectInstance they’ll return a bool – “true” for successful playback and “false” instead of throwing the InstancePlayLimitException. XNA Framework will dispose these sounds as soon as they’re done playing.

// this will play a 2D fire and forget sound effect with default

// volume, pitch and pan

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

bool effectPlayed = effect.Play();

// This will play a 2D fire and forget sound effect with specified volume, pitch and pan

bool effectPlayed = effect.Play(0.8f, 1.0f, 0.0f);

You can only playback 2D, non-looping sounds using this “fire and forget” mechanism. For everything else – there’s “Create, Configure and Play”.

“Create, Configure and Play”

The SoundEffect.CreateInstance method returns a SoundEffectInstance object. The lifetime of this instance is completely controlled by the game. In other words you need to call Dispose on the instance returned by this method when you’re done with it. The usage pattern is -
1. Create a SoundEffectInstance by calling SoundEffect.CreateInstance.
2. Configure the SoundEffectInstance by setting all the properties. As you’ll notice, some of these properties are locked into place once you call Play.
3. Call SoundEffectInstance.Play.
4. When you’re done with the sound, call SoundEffectInstance.Dispose.
```
// game controls the lifetime of the sound effect instance 
```
```
SoundEffect effect = Content.Load< SoundEffect>(“GameSoundEffect”);
```
```
SoundEffectInstance instance = effect.CreateInstance();
```
```
instance.Play();
```
```
…
```
```
instance.Dispose();
```
All SoundEffect.Play3D methods have been deprecated. Instead, game can now configure the SoundEffectInstance object returned by the SoundEffect.CreateInstance method for 2D/3D positioning and looping before playback. This configuration is locked into place on the first Play call made on the instance.
- By default all instances are created as 2D non-looping sound effects.

// this will play a 2D non-looping sound effect

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

SoundEffectInstance instance = effect.CreateInstance();

instance.Play();

Calling Apply3D on a SoundEffectInstance before calling Play configures it as a 3D positioned sound.

// this will play a 3D positioned non-looping sound effect

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

SoundEffectInstance instance = effect.CreateInstance();

instance.Apply3D(listener, emitter);

instance.Play();

Setting the “IsLooped” property before calling Play configures the instance as a looping sound.

// this will play a 2D looping sound effect

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

SoundEffectInstance instance = effect.CreateInstance();

instance.IsLooped = true;

instance.Play();

// this will play a 3D positioned looping sound effect

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

SoundEffectInstance instance = effect.CreateInstance();

instance.IsLooped = true;

instance.Apply3D(listener, emitter);

instance.Play();

If you call both Pan and Apply3D (why would you do that?) the last call wins and determines if the sound is configured as a 2D or 3D positioned.

// this will play a 2D positioned looping sound effect.

// setting the Pan in this case overrides Apply3D.

SoundEffect effect = Content.Load<SoundEffect>(“GameSoundEffect”);

SoundEffectInstance instance = effect.CreateInstance();

instance.IsLooped = true;

instance.Apply3D(listener, emitter);

instance.Pan = 1.0f;

instance.Play();

Helper Extension Class

Here’s a helper class that may ease the pain of moving your code to the new world order. Remember though, this is a helper for “Create and Configure” playback so you still need to explicitly dispose the instance when you’re done.

public static class SoundEffectExtensions

    public static SoundEffectInstance Play3D(this SoundEffect effect, AudioListener listener, AudioEmitter emitter)

        return Play3D(effect, listener, emitter, 1f, 0f, false);

    public static SoundEffectInstance Play3D(this SoundEffect effect, AudioListener listener, AudioEmitter emitter, float volume, float pitch, bool loop)

        return Play3D(effect, new AudioListener[] { listener }, emitter, volume, pitch, loop);

    public static SoundEffectInstance Play3D(this SoundEffect effect, AudioListener [] listeners, AudioEmitter emitter, float volume, float pitch, bool loop)

        SoundEffectInstance instance = effect.CreateInstance();

        instance.Volume = volume;

        instance.Pitch = pitch;

        instance.IsLooped = loop;

        instance.Apply3D(listeners, emitter);

        instance.Play();

        return instance;

    public static SoundEffectInstance Play2D(this SoundEffect effect)

        return Play2D(effect, 1f, 0f, 0f, false);

    public static SoundEffectInstance Play2D(this SoundEffect effect, float volume)

        return Play2D(effect, volume, 0f, 0f, false);

    public static SoundEffectInstance Play2D(this SoundEffect effect, float volume, float pitch, float pan, bool loop)

        SoundEffectInstance instance = effect.CreateInstance();

        instance.Volume = volume;

        instance.Pitch = pitch;

        instance.Pan = pan;

        instance.IsLooped = loop;

        instance.Play();

        return instance;

Greetings Fellow Humans

ashtat — Sat, 13 Jun 2009 00:00:00 GMT

Here I am, a little late to the MSDN blog party. But then again, my timing might not be so bad after all. Yesterday we announced the release of XNA Game Studio 3.1. Having spent a great deal of 2009 working on this baby it’s absolutely awesome to see it being set free to roam the big wide world. And it’s even more exciting to see what games you, the ultra-talented Indie developer lot, will build to entertain us!

Here’re some factoids about me –

I work on the XNA team constantly humbled by the sheer awesomeness of my coworkers.
I mainly work on the Audio and Media APIs – but then I also get to work on whatever it is that needs me working on it.
I grew up in India, the land of curry and other mythical things (oh yeah…now there’s a future post!).
I live in Seattle with my wife and our feline love-child Navin R. Johnson.
In my spare time I enjoy – playing music (guitar, piano, kazoo…), cooking and eating delicious food, brewing and drinking delicious beer etc. etc. You get the drift.

And now it’s time for me to think of posting something useful… :)