Many people may have heard of some of the ongoing “buzz” words around SmoothStreaming. SmoothStreaming is a technology born of work in the Silverlight domain. The basic idea is that allows developers to supply audio and video streams that adapt to the users current bandwidth and machine conditions (like cpu retention etc.). Actually the adaptive streaming heuristics are quite detail and work really well.

Whats so important about this technology is not so much the technology itself but the underlying platform that allows for it to work. This is MediaStreamSource a public but little known abstract class available to developers to allow them to extend the basic functionality of the media stack.

Before I get into more detail about how this facilitates smooth streaming I’m going to give a brief introduction of how MediaStreamSource works and some of the ideas it could potentially be used for. This article will be very much a simple background of MediaStreamSource. Part 2 of the series will try to further address how to use MediaStreamSource and go into detail about the API.

MediaStreamSource at heart allows you the developer to directly send “un-containerized” media samples to the Silverlight media stack for processing and display. If we look at most video formats today they consist of 2 main ideas:

1. The underlying stream codec. This is a compression format applied to the raw media data since raw media data is extremely large. With advents in video/audio compression advanced formats like H.264, VC-1, AAC have been developed which are compressions algorithms specifically built to cater for the kind of redundant information found in audio/video data. The important point here is the codec provides a compressed bit stream of the original data, usually broken up into peices called samples. (In video this usually represents a frame, in audio a slice of time).
2. The container. This is the file that organizes the bit stream samples above into some structure that makes them easy to index and read by a media playback stack. The container file contains information about the format of the streams, details about the media itself (width, height, duration), and most importantly it contains the sample index. This index is used to find information about samples such as duration to display a sample for, the synchronization time for a sample used for audio/video synchronization. This is the sort of information required to effectively playback the seperate audio/video sample streams in a synchronized fashion.

MediaStreamSource requires developers to implement 2 above themselves, while still providing the framework to decode and playback the coded sample bit stream from 1 above.

So why would anyone want to implement 2 above themselves, this seems like a lot of work? To give an example lets now discuss the SmoothStreaming case. For SmoothStreaming we need to be able to change the quality of the audio/video being played back midstream depending on a number of different heuristics like network bandwidth, cpu performance etc.

If we consider that a video or audio bit stream is broken into samples as mentioned above, then all we’d need to do would be to change the samples were sending to the Silverlight plugin when we detect a change in conditions effecting our heuristic. In essence this is mostly what SmoothStreaming does. It packages small sets of samples into “chunks” and switches on the fly what quality the “chunks” we are downloading are, and sends it through MediaStreamSource down to the Silverlight plugin.

This scenario could be implemented directly into the stack, but this is just one of the flexible uses that MediaStreamSource provides for us, to make scenarios similar to these available.

MediaStreamSource was made initially available in Silverlight 2. It was in Silverlight 3 that the scope of MediaStreamSource expanded significantly when the ability to playback raw audio and video frames became available in Silverlight. Before this Silverlight was capable of playing back a predetermined set of audio/video codec's including VC-1, wma, and mp3. With raw audio and video support this means the user could provide their own container and decoder and playback any format in Silverlight that they wish with their decoder platform, making use of the rest of the Silverlight RIA stack and media playback framework.

Don’t get me wrong building a decoder is no simple endeavor, but with the possibility open, and some smart third-party developers out there re-usable libraries for different codec's are starting to emerge. Beyond this one of my many hobbies includes attempting to provide old console emulation in Silverlight through MediaStreamSource. It seems to be an idea shared by others as well: http://10rem.net/blog/2009/03/18/silverlight-3-%E2%80%93-creating-video-from-raw-bits-using-a-mediastreamsource.

I hope you found this overview of MediaStreamSource useful and can get a basic feel for what it’s about. In the coming articles I will describe the interface in more detail, and how it can be used in applications that you’re looking to build. Hopefully the ideas for use can be extended far beyond what they were imagined for (I know games emulation wasn’t the number 1 scenario!!)

Check out MSDN docs here:

http://msdn.microsoft.com/en-us/library/system.windows.media.mediastreamsource(VS.95).aspx