Windows CE Networking Team WebLog : Author: Thais Melo

Writing a network audio driver in Windows CE – Part 5 - Power consumption

cenet — Thu, 15 Dec 2005 23:36:00 GMT

Writing a network audio driver in Windows CE – Part 5 - Power consumption

This blog will discuss power consumption, the fifth bullet point in my first blog about networked audio drivers. This is mainly a concern in portable devices using wireless networks, where battery life is important and wireless radio power consumption is usually significant.

In an audio driver that plays directly to local hardware (local audio driver or LAD), typical distribution of power consumptions is in:

· Higher CPU load to process audio from storage to output:

o If audio data is encoded, decoding audio data;

o General processing of audio data in the driver.

· Disk, reading audio data from storage source:

o Spinning/seeking/reading in a hard disk;

o Reading from flash memory.

· Audio hardware, playing back the reconstructed audio signal to:

o Headphone jacks;

o System speakers.

These activities generally consume a little extra power, optimization from the driver behavior is usually not needed, and this extra power usage ceases when the playing of the audio is done.

For a networked audio driver (NAD) all of the above, with the exception of the outputting the audio signal to hardware, still apply. In addition, there is power used to send data over the network to the remote device responsible for playback (the sink). In a wired network, there is usually little or none extra power involved in sending data through the network. The wireless case is more interesting, since the cost of having a wireless (i.e. Bluetooth, 802.11) radio on is usually fairly high. Moreover, wireless radios usually have varying levels of power consumption depending on what mode they are on .

For an example of how this can create non-trivial design decisions, when a user requests to play a track, if the source and the sink side don’t have an active connection, a NAD might have to negotiate a connection with a sink device before it starts playing the track. Establishing this connection consumes some power and takes some amount of time, thus adding some latency between the time when the user requests a play to the data actually getting played in the sink side. If the connection is already up, the data can be sent over the network right away without the delay for connection setup, but there is generally some cost involved in keeping the connection up. After the track is done playing, for how long should the connection be maintained (possibly not allowing the wireless radio to go into a lower/no power consumption mode) considering the user could potentially request that a new track be played soon after the initial one is done? The answer to that question is implementation specific, but can follow from:

· How much power is saved by disconnecting the connection;

· How likely is the user to request that a sound be played again;

· What kind of user experience the driver should be optimized for;

· What the underlying hardware is, and how it behaves in relation to power consumption (for example, certain Bluetooth radios have power save modes built into hardware that are automatically activated and transparent to upper layers).

[Author : Thais Melo]

Writing a network audio driver in Windows CE – Part 4 - Transient Errors

cenet — Tue, 15 Nov 2005 22:18:00 GMT

This blog will discuss transient errors, the fourth bullet point in my first blog about networked audio drivers.

In an audio driver that plays directly to local hardware (local audio driver or LAD), the audio is usually sent to hardware in fully decoded (PCM) data mode and errors are rare and usually permanent. Thus most audio frameworks are designed with such assumption built in.

In networks however, transient errors are common and expected. For instance, most networks can lose packets. Certain networks can re-order packets. Networks can also be temporarily disconnected. Wireless networks can have their bandwidth decreased by noise or close to out of range situations. Thus networked audio drivers (from here on NADs) need to be aware of the situations above and handle them graciously:

· If lost packets are possible/expected:

o As mentioned in the bandwidth blog, to save bandwidth, a compression method might be used by a NAD before the data is sent over the network. This compression method should be chosen carefully to assure that if a packet is lost, all (stateless compression) or most of the data that is not lost can still be decoded.

o If packets are being lost often due to network saturation, action might be needed from the NAD to avoid consistent sound degradation. This case is equivalent to diminished bandwidth, and is treated in that entry.

· Packet re-ordering:

o If packet re-ordering can happen in the network, the source side should have a time stamp mechanism so the sink side can determined in which order the packets were sent.

· If temporary disconnections are possible/expected:

o Particularly in wireless networks, the sink and source sides might not be permanently connected due to power concerns. If a request to play audio is made, a connection might need to be started. The driver could do this transparently (connection started when the user hit play on a song for example) or ask the user to connect if currently disconnected.

o If the sink becomes disconnected in the middle of playback, the NAD should behave graciously, possibly warning the user about the error and stopping playback.

· In a wireless network where interference/noise or low signal strength condition can decrease bandwidth temporarily, see bandwidth and/or timing blog for recommended solutions.

[Author: Thais Melo]

Writing a network audio driver in Windows CE – Part 3 - Bandwidth

cenet — Mon, 24 Oct 2005 22:42:00 GMT

This blog will discuss bandwidth, the second bullet point in my first blog about networked audio drivers.

In an audio driver that plays directly to local hardware (local audio driver or LAD), bandwidth is usually not a concern since it is only dependent on internal components (disk bandwidth, bus speed, processor speed, DMAs, etc) which provide more than enough bandwidth to play music, is well known and rarely varies.

A networked audio driver (or NAD) on the other hand sends packets down the network stack it is using, which then go over some medium to a different device, where it gets played. So it depends in local bandwidth (above paragraph) and network bandwidth. I will from here on only discuss network bandwidth.

Networks can only send a limited amount of data, usually have more than one user and are used for more than one purpose. This means that using network bandwidth efficiently is important. One way to do so is to compress the music being transferred in a format that both the source and sink understand. The decision on what type/amount of compression to use could be:

fixed (registry setting or hard coded in both source and sink);
made at connect time during a connect handshake or some other connecting mechanism;
made during streaming if compression type/amount needs to change.

A NAD using a network where the available bandwidth may vary should also be designed to adapt to those situations. For example, on a wireless medium, bandwidth decreases if there is noise in the channel or distance between source and sink increases. In both wired and wireless mediums, if other users/applications are using the network, the bandwidth available for the NAD decreases. As mentioned in the timing blog, one possible solution is to decrease the bit rate of the audio data being streamed. Another solution is to temporarily stop streaming until there is again acceptable bandwidth and possibly notify the user in some way. If real time behavior is expected, simply streaming the audio data at a lower bandwidth than required will create poor user experience (depending on the sink implementation, it could create skipping, playing the data at an incorrect rate, etc).

[Author: Thais Melo]

Writing a network audio driver in Windows CE – Part 2 - Timing

cenet — Wed, 19 Oct 2005 03:19:00 GMT

[Revision 10/20/05 - minor changes to make content clearer]

This blog will discuss timing, the first bullet point in my first blog about networked audio drivers.

In an audio driver that plays directly to local hardware (LAD), the mechanism for timing is usually relatively simple. The driver writes to hardware buffers, stops writing when they are full, gets signaled when there are free buffers again. A networked audio driver (NAD) in the other hand sends packets down the network stack it is using, which then go over some medium to a different device, where it gets played.

Note that some of the issues discussed below are also valid for a LAD, but they become more visible/bigger issues in NADs, because NADs might have higher local CPU usage, less information about sink device, less control over sink device, greater latency and performance issues due to the network they stream over, etc.

One common system for NADs is to establish a connection through some sort of handshake where the codec type/codec settings/device settings are negotiated and then initiate streaming which is a one way communication from the source device to the sink device.

Using the above mechanism, the sink’s timing implementation is just to play the sounds at the correct speed, and buffer everything else it receives until it is needed. The source implementation is more interesting due to the challenges below:

1. It needs to send data fast enough so that the sink has enough data to play at any one time.

2. It must not send data so fast that the sinks buffers get overrun or latency due to excessive buffering becomes strongly noticeable.

3. If there are no built-in throttling mechanisms in the network technology it is using, it must send data to the network at a speed that will not adversely flood the network.

If 1 is not followed, the sink will not have enough data to play, which will cause either skips or quality degradation if the sink attempts some skip mitigation technique. A simple way for the source to figure out how fast it needs to send data is to follow the audio sampling rate (kbit/s) of the audio data it is streaming (and possibly adjust data rate if there is extra data per packet), but trying to figure out what to do when this data rate is not possible due to performance problems in the source device or network presents more design options.

If the issue is permanent, probably the best solution is to renegotiate the streaming rate between the devices to lower quality and thus lower kbit/s, or use a different codec.

But sometimes performance issues can be intermittent and allow for solutions that will not yield a decrease of quality. For example, assume the music is being streamed from a device where at some point a higher priority application starts using most of the CPU cycles for a short period of time, temporarily starving the NAD, which will yield a skip on the sink side. In the case where the skip in inevitable, the driver might choose to make the skip longer to fill all local buffers and/or temporarily decrease system load before starting to stream again to avoid multiple skips when the system becomes busy. This creates better user experience, and much as one skip is annoying, 100 skips in 10 seconds are considerably worst.

On point 2, to avoid sink side overrun, the source side needs to ensure that:

· The data is not consistently sent faster than the sink can consume it;

o If the source sends more data than the sink can use consistently, even if it is a very small amount, eventually it will completely fill the buffers in the sink side which will have to start to use some mechanism to throw data away.

· There are no burst of data sent that are larger than the buffers in the sink side.

o For instance, for efficiency reasons the source side might decide to send n milliseconds worth of audio data to the source at a time for efficiency reasons. The amount of data in n milliseconds must never be bigger than the size of the buffers in the sink side.

For latency issues, the sink side buffering can also be important. For example, if all the source can do is send data bits to the sink and no control messages in the lines of “flush buffer”, if n milliseconds of data are buffered, several user scenarios can be affected, for example:

· It will take n milliseconds to stop playing from the time the source stops sending data;

· If user decides to change the data that is being streamed (i.e. skip to another track mid-track), it will take n milliseconds for the change to reflect in the sink;

· If there are sounds created in real-time (i.e. system sounds) those will only be played n milliseconds later;

· If video is being played concurrent to the sounds, the sound and video will become unsynchronized.

If (n + network latency time) is small enough this is quite imperceptible to the user, but otherwise this can be a serious user experience issue, so it is important to design buffering appropriately.

I will not discuss 3 here since there are a lot of network timing information published, significant research in this area and most networking technologies supply some sort of load control/balancing (for some examples, see how TCP does this at the IETF website).

Unfortunately, I don’t know of any sample audio drivers in CE which have timing mechanisms built in other than interrupts, so the writer of a NAD would have to implement timing themselves.

There are of course many ways this could be done, but a couple of possible ways would be:

· Minor changes to wavdev sample driver, and add a middle layer between the stack and the driver that blocks the driver on writes for the necessary amount of time for the appropriate data rate.

o The middle layer would know how fast the data should go to the network, how much has been sent at any one point, and the time elapsed;

o If the audio driver tries to send a data packet at a time before any more data in meant to be sent to the sink, it is blocked until the appropriate time;

o If it tries to send data at or after the appropriate time it is allowed to do so without blocking.

o Some allowance for buffering should be made, so instead of one hard number for data sent in time n, it might be necessary to use 2 or more thresholds.

· Change wavdev (or write an audio driver from scratch) to do its own timing on writes (sends):

o This solution does not need any extra layers for timing, but involves more changes on the sample driver;

o Most of the above changes explained for adding an extra layer to block writes apply, but the driver itself needs to stop and trigger its writes.

The choice of which way to implement this of course depends on the technologies being used, and other design trade-offs.

Author Note: I apologize for posting this one late for the data mentioned in my initial blog.

[Author: Thais Melo]

Writing a network audio driver in Windows CE – Part 1

cenet — Mon, 10 Oct 2005 22:17:00 GMT

[Updated to reflect status of other parts of this blog]

In this series of blogs I will discuss the implementation of an audio driver that starts from the wavdev sample and streams audio over a network in real time. In part 1 I will list the main things that one has to worry about only when writing a network audio driver, and I will discuss each bullet point listed below in further blogs (parts 2 through 6) that will be posted in the date mentioned in the bullet point. If there is anything in particular that I listed (or didn’t list) that you want to see described in more detail, please let me know.

While there are multiple ways for writing an audio driver for Windows CE, probably the easiest way to get started is using the sample driver code provided with Windows CE.

For more information on the sample driver please see:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wceddk40/html/cxconsamplewavedevdriver.asp

Starting from the wavdev driver most of the upper layer work is done, meaning only hardware specific changes need to be made. Please note that this is only sample code, and as the documentations suggests, you are encouraged to change the sources in any part of the code to fit your needs.

In an audio driver that plays to headphones or speakers, usually all the hardware specific layer does is write to certain hardware buffers obeying hardware interrupts. Audio drivers that write to a network are slightly different in that they write to a network buffer that then gets sent down the network stack and over some medium to a remote device.

Some important differences for network audio drivers, which I will treat in more detail in future blogs:

· Timing - There are no hardware interrupts, so either the driver or the network layers below it, must be able to throttle the speed at which data is sent to avoid overrunning the buffers in the receiving side, the buffers on lower layers of the network stack or the network itself. Further details in Part 2 – estimated post date 10/17/05.

· Bandwidth - Most networks can only send a limited amount of data; this means sending raw high quality PCM data might not be acceptable. The driver might need a mechanism to figure out how much data can be sent over the network and in which format can it be sent (which formats does the other side understand). Further details in Part 3 – estimated post date 10/24/05.

· Latency – Latency in hardware is usually very low. When transmitting data over the network this can increase significantly. Also, in general, buffering data and sending in large chunks improves bandwidth but increases latency. Further details in Part 2.

· Transient Errors – Certain networks can lose packets, it is important to consider if this is acceptable. Also, wireless networks can have their bandwidth decreased by noise or close to out of range situations; if using wireless networks are being used it is important for the driver to be able to cope with this situations. Networks can also be temporarily disconnected, thus the driver needs to be aware of disconnections/connections, and the correct actions to take in such cases. Note that most audio frameworks do not deal with this type of errors graciously because they where in general designed for writing to local hardware, where transient errors are extremely rare. Further details in Part 4 – estimated post date 11/14/05.

· Power consumption - This mostly only applies to portable devices, especially to if the driver is using wireless networks. For instance, design decisions need to be made to when to connect and disconnect to the remote device when the driver is idle or back in use. Further details in Part 5 – estimated post date 11/28/05.

[Author: Thais Melo]