Coding4Fun : web miscellaneous

Silverlight 3 File Transfer Application

Coding4Fun — Fri, 04 Sep 2009 09:43:00 GMT

Author: Giovanni Montrone
Difficulty: Intermediate
Time Required: 5-10 hours
Cost: Free
Software: Visual Web Developer Express SP1 (or Visual Studio 2008 SP1), Silverlight 3 SDK, Silverlight 3 Tools for VS, Silverlight Toolkit
Hardware: None
Use it Now: Run the Application
Source Download: CodePlex

Introduction

Every once in a while I will run into a situation where I need to send a file to someone, but struggle to find an easy way of doing it. Instant messenger programs usually work until you run into a situation where someone just cannot send or receive a file from chat client whether it’s due to firewalls, differing client versions, or multi-IM incompatibilities. Other times, I will try to send the file via email just to find out that the person’s email server blocks specific extensions. This quick application will allow two users to quickly and easily connect to a Silverlight 3 client and send each other files.

Overview

In this application, a user will connect to the Silverlight application and choose to either host or join a session. If a user decides to host a session, he or she will be given a random eight character session key, and he or she will be in a waiting status until another user connects. When a user wishes to connect to the host, he or she can supply the host’s session key which will establish a connection between the two users. Once connected, the users will be able to send files to one another, and will also be able to chat with each other via simple text messages.

Polling Duplex

Sending a file from one client application to another requires a common central point in order to properly route the message from one user to another. Since it is easy to host a Silverlight application in an ASP.NET page, we will use an ASP.NET back end to manage all communications between all connected users. In order to do this, we must be able to host a service that is able to accept incoming messages from the Silverlight client, and perform a push back out to the intended recipient. This is accomplished by using the WCF Polling Duplex (System.ServiceModel.PollingDuplex.dll) channel. Silverlight 3 allows us to directly add a service reference to a service of this type, and handles all of the complexities for us. I began by using the DuplexService.cs file from a sample application from MIX09 when Silverlight 3 beta was released. The code starts us out with a couple of base abstract classes, and interfaces that we must inherit from in order to create our own service.

FileSendService

The two main things we need to do in order to create our own service is to define custom message types that will be used for communication, and override the base DuplexService class to properly handle these messages. We will create our custom message types by using DuplexMessage (defined in DuplexService.cs) as our base class. Our message classes must be defined with the [DataContract] attribute, and the the member variables must be public and contain the [DataMember] attribute. This will allow our Silverlight client project to share these definitions using a service reference. Additionally, the Duplex message class must have the [KnownType] attribute for each descendant message created.

[KnownType(typeof(HostSessionMessage))]
[KnownType(typeof(JoinSessionMessage))]
[KnownType(typeof(FileBeginUploadMessage))]
[KnownType(typeof(FileTransferBytesMessage))]
public class DuplexMessage { }

[DataContract]
public class HostSessionMessage : DuplexMessage
{
    [DataMember]
    public string Username;
}

[DataContract]
public class JoinSessionMessage : DuplexMessage
{
    [DataMember]
    public string Username;
    [DataMember]
    public string SessionKey;
}

[DataContract]
public class FileBeginUploadMessage : DuplexMessage
{
    [DataMember]
    public string FileName;
    [DataMember]
    public long TotalBytes;
}

[DataContract]
public class FileTransferBytesMessage : DuplexMessage
{
    [DataMember]
    public long StartByte;
    [DataMember]
    public long PacketSize;
    [DataMember]
    public byte[] Bytes;
    [DataMember]
    public bool EndFile;
}

<DataContract(Namespace := "http://samples.microsoft.com/silverlight2/duplex"), 
KnownType(GetType(HostSessionMessage)), KnownType(GetType(JoinSessionMessage)), KnownType(GetType(FileBeginUploadMessage)), KnownType(GetType(FileTransferBytesMessage))> _
Public Class DuplexMessage
End Class

<DataContract()> _
Public Class HostSessionMessage
    Inherits DuplexMessage
    <DataMember()> Public Username As String
End Class

<DataContract()> _
Public Class JoinSessionMessage
    Inherits DuplexMessage
    <DataMember()> Public Username As String
    <DataMember()> Public SessionKey As String
End Class

<DataContract()> _
Public Class FileTransferBytesMessage
    Inherits DuplexMessage
    <DataMember()> Public StartByte As Long
    <DataMember()> Public PacketSize As Long
    <DataMember()> Public Bytes() As Byte
    <DataMember()> Public EndFile As Boolean
End Class

<DataContract()> _
Public Class FileBeginUploadMessage
    Inherits DuplexMessage
    <DataMember()> Public FileName As String
    <DataMember()> Public TotalBytes As Long
End Class

The next step is to create our FileSendService class which descends from the DuplexService class as described above. We override the OnMessage method so that we can do custom processing based on the type of message sent as shown below.

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class FileSendService : DuplexService
{
    private List<SessionConnectionInfo> sessionConnections = new List<SessionConnectionInfo>();

    {...}

    protected override void OnMessage(string sessionId, DuplexMessage data)
    {
        if (data is HostSessionMessage)
            CreateHostSession(data as HostSessionMessage);
        else if (data is JoinSessionMessage)
            JoinSession(data as JoinSessionMessage);
        else if (data is FileBeginUploadMessage)
            StartSendFile(data as FileBeginUploadMessage);
        else
            SendMessage(data);
    }
    
}


        else if (data is JoinSessionMessage)
            JoinSession(data as JoinSessionMessage);
        else if (data is FileBeginUploadMessage)
            StartSendFile(data as FileBeginUploadMessage);
        else
            SendMessage(data);
    }
    
}


    }
    
}

Public Class FileSenderServiceFactory
    Inherits DuplexServiceFactory(Of FileSendService)
End Class

<AspNetCompatibilityRequirements(RequirementsMode := AspNetCompatibilityRequirementsMode.Allowed)> _
Public Class FileSendService
    Inherits DuplexService
    Private sessionConnections As New List(Of SessionConnectionInfo)()

    ...

    Protected Overrides Sub OnMessage(ByVal sessionId As String, ByVal data As DuplexMessage)
        If TypeOf data Is HostSessionMessage Then
            CreateHostSession(TryCast(data, HostSessionMessage))
        ElseIf TypeOf data Is JoinSessionMessage Then
            JoinSession(TryCast(data, JoinSessionMessage))
        ElseIf TypeOf data Is FileBeginUploadMessage Then
            StartSendFile(TryCast(data, FileBeginUploadMessage))
        Else
            SendMessage(data)
        End If
    End Sub

End Class

Since we are going to have to keep track of each host and user connected to that host, we will create a class, SessionConnectionInfo, to manage the relevant data. When a user decides to host a session, our OnMessage method will receive a HostSessionMessage which will scan our List<SessionConnectionInfo> to see if there is another host with the same username. If not found, a new SessionConnectionInfo object is created along with a randomly generated session key. When a user attempts to join a session, the OnMessage method will receive a JoinSessionMessage containing a session key and the name of the user joining the session. The service will search for a SessionConnectionInfo object containing that session key, and if found, establishes a connection between the two users.

public class SessionConnectionInfo
{
    public string HostUserName { get; set; }
    public string ConnectedUsername { get; private set; }
    public string SessionKey { get; set; }

    public string ConnectedUserInternalSession { get; private set; }
    public string HostInternalSession { get; set; }

    public bool UserConnected
    {
        get { return ConnectedUserInternalSession != string.Empty; }
    }

    public SessionConnectionInfo()
    {
        ConnectedUserInternalSession = string.Empty;
        ConnectedUsername = string.Empty;
    }

    ....
}

Public Class SessionConnectionInfo
    Private privateHostUserName As String
    Public Property HostUserName() As String
        Get
            Return privateHostUserName
        End Get
        Set(ByVal value As String)
            privateHostUserName = value
        End Set
    End Property
    Private privateConnectedUsername As String
    Public Property ConnectedUsername() As String
        Get
            Return privateConnectedUsername
        End Get
        Private Set(ByVal value As String)
            privateConnectedUsername = value
        End Set
    End Property
    Private privateSessionKey As String
    Public Property SessionKey() As String
        Get
            Return privateSessionKey
        End Get
        Set(ByVal value As String)
            privateSessionKey = value
        End Set
    End Property

    Private privateConnectedUserInternalSession As String
    Public Property ConnectedUserInternalSession() As String
        Get
            Return privateConnectedUserInternalSession
        End Get
        Private Set(ByVal value As String)
            privateConnectedUserInternalSession = value
        End Set
    End Property
    Private privateHostInternalSession As String
    Public Property HostInternalSession() As String
        Get
            Return privateHostInternalSession
        End Get
        Set(ByVal value As String)
            privateHostInternalSession = value
        End Set
    End Property

    Public ReadOnly Property UserConnected() As Boolean
        Get
            Return ConnectedUserInternalSession <> String.Empty
        End Get
    End Property

    Public Sub New()
        ConnectedUserInternalSession = String.Empty
        ConnectedUsername = String.Empty
    End Sub


End Class

Client Side

Now that we have the basic server side complete, we can add a service reference, and create the necessary service reference. Our service is hosted within ASP.NET via a simple xml file (see FileSendService.svc), which allows our Silverlight project to see it.

Be sure to mark the checkbox “Always generate message contracts”. Note that your web portion of the project must be compiled before you will be able to “Discover” the service.

We will now have access to the DuplexServiceClient class which will allow us to send and receive messages to the server. We instantiate the service as shown below. In order to do this, you must manually add a reference to System.ServiceModel.PollingDuplex assembly.

private DuplexServiceClient fileDuplexService;

private CustomBinding binding = new CustomBinding(
    new PollingDuplexBindingElement(),
    new BinaryMessageEncodingBindingElement(),
    new HttpTransportBindingElement());

public MainPage()
{
    InitializeComponent();
    fileDuplexService = new DuplexServiceClient(binding, new EndpointAddress("http://localhost:9797/FileSendService.svc"));
    ...
}

Private fileDuplexService As DuplexServiceClient

Private binding As New CustomBinding(New PollingDuplexBindingElement(), New BinaryMessageEncodingBindingElement(), New HttpTransportBindingElement())
    
Public Sub New()
    InitializeComponent()
    fileDuplexService = New DuplexServiceClient(binding, New EndpointAddress("http://localhost:9797/FileSendService.svc"))
End Sub

Note that at the time of this writing, adding the service reference does not properly create the ServiceReferences.ClientConfig file. This is the reason the above is done using code.

Now that our service is setup, we need to set it up to handle sending and receiving of messages. In order to send a message, we must first create a DupexMessage (or a descendant), and use the SendToServiceAsync method. This requires a SendToService object which will contain the message, and an optional userState object which we can use to tag the request. In this case, we will pass an enum value which represents the status of our upload. In the example below, when the user clicks on the send file button, after choosing the file, we open our file, and send a FileBeginUpload message which contains the file name and its size. Note that the server is set to deny any file that is greater than 20 million bytes.

private void btnSendFile_Click(object sender, RoutedEventArgs e)
 {
     OpenFileDialog openFileDialog = new OpenFileDialog();
     openFileDialog.Multiselect = false;
     openFileDialog.ShowDialog();
     if (openFileDialog.File != null)
     {
         fileToSend = openFileDialog.File.OpenRead();

         FileBeginUploadMessage fsm = new FileBeginUploadMessage();
         fsm.FileName = openFileDialog.File.Name;
         fsm.TotalBytes = openFileDialog.File.Length;
         fileDuplexService.SendToServiceAsync(new SendToService(fsm), FileSendState.FileStart);
         ....
     }
 }

Private Sub btnSendFile_Click(ByVal sender As Object, ByVal e As RoutedEventArgs)
    Dim openFileDialog As New OpenFileDialog()
    openFileDialog.Multiselect = False
    openFileDialog.ShowDialog()
    If openFileDialog.File IsNot Nothing Then
        fileToSend = openFileDialog.File.OpenRead()
        Dim fsm As New FileBeginUploadMessage()
        fsm.FileName = openFileDialog.File.Name
        fsm.TotalBytes = openFileDialog.File.Length
        totalBytesSent = 0
        fileDuplexService.SendToServiceAsync(New SendToService(fsm), FileSendState.FileStart)
        ....
    End If
End Sub

The service object fires two main events that need to be handled: SendToServiceCompleted and SendToClientReceived. SendToServiceCompleted is the event which is fired after the server acknowledges that it has received and processed a message that the client has sent. Once the server receives and processes the FileBeginUploadMessage, the event handler below will receive the results. In this case, if there isn’t an error, and the value of userState is FileSendState.FileStart, the method will send a FileTransferBytesMessage which will send the bytes from the file.

private void FileDuplexServiceSendToServiceCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
    if (e.Error == null)
    {
        {...}
        if ((FileSendState)e.UserState == FileSendState.FileEnd)
        {
            fileToSend.Close();
            fileProgress.Value = 100;
            return;
        }
        if ((FileSendState)e.UserState == FileSendState.FileStart || (FileSendState)e.UserState == FileSendState.FileContinue)
        {

            ...
            FileTransferBytesMessage fileMessage = new FileTransferBytesMessage();
            fileMessage.StartByte = totalBytesSent;
            fileMessage.EndFile = false;
            fileMessage.PacketSize = CHUNK;

            ...

            byte[] bytes = new byte[numBytesToRead];
            fileToSend.Read(bytes, 0, numBytesToRead);
            totalBytesSent += numBytesToRead;
            fileMessage.Bytes = bytes;

            if (fileMessage.EndFile)
                fileDuplexService.SendToServiceAsync(new SendToService(fileMessage), FileSendState.FileEnd);
            else
                fileDuplexService.SendToServiceAsync(new SendToService(fileMessage), FileSendState.FileContinue);
        }
    }
}

Private Sub FileDuplexServiceSendToServiceCompleted(ByVal sender As Object, ByVal e As System.ComponentModel.AsyncCompletedEventArgs)
    If e.Error Is Nothing Then
        If e.UserState Is Nothing Then
            Return
        End If
        If CType(e.UserState, FileSendState) = FileSendState.FileEnd Then
            fileToSend.Close()
            fileProgress.Value = 100
            Return
        End If
        If CType(e.UserState, FileSendState) = FileSendState.FileStart OrElse CType(e.UserState, FileSendState) = FileSendState.FileContinue Then
            ...
            Dim fileMessage As New FileTransferBytesMessage()
            fileMessage.StartByte = totalBytesSent
            fileMessage.EndFile = False
            fileMessage.PacketSize = CHUNK
            ...
            Dim bytes(numBytesToRead - 1) As Byte
            fileToSend.Read(bytes, 0, numBytesToRead)
            totalBytesSent += numBytesToRead
            fileMessage.Bytes = bytes

            If fileMessage.EndFile Then
                fileDuplexService.SendToServiceAsync(New SendToService(fileMessage), FileSendState.FileEnd)
            Else
                fileDuplexService.SendToServiceAsync(New SendToService(fileMessage), FileSendState.FileContinue)
            End If
        End If
    End If
End Sub

The SendToClientReceived event is fired when a message has been sent from the service. In this case, the client will check to see which message was received and process it accordingly. The method below shows that when a FileBeginUpload message is received, the user is prompted to accept or deny the file. If the user does not accept the file, a FileDenyMessage will be sent which will notify the sender to stop sending bytes. If accepted, file bytes are added to a buffer.

private void FileDuplexServiceSendToClientReceived(object sender, SendToClientReceivedEventArgs e)
{
    if (e.Error == null)
    {
        if (e.request.msg is ClientConnectedMessage)
        {
            ClientConnectedMessage msg = (ClientConnectedMessage)e.request.msg;
            AddMsgToListbox(msg.Username + " has just connected.");
            connectedTo = msg.Username;
            UIState = UIState.Chat;
        }

        else if (e.request.msg is HostSessionServerMessage)
        {
            HostSessionServerMessage hssm = e.request.msg as HostSessionServerMessage;
            if (hssm.Failed) {...}
            SessionCreated(hssm);

        }
        else if (e.request.msg is JoinSessionServerMessage)
        {
            JoinSessionServerMessage jssm = e.request.msg as JoinSessionServerMessage;
            if (jssm.Failed) {....}
            SessionJoined(jssm);
        }
        else if (e.request.msg is FileBeginUploadMessage )
        {
            FileBeginUploadMessage fsm = (FileBeginUploadMessage)e.request.msg;

            int sizeInKB = (int)fsm.TotalBytes / 1024;
            totalRevd = 0;
            if (MessageBox.Show(connectedTo + " would like to send you the file: " + fsm.FileName + ", Size: " + sizeInKB + ".  Would you like to receive this file?", "File Upload", MessageBoxButton.OKCancel) == MessageBoxResult.OK)
            {
                bytesReceived = new List<byte>((int)fsm.TotalBytes);
                fileNameReceiving = fsm.FileName;
                ....
            }
            else
            {
                fileDuplexService.SendToServiceAsync(new SendToService(new FileDenyMessage()));
            }
        }
        else if (e.request.msg is FileTransferBytesMessage)
        {
            if (bytesReceived == null)
                return;
            FileTransferBytesMessage fm = (FileTransferBytesMessage)e.request.msg;
            bytesReceived.AddRange(fm.Bytes);
            ....
        }
        else {....}

    }
}

Private Sub FileDuplexServiceSendToClientReceived(ByVal sender As Object, ByVal e As SendToClientReceivedEventArgs)
    If e.Error Is Nothing Then
        If TypeOf e.request.msg Is ClientConnectedMessage Then
            Dim msg As ClientConnectedMessage = CType(e.request.msg, ClientConnectedMessage)
            AddMsgToListbox(msg.Username & " has just connected.")
            connectedTo = msg.Username
            UIState = UIState.Chat

        ElseIf TypeOf e.request.msg Is HostSessionServerMessage Then
            Dim hssm As HostSessionServerMessage = TryCast(e.request.msg, HostSessionServerMessage)
            If hssm.Failed Then ...

            SessionCreated(hssm)

        ElseIf TypeOf e.request.msg Is JoinSessionServerMessage Then
            Dim jssm As JoinSessionServerMessage = TryCast(e.request.msg, JoinSessionServerMessage)
            If jssm.Failed Then ...

            SessionJoined(jssm)
        ElseIf TypeOf e.request.msg Is FileBeginUploadMessage Then
            Dim fsm As FileBeginUploadMessage = CType(e.request.msg, FileBeginUploadMessage)

            Dim sizeInKB As Integer = CInt(Fix(fsm.TotalBytes)) / 1024
            totalRevd = 0
            If MessageBox.Show(connectedTo & " would like to send you the file: " & fsm.FileName & ", Size: " & sizeInKB & " KB.  Would you like to receive this file?", "File Upload", MessageBoxButton.OKCancel) = MessageBoxResult.OK Then
                bytesReceived = New List(Of Byte)(CInt(Fix(fsm.TotalBytes)))
                fileNameReceiving = fsm.FileName
                ...
            Else
                fileDuplexService.SendToServiceAsync(New SendToService(New FileDenyMessage()))
            End If
        ElseIf TypeOf e.request.msg Is FileTransferBytesMessage Then
            If bytesReceived Is Nothing Then
                Return
            End If
            Dim fm As FileTransferBytesMessage = CType(e.request.msg, FileTransferBytesMessage)
            bytesReceived.AddRange(fm.Bytes)
            ...
        ElseIf 
          ...

    End If
End Sub

When the file is complete, the user will be presented with two buttons. One button will allow the user to save, and the other will allow the user to discard/ignore the file. If the user chooses to save, a SaveFileDialog is created, and we use the filename to assign the default extension and filter so that when the user saves the file, no extension needs to be typed in. Once the dialog appears and the user enters the file name, the bytes will be written to disk. Finally we send a message back to the server which will notify the sender that user has done something with the file. During file send operations, the Send File button is disabled, and becomes re-enabled once a file has been received, cancelled, or denied.

private void btnSaveFile_Click(object sender, RoutedEventArgs e)
{
    SaveFileDialog sfd = new SaveFileDialog();
    string extension = GetExtension(fileNameReceiving);
    sfd.DefaultExt = extension;
    sfd.Filter = extension +  " Files|" + extension;

    if (sfd.ShowDialog() == true)
    {
        using (Stream fsx = sfd.OpenFile())
        {
            byte[] fBytes = bytesReceived.ToArray();
            fsx.Write(fBytes, 0, fBytes.Length);
            fsx.Close();
        }

        fileDuplexService.SendToServiceAsync(new SendToService(new FileReceivedMessage()));
        UIState = UIState.Chat;
        btnSendFile.IsEnabled = true;
    }
}

Private Sub btnSaveFile_Click(ByVal sender As Object, ByVal e As RoutedEventArgs)
    Dim sfd As New SaveFileDialog()
    Dim extension As String = GetExtension(fileNameReceiving)
    sfd.DefaultExt = extension
    sfd.Filter = extension & " Files|" & extension

    If sfd.ShowDialog() = True Then
        Using fsx As Stream = sfd.OpenFile()
            Dim fBytes() As Byte = bytesReceived.ToArray()
            fsx.Write(fBytes, 0, fBytes.Length)
            fsx.Close()
        End Using

        fileDuplexService.SendToServiceAsync(New SendToService(New FileReceivedMessage()))
        UIState = UIState.Chat
        btnSendFile.IsEnabled = True
    End If
End Sub

Conclusion

Overall, the application does what I set out for it to. It allows a user to send a file to another user and even does basic chat. There is definitely a lot of room for improvement, and some general additions that can make this more robust. Right now we keep a list of connections, but never do basic pinging to see if the clients are still there. The only way to know if a client has disconnected is when a message fails, or when the user decides to hit the disconnect button. Better maintenance of this list is needed. The file bytes are basically being stored in memory, which is why i set a file size limit, but I’m sure with things like Silverlight local storage, we can increase the limit.

Thanks

I would like to thank Brian Peek for taking the time to review my article and test the code.

Additional Notes

The ASP.NET project requires a file called System.ServiceModel.PollingDuplex.dll to be added as a reference. Somewhere between Silverlight 3 Beta and Silverlight 3 RTW, the file was no longer available via the basic add .NET reference link. In order to add the file, I had to browse for it in %Program Files%\Microsoft SDKs\Silverlight\v3.0\Libraries\Server. For 64 bit users, it will be the Program Files (x86) folder.

For ease of use, I used a static port (9797) for this project. When deploying to a server, you must rename all references of http://localhost:9797 in the Silverlight project. This will remain true until the config file is supported.

The user interface uses the Silverlight Toolkit TwilightBlue theme. For more information about this, see http://www.codeplex.com/Silverlight

Adding Sidetone to Skype

Coding4Fun — Wed, 10 Jun 2009 00:44:00 GMT

Ever use a headset with Skype – and were frustrated that it was too quiet? This article shows how to add the sound of your own voice to the headset and not feel exhausted from shouting to be heard.

Compiled version: Download
Source Code: Download C#
Difficulty: Intermediate
Time Required: 1-3 hours
Cost: free
Software Needed: Skype (Sype4Com), DirectX, Visual Basic or Visual C# Express
Hardware: none

Skype: I find your lack of feedback disturbing.

This article will discuss how to build a software tool that makes it easier to talk into Skype with a headset.

Recently my family began experimenting with Skype, but we found that talking over Skype can be exhausting with headphones. After doing a little research, I learned that the problem was that we only heard the other party. You’d think that this is a good thing, but we have a social brain that works hard to gauge our behavior and adjust.

The answer is to feed a little bit of the microphone back into the earpiece, so that our brain knows how loud we're talking. This is called side tone in the telecommunication industry. Without it, we talk louder and louder until we’re sure that we’ll be heard.

I couldn’t find a Skype plug-in to do this… but I had just read a Coding4Fun article, by Mark Heath, about adding audio effects to Skype. I decided I was going to write a tool to add this feedback.

How to use the tool

First, let’s take a look at how to use the application. Once it is running, there will be a microphone icon in the lower right hand corner of the screen:

Figure 1: Icon in the system notification tray

Clicking on it, will get the application control window:

Figure 2: The application's controls

Let’s look at the side tone controls:

The “Side Tones” check box enables or disables the side tone. If you have headphones, you'll want this on; if you have loud speakers, you'll want it off.
The slider controls the volume of the microphone in the headset. You can change it while talking. The volume will vary with microphone and headset.

You can do a “sound check” to see if the feedback is working by clicking on the Sound Check button, and adjusting the volume. I found that the volume setting that works best in a conversation is much, much lower than what works in a sound check.

Next, let’s look at the AGC (“Automatic Gain Control”) section. When making a call, the software can automatically adjust how loud you sound to the other party:

When “Low Pass” is checked, the headphone feedback is at 44100 samples/sec. The microphone sound is filtered to keep only the sounds below 8 Khz, converted to 16000/samples per second and sent to Skype. When it is not checked, headphone feedback is at 16000 samples/sec, and sent to Skype without the low pass filter.
When "Skype AutoGain" is checked, it signals to Skype that Skype can use its own algorithm. When clear, Skype is told not to apply any adjustments.
When "AutoGain" is checked, the custom Automatic Gain Control algorithm is used.

The Automatic Gain Control has three sliders:

The “Cutoff” slider controls the distinction between background noise and conversation. Sound below this level is cutoff and silence is sent to Skype. This will vary with microphone – more sensitive (expensive) microphones will pick more background noise and be better with a higher setting.
The “Normal” slider controls the volume when you are talking normally. The Gain Control tries to raise the volume to this level.
The “Loud” slider controls the volume when you talk exceptionally loud. This rarely happens, but when you do talk louder than the Normal level, the Gain Control tries to adjust the volume to this level.

How the Software Works

I originally started this project by creating an effect for the Skype Voice Changer. My plan was to open a WaveStream, begin playing it on the headphones and copy the microphone stream to it.

I quickly found that this was not the way to add feedback. The underlying “WaveOut” system had a huge latency: Everything I heard in the headphones was at least a second or more behind what I was saying. This made it even harder to talk than before, and I had to abandon it.

While researching the problem I found a DirectSound code sample that I could modify into doing what I wanted. (This initial prototype didn’t coordinate with Skype – it attached to the microphone and copied the sounds to the output, at a lower volume. But it was on always on.) The sound in the headphones is still slightly behind (the microphone) but is barely perceptible, and we’ll muffle it a bit more to make it less distinguishable.

From here on I shall describe the major – or technically interesting – components of the program. We’ll look at:

DirectSound and Circular buffers
Sample Window and Sizing the buffers
Using WaitHandle’s to Synchronize with DirectSound
IIR Filters
Automatic Gain Control
Estimating loudness

Note: I won’t be describing how to connect to Skype. Mark Heath’s description is very good.

DirectSound and Circular Buffers

The DirectSound code is in AudioLoop.cs. The module sets DirectSound to capture sound from the default recording device at 16bits / sample at either 16000 or 44100 samples per second. The capture and playback buffers are configured in “looping” mode to act as circular buffers. The capture buffer eventually overwrites samples, and we’ll lose them if we don’t act fast enough; if we don’t update the playback buffer, it will repeat the same sound over and over.

The StartMicrophone() procedure sets up the capture and playback buffers. Then it creates a thread to do the work. The thread is at a high priority so that if the OS has a choice between (say) email or processing sound, it does the sound.

The StopMicrophone() procedure stops the worker thread and cleans up the resources.

The software processes a fixed number of samples at a time, called the “sample window.” The buffers are several times the size of sample window, so that the system can keep capturing and playing while the software is processing them. The sound processing loop is the heart of the application:

Wait for the next sample window to be ready
Copy the samples from the capture buffer
Process the samples and send the results to the playback buffer
Process the samples and sends the results to Skype

while (Runnable)
{
    for (int I = 0, offset = 0; Runnable && I < _bufferPositions; I++, offset += bufferSize)
    {
        // Wait for the sample areas to be ready
        notificationEvent[I].WaitOne(Timeout.Infinite, true);

        // Get the sound samples
        byte[] buffer = (byte[]) captureBuffer.Read(offset, typeof (byte), LockFlag.None, bufferSize);

        // Convert samples to 16bit PCM 
        for (int L = buffer.Length, J = 10, K = 0; K < L; K += 2)
            PCM16Buffer[J++] = (Int16) ((UInt16) buffer[K] | (((UInt16) buffer[K + 1]) << 8));

        // Play them out to the ear, if applicable
        if (null != playbackBuffer)
        {
            // Perform a low pass filter to "muffle" the sound
            Butterworth(PCM16Buffer, 10, LPSample, Coefs);

            // put the muffled sample into the output buffer
            // -- The lock flag seems to work, but others may work too
            playbackBuffer.Write(Idx, LPSample, LockFlag.None);
            Idx += buffer.Length;
            if (Idx >= 4*bufferSize)
                Idx -= 4*bufferSize;
            if (!playing)
            {
                playbackBuffer.Volume = _Volume;
                playbackBuffer.Play(0, BufferPlayFlags.Looping);
                playing = true;
            }
        }

        // Process the sound and deliver it to Skype
        if (null != outStream)
        {
            int L = AGC.Process(PCM16Buffer, 10, buffer);
            if (0 != L)
                outStream.BeginSend(buffer, 0, L, SocketFlags.None, SendCallback, null);
            // Note: could send out pink noise if L == 0
        }

        // Move the sliding window of the previous 10 samples into the start
        // of the PCM16Buffer
        for (int K = 0, J = PCM16Buffer.Length - 10; K < 10; K++, J++)
            PCM16Buffer[K] = PCM16Buffer[J];
    }
}

Note the “for” loop at the bottom of the code. This preserves the last 10 incoming samples at the start of the buffer. This is needed to make the sound processing smooth, and will be discussed a bit later.

Sample Window and Sizing the Buffers

How big should the sample window be? This is bit of a trade off in responsiveness and design complexity.

I chose a window big enough to hold 10 milliseconds of sound. Since the ear is sensitive sound to delays of even 30 milliseconds, I cut this done so that a delay wouldn’t be perceptible. (When I tried a 50 millisecond window, my voice came out the headphone sounding like an echo... and I found myself talking slower and slower.) The sample window could be made smaller, but I am sure that there is a point where the OS won’t schedule the audio loop to wake-n-run more frequently. And, as the sample window gets smaller, the processing may drop in quality, because it doesn’t have enough to work with.

The capture buffer is 8 times the size of the sample window. This ratio is arbitrary, but I wanted the buffer to be about an order of magnitude larger. My rationale is that if the processing falls behind, the sound – for the Skype call – won’t be dropped. I feel that it is more important to preserve sound quality for the other party than to preserve the quality of feedback.

The playback buffer is four times the sample window. I wanted it small, so that if the processing fell behind, the replaying of a sound will seem to be a continuation of a current sound.

When writing the sound to the playback buffer, we have to track where in the buffer to put the samples. I tried to use GetCurrentPosition() to find where to write to next into the playback buffer; this created terrible sound. Instead, the software uses a local variable to track where to write next.

DirectSound and Notifications

How do we keep in sync with the sound capture – how do we know when a sample buffer is ready?

The application gives a table of buffer indices and WaitHandle’s to DirectSound. When the capture buffer’s write index reaches one of those indices, it signals the corresponding WaitHandle. The worker thread cycles performs a WaitOne() one each of the WaitHandle’s, one at a time. As a convenience, we use a specific kind of WaitHandle called AutoResetEvent. This type of WaitHandle sets itself back to a “wait” state once WaitOne() returns.

If the thread has gotten behind, the WaitOne will return immediately, the loop processes the sample, and begins to catch up with the work.

We must use a separate AutoResetEvent for each of the 8 capture windows. The AutoResetEvent doesn’t tell us if it was signaled multiple times. If only one AutoResetEvent handle were used, it wouldn’t know that two (or more) sample windows were ready. Instead, it would process just one, falling further behind, adding latency. This would happen randomly overtime, and be hard to test consistently.

IIR: Infinite Impulse Response Filters

This project came together so quickly, so easy – once I found the right approach – that I couldn't resist getting fancy. I added a low-pass filter to muffle the feedback a little. And I added automatic gain control, as an experimental option.

For both of these I used a filtering algorithm called “IIR” (this stands for Infinite Impulse Response – but that term is a confusing mouthful, so let’s just call it IIR). IIR is a special purpose virtual machine. Low-pass filters, high-pass filters, combinations of those filters, and even equalizers, can be specified, and use very specific techniques (like a compiler) to convert them into an IIR implementation.

(You could, instead, “compile” the filters to be the resistor values to use in a hardware circuit. That’s programming in solder!)

The machine code for these IIR virtual machines is just two list of coefficients, called A & B. The software emulator is code that looks like the following bit of code:

Out[0] = Sample[0];
Out[1] = Sample[1];
for (int Idx = 2; Idx < N; Idx++)
{
    Out[Idx] =
    B0 * Sample[Idx]
 + B1 * Sample[Idx - 1]
 + B2 * Sample[Idx - 2]
        // … more like this …
        // Next, the feed back
 - A1 * Out[Idx - 1]
 - A2 * Out[Idx - 2]
        // … more like this …
 ;
}

IIRs are easy to implement - and take less CPU power than other methods. But sometimes they sound poor; if they sound too bad, you’ll want to use a different technique. I found that the low-pass filters in this project work will for some microphones, and add a slight crackle to others.

Example Low Pass Filter

For the low pass filter to create the muffling, I used a Butterworth filter, using the code below. It takes a buffer of signed, 16-bit samples, and then converts the 16-bit values into a byte array suitable for the sound buffer.

The filter code is a bit different than the example code in the previous section. Most of the differences are for speed.

This code doesn’t use a buffer for the old values, instead uses separate variables for the elements of the buffer. It uses I_0,I_1,I_2 instead of Sample[Idx], Sample[Idx-1], and Sample[Idx-2]. It also uses O_1, and O_2 instead of Out[Idx-1] and Out[Idx-2].
The A and B coefficients are put into a single array. It also adds two of the sample input values, and is missing a coefficient; this is because the B0 and B2 coefficients are always the same for this kind of filter.

There is one difference that is not for speed. These are tricks done to make the filter smooth, and needed because the sample window is so small. They preserve the state of filter. If we didn’t preserve them, the filter would be starting and stopping so frequently that it would add distracting clicks to the output sound. The filters performance would be weakened, because the sample window isn’t big enough to hold sounds lower than (about) 200 Hz. Preserving these values, the filter isn’t starting and stopping, and doesn’t really know about the sample window. All of the IIR filters in this program use similar techniques.

O_1, O_2 are explicitly preserved across calls by being stored in class variables
I_1 and I_2 are preserved by the audio loop (remember the warning about preserving 10 samples at the bottom of the loop?) The audio loop preserves the last 10 samples at the start of InBuffer. When this procedure is called, it retrieves the last two samples.

static double O_1 = 0.0, O_2=0.0;

static void Butterworth(Int16[] InBuffer, int Ofs, byte[] OutBuffer, double[] Coefs)
{
  double C0=Coefs[0], C1=Coefs[1], C2 = Coefs[2], C3=Coefs[3];
  double I_1=InBuffer[Ofs-1], I_2= InBuffer[Ofs-2];
  for (int L = InBuffer . Length, J=0, I = Ofs; I < L; I++)
  {
     double I_0 = InBuffer[I];

     // Filter the samples
     double A = (I_0 + I_2) * C0 + I_1 * C1;
     I_2 = I_1;
     I_1 = I_0;

     A = A - O_1 * C2 - O_2 * C3;
     O_2 = O_1;
     O_1 = A;

     // Convert it back to 16 bit
     Int16 S;
     if (A < -32767) S = -32767;
     if (A > 32767)  S = 32767;
     else S = (Int16) A;

     // Store it
     OutBuffer[J++] = (byte)(S & 0xFF);
     OutBuffer[J++] = (byte)(((UInt16)S >> 8) & 0xFF);
  }
}

Automatic Gain Control

I decided next to tackle a problem where my wife's voice did not carry well on calls. This happens a lot to her with cell phones – and answering machines. I was pretty sure that the problem was poor automatic-gain-control (AGC). The typical amplifier in a headset (and in Skype) estimates how loud our voice is, then increases – or decreases – the volume to a reasonable level. It was deciding that my wife's voice was background noise, and cutting her off.

I chose to write an alternate gain control that amplified the sound and passed it to Skype. That way we'd have four to choose from: The one built into the Microphone, the Soundcard’s, Mine, and Skype’s. (To be fair, these automatic gain controls work well in most cases).

The main portion of the gain control is implemented in the file GainControl.cs. The control algorithm is:

Calculate (or estimate) how loud we are currently talking (using the Analyze() procedure)
If the loudness is very low, no one is talking… so set the output to zero. (Without this step, the volume of noise and hum would be cranked up)
Otherwise, compute the guess-gain by dividing how loud the sound of our voice should be by how loud it currently is
Compute the gain (called MaxGain) at which the sound will start clipping. If the guess-gain is louder than this, reduce it to MaxGain.
The software adjusts the gain for a gentle transition – especially in the case when we go from absolute quiet, to the start of talking. It does this by tracking the gain (called PrevGain) used in previous sample and the current one.
Multiple all the samples by this gain value.
If the sample rate is greater than 16000 samples/sec
1. Do a low pass filter at 8 khz (again in IIR form). This helps prevent artifacts from down sampling
2. Resample the sound to 16000

The portion of code that calculates the gain looks like (CutOff_dB, LowGain_dB, and TgtGain_dB are the three slider values):

if (!AutoGain)
    Gain = 1.0;
else
{
    double MaxGain;
    double dB = Analyze(InBuffer, Ofs, out MaxGain);

    if (dB < CutOff_dB)
        Gain = 0.0;
    else if (dB < LowGain_dB + 4.0)
    {
        Gain = Math.Exp((LowGain_dB - dB) * db2log);
    }
    else
    {
        Gain = Math.Exp((TgtGain_dB - dB) * db2log);
    }
    Gain = (0.4 * Gain + 0.6 * PrevGain);
    if (Gain > MaxGain)
        Gain = MaxGain;
    PrevGain = Gain;
}

// Skip further process if there is silence
if (0.0 == Gain)
{
    return 0;
}

If you look at the code, you’ll see that we don’t compare directly with LowGain_db; rather we compare the estimate volume with LowGain_db+4. This gives a little “hysteresis” – if we raise our voice momentarily, the software won’t suddenly make it the highest possible volume. Instead, the software lowers the volume a little bit.

When the software changes the sample rate, it basically needs to know how many input samples to skip. At the start of a call, the software computes this, calling it InInc:

// Calculate how we resample to 16Khz
InInc = (int)(1024.0 * SampleRate / 16000.0);

The process of applying the gain adjustment, performing a low pass filter and re-sampling is below:

int NumSamples = InBuffer.Length;
int End = OutBuffer.Length;
int NextIdxForOut = -InInc;
int OutIdx = 0;
for (int I = Ofs; OutIdx < End && I < NumSamples; I++)
{
    // Retrieve the sample
    Int16 S = InBuffer[I];
    double I_0 = S;

    // Apply Gain
    I_0 *= Gain;

    // 8khz low pass filter 
    if (DoLP)
    {
        // Simple Butterworth 8KHz low-pass filter
        double A = (I_0 + DS_I_2) * LP[0] + DS_I_1 * LP[1];
        DS_I_2 = DS_I_1;
        DS_I_1 = I_0;
        I_0 = A - DS_O_1 * LP[2] - DS_O_2 * LP[3];
        DS_O_2 = DS_O_1;
        DS_O_1 = I_0;

        // Change sample rate
        int Tmp = NextIdxForOut / 1024;
        if (I < Tmp)
            continue;
        NextIdxForOut += InInc;
    }

    // Convert it back to 16 bit
    S = (Int16)I_0;

    // Store it
    OutBuffer[OutIdx++] = (byte)(S & 0xFF);
    OutBuffer[OutIdx++] = (byte)(((UInt16)S >> 8) & 0xFF);
}

Estimating Volume

How loud “it should be” is controlled by a slider on the screen. The software estimates how loud the sound is by using an algorithm devised by David Robinson that takes into account how it sounds to a person. This way we can increase the gain on hard to hear sounds, and reduce the gain on sounds that a person is very sensitive to.

The loudness estimator, implemented in the Analysis procedure in GainAnalysis.cs, uses the following algorithm:

Uses a combination of two (IIR) filters to make the sound to reflect how our ears hear it.
Compute the Mean-Square value of the filtered sound (called MS)
Track the last 750 ms of these values.
Make a sorted copy of these.
Retrieve the first non-zero value at least 95% of the way into the buffer. This is so that we don’t take the loudest sample and assume that is how the person is talking.
If MS (the value computed in step 2) is much quieter, use that one instead
Convert this value into dB scale by performing a logarithm on it.

The code to “normalize” the sound into how a person hears it is below. Along the way it computes the square of the samples (used in step 2). Like the earlier IIR filters, these preserves their variables across the calls. The first IIR is a yulewalk filter, but it preserves it old intermediary values in an array. Like the trick in AudioLoop, where we copy the last 10 samples into the start of the current buffer, the analysis procedure copies the last 10 immediate values into the start of YuleTmp array.

The output of the yulewalk filter is feed into a 150Hz high pass filter. It is essentially the same as the low-pass filter described earlier.

for (int L = Samples.Length, N = Ofs; N < L; N++)
{
    int _V = Samples[N];
    double V = _V;
    if (_V > MaxSample) MaxSample = _V;
    if (-_V > MaxSample) MaxSample = -_V;

    // Perform yulewalk filter
    double S = V * YuleCoefs[0];
    for (int J = N - 1, I = 1; I < 11; I++, J--)
        S += Samples[J] * YuleCoefs[I];
    for (int J = N - 1, I = 11; I < 21; I++, J--)
        S -= YuleTmp[J] * YuleCoefs[I];

    // Store for the feedback into the next stage of the yule walk
    YuleTmp[N] = S;


    // Perform butterworth high-pass filter stage, using S as an input
    double Accum =
       (S + GA_I_2) * HPCoefs[0]
       + GA_I_1 * HPCoefs[1];
    GA_I_2 = GA_I_1;
    GA_I_1 = S;
    Accum = Accum - GA_O_1 * HPCoefs[2] - GA_O_2 * HPCoefs[3];

    GA_O_2 = GA_O_1;
    GA_O_1 = Accum;

    // The square of the filtered results
    Sum += Accum * Accum;
}

// Copy the intermediate yulewalk state for the next
// (this is needed since we are looking at a fairly small time window)
for (int I = 0, J = YuleTmp.Length - 10; I < 10; I++)
    YuleTmp[I] = YuleTmp[J];

The mean-squared is computed:

// The mean square of the filtered results
double MS = Sum / NumSamples;

Tracking the last 750ms of samples is a simple matter of putting it into a circular queue:

MSQueue[QIdx++] = MS;
if (QIdx >= MSQueue . Length)
   QIdx = 0;

Next, is the code to finding the first non-zero value 95% of the way into the buffer. It is a straightforward copy-the-array, sort it, and fetch:

Array.Copy(MSQueue, SortedQ, MSQueue.Length);
Array.Sort(SortedQ);

// Return the 95% 
double X = SortedQ[Q95Idx]; 
for (int I = Q95Idx +1; X < 400.0 && I < SortedQ . Length; I++)
      X = SortedQ[I];

Next, override this value if the current sample window is very, very quiet – that is, the user stopped talking. (If we don’t do this, we’ll amplify background noise between words)

if (MS < X * 0.40 && MS < 12800.0)
   X = MS;

Finally, convert the result into decibels (or a reasonable approximation of a decibel)

return 10.0 * Math . Log10 (X * 0.5 + double . Epsilon);

Note: The logarithm function takes a positive, non-zero floating point number. However, the value we pass to it can be zero; if we pass a zero, though, the Logarithm function would return a bad value. The simplest thing to do is check to see if the value we are passing is “zero” and not call Logarithm. However, I learned a long time ago to just add “epsilon” to the value to whatever we pass. This can really improve performance on number crunching.

Conclusion

This concludes how to add a little bit feedback and fancy amplification to you Skype phone calls.

If you want to try this out, the download link for the source code is at the top of the article!

If you’d like to experiment further, here are ideas of what can be done:

DirectSound has echo cancellation and noise suppression. These seem desirable, esp. if you wanted to try making your own speakerphone. I was not able to get them to work and I would love to learn how.
I’m sure it is possible to trim even more latency off of the side-tone playback, and I would be interested in learning better techniques to do so.
Another would be to create the ideal equalizer from Robinson’s Equal Loudness model, and use that filter and amplify the sound.
It might be useful for the other party to control the settings, so that they decide when your voice has the right volume.

Resources and References

David Robinson (while doing his PhD at University of Essex on human hearing). His algorithm is intended to find the overall loudness of an album. He includes MATLAB scripts to compute coefficients at custom sample rates, as well as some pre-computed coefficients.
Mark Heath’s Skype Voice Changer
Derek Pierson’s DirectX tutorial

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded software. Before that he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.

Silverlight Dynamic Video Puzzle

Coding4Fun — Tue, 22 Jan 2008 22:02:00 GMT

My blog title says it all, Monkey see, Monkey Build. I saw Microsoft Surface's video puzzle and I needed to build it. I took this opportunity to play with Microsoft's new technology, Silverlight and build my own puzzle game. I couldn't find anything out on the Internet regarding dynamic video creation with Silverlight so I took it upon myself to do this. The screen shots in this demo will be in c#, however, everything should hold true for VB.

Clint Rutkas - Academic Developer Evangelist - Microsoft
Monkey see, Monkey Build

Difficulty: Intermediate
Time Required: 6-10 hours
Cost: Free
Software: Visual Studio Express, Silverlight SDK and Runtime for ~~1.1~~ 2.0 beta2
Download: ~~Download c#~~ - ~~Download VB~~ Updated to Silverlight 2.0 Beta - Download c#

Updates:

Fixed the Silverlight.Js, used the new version that is included with the 1.1 Refresh SDK so the proper installer will prompt now. Downloads are corrected also too.
6/20/2008 - Verified the solution works with Silverlight 2.0beta2.

Spec's, Requirements and headaches

So I had a few mental requirements for the application to "entertain" myself.

Be able to rotate the video blocks
Be able to translate
Be able to increase the difficulty of the puzzle on the fly.
Zero interaction with the keyboard

Simple, right? You want to see a demo too see some awesomeness? No problem, head over to my site and see it in person.

Microsoft Surface Video Puzzle

Clint's Silverlight Video Puzzle

To the Internet Batman!

To help save time building this, I first when out and attempting to see if anything has been done anything close to this before. I found

Also I need 2 small controls which I found out are in the Silverlight SDK. I needed horizontal slider controls for this app. While I know a text box control would have worked just as well, see requirement 4.

On the Silverlight community site, there is also a very nice demo showing the photo application in Surface with source code. This is where I learned the majority of what I needed. This application was very close to where I needed to go with mine. However, while it was close, it wasn't perfect. Parts of the photo application code are pretty much copied. This isn't the best way of doing this, but it is one way of doing it.

Toolbelt: check, Keyboard: check, Band-aids: check

So on to building it. I first started off with a new project and some XAML.

After clicking "OK", I get a very nice blank project.

We'll want to add an additional XAML object for the video blocks. Go to "Project -> Add New Item", select Silverlight Page, name it "VideoBlock.xaml" and click "OK".

So now we have everything we need for the most part, grab any video you have, if you don't have one, go to a site like TeamXbox and grab one from there. I used Windows Movie Maker and grabbed a snippet from The Office (best show ever).

In addition to all these files, we need to add in a reference to the Silverlight SDK to get a hold of the Slider controls. Go to "Project -> Add Reference", and add in the Silverlight.Samples.Controls.dll.

While we have a reference in the application, we still need the XAML to know about this also. So we'll add a line to link this in. It is the xmlns line that I bolded, XML name space, real clever, eh? We'll also change the background color to a gray so we can see it more easy.

<Canvas x:Name="parentCanvas"
        xmlns="http://schemas.microsoft.com/client/2007" 
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
        Loaded="Page_Loaded" 
        x:Class="c4f_SilverlightVideoDemo.VideoBlock;assembly=ClientBin/c4f_SilverlightVideoDemo.dll"
        xmlns:uicontrol="clr-namespace:Silverlight.Samples.Controls;assembly=ClientBin/Silverlight.Samples.Controls.dll"
        Background="#CCCCCC" Width="640" Height="480">
</Canvas>

Silverlight 101

So in Silverlight, you have brushes which can "painted" onto objects. One of these brushes is called the VideoBrush which I'll talk about later on. For playing videos, one would use the MediaElement. Lets add this to our canvas.

<MediaElement x:Name="media" Source="theOffice.wmv" />

Now if we run the application at this point, we'll get the video playing. Lets sit back with some popcorn and enjoy this victory for a second or two. Now that we have this, lets hide it since this isn't what we want. We do that by adding in an XML attribute Opacity and setting it to 0.

On top of that element, lets add on some TextBlock elements and the sliders.

<Canvas x:Name="parentCanvas"
        xmlns="http://schemas.microsoft.com/client/2007" 
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
        Loaded="Page_Loaded" 
        x:Class="VideoCreatedObjectTest.Page;assembly=ClientBin/VideoCreatedObjectTest.dll"
        xmlns:uicontrol="clr-namespace:Silverlight.Samples.Controls;assembly=ClientBin/Silverlight.Samples.Controls.dll" 
        Background="#CCCCCC" >
    <MediaElement x:Name="media" Source="theOffice.wmv" Opacity="0" />
    
    <TextBlock x:Name="sliderX" Canvas.Top="5" Canvas.Left="5" Text="X (01):" />
    <TextBlock x:Name="sliderY"  Canvas.Top="21" Canvas.Left="5" Text="Y (01):" />
    
    <uicontrol:Slider x:Name="hSliderX" Canvas.Top="8" Canvas.Left="55" />
    <uicontrol:Slider x:Name="hSliderY" Canvas.Top="25" Canvas.Left="55" />
</Canvas>

So now when we run the application, we get a blank screen but we can hear the audio from the video.

Super exciting I know. Now if you resize the browser, you'll notice also the canvas isn't resizing automatically. Lets fix that.

So in the code behind lets add in a new event on the BrowserHost object with the Resize event. We'll attach the Browser_Resize event to it.

private void BrowserHost_Resize(object sender, EventArgs e)
{
    // Set size to host size
    Width = BrowserHost.ActualWidth;
    Height = BrowserHost.ActualHeight;
}

VB.Net

Private Sub BrowserHost_Resize(ByVal sender As Object, ByVal e As EventArgs)
    ' Set size to host size
    Width = BrowserHost.ActualWidth
    Height = BrowserHost.ActualHeight
End Sub

Video as Paint?

So now that we have that small part done, lets get into more of the nitty gritty with video. As I said earlier, you'll use a VideoBrush to paint on the video.

We'll use the brush as the Fill for the rectangle we'll be using to show off the clip of the video we want.

Still not moving? Try explosives?

While doing this project, I discovered an interesting thing when using the VideoBrush and dynamically appending items on. Having items in the XAML rather than programmatically creating everything caused it not to work properly. Nothing would appear.

Throwback to how UI's use to be done

The XAML for this is a simplified version of the Surface demo Photo.XAML. You have a 3 Rectangles, 1 for the video, 1 for translation and 1 for rotation. Simple, no? Doing all this by hand gets a bit annoying since you need to nest everything properly.

Here is the more interesting VideoBrush code. You need the name from the MediaElement along with the how much of an offset you want the video. Since your moving the "camera" in, this value needs to be negative.

public delegate void SetActiveVideo(VideoBlock videoBlock, ActionType actionType, Point photoCenter, Point lastPosition);
public VideoBlock(string mediaName, int offsetX, int offsetY, int rectWidth, int rectHeight, SetActiveVideo functionPointer)
{
    // additional code here
    Rectangle video = new Rectangle();
    VideoBrush vb = new VideoBrush();
    TranslateTransform videoTransform = new TranslateTransform();
    videoTransform.X = -1 * offsetX;
    videoTransform.Y = -1 * offsetY;

    vb.SourceName = mediaName;
    vb.Transform = videoTransform;
    vb.Stretch = Stretch.None;
    vb.AlignmentX = AlignmentX.Left;
    vb.AlignmentY = AlignmentY.Top;

    video.Width = Width;
    video.Height = Height;
    video.Fill = vb;
    // additional code here
}

VB.Net

Public Delegate Sub SetActiveVideo(ByVal videoBlock As VideoBlock, ByVal actionType As ActionType, ByVal photoCenter As Point, ByVal lastPosition As Point)
Public Sub VideoBlock(ByVal mediaName As String, ByVal offsetX As Integer, ByVal offsetY As Integer, ByVal rectWidth As Integer, ByVal rectHeight As Integer, ByVal functionPointer As SetActiveVideo)
      ' additional code here
      Dim video As Rectangle = New Rectangle
      Dim vb As VideoBrush = New VideoBrush
      Dim videoTransform As TranslateTransform = New TranslateTransform
      videoTransform.X = ((1 * offsetX)  * -1)
      videoTransform.Y = ((1 * offsetY)  * -1)
      vb.SourceName = mediaName
      vb.Transform = videoTransform
      vb.Stretch = Stretch.None
      vb.AlignmentX = AlignmentX.Left
      vb.AlignmentY = AlignmentY.Top
      video.Width = Width
      video.Height = Height
      video.Fill = vb
      ' additional code here
End Sub

There is a mouse on my object!

So now we have our video block element prepped and ready. There are additional events added in, but they are fairly boiler plate. When looking at this code, you see I pass in a delegate, a function pointer if you will. Why do I do this? This is how the mouse position is calculated. I bet this could be redone better, but this is how I did it. The parent element, the canvas, has the mouse position I need to calculate proper translation and rotation, however, mouse events on the VideoBlock object will give me the mouse position only on that element. So I may be at point 150, 150 on the screen, on the VideoBlock object, I'm actually at 10,15.

Additionally, this approach allows me to move the VideoBlock to the top of the objects. Once again, this could be done other ways, this is how I chose to do it.

So on all my mouse related all boils down to doing the exact same function. Since events are bubbled, a mouse movement on a VideoBlock is also a mouse movement on the root canvas.

private void root_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.Selecting, e);
}

private void translateControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.Moving, e);
}

private void rotateScaleControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.RotatingScaling, e);
}

private void HandleMouseLeftButtonDown(ActionType actionType, MouseEventArgs e)
{
    if (functionPointer != null)
        functionPointer(this, actionType,
            new Point(translateTransform.X + rotateTransform.CenterX,
                  translateTransform.Y + rotateTransform.CenterY), e.GetPosition(null));
}

VB.Net

    Private Sub root_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Selecting, e)
    End Sub
    
    Private Sub translateControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Moving, e)
    End Sub
    
    Private Sub rotateScaleControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.RotatingScaling, e)
    End Sub
    
    Private Sub HandleMouseLeftButtonDown(ByVal actionType As ActionType, ByVal e As MouseEventArgs)
        If (Not (functionPointer) Is Nothing) Then
            functionPointer(Me, actionType, New Point((translateTransform.X + rotateTransform.CenterX), 
              (translateTransform.Y + rotateTransform.CenterY)), e.GetPosition(Nothing))
        End If
    End Sub

You built a car out of all these parts?

So we move back from VideoBlock.xaml.cs to Page.xaml.cs

We've created our primary object for painting video, lets populate it. First, how do we know how big the media file is? With the MediaOpened event! And we'll restart the video with the MediaEnded event.

So here is the full Page_Load event from Page.xaml.cs

public void Page_Loaded(object o, EventArgs e)
{
    // Required to initialize variables
    InitializeComponent();

    BrowserHost.Resize +=new EventHandler(BrowserHost_Resize);
    media.MediaOpened += new EventHandler(media_MediaOpened);
    media.MediaEnded += new EventHandler(media_MediaEnded);
    MouseLeftButtonUp += Page_MouseLeftButtonUpOrLeave;
    MouseLeave += Page_MouseLeftButtonUpOrLeave;
    MouseMove += Page_MouseMove;

    hSliderX.Range = new ValueRange(1, 20);
    hSliderY.Range = new ValueRange(1, 20);
    hSliderX.ValueChanged += new EventHandler(slider_ValueChanged);
    hSliderY.ValueChanged += new EventHandler(slider_ValueChanged);

    hSliderX.SetValue(ZIndexProperty, 500);
    hSliderY.SetValue(ZIndexProperty, 500);
    sliderX.SetValue(ZIndexProperty, 500);
    sliderY.SetValue(ZIndexProperty, 500);
}

VB.Net

    Public Sub Page_Loaded(ByVal o As Object, ByVal e As EventArgs)
        ' Required to initialize variables
        InitializeComponent
        AddHandler BrowserHost.Resize, AddressOf Me.BrowserHost_Resize
        AddHandler media.MediaOpened, AddressOf Me.media_MediaOpened
        AddHandler media.MediaEnded, AddressOf Me.media_MediaEnded
        MouseLeftButtonUp = (MouseLeftButtonUp + Page_MouseLeftButtonUpOrLeave)
        MouseLeave = (MouseLeave + Page_MouseLeftButtonUpOrLeave)
        MouseMove = (MouseMove + Page_MouseMove)
        hSliderX.Range = New ValueRange(1, 20)
        hSliderY.Range = New ValueRange(1, 20)
        AddHandler hSliderX.ValueChanged, AddressOf Me.slider_ValueChanged
        AddHandler hSliderY.ValueChanged, AddressOf Me.slider_ValueChanged
        hSliderX.SetValue(ZIndexProperty, 500)
        hSliderY.SetValue(ZIndexProperty, 500)
        sliderX.SetValue(ZIndexProperty, 500)
        sliderY.SetValue(ZIndexProperty, 500)
    End Sub

So now that we have the media loaded, we can use the media.NaturalVideoWidth and media.NaturalVideoHeight properties to find out exactly how big the video is! I have a function that is fired off when the page is first loaded or a slider value has changed. This is how we chop it up.

private void prepVideoObjects()
{
    int totalPhotosX = (int)hSliderX.Value;
    int totalPhotosY = (int)hSliderY.Value;

    if (lastX != totalPhotosX || lastY != totalPhotosY)
    {
        lastX = totalPhotosX;
        lastY = totalPhotosY;

        sliderX.Text = string.Format("X ({0}):", totalPhotosX.ToString("D2"));
        sliderY.Text = string.Format("Y ({0}):", totalPhotosY.ToString("D2"));

        int rectWidth = (int)Math.Floor(media.NaturalVideoWidth / totalPhotosX);
        int rectHeight = (int)Math.Floor(media.NaturalVideoHeight / totalPhotosY);
        Random random = new Random();

        // remove all old videoblocks
        for (int i = Children.Count - 1; i > 0; i--)
        {
            if (Children[i].GetType() == typeof(VideoBlock))
                Children.RemoveAt(i);
        }

        for (int x = 0; x < totalPhotosX; x++)
        {
            for (int y = 0; y < totalPhotosY; y++)
            {
                VideoBlock vb = new VideoBlock(media.Name, x * rectWidth, y * rectHeight, rectWidth, rectHeight, SetActivePhoto);
                shuffle(vb, random);
                Children.Add(vb);
            }
        }
    }
}

VB.Net

Private Sub prepVideoObjects()
        Dim totalPhotosX As Integer = CType(hSliderX.Value,Integer)
        Dim totalPhotosY As Integer = CType(hSliderY.Value,Integer)
        If ((lastX <> totalPhotosX)  _
                    OrElse (lastY <> totalPhotosY)) Then
            lastX = totalPhotosX
            lastY = totalPhotosY
            sliderX.Text = String.Format("X ({0}):", totalPhotosX.ToString("D2"))
            sliderY.Text = String.Format("Y ({0}):", totalPhotosY.ToString("D2"))
            Dim rectWidth As Integer = CType(Math.Floor((media.NaturalVideoWidth / totalPhotosX)),Integer)
            Dim rectHeight As Integer = CType(Math.Floor((media.NaturalVideoHeight / totalPhotosY)),Integer)
            Dim random As Random = New Random
            ' remove all old videoblocks
            Dim i As Integer = (Children.Count - 1)
            Do While (i > 0)
                If (Children(i).GetType = GetType(VideoBlock)) Then
                    Children.RemoveAt(i)
                End If
                i = (i - 1)
            Loop
            Dim x As Integer = 0
            Do While (x < totalPhotosX)
                Dim y As Integer = 0
                Do While (y < totalPhotosY)
                    Dim vb As VideoBlock = New VideoBlock(media.Name, (x * rectWidth), (y * rectHeight), rectWidth, rectHeight, SetActivePhoto)
                    shuffle(vb, random)
                    Children.Add(vb)
                    y = (y + 1)
                Loop
                x = (x + 1)
            Loop
        End If
    End Sub

Last but not least, lets see how the mouse movements are handled on the page.

private void Page_MouseMove(object sender, MouseEventArgs e)
{
    if (null != activeVideoBlock)
    {
        // Perform the appropriate transform on the active photo
        Point position = e.GetPosition(null);
        switch (currentActionType)
        {
            case VideoBlock.ActionType.Moving:
                // Move it by the amount of the mouse move
                activeVideoBlock.Translate(position.X - currentLastPosition.X, position.Y - currentLastPosition.Y);
                break;
            case VideoBlock.ActionType.RotatingScaling:
                // Rotate it according to the angle the mouse moved around the photo's center
                double radiansToDegrees = 360 / (2 * Math.PI);
                double lastAngle = Math.Atan2(currentLastPosition.Y - currentVideoCenter.Y, currentLastPosition.X - currentVideoCenter.X) * radiansToDegrees;
                double currentAngle = Math.Atan2(position.Y - currentVideoCenter.Y, position.X - currentVideoCenter.X) * radiansToDegrees;
                activeVideoBlock.Rotate(currentAngle - lastAngle);

                break;
        }
        currentLastPosition = position;
    }
}

VB.Net

    Private Sub Page_MouseMove(ByVal sender As Object, ByVal e As MouseEventArgs)
        If (Not (activeVideoBlock) Is Nothing) Then
            ' Perform the appropriate transform on the active photo
            Dim position As Point = e.GetPosition(Nothing)
            Select Case (currentActionType)
                Case VideoBlock.ActionType.Moving
                    ' Move it by the amount of the mouse move
                    activeVideoBlock.Translate((position.X - currentLastPosition.X), (position.Y - currentLastPosition.Y))
                Case VideoBlock.ActionType.RotatingScaling
                    ' Rotate it according to the angle the mouse moved around the photo's center
                    Dim radiansToDegrees As Double = (360 / (2 * Math.PI))
                    Dim lastAngle As Double = (Math.Atan2((currentLastPosition.Y - currentVideoCenter.Y), (currentLastPosition.X - currentVideoCenter.X)) * radiansToDegrees)
                    Dim currentAngle As Double = (Math.Atan2((position.Y - currentVideoCenter.Y), (position.X - currentVideoCenter.X)) * radiansToDegrees)
                    activeVideoBlock.Rotate((currentAngle - lastAngle))
            End Select
            currentLastPosition = position
        End If
    End Sub

Wrapping it up

So as you can see, Silverlight is pretty powerful once you figure out how to tame it. I used example base code and modified it to suit my own needs to dynamically alter a video in real time. There are a few code tweaks that could happen to improve this code and I'm betting with some additional effort, one could remove the delegate. In addition, you could add in edge detection code and a timer to make this into a true video puzzle game. This was a pet project I worked on while I was at the airport. This demo can be downgraded to Silverlight 1.0 also. I don't believe anything I did was really Silverlight 1.1 (now 2.0) only.

Clint's Bio:

Clint is an academic developer evangelist for Microsoft. His two primary development languages are C# and JavaScript. He has built a Disco Dance Floor too! In his off time, he whips up other random weird projects and does twenty something activities with his friends. His next two big projects are an automated bartender and a skateboard segway. Clint’s blog is betterthaneveryone.com and can be emailed at crutkas@microsoft.com if you have any question.

Silverlight 8-Ball

Coding4Fun — Thu, 20 Sep 2007 03:03:50 GMT

	In this article, I will discuss how I have built a 2 player 8-Ball game in Silverlight (play here). I will explain how I used the Expression tools to design the graphics and various .NET techniques to enable user control and game animation.
Justin Petersen Coding4Fun at Clarity Difficulty: Intermediate Time Required: 6-10 hours Cost: Free Software: Visual Studio 2008 Beta 2, Silverlight 1.1 Alpha Refresh, Expression Design, Expression Blend 2 Preview Hardware: Download: Download

In this article, I will discuss how I have built a 2 player 8-Ball game in Silverlight (play here). I will explain how I used the Expression tools to design the graphics and various .NET techniques to enable user control and game animation.

The following components must be implemented to complete this game:

Graphics (pool table, pool ball, and pool stick)
Vector animation and collision physics
User interaction
Game state and control

Before I continue, I’d like to point out that I did not originate all of the code and XAML for this solution. I came up with the idea of creating a Silverlight 8-Ball game after stumbling upon a 2-D “bouncing bubble” animation here: http://www.bubblemark.com/ created by Alexey Gavrilov.

The original intent for this solution was to compare the performance of the bouncing bubbles across various platforms. I found this very useful, but was personally interested in making something fun and interactive out of it.

Getting Started

Prerequisites

The following tools were used to implement this solution:

Visual Studio 2008 Beta 2
Silverlight 1.1 Alpha Refresh
Expression Design
Expression Blend

Learning more about game development

If you'd like to learn more about game development, I found a number Coding4Fun posts under Gaming helpful. In particular, the 2D Game Primer by "ZMan" gives a good overview of the basics (e.g. GameLoop, Sprites, etc.)

Graphics

I am not much of a graphic designer. In fact, I've spent the majority of my career designing and implementing line of business apps for enterprise customers (typically not a very graphically intensive task). So I believe it is a decent testament to the effectiveness of the Microsoft Expression suite that I could, in short order, create the graphics for this game.

I used Expression Design to build the graphics of the pool table. Modeling my design after the standard dimension of a 9 foot billiards table, I drew the main structure as a rectangle with rounded corners. I drew a smaller green rectangle centered over it for the playing surface (of course following standard dimensions). Next, I created 6 black circles placed one level behind the playing surface to give the impression of pockets. Lastly, I added the wood texture to the outside rectangle (which was super easy as there is a set of wood textures available by default), and drew a simple rounded rectangle beneath the table to serve as a status bar.

I also created the graphics for my pool stick using Expression Design. This consisted of a few tapered polygons and 2 half circles for the butt and tip.

After exporting the associated XAML, I imported each element into my project. From here, I could view and refine the design through a scaled down visual editor or the associated XAML.

Lastly, I inspected the XAML markup for the pool ball I reused from Alexey's Bubblemark solution. I needed to understand the properties of this object so that I could modify the color of the balls later on.

Vector Animation and Collision Physics

The root of Silverlight8Ball is a Storyboard (x:Name=”GameLoop”) that acts as a triggering mechanism for determining ball velocity, direction, and redirection due to collision. Whereas some Storyboards are implemented with predetermined paths and timelines directly in XAML, the action taken at each tick must be determined dynamically in this game. So we simply set the duration to 00:00:0, and handle the Completed event to trigger our positioning logic. This design effectively creates what game developers call a “Game Loop” that triggers continuous processing to determine application state.

The resulting “GameLoop_Completedhandler” becomes the root processing agent for all of the game’s ball movement logic.

   1: void GameLoop_Completed(object sender, EventArgs e)   2: {   3:     switch (m_ActionState)   4:     {   5:         case m_ActionStates.BallsMoving:   6:     7:             // prep   8:             List<Ball> removeList = new List<Ball>();   9:             bool someBallsAreMoving = false;  10:    11:             // move each ball  12:             foreach (Ball ball in m_GameBalls)  13:             {  14:                 ball.Move();  15:                 if (ball.InPocket) removeList.Add(ball);  16:                 else if (ball.IsMoving) someBallsAreMoving = true;  17:             }  18:    19:             // store balls sunk on shot and update ui   20:             foreach (Ball ball in removeList)  21:             {  22:                 sunkBalls.Add(ball);  23:                 RemoveBall(ball);  24:             }  25:    26:             if (someBallsAreMoving)  27:             {  28:                 // update vectors for ball collisions  29:                 for (int i = 0; i < m_GameBalls.Count; i++)  30:                 {  31:                     for (int j = i + 1; j < m_GameBalls.Count; j++)  32:                     {  33:                         m_GameBalls[i].DoCollide(m_GameBalls[j]);  34:                     }  35:                 }  36:             }  37:             else  38:             {  39:                 // determine shot results  40:                 ShotResults results = EvaluateShot();  41:    42:                 // apply state and ui changes  43:                 UpdateGameState(results);  44:             }  45:    46:             break;  47:     }  48:    49:     // restart the storyboard  50:     if (m_IsRunning)  51:     {  52:         GameLoop.Begin();  53:     }  54:    55: }

   1: Private Sub GameLoop_Completed(ByVal sender As Object, ByVal e As EventArgs)   2:     3:     Select Case m_ActionState   4:     5:         Case m_ActionStates.BallsMoving   6:     7:             ' prep   8:             Dim removeList As List(Of Ball) = New List(Of Ball)()   9:             Dim someBallsAreMoving As Boolean = False  10:    11:             ' move each ball  12:             For Each ball As Ball In m_GameBalls  13:                 ball.Move()  14:                 If ball.InPocket Then  15:                     removeList.Add(ball)  16:                 ElseIf ball.IsMoving Then  17:                     someBallsAreMoving = True  18:                 End If  19:             Next ball  20:    21:             ' store balls sunk on shot and update ui   22:             For Each ball As Ball In removeList  23:                 sunkBalls.Add(ball)  24:                 RemoveBall(ball)  25:             Next ball  26:    27:             If someBallsAreMoving Then  28:                 ' update vectors for ball collisions  29:                 For i As Integer = 0 To m_GameBalls.Count - 1  30:                     For j As Integer = i + 1 To m_GameBalls.Count - 1  31:                         m_GameBalls(i).DoCollide(m_GameBalls(j))  32:                     Next j  33:                 Next i  34:             Else  35:                 ' determine shot results  36:                 Dim results As ShotResults = EvaluateShot()  37:    38:                 ' apply state and ui changes  39:                 UpdateGameState(results)  40:             End If  41:    42:     End Select  43:    44:     ' restart the storyboard  45:     If m_IsRunning Then  46:         GameLoop.Begin()  47:     End If  48:    49: End Sub

The above code snippets demonstrate the full functionality of our GameLoop. However, the primary animation and collision logic are handled by "ball.Move();" and "m_GameBalls[i].DoCollide(m_GameBalls[j]);" lines (in C# example). The Move() function of each ball applies its current vector (i.e. x and y "velocities") to determine its next position and updates the ball's UI element coordinates. Once the ball is advanced, the DoCollide function checks if two balls have collided. If so, their vectors are adjusted accordingly.

   1: public bool DoCollide(Ball b)   2: {   3:     // calculate some vectors    4:     double dx = this._x - b._x;   5:     double dy = this._y - b._y;   6:     double dvx = this._vx - b._vx;   7:     double dvy = this._vy - b._vy;   8:     double distance2 = dx * dx + dy * dy;   9:       10:     if (Math.Abs(dx) > this._d || Math.Abs(dy) > this._d)  11:         return false;  12:     if (distance2 > this._d2)  13:         return false;  14:    15:     // make absolutely elastic collision  16:     double mag = dvx * dx + dvy * dy;  17:    18:     // test that balls move towards each other      19:     if (mag > 0)  20:         return false;  21:    22:     mag /= distance2;  23:    24:     double delta_vx = dx * mag;  25:     double delta_vy = dy * mag;  26:    27:     this._vx -= delta_vx;  28:     this._vy -= delta_vy;  29:    30:     b._vx += delta_vx;  31:     b._vy += delta_vy;  32:    33:     return true;  34: }

   1: Public Function DoCollide(ByVal b As Ball) As Boolean   2:     ' calculate some vectors    3:     Dim dx As Double = Me._x - b._x   4:     Dim dy As Double = Me._y - b._y   5:     Dim dvx As Double = Me._vx - b._vx   6:     Dim dvy As Double = Me._vy - b._vy   7:     Dim distance2 As Double = dx * dx + dy * dy   8:     9:     If Math.Abs(dx) > Me._d OrElse Math.Abs(dy) > Me._d Then  10:         Return False  11:     End If  12:     If distance2 > Me._d2 Then  13:         Return False  14:     End If  15:    16:     ' make absolutely elastic collision  17:     Dim mag As Double = dvx * dx + dvy * dy  18:    19:     ' test that balls move towards each other      20:     If mag > 0 Then  21:         Return False  22:     End If  23:    24:     mag /= distance2  25:    26:     Dim delta_vx As Double = dx * mag  27:     Dim delta_vy As Double = dy * mag  28:    29:     Me._vx -= delta_vx  30:     Me._vy -= delta_vy  31:    32:     b._vx += delta_vx  33:     b._vy += delta_vy  34:    35:     Return True  36: End Function

User Interaction

While the GameLoop Storyboard does a good job at managing ball movement, I didn't think it was the best option for handling pool stick and q-ball control. During times of user control (stick aiming and q-ball "ball in hand"), I chose to use mouse movement and click events to determine object position and angle. I created my own enumeration to manage switching between ball movement and user control states. The rest involves some simple statistics and Silverlight XAML transformations.

Pool Stick Aiming and Ball-In-Hand

During a scratch, the MouseMove event allows the affect of moving the q-ball for placement. During aiming, it allows the user to rotate the poolstick around the q-ball.

   1: void Page_MouseMove(object sender, MouseEventArgs e)   2: {   3:     4:     m_MousePoint = e.GetPosition(this);   5:     6:     switch (m_ActionState)   7:     {   8:         case m_ActionStates.Scratch:   9:             m_QBall.MoveAbsolute(m_MousePoint.X, m_MousePoint.Y);  10:             break;  11:         case m_ActionStates.Aiming:  12:             m_PoolStick.Rotate(m_MousePoint.X, m_MousePoint.Y, m_QBall.BallCenterX, m_QBall.BallCenterY);  13:             break;  14:     }        15:       16: }

   1: Private Sub Page_MouseMove(ByVal sender As Object, ByVal e As MouseEventArgs)   2:     3:     m_MousePoint = e.GetPosition(Me)   4:     5:     Select Case m_ActionState   6:         Case m_ActionStates.Scratch   7:             m_QBall.MoveAbsolute(m_MousePoint.X, m_MousePoint.Y)   8:         Case m_ActionStates.Aiming   9:             m_PoolStick.Rotate(m_MousePoint.X, m_MousePoint.Y, m_QBall.BallCenterX, m_QBall.BallCenterY)  10:     End Select  11:    12: End Sub

The RotateTransform Rocks

The Silverlight RotateTranform is very powerful in this case. It allows me to simply rotate the pool stick object around a center point (the center of the q-ball) using a relative angle between the mouse point and the ball.

   1: public void Rotate (double mouseX, double mouseY, double ballX, double ballY)   2: {   3:     double vx = ballX - mouseX;   4:     double vy = ballY - mouseY;   5:        6:     radians = Math.Atan2(vy, vx);   7:     double angle = radians * (180/Math.PI);   8:        9:     rootCanvas.RenderTransform = new RotateTransform   10:     {   11:         CenterX = Model.stickBuffer,   12:         CenterY = Model.stickHeight/2,   13:         Angle = angle   14:     };  15:    16: }

   1: Public Sub Rotate(ByVal mouseX As Double, ByVal mouseY As Double, ByVal ballX As Double, ByVal ballY As Double)   2:     Dim vx As Double = ballX - mouseX   3:     Dim vy As Double = ballY - mouseY   4:     5:     radians = Math.Atan2(vy, vx)   6:     Dim angle As Double = radians * (180 / Math.PI)   7:     8:     Dim transform As RotateTransform = New RotateTransform()   9:     transform.CenterX = Model.stickBuffer  10:     transform.CenterY = Model.stickHeight / 2  11:     transform.Angle = angle  12:    13:     rootCanvas.RenderTransform = transform  14:    15: End Sub

Power Control and Q-Ball Placement

Based on the ActionState, the mouse up and mouse down events manage user interaction during power adjustment and q-ball placement.

   1: void Page_MouseLeftButtonUp(object sender, MouseEventArgs e)   2: {   3:     if (this.m_ActionState == m_ActionStates.AdjustingPower)   4:     {   5:         m_QBall.Strike(m_PoolStick.power, m_PoolStick.radians);   6:     7:         m_PoolStick.StopPowerMovement();   8:         this.Children.Remove(m_PoolStick);   9:         this.m_ActionState = m_ActionStates.BallsMoving;  10:     }  11: }  12:    13: void Page_MouseLeftButtonDown(object sender, MouseEventArgs e)  14: {  15:     switch (m_ActionState)  16:     {  17:         case m_ActionStates.Aiming:  18:             m_PoolStick.StartPowerMovement();  19:             this.m_ActionState = m_ActionStates.AdjustingPower;  20:             break;  21:         case m_ActionStates.Scratch:  22:             ResetPoolStick();  23:             m_ActionState = m_ActionStates.Aiming;  24:             break;  25:     }  26: }

   1: Private Sub Page_MouseLeftButtonUp(ByVal sender As Object, ByVal e As MouseEventArgs)   2:     If Me.m_ActionState = m_ActionStates.AdjustingPower Then   3:         m_QBall.Strike(m_PoolStick.power, m_PoolStick.radians)   4:     5:         m_PoolStick.StopPowerMovement()   6:         Me.Children.Remove(m_PoolStick)   7:         Me.m_ActionState = m_ActionStates.BallsMoving   8:     End If   9: End Sub  10:    11: Private Sub Page_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)  12:     Select Case m_ActionState  13:         Case m_ActionStates.Aiming  14:             m_PoolStick.StartPowerMovement()  15:             Me.m_ActionState = m_ActionStates.AdjustingPower  16:         Case m_ActionStates.Scratch  17:             ResetPoolStick()  18:             m_ActionState = m_ActionStates.Aiming  19:     End Select  20: End Sub

Game State and Control

The final component to this application is the "business rules" behind the game. A game of 8-Ball is governed by the results of each player's shot. Whether that shot ends the game or whose shot it is next is based on what balls are sunk.

I created another enumeration for ShotResults which defines each potential outcome: GoAgain, NextPlayer, Scratch, ScratchOnEight, PrematureEightBall, and Win.

   1: private void UpdateGameState(ShotResults results)   2: {   3:     string resultText = "";   4:        5:     switch (results)   6:     {   7:         // update game states   8:         case ShotResults.PrematureEightBall:   9:         case ShotResults.ScratchOnEight:  10:             resultText = ProcessGameEnd(false, results);  11:             m_ActionState = m_ActionStates.GameOver;  12:             break;  13:         case ShotResults.NextPlayer:  14:             resultText = "Player looses turn";  15:             ChangeCurrentPlayer();  16:             ResetPoolStick();  17:             m_ActionState = m_ActionStates.Aiming;  18:             break;  19:         case ShotResults.GoAgain:  20:             resultText = "Nice job. Go again.";  21:             ResetPoolStick();  22:             m_ActionState = m_ActionStates.Aiming;  23:             break;  24:         case ShotResults.Scratch:  25:             resultText = "Scratch. Player looses turn.";  26:             ChangeCurrentPlayer();  27:             AddBall(m_QBall);  28:             m_ActionState = m_ActionStates.Scratch;  29:             break;  30:         case ShotResults.Win:  31:             resultText = ProcessGameEnd(true, results);  32:             m_ActionState = m_ActionStates.GameOver;  33:             break;  34:     }  35:    36:     sunkBalls.Clear();  37:    38:     this.Text_Status.Text = resultText;  39: }

   1: Private Sub UpdateGameState(ByVal results As ShotResults)   2:     Dim resultText As String = ""   3:     4:     Select Case results   5:         ' update game states   6:         Case ShotResults.PrematureEightBall, ShotResults.ScratchOnEight   7:             resultText = ProcessGameEnd(False, results)   8:             m_ActionState = m_ActionStates.GameOver   9:         Case ShotResults.NextPlayer  10:             resultText = "Player looses turn"  11:             ChangeCurrentPlayer()  12:             ResetPoolStick()  13:             m_ActionState = m_ActionStates.Aiming  14:         Case ShotResults.GoAgain  15:             resultText = "Nice job. Go again."  16:             ResetPoolStick()  17:             m_ActionState = m_ActionStates.Aiming  18:         Case ShotResults.Scratch  19:             resultText = "Scratch. Player looses turn."  20:             ChangeCurrentPlayer()  21:             AddBall(m_QBall)  22:             m_ActionState = m_ActionStates.Scratch  23:         Case ShotResults.Win  24:             resultText = ProcessGameEnd(True, results)  25:             m_ActionState = m_ActionStates.GameOver  26:     End Select  27:    28:     sunkBalls.Clear()  29:    30:     Me.Text_Status.Text = resultText  31: End Sub

Conclusion

This was a fun and educational exercise for me (always a good combination). I learned enough about graphics design to at least fear it less, especially with the use of the Expression suite. It also opened my perspective of Silverlight to understand how it will empower us to implement more rich interactive applications. The most interesting part for me, however, was the change in perspective I had to take in implementing an "always on" game. As simple as a game of 8-ball is, it definitely required an adjustment in thinking from the event-driven model of programming I'm accustomed to using in my every day work.

If you have any questions about this solution or have suggestions for improvement, feel free to contact me at jpetersen@claritycon.com

Where am I? What am I doing?

Coding4Fun — Sat, 01 Sep 2007 01:17:53 GMT

	Do have a GPS receiver laying around? Do you travel? Do you blog? If you answered yes to these questions, you might be interested in this Windows Live Writer plugin. People who travel and like to blog about it can just click a button to insert their current coordinates along with a Virtual Earth Map.
Arian Kulp Arian's Blog Difficulty: Intermediate Time Required: 1-3 hours Cost: $50 and up (depending on hardware choice) Software: Visual Basic or Visual C# Express Editions, Windows Live Writer Hardware: A GPS receiver that supports the NMEA protocol (they all should...) Download: C# Download VB Download MSI Download (binary only, ready to install!)

Introduction

I've had a yellow Garmin etrex GPS unit in the home for about two years now. My family enjoys geocaching (though we don't go nearly often enough...), and it was a good price. The unit has a small screen, but it lets you enter coordinates, then track your way to them. Really, just having your current coordinates is a pretty cool thing!

I've always enjoyed playing with GPS software, and I've liked the idea of creating my own. Unfortunately, I didn't know how to interface with the hardware, so I was out of luck. After buying a serial/USB cable, I was half-way there, but what to do with the data stream?

In this article, learn how to read the stream of data from (hopefully) any GPS unit, then create a plugin for Windows Live Writer so you can add your current location to any blog entry. Scott Hanselman created a GPS interface in Where the Heck am I? Connecting .NET 2.0 to a GPS back in October of 2006. As always with Scott, it was a great article, but I didn't come across it until I'd made it through a lot of roadblocks already! My result at this point works pretty well so I decided to stick with it, plus it integrates with Windows Live Writer.

Figure 1: The plugin in action

If you want, download the source code from the links above, or just follow along with the article. If you haven't already, download the appropriate version of Visual Studio 2005 Express Edition. The plugin will run on either XP or Vista.

Choosing your Hardware

Though my main GPS receiver is the Garmin etrex, I also recently bought an Ambicom unit with no screen -- just a Bluetooth link. It's the perfect device just for obtaining information for a PDA, laptop, or other Bluetooth-connected device if you don't need display on the device. It uses the SPP profile for data exchange, so it's just another COM port as well.

My other unit, the Garmin, has four metal dots hidden under a rubber flap. A number of sites provide information on the pin-outs, or you can buy a cable (serial). I bought a serial-USB interface to avoid the serial port on the back of the machine. My assumption is that all GPS units end up with a serial connection via virtual or physical COM port. If I'm wrong, you may need to do some extra work to get the data.

Speaking NMEA

Speaking about data... once you have a connection to the unit (4800 bps, 8-N-1), you will have a continuously updated stream of information flowing in. The format of this data is defined by the National Marine Electronics Association (NMEA), and is an older format than most people would expect. Interpreting the information is pretty easy once you understand the format (of course, that's often the case!).

Sample NMEA Data

$GPRMB,V,,,,,,,,,,,,A,N*13

$GPGGA,,,,,,0,00,,,M,,M,,*66

$GPGSA,A,1,,,,,,,,,,,,,,,*1E

$GPGSV,3,1,10,02,60,

$GPRMC,,V,,,,,,,210807,0.2,E,N*36

241,00,04,78,042,00,05,24,314,00,09,20,252,00*7F

$GPGSV,3,2,10,10,07,185,00,12,44,307,00,17,36,089,00,20,08,035,00*7A

$GPGSV,3,3,10,28,05,147,00,30,08,319,00*75

$GPGLL,,,,,,V,N*64

$GPBOD,,T,,M,,*47

$PGRME,,M,,M,,M*00

There are three things to notice at first glance (four if you're really on the ball!). First, every line starts with a dollar sign. Second, that dollar sign is always followed by five upper-case letters (not all begin with "G" though). Third, the data following the capital letters is comma-delimited. If you looked really close, you might have noticed that every line ends with an asterisk, then two more characters that appear to be in hexadecimal format.

The capital letters after the dollar sign represent the message ID. All standard NMEA messages start with "G." In this example, the line beginning with "P" is a Garmin-specific message. It's proprietary, but it's also documented and freely available from their web site.

With each line being comma-delimited, processing is pretty easy at a high-level. From the System.IO.Ports.SerialPort class, you can call ReadLine() to get a line of data as a string. Each line of data is called a "sentence" in the standard. From there, the Split() method can be used to return an array for the "words."

That hexadecimal number is the checksum for the sentence. It's a simple formula too. Take, in order, every character between (not including) the dollar sign and asterisk, and XOR them together. The first character seeds the checksum (in other words, a one-character sentence's checksum would be equal to that one character). The LocationInterface class handles all of the serial I/O and NMEA protocol implementation (the little there is!).

Visual Basic

Public Shared Function GetChecksum(ByVal sentence As String) As String
    ' A sentence can't be shorter than 9 characters ($XXXXX*0x)
    If sentence.Length < 9 Then
        Return ""
    End If

    ' Seed with the second character (remember, skip the $)
    Dim checksum As Integer = Convert.ToByte(sentence(1))
    Dim loc As Integer = 0, [end] As Integer = sentence.Length - 3

    ' Loop through the characters
    For Each c As Char In sentence
        ' XOR every character between the dollar sign and the asterisk
        If loc > 1 AndAlso loc < [end] Then
            checksum = checksum Xor Convert.ToByte(c)
        End If
        loc += 1
    Next

    ' "X2" is the format string for hexadecimal
    Return checksum.ToString("X2")
End Function

Visual C#

// Calculates the checksum for a sentence
public static string GetChecksum(string sentence)
{
    // A sentence can't be shorter than 9 characters ($XXXXX*0x)
    if( sentence.Length < 9 ) return "";

    // Seed with the second character (remember, skip the $)
    int checksum = Convert.ToByte(sentence[1]); // Second character
    int loc = 0, end = sentence.Length - 3;

    // Loop through the characters
    foreach (char c in sentence)
    {
        // XOR every character between the dollar sign and the asterisk
        if (loc > 1 && loc < end)
        {
            c2 = c2 ^ Convert.ToByte(c);
        }
        loc++;
    }
    // "X2" is the format string for hexadecimal
    return checksum.ToString("X2");
}

With this handy method, you can validate every line of data that you have before bothering to process it. If a line's bad, don't make a guess. The nature of NMEA is to repeat data frequently. If you missed it this time, you'll have it a second or so later probably. In my code, when I want to get the next data sentence, I call my ReadNextGoodLine() method. This calls ReadLine() on the SerialPort, and validates the checksum. It will try up to ten times before returning a NULL.

Visual Basic

Private Function ReadNextGoodLine(ByVal port As SerialPort) As String
    ' Must read the next line.  This may not be a complete NMEA line
    ' (or even not an NMEA line at all!)
    Dim line As String = Nothing
    For i As Integer = 0 To 9

        ' Up to ten tries...
        line = port.ReadLine()

        If Not IsSentenceValid(line) Then
            line = Nothing
        Else
            Exit For

        End If
    Next

    Return line
End Function

Visual C#

private string ReadNextGoodLine(SerialPort port)
{
    // Must read the next line.  This may not be a complete NMEA line
    // (or even not an NMEA line at all!)
    string line = null;

    // Up to ten tries...
    for (int i = 0; i < 10; i++)
    {
        line = port.ReadLine();

        if (!IsSentenceValid(line)) line = null;
        else break;
    }

    return line;
}

Now that I can read sentences and validate them, I can also create a basic auto-scan GPS search. From the SerialPort class, I can call GetPortNames(). This returns a string array of port names. I can instantiate each port by name, call open, then call ReadNextGoodLine(). If it returns NULL (couldn't find valid NMEA after ten lines), assume it isn't a GPS unit and keep moving to the next port. The ComPortName property saves the name, and can also be set manually to avoid the search.

Of Latitudes and Longitudes...

One thing I learned from this project, is just how much math is involved with GPS! You take it for granted when the device shows coordinates, but there is so much more going on in there.

Recall that GPS is all about geosynchronous satellites sending out time signals. Based on the time delay from satellites with known geostationary locations, you can locate your position. As any experienced GPS user knows, however, when you first turn on the unit, or if you are in a house, underground, or often even wooded areas, the unit will be searching for a signal and won't display anything. The $GPRMC sentence continues to arrive, but the coordinates may be blank, or indicate the lack of a "fix."

$GPRMC,153538,A,4140.5555,N,09100.0000,W,000.0,246.0,310807,,,A*66

The GPRMC consists of a lot of information, including latitude, longitude, speed, heading, and UTC date/time. If you break up the sentence based on commas, the data words that I concentrate in the sample code are 3 (latitude), 4 (N/S), 5 (longitude), 6 (E/W), and 2 (A means fix, V or blank is no fix).

Visual Basic

Public Function ParseGPRMC(ByVal sentence As String) As CoordinatePair
    Dim loc As CoordinatePair = Nothing
    Dim fix As Boolean = False

    ' Split the sentence into words
    Dim words As String() = sentence.Split(","c)

    ' Not all sentences have data populated
    If words(3) <> "" AndAlso words(4) <> "" AndAlso words(5) <> "" AndAlso words(6) <> "" Then
        ' NMEA coordinates are in DM (Degrees-Minutes) format 
        Dim lat As New Coordinate(Integer.Parse(words(3).Substring(0, 2)), _
            Decimal.Parse(words(3).Substring(2)), _
            DirectCast([Enum].Parse(GetType(HemisphereOptions), words(4)), HemisphereOptions))
        Dim lng As New Coordinate(Integer.Parse(words(5).Substring(0, 3)), _
            Decimal.Parse(words(5).Substring(3)), _
            DirectCast([Enum].Parse(GetType(HemisphereOptions), words(6)), HemisphereOptions))

        loc = New CoordinatePair(lat, lng)

        ' Does the device currently have a satellite fix?
        If words(2) = "A" Then
            ' Fix has been obtained
            loc.Fixed = True
        ElseIf words(2) = "V" Then
            ' Fix has been lost
            loc.Fixed = False
        End If

    End If

    Return loc
End Function

Visual C#

public CoordinatePair ParseGPRMC(string sentence)
{
    CoordinatePair loc = null;

    // Split the sentence into words
    string[] words = sentence.Split(',');

    // Not all sentences have data populated
    if (words[3] != "" && words[4] != "" && words[5] != "" && words[6] != "")
    {
        // NMEA coordinates are in DM (Degrees-Minutes) format 
        Coordinate lat = new Coordinate(int.Parse(words[3].Substring(0, 2)), 
            decimal.Parse(words[3].Substring(2)), 
            (HemisphereOptions)Enum.Parse(typeof(HemisphereOptions), words[4]));
        Coordinate lng = new Coordinate(int.Parse(words[5].Substring(0, 3)), 
            decimal.Parse(words[5].Substring(3)),
            (HemisphereOptions)Enum.Parse(typeof(HemisphereOptions), words[6]));

        loc = new CoordinatePair(lat, lng);
        
        // Does the device currently have a satellite fix?
        if (words[2] == "A") loc.Fixed = true; // Fix has been obtained
        else if (words[2] == "V") loc.Fixed = false; // Fix has been lost
    }

    return loc;
}

The latitude and longitude values in NMEA data are expressed as DM format. DM is short for Degrees-Minutes, and actually the minutes are expressed in decimal form. In the above example, "4140.5555" is meant to be parsed as 41 degrees and 40.5555 minutes. Longitude uses three digits for degrees, so "09100.0000" is 091 degrees and 00.0000 minutes. Degrees-Minutes is just one of three common ways to express coordinates though. Degrees-Minutes-Seconds splits the minutes into minutes and seconds. If you think of it like time, it's easier. If the decimal minutes is .50000, then that's half of 60, or 30 minutes. A number like .55555 would spill into the seconds place as well. You can also use Degrees-Decimal which simply uses one decimal number to express the degrees along with any fractional "extra." Just remember that the source format is Degrees-Minutes, and given any format, you can convert to any other. My Coordinate class provides ToString() methods to return in all three representations, and internally it stores it as int degrees and minutes and decimal seconds. For more information on these numbers, Wikipedia has a great article.

Plugging In

Functioning as a Live Writer plugin requires that you do several things. First of all, you will need to download and install the Windows Live Writer SDK. This gives you access to an assembly to add a reference to in your project. Your entry point class should import the WindowsLive.Writer.Api namespace, and inherit ContentSource. You'll need to add the WriterPlugin attribute with values for a unique GUID (use the Tools | Create GUID menu item in Visual Studio to help with that), and set a name at least.

The EditOptions method must be overridden if you provide an Options dialog (in this sample, it's just a message box. The CreateContent method is the meat of the plugin. This can do anything, then assign a string to the newContent ref parameter. It returns a DialogResult to indicate if it was OK'd or Cancel'd.

The InsertableContentSource attribute lets you set the SideBar text property, and the WriterPlugin attribute has a property for ImagePath to create an image link:

Figure 2: Ready to use

When CreateContent is called, the sample plugin shows the LocationPreviewForm dialog. This in turn triggers the GPSInit method in LocationInterface to find and connect to the GPS unit. If it's found, the COM port and coordinates are shown. All GPS activity is performed in a BackgroundWorker to maintain a responsive user interface. The Title and Description fields are used for a pushpin in a Virtual Earth embedded AJAX map. This is a full-blown instance with the scrolling and zooming.

Figure 3: The plugin dialog

Actually, I would have liked to use a static bitmap, but I'm not sure how to obtain good free mapping images in a small amount of code, other than services like MapPoint which aren't free. Changing this aspect of the code would be very easy in the GetOutput method, and you could remove the image/map altogether and simply show the coordinates or other information. It's not exactly extensible, but it's easy to modify.

Finally, in order to simplify debugging, I created a post-build event to copy the plugin DLL to the Windows Live Writer Plugins directory:

XCOPY /D /Y /R "$(TargetPath)" "C:\Program Files\Windows Live Writer\Plugins\"

I also went into the Debug settings for the project for the project and set the Start Action to Start external program, setting it to the path to the EXE:

C:\Program Files\Windows Live Writer\WindowsLiveWriter.exe

Now, when you press F5, it copies the DLL and attaches the debugger to a new instance of Windows Live Writer for Edit-and-Continue and other debugging. It works well!

One thing to remember, is that Live Writer locks the DLL's when it's running, so you will get an error if you even build while it's running, since the DLL can't be copied.

Next Steps

At this point, I'd like to create more flexibility on what the plugin does with the coordinates. It could locate images through various sources, include altitude, speed, and heading when available, or just pretty up the embedded information. I'd like to work on the GPS classes a bit to make them more robust, handling some of the other messages, and support raising events to be used in other applications. Developing GPS applications is conceptually pretty easy once you have the data. It's just a matter of exposing the data in a good way.

Conclusion

Working with GPS data was much easier than I expected. I had some Bluetooth/USB issues at first, but working with the data was fun and gave me lots of ideas! See what you can do with location information to enhance some existing application, or think up something entirely new. Have fun with the code and as usual, contact me through my blog for comments, complaints, or questions.

Arian Kulp is an independent software developer and writer working in the Midwest. He has been coding since the fifth grade on various platforms, and also enjoys photography, nature, and spending time with his family. Arian can be reached through his web site at http://www.ariankulp.com.