By now you’ve probably seen Larry’s video in which he explores the WEI Share (“We” share) project and what it does. Now we’re going to show you what went in to creating it, from building the desktop portion to grab the score to compiling the data to setting it up on Azure.

Creating the Silverlight App and Web Service – Peter B.

We originally set out to create a Silverlight Application that would display uploaded Windows Experience Index (WEI) Scores via a Web Service and Desktop Application. At the time, we created the application in Silverlight 3 and used a WCF Service to integrate with not only the desktop application but also the Silverlight application. Since we weren’t sure exactly where the WCF Service was being hosted in relation to the Web Site, one of the things we did was make it so the location of the WCF Service can be fairly easily changed. We ended up going with Silverlight Parameters to pass in the location of the WCF Service. Doing this was fairly easy, but it took some set up. The first step involved modifying App.xaml.cs to the following:

private void Application_Startup(object sender, StartupEventArgs e)
{
    //Insert InitParams into Global Resources
    foreach (string key in e.InitParams.Keys)
    {
        this.Resources.Add(key, e.InitParams[key]);
    }

    this.RootVisual = new MainPage();
}

Once this was done, we then had to modify the code behind for the main Silverlight app in MainPage.xaml.cs, like so:

WindowsExperienceServiceClient serviceClient;

public MainPage()
{
    InitializeComponent();

    if (App.Current.Resources.Contains("WCFServiceLocation")
        && !String.IsNullOrEmpty(App.Current.Resources["WCFServiceLocation"] as string))
    {
        EndpointAddress endpointAddress = new EndpointAddress(
            App.Current.Resources["WCFServiceLocation"].ToString());
        serviceClient = new WindowsExperienceServiceClient(
            "CustomBinding_IWindowsExperienceService", endpointAddress);
    }
    else
    {
        serviceClient = new WindowsExperienceServiceClient();
    }

    myDelay.Duration = TimeSpan.FromMilliseconds(2000);
    myDelay.Completed += new EventHandler(myDelayCompleted);
    myDelay.Begin();

    this.Loaded += new RoutedEventHandler(MainPage_Loaded);
}

In the above example, I checked for the parameter, and finding that it existed I changed the Endpoint Address of the WCF Service (without an existing parameter, it uses the default address). After this was complete, we had to pass in a location similar to the following in the Default.aspx page’s object tag for the Silverlight control:

<param name="initparams" value="WCFServiceLocation=http://<%=Request.Url.Host %>:8080/windowsexperienceservice.svc" />

This made changing the location of the WCF Service much easier, allowing the flexibility we later needed.

Within the WCF Service, we separated our code using a three layered approach utilizing separate folders for each layer “DataAccessLayer”, “BusinessLogicLayer”, and “BusinessEntities”. Within the BusinessEntities folder, we had the base classes containing each business entity. As an example, we had the WindowsAssessment class, which contained all of the properties of the data sent from the Desktop Application to the WCF Service, including scores. Below, you can see part of what this looked like:

[DataContract]
public class WindowsAssessment
{
    public Guid? WindowsAssessmentID { get; set; }
    [DataMember]
    public decimal? SystemScore { get; set; }
    [DataMember]
    public decimal? AverageScore { get; set; }
    [DataMember]
    public decimal? MemoryScore { get; set; }
    [DataMember]
    public decimal? CpuScore { get; set; }
    [DataMember]
    public decimal? CpuSubAggScore { get; set; }
    [DataMember]
    public decimal? VideoEncodeScore { get; set; }

    /* additional properties... */
}

You’ll notice above that [DataContract] and [DataMember] was set above the class and properties so that it could be used as part of the WCF Service.

Within the DataAccessLayer folder, we utilized Linq2Sql. We simply created a dbml file, and brought over the few assets we created in the database by dragging and dropping the tables from the Data Source into the diagram. Then, within the WindowsAssessment.cs file of the Data Access Layer, we created methods that used Linq2Sql for our basic CRUD, taking advantage of Linq to easily create paging that we can search, filter, and orderby. Finally, within the Business Logic Layer, we utilized the Business Entities and created different methods that called the DAL and also took advantage of HttpRuntime.Cache.

Once all of the layers were complete, the Desktop and Silverlight Application were free to communicate back and forth, allowing the Desktop App to send a score and the Silverlight App to display it.

Upgrading to Silverlight 4 and Azure

Unfortunately, after we completed all of this, Silverlight 4 released. We quickly moved to upgrade to Silverlight 4 and, while we were at it, also modify the app to utilize Windows Azure.

One of the first things I encountered was that the WCF Service was not going to be within the Web Site anymore. Azure has roles in which you create for your project. In this particular instance I needed 2 web roles, one for the website, one for the WCF. In the end my Solution looked like the following:

Working with WCF as a WCF Role, I noticed that there were a few known issues, all of which were documented here: http://code.msdn.microsoft.com/wcfazure/Wiki/View.aspx?title=KnownIssues&referringTitle=Home. Luckily, with another developer’s help, we were able to pinpoint some of the issues and once we addressed them, we really didn’t have much more to worry about when working with WCF.

The next hurdle was moving over the database. The easiest way to do this is to use the tool, SQL Azure Migration Wizard, which can be found on CodePlex at http://sqlazuremw.codeplex.com/.

Creating the Desktop Experience – Peter J

I was tasked with creating a set of applications for showcasing user’s Windows Experience Index (WEI) scores. The set of applications included a desktop application created using Windows Presentation Foundation (WPF) for retrieving the scores and other system data, and a web site using Silverlight for displaying the information.

I needed to create the desktop application. Accordingly, there were some initial pre-requisites and other considerations we had to deal with:

• Limiting the desktop application to only run in Windows 7
• Determining where the relevant WEI data is stored in the system
• Retrieving the relevant information for all areas
• Remote Service connectivity—in this case we used a WCF service
• Not forcing the user to install/unzip anything (just download and run)

After scouring the web, I found a site (http://www.codeguru.com/cpp/w-p/system/systeminformation/article.php/c8973) with just the information I needed:

/// <summary>
/// Determines if the current version of windows is valid (Vista, 7, Server 2008)
/// </summary>
/// <returns></returns>
public static bool IsValidWindowsVersion()
{
    System.OperatingSystem osInfo = System.Environment.OSVersion;
    //6 - Vista, 7, Server 2008
    if (osInfo.Version.Major == 6)
    {
        //windows 7 = 1
        if(osInfo.Version.Minor == 1)
            return true;
    }

    return false;
}

The next major hurdle to overcome was determining where all of the information for the system was stored.

I didn’t really know where to start, so I started looking at MSDN and the search engines for the best way to get this information and I came across a term I had only heard in passing: WMI. Windows Managed Instruction (WMI) allows for querying the system for data, as well as other things that I did not use it for so we will skip all that. You can learn more on WMI on your own time here: http://msdn.microsoft.com/en-us/library/aa394582(VS.85).aspx. I also found a nifty tool from Microsoft called WMI Creator (http://www.microsoft.com/downloads/details.aspx?familyid=2cc30a64-ea15-4661-8da4-55bbc145c30e&displaylang=en) to help facilitate my search. I was blown away by the sheer amount of data and left with no idea of where to start. So, it was back to MSDN to figure out what happens when the Experience Index is run. This made my task much easier since the Experience Index creates its own XML files with most of the data I need.

I retrieved a list of files and used the one that was within a day of the current date/time. This was easy stuff, till I had to determine what to do if they didn’t have an Experience Index. If they didn’t, I’d just run the Experience Index on the fly. To do this, set the program to tell the user that they do not have a current enough score and that we’ll update the information for them by running the Windows System Assessment Tool (WinSAT).

The initial version was not very good since it was modal and caused the application to not refresh until the Experience Index finished running. Needless to say, I was not happy. What I did do, though, was setup a child form that would launch the WinSAT and query every 5 seconds to see if it was still running. Here is the initial call to load the dialog windows and, if successful, check again for a valid Experience Index to post:

/// <summary>
/// Launches the dialog that handles launching WinSat and checking if it is running.
/// </summary>
private void SetupAndRunWinSAT()
{
    WinSatProgress progressWindow = new WinSatProgress();
    bool? dialogResult = progressWindow.ShowDialog();
    if (dialogResult.HasValue && dialogResult.Value)
        CheckForAndProcessScore();
}

Now we are getting into the good stuff, so I will touch on the actual running of the WinSAT and querying the system for a running process.

Launching the WinSAT is pretty simple since there is an environment variable that points to its path:

Environment.GetFolderPath(Environment.SpecialFolder.System) + @"\WinSat.exe"

I only started a basic process and told it to run. I did not tell it to run behind the scenes because I wanted the user to be sure something was actually happening since the WinSAT can take any number of minutes depending on the system configuration. I also setup a global variable (WinsatExecutableExited) used elsewhere in the application to query to see if the process exited:

/// <summary>
/// Launches the winsat program
/// </summary>
public static bool RunWinSatProgram()
{
    bool isLaunched = false;
    WinsatExecutableExited = false;

    System.Diagnostics.Process winSatProcess = new System.Diagnostics.Process();

    winSatProcess.StartInfo.FileName = Helpers.Globals.WinSatExecutable;
    winSatProcess.StartInfo.Arguments = Helpers.Globals.WinSatExecutableParameters;
    winSatProcess.EnableRaisingEvents = true;

    winSatProcess.Exited += new EventHandler(WinSatProcess_Exited);

    try
    {
        winSatProcess.Start();
        isLaunched = true;
    }
    catch
    { }

    return isLaunched;
}

You’ll notice the exited event handler. I put that there so that when the process exited, it set the flag. However, I was having issues with it not always getting executed on some systems. To accommodate for this, I setup a DispatchTimer in the dialog window that would check every 5 seconds to see if the global WinsatExecutableExited flag was true or if the WinSAT process was still running. In order to check if the process is still running, I used WMI. Yes, yes I know—why, since I hadn’t yet done much of anything with it, am I using WMI for this? Well, I won’t bore you with details, but I found an example online and modified it to fit what I need (http://www.techimo.com/forum/applications-operating-systems/117464-need-vbulletin-script-check-see-if-process-running.html). Bet you weren’t expecting that were you? (Or, maybe you were.) Long story short, I set up a method that I can pass a process name to, and it returns true/false if it is still running or null if it can’t determine anything:

/// <summary>
/// used to determine if the process is still running
/// http://www.techimo.com/forum/applications-operating-systems/117464-need-vbulletin-script-check-see-if-process-running.html
/// </summary>
/// <param name="processName"></param>
/// <returns></returns>
public static bool? IsProcessStillRunning(string processName)
{
    bool? response = null;

    try
    {
        ManagementObjectSearcher searcher =
            new ManagementObjectSearcher("root\\CIMV2",
            "SELECT * FROM Win32_Process");

        foreach (ManagementObject queryObj in searcher.Get())
        {
            if(string.Compare(queryObj["Name"].ToString(), 
                processName, 
                StringComparison.OrdinalIgnoreCase) == 0)
            {
                response = true;
                break;
            }
        }

        if(!response.HasValue)
            response = false;
    }
    catch (Exception ex)
    {
        System.Windows.MessageBox.Show(ex.Message + 
            Environment.NewLine + ex.StackTrace);
    }

    return response;
}

Neat, huh? That’s what happens every time the dialog counts down until it either finishes or can’t—for whatever reason—determine the status of the WinSAT.

Now that the file is generated, we want to get all of the data goodness out of it so we can post our data to the service. Since this is an XML file, it was easily loaded into a XElement and traversed as needed. Now we come to another sticky point, retrieving other system data not stored in the XML file. This is where the web and that WMI Creator application came in handy. I would look online to find what I needed, and then leveraged WMI Creator to generated my base code for me. I setup an object for the retrieved data and here’s what it looks like to query the system for the memory type and speed:

/// <summary>
/// used to retrieve the memory info
/// </summary>
/// <returns></returns>
public static List<WMIItem> GetPhysicalMemory()
{
    List<WMIItem> response = new List<WMIItem>();

    ManagementObjectSearcher searcher =
        new ManagementObjectSearcher("root\\CIMV2",
        "SELECT * FROM Win32_PhysicalMemory");

    foreach (ManagementObject queryObj in searcher.Get())
    {
        response.Add
            (
                new WMIItem()
                {
                    Key = "MemoryType",
                    Value = queryObj["MemoryType"]
                }
            );
        response.Add
            (
                new WMIItem()
                {
                    Key = "Speed",
                    Value = queryObj["Speed"]
                }
            );

        break;
    }

    return response;
}

Now is about where I have all of my information ready and need to send it to the server. Setting up the service connection for initial testing is no problem. It’s a standard Windows Communication Foundation (WCF) service. But here comes the hard part—setting it up so the service does not need information from a config file. Back to the web we go, since I am like the other 100% lazy programmers who prefer to just add a Service reference because it is easy. Well, there are two parts to this: generating the code class and setting up the service connection information.

Generating the code class is easy—you use a command line to generate it. It was a little complex since I was generating it for asynchronous calls to the service, but this was nothing MSDN couldn’t help me with. Speaking of, here’s the documentation from MSDN: http://msdn.microsoft.com/en-us/library/ms730059.aspx

The other part was sending up the binding endpoint. Most of the information I needed was generated when I generated the code file form the service. Now I just needed to determine how to put it all together. MSDN to the rescue yet again: http://msdn.microsoft.com/en-us/library/ms733107.aspx

What I ended up doing was creating a CustomBinding and specifying the same information/objects the system would have created for me using the values in the config file:

/// <summary>
/// The wcf service reference member
/// </summary>
WindowsExperienceServiceClient experienceClient = null;

CustomBinding myBinding = new CustomBinding();
myBinding.Elements.Add(Helpers.Globals.CustomServiceBinding);
myBinding.Elements.Add(Helpers.Globals.CustomTransportBinding);
experienceClient = 
   new WindowsExperienceServiceClient(
      myBinding, 
      Helpers.Globals.CustomEndPoint);

The global bindings and endpoints:

/// <summary>
/// The binary encoding objct for the custom binding
/// </summary>
public static BinaryMessageEncodingBindingElement CustomServiceBinding
{
    get
    {
        BinaryMessageEncodingBindingElement binaryEncoding = 
new BinaryMessageEncodingBindingElement();
        binaryEncoding.MaxReadPoolSize = 64;
        binaryEncoding.MaxSessionSize = 2048;
        binaryEncoding.MaxWritePoolSize = 16;
        binaryEncoding.ReaderQuotas.MaxDepth = 32;
        binaryEncoding.ReaderQuotas.MaxStringContentLength = 8192;
        binaryEncoding.ReaderQuotas.MaxArrayLength = 16384;
        binaryEncoding.ReaderQuotas.MaxBytesPerRead = 4096;
        binaryEncoding.ReaderQuotas.MaxNameTableCharCount = 16384;

        return binaryEncoding;
    }
}

/// <summary>
/// The transport element for the custom binding
/// </summary>
public static HttpTransportBindingElement CustomTransportBinding
{
    get
    {
        HttpTransportBindingElement transportBinding = 
            new HttpTransportBindingElement();
        transportBinding.ManualAddressing = false;
        transportBinding.MaxBufferPoolSize = 524288;
        transportBinding.MaxReceivedMessageSize = 65536;
        transportBinding.AllowCookies = false;
        transportBinding.AuthenticationScheme = 
            System.Net.AuthenticationSchemes.Anonymous;
        transportBinding.BypassProxyOnLocal = false;
        //decompressionEnabled="true" 
        transportBinding.HostNameComparisonMode = 
            HostNameComparisonMode.StrongWildcard;
        transportBinding.KeepAliveEnabled = true;
        transportBinding.MaxBufferSize = 65536;
        transportBinding.ProxyAuthenticationScheme = 
            System.Net.AuthenticationSchemes.Anonymous;
        transportBinding.Realm = string.Empty;
        transportBinding.TransferMode = TransferMode.Buffered;
        transportBinding.UnsafeConnectionNtlmAuthentication = false;
        transportBinding.UseDefaultWebProxy = true;

        return transportBinding;
    }
}

/// <summary>
/// the endpoint used for the service communication
/// </summary>
public static EndpointAddress CustomEndPoint
{
    get
    {
        return new EndpointAddress(
            Properties.Resources.WEIServiceEndpoint);

    }
}

Yes, it is just that easy. All I had to do after that was setup the call completed handler and then call the service method and then it was all done.

Conclusion

This was a really fun and challenging project for us. While we’d worked extensively in Silverlight, the desktop development was relatively new to us. After getting our bearings, we found it pretty simple to translate our Web development skills to the desktop environment with WPF.

If you want to try this out, the download link for the source code is at the top of the article!

About The Authors

Lincoln Anderson, Peter Brownstein, and Peter Juchniewicz are all developers at web design company 352 Media Group. The company, which focuses on emerging technologies like Microsoft Silverlight Design, Windows Phone 7 App Development, and Microsoft Surface Development, has worked on other cool tools like the Coding4Fun Windows Phone 7 T-Shirt Cannon and the Tweetcraft WOW Twitter client.

Want to know if the earth is shaking?  How about if someone is moving your laptop?  Create a utility to detect movement so you’ll know right away when something is happening!

Introduction

The Windows 7 sensors platform exposes a standard way of communicating with hardware for determining location, light, magnetic north (compass), distance, movement, and much more.  Through the accelerometer, movement is measured as force (G's) in up to three dimensions.  The Freescale Badge board that I use for sensor development includes ambient light, touch, and a 3D accelerometer.

Working with the three axes of accelerometer data is incredibly easy.  Each sensor data report indicates the number of G’s felt along each axis.  You can watch the values of movement in order to know more about the environment.

My first thought with this application was to create a seismometer to measure ground tremors, but I’m not so sure the one that I have would be nearly sensitive for that.  Then it occurred to me that the accelerometer would work great to detect a laptop being moved.  I see people leaving their laptop on a table and going up for a refill at a coffee shop.  When I do that I always feel pretty nervous.  Knowing that it could sound an alarm if someone moved it would make me feel more secure. 

For this application, you’ll need a Windows 7 system, with a built-in or hardware add-on accelerometer.  Some hard drives have them, some Lenovo laptops have them, but you need to be able to access the sensor using standard Windows 7 Devices and Sensors drivers.

With hardware in place, download Visual Studio 2010 Express, the Windows API Code Pack, and the Sensors and Location API, then you’re ready to get started!

Good Vibrations

The accelerometer is an interesting sensor.  You wiggle something physical around and get numbers that change.  The numbers represent how many G’s are being experienced in up to three dimensions.  This makes it trivial to see when there is movement.  With some additional work you can figure out more fine-grained movement information, but we just need to know how much things are moving around.

Using the Windows 7 managed sensors API, we can instantiate the accelerometer and start reading values with only a small amount of work.  Even better, due to the design of the sensors API we can access it without locking it.  In other words, unlike reading data from a serial port or other hardware connection, applications can share devices.

In this application, I wanted a UI with a lots of the raw and calculated values showing.  This includes the G’s for the individual axes, the net magnitude, and even a chart of data.  In a WPF application, you need public properties in order to participate in databinding.  You also need to be able to alert listeners when your values change.  Unfortunately, the X, Y, and Z values from the sensor report are not easily bindable due to their indexed nature.  They also have no way of alerting listeners (like the UI) that they have been updated.

The key to creating the right properties is using the INotifyPropertyChanged interface.  This interface has only a single member – the PropertyChanged event.  Whenever a property value changes, raise this event and any interested listeners will know to grab the new value.

Visual Basic

#Region "Magnitude Property"
    Private _magnitude As Double
    Public Property Magnitude() As Double
        Get
            Return _magnitude
        End Get
        Set(ByVal value As Double)
            _magnitude = value
            OnPropertyChanged("Magnitude")
        End Set
    End Property
#End Region

#Region "INotifyPropertyChanged Members"

    Public Event PropertyChanged As PropertyChangedEventHandler Implements INotifyPropertyChanged.PropertyChanged

    Private Sub OnPropertyChanged(ByVal prop As String)
        RaiseEvent PropertyChanged(Me, New PropertyChangedEventArgs(prop))
    End Sub
#End Region

Visual C#

#region Magnitude Property
private double _magnitude;
public double Magnitude
{
    get { return _magnitude; }
    set
    {
        _magnitude = value;
        OnPropertyChanged("Magnitude");
    }
}
#endregion
#region INotifyPropertyChanged Members

public event PropertyChangedEventHandler PropertyChanged;

private void OnPropertyChanged(string prop)
{
    if (PropertyChanged != null)
        PropertyChanged(this, new PropertyChangedEventArgs(prop));
}

#endregion

This pattern is found in every public property that you want to enable for databinding.  It’s great because you can do it on a class for WPF or Silverlight, yet it’s not creating a dependency on them.  You can subscribe to this event in some apps, and simply ignore the event in other applications.  Being able to reuse business objects this way is a big deal to me.

Now the UI can display new values as they change without the old Windows forms-style code (tbValue.Text = “value”).  The other part was to actually sound an alarm if the acceleration was more than a preset value.  This is a simple comparison.  This threshold value is called “sensitivity” and accepts a value from 0-3.  This is value is also a property in the view model.  If the measured magnitude value is higher than the threshold, then an alert displays and the system Exclamation sound plays:

Visual Basic

If Me.Magnitude > Me.Sensitivity Then
    Status = String.Format("Big movement! ({0} G's)", Magnitude)
    System.Media.SystemSounds.Exclamation.Play()
End If

if (this.Magnitude > this.Sensitivity)
{ 
    Status = string.Format("Big movement! ({0} G's)", Magnitude);
    System.Media.SystemSounds.Exclamation.Play();
}

In XAML, a Slider control is bound to the Sensitivity property in TwoWay mode. The slider is set with minimum and maximum values. Since sliders typically display their current value, you need to add a Label too.  It’s not automatic.  The Label’s content is actually bound to the Value property of the Slider control.  So, although it sounds complicated to require an extra field for the slider’s label, the element databinding makes it simple.

XAML

<Label Content="Sensitivity (1-3):" FontWeight="bold"/>
<Grid>
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width=".75*" />
        <ColumnDefinition Width=".25*" />
    </Grid.ColumnDefinitions>
    <Slider x:Name="slider" Margin="10,0,10,0" Value="{Binding Sensitivity, Mode=TwoWay}" Minimum="0" Maximum="3"  />
    <Label Content="{Binding Value, ElementName=slider}" FontWeight="bold" Grid.Column="1"/>
</Grid>

Getting the Data

Most sensors support the concept of auto-updating their data on a minimum interval.  I decided to do it on my own using a background thread.  This guaranteed a constant interval making it nicer for charting.  It wasn’t really necessary, but what I found was no readings during no movement, and a flood of readings when things changed.

As long as the sensor is enabled, you can subscribe to the DataReportChanged event.  Then you can access the CurrentAcceleration property.  Individual axes are indexed rather than exposed as direct properties.  I suppose this is more flexible for far-future 11-dimensional accelerometers…

Visual Basic

Private Sub UpdateData()
    _optionsControl.Dispatcher.BeginInvoke(Sub() UpdateDataInvoked())
End Sub

Private Sub UpdateDataInvoked()
    If (_accelSensor Is Nothing) OrElse (_accelSensor.TryUpdateData() = False) Then
        Return
    End If

    Dim currentAccel = _accelSensor.CurrentAcceleration
    Dim sampleTime As DateTime = DateTime.Now

    Me.LastDataReportX = currentAccel(AccelerationAxis.X)
    Me.LastDataReportY = currentAccel(AccelerationAxis.Y)
    Me.LastDataReportZ = currentAccel(AccelerationAxis.Z)

    Me.Magnitude = Math.Sqrt(Math.Pow(Me.LastDataReportX, 2) + Math.Pow(Me.LastDataReportY, 2) + Math.Pow(Me.LastDataReportZ, 2))

    If _perfCounter IsNot Nothing Then
        _perfCounter.RawValue = Convert.ToInt64(Me.Magnitude * 10)
    End If

    ReadingsX.Add(New SensorReading() With { _
        .Reading = Me.LastDataReportX, _
        .Timestamp = sampleTime _
    })
    ReadingsY.Add(New SensorReading() With { _
        .Reading = Me.LastDataReportY, _
        .Timestamp = sampleTime _
    })
    ReadingsZ.Add(New SensorReading() With { _
        .Reading = Me.LastDataReportZ, _
        .Timestamp = sampleTime _
    })

    If ReadingsX.Count > 20 Then
        ReadingsX.RemoveAt(0)
    End If
    If ReadingsY.Count > 20 Then
        ReadingsY.RemoveAt(0)
    End If
    If ReadingsZ.Count > 20 Then
        ReadingsZ.RemoveAt(0)
    End If

    If Me.Magnitude > Me.Sensitivity Then
        Status = String.Format("Big movement! ({0} G's)", Magnitude)
        System.Media.SystemSounds.Exclamation.Play()
    End If
End Sub

Visual C#

void UpdateData()
{
    _optionsUI.Dispatcher.Invoke(new Action(
        delegate()
        {
            if ((_accelSensor == null) || (_accelSensor.TryUpdateData() == false)) return;
                    
            var currentAccel = _accelSensor.CurrentAcceleration;
            DateTime sampleTime = DateTime.Now;

            this.LastDataReportX = currentAccel[AccelerationAxis.X];
            this.LastDataReportY = currentAccel[AccelerationAxis.Y];
            this.LastDataReportZ = currentAccel[AccelerationAxis.Z];

            this.Magnitude =
                Math.Sqrt(Math.Pow(this.LastDataReportX, 2) + Math.Pow(this.LastDataReportY, 2) +
                            Math.Pow(this.LastDataReportZ, 2));

            if (_perfCounter != null) _perfCounter.RawValue = Convert.ToInt64(this.Magnitude * 10);

            ReadingsX.Add(new SensorReading { Reading = this.LastDataReportX, Timestamp = sampleTime });
            ReadingsY.Add(new SensorReading { Reading = this.LastDataReportY, Timestamp = sampleTime });
            ReadingsZ.Add(new SensorReading { Reading = this.LastDataReportZ, Timestamp = sampleTime });

            if (ReadingsX.Count > 20) ReadingsX.RemoveAt(0);
            if (ReadingsY.Count > 20) ReadingsY.RemoveAt(0);
            if (ReadingsZ.Count > 20) ReadingsZ.RemoveAt(0);

            if (this.Magnitude > this.Sensitivity)
            { 
                Status = string.Format("Big movement! ({0} G's)", Magnitude);
                System.Media.SystemSounds.Exclamation.Play();
            }
    }));
}

These ReadingsX/Y/Z collections contain the last 20 readings and are used in the next section…

Flooded with Data

With so much data flowing in, it can be hard to really visualize it.  This is why I decided that it made sense to chart it.  There are a few charting options out there that are free and a few that are commercial.  Some free options include Visifire, and Dynamic Data Display (D3).  I chose to use Visifire with a line graph and have a data series for each axis.  I’m not sure how useful this information is, but it looks nice!

Behind the scenes, I have three instances of ObservableCollection of type SensorReading.  SensorReading simply holds a data point and a point in time.  In XAML, I’m able to perform full data-binding from these collections to datasets in the graph.  Each time I add a value I check to see if I have over twenty items, and if so I remove the oldest one (element zero).  I’m sure I could graph more than that, but it seemed sufficient.

The other use for the data was to send it to the Windows Performance Monitor.  You’ve probably seen the charts of the CPU load, memory, disk write, and other low-level data, but you may not realize that you can track your own data this way as well.  Some companies add performance counters for number of simultaneous users, failed logins, cache misses, or whatever else is relevant to their application.  Adding counters for your own data is really easy.  You can choose to either have a number of items, or an increment/decrement counter to keep track of something.

Start by creating a category, then create the counter.  You can set names, descriptions, and data types.  For this project, I crated a category called “Sensors” and a counter called “Net force.”  This is of type NumberOfItems64 so I set the RawValue property with each new value.  The only caveat with performance counters, is that if you add the counter while Performance Monitor is running you won’t see it until you restart it.

Visual Basic

Private Sub CreatePerfCounter()
    If (PerformanceCounterCategory.Exists(PerfCategoryName)) Then
        PerformanceCounterCategory.Delete(PerfCategoryName)
    End If

    ' Create a collection of type CounterCreationDataCollection.
    Dim CounterCreationData As New CounterCreationDataCollection()

    ' Create the counter, set properties, and add to collection
    Dim ccd As New CounterCreationData()
    With (ccd)
        .CounterName = PerfNetForceName
        .CounterHelp = "The net acceleration value (in G's)"
        .CounterType = PerformanceCounterType.NumberOfItems64
    End With

    CounterCreationData.Add(ccd)

    ' Create the category and pass the collection to it.
    _category = PerformanceCounterCategory.Create(PerfCategoryName, _
        "The Sensors performance object tracks the value of various sensors over time", _
        PerformanceCounterCategoryType.SingleInstance, CounterCreationData)

    _perfCounter = New PerformanceCounter(PerfCategoryName, PerfNetForceName, False)
End Sub

private void CreatePerfCounter()
{
    if (PerformanceCounterCategory.Exists(PerfCategoryName))
    {
        PerformanceCounterCategory.Delete(PerfCategoryName);
    }

    // Create a collection of type CounterCreationDataCollection.
    var counterCreationData = new CounterCreationDataCollection();

    // Create the counter, set properties, and add to collection
    counterCreationData.Add(new CounterCreationData
    {
        CounterName = PerfNetForceName,
        CounterHelp = "The net acceleration value (in G's)",
        CounterType = PerformanceCounterType.NumberOfItems64
    });

    // Create the category and pass the collection to it.
    _category =
        PerformanceCounterCategory.Create(
        PerfCategoryName,
        "The Sensors performance object tracks the value of various sensors over time",
        PerformanceCounterCategoryType.SingleInstance,
        counterCreationData);

    _perfCounter = new PerformanceCounter(PerfCategoryName, PerfNetForceName, false);
}

Utility Runner

This is not the first article to use my MEF-based Utility Runner application as the base.  As usual, you’ll need to install that package (listed at the top in the prerequisites) in order for this to work.  You’ll also need to go to your project’s properties, to Debug, and tell it to use a specific executable for debugging.

If you use Utility Runner and want to add this is a plugin, go into the bin\debug\Addins folder and create a ZIP file from the contents of the MEFUtil-Seismo.util folder and rename it to MEFUtil-Seismo.util.  From the Addins tab in Utility Runner, click the Install Addin From File command from within Utility Runner.

Possible Enhancements

In addition to a sound, it might make sense to support other notifications like growl, Twitter, or email.  It might also make sense to respond to changes in a different sensor, for example ambient light.  If I turn off the light in my office, it could sound an alarm if the light went on and I didn’t sign in within a certain amount of time.  That would require a change to monitor a different sensor (not difficult), and to check to see if the desktop is locked or not.

Conclusion

Sensors really change the nature of your code by allowing you to interact with the real world.  If your system isn’t equipped with sensors (like mine, unfortunately), make do with an add-on/prototyping board so you can try things out.  These sensors are basically standard on smartphones these days, and will quickly become common on laptops as well.  Windows 7 included a great feature by providing common hardware and device support.  Take advantage of it today!

About Arian

Arian Kulp is a software developer living in Western Oregon.  He creates samples, screencasts, demos, labs, and articles; speaks at programming events about data, UI, Silverlight, and more; and enjoys spending time with his family.

The XNA team just released a wealth of information on developing for Windows Phone 7.  This is the first of 4 releases for this series on WP7 game development by this team.  They’ll talk about the actual platform, to performance to adding in the final polish to make your game really pop.

What just got released is the following:

• Catapult Wars Lab
• Orientation Sample
• Sound and Music Sample
• 2D Graphics Sample
• Particles Sample
• 2D Asset Types Article
• Intro to C# From Objective-C and Java Article
• Gestures Article

Go head over to Creator’s club to find out more:  http://creators.xna.com/en-US/education/gettingstarted

Trossen Robotics has a pretty awesome RFID kit with a RedBee RFID reader by Robotic Connections.  Here are a few specs from the reader.

• Supports all EM41xx family 125 kHz RFID tags
• USB, XBee, and TTL Serial communication
• Serial Communication Protocol and Shared Serial Devices Protocol (SSDP)
• .NET programming API
• Operates as a slave device to PC or microcontroller
• Onboard programmable microcontroller
• Xbee point to point
• Xbee Mesh networking
• 3 power input options (Barrel Connector, USB, or Vcc)
• 4 Configurable I/O pins
• Valid/Invalid Tag LED
• Xbee Comm LED

There is even a drywall with a gangbox cover even!  You can bet we’ll be releasing a project with one of these shortly.

David Rousset created a 4 part write up on explaining a demo built around Silverlight 4, Windows Azure and XNA on Windows Phone 7:

• Showing the final result in action as well as the SL4 application to play with
  Silverlight 4 XNA Platformer Level Editor for Windows Phone 7: overview (1/4)
• Details on the SL4 app and the source code of the SL app to download
  SL4 XNA Platformer Level Editor for WP7: details on the Silverlight App (2/4)
• Details on the Windows Azure storage
  SL4 XNA Platformer Level Editor for WP7: Azure storage (3/4)
• How I’ve modified the XNA Platformer game and the source code of the game to download
  SL4 XNA Platformer Level Editor for WP7: the XNA game (4/4)

For those interested in creating a level based game, this could be a great starting point.

This article explains how to write a Windows Phone 7 photo effects application.

Introduction

At the MIX 2010 conference, Microsoft announced its new mobile strategy with the Windows Phone 7 platform. Windows Phone 7 is a completely new mobile OS, and the best part is that Silverlight is THE technology for Windows Phone 7 application development. Apps are written with a special version of Silverlight based on Silverlight 3 plus some additions. Games with 3D hardware acceleration can be made with the XNA Game Studio 4

Most of the first technical previews have stated that the Windows Phone has a pretty impressive photo application, and we can surely expect some pretty good cameras from the different phone hardware vendors. A Windows Phone 7 is required to have at least a 5 Megapixel camera. Another cool feature requirement of the Windows Phone camera system is called “pocket to picture.” This represents a special hardware button that launches the camera app even if the phone is locked. This feature will be great for capturing all those spontaneous moments in life. How often have you missed such snapshots because you were fumbling to unlock your phone and starting the camera app? I know this has happened to me quite often.

The ability to so quickly take pictures will result in some very cool photos. The original pictures might already look nice, but some will look even better if they are enhanced with a nice image effect. It’s in this manner that this article and the PicFx application come into play.

This is the first article in a short series describing how to write picture effects applications for Windows Phone. The first article will show how to create the base Windows Phone application and how to implement some basic effects. Spoiler: The next part will demonstrate how to write some more advanced effects you probably don’t want to miss.

The App in Action

The video below demonstrates the application’s features and also shows how the application can be used. It was recorded with the application running in the emulator.

How it works

The following sections will describe how this picture effects application was made. We will drill down from the User Interface (UI) Crust through the UI Mantle until we finally reach the Effects Core. Cross your fingers so a leak doesn’t stop us.

The User Interface

We start our journey on the surface of the application.

The app uses the so-called Pivot control as the main UI template (Figure 1). Along with the Panorama control, the Pivot control is the dominating UI template used in standard Windows Phone applications / hubs. Both controls usually contain multiple items or views that can be navigated with a flick gesture. You can read more about these controls and the general design of Windows Phone applications in the official Windows Phone UI Design and Interaction Guide.

Since the official Silverlight version of the Pivot and Panorama controls haven’t been released yet, we’ll use this nice open source implementation from CodePlex by Stephane Crozatier.

Figure 1: The PicFx application layout using the Pivot control

The relevant part from the MainPage.xaml:

XAML

<phoneCtrls:PivotControl Name="PivotCtrl" >

    <phoneCtrls:PivotItem 
            Name="PivotItemPic" 
            Title="PicFx" 
            Header="Picture " >
        <Image Name="Viewport" />
    </phoneCtrls:PivotItem>

    <phoneCtrls:PivotItem  
            Name="PivotItemFx" 
            Title=" " 
            Header="Effects " >
        <phoneCtrls:PivotItem.Resources>
            <vm:EffectItems x:Key="effects"/>
        </phoneCtrls:PivotItem.Resources>
        <ListBox Name="ListBoxEffects" 
                        SelectionMode="Single" 
                        ItemsSource="{StaticResource effects}" 
                        SelectionChanged="ListBox_SelectionChanged" >
            <ListBox.ItemsPanel>
                <ItemsPanelTemplate>
                    <controlsToolkit:WrapPanel Width="480" />
                </ItemsPanelTemplate>
            </ListBox.ItemsPanel>
            <ListBox.ItemTemplate>
                <DataTemplate>
                    <StackPanel Orientation="Vertical" Margin="14" >
                        <Image Width="128"
                                    Height="128" 
                                    Source="{Binding Thumbnail}" />
                        <TextBlock Text="{Binding Name}" 
                                   FontSize="{StaticResource PhoneFontSizeNormal}" 
                                   VerticalAlignment="Center" 
                                   HorizontalAlignment="Center" />
                    </StackPanel>
                </DataTemplate>
            </ListBox.ItemTemplate>
        </ListBox>
    </phoneCtrls:PivotItem>
</phoneCtrls:PivotControl>

The first Pivot item contains only the Image showing the selected picture. The second view contains a ListBox control with the thumbnails of the effects (see Figure 1). The list is data-bound to the StaticResource “effects,” an instance of the EffectItems class that is derived from ObservableCollection<EffectItem> and consists of EffectItem elements:

C#

public class EffectItems : ObservableCollection<EffectItem>
{
   public EffectItems()
   {
      Add(new EffectItem(new BlackWhiteEffect(), "data/icons/BlackWhite.png"));
      Add(new EffectItem(new SepiaEffect(), "data/icons/Sepia.png"));
   }
}

public class EffectItem
{
   public IEffect Effect { get; private set; }
   public string Name { get; private set; }
   public ImageSource Thumbnail { get; set; }

   public EffectItem(IEffect effect)
   {
      Effect = effect;
      Name = effect.Name;
   }

   public EffectItem(IEffect effect, string thumbnailRelativeResourcePath)
      : this(effect)
   {
      // Load the thumbnail from the resource stream 
      // using the WriteableBitmapEx lib
      Thumbnail = new WriteableBitmap(0, 0).
            FromResource(thumbnailRelativeResourcePath);
   }
 }

The ListBox.ItemTemplate defines the visual representation of the data (EffectItem). The Image is data-bound to the Thumbnail property and the TextBlock is data-bound to the Name property of the EffectItem. The two controls are stacked together. Each Thumbnail has a size of 128 x 128 pixels and a Margin of 14 is used between the items so they get evenly spaced. Please also note the use of a StaticResource “PhoneFontSizeNormal” rather than a fixed FontSize. You can find all the predefined styles in the file %ProgramFiles%\SDKs\Windows Phone\v7.0\Design\ThemeResources.xaml. These styles, margins, and sizes are important because the Windows Phone is a multi-touch device that supports various themes. The Windows Phone UI Design and Interaction Guide provides more information about this topic.

The small thumbnail image is compiled into the assembly’s resources. The WriteableBitmapEx’ FromResource extension method is then used to load the actual image data at runtime. The FromResource method takes the relative path of the image and automatically builds the Pack URI that is needed to load data from the assembly’s resources. Please note that it’s recommended to add larger images as “Content” and not as “Resource” to keep the assembly file small.

The ListBox.ItemsPanel specifies the layout of the items. A WrapPanel is used here so the items are stacked horizontally and wrapped when the panel limit is reached. The ItemsPanelTemplate specifies a StackPanel as default, which wastes precious space here. The WrapPanel originally comes from the Silverlight Toolkit and was ripped by Matthias Shapiro and compiled for the Windows Phone 7.

The Application Bar

The Windows Phone Application Bar is a very useful system control that can contain up to four icon buttons and a set of text menu items. It’s normally used as a toolbar to provide quickly accessible functionality. The Application Bar is easy to use and very handy for Windows Phone app development.

Figure 2: The Application Bar from the PicFx app

The relevant part from the MainPage.xaml :

XAML

<phone:PhoneApplicationPage.ApplicationBar>
    <shell:ApplicationBar IsVisible="True">
        <shell:ApplicationBarIconButton 
                Text="Choose" 
                IconUri="/data/appbar/appbar.folder.rest.png" 
                Click="ApplicationBarIconFolderButton_Click"/>
        <shell:ApplicationBarIconButton 
                Text="Take" 
                IconUri="/data/appbar/appbar.feature.camera.rest.png" 
                Click="ApplicationBarIconCameraButton_Click"/>
        <shell:ApplicationBarIconButton 
                Text="Save" 
                IconUri="/data/appbar/appbar.save.rest.png" 
                Click="ApplicationBarIconSaveButton_Click"/>
    </shell:ApplicationBar>
</phone:PhoneApplicationPage.ApplicationBar>

The MainPage is derived from the PhoneApplicationPage class, which defines the ApplicationBar property. Three ApplicationBarIconButtons are added here. Each one needs to have set a Text and an Icon. Microsoft provides a set of common application bar icons that are used in the PicFx app. The set contains 32 dark and 32 light icons along with the corresponding vector versions.

The Code Behind

The UI is properly set up, the user is able to select an effect and click on the Application Bar buttons, but nothing will actually happen. Now it’s time to breathe life into the UI elements.

Choosing a Picture

First of all we need to get a picture. One way to do this is to call the Windows Phone picture hub from our app and let the user select an image from the library. Fortunately, this task is pretty easy when using the Windows Phone Task API:

C#

private void Initialize()
{
   // ...
   // Init tasks
   photoChooserTask = new PhotoChooserTask();
   photoChooserTask.Completed += 
      PhotoProviderTaskCompleted;
   // ...
}

A member variable of the PhotoChooserTask is instantiated in the Initialize method and an event handler for the Completed event is attached. The Initialize method is called in the Loaded event handler of the page:

C#

private void PhotoProviderTaskCompleted
   (object sender, PhotoResult e)
{
   // Load the photo from the task result
   if (e != null && e.TaskResult == TaskResult.OK)
   {
      var bmpi = new BitmapImage();
      bmpi.SetSource(e.ChosenPhoto);
      original = new WriteableBitmap(bmpi);
   }
}

The Chooser's Completed event provides a PhotoResult that contains the TaskResult, the ChosenPhoto as a Stream, and some other properties. If the user hasn't cancelled the operation, the stream is used as the source of a BitmapImage, which is then assigned to a WriteableBitmap member variable.

To launch the picture application, the PhotoChooserTask.Show method is called when the corresponding Application Bar button is clicked:

C#

private void ApplicationBarIconFolderButton_Click
   (object sender, EventArgs e)
{
   photoChooserTask.Show();
}

The Show method launches the Windows Phone picture hub. The current version of the Windows Phone operating system allows only one application to run at the same time, and our app is therefore terminated when the Chooser is started. Here's where the concept of tombstoning comes into play.

After the user selected an image or pressed the back button, the Chooser's Completed event is raised. You might have noticed the black screen that appears after the choose operation when you debug the application. As a result of the app termination, the Visual Studio debugging session is stopped. The Windows Phone 7 emulator detected that the app was started in a debug context and the emulator now waits at the black screen for re-attaching. So, just go back to Visual Studio and hit F5 (“Start Debugging”), then the debugger is reattached and the app continues. Of course, it's also possible to “Start Without Debugging” in Visual Studio.

Taking a Picture

Another way to get a picture into the app is to launch the camera application and let the user take an on-the-fly picture with the phone’s camera:

C#

private void Initialize()
{
   // ...
   // Init tasks
   cameraCaptureTask = new CameraCaptureTask();
   cameraCaptureTask.Completed += 
      PhotoProviderTaskCompleted;
   // ...
}

A member variable of the CameraCaptureTask is also instantiated in the Initialize method and the same event handler used for the PhotoChooserTask is attached.

To launch the camera application, the CameraCaptureTask.Show method is called when the corresponding Application Bar button is clicked:

C#

private void ApplicationBarIconCameraButton_Click
   (object sender, EventArgs e)
{
   cameraCaptureTask.Show();
}

Obviously, the CameraCaptureTask API works the same way as the PhotoChooserTask.

The emulator doesn’t support the camera application; therefore, the corresponding Application Bar button is disabled in the Initialize method when the app runs in the emulator:

C#

// Disable the camera button if the 
// app runs in the emulator
// The cast is needed since the 
// Buttons property is an IList of objects
var buttons = ApplicationBar.Buttons.Cast<ApplicationBarIconButton>();
var btn = buttons.Where(b =>    
                  b.IconUri.ToString().
                     ToLower().Contains("camera")).FirstOrDefault();
if (btn != null)
{
   btn.IsEnabled = 
      (Microsoft.Devices.Environment.DeviceType != DeviceType.Emulator);
}

In the current beta of the Windows Phone developer tools, the ApplicationBar has some limitations since it’s rendered by the phone shell and is not at all a Silverlight element. This means that there's no Name property on the ApplicationBarIconButton, x:Name doesn't initialize the member variable, and data binding isn’t possible. This is the reason for the above code, which uses the Buttons property and disables the button that has a camera icon if the app is executed in the emulator. As an alternative, the whole Application Bar can be set in the code-behind, including the IsEnabled property, and a working member variable for the button can be initialized.

Resizing a Picture

Both Tasks return the full-sized image. These images can be large, depending on the phone’s camera, and as a result the image processing can take quite a long time. Image processing is the operation where the actual effect is applied to an image. Until the picture is saved, however, only the Image control size is needed. That’s why a resized version of the picture is used until saving. This speeds the interactive process up a lot:

C#

// From MaingPage.ResizeAndShowImage
// Fast and simple resize by using UIElement rendering
if (Viewport != null)
{
   Viewport.Source = bitmap;
   resized = new WriteableBitmap(Viewport, null);
}

The WriteableBitmap resized member variable is being newly instantiated from the Viewport user control. This means that the Viewport image control is rendered at its actual size and the rendered pixels are used for the new and smaller WriteableBitmap instance where the effect gets applied.

Applying an Effect

When the user selects an effect item in the list, the SelectionChanged event is fired and the following event handler and methods are being called:

C#

// From the MainPage class
private void ListBox_SelectionChanged
   (object sender, SelectionChangedEventArgs e)
{
   if (resized != null)
   {
      // Apply selected effect and show image
      var result = ApplySelectedEffect(resized);
      ShowImage(result);
   }
}

private WriteableBitmap ApplySelectedEffect
   (WriteableBitmap bitmap)
{
   // Apply selected effect
   var result = bitmap;
   if (ListBoxEffects != null)
   {
      var item = ListBoxEffects.SelectedItem as EffectItem;
      if (item != null)
      {
         result = item.Effect.Process(bitmap);
      }
   }
   return result;
}

private void ShowImage(ImageSource result)
{
   // Show image and scroll to start page if needed
   if (Viewport != null)
   {
      Viewport.Source = result;
      if (PivotCtrl.SelectedIndex != 0)
      {
         PivotCtrl.ScrollPrev();
      }
   }
}

The event handler calls the ApplySelectedEffect method with the resized bitmap as parameter. The ApplySelectedEffect method casts the SelectedItem to an EffectItem and then applies the underlying effect to the bitmap. Afterwards, the bitmap is set as source of the Viewport Image control in the ShowImage method. As the last step, the Pivot control’s ScrollPrev method is called, the Pivot control is scrolled to the image view, and the user is presented with the applied effect.

Removing an Effect

If the user wants to remove the effect selection and see the original image again, he or she can hit the phone’s hardware back button. This functionality is accomplished by attaching an event handler to the page’s BackKeyPress event:

C#

private void PhoneApplicationPage_BackKeyPress
   (object sender, CancelEventArgs e)
{
   if (ListBoxEffects != null && 
      ListBoxEffects.SelectedItem != null)
   {
      ListBoxEffects.SelectedItem = null;
      e.Cancel = true;
   }
}

The selected item of the ListBox is set to null which raises the SelectionChanged event (see above). The Cancel property of the event argument is set to true. Therefore, the app is not closed, which is the default functionality of the back key. The app will be closed if no effect was applied (ListBoxEffects.SelectedItem == null).

Saving a Picture

After the user applied an effect he probably wants to save the result back to the picture library / photo album. But there's nothing like a PhotoSaveTask available in the Silverlight SDK. Fortunately, the Windows Phone's XNA MediaLibrary comes to the rescue. Only a reference to the Microsoft.Xna.Framework assembly is needed to use it in our Windows Phone Silverlight application. To make this task a bit easier, I wrote some reusable extension methods for the WriteableBitmap:

C#

// From the file WriteableBitmapMediaLibraryExtensions.cs
// Saves the WriteableBitmap encoded as JPEG 
// to the Media library.
// The quality for JPEG encoding 
// has to be in the range 0-100, 
// where 100 is the best quality with the largest size.
public static void SaveToMediaLibrary
   (this WriteableBitmap bitmap, string name, int quality)
{
   using (var stream = new MemoryStream())
   {
      // Save the picture to the WP7 media library
      bitmap.SaveJpeg(
         stream, 
         bitmap.PixelWidth, 
         bitmap.PixelHeight, 
         0, 
         quality);
      stream.Seek(0, SeekOrigin.Begin);
      new MediaLibrary().SavePicture(name, stream);
   }
}

The SaveToMediaLibrary method expects a name for the picture and the quality for JPEG encoding. The quality has to be in the range from 0 to 100, where 100 is the best image quality. Though a higher quality yields a larger file size, it’s not recommended to use a quality below 70.

The MediaLibrary's SavePicture method expects a stream or a byte array as parameter containing an image encoded in the JPEG format. Therefore, the WriteableBitmap is encoded as JPEG with the built-in SaveJpeg extension method. The SaveJpeg method expects the targetStream, the width and height of the target, the orientation, which is not used at the moment, and the quality. The width and height parameters can be useful when a scaled version of the WriteableBitmap should be saved as JPEG.

The image will be saved when the disk floppy button in the ApplicationBar is clicked:

C#

private void ApplicationBarIconSaveButton_Click
   (object sender, EventArgs e)
{
   var result = ApplySelectedEffect(original);
   var name = String.Format(
      "PicFx_{0:yyyy-MM-dd_hh-mm-ss-tt}.jpg", 
      DateTime.Now);
   result.SaveToMediaLibrary(name);
}

The effect is applied to the original sized bitmap and a filename based on the current date and time is created. This filename is then passed to the SaveToMediaLibrary extension method, which will save the edited picture to the phone’s media library / picture hub.

The Effects

Now that we have examined the UI crust and the underlying mantle, it’s time to drill further into the core of our application—the image effects.

The effects selectable by the end-user are made out of several base effects, which can be differently combined. Therefore, it’s possible to make a wide range of different composite effects (outer core) using various combinations of base effects (inner core). This approach also guarantees high code reusability.

Regardless of whether the effect is a base effect or a composite effect, both implement the same IEffect interface (Figure 3). This interface is quite small. The Process method takes a WriteableBitmap as parameter and returns a WriteableBitmap as result of the image processing.

Figure 3: The IEffect interface

The Black & White Effect

This composite effect converts the input image to a grayscale image and slightly increases the contrast. The classic Black & White effect is often used for portrait photos. Most pictures tend to look better when they’re converted to an old school gray image. One might think that’s the reason why I use this effect on my profile picture.

Figure 4: The Black & White effect applied to the sample picture

The Black & White composite effect internally uses both the BrightnessContrast and Tint base effects (Figure 5). Therefore, it contains almost no code at all.

Figure 5: The class diagram of the BlackWhiteEffect

This is the complete code of the BlackWhiteEffect class without XML comments:

C#

public class BlackWhiteEffect : IEffect
{
   readonly TintEffect tintFx;
   readonly BrightnessContrastEffect contrastFx;

   public string Name { get { return "Black & White"; } }

   public float ContrastFactor 
   {
      get { return contrastFx.ContrastFactor; }
      set { contrastFx.ContrastFactor = value; }
   }

   public BlackWhiteEffect()
   {
      tintFx = TintEffect.White;
      contrastFx = new BrightnessContrastEffect 
         { ContrastFactor = 0.15f };
   }

   public WriteableBitmap Process(WriteableBitmap input)
   {
      return contrastFx.Process(tintFx.Process(input));
   }
}

The Base Effects

The Black & White effect of the Outer Core doesn’t have much code. So, where’s all the code? Good question. Let’s dig deeper into the Inner Core to see where the magic is happening.

The Tint Effect

The Tint effect converts each pixel to gray and tints it with a parameterized color:

C#

public WriteableBitmap Process(WriteableBitmap input)
{         
   // Prepare some variables
   var result = new WriteableBitmap(
      input.PixelWidth, 
      input.PixelHeight);
   var resultPixels = result.Pixels;
   var inputPixels = input.Pixels;
   var ta = Color.A;
   var tr = Color.R;
   var tg = Color.G;
   var tb = Color.B;

   for (int i = 0; i < inputPixels.Length; i++)
   {
      // Extract color components
      var c = inputPixels[i];
      var a = (byte)(c >> 24);
      var r = (byte)(c >> 16);
      var g = (byte)(c >> 8);
      var b = (byte)(c);

      // Convert to gray with constant factors 
      // 0.2126, 0.7152, 0.0722
      int gray = (r *  6966 + g * 23436 + b *  2366) >> 15;

      // Apply Tint color
      a = (byte)((a    * ta) >> 8);
      r = (byte)((gray * tr) >> 8);
      g = (byte)((gray * tg) >> 8);
      b = (byte)((gray * tb) >> 8);

      // Set result color
      resultPixels[i] = (a << 24) | 
         (r << 16) | 
         (g << 8) | 
         b;
   }

   return result;
}

First, the result WriteableBitmap is initialized with the same size as the input image and the values of the Pixels property are assigned to local variables. Using the local variables inside the processing loop is much faster than using the Pixels property, which means that a method is being called each time. The WriteableBitmap uses the RGB color space to represent the pixels and is actually just a 32 bit integer array that stores the alpha, red, green, and blue (ARGB) byte components for all the pixels in a 1D array.

Inside the processing loop the red, green, and blue color components of each input pixel are extracted. Then, each component is multiplied with a constant factor and the elements are added to a scalar value. The result actually represents the luminance of the pixel. The human eye is known to be more sensitive to green colors, which is the reason why the green component is weighted more.

The factors aren’t applied as a floating-point multiplication; instead, the components are multiplied with large integer values in the range 2¹⁵. That’s why the result is then divided by 32768 (2¹⁵) using a cheap bit shift operation. Integer operations are usually faster on most non-specialized CPUs, and on mobile CPUs. But this has to be proven in the future with a real Windows Phone device at hand. One thing is clear, the integer operations perform better than floating point operations in the emulator.

In the last processing step the actual tint Color property is applied by multiplying each component of the color with the gray value of the input pixel. Then the byte color components are combined into the result integer pixel. Again, only fast integer operations are used here.

The Black & White effect uses white as tint color, which represents 255, 255, 255 as a byte in the common additive RGB system or 1f, 1f, 1f as a normalized floating point. This means the gray color is used directly and no tinting is actually applied.

The Brightness and Contrast Modification Effect

The second and last base effect from the Black & White effect is the brightness and contrast modification. The Black & White effect only increases the contrast, but the BrightnessContrastEffect can alter contrast and brightness in the same process since both modifications are commonly used together:

C#

public WriteableBitmap Process(WriteableBitmap input)
{         
   // Prepare some variables
   var result = new WriteableBitmap(
      input.PixelWidth, 
      input.PixelHeight);
   var resultPixels = result.Pixels;
   var inputPixels = input.Pixels;
   // Convert to integer factors
   var bfi = (int)(BrightnessFactor * 255);
   var cf = (1f + ContrastFactor) / 1f;
   cf *= cf;
   var cfi = (int)(cf * 32768);

   for (int i = 0; i < inputPixels.Length; i++)
   {
      // Extract color components
      var c = inputPixels[i];
      var a = (byte)(c >> 24);
      var r = (byte)(c >> 16);
      var g = (byte)(c >> 8);
      var b = (byte)(c);
            
      // Modify brightness (addition)
      if (bfi != 0)
      {
         // Add brightness
         int ri = r + bfi;
         int gi = g + bfi;
         int bi = b + bfi;

         // Clamp to byte boundaries
         r = (byte)(ri > 255 ? 255 : (ri < 0 ? 0 : ri));
         g = (byte)(gi > 255 ? 255 : (gi < 0 ? 0 : gi));
         b = (byte)(bi > 255 ? 255 : (bi < 0 ? 0 : bi));
      }
 
      // Modify contrast (multiplication)
      if (cfi != 0)
      {
         // Transform to range [-128, 127]
         int ri = r - 128;
         int gi = g - 128;
         int bi = b - 128;

         // Multiply contrast factor
         ri = (ri * cfi) >> 15;
         gi = (gi * cfi) >> 15;
         bi = (bi * cfi) >> 15;

         // Transform back to range [0, 255]
         ri = ri + 128;
         gi = gi + 128;
         bi = bi + 128;

         // Clamp to byte boundaries
         r = (byte)(ri > 255 ? 255 : (ri < 0 ? 0 : ri));
         g = (byte)(gi > 255 ? 255 : (gi < 0 ? 0 : gi));
         b = (byte)(bi > 255 ? 255 : (bi < 0 ? 0 : bi));
      }  

      // Set result color
      resultPixels[i] = (a << 24) | 
         (r << 16) | 
         (g << 8) | 
         b;
   }

   return result;
}

A with most of the effects, the result WriteableBitmap is first initialized with the same size as the input image and the Pixels property is locally referenced. The range of the ContrastFactor and the BrightnessFactor floating point properties is normalized to [-1, 1], which makes the class easier to use. But floating operations are slower than integer operations; therefore, local, scaled integer variables are initialized for these properties, too.

Brightness

Inside the loop, the color components of each input pixel are again extracted. Then it’s determined whether the Brightness should be modified. The Brightness modification is actually an addition or subtraction of the pixel’s luminance. In terms of vector arithmetic, it can be described as a translation. But I don’t want to bore you with mathematical details and, as always, a picture is worth a thousand words:

Figure 6: The brightness transfer functions from left to right: 1:1, subraction, addition

Figure 6 shows three different brightness transfer functions along with the picture resulting from when the function is applied to the sample image. The y-axis represents the input value and the x-axis represents the corresponding output value for the pixel’s luminosity. The left image is the original. The function curve in the middle has a negative offset, which means that a subtraction is performed and the brightness gets decreased. The right curve has a positive offset which means that an addition is performed and the brightness gets increased. So the brightness modification is actually the y-intercept change of the linear luminance transfer function.

Contrast

On the other hand, the Contrast modification is a multiplication of the pixel’s luminance. It’s a uniform scaling of the color vector (Figure 7). In the above code the value of the pixel components is transformed to the range [-128, 127] and then multiplied with the contrast factor. Due to the range transformation, darker pixels get darker and lighter pixels get lighter when the contrast is increased with a positive factor. Then the pixel is transformed back to the range [0, 255] and the boundaries of the color byte components are checked. The last processing step is the combination of the byte components into the result integer pixel. Again, everything was done with only fast integer operations.

Figure 7: The contrast transfer functions from left to right: 1:1, scale down, scale up

The contrast transfer function in the middle of Figure 7 has a factor below 1, which means the slope is shallower and the contrast gets decreased. The right curve has a factor above 1, which means the slope is steeper and the contrast gets increased. The contrast modification is actually the slope change of the linear luminance transfer function.

The Sepia Effect

After the boring details of the Inner Core effects, it’s time to use these base effects in a different combination to make an interesting Outer Core effect.

The Sepia effect is another composite effect that tints the image and changes the contrast. Like the Black & White, the Sepia effect is also a very popular image effect to give a photo the old-touch. Sepia is some kind of a brown color tint.

Figure 8: The Sepia effect applied to the sample picture

The Sepia effect is basically the same as the Black & White effect, only with slightly other parameter values and a different tint color of course. Actually, only the contrast factor is a bit less than the Black & White effect and the tint color is Sepia (230, 179, 77).

Figure 9: The class diagram of the SepiaEffect

The code of the Sepia effect in its whole glory:

C#

public class SepiaEffect : IEffect
{
   readonly TintEffect tintFx;
   readonly BrightnessContrastEffect contrastFx;

   public string Name { get { return "Sepia"; } }

   public float ContrastFactor 
   {
      get { return contrastFx.ContrastFactor; }
      set { contrastFx.ContrastFactor = value; }
   }

   public SepiaEffect()
   {
      tintFx = TintEffect.Sepia;
      contrastFx = new BrightnessContrastEffect 
         { ContrastFactor = 0.05f };
   }

   public WriteableBitmap Process(WriteableBitmap input)
   {
      return contrastFx.Process(tintFx.Process(input));
   }
}

Conclusion

We drilled down from the UI Crust with the Pivot control template and the Windows Phone Application Bar through the UI Mantle with all the IO functionality. Then we drilled further into the Effects Core with the Black & White and Sepia effects. Finally, we reached the Inner Core and learned how the BrightnessContrast and Tint base effects work.

I hope you didn’t get lost during our long journey because this here isn’t the end. The second article of this series will describe how to implement advanced effects (the cool stuff) and many more.

About The Author

René Schulte is a .Net / Silverlight developer and Microsoft Silverlight MVP passionate about real-time computer graphics, physics, AI, and algorithms. He loves C#, Shaders, Augmented Reality, and computer vision. He started the SLARToolkit, the WriteableBitmapEx, and the Matrix3DEx Silverlight open source projects, and he has a Silverlight website powered by real time soft body physics. He is also a regular author for Microsoft's Coding4Fun. Contact information can be found on his Silverlight website, his blog, or via Twitter.

In this article, Brian Peek describes how he created his recreation of the game show Jeopardy for Visual Studio Live! 2010 in Redmond

For Visual Studio Live 2010 in Redmond, I was asked to help plan the attendee event for the conference.  We decided to recreate the game show Jeopardy and allow 3 teams of 3 people to play: Microsoft Employees vs. Visual Studio Live Speakers vs. Visual Studio Live Attendees.  To pull this off, I decided to create some big, red buttons to allow the players to ring in, and some software to drive the game itself.

The Buzzers

I debated long and hard on whether to create hand-held buzzers as seen on Jeopardy, or to go with larger table-top buzzers.  I opted for the latter to keep things a bit simpler.  I searched around for inexpensive big red buttons, and eventually stumbled across the Staples “Easy” button which says “That was easy” when you smack it.  I figured with a little hacking, I could repurpose these into buzzers that could be used for the game.

To turn this into a usable buzzer, flip it over, remove the foam black feet, batteries, and battery door as shown.

Next, unscrew the four screws and disassemble the button.

Inside, you’ll find a small circuit board with a rubber button mounted on top.  I was able to trace out the switch and determine the two best solder points to extend the switch with some two-conductor wire.

Cut a length of the wire to suit your needs, then strip one end along with the 2 wires inside.

Next, cut off the resistor next to the white rubber button, seen below the arrow in the above picture.  This will make it a bit easier to solder the wire.

Finally, solder the wire onto the two points I noted earlier.  It doesn’t matter which wire goes to which contact.  I also decided to remove the metal plate under the circuit board.  This makes the button very stiff and “clicky”, which I didn’t really like.  This isn’t necessary so you decide how you want the button to behave.

Before we can close up the button some plastic needs to be cut out so the wire can feed out the back.  Cut the red plastic near the battery compartment as shown.

Next, notch out the back of the grey ring, run the wire down through these notches, and close everything back up.

The other end of that wire gets connected to the Phidget 8/8/8 board.  Simply strip the wires at the other end, and insert one into a terminal marked G (ground), and the other into one of the numbered Inputs.  Again, it doesn’t matter which wire goes to which terminal.  You will wind up putting more than one wire into the Ground terminal since there are only two on the board.

The Software

There are two pieces to the game:  The main game board, and the scoring console.  I have some ideas to perhaps put these together into a single application, but for now, they are separate.

The game board application runs on a computer and can be directly duplicated to a projector for display for a large crowd.  The scoring console consists of two windows, one which will be for the operator only, and one which will be projected and shown to the players/audience.  So, you will need a machine capable of multiple monitor output for that.

I’m going to give an overview of the two applications here.  I took some shortcuts and have a few things that I’m not entirely proud of, but with this being a side project with a looming deadline, I did what I had to do.    There are some places where a more XAML-based approach could have been used, some areas where databinding could have been used more effectively, the software isn’t overly configurable, etc.  I hope to address some of these things in a future version and I will update the code/article if/when that happens.

The Game Board

This application displays the main Jeopardy board, but also has an “admin console” where the categories are edited and saved.

Let’s start with the admin side.  The window is created with a Grid layout with 6 columns and 7 rows, the last row being a series of 6 buttons to load/save the data for round 1, 2, and 3 (Final Jeopardy).

In code, those cells are populated with a UserControl named EditorBox that I created.

   1: private void Window_Loaded(object sender, RoutedEventArgs e)

   2: {

   3:     for(int col = 0; col < 6; col++)

   4:     {

   5:         for(int row = 0; row < 6; row++)

   6:         {

   7:             EditorBox tb = new EditorBox();

   8:             tb.SetValue(Grid.RowProperty, row);

   9:             tb.SetValue(Grid.ColumnProperty, col);

  10:             EditorGrid.Children.Add(tb);

  11:  

  12:             _boxes[col,row] = tb;

  13:         }

  14:     }

  15: }

The EditorBox is a very simple control which contains a checkbox to toggle the Daily Double on or off for that square, and a styled textbox.  These controls are hooked up with some data binding to two dependency properties on the control itself, which are set when the data is loaded from the XML file.

Data is stored internally in a List of Category and Clue objects.

   1: public class Category

   2: {

   3:     public string Text { get; set; }

   4:     public Clue[] Clues { get; set; }

   5: }

   6:  

   7: public class Clue

   8: {

   9:     public string Value { get; set; }

  10:     public string Text { get; set; }

  11:     public bool DailyDouble { get; set; }

  12: }

When the data is saved, the screen of EditorBoxes is enumerated and placed into Category and Clue objects.  That master list is then persisted to an XML file using the XmlSerializer.  To load the data, this process is reversed, reading the XML file, placing the data into the Category and Clue objects, and then assigned to the appropriate EditorBox controls.

The game window consists of a number of user controls.  The first is the CategoryWindow which is the white text displayed at the top of each category column.  The boxes below this window are DollarWindow controls which, when double-clicked, display a ClueWindow control.  A single CategoryWindow and five ClueWindow controls make up a CategoryStrip.  Six CategoryStrip controls arranged side-by-side make up the full GameWindow.  There is also an ImageWindow control which is used to display the Daily Double image.  All of these controls contain various dependency properties for things like the clue text, the dollar value, whether it’s a Daily Double, etc.  These items are bound when the game board is loaded, which is handled by some keyboard input.  Pressing F1 will load round 1, F2 will load round 2, and F3 will load the Final Jeopardy round.  Pressing the Tab key will display the admin editor window.

One issue I ran into when building the application was displaying the clue text so it would fill the display window, but then zoom in while keeping the same formatting and wrapping.  I was able to achieve this by wrapping a TextBlock with a Border and then a Viewbox.  The ViewBox allows the content to stretch and resize without changing the contents of the window, and the Border allows the text to be centered vertically.  I was then able to use a quick Storyboard to zoom the clue window in from its position on the board to fill the screen when the clue window is double-clicked.

   1: <EventTrigger RoutedEvent="UserControl.Loaded" SourceName="clueWindow">

   2:     <EventTrigger.Actions>

   3:         <BeginStoryboard>

   4:             <Storyboard>

   5:                 <DoubleAnimation From="0" To="1280" Duration="00:00:00.25" Storyboard.TargetProperty="Width"/>

   6:                 <DoubleAnimation From="0" To="960"  Duration="00:00:00.25" Storyboard.TargetProperty="Height"/>

   7:                 <DoubleAnimation To="0" Duration="00:00:00.25" Storyboard.TargetProperty="(Canvas.Top)"/>

   8:                 <DoubleAnimation To="0" Duration="00:00:00.25" Storyboard.TargetProperty="(Canvas.Left)"/>

   9:             </Storyboard>

  10:         </BeginStoryboard>

  11:     </EventTrigger.Actions>

  12: </EventTrigger>

Sound Effects

There are two sound effects that are played during the game.  The first is the Daily Double sound effect when the clue is revealed, and the second is the Final Jeopardy “think” music.  Both of these are stored as WAV files and copied to the output directory.  To play the sounds, the SoundPlayer object is used.  This takes a path to the WAV file to load.  Then, calling the Play method will play the sound.

   1: SoundPlayer sp = new SoundPlayer("Sounds\\think.wav");

   2: sp.Play();

The Score Board

As I said earlier, this application consists of two Windows, one of which is for the operator, the other of which is for display.

The operator console consists of three Calculator user controls, one per team.  This control contains buttons to represent each dollar value on the board, along with a + and – button.  Scores can be easily added and subtracted based on correct and incorrect answers.  The score value from these columns are bound to the display window’s score area for each team.

The display window consists of three ScoreDisplay user controls, one per team.  A ScoreDisplay contains a TextBlock to display the score for the player, and a TextBlock below it which shows the player’s name.  This area changes to a blue/white gradient when the player rings in.  Above and below these displays is a wide rectangle which also changes to a blue/white gradient when the buzzers have been activated after a question is read.

The buzzers are also wired into this application using the standard Phidgets API.  When the application starts, the Phidget board is opened and an event is wired up to listen for changes to the inputs on the board.  We also read the app.config file to determine which inputs on the board are tied to which buzzer/player.  This will allow us to know when a player has pressed their big red button to buzz in.

Now, to follow the rules of Jeopardy, we have to notify the players that they are allowed to ring in, figure out who rung in first, “light up” their display, and allow them to answer.  If they are correct, the buzzers are reset and the next question is selected.  If they are incorrect, but buzzers are re-activated, however the player who answered incorrectly is locked out and cannot ring in again.

When the host finishes reading the question, the scoreboard operator presses F1 to turn on the buzzers.  This swaps the style of the Rectangle to the gradient version.  When the players ring in, the event we wired up earlier is called on each button press.  This figures out who rung in based on the input of the Phidget that was activated, lights up the appropriate boxes, and locks out the other buzzers.

   1: void _ik_InputChange(object sender, Phidgets.Events.InputChangeEventArgs e)

   2: {

   3:     if(!e.Value)

   4:         return;

   5:  

   6:     lock(_answerLock)

   7:     {

   8:         // we already have our "winner" so dump out

   9:         if(_answering || !_canAnswer)

  10:             return;

  11:  

  12:         // we're on a different thread, so invoke to the UI thread

  13:         this.Dispatcher.Invoke(new Action(delegate 

  14:         {

  15:             // if player 1 can answer, and we're coming from the player 1 input on the Phidget

  16:             if(!_player1Locked && _player1Inputs.Contains(e.Index.ToString()))

  17:             {

  18:                 // we're in answer mode, player 1 can't ring in again, and we light up the board

  19:                 _answering = true;

  20:                 _player1Locked = true;

  21:                 _sdw.Score1Display.Answering = true;

  22:                 _sdw.BuzzersActive(false);

  23:                 Player1Status.Background = Brushes.Green;

  24:             }

  25:  

  26:             if(!_player2Locked &&_player2Inputs.Contains(e.Index.ToString()))

  27:             {

  28:                 _answering = true;

  29:                 _player2Locked = true;

  30:                 _sdw.Score2Display.Answering = true;

  31:                 _sdw.BuzzersActive(false);

  32:                 Player2Status.Background = Brushes.Green;

  33:             }

  34:  

  35:             if(!_player3Locked &&_player3Inputs.Contains(e.Index.ToString()))

  36:             {

  37:                 _answering = true;

  38:                 _player3Locked = true;

  39:                 _sdw.Score3Display.Answering = true;

  40:                 _sdw.BuzzersActive(false);

  41:                 Player3Status.Background = Brushes.Green;

  42:             }

  43:         }));

  44:     }

  45: }

When the player has responded, the proper score value is added or subtracted using the buttons on the operator screen, and one of two things happens.  If the answer was correct (the operator pressed +), the buzzers are reset for the next question.  If the answer was incorrect (the operator pressed –), that player is locked out and the buzzers are re-activated for the other players to buzz in.  If all three players guess incorrectly, or the question isn’t answered, the operator can press F3 to reset the buzzers for the next question.

Using the Application

If you download the source or binaries, you will need to install the fonts included in the archive if you want to get the full Jeopardy effect.  In addition, you will need to build the buzzers, setup the Phidgets board, etc. as described above to run the scoring console.  The app.config for the scoring console contains 3 lines, one per player, which need to be set with the number of the input port the buzzer is tied to.  So, for example, if you have Player 1’s buzzer tied to input #3, the line would look like:

<add key="Player1Inputs" value="3" />

With the fonts and hardware in place, just run each application on their respective computers.  In the game board application, press the Tab key to load the editor and enter your questions for each round.  When playing the game, press F1/F2/F3 to load round 1/2/3 appropriately.

Conclusion

And that’s really all there is to it.  Two applications, some WPF, some user controls, some buttons and a Phidgets board.  Now you can host your own Jeopardy games at the office to make training a bit more interesting, or even play a game with some friends for fun.

I will probably update this software/article a bit more as we’ll probably be doing another game at Visual Studio Live in Orlando in November, so keep an eye out for that.

Bio

Brian is a Microsoft C# MVP who has been actively developing in .NET since its early betas in 2000, and who has been developing solutions using Microsoft technologies and platforms for even longer. Along with .NET, Brian is particularly skilled in the languages of C, C++ and assembly language for a variety of CPUs. He is also well-versed in a wide variety of technologies including web development, document imaging, GIS, graphics, game development, and hardware interfacing. Brian has a strong background in developing applications for the health-care industry, as well as developing solutions for portable devices, such as tablet PCs and PDAs. Additionally, Brian has co-authored the book "Coding4Fun: 10 .NET Programming Projects for Wiimote, YouTube, World of Warcraft, and More" published O'Reilly. He previously co-authored the book "Debugging ASP.NET" published by New Riders.  Brian is also an author for MSDN's Coding4Fun website.

There is a new .NET Micro Framework board that is 100% open source now.  The netduino from Secret Labs can leverage the arduino shields!  Their projects page, http://www.netduino.com/projects/, has a few projects like how to get a blinking LED to work and read data from a color sensor to determine was color is in front of it.

Netduino combines 20 GPIOs with SPI, I2C, 2 UARTs (1 RTS/CTS), 4 PWM and 6 ADC channels.  Best of all with NETMF 4.1, you can use Visual Studio 2010!

Introduction

Last year, Clint Rutkas contacted me about building a project involving lasers, graffiti and code. When you get a request like that, you never decline, especially when lasers are involved. The project requirements sounded quite simple: create an application that can, using a laser pointer as a virtual spray can, draw virtual graffiti on the side of a building. It sounded a little daunting to me at first, but after it was broken up into small digestible pieces, the project ended up not being very complicated.

Below is the basic hardware setup of the Laser Graffiti System.

Building a Laser Tracker Engine

Giving Sight to the Blind

The very first thing that needed to be proven was the ability to track a laser point. The most obvious solution was to give our application some vision with a webcam. Since this was going to be a WPF application, I decided to use my open-source project, WPF MediaKit, which comes with a webcam control called the VideoCaptureElement.

At the time, VideoCaptureElement was not very robust, and it occurred to me that my control wasn’t very useful beyond looking at your webcam in a WPF app! I needed a way to get high-performance access to every pixel of every frame the webcam spat out. This was a good time to add that ability and increase value in my project. Without getting into the gory details of DirectShow and p/invoke, I was able to add a hook to pass me the pixel buffer, where I would wrap it in a Bitmap class and raise an event for each frame.

The XAML required to get each video sample from the web camera.

<MediaKit:VideoCaptureElement LoadedBehavior="Play"
    RenderOptions.BitmapScalingMode="LowQuality"
    UnloadedBehavior="Manual"
    DesiredPixelWidth="{Binding DesiredPixelWidth}"
    DesiredPixelHeight="{Binding DesiredPixelHeight}"
    VideoCaptureDevice="{Binding SelectedItem, ElementName=videoCapDevices}"
    FPS="30"
    EnableSampleGrabbing="True"
    NewVideoSample="VideoCaptureElement_NewVideoSample"
    Margin="0"
    x:Name="videoElement"
    Stretch="Fill"
    MinHeight="380"
    MinWidth="500" />

Did I Mention Lasers?

Now that I had the ability to look at every pixel sent to me, I needed to make sense of it all. In order to complete the project, we have to be able to find a laser point shined on a wall. The difficulty lies in the fact that a laser can be of varying size and color. For example, if you take two consecutive video frames with the same green laser point, you will find the point may slightly differ between the two. This is because we are dealing with analog data and slightly changing lighting conditions. Whatever algorithm we choose to find the laser must take this into account.

There are many advanced ways to use video analytics to find the laser pointer, but this project did have a deadline, and I wanted to keep it fun, so I went with a simple method, which allows a user to filter the video based off ranges of hue, saturation, and luminance. To do this, we make use of the AForge image processing library. AForge is an open-source library that comes with tons of useful utilities for just what we want to do.

Filtering pixels and counting blobs…

As previously mentioned, we need to first filter the image based on hue, saturation, and luminance. Luckily, the AForge library comes with such a filter, so there is not much work to be done here:

/* This AForge class helps us filter out the pixels we do not want */
var hsl = new HSLFiltering
{
    Hue = new IntRange(HueMinimum, HueMaximum),
    Saturation = new DoubleRange(SaturationMinimum, SaturationMaximum),
    Luminance = new DoubleRange(LuminanceMinimum, LuminanceMaximum)
};

var bitmapData = bitmap.LockBits(
    new Rectangle((int)targetSearchArea.X, 
        (int)targetSearchArea.Y, 
        (int)targetSearchArea.Width, 
        (int)targetSearchArea.Height),
    System.Drawing.Imaging.ImageLockMode.ReadWrite, 
    m_lastBitmap.PixelFormat);


/* Apply the AForge filter.  Doing it "in place" is more efficient as
 * new bitmap does not have to be allocated and copied */
hsl.ApplyInPlace(bitmapData);

So, what needs to be done now that we have a filtered image showing only a laser dot? The answer: Blob counting! A blob, in the context of video analytics, is a set of pixels that are all touching, and AForge comes with a class ready to tackle that too!

The AForge blob counter takes in a gray-scale image for processing, and then returns all blobs found in the image. The information included on a blob includes the rectangular pixel area where the laser was found in the video frame. The center of that area should be the center of our laser. It’s important to note that if the HSL filter is improperly calibrated, we will get an overabundance of blobs and there will accordingly be no way we can tell if it is the laser point or not.

/* Create an initialize our blob counter */
var blobsCounter = new BlobCounter
    {
       FilterBlobs = true,
       ObjectsOrder = ObjectsOrder.Size,
       MinHeight = BlobMinimumHeight,
       MinWidth = BlobMinimumWidth,
       MaxWidth = 25,
       MaxHeight = 25
    };

/* We first let the blob counter process our image */
blobsCounter.ProcessImage(grayImage);

/* Retrieve a list of blobs that were found */
Blob[] blobs = blobsCounter.GetObjects(grayImage);

The early laser tracking prototype. We can literally have it track anything, based off color.

Building the Laser Graffiti Application

Our Rocket Engine Needs a Rocket

At this point, I had a working proof of concept and a laser tracker. What I needed now was an application to use it. You know, the laser-graffiti application.

There are many ways to construct a WPF application and one pattern that’s been getting a little bit of traction is MVVM. I chose to make a composite application using Prism and MVVM. Prism is a great guidance library pack for making composite WPF applications with tons of tools to help in common scenarios.

Composite What? Huh?

When developing applications, usually of the larger variety, things can become complex. Complexity turns into a mess. A mess turns into time investment. Time investment turns into money. The more money spent, the fewer bonuses you will receive. To make sure we get that end-of-year pay raise, we want to control complexity from the get-go by making our code and our internal systems loosely coupled. A composite application is just a collection of loosely coupled systems that is composited at runtime.

The advantages of a composite application are:

· Components can be developed independently

· Components can be swapped out more easily than if they were tightly coupled

· The application remains extendable, meaning you can more easily extend your application

· Maintenance is much clearer with less chance of breaking other components

· Unit testing is more feasible

Composite Application Entrails

There are many, many excellent resources about Prism available on the internet, so I’d rather not compete with them. I did, however, want to cover the basics, and at least how I’ve used it within the Laser Graffiti application.

The Shell Assembly – This contains the main entry point of the application and the initial or main UI container for the application. Before anything else, this assembly will initialize something called a bootstrapper. The bootstrapper simply does any pre-initialization that needs to be done before initializing the rest of the application.

Module Assemblies – Each module assembly can be considered a component. The bootstrapper initializes each component and then the module spins up any application services and adds any user interface to the shell. For instance, the MediaKit module handles the capture of the video and the AForgeModule handles the vision routines.

The Infrastructure Assembly – This contains any base classes, well known/shared interfaces or general infrastructure code. Simply put, most if not all modules will have reference to this.

Rendering the Graffiti

I needed a way to draw the graffiti to a projector. Overall, this will be accomplished by drawing the graffiti to a window. The window will be set to maximized with no border. I also needed to write the code to draw the graffiti. Initially, I just used the WPF InkCanvas control, but I found it wasn’t well tuned for what I wanted to do with it. So, I decided XNA would give the most performance and flexibility with the level of effects desired for drawing graffiti.

I had never really done any XNA development, so this project is certainly not built using best practices. The important part to notice is that the project is more or less a regular XNA application with the exception that the executable is added as a library reference. This is because we instantiate the XNA from our WPF application’s process. The following snippet of code accomplishes this:

private void ShowGraffitiWall(bool show)
{
    if (m_game != null)
        m_game.Exit();

    if (!show) 
        return;

    var t = new Thread((ThreadStart)delegate
    {
        m_game = new XnaGraffitiGame();
        m_game.SetScreen(m_lastScreen);
        m_game.Effect = m_lastGraffitiEffect;
        m_game.Run();
        m_game.Dispose();
    });

    t.SetApartmentState(ApartmentState.STA);
    t.IsBackground = true;
    t.Start();
}

We instantiate the XNA window this way because it allows our WPF code to send the XNA messages of where the laser pointer was detected.

Drawing with the GPU

A blanket statement about XNA is that it is a drawing API. XNA applications can get quite complex when you add in things like 3D or shaders, but for this article, I’m only covering the required simple stuff. That said, I do make the assumption that the reader has had at least a few hours of XNA experience.

Our primary goal is to render virtual graffiti using XNA. This means we need to be able to draw a line. I started by creating a PNG file that would act as a “brush.” In XNA, these 2D graphics are also known as sprites. The brush sprite looks like this:

Now we can use XNA’s SpriteBatch class to draw our brush. The first problem I had, though, was that most XNA tutorials show drawing the sprite, continue, clear the screen, draw the sprite, continue, etc. I needed to retain previous brush strokes, which means drawing onto an intermediate drawing surface (RenderTarget2D), then drawing that to the screen.

The next issue I had was that the line I was trying to draw was very broken. See, the laser detection code runs at a maximum 30 FPS (the limit of my web camera). If I moved the laser fast enough, the screen would just look like a bunch of random points were being drawn. I had to move the laser very slowly to make it appear like a line. I remember reading an excellent article detailing a project by Rick Barraza, which led me to a link to some popular algorithms for creating a line where your input is two, 2D points.

So, drawing a line required only a simple, two-step process of A) getting a new point from laser detection, and B) drawing a line from last_laser_coordinate to current_laser_coordinate. After putting that together, we can now produce some useful looking graffiti!

Adding Crazy Graffiti Effects

The excitement of drawing simple lines in XNA wore off pretty fast. After all, the GPU is capable of some pretty amazing things. What about drawing with fire? Fire isn’t boring:

The fire effect is done using a method slightly similar to the solid color sprite method described earlier. However, since it uses a particle system, the fire effect is much more complex. Each particle has its own shader to control position and color. With the time constraints of this project, I had modified code from this XNA tutorial. There weren’t many changes from the tutorial’s code, but a big hurdle was that the particles took 3D coordinates instead of the 2D coordinates I needed to draw the graffiti. I found a small snippet to handle that, though I was informed by other XNA gurus that there is an easier way to accomplish this. If you are curious, this is what it looks like (and it’s not for the faint of heart):

protected Vector3 ScreenPointToVector3(Point coords)
{
    Matrix viewMatrix = 
        Matrix.CreateTranslation(0, -25, 0) *
        Matrix.CreateRotationY(
            MathHelper.ToRadians(CameraRotation)) *
        Matrix.CreateRotationX(
            MathHelper.ToRadians(CameraArc)) *
        Matrix.CreateLookAt(
            new Vector3(0, 0, -CameraDistance),
            new Vector3(0, 0, 0), 
            Vector3.Up);

    float aspectRatio = 
        (float)m_game.GraphicsDevice.Viewport.Width /
            (float)m_game.GraphicsDevice.Viewport.Height;

    Matrix projectionMatrix = 
        Matrix.CreatePerspectiveFieldOfView(
            MathHelper.PiOver4,
            aspectRatio,
            1, 10000);

    Vector3 nearScreenPoint = new Vector3(coords.X, coords.Y, 0);
    Vector3 farScreenPoint = new Vector3(coords.X, coords.Y, 1);
    Vector3 nearWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            nearScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);
    Vector3 farWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            farScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);

    Vector3 direction = farWorldPoint - nearWorldPoint;

    float zFactor = -nearWorldPoint.Z / direction.Z;
    Vector3 zeroWorldPoint = nearWorldPoint + direction * zFactor;

    return zeroWorldPoint;
}

The End Result

Web-camera device selection:

Fine tuning the laser detection:

About Jeremiah

Jeremiah Morrill is a software developer and 2010 MVP living in Las Vegas, Nevada. He owns a software company called HJT, http://www.hjtcentral.com/, with a couple partners. There he focuses on multimedia and rich user interfaces with WPF and Silverlight. Jeremiah spends a lot of his free time learning, listening and helping others in the online development communities. 

Friend of Coding4Fun, René Schulte, has an full write-up on how to do some simple drawing, loading, and saving with Windows Phone 7.  René even used multi-touch as well!

He leverages the WriteableBitmapEx class that extends out the WriteableBitmap Silverlight class.

To do a PhotoChooser, the code is simpler than one may think.  Since you're app will navigate away, it works with events.

public MainPage()
{
   // ...

   // Init chooser
   photoChooserTask = new PhotoChooserTask();
   photoChooserTask.Completed += PhotoChooserTaskCompleted;
   
   // ...
}

private void ApplicationBarIconOpenButton_Click(object sender, EventArgs e)
{
   photoChooserTask.Show();
}

private void PhotoChooserTaskCompleted(object sender, PhotoResult e)
{
   if (e.TaskResult == TaskResult.OK)
   {
      // Load original image and invalidate bitmap so it gets newly rendered
      var bitmapImage = new BitmapImage();
      bitmapImage.SetSource(e.ChosenPhoto);
      Viewport.Source = bitmapImage;
      bitmap = null;
   }
}