March, 2010

One common question I hear is:

How can you consume the raw image provided by the core APIs in a WPF application? How do you display it on the screen?

I just extracted it from the code I used to “process the image” created by the cup on the monster demo.

Here is a simple answer to that question, the code is “AS IS” – use at your own risk.

Follow these steps:

1. Add the Microsoft.Surface.Core assembly to your project so you can reference it.

2. After your SurfaceWindow is initialized, do the necessary setup so that you can start receiving Raw Image events and data.

// This code released under the terms of the

// Microsoft Public License (MS-PL, http://opensource.org/licenses/ms-pl.html.)

/// <summary>

/// Default constructor.

/// </summary>

public SurfaceWindow1()

{

InitializeComponent();

SourceInitialized += new EventHandler(InitializeCore);

// Add handlers for Application activation events

AddActivationHandlers();

}

void InitializeCore(object sender, EventArgs e)

{

// Create a target for surface input, and start

// receiving normalized raw images.

            contactTarget = new ContactTarget(
                            new WindowInteropHelper(this).Handle,
                            EventThreadChoice.OnCurrentThread);

// See step 3 for definition of OnContactTargetFrameReceived.

contactTarget.FrameReceived += OnContactTargetFrameReceived;

contactTarget.EnableInput();

contactTarget.EnableImage(ImageType.Normalized);

}

3. Now that you are receiving the raw image, do something with it. I my case I write it to a WriteableBitmap so that I can use it as a source to an Image in my visual tree. I write to the bitmap every 100 milliseconds.

// This code released under the terms of the

// Microsoft Public License (MS-PL, http://opensource.org/licenses/ms-pl.html.)

// Assuming the following members of the class

private byte[] normalizedImage;

private ImageMetrics normalizedImageMetrics;

private static WriteableBitmap writeableBitmap;

// remembers the last time we showed the raw image.

private long oldTimeStamp;

/// <summary>

/// Handler for the FrameReceived event. Here we get the

/// rawimage data from FrameReceivedEventArgs object.

/// </summary>

/// <param name="sender">ContactTarget that received

/// the frame</param>

/// <param name="e">Object containing information about

/// the current frame</param>

private void OnContactTargetFrameReceived(

object sender,

FrameReceivedEventArgs e)

{

long now = DateTime.Now.Ticks;

lock (this)

{

long ticksDelta = Math.Abs(now - oldTimeStamp);

// Update image every 100 milliseconds.

if (ticksDelta > 100 * 10000)

{

int paddingLeft, paddingRight;

Rect imageBound = new Rect();

imageBound.X = 0;

imageBound.Y = 0;

imageBound.Width = 1024;

imageBound.Height = 768;

if (e.TryGetRawImage(

ImageType.Normalized,

(int)imageBound.Left,

(int)imageBound.Top,

(int)imageBound.Width,

(int)imageBound.Height,

out normalizedImage,

out normalizedImageMetrics,

out paddingLeft,

out paddingRight))

{

WriteToImage(normalizedImage,
normalizedImageMetrics);

imageToShow.Source = writeableBitmap;

}

oldTimeStamp = now;

}

/// <summary>

/// The WriteToImage method updates the

/// WriteableBitmap by using unsafe code to write

/// a pixel into the back buffer.

/// </summary>

/// <param name="normalizedImage">The raw image</param>

/// <param name="metrics">The size of the raw image</param>

static void WriteToImage(byte[] image, ImageMetrics metrics)

{

int maxRow = metrics.Height;

int maxCol = metrics.Width;

if (writeableBitmap == null)

{

writeableBitmap =

new WriteableBitmap(

maxCol,

maxRow,

96,

PixelFormats.Bgr32, null);

}

writeableBitmap.Lock();

unsafe

{

for (int y = 0; y < maxRow; y++)

{

for (int x = 0; x < maxCol; x++)

{

int color_data = 0;

int pBackBuffer = (int)writeableBitmap.BackBuffer;

pBackBuffer += y * writeableBitmap.BackBufferStride;

pBackBuffer += x * 4;

// Set Red, Green, Blue respectively

color_data = image[x + y * metrics.Stride] * 2 << 16;

color_data |= image[x + y * metrics.Stride] * 2 << 8;

color_data |= image[x + y * metrics.Stride] * 2 << 0;

*((int*)pBackBuffer) = color_data;

}

try

{

writeableBitmap.AddDirtyRect(new Int32Rect(0, 0, maxCol, maxRow));

}

catch (Exception ex)

{

string s = ex.ToString();

}

finally

{

writeableBitmap.Unlock();

}

That was it! I hope this is helpful. Let me know if you have questions.

Luis Cabrera
Platform Program Manager

Microsoft Surface

For the uninitiated to the Penny Arcade Expo, I can best describe it as a gaming conference by the fans, for the fans. This conference is normally held in Seattle to sell out and capacity crowds. This year Gabe and Tycho gave the fans (which includes themselves) even more by expanding to Boston with PAX East. This has made it accessible to even more fans who couldn't make the trek to Seattle. Intrigued? There's a brilliant blog post that puts PAX in perspective.

I’ve been talking a lot about the Carnegie Mellon University grad student project that brought the Dungeons & Dragons proof-of-concept to Microsoft Surface. You can learn more about WHY (and read the disclosures that it’s an academic project and not a shipping application) in my previous blog post on the topic. It will help put this into perspective as to why it is relevant to Surface.

This is the second time Surface has been at PAX. The reception is just as warm here in Boston. We’re located in the tabletop gaming area, room 111. (Tabletop HQ & Library) Fans of Surface or D&D can sit down and play with the Carnegie Mellon team. If you’re at PAX, you really should stop by – even if D&D isn’t your cup of tea – you’ll find out just how accessible Surface can make this game for the uninitiated.

I’ll be reporting in throughout the weekend, so make sure to check back in.

- Eric (follow Surface on Twitter and Facebook)

Microsoft PixelSense Blog

Raise of Hands: Do you use the Surface Simulator? Do you use the Surface Simulator APIs?

Using the RawImage in a WPF application.

Surface and Carnegie Mellon at PAX with D&D