UK StudentZine Blog : Silverlight

Pushing Paint applications into Web 2.0 using Silverlight

ukstudentzine — Thu, 09 Apr 2009 15:51:00 GMT

Paint applications on the desktop has been developing for a long time however there are few available on the internet, even less that offer the same control that you would find on a desktop paint application, even one as simple as Paint. Let me show you how you can make your own online paint application in Silverlight.

One of the reasons this hasn’t been done before in Silverlight is the way it renders graphics and shapes. It uses Vectors; which are great if you want to create shapes that look good at any resolution or zoom level but not so good for us. Why? Well you can’t say what colour is at point [5, 5] for example if we load in an image and then draw a shape on top. Or what if we want to draw a pen line, well we could use the Ink tools but they are all separate and distinct and in the end we can’t bring them all together in a saveable image. Thus we need to create our own pixel based graphics library that incorporates the required functionality.

Luckily most of the hard work was figured out for us by Joe Stegman who incorporated a PNG encoder to link between the Silverlight Image control and a pixel level graphics array of bytes accessible via Get and Setpixel functions.

We need to increase the functionality of this, so I went through the GDI+ graphics library and copied most of the function signatures for the ones that would be useful (or easy to implement) and then added a few additional ones that would help the applications performance and scalability, below is the class diagram for the finished graphics library.

As you can see we have quite the selection of Methods, quite a few of them we don’t actually require for this project however I thought it would be useful if the graphics library was used elsewhere.

Now let’s talk about how the graphics library actually works and what goes on under the hood. The image data is stored in a byte array. Each screen pixel requires 4 bytes (representing A, R,G,B) and then a parity byte at the end of each row, thus for an image of size [w, h] the required bytes are (w * 4 + 1) * h. Set and Get Pixel process the coordinates and read / write from a position in the byte array.

Some modifications were made to the PNG encoding library, such as altering the way it copies arrays to use System.Array.Copy() to increase speed, as we will constantly be updating as fast as possible every millisecond counts!

One of the first functions we needed to create was a DrawLine() function; the default signature of which being DrawLine(Color color, int x1, int y1, int x2, int y2) this function would effectively walk the line in both horizontal and then vertical 1 pixel increments and plot the resulting pixel using Bresenham's algorithm, this gave a fast and effective algorithm however it lacks anti-aliasing though for this implementation that is acceptable. To improve performance we bypass SetPixel and set the bytes directly otherwise we would constantly have to recalculate the colour values. Once we have a line drawing function in place others come naturally, such as DrawRectangle() etc.

public void DrawLine(Color color, int x1, int y1, int x2, int y2)

{

byte red = color.R;

byte green = color.G;

byte blue = color.B;

byte alpha = color.A;

int start;

// Uses Bresenham's algorithm to draw a line

// Work out the start, end and step points

double startX = x1;

int endX = x2;

double yStep = (y2 - y1) / (endX - startX);

double yPos = y1;

if (endX < startX)

{

startX = x2;

endX = x1;

yStep = (y1 - y2) / (endX - startX);

yPos = y2;

}

// First we traverse the x plane

for (int x = (int)startX; x < endX; x++)

{

start = _rowLength * (int)yPos + x * 4 + 1;

if (x < Width && yPos < Height && x > 0 && yPos > 0)

{

// Set the buffer directly for increased speed

_buffer[start] = red;

_buffer[start + 1] = green;

_buffer[start + 2] = blue;

_buffer[start + 3] = alpha;

}

yPos += yStep;

}

// Secondly we traverse the y plane

double startY = y1;

int endY = y2;

double xStep = (x2 - x1) / (endY - startY);

double xPos = x1;

if (endY < startY)

{

startY = y2;

endY = y1;

xStep = (x1 - x2) / (endY - startY);

xPos = x2;

}

for (int y = (int)startY; y < endY; y++)

{

start = _rowLength * y + (int)xPos * 4 + 1;

if (xPos < Width && y < Height && xPos > 0 && y > 0)

{

// Set the buffer directly for increased speed

_buffer[start] = red;

_buffer[start + 1] = green;

_buffer[start + 2] = blue;

_buffer[start + 3] = alpha;

}

xPos += xStep;

}

Drawing an ellipse is slightly more complex, for this I used a modified version of the ellipse rendering code found on Wikipedia. It creates lines connecting 72 splines around the ellipse at the key positions; I choose 72 as an arbitrary number that provides a high level of detail.

private Point[] GetEllipsePoints(int x, int y, int width, int height)

{

// Iterate through 72 points

int steps = 72;

double angle = Math.PI / 4;

double centerX = (double)width / 2;

double centerY = (double)height / 2;

// Store for the keypoints around the Ellipse

List<Point> points = new List<Point>();

// Angle is given by Degree Value

double beta = -angle * (Math.PI / 180); //(Math.PI/180) converts Degree Value into Radians

double sinbeta = Math.Sin(beta);

double cosbeta = Math.Cos(beta);

for (double i = 0; i < 360; i += 360 / (double)steps)

{

double alpha = i * (Math.PI / 180);

double sinalpha = Math.Sin(alpha);

double cosalpha = Math.Cos(alpha);

double X = x + centerX + (centerX * cosalpha * cosbeta - centerY * sinalpha * sinbeta);

double Y = y + centerY + (centerX * cosalpha * sinbeta + centerY * sinalpha * cosbeta);

points.Add(new Point(X, Y));

}

return points.ToArray();

}

Bucket fill proved interesting, from a starting point it would iterate through till an area enclosed by the same original colour was filled. This was achieved by first creating a copy of the buffer store, though in the form of a 2D colour grid, and then using a queue to process each item. The first item in the queue is the original point pasted to the function. The queues iteration loop would check the four surrounding pixels to see if they are the same as the origin pixel colour and if so, would change its colour and add the position to the queue if the square hadn’t already been processed. Once the queue is empty the bucket fill is complete.

public void BucketFill(Color color, int x, int y)

{

// Store the colour of the origin pixel

Color OriginPixelColour = GetPixel(x, y);

// First to make things easier we need to create a backup of the graphic

Color[,] _backupGrid = CreateColourGrid();

bool[,] ProgressGrid = new bool[Width, Height];

// Create a queue of all the points we need to check through

Queue<Point> ProgressQueue = new Queue<Point>();

// Set the initial queue item to the origin

ProgressQueue.Enqueue(new Point(x, y));

ProgressGrid[x, y] = true;

// Iterate through the queue till there are no items

{

// Dequeue the item

Point QueueItem = ProgressQueue.Dequeue();

int X = (int)QueueItem.X;

int Y = (int)QueueItem.Y;

// Change the colour

SetPixel(X, Y, color);

// Now go through the different directions

if (Y >= 1 && !ProgressGrid[X, Y - 1] && _backupGrid[X, Y - 1] == OriginPixelColour)

{

ProgressGrid[X, Y - 1] = true;

ProgressQueue.Enqueue(new Point(X, Y - 1));

}

if (Y < Height - 1 && !ProgressGrid[X, Y + 1] && _backupGrid[X, Y + 1] == OriginPixelColour)

{

ProgressGrid[X, Y + 1] = true;

ProgressQueue.Enqueue(new Point(X, Y + 1));

}

if (X >= 1 && !ProgressGrid[X - 1, Y ] && _backupGrid[X - 1, Y] == OriginPixelColour)

{

ProgressGrid[X - 1, Y] = true;

ProgressQueue.Enqueue(new Point(X - 1, Y));

}

if (X < Width- 1 && !ProgressGrid[X + 1, Y] && _backupGrid[X + 1, Y] == OriginPixelColour)

{

ProgressGrid[X + 1, Y] = true;

ProgressQueue.Enqueue(new Point(X + 1, Y));

}

while (ProgressQueue.Count != 0);

}

Initialisation for the image was moved to the constructor method to improve perceived performance as there would not be a slight lag the first time the image is drawn to.

public Bitmap(int width, int height)

{

this.Width = width;

this.Height = height;

// Initalise everything

_rowLength = _width * 4 + 1;

_buffer = new byte[_rowLength * _height];

_init = true;

}

Other functions such as DrawImage() just transfer the bytes from a previously defined Bitmap to the current object, due to their simplicity I thought it would be best to carry on and not go into much detail about them.

So now we have a fully functional dynamically generated Bitmap class. Moving on we must now go ahead and implement the front end this plugs into. To simplify things the window will be split into several Grids, one to house the main paint window, status bar, tool bar and menu. For the moment we will concentrate on the main window and implement that.

In the main windows code we need to declare two bitmaps using the library we just created and then one using Silverlights graphics library.

public Bitmap(int width, int height)

{

this.Width = width;

this.Height = height;

// Initalise everything

_rowLength = _width * 4 + 1;

_buffer = new byte[_rowLength * _height];

_init = true;

}

_Frontbuffer is where the main drawing will take place before it is drawn to the screen; _Backbuffer is used to hold temporary graphics information for certain operations and brushes.

The image itself will consist of an Image control ‘DrawSurface’ surrounded by 3 Rectangles for the different shadows to give the image a 3d relief effect ‘DrawSurfaceShadowRight’, ‘DrawSurfaceShadowBottom’ and ‘DrawSurfaceShadowCorner’.

The shadow fill of the rectangles is created by a LinearGradientBrush going from Gray to LightSteelBlue (the paint windows background colour). The positions of the Shadows are based on the dimensions of the internal drawable bitmap _FrontBuffer although offset northeast by 5 pixels to give the effect the canvas is hovering. We package this up as a function that we call whenever the dimensions of _FrontBuffer changes.

private void SetupShadows()

{

// Set the dimensions and margins

DrawSurfaceShadowRight.Height = _Frontbuffer.Height - 5;

DrawSurfaceShadowRight.Margin = new Thickness(30 + _Frontbuffer.Width, 35, 0, 0);

DrawSurfaceShadowBottom.Width = _Frontbuffer.Width - 5;

DrawSurfaceShadowBottom.Margin = new Thickness(35, 30 + _Frontbuffer.Height, 0, 0);

DrawSurfaceShadowCorner.Margin = new Thickness(30 + _Frontbuffer.Width, 30 + _Frontbuffer.Height, 0, 0);

// Set up the gradient colours and offsets

GradientStopCollection _GSC = new GradientStopCollection() {

new GradientStop() { Color = Colors.Gray, Offset = 0 },

new GradientStop() { Color = Color.FromArgb(255, 176, 196, 222), Offset = 1 } };

// Set the right, bottom and corner gradients

DrawSurfaceShadowBottom.Fill = new LinearGradientBrush(_GSC, 90);

DrawSurfaceShadowRight.Fill = new LinearGradientBrush(_GSC, 0);

DrawSurfaceShadowCorner.Fill = new LinearGradientBrush(_GSC, 45);

}

Different tools in Silverpaint are used by storing the ID for the tool which is in the form of a string eg. Pen, Line, BucketFill etc. The type of tool is checked on various mouse events on the Drawable surface depending on how they effect it.

The source code for SilverPaint is available for download at SilverPaints page on MSDN.

Mix Conference 2009

ukstudentzine — Thu, 09 Apr 2009 15:50:00 GMT

The Mix Conference held each year in Las Vegas is a showcase for Microsoft’s vision of internet technologies and this year was no exception. It bridges the gap between Web professionals and industry leaders in a way that allows them to explore the future of the web together.

This year it was held between the 18^th and 20^th of March. For those not able to attend the conference videos on demand of various sessions have been made available to download and view on the Mix Website. http://visitmix.com/

There was a lot to talk about this year at the initial Keynote, held by Scott Guthrie and Bill Buxton. Formost on the agenda was the forthcoming Version of Silverlight 3, now available for download in Beta 1. To continue the ‘next generation web’ experience Internet Explorer 8 was finalised and demonstrated; providing a faster, easier and more secure web browsing experience.

If you want to try out the Beta of Silverlight 3 then pop over to MSDN and download the Visual Studio Tools

Visualising media search better (Part 2) - Kevin Pfister

ukstudentzine — Tue, 17 Feb 2009 17:31:00 GMT

The way Image search results look now hasn’t really changed much in the last few years, I ask the simple question ...why? In this article I want to try something that hasn’t been done before; combining Deep Zoom and Live Image Search. I call it DeepSearch ...Sounds awesome!

In the last article we used a pseudo 3D effect in Silverlight to show one way of visualising Images we have searched for.

However just showing the images was a bit bland, even with a 3d effect. We needed to actually be able to manipulate the images better, and so people I give you DeepSearch!

This has only been made possible by the changes made to the MultiScaleImage control (Deep Zoom to us common people) within Silverlight Version 2 which allows us to dynamically generate our own MultiScaleTileSource. In the ye olde days the MultiScaleImage control would link to an external XML file via the Source property; however we can override this via the abstract class MultiScaleTileSource which has a method GetTileLayers() that we can override.

protected abstract void GetTileLayers(

int tileLevel,

int tilePositionX,

int tilePositionY,

IList<Object> tileImageLayerSources

)

Where this gets interesting now for us is creating the Dynamic tiles that we are going to supply to the MultiScaleImage control. Silverlight 2 doesn’t contain really anything in the way of pixel based bitmap manipulation client side, so what we need to do is push all that code over to the server. Mike Ormond and Mike Taulty have both done great blog posts that explain this problem and a way to get round it. I will be using the same type of methods they suggest. We will remote the requests to server side code so using an ASP.Net ashx handle so that we can use System.Drawing to enable us to have the bitmap manipulation we so desperately need, the only problem as both the Mike’s so rightly point out is the problem with the lack of caching causing server side congestion which will cause our code some problems later. Now to start off with I used most of the code base downloadable from Mike Taulty’s blog entry to speed things up a bit... here’s it running with quite a lot of changes made to the Silverlight front end. (One of the ideas of how it could have looked)

Some of you may notice that in Mikes DeepZoom image only showed the number whereas mine also shows the text searched for; in this case Layer. All that had to be done was simply link send the search text to the ashx handler along with the rest of the deep zoom configurations.

string source =

string.Format(

"http://localhost:62586/ClientBin/handler.ashx?" +

"tileLevel={0}&tilePositionX={1}&tilePositionY={2}&tileSize={3}" +

"&imageWidth={4}&imageHeight={5}&imageQuery={6}",

tileLevel, tilePositionX, tilePositionY, tileSize,

imageWidth, imageHeight, QueryString);

Having the bitmaps being generated server side also has a benefit to us as well; we will also be processing the search results server side as well due to Silverlight using a subset of System.Xml and also is a simple way to negate Silverlight’s Cross-Domain problem.

To use the Live Search API’s as part of our project we need to download the Live Search SDK and then acquire an AppID; for more information on the Live Search SDK you can visit the site.

There is a great example as part of the Live Search SDK which shows how to perform a Live Image Search and display the results; we only need to alter the context as it has been designed to run from a console rather than a web service... easy enough you would think.

However there is a big problem that we will have to overcome, the example I mentioned above will just search for a set of images and display the links to the results. If we plug this into the Server code; based on the code example Mike T provides then every time we want to change the depth or viewport we would have to perform the same search again and download the same set of images; thus we need to implement a simple image caching system on the server.

To do this we need to implement SessionState variables in our ashx handler class. This can be done by firstly importing System.Web.SessionState and having the class inherit IRequiresSessionState. What we are going to store in the SessionState is the search results of a given search and also the images that are retrieved. Now this will be A LOT of data if it were pushed out onto a production server however I am only using this method as this is really only a proof of concept. To store the search results I am made a struct to store the information retrieved from a search.

private struct SearchItem

{

public string MediaURL;

public string PageTitle;

public string PageURL;

public Size MediaSize;

}

We need to replace the example Live Search SDK code so it will now store search items in this type of struct and then store them as a List<SearchItem> in the SessionState using the SearchQuery as the key. Also as a sidenote, the SessionState is saved as part of the HttpContext we need to promote the variable to a class variable rather than local.

// List of the image results

List<SearchItem> _SearchItems = new List<SearchItem>();

// Save the Image results.

foreach (XmlNode result in results)

{

SearchItem _Result = new SearchItem() {

MediaURL = result.SelectSingleNode("./mms:MediaUrl", nsmgr).InnerText,

PageTitle = result.SelectSingleNode("./mms:Title", nsmgr).InnerText,

PageURL = result.SelectSingleNode("./mms:Url", nsmgr).InnerText,

MediaSize = new Size(

int.Parse(result.SelectSingleNode("./mms:Width", nsmgr).InnerText),

int.Parse(result.SelectSingleNode("./mms:Height", nsmgr).InnerText)),

ThumbnailURL = result.SelectSingleNode("./mms:Thumbnail/mms:Url", nsmgr).InnerText };

_SearchItems.Add(_Result);

}

// Save the results to the session state

httpContext.Session[queryString] = _SearchItems;

Now to test if this all work I changed the rendering code so instead of displaying text in the deep zoom it would instead display the first image of the search results list; in this case... Bill Gates

Now comes the big question; how do we want to display the images within the Deep Zoom? To give you one idea of what we could do is just panning the images across and then wrapping around at the edges, and so I give you the base of our deep zoom search.

One thing I had to do which I found very interesting was that if I just let told the graphics device to render the bitmap that sometimes it wouldn’t render it to the correct size so I made it render to the size that the search engine returned instead. To make transfer between server and client a little faster I also altered the context.Response.ContentType of the handler response to use jpeg instead of png, hopefully the lossy compression shouldn’t be noticable too much when compared to the bandwidth savings.

context.Response.ContentType = "img/jpeg";

...

bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);

Let’s go back to the UI now seeing as the search functionality is almost done. Now at the start I mentioned that the project was based on Mike Taulty’s underlying codebase, however he didn’t implement the full set of Deep Zoom mouse controls, so we have to add in MouseWheel support which isn’t as hard as you would think as any project created by Deep Zoom Composer already contains this code.

With some final adjustments to the UI with a nice shiny welcome page; making it more like a normal search engine I think I’ve done a good job on dull winter’s afternoon.

Follow the continuing development of DeepSearch at my Blog: www.blogplusequals.blogspot.com

If you want to try out DeepSearch, download the source code from MSDN

A better way to visualising media search (Part 1)

ukstudentzine — Fri, 16 Jan 2009 14:30:00 GMT

Search... the most important aspect of the internet; without it no one would be able to explore easily the vast amount of content available in an efficient manner, but could it look better?

Over the years search has come a long way, improving its algorithms and search techniques with the addition of other search categories such as image and video. We have reached the point now where the quality of results are sublime yet the search interface is still little different from where we originally started, therefore it’s time for a face lift.

So how do we go about this then? Well most search engines nowadays such as Live Search and Google have their own API sets that allow developers to utilise their search tools, then we need a way to merge the results and then output them in a graphically appealing way. There are several search visualisation plugins available already online but I thought I would try and make my own.

All this as usual I’ll be doing in Silverlight, allowing us to create a very rich and powerful interface. The only problem is that Silverlight doesn’t support much in the way of 3d graphics so we have to create a pseudo 3d impression.

The 3d effect I will be using is a common one that gets used a lot these days; a wall of images that you can scroll along, creating a highly tactile surface. When you click on an image it will scale it later and give more information about it.

Now for the coding...

One of the first problems I had was trying to create the boxes to house each of the image search results in the visualisation. The problem I had was that the shape couldn’t be achieved just by using a rectangle and applying transformations to it, I had to use the Path shape to create the custom shape I required to give the pseudo 3d effect I wanted.

Another problem that I have run into a few times is the quality of image scaling in Silverlight, although in WPF you can adjust these settings it seems locked in Silverlight. In our application this is a problem when the images become smaller as they go off into the pseudo distance, to get round this I altered the opacity of the images as they go further away till they fade out completely just before they hit the scaling problems. Once completely faded or off screen I make the objects visibility collapsed thus to increase rendering performance of the application.

Coming in the next issue I will be adding to the visual effect and adding tagging so you can play with the images in the search that you like along with talking more about what goes into search itself and how for images finding what you want can be even harder. Along with this I will be giving out the source code for you to play with, however for the time being you can see the progress of the application by visiting my blog at www.blogplusequals.blogspot.com