Webcam Based Laser Tracking for Human-Computer Interaction
| | In this article, we will put together a program which will allow us to move the mouse cursor on our computers with a laser pointer, and even generate mouse clicks using only a webcam for computer vision. One really cool use of this would be with a projector. With a projector, you could use this application and turn its projected image into an interactive screen, and even open files/menus by just pointing at them with a laser pointer! |
| Ashish Derhgawen
Difficulty: Intermediate Time Required: 1-3 hours Cost: Free Hardware: Webcam, Laser Pointer |
I will be making use of Andrew Kirillov's motion detection code for image acquisition, and the AForge.NET framework, for certain image processing tasks. I'll start off by showing you how the image processing works for this program, and then I'll walk you through the important parts of the code. So, let's begin!
Image processing and laser tracking...
The laser tracking code works by finding the brightest pixel in the webcam's field of view. It is pretty much similar to the code I had written for my previous article - Laser Tracking Camera. However, since quick image processing is necessary for this program to work in real-time, I have modified the code a bit, and now it's much faster than before. The two image processing techniques I have used for optimization are - image cropping and image resizing.
Image cropping is basically just trimming off the edges of an image to keep unwanted things outside the camera's view. Image cropping also reduces the number of pixels in an image, and makes image processing fast. On the other hand, resizing images reduces the pixel resolution of an image. The pixel resolution of an image describes the resolution of an image with the set of two positive integer numbers, where the first number is the number of pixel columns (width), and the second is the number of pixel rows (height). (Examples: 320x 40, 640x480, 1024x768). An image from Wikipedia illustrates:
![clip_image002[4]](http://coding4fun.net/images/WebcamBasedLaserTrackingforHumanComputer_11629/clip_image0024_thumb.jpg)
The picture on the left is sampled at 20x20, and the image on the right is sampled at a higher pixel resolution, 100x100. Obviously, the higher resolution image has more detail. However, in image processing, while increased resolution results in greater information and detail, it comes at the price of memory and computing speed.
For cropping the video feed from a webcam, you can click-and-drag a selection rectangle around the area you would like to keep, and the program would remove everything else.
![clip_image003[4]](http://coding4fun.net/images/WebcamBasedLaserTrackingforHumanComputer_11629/clip_image0034_thumb.jpg)
The selection rectangle is a semi-transparent rubber band rectangle (picture above). A rubber band rectangle (aka focus rectangle), is a reversible rectangle that tracks with the mouse pointer while you move it around. This makes it easy to select the area you want to work on. If you are working with a projector, you can easily select its projected area, and the program would map its coordinates to the coordinates on your computer screen, and remove everything else.
Here's how you can make a rubber-band rectangle between two points (say x1,y1 and x2, y2):
//Compute width and height of rectangle
int width = Math.Abs(x2 - x1);
int height = Math.Abs(y2 - y1);
//Decide x and y of rectangle
int x = 0, y = 0;
if (x1 < x2)
x = x1;
else if (x1 > x2)
x = x2;
if (y1 < y2)
y = y1;
else if (y1 > y2)
y = y2;
//Draw the rectangle
Rectangle rect = new Rectangle(x, y, width, height);
Graphics g = Graphics.FromImage(image);
g.FillRectangle(new SolidBrush(Color.FromArgb(60, 184, 184, 0)), rect); //Draws a semi-transparent rectangle using alpha blending
g.Dispose();
You will find this code in the ProcessFrame event in MotionDetector1.cs. Here, x1 and y1 are updated in the MouseDown event of the camera window, whereas x2 and y2 are updated in the MouseMove event.
After we know how big the selected area is, we can trim the edges by using the Bitmap.Clone method:
Bitmap tmpImage0 = image.Clone(new Rectangle(x, y, width, height),
System.Drawing.Imaging.PixelFormat.Format24bppRgb);
For resizing the video feed by a certain factor, adjust the resize factor numeric up/down control. If you take a resize factor, say 2, then the program would divide the length and width of the webcam images by 2, and the resulting images would actually be 2^2 = 4x smaller. For example, if your webcam captures images of size 320 by 240, the total number of pixels in each image would be 320 x 240 = 76800. If we resize these images by a factor of 2, the resulting 160 by 120 size images would contain only 160 x 120 = 19200 pixels (which is 4x less than 76800).
![clip_image004[4]](http://coding4fun.net/images/WebcamBasedLaserTrackingforHumanComputer_11629/clip_image0044_thumb.jpg)
Here's how we can resize an image by a given factor using the Resize filter in the AForge.Imaging library:
AForge.Imaging.Filters.Resize resize = new Resize(image.Width / resizeFactor, image.Height / resizeFactor, InterpolationMethod.NearestNeighbor);
Bitmap tmpImage1 = resize.Apply(tmpImage0);
Just keep in mind that the lesser you resize your images, the more tracking resolution you'd get while tracking lasers. However, since tracking resolution comes at the cost of memory and computing speed, pick a resize factor which works best for your computer and webcam. A resize factor of 2 or 3 usually works fine for me.
After we're done with cropping and resizing, the program can go through the pixels on the image and find the brightest pixel (I'm assuming this would be the laser dot):
for (int y = 0; y < 240; y++)
{
for (int x = 0; x < 320; x++)
{
byte red, green, blue;
red = uBitmap.GetPixel(x, y).red;
green = uBitmap.GetPixel(x, y).green;
blue = uBitmap.GetPixel(x, y).blue;
float brightness = (299 * red + 587 * green + 114 * blue) / 1000;
if (brightness > certainValue)
// Do something
}
}
Here, uBitmap, is an instance of the UnsafeBitmap class I had discussed in my previous article.
NOTE: If you find brightness tracking unreliable in certain lighting conditions, change the above code to perform color detection instead of brightness detection by writing something like "if ((red > 220) && (green < 170) && (blue < 170))" instead of writing "if (brightness > certainValue)".
After our program has the x and y coordinates of the laser dot, it can compute its position on the screen:
//Get screen width and height
int screenWidth = Screen.GetBounds(this).Width;
int screenHeight = Screen.GetBounds(this).Height;
float cursorX, cursorY;
//Compute location of laser dot into camera coordinates
cursorX = ((float)screenWidth / imageWidth) * x;
cursorY = ((float)screenHeight / imageHeight) * y;
//Set cursor position
Cursor.Position = new Point((int)cursorX, (int)cursorY);
Now that you know how the code works, lets move on to the next section, in which I will tell you how to use it.
Some notes on using the software
- The program searches for the brightest pixel in the camera's field of view, so the lighting conditions of your room can affect its performance. Adjust the brightness threshold and lighting conditions so that nothing (except the laser) exceeds the brightness threshold. If this doesn't work, change the brightness detection code to perform color detection instead (as described earlier).
![clip_image005[4]](http://coding4fun.net/images/WebcamBasedLaserTrackingforHumanComputer_11629/clip_image0054_thumb.jpg)
- If you want to reset video cropping to normal, just right-click anywhere in the camera window.
- For clicking on something with a laser pointer, just turn off the laser when the mouse cursor is over the item you want to click on. Turning the laser off simulates a left button, single-click. You might also want to change your Folder Options to allow files/folders to be opened with single-clicks.
- While selecting a resize factor, remember that computing speed would come at the cost of tracking resolution. Say, for example, if I use a resize factor of 1, I'd get pretty good tracking resolution, but my program would slow down to a crawl. On the other hand, if I pick a resize factor greater than 4 or 5, I'd get a decent frame rate, but horrible tracking resolution. These values could be different for you (depending on your webcam and computer). For my computer, a resize factor of 2 or 3 works fine and I get about 27-30 frames/second.
Conclusion
We have reached the end of this article, and I hope you enjoyed reading it. Now, here's some homework for you: Try implementing features like double clicking, right clicking and click-drag. If you've watched the video for this article, you must have noticed that I was drawing stuff on MS-Paint using this program. For accomplishing this, I had to modify my code to perform click-drag. If you can think of an easier, laser-based way to switch between different clicking modes, I think that would be very cool. So, use your ideas, and if you end up doing something cool, I'd love to hear about it. :) Have fun!
About the Author
Ashish Derhgawen is an IT student, currently living in New Delhi, India. He has been coding since fourth grade. Some of his other interests are harmonica playing, wildlife and cricket. When he’s not at school, he spends his time working on unusual projects related to robotics, webcams, and electronics besides others. You can reach Ashish through his blog at http://ashishrd.blogspot.com.
