This is the second post of the "Reallistic Human Movements with Silverlight" series. I'll spend a little time talking about the .bvh file format structure. I’m still working on the code and that’s why I’ll make this post short in order to get back to the mocap code. There is a link to more relevant info at the bottom.
I updated to the motion capture source code (download attachment to this blog) that contains a major fix related to joint rotation.
The latest sample is here: http://nokola.com/mocap.html
Some people thought the sample is a saved video – it is not. The sample shows rendering of 3D motion capture data at real time on Silverlight. Try out the new “body fat” slider! If you click “Add another person” then “Show Info” you’ll see the x, y, z angles for joint rotations in realtime.
Before going on, a quick overview of one way animation studios do motion capture today:
1. A human (e.g. actor or martial artist) is placed in a special costume for motion capture
2. There are numerous cameras surrounding that person (e.g. 24 cameras) that take videos of the person
3. Software processes the videos to "understand" where the person's hands, feet, etc are and saves this information in a motion capture file
One such file format is .BVH (Biovision motion capture).
The bvh contains 2 sections:
1. Skeleton data describing the hierarchy and initial pose of the human skeleton whose motion is being captured
2. Frame data describing how the human's joints move over time
The skeleton data looks something like this:
HIERARCHY
ROOT Hips
{
OFFSET 0.00 0.00 0.00
CHANNELS 6 Xposition Yposition Zposition Zrotation Xrotation Yrotation
JOINT Chest
{
OFFSET 0.00 5.21 0.00
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT Neck
{
OFFSET 0.00 18.65 0.00
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT Head
{
OFFSET 0.00 5.45 0.00
CHANNELS 3 Zrotation Xrotation Yrotation
End Site
{
OFFSET 0.00 3.87 0.00
}
}
}
…
For each joint, we have its 3D offset relative to its parent joint (Hips is the parent of Chest) and its channel information.
You can read more about this here: http://www.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/BVH.html
One thing missing from the above page is that it is very important to note the order of the channels –this is the order in which you have to rotate the joints. For example order of “Zrotation Xrotation Yrotation” means that you have to rotate by z,x,y in that order.
The bug that I fixed in my code was related to that. If you don’t follow the order in the file, you’ll end up with weirdly moving figures or jitter in the playback.
The second part of the .bvh file is the frame information – I encourage you to read about it here: http://www.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/BVH.html
This resource explains it pretty well!
More Info
One person also asked me “how do I capture my own motion data?” Usually this is pretty costly (e.g. $10,000-$20,000), but could be made with lower cost too!
I’ll talk a little bit about this in the next post. In the meantime you can search for “motion capture single camera” or "markerless motion capture" on the internet. This seems to be the cheaper way to do it (cheaper as in $50-$500). Recent scientific developments (last few months) in markerless motion capture really open the possibility for the public to use this technology!
Attachment(s): MocapView.zip
Hoho! yes! How to make reallistic human movements in Silverlight?
Check out this stickman animation demo here: http://www.nokola.com/mocap.html
Download source code for the above sample in the attachments to this post below.
You're looking at 3D realtime animation of stick man figures using motion data found on the internet.
The motion capture viewer took me the whole day yesterday - still nothing compared to doing these animations manually frame by frame.
As a start, you need movement data (motion capture or mocap data). You can find a lot of that online. This site http://sites.google.com/a/cgspeed.com/cgspeed/motion-capture/3dsmax-friendly-release-of-cmu-motion-database contains thousands of human motions - e.g. jump, punch, kick, wash windows, mop floor, and other.
I used the .BVH files - they have a short description of the skeleton being animated (joints, head etc) as well as rotational information (e.g. "wrist should be rotated at 50 degrees at time X")
Since this is a work in progress, I'm just publishing this quick blog post - more info will come later, because I want to make the animations transition into each other.
In the future I'll talk about the motion capture file formats and some challenges I faced (and are still facing!) implementing the above sample.
Here's a quick overview of motion capture from Wikipedia:
"Motion capture, motion tracking, or mocap are terms used to describe the process of recording movement and translating that movement onto a digital model. Initially invented in Scotland, it is used in military, entertainment, sports, and medical applications. In filmmaking it refers to recording actions of human actors, and using that information to animate digital character models in 3D animation. When it includes face, fingers and captures subtle expressions, it is often referred to as performance capture."
http://en.wikipedia.org/wiki/Motion_capture
Motion capture is used in games, movie special effects, and now in demos like the one above :)
Attachment(s): MocapView.zip
I decided to give a third spin of Joe Stegman's dynamic image generation code. This time, it's many times faster (about 10x) than the original implementation and the png is generated in-place (no recoding necessary).
I used some hacky optimizations :) For example, replacing the big CRC loop with a constant (0), definitely improves speed a lot! :)
There are no more intermediate memory streams or buffers: whenever SetPixel() is called, the value is written directly into PNG. Also all the header and size information is created just once. That also helped performance quite a bit.
With the new code my raindrops sample runs 20% faster (90% of the time used in calculating the drops): www.nokola.com/raindrops
Download the source code containing the new PngEncoder class here: www.nokola.com/sources/water.zip
Here's how to use the new image generator class:
PngEncoder surface = new PngEncoder(640, 480); // image dimension
surface.SetPixelSlow(40, 30, 200, 135, 32, 255); // set pixel at (40,30) with color RGBA=(200,135,32,255)
// draw a white horizontal line fast
int rowStart = surface.GetRowStart(30);
for (int i = 0; i < 10; i++) {
// SetPixelAtRowStart() is good for blitting/copying existing images onto this one
surface.SetPixelAtRowStart(i + 40, rowStart, 255, 255, 255, 255);
}
// display the image
BitmapImage img = new BitmapImage();
img.SetSource(surface.GetImageStream());
imgWater.Source = img; // this is just a normal Silverlight Image
And, here's the GetImageStream() function for comparison with the previous implementation:
public Stream GetImageStream()
{
MemoryStream ms = new MemoryStream();
ms.Write(_buffer, 0, _buffer.Length);
ms.Seek(0, SeekOrigin.Begin);
return ms;
}
Compare this to the previous function, that had numerous for()-s, encoding logic, etc :)
hoho! I'm very happy for doing this!
Edit: here's the source code for the SetPixelXX functions - to prove they are actually not slow :)
public void SetPixelSlow(int col, int row, byte red, byte green, byte blue, byte alpha)
{
int start = _rowLength * row + col * 4 + 1;
int blockNum = start / _blockSize;
start += ((blockNum + 1) * 5);
start += _dataStart;
_buffer[start] = red;
_buffer[start + 1] = green;
_buffer[start + 2] = blue;
_buffer[start + 3] = alpha;
}
public void SetPixelAtRowStart(int col, int rowStart, byte red, byte green, byte blue, byte alpha)
{
int start = rowStart + (col << 2);
_buffer[start] = red;
_buffer[start + 1] = green;
_buffer[start + 2] = blue;
_buffer[start + 3] = alpha;
}
public int GetRowStart(int row)
{
int start = _rowLength * row + 1;
int blockNum = start / _blockSize;
start += ((blockNum + 1) * 5);
start += _dataStart;
return start;
}
Last night, inspired by the cool taskbar in Windows 7, I decided to see how easy it will be to make a similar control in Silverlight.
Here's the screenshot of the first stab on the Win7 taskbar in Silverlight:
Click here: http://www.nokola.com/NavSeven to see it in action.
There are a lot of things to be improved on the above control, but it was easy to do this first version.
Download the source code here: http://www.nokola.com/sources/NavSeven.zip
Attachment(s): buttonseven.jpg
Submission deadline is April 30, 2009
Check it out:
http://www.serverquestcontest.com/
Too bad I can't participate (I'm not eligible because I'm Microsoft employee)
Whoever reads this, please do your best! Thank you! :)
I'll still be doing my own game thingie in the meantime, even without a chance to win $5000
Last week I created a document that incorporates my experiences developing the Shock Silverlight online game so far.
The document discusses the basics of online games in Silverlight and has a lot of practical examples (code) and a working project that you can use to build your own game.
Don't make the mistakes I did! Happy coding!
Open Document: Anatomy of a Silverlight Game.docx
Download Source Code Project
Play Sample Game
Here are the abstract and contents of the doc:
Abstract:
Typical online games have a start screen, levels, storyline, transitions, high scores, chat, and other features that can take significant amount of time to develop.
There are challenges when building online games, such as full screen support and scaling; speed optimization; animations; real-time calculations.
Learn how to develop compelling Silverlight games faster, by using a base framework for the common game elements outlined above.
Contents:
About the Author
About this Document
Goal
Scope
Target Audience
Getting Started
1. Introduction: Why Care About the “Details”?
2. Components of an Online Game
3. Screen Layout
Visual Layers that Make Up Your Game
Game Interface Layer
In-game Controls Layer
Sounds Layer
Start Screen Layer
Main Screen Popup Controls Layer
4. Silverlight Game Internals
The Game Loop
Some Classes That You’ll Likely Use in Every Game
The Game Class
Defaults Class
Globals Class
Enumeration Classes
Sounds Class
Other Classes
Keyboard and Mouse
Full Screen Support
Returning from Full Screen to Embedded Mode
Aspect Ratio
More Info
Transitions
Sound and Music
5. Networking
Score
Chat and Multiplayer
Network Speed Optimizations
6. Host Integration
7. Cheat Protection
8. Tips and Tricks
9. Summary
Happy Reading!
Attachment(s): Anatomy of a Silverlight Game.docx
I'm very proud to announce version 2 or my online Silverlight game "Shock":
Play it here:http://www.nokola.com/shock
Edit: I appreciate if you leave your comments and suggestions in the comment field for the game.
Also, if you like it, please vote for it on http://silverlight.net/community/gallerydetail.aspx?cat=sl2&sort=2
Sub-set of the source code (v1, playable) is here: Download Shock source code
Attachment(s): Shock2.jpg
I found a pretty cool chill-out style game in Silverlight. Just go and draw your first animal :) it's pretty nice to see those things come to life.
Also there are some cool things you can play with friends and share the animals and joy :)
You don't need an account to play
http://shidonni.com/
Check them out - including screenshots - on Karl’s (one of my friends) site:
http://karlshifflett.wordpress.com/xaml-power-toys/
XAML Power Toys
Speed up production of Business (LOB) Applications in WPF and Silverlight!
Automatically create XAML views from VB or C# Code:
· Create WPF or Silverlight DataGrid For Class - quickly create a WPF DataGrid complete with bindings that are easily associated with properties on the source class
· Create WPF ListView For Class - quickly create a ListView complete with bindings that are easily associated with properties on the source class
· Create Business Form For Class - quickly create a form complete with bindings that are easily associated with properties on the source class
· Create Business Form - quickly create a form complete with bindings if desired
· Drag and Drop class fields to create controls - display a list of class fields similar to Microsoft Access, then drag fields to create controls
Speed up common XAML editing tasks
· Group Into - allows selecting one or more controls and group them inside a new parent control
· Minimize XAML - allows selecting of one or more controls and will remove all MinWidth, MinHeight, x:Name, Name, Margin properties and will set all row and column definitions to Auto.
· Remove Margins - allows selecting one or more controls and removes the Margin property from them
· Quickly Add/Remove Grid Column and Rows - allows selecting a grid and then add or remove rows and columns
· Specify Defaults For Created Controls - allows customizing the initial properties that are set when the software creates a new control
http://karlshifflett.wordpress.com/xaml-power-toys/
In This Tutorial:
· Template project and simple 3-step instructions to kick start WPF pixel shaders
· Introduction to Pixel Shaders in WPF
· Source Code for 3 pixel shaders
· Links to 2 pixel shader sources on the web
Using Shaders in WPF: The Easy Way
Pixel shaders are a great way to enhance your WPF program in many ways. I show just few of the possibilities here and provide a good framework to start developing your own effects.
3 steps to kick start you with WPF pixel shaders:
1. Install DirectX SDK
2. Download and open the Shader template project from Nikola’s site: http://www.nokola.com/sources/ShaderEffectTemplate.zip
3. Start coding your effect in MyEffect.fx
Next: Read the Beginner’s Presentation to WPF Pixel Shaders
Downloads
Prerequisites: DirectX SDK from Microsoft.com
View the Beginner’s Presentation to WPF Pixel Shaders here: http://blogs.msdn.com/nikola/attachment/8924657.ashx
Download the template project here: http://www.nokola.com/sources/ShaderEffectTemplate.zip
Download these Pixel Shader samples: Twirl UI, Blobs and Lights: http://www.nokola.com/sources/ShaderEffects.zip Note that you have to enable each effect by itself for best view J Just comment out the other 2 effects when playing at the sample.
Check out these 2 cool pixel shader tutorials:
Grayscale effect: http://bursjootech.blogspot.com/2008/06/grayscale-effect-pixel-shader-effect-in.html
Wave reflection effect: http://rakeshravuri.blogspot.com/2008/07/wave-reflection-shader-effect-in-wpf.html
Hope you like it!
Attachment(s): Beginner’s WPF Pixel Shaders.pptx
In this tutorial:
- Creating a cool (and easy) Vista-like "ray of light" effect in Silverlight
- Creating images for splash screens
The demo below shows a "core" that emits rays of light as a hypothetical download progresses.
Download source code
Images used to create this sample:
Bottom:
Top:
Just right-click, save them and they are yours! Since I created them myself from scratch, you are free to use them in your projects!
One requirement if you use them: please send me an e-mail at nokola@nokola.com. I appreciate it a lot! Please also link to this site somewhere in the app credits or from your site.
Structure of the Sample
The sample displays 2 images. The bottom image is the “rays” view and the top image contains the “core”.
Having the rays be more intense in the center and less intense at the end is nice for 2 reasons:
· The image is visually appealing
· If we animate the Opacity of the rays from 0 to 1 it would look as if the rays “shine out” gradually
I use the second fact to make my life easier. The sample initially displays the rays image with Opacity=0 and the core image on top with Opacity=1.
When the user clicks the Opacity of the rays is animated using this simple XAML animation:
<Storyboard x:Name="coolStartup" Completed="coolStartup_Completed">
<DoubleAnimation Storyboard.TargetName="imgAnim" Storyboard.TargetProperty="Opacity"
To="1" Duration="00:00:03.00" />
</Storyboard>
Creating the Images
Another challenging part when making this sample is creating the actual images from scratch. Fortunately there are a bunch of good tutorials that you can use.
For example, check this tutorial: http://www.gimptalk.com/tutorial/abstract-art-(-good-for-background-)-33124-1.html. Then on the last step separate the rays image from the core image.
Make sure to convert the black (or white) background to transparency (Colors | Color To Aplha in this case).
I hope you like this sample! I enjoy it a lot!
In This Tutorial:
· C# compiler demo for the web - type code in a web page, compile and execute it on the client
· How to enable dynamic code compilation for a Silverlight application
· How to execute third party DLL from within a Silverlight application
Download source code
View sample
What Is It Good For?
· The compiled code is fast. It runs full speed on the client.
· It enables very flexible programs easily. Here are some example uses:
o Procedural texture generation (Perlin noise, 3-d textures, clouds, landscapes, and other). This is my next demo, btw.
o Fractals. The ability to compile code on-the-fly is very nice for experiments with fractals
o Advanced option that enables huge flexibility of LOB apps. Good for filtering database queries and such
o Simple plugins to extend a program
· As a side effect of using this technique, you’ll learn how to delay load Silverlight code – usable for making huge apps that load themselves part by part only when needed
· It’s good for educational purposes. Inspire more people to try and start coding!
Design
The C# compiler sample does not compile on the client machine. Instead, the C# source is sent to a web service and compiled there.
Then on the client side Silverlight loads the DLL that is returned from the web service and executes it.
Implementation
Perhaps the most interesting part of this sample is loading and assembly at runtime from within the Silverlight client application:
I’ve seen several heavy and not-so-obvious implementations on the web. It’s just a few lines of code, once you know how.
AssemblyPart part = new AssemblyPart();
MemoryStream stream = new MemoryStream(e.Result.AssemblyRawData);
_compiledAssembly = part.Load(stream);
Another interesting part is the web service compilation code:
With small modification, you can change it to target VB or WPF. Look at CompileHelper.cs in the source code.
Microsoft.CSharp.CSharpCodeProvider provider;
CompilerParameters parameters;
CompilerResults results;
parameters = new System.CodeDom.Compiler.CompilerParameters();
parameters.GenerateInMemory = true;
parameters.OutputAssembly = outputAssemblyFileName;
parameters.TreatWarningsAsErrors = false;
parameters.WarningLevel = 4;
parameters.TempFiles.KeepFiles = false;
if (TargetFramework == TargetFramework.Silverlight)
{
// Silverlight only: ignore the rsp file, because we want to replace the
// default desktop .net framework mscorlib and other "default" dlls
// with the Silverlight ones
parameters.CompilerOptions = " /nostdlib ";
}
parameters.ReferencedAssemblies.AddRange(referencedAssemblies);
try
{
provider = new Microsoft.CSharp.CSharpCodeProvider();
results = provider.CompileAssemblyFromSource(parameters, text);
errors = new List<ErrorInfo>(results.Errors.Count);
foreach (CompilerError error in results.Errors)
{
ErrorInfo info = new ErrorInfo();
info.Column = error.Column;
info.ErrorNumber = error.ErrorNumber;
info.ErrorText = error.ErrorText;
info.FileName = error.FileName;
info.IsWarning = error.IsWarning;
info.Line = error.Line;
errors.Add(info);
}
if (results.Errors.Count == 0)
{
return results.CompiledAssembly;
}
return null;
}
catch (Exception e)
{
Debug.WriteLine(e.Message);
errors = new List<ErrorInfo>();
return null;
}
Here are the 7 easy steps to learn C#, in my opinion: http://www.nokola.com/trycsharp/LearnCSharp.aspx
That’s all. Hope you’ll like this sample!
This is very good for web sites that have ASP.NET Authentication (or plan to have) and have Silverlight apps.
Brad Abrams, from his blog:
"Here is what I plan to show:
1. Login\Logout
2. Save personalization settings
3. Enable custom UI based on a user's role (for example, manager or employee)
4. A custom log-in control to make the UI a bit cleaner
"
http://blogs.msdn.com/brada/archive/2008/05/03/accessing-the-asp-net-authentication-profile-and-role-service-in-silverlight.aspx
Source code available on the above post.
In this tutorial:
- creating and rendering water ripples
- optimized image generation
- JPEG decoding component for Silverlight
Source code for this sample
Algorithm
The algorithm is based on Hugo Elias’ 2D water tutorial.
On each render step we have the state of the water for the current frame and the previous frame.
The state is stored in two 2-dimensional arrays of integers that are as big as the image.
For each pixel position of the image we store the height of the water (or wave) in that position. 0 means “sea level”. Larger than 0 means that we have a raised wave, less than zero means that we have wave below “sea level”. We need information for both raised and low waves in order to be able to combine them.
On each render step we use data from the current frame (Buffer2) and the previous frame (Buffer1) and write the results into Buffer1.
damping = some non-integer between 0 and 1 (I use 0.94)
for every non-edge element:
loop
Buffer2[x, y] = (Buffer1[x-1,y]
Buffer1[x+1,y]
Buffer1[x,y+1]
Buffer1[x,y-1]) / 2 - Buffer2[x,y]
Buffer2[x,y] = Buffer2[x,y] * damping
end loop
Swap the buffers
Display Buffer1
end loop
You can go ahead and look at Hugo’s explanation about why does this work, or continue reading here.
Because the 2 buffers contain consecutive steps for the water, we can get the water velocity at a given location [x, y] by subtracting: Buffer2[x, y] – Buffer1[x, y]
Also we want the waves to spread out, so we smooth the buffers on every frame:
Smoothed[x,y] = (Buffer1[x-1, y] +
Buffer1[x+1, y] +
Buffer1[x, y-1] +
Buffer1[x, y+1]) / 4
In the actual algorithm we multiply the smoothed value by 2 in order to decrease the effect of velocity.
And last, the waves lose energy as they travel:
Buffer2[x,y] = Buffer2[x,y] * damping
Rendering the Water
The render buffer contains heights of the water in each pixel. We’ll render it using shading and refraction.
The shading variable below determines the direction and intensity of the light. For example, if you set shading = xoffset, you’ll get light straight from the left. I decided to set the light at the bottom-right part of the screen.
For every non-edge pixel in the buffer
Xoffset = buffer[x-1, y] – buffer[x+1, y]
Yoffset = buffer[x, y-1] – buffer[x, y+1]
shading = (xoffset - yoffset) / 2
// note: x+xoffset and y+yoffset do not wrap around the texture
t = texture[x+Xoffset, y+Yoffset]
resultColor = t + Shading
// make sure the color is within limits
resultColor = SaturateTo255Max(resultColor)
plot pixel at (x,y) with color resultColor
End loop
Creating Drops/Splashes
I made a simple function to create a circular splash, although you can create different splashes to simulate dropping irregular shapes into the water or other motion effects (e.g. star, line or use the letters of your name).
The function creates a splash given its radius at location (cx, cy). The splash begins below water level and rises above at the end.
Splash(cx, cy, radius):
for each y from (cy - radius) to (cy + radius)
for each x from (cx - radius) to (cx + radius)
dist = distance from point (x,y) to (cx, cy)
if (dist < radius) // if within splash circle
buffer1[x, y] = 255 - (512 * 1 - dist / radius)
end if
end loop
end loop
The Silverlight Side
The rendering loop is called every 60ms and does this:
1. Add a random splash (rain drop) on the image
2. Calculate the next frame to render
3. Display the next frame
There are 3 components used to render each frame:
Renderer has the “raw” buffers of integers containing wave height for each pixel in the frame. On each frame the renderer mixes its raw buffer with the background image (decoded using FluxJpeg.Core.Image component) and outputs each it to a dynamic image generation surface (EditableImage).
Links
Original 2D water algorithm (Hugo Elias)
This is the algorithm used as a base for the sample.
Dynamic image generation (Joe Stegman)
Optimized dynamic image generation based on Joe's (Nokola)
I’m using optimized version of Joe’s algorithm to render the effect on screen.
Silverlight JPEG encoder/decoder in C# - FJCore on fluxcapacity.net
The binary is used to decode the JPEG image in Silverlight.
I needed a good and relatively fast dynamic image generation code for Silverlight (for the next sample, you’ll see).
Joe Stegman has an excellent post including source code for generating images in Silverlight: http://blogs.msdn.com/jstegman/archive/2008/04/21/dynamic-image-generation-in-silverlight.aspx
Joe’s code is simple and easy to understand, but can be optimized for speed quite a bit.
After doing some profiling (looking in Task Manager) and optimizations, I managed to speed up the code more than twice. To find out the slowest functions, I just ran and image generation in a loop and looked at the CPU usage in Task Manager when disabling each of the functions.
These are the things I updated:
· All the image data is not copied anymore in the WriteChunk() function
· One big array was initialized on every frame in Encode()
· Changed SetPixel() to remove some checks in Release mode
· The Adler32 algorithm was amazingly slow (it’s doing two prime number divisions for each pixel)…after looking at Wikipedia I found a trick that makes the adler use 5550 times less divisions J
The optimized source is here: http://blogs.msdn.com/nikola/attachment/8690157.ashx
Further optimizations
There can be more optimizations, for example the PngEncoder always works with the same image width/height, so it may avoid some calls to write to a memory stream and bulk several calls into one.
The second memory stream in Encode() can be removed.
Maybe the image can be stored png-encoded and just updated by SetPixel. This way the stream creation will be very fast. This will be good if for example only 10% of the pixels are changed each frame.
I hope someone else will pick up the source finish what we started J (or I’ll finish it later)
Note: this is a remake of a post that was accidentally deleted.
Attachment(s): EditableImage.zip
