David Rousset - sharing about HTML5, Windows 8 & Gaming

Tutorial part 3: learning how to write a 3D soft engine in C#, TS or JS – loading meshes exported from Blender

2013-06-17T15:55:29Z

In the previous tutorial learning how to write a 3D soft engine from scratch in C#, TS or JS – drawing lines & triangles, we’ve learned how to draw lines & triangles and we really started to see the 3D side our meshes thanks to this wireframe rendering. But we’ve only displayed a cube… And even a simple cube already has 12 faces! Are we going to be forced to handle ourselves all the faces for more complex objects this way? Hopefully not.

3D modelers help the collaboration between 3D designers and developers. The designer can use its favorite tools to build his scenes or meshes (3D Studio Max, Maya, Blender, etc.). Then he will export his work into a file that will be loaded by the developers. The developers will finally push the meshes into his real time 3D engine. There are several file formats available on the market to serialize the job done by the artists. In our case, we’re going to use JSON. Indeed, David Catuhe has done an export library for Blender that output a .babylon file using JSON. We’re then going to see how to parse that file and display the meshes in our lovely soft engine.

Blender is a free 3D modeler you can download here: http://www.blender.org/download/get-blender/

You can write plug-ins in Python. That’s what we’ve done for the exporter.

By following this tutorial, you will be able to have such a result:

And you’ll see that you’ve already done most of the job in the 2 previous tutorials to do that.

Install the Babylon exporter and generate your own scene with Blender

Once you’ll have installed Blender, please download our Babylon exporter from here: io_export_babylon.py

Copy this file into the \script\addons directory where you’ve installed Blender (for instance “C:\Program Files\Blender Foundation\Blender\2.67\scripts\addons” in my specific case).

You need to active our plug-in in the user preferences. Go to “File” –> “User Preferences” and the “Addons” tab. Search for “babylon” and activate it by checking the case.

Do whatever you want with Blender. If you’re like me, really bad at building 3D meshes, here is a cool option that will impress your friends during geeks parties: “Add” –> “Mesh” –> “Monkey”:

You should then obtain a screen like that:

Last step is to export it into the .babylon file format (our JSON file). “File” –> “Export” –> “Babylon.js”

Name the file “monkey.babylon”.

Note: this monkey is named Suzanne and is very well-known in the 3D/gaming community. By knowing her, you’re now a proud member of this cool community! Welcome onboard! ;)

Loading the exported JSON file and displaying its meshes

As I was telling you at the beginning of this article, we’ve already built all the needed logic to display more complex meshes like Suzanne. We’ve got our Face, Mesh & Vertex logic. This is all we need for now.

In the JSON format, the Babylon exporter is adding more details than we currently need. For instance, it also adds potential details about the textures, the lights, etc. That’s why, we’re going to parse the file and jump directly to areas we’re only interesting in: the vertices and the faces’ indices to build our wireframe rendering.

Note: for C# developers, you need to install Json.NET from Newtonsoft via nuGet like we’ve done in the first tutorial to add SharpDX. Indeed, JSON parsing is not natively supported in .NET like inside a browser using JavaScript.

Let’s start by adding the loading logic inside the Device object:

C#
TypeScript
JavaScript

// Loading the JSON file in an asynchronous manner
public async Task<Mesh[]> LoadJSONFileAsync(string fileName)
{
    var meshes = new List<Mesh>();
    var file = await Windows.ApplicationModel.Package.Current.InstalledLocation.GetFileAsync(fileName);
    var data = await Windows.Storage.FileIO.ReadTextAsync(file);
    dynamic jsonObject = Newtonsoft.Json.JsonConvert.DeserializeObject(data);

    for (var meshIndex = 0; meshIndex < jsonObject.meshes.Count; meshIndex++)
    {
        var verticesArray = jsonObject.meshes[meshIndex].vertices;
        // Faces
        var indicesArray = jsonObject.meshes[meshIndex].indices;

        var uvCount = jsonObject.meshes[meshIndex].uvCount.Value;
        var verticesStep = 1;

        // Depending of the number of texture's coordinates per vertex
        // we're jumping in the vertices array  by 6, 8 & 10 windows frame
        switch ((int)uvCount)
        {
            case 0:
                verticesStep = 6;
                break;
            case 1:
                verticesStep = 8;
                break;
            case 2:
                verticesStep = 10;
                break;
        }

        // the number of interesting vertices information for us
        var verticesCount = verticesArray.Count / verticesStep;
        // number of faces is logically the size of the array divided by 3 (A, B, C)
        var facesCount = indicesArray.Count / 3;
        var mesh = new Mesh(jsonObject.meshes[meshIndex].name.Value, verticesCount, facesCount);

        // Filling the Vertices array of our mesh first
        for (var index = 0; index < verticesCount; index++)
        {
            var x = (float)verticesArray[index * verticesStep].Value;
            var y = (float)verticesArray[index * verticesStep + 1].Value;
            var z = (float)verticesArray[index * verticesStep + 2].Value;
            mesh.Vertices[index] = new Vector3(x, y, z);
        }

        // Then filling the Faces array
        for (var index = 0; index < facesCount; index++)
        {
            var a = (int)indicesArray[index * 3].Value;
            var b = (int)indicesArray[index * 3 + 1].Value;
            var c = (int)indicesArray[index * 3 + 2].Value;
            mesh.Faces[index] = new Face { A = a, B = b, C = c };
        }

        // Getting the position you've set in Blender
        var position = jsonObject.meshes[meshIndex].position;
        mesh.Position = new Vector3((float)position[0].Value, (float)position[1].Value, (float)position[2].Value);
        meshes.Add(mesh);
    }
    return meshes.ToArray();
}

// Loading the JSON file in an asynchronous manner and
// calling back with the function passed providing the array of meshes loaded
public LoadJSONFileAsync(fileName: string, callback: (result: Mesh[]) => any): void {
    var jsonObject = {};
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.open("GET", fileName, true);
    var that = this;
    xmlhttp.onreadystatechange = function () {
        if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
            jsonObject = JSON.parse(xmlhttp.responseText);
            callback(that.CreateMeshesFromJSON(jsonObject));
        }
    };
    xmlhttp.send(null);
}

private CreateMeshesFromJSON(jsonObject): Mesh[] {
    var meshes: Mesh[] = [];
    for (var meshIndex = 0; meshIndex < jsonObject.meshes.length; meshIndex++) {
        var verticesArray: number[] = jsonObject.meshes[meshIndex].vertices;
        // Faces
        var indicesArray: number[] = jsonObject.meshes[meshIndex].indices;

        var uvCount: number = jsonObject.meshes[meshIndex].uvCount;
        var verticesStep = 1;

        // Depending of the number of texture's coordinates per vertex
        // we're jumping in the vertices array  by 6, 8 & 10 windows frame
        switch (uvCount) {
            case 0:
                verticesStep = 6;
                break;
            case 1:
                verticesStep = 8;
                break;
            case 2:
                verticesStep = 10;
                break;
        }

        // the number of interesting vertices information for us
        var verticesCount = verticesArray.length / verticesStep;
        // number of faces is logically the size of the array divided by 3 (A, B, C)
        var facesCount = indicesArray.length / 3;
        var mesh = new SoftEngine.Mesh(jsonObject.meshes[meshIndex].name, verticesCount, facesCount);
                
        // Filling the Vertices array of our mesh first
        for (var index = 0; index < verticesCount; index++) {
            var x = verticesArray[index * verticesStep];
            var y = verticesArray[index * verticesStep + 1];
            var z = verticesArray[index * verticesStep + 2];
            mesh.Vertices[index] = new BABYLON.Vector3(x, y, z);
        }
                
        // Then filling the Faces array
        for (var index = 0; index < facesCount; index++) {
            var a = indicesArray[index * 3];
            var b = indicesArray[index * 3 + 1];
            var c = indicesArray[index * 3 + 2];
            mesh.Faces[index] = {
                A: a,
                B: b,
                C: c
            };
        }
                
        // Getting the position you've set in Blender
        var position = jsonObject.meshes[meshIndex].position;
        mesh.Position = new BABYLON.Vector3(position[0], position[1], position[2]);
        meshes.push(mesh);
    }
    return meshes;
}

// Loading the JSON file in an asynchronous manner and
// calling back with the function passed providing the array of meshes loaded
Device.prototype.LoadJSONFileAsync = function (fileName, callback) {
    var jsonObject = {};
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.open("GET", fileName, true);
    var that = this;
    xmlhttp.onreadystatechange = function () {
        if(xmlhttp.readyState == 4 && xmlhttp.status == 200) {
            jsonObject = JSON.parse(xmlhttp.responseText);
            callback(that.CreateMeshesFromJSON(jsonObject));
        }
    };
    xmlhttp.send(null);
};
Device.prototype.CreateMeshesFromJSON = function (jsonObject) {
    var meshes = [];
    for(var meshIndex = 0; meshIndex < jsonObject.meshes.length; meshIndex++) {
        var verticesArray = jsonObject.meshes[meshIndex].vertices;
        // Faces
        var indicesArray = jsonObject.meshes[meshIndex].indices;

        var uvCount = jsonObject.meshes[meshIndex].uvCount;
        var verticesStep = 1;

        // Depending of the number of texture's coordinates per vertex
        // we're jumping in the vertices array  by 6, 8 & 10 windows frame
        switch(uvCount) {
            case 0:
                verticesStep = 6;
                break;
            case 1:
                verticesStep = 8;
                break;
            case 2:
                verticesStep = 10;
                break;
        }

        // the number of interesting vertices information for us
        var verticesCount = verticesArray.length / verticesStep;
        // number of faces is logically the size of the array divided by 3 (A, B, C)
        var facesCount = indicesArray.length / 3;
        var mesh = new SoftEngine.Mesh(jsonObject.meshes[meshIndex].name, verticesCount, facesCount);

        // Filling the Vertices array of our mesh first
        for (var index = 0; index < verticesCount; index++) {
            var x = verticesArray[index * verticesStep];
            var y = verticesArray[index * verticesStep + 1];
            var z = verticesArray[index * verticesStep + 2];
            mesh.Vertices[index] = new BABYLON.Vector3(x, y, z);
        }

        // Then filling the Faces array
        for(var index = 0; index < facesCount; index++) {
            var a = indicesArray[index * 3];
            var b = indicesArray[index * 3 + 1];
            var c = indicesArray[index * 3 + 2];
            mesh.Faces[index] = {
                A: a,
                B: b,
                C: c
            };
        }

        // Getting the position you've set in Blender
        var position = jsonObject.meshes[meshIndex].position;
        mesh.Position = new BABYLON.Vector3(position[0], position[1], position[2]);
        meshes.push(mesh);
    }
    return meshes;
};

You will probably wonder why we’re jumping by 6, 8 & 10 in the vertices array to take our 3D coordinate (X, Y, Z) of our vertices. Again, this is because the Babylon exporter adds more details that we currently need for our wireframe rendering. That’s why, we’re filtering those details using this frame approach. This logic is specific to our file format. If you want to load the export from other (like the one from three.js), you’ll just have to identity where to retrieve the vertices and faces indices in another file format.

Note: to be able to load our .babylon files, TypeScript/JavaScript developers, you need to define a new MIME type “application/babylon” targeting the extension “.babylon”. In IIS, you need to declare it inside your web.config:

  <system.webServer>
    <staticContent>
      <mimeMap fileExtension=".babylon" mimeType="application/babylon" />
    </staticContent>
  </system.webServer>

C# developers, you need to change the properties of the file you will include in the solution. Switch “Build Action” to “Content” and always copy to the output directory:

Otherwise, the file won’t be found.

Finally, we now need to update the equivalent of our main function to call this new LoadJSONFileAsync function instead of creating manually our cube. As we will also have potentially several meshes to animate, we need also to change the rotation values during each tick to every meshes loaded:

C#
TypeScript
JavaScript

private Device device;
Mesh[] meshes;
Camera mera = new Camera();

private async void Page_Loaded(object sender, RoutedEventArgs e)
{
    // Choose the back buffer resolution here
    WriteableBitmap bmp = new WriteableBitmap(640, 480);

    // Our Image XAML control
    frontBuffer.Source = bmp;
            
    device = new Device(bmp);
    meshes = await device.LoadJSONFileAsync("monkey.babylon");
    mera.Position = new Vector3(0, 0, 10.0f);
    mera.Target = Vector3.Zero;

    // Registering to the XAML rendering loop
    CompositionTarget.Rendering += CompositionTarget_Rendering;
}

// Rendering loop handler
void CompositionTarget_Rendering(object sender, object e)
{
    device.Clear(0, 0, 0, 255);

    foreach (var mesh in meshes) {
        // rotating slightly the meshes during each frame rendered
        mesh.Rotation = new Vector3(mesh.Rotation.X + 0.01f, mesh.Rotation.Y + 0.01f, mesh.Rotation.Z);
    }

    // Doing the various matrix operations
    device.Render(mera, meshes);
    // Flushing the back buffer into the front buffer
    device.Present();
}

///<reference path="SoftEngine.ts"/>

var canvas: HTMLCanvasElement; 
var device: SoftEngine.Device; 
var meshes: SoftEngine.Mesh[] = [];
var mera: SoftEngine.Camera;

document.addEventListener("DOMContentLoaded", init, false);

function init() {
    canvas = <HTMLCanvasElement> document.getElementById("frontBuffer");
    mera = new SoftEngine.Camera();
    device = new SoftEngine.Device(canvas); 

    mera.Position = new BABYLON.Vector3(0, 0, 10);
    mera.Target = new BABYLON.Vector3(0, 0, 0);

    device.LoadJSONFileAsync("monkey.babylon", loadJSONCompleted)
}

function loadJSONCompleted(meshesLoaded: SoftEngine.Mesh[]) {
    meshes = meshesLoaded;
    // Calling the HTML5 rendering loop
    requestAnimationFrame(drawingLoop);
}

// Rendering loop handler
function drawingLoop() {
    device.clear();

    for (var i = 0; i < meshes.length; i++) {
        // rotating slightly the mesh during each frame rendered
        meshes[i].Rotation.x += 0.01;
        meshes[i].Rotation.y += 0.01;
    }

    // Doing the various matrix operations
    device.render(mera, meshes);
    // Flushing the back buffer into the front buffer
    device.present();

    // Calling the HTML5 rendering loop recursively
    requestAnimationFrame(drawingLoop);
}

var canvas;
var device;
var meshes = [];
var mera;

document.addEventListener("DOMContentLoaded", init, false);

function init() {
    canvas = document.getElementById("frontBuffer");
    mera = new SoftEngine.Camera();
    device = new SoftEngine.Device(canvas);
    mera.Position = new BABYLON.Vector3(0, 0, 10);
    mera.Target = new BABYLON.Vector3(0, 0, 0);
    device.LoadJSONFileAsync("monkey.babylon", loadJSONCompleted);
}

function loadJSONCompleted(meshesLoaded) {
    meshes = meshesLoaded;
    // Calling the HTML5 rendering loop
    requestAnimationFrame(drawingLoop);
}

// Rendering loop handler
function drawingLoop() {
    device.clear();

    for (var i = 0; i < meshes.length; i++) {
        // rotating slightly the mesh during each frame rendered
        meshes[i].Rotation.x += 0.01;
        meshes[i].Rotation.y += 0.01;
    }

    // Doing the various matrix operations
    device.render(mera, meshes);
    // Flushing the back buffer into the front buffer
    device.present();

    // Calling the HTML5 rendering loop recursively
    requestAnimationFrame(drawingLoop);
}

You should now have a 3D engine which is capable of loading meshes exported by Blender and animate them in a wireframe rendering mode! I don’t know about you, but I was very excited to reach this stage. :)

Again, you can download the solutions containing the source code:

- C# : SoftEngineCSharpPart3.zip
- TypeScript : SoftEngineTSPart3.zip
- JavaScript : SoftEngineJSPart3.zip or simply right-click –> view source on the first embedded iframe

So, what’s next now? Well, we need to fill the triangles. This is called rasterization. We will also handle what we name a Z-Buffer to have a proper rendering. In the next tutorial, you’ll then learn how to obtain something like that:

We will fill the triangles with a random color. See you in the fourth tutorial!

David

Tutorial part 2: learning how to write a 3D soft engine from scratch in C#, TS or JS – drawing lines & triangles

2013-06-14T15:31:29Z

Now that we have built the core of our 3D engine thanks to the previous tutorial Tutorial series- learning how to write a 3D soft engine from scratch in C#, TypeScript or JavaScript, we can work on enhancing the rendering. The next step is then to connect the dots to draw some lines in order to render what you probably know as a “wireframe” rendering.

In this tutorial, uou will learn how to draw lines, what a face is and how cool is the Bresenham algorithm to draw some triangles.

Thanks to that, at the end, you will know how to code something as cool as that:

Yes! Our 3D rotating cube starts to really live on our screens!

First basic algorithm to draw a line between two points

Let’s start by coding a simple algorithm. To draw a line between 2 vertices, we’re going to use the following logic:

- if the distance between the 2 points (point0 & point1) is less than 2 pixels, there’s nothing to do
- otherwise, we’re finding the middle point between both points (point0 coordinates + (point1 coordinates – point0 coordinates) / 2)
- we’re drawing that point on screen
- we’re launching this algorithm recursively between point0 & middle point and between middle point & point1

Here is the code to do that:

C#
TypeScript
JavaScript

public void DrawLine(Vector2 point0, Vector2 point1)
{
    var dist = (point1 - point0).Length();

    // If the distance between the 2 points is less than 2 pixels
    // We're exiting
    if (dist < 2)
        return;

    // Find the middle point between first & second point
    Vector2 middlePoint = point0 + (point1 - point0)/2;
    // We draw this point on screen
    DrawPoint(middlePoint);
    // Recursive algorithm launched between first & middle point
    // and between middle & second point
    DrawLine(point0, middlePoint);
    DrawLine(middlePoint, point1);
}

public drawLine(point0: BABYLON.Vector2, point1: BABYLON.Vector2): void {
    var dist = point1.subtract(point0).length();

    // If the distance between the 2 points is less than 2 pixels
    // We're exiting
    if (dist < 2)
        return;

    // Find the middle point between first & second point
    var middlePoint = point0.add((point1.subtract(point0)).scale(0.5));
    // We draw this point on screen
    this.drawPoint(middlePoint);
    // Recursive algorithm launched between first & middle point
    // and between middle & second point
    this.drawLine(point0, middlePoint);
    this.drawLine(middlePoint, point1);
}

Device.prototype.drawLine = function (point0, point1) {
    var dist = point1.subtract(point0).length();

    // If the distance between the 2 points is less than 2 pixels
    // We're exiting
    if(dist < 2) {
        return;
    }

    // Find the middle point between first & second point
    var middlePoint = point0.add((point1.subtract(point0)).scale(0.5));
    // We draw this point on screen
    this.drawPoint(middlePoint);
    // Recursive algorithm launched between first & middle point
    // and between middle & second point
    this.drawLine(point0, middlePoint);
    this.drawLine(middlePoint, point1);
};

You need to update the rendering loop to use this new piece of code:

C#
TypeScript
JavaScript

for (var i = 0; i < mesh.Vertices.Length - 1; i++)
{
    var point0 = Project(mesh.Vertices[i], transformMatrix);
    var point1 = Project(mesh.Vertices[i + 1], transformMatrix);
    DrawLine(point0, point1);
}

for (var i = 0; i < cMesh.Vertices.length -1; i++){
    var point0 = this.project(cMesh.Vertices[i], transformMatrix);
    var point1 = this.project(cMesh.Vertices[i + 1], transformMatrix);
    this.drawLine(point0, point1);
}

for (var i = 0; i < cMesh.Vertices.length -1; i++){
    var point0 = this.project(cMesh.Vertices[i], transformMatrix);
    var point1 = this.project(cMesh.Vertices[i + 1], transformMatrix);
    this.drawLine(point0, point1);
}

And you should now obtain something like that:

I know this looks weird but this was the expected behavior. It should help you starting to understand what you need to do to display a 3D mesh. But to have a better rendering, we need to discover a new concept.

Displaying faces with triangles

Now that we know how to draw lines, we need a better way to render the mesh with them. The simplest geometric 2D shape is a triangle. The idea in 3D is then to draw all our meshes by using those triangles. We then need to split each side of our cube into 2 triangles. We’re going to do this “manually” but we’ll see in the next tutorial that 3D modelers are doing this step automatically for us now.

To draw triangles, you need to have 3 points/vertices. A face is then simply a structure containing 3 values which are indexes pointing to the proper vertices array of the mesh to be rendered.

To be understand this concept, let’s take our previous figure with a Cube displayed by Blender:

We have 4 vertices displayed on this figure with the following indices: 0, 1, 2, 3. To draw the upper side of the cube, we need to draw 2 triangles. The first one, Face 0, will be drawn with 3 lines from vertex 0 (-1, 1, 1) to vertex 1 (1, 1, 1), from vertex 1 (1, 1, 1) to vertex 2 (-1, –1, 1) and finally from vertex 2 (-1, –1, 1) to vertex 0 (-1, 1, 1). The second triangle, Face 1, will be drawn with the lines from vertex 1 to vertex 2, vertex 2 to vertex 3 and vertex 3 to vertex 1.

The equivalent code would be something like that:

var mesh = new SoftEngine.Mesh("Square", 4, 2);
meshes.Add(mesh);
mesh.Vertices[0] = new Vector3(-1, 1, 1);
mesh.Vertices[1] = new Vector3(1, 1, 1);
mesh.Vertices[2] = new Vector3(-1, -1, 1);
mesh.Vertices[3] = new Vector3(1, -1, 1);

mesh.Faces[0] = new Face { A = 0, B = 1, C = 2 };
mesh.Faces[1] = new Face { A = 1, B = 2, C = 3 };

If you want to draw to whole cube, you need to find the 10 remaining faces as we’ve got 12 faces for the 6 sides of our cube to draw.

Let’s now define the code for a Face object. It’s a very simple object as this is just a set of 3 indexes. Here’s the code of Face and the new Mesh definition which also now use it:

C#
TypeScript
JavaScript

namespace SoftEngine
{
    public struct Face
    {
        public int A;
        public int B;
        public int C;
    }
    public class Mesh
    {
        public string Name { get; set; }
        public Vector3[] Vertices { get; private set; }
        public Face[] Faces { get; set; }
        public Vector3 Position { get; set; }
        public Vector3 Rotation { get; set; }

        public Mesh(string name, int verticesCount, int facesCount)
        {
            Vertices = new Vector3[verticesCount];
            Faces = new Face[facesCount];
            Name = name;
        }
    }
}

///<reference path="babylon.math.ts"/>

module SoftEngine {
    export interface Face {
        A: number;
        B: number;
        C: number;
    }

    export class Mesh {
        Position: BABYLON.Vector3;
        Rotation: BABYLON.Vector3;
        Vertices: BABYLON.Vector3[];
        Faces: Face[];

        constructor(public name: string, verticesCount: number, facesCount: number) {
            this.Vertices = new Array(verticesCount);
            this.Faces = new Array(facesCount);
            this.Rotation = new BABYLON.Vector3(0, 0, 0);
            this.Position = new BABYLON.Vector3(0, 0, 0);
        }
    }
}

var SoftEngine;
(function (SoftEngine) {
    var Mesh = (function () {
        function Mesh(name, verticesCount, facesCount) {
            this.name = name;
            this.Vertices = new Array(verticesCount);
            this.Faces = new Array(facesCount);
            this.Rotation = new BABYLONTS.Vector3(0, 0, 0);
            this.Position = new BABYLONTS.Vector3(0, 0, 0);
        }
        return Mesh;
    })();
    SoftEngine.Mesh = Mesh;    
})(SoftEngine || (SoftEngine = {}));

We now need to update our Render() function/method of our Device object to iterate through all the faces defined and to draw the triangles associated.

C#
TypeScript
JavaScript

foreach (var face in mesh.Faces)
{
    var vertexA = mesh.Vertices[face.A];
    var vertexB = mesh.Vertices[face.B];
    var vertexC = mesh.Vertices[face.C];

    var pixelA = Project(vertexA, transformMatrix);
    var pixelB = Project(vertexB, transformMatrix);
    var pixelC = Project(vertexC, transformMatrix);

    DrawLine(pixelA, pixelB);
    DrawLine(pixelB, pixelC);
    DrawLine(pixelC, pixelA);
}

for (var indexFaces = 0; indexFaces < cMesh.Faces.length; indexFaces++)
{
    var currentFace = cMesh.Faces[indexFaces];
    var vertexA = cMesh.Vertices[currentFace.A];
    var vertexB = cMesh.Vertices[currentFace.B];
    var vertexC = cMesh.Vertices[currentFace.C];

    var pixelA = this.project(vertexA, transformMatrix);
    var pixelB = this.project(vertexB, transformMatrix);
    var pixelC = this.project(vertexC, transformMatrix);

    this.drawLine(pixelA, pixelB);
    this.drawLine(pixelB, pixelC);
    this.drawLine(pixelC, pixelA);
}

for (var indexFaces = 0; indexFaces < cMesh.Faces.length; indexFaces++)
{
    var currentFace = cMesh.Faces[indexFaces];
    var vertexA = cMesh.Vertices[currentFace.A];
    var vertexB = cMesh.Vertices[currentFace.B];
    var vertexC = cMesh.Vertices[currentFace.C];

    var pixelA = this.project(vertexA, transformMatrix);
    var pixelB = this.project(vertexB, transformMatrix);
    var pixelC = this.project(vertexC, transformMatrix);

    this.drawLine(pixelA, pixelB);
    this.drawLine(pixelB, pixelC);
    this.drawLine(pixelC, pixelA);
}

We finally need to declare the mesh associated with our Cube properly with its 12 faces to make this new code working as expected.

Here is the new declaration:

C#
TypeScript
JavaScript

var mesh = new SoftEngine.Mesh("Cube", 8, 12);
meshes.Add(mesh);
mesh.Vertices[0] = new Vector3(-1, 1, 1);
mesh.Vertices[1] = new Vector3(1, 1, 1);
mesh.Vertices[2] = new Vector3(-1, -1, 1);
mesh.Vertices[3] = new Vector3(1, -1, 1);
mesh.Vertices[4] = new Vector3(-1, 1, -1);
mesh.Vertices[5] = new Vector3(1, 1, -1);
mesh.Vertices[6] = new Vector3(1, -1, -1);
mesh.Vertices[7] = new Vector3(-1, -1, -1);

mesh.Faces[0] = new Face { A = 0, B = 1, C = 2 };
mesh.Faces[1] = new Face { A = 1, B = 2, C = 3 };
mesh.Faces[2] = new Face { A = 1, B = 3, C = 6 };
mesh.Faces[3] = new Face { A = 1, B = 5, C = 6 };
mesh.Faces[4] = new Face { A = 0, B = 1, C = 4 };
mesh.Faces[5] = new Face { A = 1, B = 4, C = 5 };

mesh.Faces[6] = new Face { A = 2, B = 3, C = 7 };
mesh.Faces[7] = new Face { A = 3, B = 6, C = 7 };
mesh.Faces[8] = new Face { A = 0, B = 2, C = 7 };
mesh.Faces[9] = new Face { A = 0, B = 4, C = 7 };
mesh.Faces[10] = new Face { A = 4, B = 5, C = 6 };
mesh.Faces[11] = new Face { A = 4, B = 6, C = 7 };

var mesh = new SoftEngine.Mesh("Cube", 8, 12);
meshes.push(mesh);
mesh.Vertices[0] = new BABYLON.Vector3(-1, 1, 1);
mesh.Vertices[1] = new BABYLON.Vector3(1, 1, 1);
mesh.Vertices[2] = new BABYLON.Vector3(-1, -1, 1);
mesh.Vertices[3] = new BABYLON.Vector3(1, -1, 1);
mesh.Vertices[4] = new BABYLON.Vector3(-1, 1, -1);
mesh.Vertices[5] = new BABYLON.Vector3(1, 1, -1);
mesh.Vertices[6] = new BABYLON.Vector3(1, -1, -1);
mesh.Vertices[7] = new BABYLON.Vector3(-1, -1, -1);

mesh.Faces[0] = { A:0, B:1, C:2 };
mesh.Faces[1] = { A:1, B:2, C:3 };
mesh.Faces[2] = { A:1, B:3, C:6 };
mesh.Faces[3] = { A:1, B:5, C:6 };
mesh.Faces[4] = { A:0, B:1, C:4 };
mesh.Faces[5] = { A:1, B:4, C:5 };

mesh.Faces[6] = { A:2, B:3, C:7 };
mesh.Faces[7] = { A:3, B:6, C:7 };
mesh.Faces[8] = { A:0, B:2, C:7 };
mesh.Faces[9] = { A:0, B:4, C:7 };
mesh.Faces[10] = { A:4, B:5, C:6 };
mesh.Faces[11] = { A:4, B:6, C:7 };

var mesh = new SoftEngine.Mesh("Cube", 8, 12);
meshes.push(mesh);
mesh.Vertices[0] = new BABYLON.Vector3(-1, 1, 1);
mesh.Vertices[1] = new BABYLON.Vector3(1, 1, 1);
mesh.Vertices[2] = new BABYLON.Vector3(-1, -1, 1);
mesh.Vertices[3] = new BABYLON.Vector3(1, -1, 1);
mesh.Vertices[4] = new BABYLON.Vector3(-1, 1, -1);
mesh.Vertices[5] = new BABYLON.Vector3(1, 1, -1);
mesh.Vertices[6] = new BABYLON.Vector3(1, -1, -1);
mesh.Vertices[7] = new BABYLON.Vector3(-1, -1, -1);

mesh.Faces[0] = { A:0, B:1, C:2 };
mesh.Faces[1] = { A:1, B:2, C:3 };
mesh.Faces[2] = { A:1, B:3, C:6 };
mesh.Faces[3] = { A:1, B:5, C:6 };
mesh.Faces[4] = { A:0, B:1, C:4 };
mesh.Faces[5] = { A:1, B:4, C:5 };

mesh.Faces[6] = { A:2, B:3, C:7 };
mesh.Faces[7] = { A:3, B:6, C:7 };
mesh.Faces[8] = { A:0, B:2, C:7 };
mesh.Faces[9] = { A:0, B:4, C:7 };
mesh.Faces[10] = { A:4, B:5, C:6 };
mesh.Faces[11] = { A:4, B:6, C:7 };

You should now have this beautiful rotating cube:

Congrats! :)

Enhancing the line drawing algorithm with Bresenham

There is an optimized way to draw our lines using the Bresenham's line algorithm. It’s faster & sharper than our current simple recursive version. The story of this algorithm is fascinating. Please read the Wikipedia definition of this algorithm to discover how Bresenham build it and for which reasons.

Here are the versions of this algorithm in C#, TypeScript and JavaScript:

C#
TypeScript
JavaScript

public void DrawBline(Vector2 point0, Vector2 point1)
{
    int x0 = (int)point0.X;
    int y0 = (int)point0.Y;
    int x1 = (int)point1.X;
    int y1 = (int)point1.Y;
            
    var dx = Math.Abs(x1 - x0);
    var dy = Math.Abs(y1 - y0);
    var sx = (x0 < x1) ? 1 : -1;
    var sy = (y0 < y1) ? 1 : -1;
    var err = dx - dy;

    while (true) {
        DrawPoint(new Vector2(x0, y0));

        if ((x0 == x1) && (y0 == y1)) break;
        var e2 = 2 * err;
        if (e2 > -dy) { err -= dy; x0 += sx; }
        if (e2 < dx) { err += dx; y0 += sy; }
    }
}

public drawBline(point0: BABYLON.Vector2, point1: BABYLON.Vector2): void {
    var x0 = point0.x >> 0;
    var y0 = point0.y >> 0;
    var x1 = point1.x >> 0;
    var y1 = point1.y >> 0;
    var dx = Math.abs(x1 - x0);
    var dy = Math.abs(y1 - y0);
    var sx = (x0 < x1) ? 1 : -1;
    var sy = (y0 < y1) ? 1 : -1;
    var err = dx - dy;

    while (true) {
        this.drawPoint(new BABYLON.Vector2(x0, y0));

        if ((x0 == x1) && (y0 == y1)) break;
        var e2 = 2 * err;
        if (e2 > -dy) { err -= dy; x0 += sx; }
        if (e2 < dx) { err += dx; y0 += sy; }
    }
};

Device.prototype.drawBline = function (point0, point1) {
    var x0 = point0.x >> 0;
    var y0 = point0.y >> 0;
    var x1 = point1.x >> 0;
    var y1 = point1.y >> 0;
    var dx = Math.abs(x1 - x0);
    var dy = Math.abs(y1 - y0);
    var sx = (x0 < x1) ? 1 : -1;
    var sy = (y0 < y1) ? 1 : -1;
    var err = dx - dy;
    while(true) {
        this.drawPoint(new BABYLON.Vector2(x0, y0));
        if((x0 == x1) && (y0 == y1)) break;
        var e2 = 2 * err;
        if(e2 > -dy) { err -= dy; x0 += sx; }
        if(e2 < dx) { err += dx; y0 += sy; }
    }
};

In the render function, replace the call do DrawLine by DrawBline and you should notice that this is a bit more fluid & a bit more sharp:

If you’re observing it with attention, you should see that this version using Bresenham is less choppy than our first algorithm.

Again, you can download the solutions containing the source code:

- C# : SoftEngineCSharpPart2.zip
- TypeScript : SoftEngineTSPart2.zip
- JavaScript : SoftEngineJSPart2.zip or simply right-click –> view source on the embedded iframe

In next tutorial, you will learn how to export some Meshes from Blender, a free 3D modeler tool, into a JSON format. We will then load this JSON file to display it with our wireframe engine. Indeed, we already have everything setup to display much more complex meshes like these one:

See you in the third part: learning how to write a 3D soft engine in C#, TS or JS – loading meshes exported from Blender

David

Tutorial series: learning how to write a 3D soft engine from scratch in C#, TypeScript or JavaScript

2013-06-13T19:31:35Z

I’d to like to share with you how I’ve learned to build what’s known as a “3D soft engine” through a series of tutorials. “Software engine” means that we will use only the CPU to build a 3D engine in an old school way (remember Doom on your 80386 ?). I’ll share with you the C#, TypeScript and JavaScript versions of the code. In this list, you should then find your favorite language or at least something near your favorite one. The idea is to help you transposing the following samples & concepts on your favorite platform. You’ll find the Visual Studio 2012 C#/TS/JS solutions to download at the end also.

So why building a 3D soft engine? Well, it’s simply because it really helps understanding how modern 3D works with our GPUs. Indeed, I’m currently learning the basics of 3D thanks to internal workshops delivered within Microsoft by the awesome David Catuhe. He’s been mastering 3D for many years now and matrices operations is hard-coded in his brain. When I was young, I was dreaming to be able to write such engines but I had the feeling it was too complex for me. Finally, you’ll see that this is not – that - complex. You simply need someone that will help you understanding the underlying principles in a simple way.

Through this series, you will learn how to project some 3D coordinates (X, Y, Z) associated to a point (a vertex) on a 2D screen, how to draw lines between each point, how to fill some triangles, to handle lights, materials and so on. This first tutorial will simply show you how to display 8 points associated to a cube and how to move them in a virtual 3D world.

This tutorial is part of the following series:

1 – Writing the core logic for camera, mesh & device object (this article)
2 – Drawing lines and triangles to obtain a wireframe rendering
3 – Loading meshes exported from Blender in a JSON format
4 – Filling the triangle with rasterization and using a Z-Buffer (to be published)

By properly following this tutorial, you will have the following result at the end:

Reading prerequisites

I’ve been thinking on how to write these tutorials for a long time now. And I’ve finally decided not to explain each required principle myself. There is a lot of good resources on the web that will explain those important principles better than I. But I’ve then spent quite some time browsing the web for you to choose, according to myself, the best one to read:

- World, View and Projection Matrix Unveiled
- Tutorial 3 : Matrices that will provide you an introduction to matrices, the model, view & projection matrices.
- Cameras on OpenGL ES 2.x – The ModelViewProjection Matrix : this one is really interesting also as it explains the story starting by how cameras and lenses work.
- Transforms (Direct3D 9)
- A brief introduction to 3D: an excellent PowerPoint slides deck ! Read at least up to slide 27. After that, it’s too linked to a technology talking to GPU (OpenGL or DirectX).
- OpenGL Transformation

Read those articles by not focusing on the technology associated (like OpenGL or DirectX) or on the concept of triangles you may have seen in the figures. We will see that later on.

By reading those articles, you really need to understand that there is a series of transformations done that way:

- we start by a 3D object centered on itself
- the same object is then moved into the virtual 3D world by translation, scaling or rotation operations via matrices
- a camera will look at this 3D object positioned in the 3D world
- the final projection of all that will be done into a 2D space which is your screen

All this magic is done by cumulating transformations through matrices operations. You should really be at least a bit familiar with those concepts before running through these tutorials. Even if you don’t understand everything by reading them the first time. You should read them first. You will probably go back to those articles later on while writing your own version of this 3D soft engine. This is completely normal, don’t worry! ;) The best way to learn 3D if by experimenting and doing mistakes.

We won’t neither spend some times on how matrix operations works. The good news is that you don’t really need to understand matrices. Simply view it as a black box doing the right operations for you. I’m not a master of matrices but I’ve managed to write a 3D soft engine by myself. So you should also succeed in doing so.

We will then use libraries that will do the job for us: SharpDX, a managed wrapper on top of DirectX, for C# developers and babylon.math.js written by David Catuhe for JavaScript developers. I’ve rewritten it in TypeScript also.

Software prerequisites

We will write a WinRT/XAML Windows Store Apps in C# and/or a HTML5 application with TypeScript/JavaScript. So if you want to use the C# samples as-is, you need to install:

1 - Windows 8
2 - Visual Studio 2012 Express for Windows Store Apps. You can download it for free: http://msdn.microsoft.com/en-US/windows/apps/br211386

If you choose to use the TypeScript samples, you need to install it from: http://www.typescriptlang.org/#Download .

You will find the plug-in for Visual Studio 2012 but there are other options available: Sublime Text, Vi, Emacs: TypeScript enabled! On my side, I’ve learned TypeScript by porting the C# version of my code to TypeScript. If you’re also interested in learning TypeScript, a first good introduction is this webcast: Anders Hejlsberg: Introducing TypeScript

If you choose JavaScript, you just need your favorite IDE and a HTML5 compatible browser. :)

Please create a project named “SoftEngine” targeting the language you’d like to use. If it’s C#, add the “SharpDX core assembly” by using NuGet on your solution:

If it’s TypeScript, download babylon.math.ts. If' it’s JavaScript download babylon.math.js. Add a reference to those files in both cases.

Back buffer & rendering loop

In a 3D engine, we’re rendering the complete scene during each frame with the hope of keeping an optimal 60 frames per second (FPS) to keep fluid animations. To do our rendering job, we need what we call a back buffer. This could be seen as 2 dimensional array mapping the screen/window size. Every cell of the array is mapped to a pixel on the screen.

In our XAML Windows Store Apps, we will use a byte[] array that will act as our dynamic back buffer. For every frame being rendered in the animation loop (tick), this buffer will be affected to a WriteableBitmap acting as the source of a XAML image control that will be called the front buffer. For the rendering loop, we’re going to ask to the XAML rendering engine to call us for every frame it will generate. The registration is done thanks to this line of code:

CompositionTarget.Rendering += CompositionTarget_Rendering;

In HTML5, we’re going to use of course the <canvas /> element. The canvas element has already a back buffer data array associated to it. You can access it through the getImageData() and setImageData() functions. The animation loop will be handled by the requestAnimationFrame() function. This one is much more efficient that an equivalent of a setTimeout(function() {], 1000/60) as it’s handled natively by the browser that will callback our code only when it will be ready to draw.

Note: in both cases, you can render the frames in a different resolution that the actual width & height of the final window. For instance, you can have a back buffer of 640x480 pixels whereas the final display screen (front buffer) will be in 1920x1080. In XAML and thanks to CSS in HTML5, you will then benefit from “hardware scaling”. The rendering engines of XAML and of the browser will stretch the back buffer data to the front buffer window by even using an anti-aliasing algorithm. In both cases, this task is done by the GPU. This is why we call it “hardware scaling” (hardware is the GPU). You can read more about this topic addressed in HTML5 here: Unleash the power of HTML 5 Canvas for gaming . This approach is often used in games for instance to boost the performance as you have less pixels to address.

Camera & Mesh objects

Let’s start coding. First, we need to define some objects that will embed the details needed for a camera and for a mesh. A mesh is a cool name to describe a 3D object.

Our Camera will have 2 properties: its position in the 3D world and where it’s looking at, the target. Both are made of 3D coordinates named a Vector3. C# will use SharpDX.Vector3 and TypeScript & JavaScript will use BABYLON.Vector3.

Our Mesh will have a collection of vertices (several vertex or 3D points) that will be used to build our 3D object, its position in the 3D world and its rotation state. To identify it, it will also have a name.

To resume, we need the following code:

C#
TypeScript
JavaScript

// Camera.cs & Mesh.cs
using SharpDX;

namespace SoftEngine
{
    public class Camera
    {
        public Vector3 Position { get; set; }
        public Vector3 Target { get; set; }
    }
    public class Mesh
    {
        public string Name { get; set; }
        public Vector3[] Vertices { get; private set; }
        public Vector3 Position { get; set; }
        public Vector3 Rotation { get; set; }

        public Mesh(string name, int verticesCount)
        {
            Vertices = new Vector3[verticesCount];
            Name = name;
        }
    }
}

//<reference path="babylon.math.ts"/>
module SoftEngine {
    export class Camera {
        Position: BABYLON.Vector3;
        Target: BABYLON.Vector3;

        constructor() {
            this.Position = BABYLON.Vector3.Zero();
            this.Target = BABYLON.Vector3.Zero();
        }
    }
    export class Mesh {
        Position: BABYLON.Vector3;
        Rotation: BABYLON.Vector3;
        Vertices: BABYLON.Vector3[];

        constructor(public name: string, verticesCount: number) {
            this.Vertices = new Array(verticesCount);
            this.Rotation = BABYLON.Vector3.Zero();
            this.Position = BABYLON.Vector3.Zero();
        }
    }
}

var SoftEngine;
(function (SoftEngine) {
    var Camera = (function () {
        function Camera() {
            this.Position = BABYLON.Vector3.Zero();
            this.Target = BABYLON.Vector3.Zero();
        }
        return Camera;
    })();
    SoftEngine.Camera = Camera;    
    var Mesh = (function () {
        function Mesh(name, verticesCount) {
            this.name = name;
            this.Vertices = new Array(verticesCount);
            this.Rotation = BABYLON.Vector3.Zero();
            this.Position = BABYLON.Vector3.Zero();
        }
        return Mesh;
    })();
    SoftEngine.Mesh = Mesh;    
})(SoftEngine || (SoftEngine = {}));

For instance, if you want to describe a cube using our Mesh object, you need to create 8 vertices associated to the 8 points of the cube. Here are the coordinates on a cube displayed in Blender:

With a left-handed world. Remember also that when you’re creating a mesh, the coordinates system is starting at the center of the mesh. So, X=0, Y=0, Z=0 is the center of the cube.

This could be created via this kind of code:

var mesh = new Mesh("Cube", 8);
mesh.Vertices[0] = new Vector3(-1, 1, 1);
mesh.Vertices[1] = new Vector3(1, 1, 1);
mesh.Vertices[2] = new Vector3(-1, -1, 1);
mesh.Vertices[3] = new Vector3(-1, -1, -1);
mesh.Vertices[4] = new Vector3(-1, 1, -1);
mesh.Vertices[5] = new Vector3(1, 1, -1);
mesh.Vertices[6] = new Vector3(1, -1, 1);
mesh.Vertices[7] = new Vector3(1, -1, -1);

The most important part: the Device object

Now that we have our basic objects and we know how to build 3D meshes, we need the most important part: the Device object. It’s the core of our 3D engine.

In it’s rendering function, we will build the view matrix and the projection matrix based on the camera we will have defined before.

Then, we will iterate through each available mesh to build their associated world matrix based on their current rotation and translation values. Finally, once done, the final transformation matrix to apply is:

var transformMatrix = worldMatrix * viewMatrix * projectionMatrix;

This is the concept you absolutely need to understand by reading the previous prerequisites resources. Otherwise, you will probably simply copy/paste the code without understanding anything about the magic underneath. This is not a very big problem for further tutorials but again, it’s better to know what’s you’re coding.

Using this transformation matrix, we’re going to project each vertex of each mesh in the 2D world to obtain X,Y coordinates from their X,Y,Z coordinates. To finally draw on screen, we’re adding a small clip logic to only display visible pixels via a PutPixel method/function.

Here are the various versions of the Device object. I’ve tried to comment the code to help you understanding it as much as possible.

Note: Microsoft Windows is drawing using the BGRA color space (Blue, Green, Red, Alpha) whereas the HTML5 canvas is drawing using the RGBA (Red, Green, Blue, Alpha) color space. That’s why, you will notice some slight differences in the code between C# and HTML5.

C#
TypeScript
JavaScript

using Windows.UI.Xaml.Media.Imaging;
using System.Runtime.InteropServices.WindowsRuntime;
using SharpDX;

namespace SoftEngine
{
    public class Device
    {
        private byte[] backBuffer;
        private WriteableBitmap bmp;

        public Device(WriteableBitmap bmp)
        {
            this.bmp = bmp;
            // the back buffer size is equal to the number of pixels to draw
            // on screen (width*height) * 4 (R,G,B & Alpha values). 
            backBuffer = new byte[bmp.PixelWidth * bmp.PixelHeight * 4];
        }

        // This method is called to clear the back buffer with a specific color
        public void Clear(byte r, byte g, byte b, byte a) {
            for (var index = 0; index < backBuffer.Length; index += 4)
            {
                // BGRA is used by Windows instead by RGBA in HTML5
                backBuffer[index] = b;
                backBuffer[index + 1] = g;
                backBuffer[index + 2] = r;
                backBuffer[index + 3] = a;
            }
        }

        // Once everything is ready, we can flush the back buffer
        // into the front buffer. 
        public void Present()
        {
            using (var stream = bmp.PixelBuffer.AsStream())
            {
                // writing our byte[] back buffer into our WriteableBitmap stream
                stream.Write(backBuffer, 0, backBuffer.Length);
            }
            // request a redraw of the entire bitmap
            bmp.Invalidate();
        }

        // Called to put a pixel on screen at a specific X,Y coordinates
        public void PutPixel(int x, int y, Color4 color)
        {
            // As we have a 1-D Array for our back buffer
            // we need to know the equivalent cell in 1-D based
            // on the 2D coordinates on screen
            var index = (x + y * bmp.PixelWidth) * 4;

            backBuffer[index] = (byte)(color.Blue * 255);
            backBuffer[index + 1] = (byte)(color.Green * 255);
            backBuffer[index + 2] = (byte)(color.Red * 255);
            backBuffer[index + 3] = (byte)(color.Alpha * 255);
        }

        // Project takes some 3D coordinates and transform them
        // in 2D coordinates using the transformation matrix
        public Vector2 Project(Vector3 coord, Matrix transMat)
        {
            // transforming the coordinates
            var point = Vector3.TransformCoordinate(coord, transMat);
            // The transformed coordinates will be based on coordinate system
            // starting on the center of the screen. But drawing on screen normally starts
            // from top left. We then need to transform them again to have x:0, y:0 on top left.
            var x = point.X * bmp.PixelWidth + bmp.PixelWidth / 2.0f;
            var y = -point.Y * bmp.PixelHeight + bmp.PixelHeight / 2.0f;
            return (new Vector2(x, y));
        }

        // DrawPoint calls PutPixel but does the clipping operation before
        public void DrawPoint(Vector2 point)
        {
            // Clipping what's visible on screen
            if (point.X >= 0 && point.Y >= 0 && point.X < bmp.PixelWidth && point.Y < bmp.PixelHeight)
            {
                // Drawing a yellow point
                PutPixel((int)point.X, (int)point.Y, new Color4(1.0f, 1.0f, 0.0f, 1.0f));
            }
        }

        // The main method of the engine that re-compute each vertex projection
        // during each frame
        public void Render(Camera camera, params Mesh[] meshes)
        {
            // To understand this part, please read the prerequisites resources
            var viewMatrix = Matrix.LookAtLH(camera.Position, camera.Target, Vector3.UnitY);
            var projectionMatrix = Matrix.PerspectiveFovRH(0.78f, 
                                                           (float)bmp.PixelWidth / bmp.PixelHeight, 
                                                           0.01f, 1.0f);

            foreach (Mesh mesh in meshes) 
            {
                // Beware to apply rotation before translation 
                var worldMatrix = Matrix.RotationYawPitchRoll(mesh.Rotation.Y, 
                                                              mesh.Rotation.X, mesh.Rotation.Z) * 
                                  Matrix.Translation(mesh.Position);

                var transformMatrix = worldMatrix * viewMatrix * projectionMatrix;

                foreach (var vertex in mesh.Vertices)
                {
                    // First, we project the 3D coordinates into the 2D space
                    var point = Project(vertex, transformMatrix);
                    // Then we can draw on screen
                    DrawPoint(point);
                }
            }
        }
    }
}

///<reference path="babylon.math.ts"/>

module SoftEngine {
    export class Device {
        // the back buffer size is equal to the number of pixels to draw
        // on screen (width*height) * 4 (R,G,B & Alpha values). 
        private backbuffer: ImageData;
        private workingCanvas: HTMLCanvasElement;
        private workingContext: CanvasRenderingContext2D;
        private workingWidth: number;
        private workingHeight: number;
        // equals to backbuffer.data
        private backbufferdata;

        constructor(canvas: HTMLCanvasElement) {
            this.workingCanvas = canvas;
            this.workingWidth = canvas.width;
            this.workingHeight = canvas.height;
            this.workingContext = this.workingCanvas.getContext("2d");
        }

        // This function is called to clear the back buffer with a specific color
        public clear(): void {
            // Clearing with black color by default
            this.workingContext.clearRect(0, 0, this.workingWidth, this.workingHeight);
            // once cleared with black pixels, we're getting back the associated image data to 
            // clear out back buffer
            this.backbuffer = this.workingContext.getImageData(0, 0, this.workingWidth, this.workingHeight);
        }

        // Once everything is ready, we can flush the back buffer
        // into the front buffer. 
        public present(): void {
            this.workingContext.putImageData(this.backbuffer, 0, 0);
        }

        // Called to put a pixel on screen at a specific X,Y coordinates
        public putPixel(x: number, y: number, color: BABYLON.Color4): void {
            this.backbufferdata = this.backbuffer.data;
            // As we have a 1-D Array for our back buffer
            // we need to know the equivalent cell index in 1-D based
            // on the 2D coordinates of the screen
            var index: number = ((x >> 0) + (y >> 0) * this.workingWidth) * 4;
    

    
            // RGBA color space is used by the HTML5 canvas
            this.backbufferdata[index] = color.r * 255;
            this.backbufferdata[index + 1] = color.g * 255;
            this.backbufferdata[index + 2] = color.b * 255;
            this.backbufferdata[index + 3] = color.a * 255;
        }

        // Project takes some 3D coordinates and transform them
        // in 2D coordinates using the transformation matrix
        public project(coord: BABYLON.Vector3, transMat: BABYLON.Matrix): BABYLON.Vector2 {
            // transforming the coordinates
            var point = BABYLON.Vector3.TransformCoordinates(coord, transMat);
            // The transformed coordinates will be based on coordinate system
            // starting on the center of the screen. But drawing on screen normally starts
            // from top left. We then need to transform them again to have x:0, y:0 on top left.
            var x = point.x * this.workingWidth + this.workingWidth / 2.0 >> 0;
            var y = -point.y * this.workingHeight + this.workingHeight / 2.0 >> 0;
            return (new BABYLON.Vector2(x, y));
        }

        // drawPoint calls putPixel but does the clipping operation before
        public drawPoint(point: BABYLON.Vector2): void {
            // Clipping what's visible on screen
            if (point.x >= 0 && point.y >= 0 && point.x < this.workingWidth 
                                             && point.y < this.workingHeight) {
                // Drawing a yellow point
                this.putPixel(point.x, point.y, new BABYLON.Color4(1, 1, 0, 1));
            }
        }

        // The main method of the engine that re-compute each vertex projection
        // during each frame
        public render(camera: Camera, meshes: Mesh[]): void {
            // To understand this part, please read the prerequisites resources
            var viewMatrix = BABYLON.Matrix.LookAtLH(camera.Position, camera.Target, BABYLON.Vector3.Up());
            var projectionMatrix = BABYLON.Matrix.PerspectiveFovLH(0.78, 
                                                  this.workingWidth / this.workingHeight, 0.01, 1.0);

            for (var index = 0; index < meshes.length; index++) {
                // current mesh to work on
                var cMesh = meshes[index];
                // Beware to apply rotation before translation
                var worldMatrix = BABYLON.Matrix.RotationYawPitchRoll(
                        cMesh.Rotation.y, cMesh.Rotation.x, cMesh.Rotation.z)
                        .multiply(BABYLON.Matrix.Translation(
                            cMesh.Position.x, cMesh.Position.y, cMesh.Position.z));

                var transformMatrix = worldMatrix.multiply(viewMatrix).multiply(projectionMatrix);

                for (var indexVertices = 0; indexVertices < cMesh.Vertices.length; indexVertices++) {
                    // First, we project the 3D coordinates into the 2D space
                    var projectedPoint = this.project(cMesh.Vertices[indexVertices], transformMatrix);
                    // Then we can draw on screen
                    this.drawPoint(projectedPoint);
                }
            }
        }
    }
}

var SoftEngine;
(function (SoftEngine) {  
    var Device = (function () {
        function Device(canvas) {
            // Note: the back buffer size is equal to the number of pixels to draw
            // on screen (width*height) * 4 (R,G,B & Alpha values). 
            this.workingCanvas = canvas;
            this.workingWidth = canvas.width;
            this.workingHeight = canvas.height;
            this.workingContext = this.workingCanvas.getContext("2d");
        }

        // This function is called to clear the back buffer with a specific color
        Device.prototype.clear = function () {
            // Clearing with black color by default
            this.workingContext.clearRect(0, 0, this.workingWidth, this.workingHeight);
            // once cleared with black pixels, we're getting back the associated image data to 
            // clear out back buffer
            this.backbuffer = this.workingContext.getImageData(0, 0, this.workingWidth, this.workingHeight);
        };

        // Once everything is ready, we can flush the back buffer
        // into the front buffer. 
        Device.prototype.present = function () {
            this.workingContext.putImageData(this.backbuffer, 0, 0);
        };

        // Called to put a pixel on screen at a specific X,Y coordinates
        Device.prototype.putPixel = function (x, y, color) {
            this.backbufferdata = this.backbuffer.data;
            // As we have a 1-D Array for our back buffer
            // we need to know the equivalent cell index in 1-D based
            // on the 2D coordinates of the screen
            var index = ((x >> 0) + (y >> 0) * this.workingWidth) * 4;
    

    
            // RGBA color space is used by the HTML5 canvas
            this.backbufferdata[index] = color.r * 255;
            this.backbufferdata[index + 1] = color.g * 255;
            this.backbufferdata[index + 2] = color.b * 255;
            this.backbufferdata[index + 3] = color.a * 255;
        };

        // Project takes some 3D coordinates and transform them
        // in 2D coordinates using the transformation matrix
        Device.prototype.project = function (coord, transMat) {
            var point = BABYLON.Vector3.TransformCoordinates(coord, transMat);
            // The transformed coordinates will be based on coordinate system
            // starting on the center of the screen. But drawing on screen normally starts
            // from top left. We then need to transform them again to have x:0, y:0 on top left.
            var x = point.x * this.workingWidth + this.workingWidth / 2.0 >> 0;
            var y = -point.y * this.workingHeight + this.workingHeight / 2.0 >> 0;
            return (new BABYLON.Vector2(x, y));
        };

        // drawPoint calls putPixel but does the clipping operation before
        Device.prototype.drawPoint = function (point) {
            // Clipping what's visible on screen
            if (point.x >= 0 && point.y >= 0 && point.x < this.workingWidth 
                                             && point.y < this.workingHeight) {
                // Drawing a yellow point
                this.putPixel(point.x, point.y, new BABYLON.Color4(1, 1, 0, 1));
            }
        };

        // The main method of the engine that re-compute each vertex projection
        // during each frame
        Device.prototype.render = function (camera, meshes) {
            // To understand this part, please read the prerequisites resources
            var viewMatrix = BABYLON.Matrix.LookAtLH(camera.Position, camera.Target, BABYLON.Vector3.Up());
            var projectionMatrix = BABYLON.Matrix.PerspectiveFovLH(0.78, 
                                           this.workingWidth / this.workingHeight, 0.01, 1.0);

            for (var index = 0; index < meshes.length; index++) {
                // current mesh to work on
                var cMesh = meshes[index];
                // Beware to apply rotation before translation
                var worldMatrix = BABYLON.Matrix.RotationYawPitchRoll(
                    cMesh.Rotation.y, cMesh.Rotation.x, cMesh.Rotation.z)
                     .multiply(BABYLON.Matrix.Translation(
                       cMesh.Position.x, cMesh.Position.y, cMesh.Position.z));

                var transformMatrix = worldMatrix.multiply(viewMatrix).multiply(projectionMatrix);

                for (var indexVertices = 0; indexVertices < cMesh.Vertices.length; indexVertices++) {
                    // First, we project the 3D coordinates into the 2D space
                    var projectedPoint = this.project(cMesh.Vertices[indexVertices], transformMatrix);
                    // Then we can draw on screen
                    this.drawPoint(projectedPoint);
                }
            }
        };
        return Device;
    })();
    SoftEngine.Device = Device;    
})(SoftEngine || (SoftEngine = {}));

Putting it all together

We finally need to create a mesh (our cube), create a camera and target our mesh & instantiate our Device object.

Once done, we will launch the animation/rendering loop. In optimal cases, this loop will be called every 16ms (60 FPS). During each tick (call to the handler registered to the rendering loop), we will launch the following logic every time:

1 – Clear the screen and all associated pixels with black ones (Clear() function)
2 – Update the various position & rotation values of our meshes
3 – Render them into the back buffer by doing the required matrix operations (Render() function)
4 – Display them on screen by flushing the back buffer data into the front buffer (Present() function)

C#
TypeScript
JavaScript

private Device device;
Mesh mesh = new Mesh("Cube", 8);
Camera mera = new Camera();

private void Page_Loaded(object sender, RoutedEventArgs e)
{
    // Choose the back buffer resolution here
    WriteableBitmap bmp = new WriteableBitmap(640, 480);

    device = new Device(bmp);

    // Our XAML Image control
    frontBuffer.Source = bmp;

    mesh.Vertices[0] = new Vector3(-1, 1, 1);
    mesh.Vertices[1] = new Vector3(1, 1, 1);
    mesh.Vertices[2] = new Vector3(-1, -1, 1);
    mesh.Vertices[3] = new Vector3(-1, -1, -1);
    mesh.Vertices[4] = new Vector3(-1, 1, -1);
    mesh.Vertices[5] = new Vector3(1, 1, -1);
    mesh.Vertices[6] = new Vector3(1, -1, 1);
    mesh.Vertices[7] = new Vector3(1, -1, -1);

    mera.Position = new Vector3(0, 0, 10.0f);
    mera.Target = Vector3.Zero;

    // Registering to the XAML rendering loop
    CompositionTarget.Rendering += CompositionTarget_Rendering;
}

// Rendering loop handler
void CompositionTarget_Rendering(object sender, object e)
{
    device.Clear(0, 0, 0, 255);

    // rotating slightly the cube during each frame rendered
    mesh.Rotation = new Vector3(mesh.Rotation.X + 0.01f, mesh.Rotation.Y + 0.01f, mesh.Rotation.Z);

    // Doing the various matrix operations
    device.Render(mera, mesh);
    // Flushing the back buffer into the front buffer
    device.Present();
}

///<reference path="SoftEngine.ts"/>

var canvas: HTMLCanvasElement; 
var device: SoftEngine.Device;
var mesh: SoftEngine.Mesh;
var meshes: SoftEngine.Mesh[] = [];
var mera: SoftEngine.Camera;

document.addEventListener("DOMContentLoaded", init, false);

function init() {
    canvas = <HTMLCanvasElement> document.getElementById("frontBuffer");
    mesh = new SoftEngine.Mesh("Cube", 8);
    meshes.push(mesh);
    mera = new SoftEngine.Camera();
    device = new SoftEngine.Device(canvas);

    mesh.Vertices[0] = new BABYLON.Vector3(-1, 1, 1);
    mesh.Vertices[1] = new BABYLON.Vector3(1, 1, 1);
    mesh.Vertices[2] = new BABYLON.Vector3(-1, -1, 1);
    mesh.Vertices[3] = new BABYLON.Vector3(-1, -1, -1);
    mesh.Vertices[4] = new BABYLON.Vector3(-1, 1, -1);
    mesh.Vertices[5] = new BABYLON.Vector3(1, 1, -1);
    mesh.Vertices[6] = new BABYLON.Vector3(1, -1, 1);
    mesh.Vertices[7] = new BABYLON.Vector3(1, -1, -1);

    mera.Position = new BABYLON.Vector3(0, 0, 10);
    mera.Target = new BABYLON.Vector3(0, 0, 0);

    // Calling the HTML5 rendering loop
    requestAnimationFrame(drawingLoop);
}

// Rendering loop handler
function drawingLoop() {
    device.clear();

    // rotating slightly the cube during each frame rendered
    mesh.Rotation.x += 0.01;
    mesh.Rotation.y += 0.01;

    // Doing the various matrix operations
    device.render(mera, meshes);
    // Flushing the back buffer into the front buffer
    device.present();

    // Calling the HTML5 rendering loop recursively
    requestAnimationFrame(drawingLoop);
}

var canvas;
var device;
var mesh;
var meshes = [];
var mera;

document.addEventListener("DOMContentLoaded", init, false);

function init() {
    canvas = document.getElementById("frontBuffer");
    mesh = new SoftEngine.Mesh("Cube", 8);
    meshes.push(mesh);
    mera = new SoftEngine.Camera();
    device = new SoftEngine.Device(canvas);

    mesh.Vertices[0] = new BABYLON.Vector3(-1, 1, 1);
    mesh.Vertices[1] = new BABYLON.Vector3(1, 1, 1);
    mesh.Vertices[2] = new BABYLON.Vector3(-1, -1, 1);
    mesh.Vertices[3] = new BABYLON.Vector3(-1, -1, -1);
    mesh.Vertices[4] = new BABYLON.Vector3(-1, 1, -1);
    mesh.Vertices[5] = new BABYLON.Vector3(1, 1, -1);
    mesh.Vertices[6] = new BABYLON.Vector3(1, -1, 1);
    mesh.Vertices[7] = new BABYLON.Vector3(1, -1, -1);

    mera.Position = new BABYLON.Vector3(0, 0, 10);
    mera.Target = new BABYLON.Vector3(0, 0, 0);

    // Calling the HTML5 rendering loop
    requestAnimationFrame(drawingLoop);
}

// Rendering loop handler
function drawingLoop() {
    device.clear();

    // rotating slightly the cube during each frame rendered
    mesh.Rotation.x += 0.01;
    mesh.Rotation.y += 0.01;

    // Doing the various matrix operations
    device.render(mera, meshes);
    // Flushing the back buffer into the front buffer
    device.present();

    // Calling the HTML5 rendering loop recursively
    requestAnimationFrame(drawingLoop);
}

If you’ve managed to follow properly this first tutorial, you should obtain something like that:

If not, download the solutions containing the source code:

- C# : SoftEngineCSharpPart1.zip
- TypeScript : SoftEngineTSPart1.zip
- JavaScript : SoftEngineJSPart1.zip or simply right-click –> view source on the embedded iframe

Simply review the code and try to find what’s wrong with yours. :)

In the next tutorial, we’re going to learn how to draw lines between each vertex & the concept of faces/triangles to obtain something like that:

See you in the second part of this series: Tutorial part 2: learning how to write a 3D soft engine from scratch in C#, TS or JS – drawing lines & triangles

David

HTML5 Gaming: benchmarking your sprites animations to target all devices & browsers

2013-04-26T16:06:57Z

While meeting some game studios, I often have the same question coming over and over: if I’m writing/porting my game in HTML5, will it run well on the various targeted devices? Will it be playable or will the gameplay suffer too much? To answer to that question, I often use my own experience based on what I know and what worked well during my own tests. But I also had the feeling it wasn’t enough to provide some good advices. In the meantime, there were some obvious facts. For instance, we all know that mobile devices can’t animate as many sprites as desktop PC and preserve 60 fps. We know also that combining SVG and Canvas is a good idea to write games that scale across devices but this could also impact the performance. Moreover, even if GPU and hardware acceleration is available on mobile, their hardware architectures differ a lot from the PC and this impacts also a lot the performances. There is dozen of scenarios like that to address and to be aware of while writing HTML5 games for mobiles. But in which proportions?

With my friend David Catuhe, we then decided to measure these various scenarios and build a benchmark framework to have a better idea on what to pay attention for. It’s named the HTML5 Potatoes Gaming Bench framework. The concept is then to help you benchmarking your targeted platforms and to obtain indicators for your future games: number of simultaneous sprites supported, SVG & Canvas composition performance, usage of videos, etc. It’s a tool we’d like to provide you to help you benchmarking your own scenarios for your games.

David has published here an article explaining how we’ve built our benchmark framework: Benchmarking a HTML5 game: HTML5 Potatoes Gaming Bench . It took us time to validate it against tools like the ones that ship wit the Windows Performance Toolkit for instance. But we won’t dig into that part here. But this is an important article to read as we needed to rely on the stability of the framework for all the future benchmarks we will build on top of it.

This article will illustrate how to use this tool and will concentrate on the number of sprites you can use in your game to maintain a good frame rate on all platforms. We will also discover interesting details on how GPU are actually being used during these scenarios.

The questions I was asking to myself before benchmarking

During my various discussions with some games developers, we were often wondering ourselves a list of recurring questions:

- What the FPS of the benchmarked machine/browser with 200, 1000, 5000 sprites?
- What is the optimum number of sprites the benchmarked device can display at 30 fps and 60 fps?
- What is the impact of the resolution of the canvas used to draw the sprites? Can I use a canvas of 800x480 only or can I scale up to 1920x1080? If so, what’s the performance cost?
- What is the performance cost of using hardware scaling (i.e. setting a style.width & style.height higher than the canvas size itself)?

I was wondering also if the performance was due to GPU or CPU limited scenarios. The idea is to know if the CPU is waiting for the GPU to do its job or the opposite.

To answer all these questions, I’ve built some specific benchmarks to measure the scenario.

The core used by all benchmarks in this article

To animate my sprites, I’ve re-used some of my existing assets like these 2 articles:

- HTML5 Gaming: animating sprites in Canvas with EaselJS
- HTML5 Gaming: building the core objects & handling collisions with EaselJS

We will then use the famous EaselJS framework from the CreateJS suite as I know it runs everywhere and has already been optimized for performance.

Please read the main logic exposed in this embedded demo available on JSFiddle:

We’re going to re-use the very same logic inside our benchmark framework.

The various benchmarks are described like the simplified code below to be able to be integrated in the HTML5 Potatoes Gaming Bench framework:

(function () {
    "use strict";

    var canvas, context, stage, screen_width, screen_height, Monsters, contentManager;

    var spritesNumber, displayBench, currentWorkBench, dynamicSpritesNumberLabel;
    var shadowsEnabled = false;
    var useRAF = false;

    var init = function (workbench, spritesNumParam, shadowsEnabledParam, canvasSizeXParam, 
                         canvasSizeYParam, then) {
        // Initialization logic, creating the canvas, the EaselJS Stage, etc.
    };

    var buildFixNumberOfSprites = function () {
        // Our Monsters collection
        Monsters = new Array();

        for (var xMonsters = 0; xMonsters < spritesNumber; xMonsters++) {
            var newRandomMonster = new Monster(contentManager, shadowsEnabled, screen_width, screen_height);
            Monsters.push(newRandomMonster);
            stage.addChild(newRandomMonster);
        }

        createjs.Ticker.addEventListener("tick", tick);
        // Best Framerate targeted (60 FPS)
        createjs.Ticker.useRAF = useRAF;
        createjs.Ticker.setFPS(60);

        // Waiting 1s before doing taking first FPS due to some warm up needed on most machines
        setTimeout(displayBench, 1000);
    };

    // tick function called-back by EaselJS
    var tick = function () {
        // looping inside the Monsters collection
        for (var monster in Monsters) {
            var m = Monsters[monster];
            // Calling explicitly each tick method 
            // to launch the update logic of each monster
            m.tick();
        }

        // update the stage:
        stage.update();
    }

    var stopEaselJSBench = function () {
        // cleaning things 
    }

    var easelJSSpritesBench_200_NoShadow_640_480 = new POTATOES.GamingBench.Bench("Drawing 200 sprites 
        (640x480)", "http://blogs.msdn.com/davrous",
        function (workbench) { // Init
            init(workbench, 200, false, 640, 480, this.onInitCompleted);
        }, function () { }, function (workbench) { // End
            stopEaselJSBench();
        });

    POTATOES.GamingBench.registerBench(easelJSSpritesBench_200_NoShadow_640_480);
})();

You will need to build equivalent code to build your own benchmarks that will be monitored by the framework. The key point is to add your benchmark definition into the monitored collection via the registerBench function.

To validate my layout (but not the performance) on the various available browsers and devices, I’ve used Browser Stack. Thanks to Modern.IE, you can obtain a 3 months trial. It lets you test and validate your websites layouts on IE from 6 to 10, Firefox, Chrome, Opera, Android & iOS devices. A very useful tool you should have a look to.

Most of the below shared results were done on an Asus Ultrabook UX31A (Zenbook Touch) running Windows 8 Pro on a Core i5 integrating an HD4000 GPU from Intel. But I’ve done also some tests on the Nokia Lumia 920, on an iPad 2 under iOS 6.0, Surface RT and Xbox 360. I’m letting you testing the shared benchmarks on your own devices.

Average FPS with 200, 1000 & 5000 sprites

Let’s start by testing some arbitrary numbers of sprites to check how your device/browser will handle that.

Just below, inside an iframe, I’ve inserted a series of benchmarks that will run some sprints of 20 seconds to display in a 640x480 canvas: 200 sprites without shadow, 200 sprites with shadows enabled, 1000 sprites & 5000 sprites without shadow.

Just press the “launch prefixed sprites numbers series” button and it will run all of them.

You can also run this benchmark in full screen or in a separate window via this link: prefixed sprites numbers series benchmark.

At the end, you will have a summary of all benchmarks with a score. The score is simply the number of frames that your device was able to render in 20 seconds. The best logical score is then 1200. As you can render 60 frames per seconds * 20 seconds = 1200 frames. We’re using requestAnimationFrame when available. So it could sometimes occurs where the tick is calling us back just a bit above 16ms, that why you could have some cases where you will have 1201 like in the following screenshot:

If you’re clicking on the average FPS computed, it will display the graph of all instant measured FPS during the complete duration of the benchmark. We’re using the famous d3.js framework for that which generate some SVG.

Press the “Back” button to go back to the summary screen or simply touch the graph also.

Some results and their analysis

I’ve first run this benchmark on my Windows 8 machine with IE10, Chrome 26, Firefox 20 all with hardware acceleration enabled. The good news is that all the modern desktop browsers have some really good hardware acceleration layers in place. IE10 seems to have the best overall score (3030 on my machine) but can’t handle the shadows as well as Chrome (average 27 fps with 200 sprites vs 12 for IE10). Chrome has an overall score of 2870. It can’t maintain 60 fps with 1000 sprites on my screen whereas IE10 and Firefox don’t have any problem to achieve that. Firefox 20 has an overall score of 2913. The most difficult test for it is to handle the 200 sprites with shadows (average of 6 fps vs 12 and 27 for IE10 and Chrome). Except that, it has the exact same scores as IE10.

So, what should we conclude at this stage? The first thing is that you absolutely shouldn’t tell to your users: “please use this browser instead” as we’re talking here about building games for the web and all platforms. The users shouldn’t adapt their usages to your code, your code should adapt itself to the usages of your users on the web. This is a different story if you’re building a game only targeting the Windows Store Apps. In this case, you just have to benchmark IE10 to determine your assets limits.

But if we think about the web in general, we see that all browsers on can easily handle between 500 and 1000 sprites at 60 fps on my desktop machine. We absolutely shouldn’t use shadows if we want to keep a good frame rate. For an HTML5 desktop game targeting integrated Intel GPU like mine, you can use a bit less than 1000 animated sprites on screen without any issue. Please keep in mind that this test doesn’t take into account some collisions tests and/or the impact of a physics engine. But it already gives some interesting data.

On the mobile side, I’ve run the same series on my Nokia Lumia 920 (Windows Phone 8 is embedding IE10). It can maintain an average of 36 fps for 200 sprites without shadow. Activating shadows is a no-go as we’re falling at 1 fps.

So, we already have some interesting information to digest. The same rendering and JavaScript engine (IE10) is able to display 1000 sprites at 60 fps on a desktop PC using an integrated GPU but 200 sprites is already too much to handle for low-end hardware like the current ARM architectures. There is a 5X to 10X performance difference between a mobile and a desktop device. So the next step now is to find what is the optimum number of sprites to maintain 60 fps (for a good desktop experience) and also 30 fps (for a relatively good mobile experience).

Optimum number of sprites at 30 and 60 fps

To help you finding the best optimal lower number of sprites that will keep a 30 or 60 fps target, I’ve built the following benchmark series:

You can also run this benchmark in full screen or in a separate window via this link: launch 30 and 60 fps target series.

Some results and their analysis

This time the score provided at the end is the number of optimal sprites to maintain 30 or 60 fps. On IE10 on my Windows 8 Asus machine, the HD4000 is able to display 3750+ sprites at 30 fps and 1600 sprites at 60 fps. IE10 on my Nokia Lumia 920 running Windows Phone 8, I’m able to display 286 sprites at 30 fps and 75 at 60 fps. On Surface RT, IE10 is able to display 370+ sprites at 30 fps and 100+ sprites at 60 fps.

We’re then falling from 1600 sprites on desktop to 75 on mobile!

I’ve even gone further: I’ve benchmarked the version of IE available on the Xbox 360. Indeed, we can imagine building a HTML5 game running inside the Xbox 360 browser. I’ve done it with my game for instance. You can test it here: http://aka.ms/platformer . You can play to the game with the gamepad and we’ve got 60+ FPS! You can also play to the game on mobile/table touch devices thanks to Hand.JS (more details here: Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS).

Let benchmark it with the 30/60 fps target series:

The Xbox 360 can display 171 sprites to maintain 30 fps and 43 to maintain 60 fps.

So, using Internet Explorer as a base of comparison, Asus Zenbook > Surface RT > Nokia Lumia 920 > Xbox 360 to maintain 60 fps in HTML5 games with sprites animations.

I was also curious about changing another parameter to confirm some of my thoughts. I was convinced that the most importance piece of hardware to boost the performance in my HTML5 games was the GPU. Indeed, all modern browsers are now heavily using them to offload the job needed to be done for the layout engine. I needed then to have the same machine with 2 GPUs available. Everything else should remain the same (CPU, memory, hard drive, screen resolution, etc.). For that, I’ve been using a Sony Vaio Z13 that has an Intel Core i7 Sandy Bridge processor with an integrated HD3000 GPU plus a discrete nVidia GT330m GPU.

Using Intel HD3000, the Vaio is able to display 2900 sprites @30fps and 1100 sprites @60fps. Using the nVidia GT330m, the very same Vaio is able to display 5400 sprites @30fps and 2500 @60fps. We then have approximately a X2 performance boost by switching from the Intel GPU to the nVidia GPU. GPU really matters for HTML5 games even for 2D canvas games.

Device tested	Max sprites @30fps	Max sprites @60fps
Asus Zenbook with HD4000	3750	1600
Sony Vaio Z13 with HD3000	2900	1100
Sony Vaio Z13 with GT330m	5400	2500
Surface RT	370	100
Nokia Lumia 920	286	75
Xbox 360	171	43

Conclusion: you have 3 options here to build a cross-platforms game running fine everywhere. You’re taking the lowest value (43 sprites here) and you will never put more that this number of sprites on screen to be sure to have 60 fps everywhere on Xbox 360, Windows Phone 8, Surface RT and Windows 8 machine running HD4000 GPUs. Of course, you need to run the same benchmark on your other targeted platforms: Android tablets & phone, iOS tablets & phone, and so on to find the optimal magic number.

Taking this first option is the easiest one but it’s a pity that desktop machines are under exploited. A second option is then to build 2 versions of the game (like 2 versions of your websites): 1 for mobile and 1 for phone. You will just adjust the graphical complexity based on the performance of each platform. Or you can also take the optimal number on mobile to target 30 fps and you will be sure to run @60 fps on desktop. The idea is then to target 30 fps on mobile/tablets and 60 on desktop.

Lastly, the third and last option is to build a game engine that will itself dynamically adjust the complexity of the graphical engine based on the performance detected. Something that some game studios are frequently doing on PC for instance. It requires more work for sure and you need also to decide the kind of assets that will be displayed or not in your game without impacting the global gameplay.

But today, doing this kind of benchmark is really critical if you’re targeting the mobile markets. Use a “mobile first” approach, otherwise you will get into trouble to optimize the performance for these lower GPUs.

Impact of the canvas resolution

This time we’re going to always display the same number of sprites (500) but we’re going to vary the resolution of the canvas: 320x200, 640x480, 1024x768 and 1920x1080. I was convinced on my side that increasing the resolution would lower the average FPS for sure, even just to display some animated sprites. Well, let’s check that:

You can also run this benchmark in full screen or in a separate window via this link: launch various canvas resolutions series .

Some results and their analysis

Well, look at the result in IE10 on my machine:

On my machine, the various resolutions had 0 impact on the average framerate! This is also confirmed on my mobile WP8 devices and on Xbox 360. Only Chrome has a lower average FPS only in 320x200 for an unknown reason. This doesn’t seem logical. But it’s probably because of a specific optimization I’m not aware of. Firefox and all other rendering/JavaScript engines I’ve been testing on several devices provide the same result: resolutions has no impact on the global performance in my scenario.

Please note also that this doesn’t mean you shouldn’t take care of the resolution of your canvas for your global performance. This just means that for sprites animations, and with this specific benchmark, the resolution has no impact. The GPU seems to take the load without any problem even on mobile platforms. I was very surprised by these first results so I’ve done further tests and benchmarks.

To double check that GPUs weren’t saturated, I had the following idea. As I’ve got a laptop machine embedding 2 GPUs, a Sony Vaio Z13. It has a nVidia GT330m and an Intel HD3000 integrated GPU. The Intel HD3000 contains up to 12 scalar 128-bit execution units where the nVidia GT330m contains 48 128-bit execution units.

There are also tools that help to check the load of the GPU. For instance, GPU-Z can provide you the GPU Load in realtime. I’ve then run the last benchmark series first on the nVidia GPU with GPU-Z opened on the “Sensors” tab to verify the GPU load: 15% at 320x200, 30% at 640x480, 25% at 1024x768, 55% at 1920x1080 (don’t ask me why it’s lower in 1024 than in 640, I really don’t know why ;)).

Then, I’ve re-run the same benchmark on the HD3000: 30% at 320x200, 40% at 640x480, 62% at 1024x768 & 100% at 1920x1080. Logically, this time, the average FPS drop from 60 fps in 320x200 –> 1024x768 to 55 fps in 1080p.

Being below 60 fps is because we’re falling into a GPU limited scenario. You can’t do anything else than lowering the rendering resolution and/or display less sprites, etc. By the way, I’ve done some testing and using “hardware scaling” like described in this article: Unleash the power of HTML 5 Canvas for gaming has almost no impact on the global average FPS as this operation seems to be done with no effort by today’s GPU even on mobile. This is then a real good option to have in mind. If the frame rate is too low due to a too high GPU load, try to render your canvas in a lower resolution and stretch it full screen using this hardware scaling method.

There could be cases where the lost of frames will be due to CPU limited scenarios. To monitor that, you need profiler tools like the one embedded in most recent browsers. You will then see which parts of your code you should try to work on to regain some FPS. You need to avoid that the GPU is waiting too long for the CPU to do its job. Some cases could be optimized using the HTML5 Web Workers for instance.

But I’ve found also some very specific cases where dropping under 60 fps was due to more complex reasons. Here is one of them: GPU-Z shows a GPU load around 50% but the FPS is below 60 fps. Using the F12 tool included in IE10 to profile the code shows the following result:

It will only show that you’re limited by the drawImage function. drawImage is a native function handled by the browser. It’s probably mixing CPU usage and GPU usage. You can’t optimize this part at this level in JavaScript. So, you just need to deal with it!

Going even further

You can have a look to the set of tools available in the Windows Performance Toolkit and read the following methodology: Measuring Browser Performance with the Windows Performance Tools.

You’re also probably wondering why we have such slight FPS drops even with a relatively constant 60 average fps:

With David Catuhe, we had a precise idea why. But again it’s better to be able to verify it. I’ve then used the following methodology:

- Create a blank Visual Studio 2012 Windows Store App project. Indeed, as Windows 8 is using IE10 to run Windows Store App in HTML5, I had just to copy/paste the code of one of the benchmarks and I had a Windows Store app ready to be analyzed by Visual Studio. :) You can download this test project here if you’d like: HTML5PotatoesModernApp.zip
- I’ve setup a unique bench displaying 1000 sprites and gone into “Analyze” –> “JavaScript Analysis” –> “UI Responsiveness” –> “Launch Startup Project”:

After some processing, you will obtain such results:

We can see that this is the GC which is responsible for the drop of some frames. I remember reading interesting details about GC in this article: Are We Fast Yet? associated to this other benchmark: HTML5-Benchmark.

Conclusion

You know what I really like about this whole story? Even with a high level layer like HTML5, understanding the targeted architectures will always help you to optimize your code for your HTML5 games.

This is really what you should keep in mind to build your games that should scale across all HTML5 compatible devices. It’s now possible to have a unique code base to run everywhere. But layout & JavaScript compatibilities is just a small part of the story. You need to know which devices you’re going to target, understand their limitations & GPU characteristics and finally benchmark all of them to take the appropriate design decisions for your games.

I really hope that this article and our benchmark framework will help you in your future HTML5 games. We will soon work on similar articles focused on other topics important for HTML5 games.

David

Introduction aux Pointer Events ou comment simplifier l’usage du tactile dans vos sites web pour tous les navigateurs

2013-03-29T18:17:03Z

Aujourd’hui, la gestion du tactile (ou « touch » en anglais) est devenu obligatoire sur vos sites web. C’est un passage obligé si vous voulez que votre site soit bien adapté aux nombreux périphériques tactiles du marché : tablettes, smartphones et la panoplie de nouveaux PC tactiles. Ce n’est pas pour autant quelque chose de forcément simple à mettre à œuvre. Entre les différentes implémentations disponibles et l’obligation de continuer à supporter la souris en parallèle, il faut parfois écrire du code un peu rébarbatif à réaliser. J’aimerais ainsi vous présenter dans cet article notre approche appelée MSPointers, une technologie multi-touch émergeante permettant de mieux gérer ce scénario. J’aimerais aussi vous parler des solutions de support multi-navigateurs grâce entre autres à un polyfill que nous avons écrit en France pour rendre le support cette technologie moins complexe. Grâce à tout cela, vous devriez être en mesure de commencer à expérimenter des solutions multi-touch universelles sur vos sites web.

Avant toute chose, il faut savoir qu’il y a plusieurs modèles aujourd’hui disponibles sur le web. Ainsi, pour supporter tous les navigateurs, en plus de nos MSPointers, il vous faut jeter un œil au modèle de touch iOS en plus du bien connu modèle W3C pour gérer la souris. Cela fait donc 3 manières différentes à gérer pour l’instant dont une seule de véritablement standardisée : la souris. Heureusement, il y a une volonté grandissante de standardisation sur le tactile puisque ce dernier est devenu de plus en plus critique. En septembre dernier, Microsoft a ainsi soumis le modèle MSPointers auprès du W3C pour une standardisation et nous sommes passés depuis le 19 février au statut du Last Call Draft : http://www.w3.org/TR/pointerevents, ce qui est très bon signe. Par ailleurs, notre équipe MS Open Tech a également publié un 1er prototype pour supporter les Pointer Events au sein de Webkit.

Il faut donc s’intéresser au modèle Pointer Events en faisant abstraction de la part de marché des périphériques le supportant. Je pense que l’approche de Microsoft sur la gestion des différents types d’interactions est vraiment différente de ce qu’il se passe sur le web actuellement et mérite que l’on s’y attarde pour avoir un aperçu de ce que pourrait devenir rapidement ce futur standard. La différence principale avec les autres modèles est que le développeur va écrire son code de manière unique en visant une abstraction de toutes les formes d’entrées via ce que nous appelons un pointeur (« Pointer »). Un pointeur peut représenter n’importe quelle forme d’interaction avec l’écran : le pointeur d’une souris, un stylet, un ou plusieurs doigts. Ainsi, vous n’aurez plus besoin de perdre votre temps à coder pour chacune des formes de manière séparée.

Durant cet article, nous allons commencer par analyser le comportement d’applications fonctionnant au sein d’Internet Explorer 10 qui expose les évènements MSPointers. Nous verrons ensuite les solutions permettant de continuer à supporter tous les navigateurs tout en visant le modèle des Pointer Events. C’est quand même là bien le but du web : écrire du code pour tous les navigateurs.

Dans un second temps, nous verrons alors comment bénéficier du service de reconnaissance de gestuelles propriétaire à IE10. Ce service vous permettra de coder les fameuses « gestures » en JavaScript de manière extrêmement simple. Pour finir, comme Windows 8 et Windows Phone 8 partage le même moteur IE, le code et les concepts présentés ici seront identiques pour chacune des 2 plateformes. De la même manière, tout ce que vous apprendrez dans cet article vous permettra de réaliser exactement les mêmes choses au sein des Windows Store apps écrite en HTML5 puisque, à nouveau, c’est toujours le même moteur derrière : IE10.

Les concepts

L’idée derrière l’objet MSPointer est donc de vous permettre d’adresser la souris, le stylet et les périphériques tactiles en utilisant une seule base de code via un pattern identique à celui que vous connaissez déjà pour gérer la souris. En effet, la souris, un stylet et un point de contact tactile ont des propriétés en commun : vous pouvez déplacer un pointeur avec eux et vous pouvez aussi cliquer sur un élément avec. Il suffit alors d’adresser les scénarios communs avec exactement le même code. Le modèle de Pointers s’occupera alors d’agréger ces propriétés communes et de les exposer de manière similaire à ce que vous connaissez déjà aujourd’hui avec les évènements souris.

Les évènements communs les plus évidents sont bien sûr : MSPointerDown , MSPointerMove & MSPointerUp qui se calquent directement sur les évènements souris équivalents. Vous aurez ainsi des coordonnées en X et Y sur l’écran disponibles au sein de ces évènements.

Mais vous avez aussi des évènements plus spécifiques comme : MSPointerOver , MSPointerOut , MSPointerHover ou MSPointerCancel

Il y a également des cas où vous allez souhaiter gérer le tactile de manière différente de la souris afin de fournir une ergonomie adaptée à votre utilisateur en fonction de sa façon d’interagir avec votre site. Par ailleurs, grâce aux écrans multipoints, un utilisateur peut facilement zoomer ou faire tourner un élément à l’aide de 2 doigts ou plus. De son côté, un stylet peut fournir des informations de pression sur l’écran qu’une souris ne peut vous renvoyer. Le modèle de Pointer Events s’occupera à nouveau d’agréger ses différences de manière à vous permettre facilement d’écrire du code spécifique à chaque usage.

Note : pour tester les exemples qui vont suivre, il serait bien entendu préférable de disposer d’un écran tactile branché sur un périphérique Windows 8/RT ou sur un téléphone Windows Phone 8. Cependant, vous avez quelques solutions alternatives :

1. Vous aurez un 1^er niveau d’expérience tactile via le simulateur Windows 8 disponible gratuitement avec la suite de développement Visual Studio 2012 Express. Pour plus de détails à ce sujet, rendez-vous dans cet article : Tester et déboguer les Touch Events d’IE10 et votre Responsive Web Design grâce au simulateur de Windows 8 et VS 2012
2. Utilisez un service de test multi-navigateurs comme BrowserStack pour tester IE10 même si vous n’avez pas de machines Windows 7/8 à votre disposition. Vous pouvez vous servir gratuitement du service BrowserStack pendant 3 mois, grâce à l’opération menée par l’équipe d’Internet Explorer sur modern.IE.

Par ailleurs, je vous conseille en complément de cet article cette excellente session des Microsoft Techdays 2013: HTML5, JavaScript et CSS3: Concevoir des interfaces tactiles à destination de Windows 8 et du web (WEB202) animée par Philippe Didiergeorges et Etienne Margraff.

Gérer les évènements tactiles simples

Etape 1 : ne rien faire d’autre que d’ajouter une ligne de CSS

Ok, commençons par les bases. Vous pouvez très facilement reprendre n’importe lequel de votre code JavaScript exploitant les évènements souris et il fonctionnera tel quel avec les stylets et périphériques tactiles sous Internet Explorer 10. En effet, IE10 lance les évènements souris en dernier ressort si vous ne gérez pas directement les Pointer Events dans votre code. C’est pour cela que vous pouvez cliquer sur un bouton ou n’importe lequel des éléments d’une page web avec vos doigts même si le développeur n’avait jamais imaginé que quelqu’un le fasse un jour ainsi. Bref, n’importe quel code s’abonnant aux évènements mousedown et/ou mouseup fonctionnera sans aucune modification du tout. Mais qu’en est-il de mousemove ?

Pour le savoir, passons en revue le comportement par défaut en analysant le résultat de cet exemple de code :

<!DOCTYPE html>
<html>
<head>
    <title>Touch article sample 1</title>
</head>
<body>
    <canvas id="drawSurface" width="400px" height="400px" style="border: 1px dashed black;">
    </canvas>
    <script>
        var canvas = document.getElementById("drawSurface");
        var context = canvas.getContext("2d");
        context.fillStyle = "rgba(0, 0, 255, 0.5)";

        canvas.addEventListener("mousemove", paint, false);

        function paint(event) {
            context.fillRect(event.clientX, event.clientY, 10, 10);
        }
    </script>
</body>
</html>

Ce code s’occupe simplement de dessiner des carrés bleus de 10 pixels par 10 pixels en suivant les mouvements de la souris au sein d’un canvas HTML5. Pour le tester, il vous suffit donc de bouger le curseur de la souris à l’intérieur de la boîte délimitée par les pointillés. Si vous avez un écran tactile, essayez d’interagir avec le canvas pour découvrir par vous-même le comportement actuel :

Exemple 0 : comportement par défaut si vous ne faîtes rien

Résultat : seul MouseDown/Up/Click fonctionnent au touch

Vous voyez ainsi que lorsque vous bougez la souris au sein du canvas, cela dessine des séries de petits carrés bleus. Mais si vous utilisez l’écran tactile à la place, vous ne pourrez uniquement afficher un seul carré bleu à l’endroit précis où vous taperez le canvas avec l’un de vos doigts. Dès que vous allez essayer de bouger votre doigt à l’intérieur du canvas, le navigateur va naturellement faire défiler le contenu de la page puisque c’est le comportement qu’on lui demande d’avoir par défaut.

Il nous faut donc indiquer au navigateur de changer de comportement et de rediriger les évènements tactiles vers votre code JavaScript plutôt que d’essayer les interpréter lui-même. Pour cela, il vous suffit de cibler les éléments de votre page qui ne doivent plus réagir au comportement par défaut en appliquant cette règle CSS :

-ms-touch-action: auto | none | manipulation | double-tap-zoom | inherit;

Vous avez différentes valeurs possibles en fonction de ce que vous souhaitez filtrer ou non. Vous trouverez la signification de ces valeurs dans cet article : Guidelines for Building Touch-friendly Sites

Le cas d’usage typique est le contrôle de cartographie que vous insérez dans votre page. Vous voulez alors en général laisser l’utilisateur se déplacer dans la carte avec les gestuelles classiques du touch (pour zoomer et se déplacer) mais garder le comportement par défaut sur le reste de la page. Dans ce cas, il faut simplement appliquer la règle ci-dessus avec la valeur none uniquement sur le conteneur HTML embarquant la cartographie.

Dans notre cas, il faut ajouter ce bloc de CSS :

<style>
    #drawSurface
    {
        -ms-touch-action: none; /* Désactive tous les comportements tactiles par défaut, comme le défilement et le zoom */
    }
</style>

Cela nous donne le résultat suivant :

Exemple 1: juste après avoir ajouté -ms-touch-action: none

Résultat: comportement par défaut du navigateur annulé et MouseMove fonctionne mais uniquement avec 1 doigt à la fois

Maintenant, si vous bougez l’un de vos doigts à l’intérieur du canvas, il se comportera comme un pointeur de souris. C’est cool non ? Mais vous devriez rapidement vous poser alors la question suivante : pourquoi cela ne suit-il qu’un seul de mes doigts ? Tout simplement car nous tombons actuellement dans le cas de dernier ressort justement. IE10 se contente de mapper l’un de vos doigts au pointeur de la souris. Et autant que je sache, on utilise qu’une seule souris à la fois. Donc 1 souris === 1 point de contact tactile maximum en utilisant cette approche. Bon, alors, comment vraiment gérer le multipoints alors ?

Etape 2 : utilisez les évènements MSPointer en lieu et place de ceux de la souris

Prenez n’importe lequel de votre code existant et remplacer vos abonnements aux évènements « mousedown/up/move » par « MSPointerDown/Up/Move » et votre code deviendra alors directement compatible avec le support multi-touch d’IE10 !

Par exemple, dans l’exemple précédent, il suffit de changer cette ligne de code :

canvas.addEventListener("mousemove", paint, false);

par celle-ci :

canvas.addEventListener("MSPointerMove", paint, false);

Et vous obtiendrez alors ce résultat :

Exemple 2: utilisation de MSPointerMove à la place de mousemove

Résultat: le multi-touch fonctionne

Vous pouvez désormais dessiner autant de séries de carrés que votre écran ne supporte de points de contacts et/ou que vous n’avez de doigts ! Encore mieux, le même code marche pour gérer le tactile, la souris et le stylet. Cela veut dire par exemple que vous pouvez dessiner une série de carrés à la souris tout en en dessinant d’autres en même temps avec vos doigts.

Mais si vous souhaitez aller plus loin et changer le comportement de votre code en fonction du type d’entrée, il faut alors tester la valeur de la propriété pointerType. Par exemple, imaginez que vous souhaitez dessiner des séries de 10x10 pixels rouges pour le tactile, 5x5 pixels verts pour le stylet et 2x2 pixels bleus pour la souris. Vous devez alors remplacer le gestionnaire précédent (la fonction paint) par celui-ci :

function paint(event) {
    if (event.pointerType) {
        switch (event.pointerType) {
            case event.MSPOINTER_TYPE_TOUCH:
                // Un écran tactile est utilisé
                // On dessine en rouge un carré de 10
                context.fillStyle = "rgba(255, 0, 0, 0.5)";
                squaresize = 10;
                break;
            case event.MSPOINTER_TYPE_PEN:
                // Un stylet est utilisé
                // On dessine en vert un carré de 5
                context.fillStyle = "rgba(0, 255, 0, 0.5)";
                squaresize = 5;
                break;
            case event.MSPOINTER_TYPE_MOUSE:
                // Une souris est utilisée
                // On dessine en blue un carré de 2
                context.fillStyle = "rgba(0, 0, 255, 0.5)";
                squaresize = 2;
                break;
        }

        context.fillRect(event.clientX, event.clientY, squaresize, squaresize);
    }
}

Et vous pouvez tester le nouveau résultat ici :

Exemple 2b: test de pointerType pour différencier tactile/stylet ou souris

Résultat: Vous pouvez changer le comportement de votre code pour chacune des entrées mais depuis 2a le code ne marche maintenant uniquement dans IE10

Si vous faîtes partis des chanceux à disposer d’une machine supportant les 3 types d’entrées (comme le Sony Duo 11, la Microsoft Surface Pro ou la tablette Samsung que certains d’entre nous avions eu à la BUILD2011), vous serez en mesure de voir les différences visuelles entre les 3.

Bon, tout cela est bien sympathique mais nous avons désormais un problème de taille avec ce code. Il gère désormais bien tous les types d’entrées au sein d’IE10 mais ne marche pas du tout dans les navigateurs ne supportant pas encore le modèle de Pointer Events comme IE9, Chrome, Firefox, Opera & Safari !

Etape 3 : détection de fonctionnalité pour fournir une expérience alternative

Comme tout bon développeur web qui se respecte, vous savez déjà surement que la meilleure approche pour supporter l’ensemble des navigateurs est de faire de la détection de fonctionnalités. Dans notre cas, cela consiste à tester cette fonctionnalité :

window.navigator.msPointerEnabled

Faites attention cependant à ce test. Il vous indiquera uniquement si le navigateur actuel supporte les MSPointers. Il ne vous dira pas si un écran tactile est disponible ou pas. Pour tester la fonctionnalité tactile, il vous faut en plus vérifier msMaxTouchPoints.

En conclusion, pour avoir un code fonctionnel supportant à la fois le modèle MSPointer d’IE10 et proposant une expérience alternative dégradée vers le pointeur de souris pour les autres navigateurs, vous avez besoin du code suivant :

var canvas = document.getElementById("drawSurface");
var context = canvas.getContext("2d");
context.fillStyle = "rgba(0, 0, 255, 0.5)";
if (window.navigator.msPointerEnabled) {
    // Si les Pointer events sont supportés.
    canvas.addEventListener("MSPointerMove", paint, false);
}
else {
    canvas.addEventListener("mousemove", paint, false);
}

function paint(event) {
    var squaresize = 2;
    if (event.pointerType) {
        switch (event.pointerType) {
            case event.MSPOINTER_TYPE_TOUCH:
                context.fillStyle = "rgba(255, 0, 0, 0.5)";
                squaresize = 10;
                break;
            case event.MSPOINTER_TYPE_PEN:
                context.fillStyle = "rgba(0, 255, 0, 0.5)";
                squaresize = 5;
                break;
            case event.MSPOINTER_TYPE_MOUSE:
                context.fillStyle = "rgba(0, 0, 255, 0.5)";
                squaresize = 2;
                break;
        }
    }

    context.fillRect(event.clientX, event.clientY, squaresize, squaresize);
}

A nouveau, vous pouvez vérifier le résultat ici :

Exemple 3: détection de la fonctionnalité msPointerEnabled pour fournir une dégradation

Résultat: expérience complète au sein d’IE10 et comportement classique à la souris dans les autres navigateurs

L’exemple 3 fonctionnera alors en multipoints, à la souris et au stylet dans IE10 et uniquement à la souris dans IE9, Chrome, Firefox, Opera, Safari.

Etape 4 : supporter tous les modèles de touch

J’imagine que vous aimeriez bien aller plus loin qu’une simple expérience souris dégradée pour les navigateurs autre qu’IE10. Pour cela, il vous faut alors supporter les 3 modèles dont je parlais au début : Touch Events d’Apple, évènements souris W3C et MSPointer en attendant sa standardisation finale. Il y a 2 manières de mettre cela en place :

1 – Ecrire du code pour adresser tous les types d’évènements en parallèle comme expliqué (en anglais) dans cet article de l’équipe IE : Handling Multi-touch and Mouse Input in All Browsers
2 – Ajouter simplement une référence à Hand.JS, une librairie de polyfill en JavaScript conçue par mon ami David Catuhe comme il l’explique dans son article ici : HandJS a polyfill for supporting pointer events on every browser

En effet, David a écrit cette superbe librairie vous permettant de viser la spécification Pointer Events du W3C. Comme je vous l’ai indiqué en début d’article, Microsoft a soumis son modèle MSPointer Events au W3C pour standardisation.

Cependant, bien que déjà au statut de « Last Call », la spécification Pointer Events n’est pas encore un standard. Mais grâce à cette librairie, vous pouvez déjà implémenter dans votre code ce futur standard et être prêt lorsqu’elle deviendra adoptée par tous les navigateurs modernes. En attendant, la librairie de David s’occupera de propager les Pointer Events vers la forme actuellement propriétaires MSPointer Events dans IE10, vers les Touch Events pour les navigateurs implémentant le modèle d’Apple, voir même vers les évènements souris en dernier ressort pour les anciens navigateurs ! Je vous invite donc à lire son article pour savoir comment cela fonctionne.

Pour avoir une idée de comment utiliser cette librairie, je me suis amusé à m’en servir dans cet article : Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS qui montre comment écrire un joystick tactile virtuel utilisant les Pointer Events. Grâce à Hand.JS, il fonctionne dans IE10 sur Windows 8/RT, Windows Phone 8, iPad/iPhone et sur les périphérique Android/Firefox OS avec la même base de code !

Reconnaitre des gestuelles

Maintenant que nous savons comment fonctionnent les bases du multi-touch, regardons comment reconnaître des gestuelles simples comme taper un élément ou le maintenir puis ensuite des gestuelles plus avancées comme déplacer ou agrandir/rétrécir un élément.

IE10 fournit un objet nommé MSGesture qui va bien nous aider pour ce travail. Notez d’ailleurs que cet objet est spécifique à IE10 et ne fait pas partie de notre soumission au W3C. Combiné avec l’élément MSCSSMatrix (notre equivalent à WebKitCSSMatrix), vous serez en mesure de construire des expériences multi-touch très intéressantes le tout de manière fort simple. MSCSSMatrix représente en effet une matrice 4x4 homogène permettant un accès au DOM par script aux fonctionnalités CSS 2-D et 3-D. Mais avant de jouer avec cela, commençons naturellement par les bases de la reconnaissance de gestuelles (ou « gesture » en anglais).

Ce qu’il faut retenir en premier lieu est qu’il est d’abord nécessaire de s’abonner à l’évènement MSPointerDown. Ensuite, à l’intérieur du handler prenant en charge MSPointerDown, vous devrez choisir quels sont les pointeurs que vous souhaitez envoyer à l’objet MSGesture pour qu’il s’occupe de détecter certaines gestuelles spécifiques. Ce dernier renverra alors l’un de ces évènements : MSGestureTap, MSGestureHold, MSGestureStart, MSGestureChange, MSGestureEnd, MSInertiaStart. En effet, l’objet MSGesture travaille sur un jeu de pointeurs en entrée et s’occupe ensuite d’appliquer un algorithme de reconnaissance de gestuelles par-dessus. La seule chose que vous devez faire est donc de choisir/filtrer les pointeurs devant faire partie de la gestuelle (basé sur leurs identifiants, coordonnées à l’écran, etc… à vous d’imaginer la logique qui vous correspond). L’objet MSGesture s’occupera alors magiquement du reste.

Exemple 1 : gérer la gestuelle de maintien (hold gesture)

Nous allons voir ici comment maintenir un élément (un simple élément DIV contenant une image en arrière-plan). Cette gestuelle est souvent utilisée pour simuler les actions contextuelles à l’élément de la même manière que le bouton droit de la souris. Une fois que notre élément sera maintenu, nous allons ajouter des petits triangles aux quatre coins pour indiquer à l’utilisateur qu’il a bien sélectionné l’élément. Ces coins seront générés dynamiquement en créant 4 DIV ajoutés au-dessus de chacun des coins de l’image. Ensuite, grâce à quelques astuces CSS à base de transformation 2D et gradients linéaires, nous obtiendrons la chose suivante :

La séquence sera donc la suivante :

1 – abonnement aux évènements MSPointerDown & MSPointerHold au-dessus de l’élément HTML qui vous intéresse
2 – création d’un objet MSGesture qui visera le même élément HTML
3 – au sein du gestionnaire de MSPointerDown, ajout à l’objet MSGesture la prise en charge des différents PointerID que l’on souhaite surveiller (tous ou un sous-ensemble d’entre eux en fonction de ce que vous souhaitez réaliser)
4 – au sein du gestionnaire MSPointerHold, vérification au sein des détails fournis dans l’évènement si l’utilisateur a juste démarré la gestuelle de maintien (via le flag MSGESTURE_FLAG_BEGIN). Si oui, on ajoute les coins spéciaux, sinon, on les retire.

Cela nous amène au code suivant :

<!DOCTYPE html>
<html>
<head>
    <title>Touch article sample 5: simple gesture handler</title>
    <link rel="stylesheet" type="text/css" href="toucharticle.css" />
    <script src="Corners.js"></script>
</head>
<body>
    <div id="myGreatPicture" class="container" />
    <script>
        var myGreatPic = document.getElementById("myGreatPicture");
        // Création d’un nouveau MSGesture qui surveillera l’élément DOM myGreatPic
        var myGreatPicAssociatedGesture = new MSGesture();
        myGreatPicAssociatedGesture.target = myGreatPic;

        // Vous devez d’abord vous abonner à MSPointerDown pour pouvoir
        // avoir accès à des évènements gestuelles plus complexes 
        myGreatPic.addEventListener("MSPointerDown", pointerdown, false);
        myGreatPic.addEventListener("MSGestureHold", holded, false);

        // Une fois l’évènement levé, on se contente d’envoyer tous les pointeurs vers MSGesture
        function pointerdown(event) {
            myGreatPicAssociatedGesture.addPointer(event.pointerId);
        }

        // Cet évènement sera levé par l’objet MSGesture
        // en fonction des pointeurs fournis pendant l’évènement MSPointerDown
        function holded(event) {
            // La gestuelle commence, on ajoute les coins
            if (event.detail === event.MSGESTURE_FLAG_BEGIN) {
                Corners.append(myGreatPic);
            }
            else {
                // L’utilisateur a relâché son doigt, la gestuelle se termine
                // On retire donc les coins
                Corners.remove(myGreatPic);
            }
        }

        // Pour éviter d’avoir l’équivalent d’un menu contextuel affiché   
        // comme sur un click droit "right click" sur MSPointerUp, 
        // on annule le comportement du navigateur par défaut
        myGreatPic.addEventListener("contextmenu", function (e) {
            e.preventDefault();    
        }, false);
    </script>
</body>
</html>

Et voici le résultat :

Essayez de juste taper l’élément ou de cliquer dessus à la souris, rien ne se passe. Touchez le et maintenez seulement 1 doigt au-dessus de l’image ou effectuez un click gauche long dessus : les coins apparaissent. Relâchez votre doigt, les coins disparaissent bien.

Touchez et maintenez 2 ou plus de doigts sur l’image, rien ne se passe. En effet, la gestuelle de maintien est uniquement levée que lorsqu’un seul doigt maintient l’élément.

Note : les bords blancs, les coins et l’image de fond sont définis par CSS dans le fichier toucharticle.css. Corners.js s’occupe simplement de créer les 4 DIVs (avec la fonction append) et les place au-dessus de l’élément dans chacun des coins à l’aide de classes CSS appropriées.

Malgré tout, il reste un comportement avec lequel je ne suis pas très satisfait. Une fois que vous maintenez l’image, dès que vous bougez votre doigt ne serait-ce que très légèrement, le drapeau MSGESTURE_FLAG_CANCEL est levé et récupéré par le gestionnaire d’évènement qui retire aussitôt les coins. Je préférais retirer les coins uniquement lorsque l’utilisateur relâche son doigt n’importe où au-dessus de l’image ou aussitôt que son doigt sorte de la zone délimitée par l’image en elle-même. Pour effectuer cela, nous allons retirer les coins uniquement après les évènements MSPointerUp ou MSPointerOut. Cela nous donne le code suivant à la place :

var myGreatPic = document.getElementById("myGreatPicture");
var myGreatPicAssociatedGesture = new MSGesture();
myGreatPicAssociatedGesture.target = myGreatPic;

myGreatPic.addEventListener("MSPointerDown", pointerdown, false);
myGreatPic.addEventListener("MSGestureHold", holded, false);
myGreatPic.addEventListener("MSPointerUp", removecorners, false);
myGreatPic.addEventListener("MSPointerOut", removecorners, false);

function pointerdown(event) {
    myGreatPicAssociatedGesture.addPointer(event.pointerId);
}

function holded(event) {
    if (event.detail === event.MSGESTURE_FLAG_BEGIN) {
        Corners.append(myGreatPic);
    }
}

// On retire désormais les coins que sur Pointer Up ou Out
function removecorners(event) {
    Corners.remove(myGreatPic);
}

myGreatPic.addEventListener("contextmenu", function (e) {
    e.preventDefault();    
}, false);

Qui fournit le comportement que je cherchais vraiment à obtenir :

Exemple 2 : gestion des déplacements, rotation et mise à l’échelle

Maintenant que vous savez presque tout, vous allez voir que mettre à l’échelle, déplacer ou tourner un élément est l’enfance de l’art grâce à quelques lignes de code. Vous devez tout d’abord vous abonner à l’évènement MSGestureChange. Cet évènement va vous envoyer les différents attributs comme définis dans la documentation de l’objet MSGestureEvent comme les valeurs actuellement positionnées sur votre élément HTML.

Encore mieux, par défaut, l’objet MSGesture fournit un algorithme inertiel automatique. Cela signifie que vous pouvez prendre n’importe quel élément HTML et le jeter à travers l’écran en utilisant vos petits doigts et l’animation naturelle sera gérée pour vous.

Pour finir, afin de refléter ces changements envoyés par MSGesture, vous devez bien entendu bouger l’élément vous-même. La manière la plus simple consiste à appliquer une transformation CSS parfaitement calquée sur les valeurs de rotation, mise à l’échelle et déplacement fournies par la gestuelle reconnue. Pour cela, utilisez l’élément MSCSSMatrix.

En conclusion, si vous souhaitez mettre en place toutes ces gestuelles sympathiques et modernes dans l’exemple précédent, abonnez vous à l’évènement ainsi :

myGreatPic.addEventListener("MSGestureChange", manipulateElement, false);

Puis utilisez le gestionnaire suivant :

function manipulateElement(e) {
    // Décommenter le code suivant si vous voulez désactiver l’algorithme d’inertie automatique 
    // fournit par le mécanisme de reconnaissance de gestuelles
    // if (e.detail == e.MSGESTURE_FLAG_INERTIA)
    // return;

    // On récupère la dernière transformation CSS appliquée à l’élément
    var m = new MSCSSMatrix(e.target.currentStyle.transform); 
    e.target.style.transform = m
    .translate(e.offsetX, e.offsetY) // On bouge l’origine au centre de la gestuelle
    .rotate(e.rotation * 180 / Math.PI) // On applique la rotation
    .scale(e.scale) // La mise à l’échelle
    .translate(e.translationX, e.translationY) // Le déplacement
    .translate(-e.offsetX, -e.offsetY); // Et on remet l’origine à sa place initiale
}

Ce qui donne le résultat final suivant :

Essayez de déplacer et lancer l’image dans la zone à fond noir avec 1 ou plusieurs doigts. Essayez également de zoomer ou faire tourner l’image avec 2 ou plusieurs doigts. Le résultat est vraiment agréable et le code pour le réaliser extrêmement simple puisque toute la complexité a été prise en charge nativement par IE10.

Liens directs vers tous les exemples

Si vous n’avez pas d’écran tactile ou si vous n’avez pas de moyens simples de tester IE10 (malgré le site Modern.IE), n’hésitez pas à jeter un œil à la session Microsoft Techdays 2013 : HTML5, JavaScript et CSS3: Concevoir des interfaces tactiles à destination de Windows 8 et du web (WEB202) pour voir l’ensemble en action.

Sinon, voici la liste des exemples embarqués dans les iframes de cet article:

- Simple touch sample 0 with nothing done
- Simple touch sample 1 with CSS -ms-touch-action
- Simple touch sample 2a with basic MSPointerMove implementation
- Simple touch sample 2b with pointerType differentiation
- Simple touch sample 3 with MSPointers and mouse fallback
- MSGesture sample 1: MSGestureHold handler
- MSGesture sample 1b: MSGestureHold handler
- MSGesture sample 2: MSGestureChange

Quelques ressources associées :

- W3C Pointer Events Specification
- HandJS a polyfill for supporting pointer events on every browser : la librairie JavaScript de polyfill de David Catuhe
- Pointer and gesture events
- Go Beyond Pan, Zoom, and Tap Using Gesture Events
- IE Test Drive Browser Surface qui a grandement inspiré mes démos
- N’hésitez pas à essayer des expériences tactiles dans IE10. Vous en trouverez un joli florilège ici : http://www.internetexplorer.fr/touchgallery . Je vous conseille notamment :
- Contre Jour , un excellent jeu en HTML5 accompagné de son retour d’expérience à lire en français ici : Behind The Scenes !
- Atari Arcade Games , la suite de jeux Atari portés en HTML5 grâce à CreateJS.

Logiquement, avec tous les détails que je vous ai partagés dans cet article ainsi que les ressources associées, vous devriez être en mesure de commencer à implémenter le modèle de Pointer Events dans vos sites web & vos applications pour le Windows Store. Vous avez désormais l’opportunité de facilement augmenter l’expérience utilisateur de vos sites sous Internet Explorer 10.

David

Coding4fun: the Star Wars intro in pure CSS3 thanks to 3D transform & animations

2013-02-25T19:54:40Z

During the French Microsoft Techdays that occurred from the 12^th to the 14^th of February 2013, I’ve built a fun & stupid demo to start the Coding4Fun session. The idea was to rebuild the famous StarWars intro with the 3D scrolling text and the slide down effect with a star destroyer ship coming at the end. This useless demo was built using only some CSS3 3D transform & animation and the HTML5 audio & video tags.

Yes, it uses 0 line of JavaScript !!!

This demo is mainly based on this awesome article: Star Wars 3D Scrolling Text in CSS3 from Craig Buckler. But I’ve tweaked it a little bit by adding some parts at the beginning. The most cool part was added at the end of the text scrolling.

Here is a video of the result running inside IE10:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

And you can of course run the same demo directly in your favorite browser from here: Coding4Fun StarWars Intro in pure CSS . It lasts approximately 140 seconds to run the complete demo. Of course, you won’t have audio in this demo as John Williams’ music is copyrighted. So, if you’d like to have the complete video & audio experience, go buy the music composed by John Williams (the best composer in the universe !) and convert it to mp3 and/or ogg to map the audio tag definition you’ll find in the source code. In my case, I’ve just added 24 seconds of blank sound to synchronize my keyframes with the music itself.

If you don’t speak French, don’t pay attention to the text. I was too lazy to translate it in English as it’s using some French developer’s unique humor. I’m sure you don’t want to share that.

Update: some of my UK friends, like Martin Beeby, noted that as it’s only some pure HTML text used, you just need an online translator to target the language of your choice! For instance, try the English version or the Chinese version.

The demo itself should run fine inside IE10 on Windows 7/Windows 8/Windows Phone 8, Firefox, Safari & Chrome. Opera, which doesn’t support yet 3D transform, will then not run properly all parts. The experience is slightly better in IE10 as it’s the only one supporting the CSS Viewport specification. It’s also super fluid on my PC running a simple HD4000 GPU and on my Nokia Lumia 920. Hardware acceleration is definitely great for that!

If you’d like to understand how it works, all the demo logic is contained inside index.css which use a lot of keyframes definition with the appropriate timings. Note that it uses the standard CSS animation & transform unprefixed specification supported by IE10, Firefox and Opera. To support older experimental Webkit implementations, I’ve added a webkit-index.css file containing the very same logic but with the –webkit prefix for Chrome & Safari browsers. Let’s hope that those browsers will soon implement the unprefixed versions to join the club made of IE10, Firefox & Opera to definitely get rid of vendor prefixes on those specifications.

Compared to the original Craig’s demo, I’ve added a simple intro with some fading effects and an additional vertical scrolling at the end of the 3d text scrolling which ends to a html5 looping video.

Here is a series of images showing this final sequence done:

The final Star Detroyer is simply using the CSS 3D transform trick coupled with a CSS animation. The HTML associated with this sequence is very very simple:

<video class="fondvideo" poster="fondfixe.jpg" autoplay loop>
    <source src="hprichv.mp4" />
    <source src="hprichv.ogv" />
</video>

<img class="galaxy" src="bg_image.jpg" />

<img class="destroyer" src="stardestroyer.png" />
<div class="finalfadding"></div>

And here is the associated CSS to achieve the animations:

/* Final ship animation. The trick is to play on 3D rotation and Z translation */
/* To mimic again the first Star Destroyer animation */
@keyframes scrollship {
    0% {
        opacity: 0;
        transform: perspective(300px) translateZ(150px) rotateX(-80deg);
    }

    3% {
        opacity: 1;
    }

    100% {
        opacity: 1;
        transform: perspective(300px) translateZ(-4000px) rotateX(-80deg);
    }
}

/* Translating the galaxy image background on the Y axis to again */
/* mimic the StarWars intro */
@keyframes galaxyscrolling {
    0% {
        top: 0%;
    }

    100% {
        top: -100%;
    }
}

/* The video tag is animated in the same time as the galaxy  */
/* to achieve a continuous animation */
@keyframes videoscrolling {
    0% {
        top: 55%;
    }

    100% {
        top: -40%;
    }
}

/* Final black fading */
@keyframes finalfaddingkeyframes {
    0% {
        opacity: 0;
    }

    100% {
        opacity: 1;
    }
}

/* First image background */
.galaxy {
    width: 100%;
    height: 100%;
    position: absolute;
    top: 0%;
    animation: galaxyscrolling 15s ease-out 111s 1;
    animation-fill-mode: forwards;
}

/* background HTML5 looping video */
.fondvideo {
    position: absolute;
    width: 100%;
    height: 200%;
    top: 55%;
    animation: videoscrolling 15s ease-out 111s 1;
    animation-fill-mode: forwards;
}

/* Final destroyer animation. It's just a PNG image animated using 3D Transform */
.destroyer {
    position: absolute;
    top: -70%;
    opacity: 0;
    transform-origin: 50% 100%;
    width: 100%;
    height: 100%;
    transform: perspective(300px) translateZ(150px) rotateX(-80deg);
    animation: scrollship 120s linear 125s 1;
    animation-fill-mode: forwards;
}

/* final fading using a black div */
.finalfadding {
    position: absolute;
    width: 100%;
    height: 100%;
    opacity: 0;
    background-color: black;
    animation: finalfaddingkeyframes 5s ease-out 140s 1;
    animation-fill-mode: forwards;
}

The timing are the one currently set for the global sequence. You see for instance that the final fading only occurs after 140s. You can see also that the final ending black fading is simply done with an overlay black DIV getting from opacity 0 to 1.

The super cool HTML5 video is the earth shooted by the NASA from the space and showing some magic aurora effects:

Credit: Image Science & Analysis Laboratory, NASA Johnson Space Center

You may find other videos like that from the NASA on their website: Videos Aurora or here Crew Earth Observations Videos

Finally, to try to control the viewport to have the same experience across resolutions, I’m using the @viewport CSS specification coming from CSS Device Adaptation, Section 4.

@-webkit-viewport {width: 1366px;height: 768px;}
@-moz-viewport {width: 1366px;height: 768px;}
@-ms-viewport {width: 1366px;height: 768px;}
@-o-viewport {width: 1366px;height: 768px;}
@viewport {width: 1366px;height: 768px;}

This specification is currently only supported by IE10 in the desktop world. The demo doesn’t break in other browsers. But the experience is a bit better in IE10 as we have more control on the various resolutions thanks to the viewport rule.

Under Internet Explorer 10, try for instance to resize the window during each sequence of the demo and you’ll see what I mean. If you don’t have a way to test Internet Explorer 10, you have 2 options:

1 – Use Browser Stack to test it. You have a 3 months offering thanks to our Modern.IE website. You should have a look to it!
2 – Have a look to this short video:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

That’s all folks. I hope that some of you will enjoy this funky experiment.

David

Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS

2013-02-22T13:28:00Z

I’m currently working on several gaming projects for modern browsers and Windows 8 Store projects. Some of them are using HTML5 as a base in order to simplify multi-devices targeting. I was then looking for a unified way to address all inputs on all platforms: Windows 8/RT, Windows Phone 8, iPad, Android & FirefoxOS.

As you’ve maybe discovered in my previous article Unifying touch and mouse- how Pointer Events will make cross-browsers touch support easy, IE10 on Windows 8/RT & Windows Phone 8 implement the Pointer Events model we’ve submitted to the W3C. In order to address in a unified way this Pointer Events model and the one implemented in WebKit based browsers, we’re going to use David Catuhe’s HandJS library. Check out his blog’s post here: HandJS a polyfill for supporting pointer events on every browser. This idea is to target the Pointer model and the library will propagate the touch events to all platforms specifics.

Once I had all the technical pieces in hand, I was looking for a great way to implement a virtual touch joystick in my game. I’m not a huge fan of arrow keys buttons to be touched. On another side, the virtual analogic pad are often not very well placed. But I’ve finally discovered that Seb Lee-Delisle had already digested this and has created this awesome concept described in Multi-touch game controller in JavaScript/HTML5 for iPad . The code is available on GitHub here: JSTouchController

The idea was then to take his code and refactor the touch part to target the Pointer model instead of the original WebKit Touch approach. While working on that several months ago, I’ve discovered that Boris Smus from Google had already started to more or less do it. It’s been done while he was working on his own Pointer.js library as described in his article Generalized input on the cross-device web. However, at that time, Boris was mimicking an old version of the IE10 Pointer Events implementation and his library wasn’t working in IE10. That’s why, even if Boris’ work was awesome, we’ve decided to work on our own version. Indeed, David’s library is currently targeting the last and very recent W3C version currently in last call draft. If you’re having a look to both librairies, you’ll see also that HandJS is using some different approaches in several parts of the code. We will then use HandJS in this article to build our touch joystick.

Sample 1: Pointers tracker

This sample helps you tracking the various inputs on the screen. It tracks and follow the various fingers pressing the canvas element. It’s based on Seb’s sample available on GitHub here: Touches.html

Thanks to Hand.js, we’re going to make it compatible for all browsers. It’s even also going to track the stylus and/or the mouse based on the type of hardware you’re currently testing on!

Here is an HTML5 video of the result in IE10 running under Windows 8. You’ll find see some cyan circles tracking the fingers, then a red circle tracking the mouse and a green circle tracking the pen:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

The same webpage provides the very same result under Chrome on Windows 8 or on an iOS/Android/FirefoxOS devices (except that pen is only supported by IE10). Thanks to HandJS, write it once and it will run everywhere!

You’ve seen in the video that the cyan pointers are of type “TOUCH” whereas the red one is of type “MOUSE”. If you have a touch screen, you can experience the same result by testing this page embedded in this iframe:

This sample works fine on a Windows 8/RT touch device, a Windows Phone 8, an iPad/iPhone or Android/FirefoxOS device! If you don’t have a touch device, HandJS will automatically fallback to mouse. You should then be able to track at least 1 pointer with your mouse.

Let’s see how to obtain this result in a unified way. All the code lives in Touches.js:

"use strict";

// shim layer with setTimeout fallback
window.requestAnimFrame = (function () {
    return window.requestAnimationFrame ||
    window.webkitRequestAnimationFrame ||
    window.mozRequestAnimationFrame ||
    window.oRequestAnimationFrame ||
    window.msRequestAnimationFrame ||
    function (callback) {
        window.setTimeout(callback, 1000 / 60);
    };
})();

var pointers; // collections of pointers

var canvas,
c; // c is the canvas' context 2D

document.addEventListener("DOMContentLoaded", init);

window.onorientationchange = resetCanvas;
window.onresize = resetCanvas;

function init() {
    setupCanvas();
    pointers = new Collection();
    canvas.addEventListener('pointerdown', onPointerDown, false);
    canvas.addEventListener('pointermove', onPointerMove, false);
    canvas.addEventListener('pointerup', onPointerUp, false);
    canvas.addEventListener('pointerout', onPointerUp, false);
    requestAnimFrame(draw);
}

function resetCanvas(e) {
    // resize the canvas - but remember - this clears the canvas too.
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    //make sure we scroll to the top left.
    window.scrollTo(0, 0);
}

function draw() {
    c.clearRect(0, 0, canvas.width, canvas.height);

    pointers.forEach(function (pointer) {
        c.beginPath();
        c.fillStyle = "white";
        c.fillText(pointer.type + " id : " + pointer.identifier + " x:" + pointer.x + " y:" + 
                   pointer.y, pointer.x + 30, pointer.y - 30);

        c.beginPath();
        c.strokeStyle = pointer.color;
        c.lineWidth = "6";
        c.arc(pointer.x, pointer.y, 40, 0, Math.PI * 2, true);
        c.stroke();
    });

    requestAnimFrame(draw);
}

function createPointerObject(event) {
    var type;
    var color;
    switch (event.pointerType) {
        case event.POINTER_TYPE_MOUSE:
            type = "MOUSE";
            color = "red";
            break;
        case event.POINTER_TYPE_PEN:
            type = "PEN";
            color = "lime";
            break;
        case event.POINTER_TYPE_TOUCH:
            type = "TOUCH";
            color = "cyan";
            break;
    }
    return { identifier: event.pointerId, x: event.clientX, y: event.clientY, type: type, color: color };
}

function onPointerDown(e) {
    pointers.add(e.pointerId, createPointerObject(e));
}

function onPointerMove(e) {
    if (pointers.item(e.pointerId)) {
        pointers.item(e.pointerId).x = e.clientX;
        pointers.item(e.pointerId).y = e.clientY;
    }
}

function onPointerUp(e) {
    pointers.remove(e.pointerId);
}

function setupCanvas() {
    canvas = document.getElementById('canvasSurface');
    c = canvas.getContext('2d');
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
    c.strokeStyle = "#ffffff";
    c.lineWidth = 2;
}

Well, I think the code is pretty straightforward. I’m registering to the pointerdown/move/up event like described in my introduction article on MSPointer Events. In the pointerdown handler, I’m catching the ID, the X & Y coordinates and the type of pointers (touch, pen or mouse) inside an object generated on the fly pushed in the pointers collection object. This collection is indexed by the id of the pointers. The collection object is described in Collection.js. The draw() function is then enumerating this collection to draw some cyan/red/lime circles based on the type at the exact position where you’re touching the screen. It also adds some text on each circle’s side to display the pointer’s details. The pointermove handler updates the coordinates of the associated pointer in the collection and the pointerup/out simply removes it from the collection. Hand.JS makes this code compatible for IE10 by propagating pointerdown/move/up/out to the associated MSPointerDown/Move/Up/Out events and to the touchstart/move/end events for WebKit’s browsers.

If you wish, you can view the complete source code here: http://david.blob.core.windows.net/html5/touchjoystick/Touches.html

Sample 2: Video game controller with a simple spaceship game

Let’s now see the sample I was the most interested in. You will probably also if you’re looking for a virtual analog touch pad for your HTML5 games. The idea is to touch anywhere on the left side of the screen. At the exact position where you’ll touch the screen, it will display a simple but very efficient pad. Moving your finger will update the virtual touch pad and will move a simple spaceship. Touching the right side of the screen will display some red circles and those circles will generate some bullets getting out of the spaceship. Once again, It’s based on Seb’s sample available on GitHub here: TouchControl.html

Here is a video of the updated sample result in IE10 under Windows 8:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

If you have a touch screen, you can live test the page in this iframe:

Otherwise, you will only be able to move the ship using your mouse by clicking on the left of the screen or firing by clicking on the right side, but you won’t be able to achieve both actions simultaneously. Indeed, HandJS is providing a mouse fallback if the browser or platform doesn’t support touch.

Note: the iPad seems to suffer from an unknown bug which prevents this second iframe to work correctly. Open the sample directly in another tab to make it works on your iPad.

Let’s see again how to obtain this result in a unified way. All the code lives this time in TouchControl.js:

// shim layer with setTimeout fallback
window.requestAnimFrame = (function () {
    return window.requestAnimationFrame ||
    window.webkitRequestAnimationFrame ||
    window.mozRequestAnimationFrame ||
    window.oRequestAnimationFrame ||
    window.msRequestAnimationFrame ||
    function (callback) {
        window.setTimeout(callback, 1000 / 60);
    };
})();

var canvas,
c, // c is the canvas' context 2D
container,
halfWidth,
halfHeight,
leftPointerID = -1,
leftPointerPos = new Vector2(0, 0),
leftPointerStartPos = new Vector2(0, 0),
leftVector = new Vector2(0, 0);

var pointers; // collections of pointers
var ship;
bullets = [],
spareBullets = [];

document.addEventListener("DOMContentLoaded", init);

window.onorientationchange = resetCanvas;
window.onresize = resetCanvas;

function init() {
    setupCanvas();
    pointers = new Collection();
    ship = new ShipMoving(halfWidth, halfHeight);
    document.body.appendChild(ship.canvas);
    canvas.addEventListener('pointerdown', onPointerDown, false);
    canvas.addEventListener('pointermove', onPointerMove, false);
    canvas.addEventListener('pointerup', onPointerUp, false);
    canvas.addEventListener('pointerout', onPointerUp, false);
    requestAnimFrame(draw);
}

function resetCanvas(e) {
    // resize the canvas - but remember - this clears the canvas too. 
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    halfWidth = canvas.width / 2;
    halfHeight = canvas.height / 2;

    //make sure we scroll to the top left. 
    window.scrollTo(0, 0);
}

function draw() {
    c.clearRect(0, 0, canvas.width, canvas.height);

    ship.targetVel.copyFrom(leftVector);
    ship.targetVel.multiplyEq(0.15);
    ship.update();

    with (ship.pos) {
        if (x < 0) x = canvas.width;
        else if (x > canvas.width) x = 0;
        if (y < 0) y = canvas.height;
        else if (y > canvas.height) y = 0;
    }

    ship.draw();

    for (var i = 0; i < bullets.length; i++) {
        var bullet = bullets[i];
        if (!bullet.enabled) continue;
        bullet.update();
        bullet.draw(c);
        if (!bullet.enabled) {
            spareBullets.push(bullet);

        }
    }

    pointers.forEach(function (pointer) {
        if (pointer.identifier == leftPointerID) {
            c.beginPath();
            c.strokeStyle = "cyan";
            c.lineWidth = 6;
            c.arc(leftPointerStartPos.x, leftPointerStartPos.y, 40, 0, Math.PI * 2, true);
            c.stroke();
            c.beginPath();
            c.strokeStyle = "cyan";
            c.lineWidth = 2;
            c.arc(leftPointerStartPos.x, leftPointerStartPos.y, 60, 0, Math.PI * 2, true);
            c.stroke();
            c.beginPath();
            c.strokeStyle = "cyan";
            c.arc(leftPointerPos.x, leftPointerPos.y, 40, 0, Math.PI * 2, true);
            c.stroke();

        } else {

            c.beginPath();
            c.fillStyle = "white";
            c.fillText("type : " + pointer.type + " id : " + pointer.identifier + " x:" + pointer.x + 
                       " y:" + pointer.y, pointer.x + 30, pointer.y - 30);

            c.beginPath();
            c.strokeStyle = "red";
            c.lineWidth = "6";
            c.arc(pointer.x, pointer.y, 40, 0, Math.PI * 2, true);
            c.stroke();
        }
    });

    requestAnimFrame(draw);
}

function makeBullet() {
    var bullet;

    if (spareBullets.length > 0) {

        bullet = spareBullets.pop();
        bullet.reset(ship.pos.x, ship.pos.y, ship.angle);

    } else {

        bullet = new Bullet(ship.pos.x, ship.pos.y, ship.angle);
        bullets.push(bullet);

    }

    bullet.vel.plusEq(ship.vel);
}

function givePointerType(event) {
    switch (event.pointerType) {
        case event.POINTER_TYPE_MOUSE:
            return "MOUSE";
            break;
        case event.POINTER_TYPE_PEN:
            return "PEN";
            break;
        case event.POINTER_TYPE_TOUCH:
            return "TOUCH";
            break;
    }
}

function onPointerDown(e) {
    var newPointer = { identifier: e.pointerId, x: e.clientX, y: e.clientY, type: givePointerType(e) };
    if ((leftPointerID < 0) && (e.clientX < halfWidth)) {
        leftPointerID = e.pointerId;
        leftPointerStartPos.reset(e.clientX, e.clientY);
        leftPointerPos.copyFrom(leftPointerStartPos);
        leftVector.reset(0, 0);
    }
    else {
        makeBullet();

    }
    pointers.add(e.pointerId, newPointer);
}

function onPointerMove(e) {
    if (leftPointerID == e.pointerId) {
        leftPointerPos.reset(e.clientX, e.clientY);
        leftVector.copyFrom(leftPointerPos);
        leftVector.minusEq(leftPointerStartPos);
    }
    else {
        if (pointers.item(e.pointerId)) {
            pointers.item(e.pointerId).x = e.clientX;
            pointers.item(e.pointerId).y = e.clientY;
        }
    }
}

function onPointerUp(e) {
    if (leftPointerID == e.pointerId) {
        leftPointerID = -1;
        leftVector.reset(0, 0);

    }
    leftVector.reset(0, 0);

    pointers.remove(e.pointerId);
}

function setupCanvas() {
    canvas = document.getElementById('canvasSurfaceGame');
    c = canvas.getContext('2d');
    resetCanvas();
    c.strokeStyle = "#ffffff";
    c.lineWidth = 2;
}

The code is again very straightforward, I won’t spend time explaining it. You can view the complete source code here: http://david.blob.core.windows.net/html5/touchjoystick/TouchControl.html

In conclusion, thanks to the job done by Seb Lee-Delisle and David Catuhe, you now have all the pieces needed to implement your own virtual touch joypad for your HTML5 games. The result will work on all touch devices supporting HTML5!

David

Unifying touch and mouse: how Pointer Events will make cross-browsers touch support easy

2013-02-20T11:14:39Z

I often get questions from developers like, “with so many touch-enabled devices on phones and tablets, where do I start?” and “what is the easiest way to build for touch-input?” Short answer: “It’s complex.” Surely there’s a more unified way to handle multi-touch input on the web – in modern, touch-enabled browsers or as a fallback for older browsers. In this article I’d like to show you some browser experiments using MSPointers – an emerging multi-touch technology and polyfills that make cross-browser support, well less complex. The kind of code you can also experiment with and easily use on your own site.

First of all, many touch technologies are evolving on the web – for browser support you need to look the iOS touch event model and the W3C mouse event model in additional to at MSPointers, to support all browsers. Yet there is growing support (and willingness) to standardize. In September, Microsoft submitted MSPointers to the W3C for standardization and yesterday, we’ve already reached the Last Call Draft: http://www.w3.org/TR/pointerevents. The MS Open Tech team also recently released an initial Pointer Events prototype for Webkit.

The reason why I experiment with MSPointers is not based on device share – it’s because Microsoft’s approach to basic input handling is quite different than what’s currently available on the web and it deserves a look for what it could become. The difference is that developers can write to a more abstract form of input, called a “Pointer.” A Pointer can be any point of contact on the screen made by a mouse cursor, pen, finger, or multiple fingers. So you don’t waste time coding for every type of input separately.

The concepts

We will begin by reviewing apps running inside Internet Explorer 10 which exposes the MSPointer events API and then solutions to support all browsers. After that, we will see how you can take advantage of IE10 gestures services that will help you handling touch in your JavaScript code in an easy way. As Windows 8 and Windows Phone 8 share the same browser engine, the code & concepts are identical for both platforms. Moreover, everything you’ll learn on touch in this article will help you do the very same tasks in Windows Store apps built with HTML5/JS, as this is again the same engine that is being used.

The idea behind the MSPointer is to let you addressing mouse, pen & touch devices via a single code base using a pattern that match the classical mouse events you already know. Indeed, mouse, pen & touch have some properties in common: you can move a pointer with them and you can click on element with them for instance. Let’s then address these scenarios via the very same piece of code! Pointers will aggregate those common properties and expose them in a similar way than the mouse events.

The most obvious common events are then: MSPointerDown , MSPointerMove & MSPointerUp which directly map to the mouse events equivalent. You will have the X & Y coordinates of the screen as an output.

You have also specific events like: MSPointerOver , MSPointerOut , MSPointerHover or MSPointerCancel

But of course, there could be also some cases where you want to address touch in a different manner than the default mouse behavior to provide a different UX. Moreover, thanks to the multi-touch screens, you can easily let the user rotate, zoom or pan some elements. Pens/stylus can even provide you some pressure information a mouse can’t. The Pointer Events will still aggregate those differences and will let you build some custom code for each devices’ specifics.

Note: it would be better to test the following embedded samples if you have a touch screen of course on a Windows 8/RT device or if you’re using a Windows Phone 8. Still, you can have some options:

1. Get a first level of experience by using the Windows 8 Simulator that ships with the free Visual Studio 2012 Express development tools. For more details on how this works, please read this article: Using the Windows 8 Simulator & VS 2012 to debug the IE10 touch events & your responsive design .
2. Have a look to this video also available in other formats at the end of the article. The video demonstrates all the below samples on a Windows 8 tablet supporting touch, pen & mouse.
3. Use a virtual cross-browsing testing service like BrowserStack to test interactively if you don’t have access to the Windows 8 device. You can use BrowserStack for free for 3 months, courtesy of the Internet Explorer team on modern.IE.

Handling simple touch events

Step 1: do nothing in JS but add a line of CSS

Let’s start with the basics. You can easily take any of your existing JavaScript code that handles mouse events and it will just work as-is using some pens or touch devices in Internet Explorer 10. IE10 is indeed firing mouse events as a last resort if you’re not handling Pointers Events in your code. That’s why, you can “click” on a button or on any element of any webpage using your fingers even if the developer never thought that one day someone would have done it this way. So any code registering to mouvedown and/or mouseup events will work with no modification at all. But what about mouvemove?

Let’s review the default behavior to answer to that question. For instance, let’s take this piece of code:

<!DOCTYPE html>
<html>
<head>
    <title>Touch article sample 1</title>
</head>
<body>
    <canvas id="drawSurface" width="400px" height="400px" style="border: 1px dashed black;">
    </canvas>
    <script>
        var canvas = document.getElementById("drawSurface");
        var context = canvas.getContext("2d");
        context.fillStyle = "rgba(0, 0, 255, 0.5)";

        canvas.addEventListener("mousemove", paint, false);

        function paint(event) {
            context.fillRect(event.clientX, event.clientY, 10, 10);
        }
    </script>
</body>
</html>

It simply draws some blue 10px by 10px squares inside an HTML5 canvas element by tracking the movements of the mouse. To test it, move your mouse inside the box. If you have a touch screen, try to interact with the canvas to check by yourself the current behavior:

Sample 0 : default behavior if you do nothing

Result : only MouseDown/Up/Click works with touch

You’ll see than when you’re moving the mouse inside the canvas element, it will draw some series of blue squares. But using touch instead, it will only draw a unique square at the exact position where you will tap the canvas element. As soon as you will try to move your finger in the canvas element, the browser will try to pan inside the page as it’s the default behavior being defined.

You then need to specify you’d like to override the default behavior of the browser and tell it to redirect the touch events to your JavaScript code rather than trying to interpret it. For that, target the elements of your page that shouldn’t react anymore to the default behavior and apply this CSS rule to them:

-ms-touch-action: auto | none | manipulation | double-tap-zoom | inherit;

You have various values available based on what you’d like to filter or not. You’ll find the values described in this article: Guidelines for Building Touch-friendly Sites

The classic use case is when you have a map control in your page. You want to let the user pan & zoom inside the map area but keep the default behavior for the rest of the page. In this case, you will apply this CSS rule only on the HTML container exposing the map.

In our case, add this block of CSS:

<style>
    #drawSurface
    {
        -ms-touch-action: none; /* Disable touch behaviors, like pan and zoom */
    }
</style>

Which now generates this result:

Sample 1: just after adding -ms-touch-action: none

Result: default browser panning disabled and MouseMove works but with 1 finger only

Now, when you’re moving your finger inside the canvas element, it behaves like a mouse pointer. That’s cool! But you will quickly ask yourself this question: why does this code only track 1 finger? Well, this is because we’re just falling in the last thing IE10 is doing to provide a very basic touch experience: mapping one of your fingers to simulate a mouse. And as far as I know, we’re using only 1 mouse at a time. So 1 mouse === 1 finger max using this approach. Then, how to handle multi-touch events?

Step 2: use MSPointer Events instead of mouse events

Take any of your existing code and replace your registration to “mousedown/up/move” by “MSPointerDown/Up/Move” and your code will directly support a multi-touch experience inside IE10!

For instance, in the previous sample, change this line of code:

canvas.addEventListener("mousemove", paint, false);

to this one:

canvas.addEventListener("MSPointerMove", paint, false);

And you will get this result:

Sample 2: using MSPointerMove instead of mousemove

Result: multi-touch works

You can now draw as many squares series as touch points your screen supports! Even better, the same code works for touch, mouse & pen. This means for instance that you can use your mouse to draw some lines at the same time you are using your fingers to draw other lines.

If you’d like to change the behavior of your code based on the type of input, you can test that through the pointerType property value. For instance, let’s imagine that we want to draw some 10px by 10px red squares for fingers, 5px by 5px green squares for pen and 2px by 2px blue squares for mouse. You need to replace the previous handler (the paint function) with this one:

function paint(event) {
    if (event.pointerType) {
        switch (event.pointerType) {
            case event.MSPOINTER_TYPE_TOUCH:
                // A touchscreen was used
                // Drawing in red with a square of 10
                context.fillStyle = "rgba(255, 0, 0, 0.5)";
                squaresize = 10;
                break;
            case event.MSPOINTER_TYPE_PEN:
                // A pen was used
                // Drawing in green with a square of 5
                context.fillStyle = "rgba(0, 255, 0, 0.5)";
                squaresize = 5;
                break;
            case event.MSPOINTER_TYPE_MOUSE:
                // A mouse was used
                // Drawing in blue with a squre of 2
                context.fillStyle = "rgba(0, 0, 255, 0.5)";
                squaresize = 2;
                break;
        }

        context.fillRect(event.clientX, event.clientY, squaresize, squaresize);
    }
}

And you can test the result here:

Sample 2b: testing pointerType to test touch/pen or mouse

Result: You can change the behavior for mouse/pen/touch but since 2a the code now works only in IE10+

If you’re lucky enough to have a device supporting the 3 types of inputs (like the Sony Duo 11, the Microsoft Surface Pro or the Samsung Tablet some of you had during BUILD2011), you will be able to see 3 kind of drawing based on the input type. Great, isn’t it?

Still, there is a problem with this code. It now handles all type of inputs properly in IE10 but doesn’t work at all for browsers that don’t support the MSPointer Events like IE9, Chrome, Firefox, Opera & Safari.

Step 3: do feature detection to provide a fallback code

As you’re probably already aware, the best approach to handle multi-browsers support is to do features detection. In our case, you need to test this:

window.navigator.msPointerEnabled

Beware that this only tells you if the current browser supports MSPointer. It doesn’t tell you if touch is supported or not. To test support for touch or not, you need to check msMaxTouchPoints.

In conclusion, to have a code supporting MSPointer in IE10 and falling back properly to mouse events in other browsers, you need a code like that:

var canvas = document.getElementById("drawSurface");
var context = canvas.getContext("2d");
context.fillStyle = "rgba(0, 0, 255, 0.5)";
if (window.navigator.msPointerEnabled) {
    // Pointer events are supported.
    canvas.addEventListener("MSPointerMove", paint, false);
}
else {
    canvas.addEventListener("mousemove", paint, false);
}

function paint(event) {
    var squaresize = 2;
    if (event.pointerType) {
        switch (event.pointerType) {
            case event.MSPOINTER_TYPE_TOUCH:
                // A touchscreen was used
                // Drawing in red with a square of 10
                context.fillStyle = "rgba(255, 0, 0, 0.5)";
                squaresize = 10;
                break;
            case event.MSPOINTER_TYPE_PEN:
                // A pen was used
                // Drawing in green with a square of 5
                context.fillStyle = "rgba(0, 255, 0, 0.5)";
                squaresize = 5;
                break;
            case event.MSPOINTER_TYPE_MOUSE:
                // A mouse was used
                // Drawing in blue with a square of 2
                context.fillStyle = "rgba(0, 0, 255, 0.5)";
                squaresize = 2;
                break;
        }
    }

    context.fillRect(event.clientX, event.clientY, squaresize, squaresize);
}

And again you can test the result here:

Sample 3: feature detecting msPointerEnabled to provide a fallback

Result: full complete experience in IE10 and default mouse events in other browsers

Step 4: support all touch implementation

If you’d like to go even further and support all browsers & all touch implementations, you have 2 choices:

1 – Write the code to address both events models in parallel like described for instance in this article: Handling Multi-touch and Mouse Input in All Browsers
2 – Just add a reference to HandJS, the awesome JavaScript polyfill library written by my friend David Catuhe like described in his article: HandJS a polyfill for supporting pointer events on every browser

David wrote a cool little library that lets you write code leveraging the Pointer Events specification. As I mentioned in the introduction of this article, Microsoft recently submitted the MSPointer Events specification to W3C for standardization. The W3C created a new Working Group and it has already published a last call working draft based on Microsoft’s proposal. The MS Open Tech team also recently released an initial Pointer Events prototype for Webkit that you might be interested in.

While the Pointer Events specification is not yet a standard, you can still already implement code that supports it leveraging David’s Polyfill and be ready for when Pointer Events will be a standard implemented in all modern browsers. With David’s library the events will be propagated to MSPointer on IE10, to Touch Events on Webkit based browsers and finally to mouse events as a last resort. It’s damn cool! Check out his article to discover and understand how it works. Note that this polyfill will also be very useful then to support older browser with elegant fallbacks to mouse events.

To have an idea on how to use this library, please have a look to this article: Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS which shows you how to write a virtual touch joystick using pointer events. Thanks to HandJS, it will work on IE10 on Windows 8/RT, Windows Phone 8, iPad/iPhone & Android devices with the very same code base!

Recognizing simple gestures

Now that we’ve seen how to handle multi-touch, let’s see how to recognize simple gestures like tapping or holding an element and then some more advanced gestures like translating or scaling an element.

Thankfully, IE10 provides a MSGesture object that’s going to help us. Note that this object is currently specific to IE10 and not part of the W3C submission. Combined with the MSCSSMatrix element (our equivalent to the WebKitCSSMatrix one), you’ll see that you can build very interesting multi-touch experiences in a very simple way. MSCSSMatrix is indeed representing a 4×4 homogeneous matrix that enables Document Object Model (DOM) scripting access to CSS 2-D and 3-D Transforms functionality. But before playing with that, let’s start with the basics.

The base concept is to first register to MSPointerDown. Then inside the handler taking care of MSPointerDown, you need to choose which pointers you’d like to send to the MSGesture object to let it detect a specific gesture. It will then trigger one of those events: MSGestureTap, MSGestureHold, MSGestureStart, MSGestureChange, MSGestureEnd, MSInertiaStart. The MSGesture object will then take all the pointers submitted as input parameters and will apply a gesture recognizer on top of them to provide some formatted data as output. The only thing you need to do is to choose/filter which pointers you’d like to be part of the gesture (based on their ID, coordinates on the screen, whatever…). The MSGesture object will do all the magic for you after that.

Sample 1: handling the hold gesture

We’re going to see how to hold an element (a simple DIV containing an image as a background). Once the element will be held, we will add some corners to indicate to the user he has currently selected this element. The corners will be generated by dynamically creating 4 div added on top of each corner of the image. Finally, some CSS tricks will use transformation and linear gradients in a smart way to obtain something like this:

The sequence will be the following one:

1 - register to MSPointerDown & MSPointerHold events on the HTML element you’re interested in
2 - create a MSGesture object that will target this very same HTML element
3 - inside the MSPointerDown handler, add to the MSGesture object the various PointerID you’d like to monitor (all of them or a subset of them based on what you’d like to achieve)
4 - inside the MSPointerHold event handler, check in the details if the user has just started the hold gesture (MSGESTURE_FLAG_BEGIN flag). If so, add the corners. If not, remove them.

This leads to the following code:

<!DOCTYPE html>
<html>
<head>
    <title>Touch article sample 5: simple gesture handler</title>
    <link rel="stylesheet" type="text/css" href="toucharticle.css" />
    <script src="Corners.js"></script>
</head>
<body>
    <div id="myGreatPicture" class="container" />
    <script>
        var myGreatPic = document.getElementById("myGreatPicture");
        // Creating a new MSGesture that will monitor the myGreatPic DOM Element
        var myGreatPicAssociatedGesture = new MSGesture();
        myGreatPicAssociatedGesture.target = myGreatPic;

        // You need to first register to MSPointerDown to be able to
        // have access to more complex Gesture events
        myGreatPic.addEventListener("MSPointerDown", pointerdown, false);
        myGreatPic.addEventListener("MSGestureHold", holded, false);

        // Once pointer down raised, we're sending all pointers to the MSGesture object
        function pointerdown(event) {
            myGreatPicAssociatedGesture.addPointer(event.pointerId);
        }

        // This event will be triggered by the MSGesture object
        // based on the pointers provided during the MSPointerDown event
        function holded(event) {
            // The gesture begins, we're adding the corners
            if (event.detail === event.MSGESTURE_FLAG_BEGIN) {
                Corners.append(myGreatPic);
            }
            else {
                // The user has released his finger, the gesture ends
                // We're removing the corners
                Corners.remove(myGreatPic);
            }
        }

        // To avoid having the equivalent of the contextual  
        // "right click" menu being displayed on the MSPointerUp event, 
        // we're preventing the default behavior
        myGreatPic.addEventListener("contextmenu", function (e) {
            e.preventDefault();    // Disables system menu
        }, false);
    </script>
</body>
</html>

And here is the result:

Try to just tap or mouse click the element, nothing occurs. Touch & maintain only 1 finger on the image or do a long mouse click on it, the corners appear. Release your finger, the corners disappear.

Touch & maintains 2 or more fingers on the image, nothing happens as the Hold gesture is only being triggered only if 1 unique finger is holding the element.

Note: the white border, the corners & the background image are set via CSS defined in toucharticle.css. Corners.js simply creates 4 DIVs (with the append function) and places them on top of the main element in each corner with the appropriate CSS classes.

Still, there is something I’m not happy with in the current result. Once you’re holding the picture, as soon as you’re slightly moving your finger, the MSGESTURE_FLAG_CANCEL flag is raised and caught by the handler which removes the corners. I would rather like to remove the corners only once the user releases his finger anywhere above the picture or as soon as he’s moving his finger out of the box delimited by the picture. To do that, we’re going to remove the corners only on the MSPointerUp or the MSPointerOut. This gives this code instead:

var myGreatPic = document.getElementById("myGreatPicture");
// Creating a new MSGesture that will monitor the myGreatPic DOM Element
var myGreatPicAssociatedGesture = new MSGesture();
myGreatPicAssociatedGesture.target = myGreatPic;

// You need to first register to MSPointerDown to be able to
// have access to more complex Gesture events
myGreatPic.addEventListener("MSPointerDown", pointerdown, false);
myGreatPic.addEventListener("MSGestureHold", holded, false);
myGreatPic.addEventListener("MSPointerUp", removecorners, false);
myGreatPic.addEventListener("MSPointerOut", removecorners, false);

// Once touched, we're sending all pointers to the MSGesture object
function pointerdown(event) {
    myGreatPicAssociatedGesture.addPointer(event.pointerId);
}

// This event will be triggered by the MSGesture object
// based on the pointers provided during the MSPointerDown event
function holded(event) {
    // The gesture begins, we're adding the corners
    if (event.detail === event.MSGESTURE_FLAG_BEGIN) {
        Corners.append(myGreatPic);
    }
}

// We're removing the corners on pointer Up or Out
function removecorners(event) {
    Corners.remove(myGreatPic);
}

// To avoid having the equivalent of the contextual  
// "right click" menu being displayed on the MSPointerUp event, 
// we're preventing the default behavior
myGreatPic.addEventListener("contextmenu", function (e) {
    e.preventDefault();    // Disables system menu
}, false);

which now provides the behavior I was looking for:

Sample 2: handling scale, translation & rotation

Finally, if you want to scale, translate or rotate an element, you simply need to write a very few lines of code. You need to first to register to the MSGestureChange event. This event will send you via the various attributes described in the MSGestureEvent object documentation like rotation, scale, translationX, translationY currently applied to your HTML element.

Even better, by default, the MSGesture object is providing an inertia algorithm for free. This means that you can take the HTML element and throw it on the screen using your fingers and the animation will be handled for you.

Lastly, to reflect these changes sent by the MSGesture, you need to move the element accordingly. The easiest way to do that is to apply some CSS Transformation mapping the rotation, scale, translation details matching your fingers gesture. For that, use the MSCSSMatrix element.

In conclusion, if you’d like to handle all this cool gestures to the previous samples, register to the event like that:

myGreatPic.addEventListener("MSGestureChange", manipulateElement, false);

And use the following handler:

function manipulateElement(e) {
    // Uncomment the following code if you want to disable the built-in inertia 
    // provided by dynamic gesture recognition
    // if (e.detail == e.MSGESTURE_FLAG_INERTIA)
    // return;

    // Get the latest CSS transform on the element
    var m = new MSCSSMatrix(e.target.currentStyle.transform); 
    e.target.style.transform = m
    .translate(e.offsetX, e.offsetY) // Move the transform origin under the center of the gesture
    .rotate(e.rotation * 180 / Math.PI) // Apply Rotation
    .scale(e.scale) // Apply Scale
    .translate(e.translationX, e.translationY) // Apply Translation
    .translate(-e.offsetX, -e.offsetY); // Move the transform origin back
}

which gives you this final sample:

Try to move and throw the image inside the black area with 1 or more fingers. Try also to scale or rotate the element with 2 or more fingers. The result is awesome and the code is very simple as all the complexity is being handled natively by IE10.

Video & direct link to all samples

If you don’t have a touch screen experience available for IE10 and you’re wondering how these samples works, have a look at this video where I’m describing all samples shared in this article on the Samsung BUILD2011 tablet:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

And you can also have a look to all of them here:

Associated resources:

- W3C Pointer Events Specification
- Handling Multi-touch and Mouse Input in All Browsers : the polyfill library that should help a lot of developers in the future
- Pointer and gesture events
- Go Beyond Pan, Zoom, and Tap Using Gesture Events
- IE Test Drive Browser Surface which has greatly inspired lot of the embedded demos
- Try some awesome games in IE10 with Touch:
- Contre Jour and read a very interesting Behind The Scenes article
- Atari Arcade Games and read also this very informative article: Building Atari with CreateJS which details the choice made to support Touch in all platforms.
- Recording of the BUILD session 3-140: Touchscreen and stylus and mouse, oh my!

Logically, with all the details shared in this article and associated links to other resources, you’re now ready to implement the MSPointer Events model in your websites & Windows Store applications. You have then the opportunity to easily enhance the experience of your users in Internet Explorer 10.

David

Coding4Fun: building a Tetris-like game using CSS Grid Layout & Blend 5

2013-02-12T17:32:27Z

I’d like to share with you a mysterious internal secret kept within Microsoft for now a long time. It’s the real story behind the concept of the CSS Grid Layout imagined by Microsoft for IE10 and Windows Store Apps. Most of you probably think that this specification was designed to help developers having a better layout engine for their websites and applications. But the original motivation was completely different. The very first aim was to be able to create a Tetris-like game in an easy way! But I’m sure you’re not convinced yet. That’s why, I’m going to prove it to you using Blend 5 as a companion. Ok, let’s go!

Pre-requisites: to follow this tutorial, you need first to:

1 – Download/buy & install Windows 8 RTM on your machine: http://msdn.microsoft.com/en-US/windows/apps/br229516.aspx
2 – Download & install the free edition of Visual Studio 2012 Express RTM for Windows 8: http://msdn.microsoft.com/en-US/windows/apps/br229516.aspx that includes Expression Blend 5 for Visual Studio or use the higher versions.

Step 1: discover the secret behind the CSS Grid Layout thanks to Blend 5

Launch Expression Blend 5 and create a new HTML (Windows Store) project of type Blank App. Name it “TheRealCSSGridStory”:

Replace:

<p>Content goes here</p>

By:

<div class="mainGrid">
</div>

Let’s create a grid containing 10 columns and 20 lines whatever the screen’s resolution by using fraction units. For that, add this CSS rule:

.mainGrid {
    display: -ms-grid;
    width: 100%;
    height: 100%;
    -ms-grid-columns: (1fr)[10];
    -ms-grid-rows: (1fr)[20];
}

Select in the Live DOM the <div> mainGrid element and you should obtain this:

Let’s draw a shape inside this beautiful grid. Add this block of HTML inside the main grid:

<div class="shape1">
</div>

And insert this CSS associated with it:

.shape1 {
    -ms-grid-column: 4;
    -ms-grid-row: 3;
    -ms-grid-column-span: 3;
    -ms-grid-row-span: 2;
    background-color: red;
}

You should now see that in Blend 5:

Cool, but not yet something that looks like to a Tetris gaming shape. Let’s work on that. Add these 2 DIV inside the shape1:

<div class="line1shape1"></div>
<div class="line2shape1"></div>

and replace the previous .shape1 rule by this block of CSS:

.shape1 {
    -ms-grid-column: 4;
    -ms-grid-row: 3;
    -ms-grid-column-span: 3;
    -ms-grid-row-span: 2;
    display: -ms-grid;
    -ms-grid-columns: 1fr 1fr 1fr;
    -ms-grid-rows: 1fr 1fr;
    width: 100%;
    height: 100%;
}

.line1shape1 {
    -ms-grid-column-span: 2;
    background-color: red;
    
}

.line2shape1 {
    -ms-grid-column: 2;
    -ms-grid-row: 2;
    -ms-grid-column-span: 2;
    background-color: red;
}

The shape1 is currently spanning on 3 columns and on 2 rows. I’m then creating a new grid inside this area defined by 3 columns and 2 rows in order to have cells having exactly the same size as the cells of the main grid, whatever the resolution will be!

Once done, I’m creating 2 lines in order to mimic the Z shape of the Tetris game. You should now have this result:

Even better, play with the various views available in the Device tab and you’ll see that our game is already implementing a responsive design! This is freaking cool, isn’t it?

Here is for instance the outputs for the snapped view and the portrait view:

Let’s now resolve another problem. The Tetris grid gaming grid is composed of squares. Our current responsive design is stretching 100% width. Building a Windows 8 application for the Windows Store will most of the time meet widescreen monitors (current tablets are 1366x768 or 1920x1080 and most desktop PC have a 16/9 ratio). Let’s then assume that targeting a widescreen ratio is addressing almost all cases. To compute the proper responsive width, you need to do: 9/16 * 10/20 (the ratio of the main gaming grid) which equals to: 28.125%.

Add then this rule to target the main grid in full screen landscape mode:

@media screen and (-ms-view-state: fullscreen-landscape) {
    .mainGrid {
        width: 28.125%;
        }
}

Let’s now center the gaming grid by using… the CSS Grid Layout again! (And you should now start to believe it really was really designed for Tetris!).

Switch the body element to –ms-grid made of 1 column & 1 row:

body {
    display: -ms-grid;
    -ms-grid-columns: 1fr;
    -ms-grid-rows: 1fr;
    width: 100%;
    height: 100%;
}

Now simply add this attribute to the CSS associated to the the main grid:

-ms-grid-column-align: center;

And the grid is now centered:

At this stage, you’re probably already shocked. “How could I have missed this incredible secret?” you’re wondering to yourself. Take a breath. Now that you now THE secret, let’s continue this tutorial together to discover other awesome possibilities delivered by the CSS specifications combined together.

Step 2: moving or rotating a shape

My first idea was to try avoiding JS by using as much CSS as possible. I then first tried to use CSS3 Animations to move & animate the shape on the various rows/columns. But the bad news is that you can’t change the –ms-grid-column or –ms-grid-row values via CSS3 animations. This will then be the job of some JavaScript code.

I then started to think about how I will rotate the shape. CSS Transforms seemed to be perfectly adapted for that. Let’s check that by doing some experiments. Blend 5 is really cool for that as you can directly see in live the outcome of your changes.

Add a rotation of 90 degrees on shape1 by adding this class to its DIV element:

.shape1rotated {
    transform: rotate(90deg);
}

I’m sure you weren’t expecting this:

Problem: it’s not properly aligned to the gaming grid. To align our shape to the grid, we need some small adjustments:

.shape1rotated {
    transform-origin: 33% 50%;
    transform: rotate(90deg) translateX(-33%);
}

And we now have the same rotation as a Tetris-like game. Here are 2 screenshots before/after the rotation:

We can even go further than that by using a transition set on shape1 via this:

transition: all 1s ease-out;

And now, removing/adding the .shape1rotated class on the shape1 DIV will trigger a smooth rotation animation!

Check out the result inside Blend 5 thanks to this short video:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

At this stage, we could think that this approach is the good one to build our game. But this is unfortunately not yet the case. Here’s why. Try moving the shape by simply changing its –ms-grid-column property. Blend 5 will reflect the changes directly. When not rotated, the shape can be moved up to the 8th row:

So far so good. But when you’re rotating it by adding the .shape1rotated class to the DIV:

You see that there is still 1 row available on the right for the shape’s move. If you think we simply need to move it to the 9th row, you’re wrong! Indeed, here is what we’ll obtain on the 9th row:

You probably forgot that we’re currently moving a DIV element displaying a grid layout of 3 columns and 2 rows matching exactly the underlying gaming grid. When moving it, we really have the feeling that this is a block part of the main grid we’re moving. But for this tricks to work, we need at least 3 columns available to contain our shape element. If it’s contained in 2 columns (when set to the 9th column) or less, it will be “compressed” like in the screenshot.

There is 2 ways to resolve that.

1 – Stop using CSS Transforms and draw the rotated shape using another grid definition. For instance, using 3 div inside the shape instead of 2. But using such an approach will prevent us to have the nice CSS Animations in place.
2 – Redefined the main grid to work on 12 columns instead of 10 and we’ll use only the columns from 2 to 11 (a sort of clipping area if you wish). This will resolve our “overflow” problem.

Let’s implement the 2nd solution.

Redefine the main grid using that:

.mainGrid {
    display: -ms-grid;
    -ms-grid-columns: (1fr)[12];
    -ms-grid-rows: (1fr)[20];
    -ms-grid-column-align: center;    
    width: 100%;
    height: 100%;
}

To have also the proper ratio, you need to update the media query associated:

@media screen and (-ms-view-state: fullscreen-landscape) {
    .mainGrid {
        width: 33.75%;
        }
}

33.75% = 9/16 *12/20

Let’s also add a “virtual grid” to delimitate the space where we will be able to move the shapes. Inside the main grid DIV, insert this one:

<div class="virtualGrid">
</div>

Associated with this block of CSS:

.virtualGrid {
    -ms-grid-column: 2;
    -ms-grid-column-span: 10;
    -ms-grid-row-span: 20;
    border-right-style: dashed;
    border-left-style: dashed;
    background-color: #505A5A;
}

It will help to delimitate the gaming area with a grey background and some dashed border lines.

Now, if I’m moving the Z shape on column 9, row 2, here is the result:

If I’m rotating it with CSS Transforms, I can move it correctly on column 10:

Bonus – handling the Portrait mode:

If you’d like to support the portrait mode (which is even better for the Tetris grid game), add this CSS Media Query definition:

@media screen and (-ms-view-state: fullscreen-portrait) {
        .mainGrid {
        width: 106.66%;
        }
}

As the ratio need to be computed as = 16/9 * 12/20 = 106,66%.

Step 3: adding some code to handle part of the game logic

Now that we’ve solved the graphics part of the game using only some pure CSS & HTML code, we need the help of JavaScript to move/rotate the shape in the gaming area. We’re going to re-implement the CSS logic via a JS object that’s going to be defined thanks to WinJS.Class.

Open the “TheRealCSSGridStory” in Visual Studio 2012.

Create a TetrisShapeZ.js file in the JS directory and copy/paste this code:

(function () {
    "use strict";

    var ShapeZ = WinJS.Class.define(
    /// Constructor - columnIndex is optional. If provided, will define in which column the shape will start
        function (columnIndex) {
            // We're creating the equivalent of this HTML block :
            // <div class="shape1 ">
            //     <div class="line1shape1"></div>
            //     <div class="line2shape1"></div>
            // </div>
            this._shape1 = document.createElement("div");
            var line1 = document.createElement("div");
            var line2 = document.createElement("div");
            this._shape1.className = "shape1";
            line1.className = "line1shape1";
            line2.className = "line2shape1";
            this._shape1.appendChild(line1);
            this._shape1.appendChild(line2);
            // Boolean to indicate if the shape is in its default orientation mode or not
            // True means not rotated, false means rotated
            this._defaultOrientation = true;
            // Setting the column position in the main grid
            if (columnIndex) {
                this._currentColPos = columnIndex;
                this._shape1.style.msGridColumn = this._currentColPos;
            }
            else {
                this._currentColPos = 1;
            }
            // We always start at line 1
            this._currentLinePos = 1;
            // Boolean to know if the shape can be move/rotate or not
            // If true, this means we've reached the last line possible
            this._fixed = false;
        },
        {
            /// Specify in which HTML element displayed in CSS Grid you'd like to work with
            /// width is the number of columns of the grid & height is the number of lines
            insertIntoGrid: function (element, width, height) {
                element.appendChild(this._shape1);
                this._gridWidth = width;
                this._gridHeight = height;
                // These are the left & bottom max limit for this shape
                // when displayed in default orientation mode
                this._maxLeft = width - 3;
                this._maxBottom = height - 1;
            },
            /// Rotate the Z shape 90 degrees anti/clockwise using CSS Transforms
            /// by simply removing/adding the shape1rotated class
            rotate: function () {
                if (!this._fixed && this._defaultOrientation) {
                    // rotating 90 degrees clockwise, it will trigger also the CSS Transition
                    WinJS.Utilities.addClass(this._shape1, "shape1rotated");
                    this._defaultOrientation = false;
                    // the left limit is now +1 vs the default orientation
                    this._maxLeft = this._gridWidth - 2;
                }
                else {
                    if (!this._fixed && this._currentColPos < this._maxLeft) {
                        // removing the shape1rotated will automatically reset the shape
                        // to its initial matrix and again the CSS Transition will do the animation for you
                        WinJS.Utilities.removeClass(this._shape1, "shape1rotated");
                        this._defaultOrientation = true;
                        this._maxLeft = this._gridWidth - 3;
                    }
                }
            },
            // Internal function called by public moveLeft/moveRight
            _moveHorizontally: function (direction) {
                if (!this._fixed && (this._currentColPos < this._maxLeft || direction === -1) && (this._currentColPos > 2 || direction === 1)) {
                    this._currentColPos = this._currentColPos + direction;
                    this._shape1.style.msGridColumn = this._currentColPos;
                }
            },
            /// Move the shape on the immediate left column
            /// Test if you've reached the left limit or not
            moveLeft: function () {
                this._moveHorizontally(-1);
            },
            /// Move the shape on the immediate right column
            /// Test if you've reached the right limit or not
            moveRight: function () {
                this._moveHorizontally(1);
            },
            /// Move the shape down on the immediate below line
            /// Test if you've reached the bottom limit or not
            moveDown: function () {
                if (!this._fixed) {
                    this._currentLinePos = this._currentLinePos + 1;
                    this._shape1.style.msGridRow = this._currentLinePos;
                    if (this._currentLinePos === this._maxBottom) {
                        this._fixed = true;
                    }
                }
            }
        }
    );

    WinJS.Namespace.define("CSSTetris", { ShapeZ: ShapeZ });
} ());

Simply read the code to understand what it’s doing. It should be commented enough to be self-explicit.

Add a reference to this script file in default.html and keep only this block of HTML inside the body:

<div class="mainGrid">
    <div class="virtualGrid">
    </div>
</div>

Jump into default.js.

The cool part of having well documented the code is that we now have interesting IntelliSense details like for the constructor:

or the rotate function:

To properly use this code, add this block of JS just after processAll call:

document.addEventListener("keydown", OnKeyDown, false);
mainGrid = document.getElementsByClassName("mainGrid")[0];
myShape = new CSSTetris.ShapeZ(4);
myShape.insertIntoGrid(mainGrid, 12, 20);
init();

And add these 2 functions:

function init() {
    setInterval(function () {
        myShape.moveDown();
    }, 1000);
}

function OnKeyDown(event) {
    switch (event.keyCode) {
        case KEYCODE_X:
            myShape.rotate();
            break;
        case KEYCODE_LEFT:
            myShape.moveLeft();
            break;
        case KEYCODE_RIGHT:
            myShape.moveRight();
            break;
        case KEYCODE_DOWN:
            myShape.moveDown();
            break;
    }
}

And we’re done! We now have a very basic game using CSS Grid Layout coupled with CSS Transforms & Animations for the graphics part and a couple of JS lines of code to have the beginning of the basics of a Tetris-like game.

Here is a short video demonstrating the final result:

Download Video: MP4, WebM, HTML5 Video Player by VideoJS

You can download the final Visual Studio solution corresponding to the 3 steps of this tutorial here: http://david.blob.core.windows.net/win8/TheRealCSSGridStory.zip

So, are you now convinced that CSS Grid Layout was made to simplify the creation of Tetris-like games?

David

Tout savoir sur le parcours Windows 8/Windows Phone 8 des Techdays 2013 !

2012-12-19T14:25:08Z

Cette année, comme tous les ans depuis 5 ans maintenant, j’étais responsable du parcours Web des Techdays 2013. Mais j’avais également la chance d’être responsable du parcours Windows 8/Windows Phone 8 avec mes amis Eric Vernié, Pierre Cauchois et Alex Danvy. Nous avons essayé de créer un parcours qui couvre les sujets qui nous paraissent les plus importants pour notre plateforme mais contenant aussi les sujets que nous savons qui vous tiennent à cœur. Par ailleurs, nous avons essayé d’établir un équilibre sur les langages (XAML/C# et HTML5/JS) et sur les 2 plateformes (Win8 et WP8). Pour finir, il y aura des sessions de découvertes (niveau 100 et 200) et des sessions dites qui « piquent » (niveau 300 et 400). Bref, il devrait y en avoir pour tous les gouts !

En synthèse, nous avons ainsi réfléchit à 3 types de sessions sur le parcours 8 :

- Des sessions d’introduction aux plateformes
- Des sessions communes entre Windows 8/Windows Phone 8 vu que les 2 plateformes partagent un cœur commun ainsi que de nombreux concepts
- Des sessions spécifiques à Windows 8 et Windows Phone 8 contenant du contenu plus avancé

Je suis assez content du résultat et des orateurs que nous avons retenus. Je vous propose ainsi de découvrir le programme complet qui vous attend du 12 au 14 février 2013 au palais des congres. Réservez vos agendas !

Note : le APPxxx entre parenthèse de chaque titre vous donne le niveau de difficulté de 100 (accessible à tous) vers 400 (accrochez-vous au siège). On considère en général qu’une session développeur commence à 200.

Sessions d’introduction aux plateformes

1. Introduction au développement Windows 8 et Windows Phone 8 (APP101) qui sera animée par Eric Vernié et Alex Danvy. La session chapeau de toutes les autres qui faudra donc absolument ne pas manquer !
2. Introduction au développement Windows 8 avec XAML et .NET 4.5 (APP201) qui sera animée par Eric Vernié et Benjamin Baldacci
3. Introduction au développement Windows 8 ModernUI avec HTML5 et JavaScript (APP203) qui sera animée par votre humble serviteur
4. Introduction au développement Windows Phone 8 (APP212) qui sera animée par Jean-Sébastien Dupuy, le dernier petit nouveau de notre équipe, et par Julien Corioland
5. Quels sont les outils pour créer des jeux pour Windows 8 et Windows Phone 8 (APP204) qui sera animée par David Catuhe et moi-même. Que vous soyez un passionné, un étudiant ou un professionnel du jeu vidéo, nous vous dresserons un panorama complet des manières de développer un jeu pour Windows 8 et Windows Phone 8.

Ces sessions sont programmées sur le 1er jour (le 12 février) sauf celle sur les jeux vidéo qui aura lieu le 2eme jour (13 février). Il est indispensable de les suivre pour avoir les bases nécessaires à la compréhension des sessions qui suivent.

Sessions communes aux 2 plateformes

6. WinRT et l'accès aux données (APP202) qui sera animée par Sébastien Pertus et Benoit Laut
7. Fichier, Nouveau projet… Et après ? (APP213) qui sera animée par John Thiriet
8. Cycle de vie d'une application Windows 8 et Windows Phone 8 (APP305) qui sera animée par Christopher Maneu
9. Bien soumettre et monétiser votre application Windows 8 et Windows Phone 8 (APP208) qui sera animée par Pierre Lagarde
10. Bonnes pratiques de l'intégration de la publicité dans les applications Windows 8 et WP8 (APP209) qui sera animée par Nathalie Belval, Cyril Cathala, Pierre Lagarde et Marc Couraud
11. Plus près, plus près… NFC et la communication à proximité dans Windows 8 et Windows Phone 8 (APP220) qui sera animée par notre superbe Julie Knibbe et par Marc Gemeto
12. Créer des applications métier (LOB) pour Windows 8 et Windows Phone 8 (APP206) qui sera animée par Audrey Petit et Patrice Lamarche
13. Création de jeux en DirectX pour Windows 8 et Windows Phone 8 (APP401) qui sera animée par Simon Ferquel et David Catuhe
14. Conception applications 3 écrans sur Windows 8/WP8/Xbox 360 (APP210) qui sera animée par Romuald Boulanger, Thierry Bouquain et Pierre Lagarde

Sessions spécifiques Windows 8

15. Windows 8 : Comment développer des applications efficaces et performantes pour rendre l'expérience utilisateur unique ? (APP211) : 2 retours d’expériences intéressants qui seront présentés par Benoit Douvernelle et Guillaume Brout
16. Conception d'une application pour Windows 8 avec Blend 5 (APP207) qui sera animée par Eric Ambrosi
17. Outillage pour Windows 8 XAML (APP217) qui sera animée par Sébastien Pertus et Cyprien Autexier
18. Trucs et astuces pour rendre votre application Windows 8 plus visible (APP218) qui sera animée par Sacha Leroux et Alex Danvy
19. 10 tips pour améliorer les performances de vos applications Windows 8 (APP301) qui sera animée par Loic Rebours et Eric Vernié
20. Le développement d'un Composant WinRT (APP302) qui sera animée par Arnaud Auroux et Eric Vernié
21. Deep Dive WinJS – Profitez à 100% de son potentiel (APP304) qui sera animée par Thomas Lebrun et Jonathan Antoine
22. Optimisation des applications Windows 8/HTML5/WinJS (APP402) qui sera animée par Thomas Lebrun et Jonathan Antoine

Sessions spécifiques Windows Phone 8

23. L'histoire d'HTML5 pour les développeurs Windows Phone 8 (APP215) qui sera animée par David Catuhe et moi-même
24. Outillage pour Windows Phone 8 (APP219) qui sera animée par Fabien Décret et Stéphane Sibué
25. Géolocalisation, Speech, Intégration : Tirez parti des nouveautés de Windows Phone 8 (APP221) qui sera animée par Julie Knibbe, Samuel Blanchard et Jean-Sébastien Dupuy
26. Trucs et astuces pour rendre votre application Windows Phone 8 plus visible (APP222) qui sera animée par Rudy Huyn et Alex Danvy
27. Booster votre application Windows Phone 8 (APP303) par Jérémy Alles et Charlotte Chavancy

Bref, vous voyez qu’il y a de quoi faire en la matière : 27 sessions animées par 37 experts différents de chez Microsoft et de notre écosystème de partenaires! Sans compter le fait que le parcours Design, ergonomie et Interface naturelle géré par Michel Rousseau traitera également des applications Windows 8 et Windows Phone 8 et que certains concepts communs aux nouvelles plateformes 8 seront également abordés dans le parcours Developpement WEB (comme la gestion du tactile, les SPA et les nouveautés d’HTML5 et CSS3).

Avec tout cela, vous serez alors parfaitement armés pour concevoir vos applications à destination de ces nouvelles plateformes.

Je souhaite donc bon courage à tous les speakers que nous avons nommés pour qu’ils nous fassent des sessions d’excellente qualité. Et je vous souhaite à tous d’excellentes fêtes de fin d’année !

David