Eternal Coding - HTML5 / Windows / Kinect / 3D development

Benchmarking a HTML5 game: HTML5 Potatoes Gaming Bench

David Catuhe — Tue, 21 May 2013 10:48:39 GMT

A few weeks ago, my friend David Rousset wrote an excellent article about “benchmarking your sprites animations to target all devices & browsers”. This article used a benchmark framework called “HTML5 Potatoes Gaming Bench” to obtain a consistent scoring for various sprites tests.

I will try to explain you how we built this framework and the decision we made to make it representative of the effective performance of the tested hardware and browser.

You will see that this framework can be used to measure any scenarios you want to benchmark for your own games.

The delta time

First of all, we have to talk about the delta time. Indeed, when you wrote games or any resources intensive applications, the principle is always the same: you must have a render loop where the display is done (sprites, effects, etc.).

The user will have a feeling of great performance when the elapsed time between two rendered frames is near the well known target of (1000 / 60)ms (about 16ms).

So targeting 60 frames per second will be the duty of any game developer: No one is prone to experience lag or slowness in the responsiveness of their game.

Actually, the delta time can be defined as the elapsed time between two passes in the render loop.

To compute this value, you just have to store the current date() when you start a new frame and subtract the previous stored date(). The result is the number of milliseconds between the two frames:

var previousDate;

var computeDeltaTime = function() {
    if (!previousDate) {
        previousDate = Date.now();
        return 0;
    }

    var currentDate = Date.now();
    var delta = currentDate - previousDate;
    previousDate = currentDate;

    return delta;
};

Calling this function just before rendering a new frame will allow us to determine the current delta time:

var renderLoop = function() {
    while (renderInProgress) {
        var deltaTime = computeDeltaTime();
        
        // render your frame here
    }
};

Render loop

Actually, the render loop cannot be a while(true){} function. Indeed, JavaScript is mono-threaded so looping in a function will just block entirely your browser.

So you must be called by the browser every time a frame is requested. To do so, the W3C introduced a new API called requestAnimationFrame.

This API tells the browser that you wish to perform an animation and requests that the browser schedule a repaint of the window for the next animation frame. It also takes in account the visibility of the page in order to avoid unnecessary renders.

requestAnimationFrame is far better than a simple setTimeout API because setTimeout API can provide choppy animations and increase resource consumption by rendering not required frames.

This is the PERFECT place to render your new frame. And ovbiously this is the perfect place to evalute your delta time. (It is worth noting that requestAnimationFrame gives you a parameter indicating the time at which the function is scheduled to be called).

So the new render loop looks like this:

var renderLoop = function () {
    var deltaTime = computeDeltaTime();

    // render your frame here
    
    // Schedule new frame
    requestAnimationFrame(renderLoop);
};

// First call
requestAnimationFrame(renderLoop);

Alas, the requestAnimationFrame API is not supported by every browser according to www.caniuse.com:

To prevent the benchmark for not working on every browser, we can use this small polyfill:

POTATOES.Tools.queueNewFrame = function (func) {
    if (window.requestAnimationFrame)
        window.requestAnimationFrame(func);
    else if (window.msRequestAnimationFrame)
        window.msRequestAnimationFrame(func);
    else if (window.webkitRequestAnimationFrame)
        window.webkitRequestAnimationFrame(func);
    else if (window.mozRequestAnimationFrame)
        window.mozRequestAnimationFrame(func);
    else if (window.oRequestAnimationFrame)
        window.oRequestAnimationFrame(func);
    else {
        window.setTimeout(func, 16);
    }
};

You can note that the setTimeout API is used as a fallback for the requestAnimationFrame API.

Measuring FPS

The benchmark need a value called FPS (standing for frames per second). Computing the FPS is an easy task when you have requestAnimationFrame and the delta time.

To do so, HTML5 Potatoes Gaming Bench uses a function called handleMetrics. This function is responsible for computing the immediate FPS and the average FPS. The first one is just the FPS based on the delta time between the current frame and the previous one. The second one is the average of FPS for a given number of previous frames:

var fpsFrame = 20; // fps window frame
var fpsCap = 60;
var previousFramesDuration = [];

var handleMetrics = function () {
    previousFramesDuration.push(Date.now());

    if (previousFramesDuration.length >= fpsFrame) {

        if (previousFramesDuration.length > fpsFrame) {
            previousFramesDuration.splice(0, 1);
        }

        var avg = 0;
        for (var id = 0; id < fpsFrame - 1; id++) {
            avg += previousFramesDuration[id + 1] - previousFramesDuration[id];
        }
        avg /= fpsFrame - 1;

        POTATOES.GamingBench.currentFPS = Math.min(fpsCap, 1000.0 / (
                      previousFramesDuration[fpsFrame - 1] - previousFramesDuration[fpsFrame - 2]));
        POTATOES.GamingBench.averageFPS = Math.min(fpsCap, 1000.0 / avg);
    }
    POTATOES.Tools.queueNewFrame(handleMetrics);
};

To be sure the results are correct, I installed the Windows Performance Toolkit in order to compare the measures.

For my tests I used these benches: http://www.html5potatoes.com/gamingbench/index.htm

Two tests are used:

Canvas pixels manipulations
Processing images with web workers

The Windows Performance Analyser gave the following results:

We can see that the first test runs between 25 and 30 frames per second. The second one runs almost at full speed.

The good news is that the results computed by the Gaming Bench are similar:

The second test is not at more than 60fps because we use requestAnimationFrame which prevents unnecessary renders.

So we can consider that the measured values are consistent.

Using HTML5 Potatoes Gaming Bench

HTML5 Potatoes Gaming Bench is an open and free Framework that you can use freely

You can download all the code used previously here: http://www.html5potatoes.com/gamingbench/gamingbench.zip

The Gaming Bench was designed to provide an infrastructure to test whatever you want on a browser. So to add your own test on the gaming bench, you just have to create a Bench object with this code:

var tunnelBench = new POTATOES.GamingBench.Bench("Canvas pixels manipulations", "http://aka.ms/potatoes_tunnel",
    function (workbench) { // Init
        init(workbench, this.onInitCompleted);
    }, function () { // Run
        render = true;
        POTATOES.Tools.queueNewFrame(renderingLoop);
    }, function (workbench) { // End
        render = false;            
    });

POTATOES.GamingBench.registerBench(tunnelBench);

You have to provide 3 functions:

Init: This function is called once in order to create the DOM required by your test. It gives you the workbench parameter which is the DOM container where you can add your objects.
Run: This function is called to launch your test (this is where you can call the queueNewFrame function)
End: This function is called to stop your render and eventually clean associated ressources

The POTATOES.GamingBench.registerBench function is used to add your bench to the list of active benches.

To run the complete Gaming Bench, you have to use this code:

// Starting benchmark
POTATOES.GamingBench.start();

If you need to cancel it, just use this code:

POTATOES.GamingBench.cancel();

You can also skip the current test and go to the next one:

POTATOES.GamingBench.skipCurrent();

Finally if you want to gather results, you have to use this kind of code:

POTATOES.GamingBench.onprocessended = function () {
    // Generating the result list
    var score = 0;
    for (var index = 0; index < POTATOES.GamingBench.benches.length; index++) {
        var bench = POTATOES.GamingBench.benches[index];
        score += bench.score;

And that’s it! Feel free to take a look at the sample code to see how you can use the framework to extract all the infos you can need.

Understanding score

After running the complete Gaming Bench, you can analyse the score using a chart produced with d3.js. The score is computed by adding one point on every rendered frame. Obviously, you can modify this behavior for your own bench to take in account another metric. This is what David Rousset did for his own benches.

Here is a sample on how you can use details provided by the framework to compute a global fps:

var avgFps = 0;
for (var i = 0; i < bench.stats.length; i++) {
    avgFps += bench.stats[i].y;
}

avgFps /= bench.stats.length;

I really hope you will find this framework useful to help you create wonderful and efficient games!

Going further

Using Visual Studio’s Javascript Memory Analysis tool to find memory leaks on your Windows 8 Javascript app

David Catuhe — Wed, 24 Apr 2013 08:24:28 GMT

Javascript is a wonderful language for developing Windows 8 apps. As a untyped, dynamic language, Javascript is really flexible but sometimes if you are not careful you may leave some memory leaks in your code.

But do not worry, Visual Studio includes a great tool called Javascript Memory Analysis in order to help you find out where the leaks are.

Using the Javascript Memory Analysis tool

This tool is available in the Debug menu of your Visual Studio:

Launching the tool will launch your app and will show this screen:

The first part of the window is dedicated to the real-time monitoring of the memory used by the application:

Right after you can find a button (Take Heap Snapshot) used to capture the instantaneous state of the memory heap (The control also indicates the size of the heap and the number of active objects):

By clicking on the heap size, a more detailed view is generated:

This view presents a sorted list of the objects by retained size, which means the objects consuming the most memory that are potentially easiest to free would be listed first.

For instance, in my app, I can see that the UrzaGatherer object retains almost 60 MB (which is normal because it handles a huge list of cards):

The view can also display:

a list of types sorted by occurrence count
a list of root objects (from the garbage collector point of view)
a list of DOM objects sorted by retained size
a list of WinRT objects sorted by retained size

All of these views can potentially help you find inconsistent memory usage.

Comparing snapshots

Each time you click on Take Heap Snapshot button, the system generates a new dump of the heap for further analysis. Furthermore, each snapshot indicates the difference of the heap size and objects count against previous snapshot.

This is the key for finding memory leaks.

For instance, in my app, I will take a first snapshot, navigate to a given page, then go back to home page and finally take a second snapshot. Doing this I will be able to see if the given page produces memory leaks.

We can see here that between the two snapshots, the heap size has grown of 6.12 MB and more than 53k objects were generated and still alive.

By clicking on the heap size difference (+6.12 MB), you can view a detail of the heap size compared to previous snapshot:

The Dominators view indicates that new objects remain when we come back to the home page. You can use this view to identify unwanted objects.

In my case, the first line seems suspicious. By right-clicking on it, I can go to the Root view to see which function retains my objects:

We can see that the object is retained by a function called ImportFromTappedOut.

After looking for this function in my code, it turns out that this function is a global function which takes a function as parameter:

ImportFromTappedOut = function (deckName, custom) {

The custom parameter is used by this function to call some specific user code. The main problem is that because the ImportFromTappedOut is a global function it will retain all the code referenced by the function affected to the custom parameter and in my case I used an anonymous function that references a lot of local DOM objects:

 Tools.ImportFromTappedOut("test", function () {
     var select = document.querySelector("#addToDeck");
     var count = document.querySelector("#addToDeckCount").value;

     listView.selection.getItems().then(function (items) {
         appBar.hide();
         appBar.sticky = false;
         listView.selection.clear();
…code continue…

And this exactly the problem described by the tool:

The solution is easy: I just have to call importFromTappedOut with a null function when I leave the page in order to allow the garbage collector to collect all retained objects (1.25 MB here).

I clearly encourage you to use this tool to detect and eradicate memory leaks. It is simple and really powerful and can help you get rid of memory problems.

More about the tool

Others articles you may find useful:

Now with Javascript: Reading PDF and XPS on your Windows 8 application using WinRT

David Catuhe — Mon, 22 Apr 2013 07:10:54 GMT

Following my previous article on “Reading PDF and XPS on your Windows 8 application using WinRT”, I would like to share with you the same example but with Javascript code.

Many of you sent me a mail to get this code, so feel free to grab it here (I would like to thank Mehdi Lahlou for his original version):

http://www.catuhe.com/msdn/js.PDFReader.zip

The principle is the same as for C#: I use a FlipView control that uses a custom template to generate.

The FlipView is nothing but a standard WinJS control:

 <div data-win-control="WinJS.UI.FlipView" id="flipView">
 </div>

To open the PDF file, we can use the following code (standard WinRT code):

openPDF: function(filename) {
    return Windows.ApplicationModel.Package.current.installedLocation.getFileAsync(filename).then(function(file) {
        return Windows.Storage.FileIO.readBufferAsync(file).then(function(fileBuffer) {
            return MuPDFWinRT.Document.create(fileBuffer, MuPDFWinRT.DocumentType.pdf, 
                               Windows.Graphics.Display.DisplayProperties.logicalDpi);
        });
    });
},

Please note that the MuPDFWinRT component used here is strictly the same as the one used by C#

Once the PDF is opened we can configure the FlipView using this code:

this.openPDF("test.pdf").then(function (pdfDocument) {
    var array = [];
    for (var i = 0; i < pdfDocument.pageCount; i++) {
        array.push(i);
    }
    var dataList = new WinJS.Binding.List(array);
    var flipViewDOM = document.getElementById("flipView");
    var flipView = flipViewDOM.winControl;
    var width = flipViewDOM.clientWidth;
    var height = flipViewDOM.clientHeight;
    
    flipView.itemDataSource = dataList.dataSource;
    flipView.itemTemplate = (function (itemPromise) {
        return itemPromise.then(function (item) {
            // root element for the item
            var canvas = document.createElement("canvas");
            
            canvas.style.width = width;
            canvas.style.height = height;

            var canvasContext = canvas.getContext("2d");

            that.drawPage(pdfDocument, canvasContext, item.index);
            return canvas;
        });
    });
});

Each page will be generated using a canvas stretched to fill the FlipView. Each canvas is then filled using this code:

drawPage: function (pdfDocument, canvasContext, index) {
    var size = pdfDocument.getPageSize(index);

    canvasContext.canvas.width = size.x;
    canvasContext.canvas.height = size.y;

    var imageData = canvasContext.createImageData(size.x, size.y);

    var current = new Int32Array(size.x * size.y);

    pdfDocument.drawPage(index, current, 0, 0, size.x, size.y, false);

    // from  ARGB to ABGR
    for (var i = 0; i < current.length; i++) {

        var val = current[i];

        var cursor = i * 4;

        imageData.data[cursor] = (val >> 16) & 0xFF; //r
        imageData.data[cursor + 1] = (val >> 8) & 0xFF; //g
        imageData.data[cursor + 2] = val & 0xFF; //b
        imageData.data[cursor + 3] = 255; // a
    }            
    canvasContext.putImageData(imageData, 0, 0);

Using pdfDocument.drawPage, you can fill the image data object created by the canvas. You must then invert pixel color components to switch from ARGB to ABGR and finally paste the result Inside the canvas.

And voila!

Reading PDF and XPS on your Windows 8 application using WinRT

David Catuhe — Mon, 15 Apr 2013 09:20:40 GMT

PDF and XPS file formats are widely used across the world and you could need one day to display them inside your application.

Today I would like to share with you a simple way to do so by using an open source solution: MuPDF (a multiplatform lightweight PDF and XPS viewer).

MuPDF is already working on a WinRT solution you can grab on this GIT repo: http://git.ghostscript.com/?p=user/mvrhel/mupdf.git;a=summary

I also used the work of Libreliodev and especially their advanced port of MuPDF for WinRT.

The result is a simple but really useful Windows 8 Modern UI app that is able to display a PDF/XPS file: http://www.catuhe.com/msdn/pdfreader.zip

The solution is based on two projects:

MuPDF.winRT: a C++/CX WinRT component built upon MuPDF
PDFReader: A C#XAML project that used the WinRT component to display a PDF

For this article I created a C#/XAML application. But because MuPDF.winRT is a WinRT component, you can also use it from a HTML5/CSS3/Javascript app!

Using the WinRT component, you just have to use the following line to create a Document (which is the class used to interact with a PDF or a XPS):

pdfDocument = Document.Create(readBuffer, DocumentType.PDF, (int)Windows.Graphics.Display.DisplayProperties.LogicalDpi);

To draw a page, you just have to use the previous pdfDocument object and call the DrawPage method:

pdfDocument.DrawPage(PageNumber, buf, 0, 0, width, height, false);

This method draws a given page inside a IBuffer. To create it you just have to instantiate a WriteableBitmap and the inner PixelBuffer:

var size = Document.GetPageSize(PageNumber);
var width = (int)(size.X * ZoomFactor);
var height = (int)(size.Y * ZoomFactor);

var image = new WriteableBitmap(width, height);
IBuffer buf = new Buffer(image.PixelBuffer.Capacity);
buf.Length = image.PixelBuffer.Length;

The buffer is then copied back to the WriteableBitmap:

using (var stream = buf.AsStream())
{
    await stream.CopyToAsync(image.PixelBuffer.AsStream());
}

And finally the result is displayed on screen:

You can even display two pages on the same image:

await Task.Run(() => pdfDocument.DrawFirtPageConcurrent(PageNumber, buf, width, height));
await Task.Run(() => pdfDocument.DrawSecondPageConcurrent(PageNumber + 1, buf, width, height));

Using the mouse or the touch you can obviously navigate inside the document or zoom on pages. And with the power of vectorized PDF, you will always have the best display quality:

Optical zoom (bitmap based)

Vectorized zoom

To display these images, I used a simple FlipView so that I do not have to bother about gestures. The DataTemplate used for every page is based on a ScrollViewer in order to automaticaly handle zoom in and zoom out:

<FlipView Name="flipView">
    <FlipView.ItemTemplate>
        <DataTemplate>
            <ScrollViewer
                ZoomMode="Enabled" 
                HorizontalScrollMode="Auto"
                VerticalScrollMode="Auto"
                VerticalSnapPointsType="None"
                HorizontalSnapPointsType="None"
                HorizontalScrollBarVisibility="Auto"
                VerticalScrollBarVisibility="Auto"
                MinZoomFactor="1"            
                MaxZoomFactor="4">
                <Image Source="{Binding Image}" Width="{Binding Width}" Height="{Binding Height}" 
                       Stretch="Uniform" HorizontalAlignment="Center" Margin="0"/>
            </ScrollViewer>
        </DataTemplate>
    </FlipView.ItemTemplate>
</FlipView>

Simple, isn’t it ?

Faites partie du futur Instagram ou Facebook :)

David Catuhe — Wed, 10 Apr 2013 11:18:00 GMT

Chers développeurs, je vous propose d'inviter ceux d'entre vous qui sont intéressés à une après-midi de rencontres avec des diplômés de l'Ecole Supérieure de Commerce de Paris.

Ces diplômés ont monté des projets, trouvé et mis en place le design, fais le business plan, réalisé les études de marchés et parfois même commencé le marketing.

Il ne leur manque que VOUS :)

Que vous soyez motivés pour partager votre temps perso avec eux ou que vous ayez envie de rejoindre le projet en tant que CTO, venez assister à leurs pitchs et faites vos courses.

Si vous êtes intéressés, envoyez moi un mail. Le pitch aura lieu le 18/04 à Spark en mode "cosy".

WinRT component: How to generate a GUID?

David Catuhe — Mon, 25 Mar 2013 13:39:23 GMT

For your Windows 8 applications, you may need to generate a unique identifier (for instance, if you want to uniquely identify your users).

To do so, here is a small C++ WinRT component (I decided not to use C# to make the component light-weighted because it is pure native code and so you do not need to load the CLR to use it).

#pragma once
#include <Objbase.h>
using namespace Platform;

namespace UrzaBox
{
    public ref class Tools sealed
    {
    public:
        Tools();

        static Platform::String^ GetNewID()
        {
            GUID result;
            HRESULT hr = CoCreateGuid(&result);

            if (SUCCEEDED(hr))
            {
                Guid gd(result);
                return gd.ToString();
            }

            throw Exception::CreateException(hr);
        }
    };
}

Create a custom user control using JavaScript and WinJS for Windows 8

David Catuhe — Tue, 19 Mar 2013 13:08:31 GMT

As you may know, you can easily develop native applications for Windows 8 using JavaScript and WinJS.

Among a lot of other things, WinJS allows you to create custom control in order to factorize your UI.

Today, I would like to show you how to create this kind of control and to do so, please let me introduce you with the final version of the control:

As you can see, this is a 3D carousel that can be used to display a long list of items alongside a great (if you like it) mirror.

The sample Visual Studio 2012 project can be downloaded here.

(If you want to know how to achieve the same goal but with C# and XAML, you can read this article written by my colleague Sébastien Pertus: http://blogs.msdn.com/b/mim/archive/2013/03/19/create-a-custom-user-control-using-xaml-and-c-for-windows-8.aspx)

So you may now wonder how you can develop such a beautiful (!!) control. The response will be decomposed on 6 parts:

How to set up the layout?
Declaring and instancing the control
Using templates
Using data binding
Integrating user inputs and animations
Adding events
Optimizations

How to set up the layout?

The control itself is built using a grid (CSS3 Grid) defined as the “carousel-viewport”. Inside the viewport there are two items: one called the “carousel-container” to handle all items and one called “carousel-mirror” for the mirror (obviously).

Every item is built inside a “carousel-item-container” and contains two parts: a “carousel-item” for the standard part and a “carousel-mirror-item” for the mirror part.

You can then use the magic of CSS in order to completely style every part of the control:

.carousel-item {    
    transition: transform 0.1s ease;
}

.carousel-item:hover {
    transform: scale(0.95);
}

For instance, with these rules, I added a cool hover effect to my items. Simple, isn’t it?

Declaring and instancing the control

To declare a custom control, you just have to declare a WinJS class:

var carousel = WinJS.Class.define(
    function(element, options) {
    // Constructor
    },
    {// Instance members
    }, 
    {// Static members
    });

The first argument of the WinJS.Class.define function is the constructor of you control (used to instantiate your class), the second one is the list of instance members and the last one is the list of static members.

Inside the constructor, you have to define the root DOM elements you want to use. The instance members can be properties or public functions you want to make available.

For our control, the constructor will produce the layout with the following code:

var carousel = WinJS.Class.define(
    function(element, options) {
        this._element = element || document.createElement("div");
        this._element.winControl = this;
        this._selectedIndex = -1;

        // Root element
        this._element.style.display = "-ms-grid";
        this._element.style.msGridRows = "1fr";
        this._element.style.msGridColumns = "1fr";
        this._element.style.msTouchAction = "none";
        WinJS.Utilities.addClass(this._element, "carousel-viewport");

        // Items element
        this._itemsRoot = document.createElement("div");
        this._element.appendChild(this._itemsRoot);
        this._itemsRoot.msGridRow = "1";
        this._itemsRoot.msGridColumn = "1";
        this._itemsRoot.style.display = "-ms-grid";
        this._itemsRoot.style.msGridRows = "1fr";
        this._itemsRoot.style.msGridColumns = "1fr";
        this._itemsRoot.style.perspectiveOrigin = "50% 50%";
        WinJS.Utilities.addClass(this._itemsRoot, "carousel-container");

        // Mirror
        this._mirror = document.createElement("div");
        this._mirror.style.msGridRow = "1";
        this._mirror.style.msGridColumn = "1";
        this._mirror.style.msGridRowAlign = "stretch";
        this._mirror.style.msGridColumnAlign = "stretch";
        this._mirror.style.backgroundColor = "Black";
        WinJS.Utilities.addClass(this._mirror, "carousel-mirror");

The constructor is also responsible for initialization options:

maxVisibleItems: Count of items visible on each side of the current item
depth: Depth of each item relatively to previous
animationSpeed : Speed of the animation when you change the current item
mirrorEnabled: Define if the mirror is visible
perspective: Value indicating the amount of perspective

These options are defined by the user during the instantiation of the control:

<div id="carousel" data-win-control="DarkStone.Controls.Carousel" data-win-options="{mirrorEnabled: true}">
</div>

The instance of your class can be retrieved using the following code:

var carousel = element.querySelector("#carousel").winControl;

Every exposed properties can also be manipulated by the client. Here is for instance, how a slider can control the perspective:

element.querySelector("#perspective").onchange = function(evt) {
    carousel.perspective = element.querySelector("#perspective").value;
};

And the result:

In addition to the previous properties, our control also exposes some other useful members (always defined with the WinJS.Class.define function):

var carousel = WinJS.Class.define(
    function(element, options) {
        // Code hidden for readability
    },
{
    _doBinding: function () {
        // Code hidden for readability
    },
    refreshLayout: function () {
        // Code hidden for readability
    },
    element: {
        get: function () { return this._element; }
    },
    itemRender: { // Main item template
        set: function (value) {
            this._itemRender = value;
            this._doBinding();
        }
    },
    mirrorRender: { // Mirror template
        set: function (value) {
            this._mirrorRender = value;
            this._doBinding();
        }
    },
    itemsSource: { // Data source
        set: function (source) {
            this._source = source;
            this._doBinding();
        }
    },
    selectedIndex: { // Selected index
        get: function () { return this._selectedIndex; },
        set: function (index) {
            if (index < 0 || index >= this._items.length || index == this.selectedIndex)
                return;
            this._selectedIndex = index;
            this.refreshLayout();
        }
    },
    // more code…       
 });

Using templates

WinJS also provides a way to generate item content based on templates. For our control, I decided to use two templates: one for the item and one for the mirror.

The user can define his templates like this:

<div id="templateDiv" data-win-control="WinJS.Binding.Template">
    <img data-win-bind="src: imageUrl" class="templateImg" />
</div>
<div id="mirrorTemplate" data-win-control="WinJS.Binding.Template">
    <div class="mirrorContainer">
        <img data-win-bind="src: imageUrl" class="templateImg" />
        <div class="opacityMask"></div>
    </div>
</div>

The templates are retrieved from the web page:

var template = this._itemRender.winControl;
var mirrorTemplate= this._mirrorRender.winControl;

And used to create the DOM content:

template.render(data, item);

These templates are used by a function called createItem which takes a data from the data source and generate DOM objects using templates:

// Create item
var createItem = function (data) {
    // Creating DOM element and setting correct position
    var root = document.createElement("div");
    root.style.msGridRow = "1";
    root.style.msGridColumn = "1";
    root.style.msGridRowAlign = "center";
    root.style.msGridColumnAlign = "center";
    root.style.msTouchAction = "none";

    root.style.display = "-ms-grid";
    root.style.msGridRows = "1fr";
    root.style.msGridColumns = "1fr";
    root.style.opacity = 0;
    WinJS.Utilities.addClass(root, "carousel-item-container");


    that._itemsRoot.appendChild(root);

    var item = document.createElement("div");
    root.appendChild(item);

    WinJS.Utilities.addClass(item, "carousel-item");

    // Template
    template.render(data, item);

    if (mirrorTemplate) {
        // Clone for mirror
        var clone = document.createElement("div");
        mirrorTemplate.render(data, clone);
        WinJS.Utilities.addClass(clone, "carousel-mirror-item");
        root.appendChild(clone);
        clone.style.transform = "translateY(" + item.clientHeight + "px) rotateX(180deg)";
    }
    
    that._itemsRoot.removeChild(root);

    return root;
};

This function generates the correct class for each part of the item. This will allow you to skin them easily. For instance, using the following templates:

<div id="templateDiv" data-win-control="WinJS.Binding.Template">
</div>
<div id="mirrorTemplate" data-win-control="WinJS.Binding.Template">
</div>

And the following CSS rules:

.carousel-item {    
    width: 600px;
    height: 600px;
    background-color: red;
}

.carousel-mirror-item {
    width: 600px;
    height: 600px;
    background-color: blue;
}

You will obtain the following result:

Using data binding

The user can define data using a property called itemsSource:

itemsSource: { // Data source
    set: function (source) {
        this._source = source;
        this._doBinding();
    }
},

You can use a simple array or a WinJS.Binding.List:

var carousel = element.querySelector("#carousel").winControl;
var list = new WinJS.Binding.List();

list.push({ imageUrl: "/assets/pic01.jpg" });
list.push({ imageUrl: "/assets/pic02.jpg" });
list.push({ imageUrl: "/assets/pic03.jpg" });
list.push({ imageUrl: "/assets/pic04.jpg" });
list.push({ imageUrl: "/assets/pic05.jpg" });
list.push({ imageUrl: "/assets/pic06.jpg" });

carousel.itemsSource = list;

Once the itemsSource is set, the control just have to browse it and call createItem for each element.

If you use a WinJS.Binding.List, the control can also handle two events to dynamically support insertion and deletion of items:

// Events
if (usingBindingList) {
    this._source.addEventListener("itemremoved", function (evt) { // On item removed
        var previous = that._items.splice(evt.detail.index, 1)[0];

        that._itemsRoot.removeChild(previous);

        if (evt.detail.index < that.selectedIndex) {
            that.selectedIndex--;
        } else if (Math.abs(that.selectedIndex - evt.detail.index) <= that.maxVisibleItems) {
            that.refreshLayout();
        }
    });
    this._source.addEventListener("iteminserted", function (evt) { // On item inserted
        var current = evt.detail.index;
        var newOne = createItem(that._source.getAt(current));

        that._items.splice(evt.detail.index, 0, newOne);

        if (Math.abs(that.selectedIndex - current) <= that.maxVisibleItems) {
            that.refreshLayout();
        }
    });
}

Using this events, if the user executes this code:

element.querySelector("#button1").onclick = function () {
    list.splice(0, 1); // Removing first item
};

element.querySelector("#button2").onclick = function () {
    list.push({ imageUrl: "/assets/pic01.jpg" });  // Adding new item
    list.push({ imageUrl: "/assets/pic02.jpg" }); // Adding new item
};

The list will behave like this:

Integrating user inputs and animations

To respond to user inputs, you just have to handle pointer events. The pointer events (more about here) are used to unify mouse, touch and pens inside a single event model. So for our control, you just have to add the following code to handle all kind of inputs:

// Pointer events
this._element.onmspointerdown = function (evt) {
    initialOffset = evt.clientX;

    leftButtonDown = evt.buttons == 1;

    that._pageChanged = false;
};

this._element.onmspointermove = function (evt) {
    if (!leftButtonDown)
        return;

    var moveThreshold = 40; // 40px for a slide

    if (Math.abs(evt.clientX - initialOffset) > moveThreshold) {
        if (evt.clientX < initialOffset) {
            that.selectedIndex++;
        } else {
            that.selectedIndex--;
        }
        that._pageChanged = true;
        leftButtonDown = false;
    }
};

this._element.onmspointerup = function (evt) {
    leftButtonDown = false;
};

The principle is simple: if the user moves his finger/mouse/pen over more than 40 pixels, we can change the current item according to the direction of the movement.

To create a good looking control, you must ensure that every update of the layout is animated (fast and fluid!). To do so, you may have noticed that every “carousel-item-container” has a CSS style to define transitions:

root.style.transition = "transform " + that._animationSpeed + "s ease-in-out, opacity " + that._animationSpeed + "s linear";

This transition defines that every change on the transform and opacity properties is not immediate but animated from its original value to the final value during a specific duration.

If you want to learn more about CSS3 Transitions, you can go just here.

To take advantage of this great feature, the code just have to change the transformation and the opacity of each item.

The transformation itself (CSS 3D Transforms) allows elements styled with CSS to be transformed in two-dimensional or three-dimensional space.

The transform matrix is built using this code:

var depth = (current == this.selectedIndex) ? 0 : -this._depth;
var rotation = (current == this.selectedIndex) ? 0 : ((current < this.selectedIndex) ? -45 : 45);
var offset = (current == this.selectedIndex) ? 0 : ((current < this.selectedIndex) ? -this._baseWidth : this._baseWidth);

// Transform
item.style.transform = "rotateY(" + rotation + "deg) translate3d(" + 
                        (((current - this.selectedIndex) * this._baseWidth) + offset) + "px, 0px, " + depth + "px)";

To summarize, the items are transformed according to the following schema:

The problem you can face when dealing with CSS 3D Transforms is about zIndex. Indeed, when you move objects using matrices, something objects may overlap in a wrong order.

For resolving this issue, you just have to correctly organize the zIndex value of your items:

// Z-Index
var total = this._items.length;
item.style.zIndex = (current == this.selectedIndex) ? total + 1 : total - Math.abs(current - this.selectedIndex);

Using this code, you can ensure that items are rendered in the correct order (depending on their distance to the selected item).

The opacity is computed using this code:

// Opacity
var currentOpacity;
if (current == this.selectedIndex) {
    currentOpacity = 1;
    
} else {
    currentOpacity = 1.0 - (Math.abs(current - this.selectedIndex) / this._maxVisibleItems);
}

Every item has an opacity proportional to its distance from the current item:

Adding events

Obviously, a good control must raise events. For instance, if an user select an item, it would be nice to raise a “selectedindexchanged”.

For our control, first of all you have to “dispatch” the event with the following code:

selectedIndex: { // Selected index
    get: function () { return this._selectedIndex; },
    set: function (index) {
        if (index < 0 || index >= this._items.length || index == this.selectedIndex)
            return;
        this._selectedIndex = index;
        this.refreshLayout();

        this.dispatchEvent("selectedindexchanged", {
            selectedIndex: index
        });
    }
},

When the user changes the selectedIndex, we just have to raise the event with dispatchEvent function of the control.

But the control will not natively provide this function. To add it, you just have to call a small WinJS function:

WinJS.Class.mix(DarkStone.Controls.Carousel,
    WinJS.Utilities.createEventProperties("selectedindexchanged"),
    WinJS.UI.DOMEventMixin);

With only these lines, you can add event listener on your control like any other control:

carousel.addEventListener("selectedindexchanged", function() {

});

Optimizations

Finally, just to be sure our control works well on very low end hardware, we must ensure that only visible items are effectively handled by the DOM.

In order to achieve this goal, we just have to remove items from the DOM when their opacity is inferior or equal to 0. Obviously you must re-inject them when they become visible again:

// Opacity
var currentOpacity;
if (current == this.selectedIndex) {
    currentOpacity = 1;
    
    if (!isPresentInsideDOM) {
        this._itemsRoot.appendChild(item);
    }
} else {
    currentOpacity = 1.0 - (Math.abs(current - this.selectedIndex) / this._maxVisibleItems);

    // Virtualization
    if (currentOpacity <= 0 && isPresentInsideDOM) {
        this._itemsRoot.removeChild(item);
    } else if (currentOpacity > 0 && !isPresentInsideDOM) {
        this._itemsRoot.appendChild(item);
    }
}

Using this code, we are now ready to handle thousands of items even on a low end devices.

The control is now complete, feel free to use it in your own app

Going further

If you want to read more about some subjects I addressed in this article, here are some links you may find useful:

Complete code

The complete code can be downloaded here or you can copy/paste from here:

(function () {

    var carousel = WinJS.Class.define(
        function(element, options) {
            this._element = element || document.createElement("div");
            this._element.winControl = this;
            this._selectedIndex = -1;

            // Root element
            this._element.style.display = "-ms-grid";
            this._element.style.msGridRows = "1fr";
            this._element.style.msGridColumns = "1fr";
            this._element.style.msTouchAction = "none";
            WinJS.Utilities.addClass(this._element, "carousel-viewport");

            // Items element
            this._itemsRoot = document.createElement("div");
            this._element.appendChild(this._itemsRoot);
            this._itemsRoot.msGridRow = "1";
            this._itemsRoot.msGridColumn = "1";
            this._itemsRoot.style.display = "-ms-grid";
            this._itemsRoot.style.msGridRows = "1fr";
            this._itemsRoot.style.msGridColumns = "1fr";
            this._itemsRoot.style.perspectiveOrigin = "50% 50%";
            WinJS.Utilities.addClass(this._itemsRoot, "carousel-container");

            // Mirror
            this._mirror = document.createElement("div");
            this._mirror.style.msGridRow = "1";
            this._mirror.style.msGridColumn = "1";
            this._mirror.style.msGridRowAlign = "stretch";
            this._mirror.style.msGridColumnAlign = "stretch";
            this._mirror.style.backgroundColor = "Black";
            WinJS.Utilities.addClass(this._mirror, "carousel-mirror");

            this._items = [];
            this._source = null;
            this._itemRender = null;
            this._mirrorRender = null;
            this._pageChanged = false;
            this._baseWidth = 0;

            // Default values
            this._maxVisibleItems = 15;
            this._depth = 100;
            this._animationSpeed = 0.3;
            this._mirrorEnabled = false;
            this.perspective = 800;

            // Options
            if (options) {
                this.depth = options.depth || this._depth;
                this.animationSpeed = options.animationSpeed || this._animationSpeed;
                this.perspective = options.perspective || this.perspective;
                this._mirrorEnabled = options.mirrorEnabled || this._mirrorEnabled;
                this._maxVisibleItems = options.maxVisibleItems || this._maxVisibleItems;
            }

            if (this._mirrorEnabled) {
                this._itemsRoot.appendChild(this._mirror);
            }

            // Pointer events
            var that = this;
            var initialOffset = 0;
            var leftButtonDown = false;

            this._element.addEventListener("MSPointerDown", function (evt) {
                initialOffset = evt.clientX;

                leftButtonDown = evt.buttons == 1;

                that._pageChanged = false;
            }, false);

            this._element.addEventListener("MSPointerMove", function (evt) {
                if (!leftButtonDown)
                    return;

                var moveThreshold = 40; // 40px for a slide

                if (Math.abs(evt.clientX - initialOffset) > moveThreshold) {
                    if (evt.clientX < initialOffset) {
                        that.selectedIndex++;
                    } else {
                        that.selectedIndex--;
                    }
                    that._pageChanged = true;
                    leftButtonDown = false;
                }
            }, false);

            this._element.addEventListener("MSPointerUp", function () {
                leftButtonDown = false;
            }, false);
        },
    {
        _doBinding: function () {
            if (!this._source || !this._itemRender || (this._mirrorEnabled && !this._mirrorRender))
                return;

            var usingBindingList = (this._source.getAt !== undefined);

            var template = this._itemRender.winControl;
            var mirrorTemplate;

            if (this._mirrorEnabled && this._mirrorRender)
                mirrorTemplate = this._mirrorRender.winControl;

            var that = this;

            var onItemClick = function (evt) { // Selecting the current item
                var newOne = that._items.indexOf(evt.currentTarget.parentElement);
                if (!that._pageChanged) {
                    that.selectedIndex = newOne;
                }
            };

            // Browse data source
            for (var index = 0; index < this._items.length; index++) {
                var item = this._items[index];
                
                if (item.parentElement) {
                    this._itemsRoot.removeChild(item);
                }
            }
            this._items = [];

            // Create item
            var createItem = function (data, initalTransform) {
                var root = document.createElement("div");
                root.style.msGridRow = "1";
                root.style.msGridColumn = "1";
                root.style.msGridRowAlign = "center";
                root.style.msGridColumnAlign = "center";
                root.style.msTouchAction = "none";
                if (initalTransform)
                    root.style.transform = initalTransform;
                root.style.transition = "transform " + that._animationSpeed + "s ease-in-out, opacity " 
                                                     + that._animationSpeed + "s linear";
                root.style.display = "-ms-grid";
                root.style.msGridRows = "1fr";
                root.style.msGridColumns = "1fr";
                root.style.opacity = 0;
                root.onmousemove = function (evt) {
                    evt.preventDefault(); // Preventing picture manipulation
                };
                WinJS.Utilities.addClass(root, "carousel-item-container");

                that._itemsRoot.appendChild(root);

                var item = document.createElement("div");
                root.appendChild(item);

                item.onclick = onItemClick;
                WinJS.Utilities.addClass(item, "carousel-item");

                // Template
                template.render(data, item);

                if (mirrorTemplate) {
                    // Clone for mirror
                    var clone = document.createElement("div");
                    mirrorTemplate.render(data, clone);
                    WinJS.Utilities.addClass(clone, "carousel-mirror-item");
                    root.appendChild(clone);
                    clone.style.transform = "translateY(" + item.clientHeight + "px) rotateX(180deg)";
                }
                
                that._itemsRoot.removeChild(root);

                return root;
            };

            // Browse data source
            for (var index = 0; index < this._source.length; index++) {
                var data = usingBindingList ? this._source.getAt(index) : this._source[index];

                this._items.push(createItem(data));
            }

            this._selectedIndex = 0;
            this.refreshLayout();

            // Events
            if (usingBindingList) {
                this._source.addEventListener("itemremoved", function (evt) { // On item removed
                    var previous = that._items.splice(evt.detail.index, 1)[0];

                    if (previous.parentElement) {
                        that._itemsRoot.removeChild(previous);
                    }

                    if (evt.detail.index < that.selectedIndex) {
                        that.selectedIndex--;
                    } else if (Math.abs(that.selectedIndex - evt.detail.index) <= that.maxVisibleItems) {
                        that.refreshLayout();
                    }
                });
                this._source.addEventListener("iteminserted", function (evt) { // On item inserted
                    var current = evt.detail.index;
                    var newOne = createItem(that._source.getAt(current));

                    that._items.splice(evt.detail.index, 0, newOne);

                    if (Math.abs(that.selectedIndex - current) <= that.maxVisibleItems) {
                        that.refreshLayout();
                    }
                });
            }

            // Mirror
            this._mirror.style.transform = "translateY(" + this.element.clientHeight / 2 + "px)";
        },
        refreshLayout: function () {
            if (this._items.length == 0) {
                return;
            }

            var current = this.selectedIndex;
            var total = this._items.length;
            for (var index = 0; index < total; index++) {

                var item = this._items[current];
                var isPresentInsideDOM = (item.parentElement !== null);

                // Opacity
                var currentOpacity;
                if (current == this.selectedIndex) {
                    currentOpacity = 1;
                    
                    if (!isPresentInsideDOM) {
                        this._itemsRoot.appendChild(item);
                        isPresentInsideDOM = true;
                    }
                } else {
                    currentOpacity = 1.0 - (Math.abs(current - this.selectedIndex) / this._maxVisibleItems);

                    // Virtualization
                    if (currentOpacity <= 0 && isPresentInsideDOM) {
                        this._itemsRoot.removeChild(item);
                        isPresentInsideDOM = false;
                    } else if (currentOpacity > 0 && !isPresentInsideDOM) {
                        this._itemsRoot.appendChild(item);
                        isPresentInsideDOM = true;
                    }
                }

                // Setting opacity
                item.style.opacity = currentOpacity;

                if (isPresentInsideDOM) {
                    if (!this._baseWidth)
                        this._baseWidth = item.clientWidth;

                    var depth = (current == this.selectedIndex) ? 0 : -this._depth;
                    var rotation = (current == this.selectedIndex) ? 0 : ((current < this.selectedIndex) ? -45 : 45);
                    var offset = (current == this.selectedIndex) ? 0 : (
                                             (current < this.selectedIndex) ? -this._baseWidth : this._baseWidth);

                    // Transform
                    item.style.transform = "rotateY(" + rotation + "deg) translate3d(" 
                       + (((current - this.selectedIndex) * this._baseWidth) + offset) + "px, 0px, " + depth + "px)";

                    // Z-Index
                    item.style.zIndex = (current == this.selectedIndex) ? total + 1 : total
                       - Math.abs(current - this.selectedIndex);
                }

                current++;

                if (current == this._items.length)
                    current = 0;
            }
        },
        element: {
            get: function () { return this._element; }
        },
        itemRender: { // Main item template
            set: function (value) {
                this._itemRender = value;
                this._doBinding();
            }
        },
        mirrorRender: { // Mirror template
            set: function (value) {
                this._mirrorRender = value;
                this._doBinding();
            }
        },
        itemsSource: { // Data source
            set: function (source) {
                this._source = source;
                this._doBinding();
            }
        },
        selectedIndex: { // Selected index
            get: function () { return this._selectedIndex; },
            set: function (index) {
                if (index < 0 || index >= this._items.length || index == this.selectedIndex)
                    return;
                this._selectedIndex = index;
                this.refreshLayout();

                this.dispatchEvent("selectedindexchanged", {
                    selectedIndex: index
                });
            }
        },
        animationSpeed: {
            get: function () { return this._animationSpeed; },
            set: function (value) {
                this._animationSpeed = value;
            }
        },
        depth: {
            get: function () { return this._depth; },
            set: function (value) {
                if (this._depth === value)
                    return;

                this._depth = value;

                this._refreshLayout();
            }
        },
        perspective: {
            get: function () {
                return parseInt(this._itemsRoot.style.perspective.replace("px"));
            },
            set: function (value) {
                this._itemsRoot.style.perspective = value + "px";
            }
        },
        maxVisibleItems: {
            get: function () {
                return this._maxVisibleItems;
            },
            set: function (value) {
                if (this._maxVisibleItems === value) {
                    return;
                }

                this._maxVisibleItems = value;
                this._refreshLayout();
            }
        }
    });

    WinJS.Namespace.define("DarkStone.Controls", {
        Carousel: carousel
    });
    
    WinJS.Class.mix(DarkStone.Controls.Carousel,
        WinJS.Utilities.createEventProperties("selectedindexchanged"),
        WinJS.UI.DOMEventMixin);
})();

TechDays 2013 - Vidéos de la plénière développeurs et des sessions Coding4Fun et Geek is in da House

David Catuhe — Fri, 08 Mar 2013 07:23:14 GMT

Ben voila, tout est dans le titre

Développeurs c’est le chef

Coding4Fun 2013

Geek is in the house

Developing a WinRT component to create a video file using Media Foundation

David Catuhe — Wed, 06 Mar 2013 10:00:09 GMT

After creating a GIF file using my previous article, I propose you to use the power of WinRT to create a component in order to produce video file from your drawings (using for instance a canvas with Javascript).

To do so, we will use Media Foundation COM components. Media Foundation is the next generation multimedia platform for Windows that enables developers, consumers, and content providers to embrace the new wave of premium content with enhanced robustness, unparalleled quality, and seamless interoperability.

And as I said before, Media Foundation is based on COM components. But thanks to C++ projects for Windows Store, we can create a WinRT component on top of Media Foundation in order to use it from our Javascript or .NET projects.

This WinRT component will be called VideoGenerator.

The VideoGenerator component

The VideoGenerator class will have a main method called AppendNewFrame. This method is used to add a new frame to the video.

The class requires the following information to work:

Frame’s width
Frame’s height
Delay between frames (in milliseconds)

The class itself can be like the following:

public ref class VideoGenerator sealed
{

public:
    VideoGenerator(UINT32 width, UINT32 height, Windows::Storage::Streams::IRandomAccessStream^ stream,
                   UINT32 delay);
    virtual ~VideoGenerator();

    void AppendNewFrame(const Array<byte> ^videoFrameBuffer);
    void Finalize();
};

Obviously, this class also requires private data to work:

UINT32 videoWidth;
UINT32 videoHeight;
UINT32 fps;
UINT32 bitRate;
UINT32 frameSize;
GUID   encodingFormat;
GUID   inputFormat;

videoWidth is the width of a frame
videoHeight is the height of a frame
fps is the number of frames per second
bitRate is obviously the bits rate
frameSize is equal to videoWidth * videoHeight
encodingFormat defines the encoder to use (WMV for us)
inputFormat defines the format of the pixels sent to this class (RGB32)

Initialization

Starting with this information, here is the constructor:

VideoGenerator::VideoGenerator(UINT32 width, UINT32 height, 
                               Windows::Storage::Streams::IRandomAccessStream^ stream, UINT32 delay)
{
    videoWidth = width;
    videoHeight = height;
    fps = 25;
    bitRate = 400000;
    frameSize = videoWidth * videoHeight;
    encodingFormat = MFVideoFormat_WMV3;
    inputFormat = MFVideoFormat_RGB32;

    HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
    if (SUCCEEDED(hr))
    {
        hr = MFStartup(MF_VERSION);
        if (SUCCEEDED(hr))
        {
            hr = InitializeSinkWriter(stream);
            if (SUCCEEDED(hr))
            {
                initiated = true;
                rtStart = 0;
                rtDuration = (10000000 * delay) / 1000;
            }
        }
    }
}

This method needs to initialize COM engine (CoInitializeEx) and also needs to initialize Media Foundation (MFStartup).

To compile you will need the following includes:

#include <Windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <Mfreadwrite.h>
#include <mferror.h>
#include <wrl\client.h>
#include <memory>

Please note that the bigger the bitRate is the bigger the file will be.

The next step after initializing things is to create a sink writer (IMFSinkWriter) which is used to control writings to the output file.

The sink writer enables you to author media files by passing in uncompressed or encoded data. For example, you can use it to re-encode a video file, or to capture live video from a webcam to a file.

In our case we will use it by passing uncompressed data from .NET or javascript.

HRESULT VideoGenerator::InitializeSinkWriter(Windows::Storage::Streams::IRandomAccessStream^ stream)
{    
    ComPtr<IMFAttributes> spAttr;
    ComPtr<IMFMediaType>  mediaTypeOut;   
    ComPtr<IMFMediaType>  mediaTypeIn;           
    ComPtr<IMFByteStream> spByteStream;
    HRESULT hr = MFCreateMFByteStreamOnStreamEx((IUnknown*)stream, &spByteStream);

    if (SUCCEEDED(hr))
    {        
        MFCreateAttributes(&spAttr, 10);
        spAttr->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, true);

        hr = MFCreateSinkWriterFromURL(L".wmv", spByteStream.Get(), spAttr.Get(), &sinkWriter);
    }

    // Set the output media type.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateMediaType(&mediaTypeOut);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);     
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetGUID(MF_MT_SUBTYPE, encodingFormat);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetUINT32(MF_MT_AVG_BITRATE, bitRate);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeSize(mediaTypeOut.Get(), MF_MT_FRAME_SIZE, videoWidth, videoHeight);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeOut.Get(), MF_MT_FRAME_RATE, fps, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeOut.Get(), MF_MT_PIXEL_ASPECT_RATIO, 1, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->AddStream(mediaTypeOut.Get(), &streamIndex);   
    }

    // Set the input media type.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateMediaType(&mediaTypeIn);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetGUID(MF_MT_SUBTYPE, inputFormat);     
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeSize(mediaTypeIn.Get(), MF_MT_FRAME_SIZE, videoWidth, videoHeight);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeIn.Get(), MF_MT_FRAME_RATE, fps, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeIn.Get(), MF_MT_PIXEL_ASPECT_RATIO, 1, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->SetInputMediaType(streamIndex, mediaTypeIn.Get(), NULL);   
    }

    // Tell the sink writer to start accepting data.
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->BeginWriting();
    }

    return hr;
}

Starting from a WinRT Stream, you have to call MFCreateMFByteStreamOnStreamEx method to convert it to a IMFByteStream object. Then using MFCreateSinkWriterFromURL you can grab the sink writer.

The following part of the code is dedicated to configure the input and output format of the sink writer accordingly to defined parameters (mediaTypeOut and mediaTypeIn objects).

The mediaTypeOut is responsible for:

MF_MT_MAJOR_TYPE: The main type of the file (Video here)
MF_MT_SUBTYPE: The subtype (WMV here)
MF_MT_AVG_BITRATE: The bit rate or the compression rate if you will (Defines the quality of the video)
MF_MT_INTERLACE_MODE: Interlace mode (Progressive for us)
MF_MT_FRAME_SIZE: The size of the frame (width and height)
MF_MT_FRAME_RATE: Frame per second
MF_MT_PIXEL_ASPECT_RATIO: Aspect ratio

The mediaTypeIn is responsible for:

MF_MT_MAJOR_TYPE: The main type of the file (Video here)
MF_MT_SUBTYPE: The subtype (RGB32 here)
MF_MT_INTERLACE_MODE: Interlace mode (Progressive for us)
MF_MT_FRAME_SIZE: The size of the frame (width and height)
MF_MT_FRAME_RATE: Frame per second
MF_MT_PIXEL_ASPECT_RATIO: Aspect ratio

Adding a new frame

Once we have a sink writer, you just have to append new frame by sending a byte array to the component:

void VideoGenerator::AppendNewFrame(const Platform::Array<byte> ^videoFrameBuffer)
{
    auto length = videoFrameBuffer->Length / sizeof(DWORD);
    DWORD *buffer = (DWORD *)(videoFrameBuffer->Data);
    std::unique_ptr<DWORD[]> target(new DWORD[length]);

    for (UINT32 index = 0; index < length; index++)
    {
        DWORD color = buffer[index];
        BYTE b = (BYTE)((color & 0x00FF0000) >> 16);
        BYTE g = (BYTE)((color & 0x0000FF00) >> 8);
        BYTE r = (BYTE)((color & 0x000000FF));

#if ARM
        auto row = index / videoWidth;
        auto targetRow = videoHeight - row - 1;
        auto column = index - (row * videoWidth);
        target[(targetRow * videoWidth) + column] = (r << 16) + (g << 8) + b;
#else
        target[index] = (r << 16) + (g << 8) + b;
#endif
    }

    // Send frame to the sink writer.
    HRESULT hr = WriteFrame(target.get(), rtStart, rtDuration);
    if (FAILED(hr))
    {
        throw Platform::Exception::CreateException(hr);
    }
    rtStart += rtDuration;
}

Before writing data, you have to prepare it because:

Data must be provided using RGB format and in my case (HTML5 canvas), data is stored using BGR format
On ARM devices (detected thanks to the define #ARM), data is stored starting from the last line (and not the first line) so you have to revert them on the Y axis

Then, when your data is ready, just call the following code to append it to the output file:

HRESULT VideoGenerator::WriteFrame(
    DWORD *videoFrameBuffer,
    const LONGLONG& rtStart,        // Time stamp.
    const LONGLONG& rtDuration      // Frame duration.
    )
{
    ComPtr<IMFSample> sample;
    ComPtr<IMFMediaBuffer> buffer;

    const LONG cbWidth = 4 * videoWidth;
    const DWORD cbBuffer = cbWidth * videoHeight;

    BYTE *pData = NULL;

    // Create a new memory buffer.
    HRESULT hr = MFCreateMemoryBuffer(cbBuffer, &buffer);

    // Lock the buffer and copy the video frame to the buffer.
    if (SUCCEEDED(hr))
    {
        hr = buffer->Lock(&pData, NULL, NULL);
    }
    if (SUCCEEDED(hr))
    {
        hr = MFCopyImage(
            pData,                      // Destination buffer.
            cbWidth,                    // Destination stride.
            (BYTE*)videoFrameBuffer,    // First row in source image.
            cbWidth,                    // Source stride.
            cbWidth,                    // Image width in bytes.
            videoHeight                // Image height in pixels.
            );
    }
    if (buffer.Get())
    {
        buffer->Unlock();
    }

    // Set the data length of the buffer.
    if (SUCCEEDED(hr))
    {
        hr = buffer->SetCurrentLength(cbBuffer);
    }

    // Create a media sample and add the buffer to the sample.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateSample(&sample);
    }
    if (SUCCEEDED(hr))
    {
        hr = sample->AddBuffer(buffer.Get());
    }

    // Set the time stamp and the duration.
    if (SUCCEEDED(hr))
    {
        hr = sample->SetSampleTime(rtStart);
    }
    if (SUCCEEDED(hr))
    {
        hr = sample->SetSampleDuration(rtDuration);
    }

    // Send the sample to the Sink Writer.
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->WriteSample(streamIndex, sample.Get());
    }

    return hr;
}

The code creates a sample, fill it with your data and set time and duration before writing the sample to the sink writer.

Closing and cleaning

When you are done, you just have to call the Finalize method (and for the sake of completeness, I also add the destructor here):

VideoGenerator::~VideoGenerator()
{
    Finalize();
}

void VideoGenerator::Finalize()
{
    if (!initiated)
        return;

    initiated = false;
    sinkWriter->Finalize();
    MFShutdown();
}

Finalization is just about closing the sink writer and release Media Foundation API (MFShutdown).

Using VideoGenerator

VideoGenerator is a WinRT component so you can easily use it from .NET or Javascript. For instance, using Javascript, the client code can be something like that:

var picker = new Windows.Storage.Pickers.FileSavePicker();
picker.fileTypeChoices.insert(Flipflop.Tools.GetString("VideoFiles"), [".wmv"]);

picker.pickSaveFileAsync().then(function (file) {
    if (!file) {
        return;
    }
    file.openAsync(Windows.Storage.FileAccessMode.readWrite).then(function (stream) {
        var videoGenerator = new VideoTools.VideoGenerator(800, 600, stream, 16);

        for (var commandsIndex = 0; commandsIndex < 50; commandsIndex++) {
            var canvas = document.getElementById("canvas" + commandsIndex);
            var context = canvas.getContext("2d");

            var data = context.getImageData(0, 0, width, height);

            var bytes = data.data;
            videoGenerator.appendNewFrame(bytes);
        }

        videoGenerator.finalize();
        stream.close();
    });
});

Complete code

The complete VideoGenerator code can be found here.

Or copy/paste right here.

VideoGenerator.h

#pragma once
#include <Windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <Mfreadwrite.h>
#include <mferror.h>
#include <wrl\client.h>
#include <memory>

using namespace Platform;
using namespace Microsoft::WRL;

namespace VideoTools
{
    public ref class VideoGenerator sealed
    {
        UINT32 videoWidth;
        UINT32 videoHeight;
        UINT32 fps;
        UINT32 bitRate;
        UINT32 frameSize;
        GUID   encodingFormat;
        GUID   inputFormat;

        DWORD  streamIndex;
        ComPtr<IMFSinkWriter> sinkWriter;

        bool   initiated;

        LONGLONG rtStart;
        UINT64 rtDuration;

    private:
        HRESULT InitializeSinkWriter(Windows::Storage::Streams::IRandomAccessStream^ stream);
        HRESULT WriteFrame(DWORD *videoFrameBuffer, const LONGLONG& rtStart, 
                           const LONGLONG& rtDuration);

    public:
        VideoGenerator(UINT32 width, UINT32 height, 
                       Windows::Storage::Streams::IRandomAccessStream^ stream, UINT32 delay);
        virtual ~VideoGenerator();

        void AppendNewFrame(const Array<byte> ^videoFrameBuffer);
        void Finalize();
    };
}

VideoGenerator.cpp

#include "pch.h"
#include "VideoGenerator.h"

#pragma comment(lib, "mfreadwrite")
#pragma comment(lib, "mfplat")
#pragma comment(lib, "mfuuid")

using namespace VideoTools;

VideoGenerator::VideoGenerator(UINT32 width, UINT32 height, 
                               Windows::Storage::Streams::IRandomAccessStream^ stream, UINT32 delay)
{
    videoWidth = width;
    videoHeight = height;
    fps = 25;
    bitRate = 400000;
    frameSize = videoWidth * videoHeight;
    encodingFormat = MFVideoFormat_WMV3;
    inputFormat = MFVideoFormat_RGB32;

    HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
    if (SUCCEEDED(hr))
    {
        hr = MFStartup(MF_VERSION);
        if (SUCCEEDED(hr))
        {
            hr = InitializeSinkWriter(stream);
            if (SUCCEEDED(hr))
            {
                initiated = true;
                rtStart = 0;
                rtDuration = (10000000 * delay) / 1000;
            }
        }
    }
}

VideoGenerator::~VideoGenerator()
{
    Finalize();
}

void VideoGenerator::Finalize()
{
    if (!initiated)
        return;

    initiated = false;
    sinkWriter->Finalize();
    MFShutdown();
}

void VideoGenerator::AppendNewFrame(const Platform::Array<byte> ^videoFrameBuffer)
{
    auto length = videoFrameBuffer->Length / sizeof(DWORD);
    DWORD *buffer = (DWORD *)(videoFrameBuffer->Data);
    std::unique_ptr<DWORD[]> target(new DWORD[length]);

    for (UINT32 index = 0; index < length; index++)
    {
        DWORD color = buffer[index];
        BYTE b = (BYTE)((color & 0x00FF0000) >> 16);
        BYTE g = (BYTE)((color & 0x0000FF00) >> 8);
        BYTE r = (BYTE)((color & 0x000000FF));

#if ARM
        auto row = index / videoWidth;
        auto targetRow = videoHeight - row - 1;
        auto column = index - (row * videoWidth);
        target[(targetRow * videoWidth) + column] = (r << 16) + (g << 8) + b;
#else
        target[index] = (r << 16) + (g << 8) + b;
#endif
    }

    // Send frame to the sink writer.
    HRESULT hr = WriteFrame(target.get(), rtStart, rtDuration);
    if (FAILED(hr))
    {
        throw Platform::Exception::CreateException(hr);
    }
    rtStart += rtDuration;
}

HRESULT VideoGenerator::InitializeSinkWriter(Windows::Storage::Streams::IRandomAccessStream^ stream)
{    
    ComPtr<IMFAttributes> spAttr;
    ComPtr<IMFMediaType>  mediaTypeOut;   
    ComPtr<IMFMediaType>  mediaTypeIn;           
    ComPtr<IMFByteStream> spByteStream;
    HRESULT hr = MFCreateMFByteStreamOnStreamEx((IUnknown*)stream, &spByteStream);

    if (SUCCEEDED(hr))
    {        
        MFCreateAttributes(&spAttr, 10);
        spAttr->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, true);

        hr = MFCreateSinkWriterFromURL(L".wmv", spByteStream.Get(), spAttr.Get(), &sinkWriter);
    }

    // Set the output media type.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateMediaType(&mediaTypeOut);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);     
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetGUID(MF_MT_SUBTYPE, encodingFormat);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetUINT32(MF_MT_AVG_BITRATE, bitRate);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeOut->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeSize(mediaTypeOut.Get(), MF_MT_FRAME_SIZE, videoWidth, videoHeight);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeOut.Get(), MF_MT_FRAME_RATE, fps, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeOut.Get(), MF_MT_PIXEL_ASPECT_RATIO, 1, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->AddStream(mediaTypeOut.Get(), &streamIndex);   
    }

    // Set the input media type.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateMediaType(&mediaTypeIn);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);   
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetGUID(MF_MT_SUBTYPE, inputFormat);     
    }
    if (SUCCEEDED(hr))
    {
        hr = mediaTypeIn->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeSize(mediaTypeIn.Get(), MF_MT_FRAME_SIZE, videoWidth, videoHeight);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeIn.Get(), MF_MT_FRAME_RATE, fps, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = MFSetAttributeRatio(mediaTypeIn.Get(), MF_MT_PIXEL_ASPECT_RATIO, 1, 1);   
    }
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->SetInputMediaType(streamIndex, mediaTypeIn.Get(), NULL);   
    }

    // Tell the sink writer to start accepting data.
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->BeginWriting();
    }

    return hr;
}

HRESULT VideoGenerator::WriteFrame(
    DWORD *videoFrameBuffer,
    const LONGLONG& rtStart,        // Time stamp.
    const LONGLONG& rtDuration      // Frame duration.
    )
{
    ComPtr<IMFSample> sample;
    ComPtr<IMFMediaBuffer> buffer;

    const LONG cbWidth = 4 * videoWidth;
    const DWORD cbBuffer = cbWidth * videoHeight;

    BYTE *pData = NULL;

    // Create a new memory buffer.
    HRESULT hr = MFCreateMemoryBuffer(cbBuffer, &buffer);

    // Lock the buffer and copy the video frame to the buffer.
    if (SUCCEEDED(hr))
    {
        hr = buffer->Lock(&pData, NULL, NULL);
    }
    if (SUCCEEDED(hr))
    {
        hr = MFCopyImage(
            pData,                      // Destination buffer.
            cbWidth,                    // Destination stride.
            (BYTE*)videoFrameBuffer,    // First row in source image.
            cbWidth,                    // Source stride.
            cbWidth,                    // Image width in bytes.
            videoHeight                // Image height in pixels.
            );
    }
    if (buffer.Get())
    {
        buffer->Unlock();
    }

    // Set the data length of the buffer.
    if (SUCCEEDED(hr))
    {
        hr = buffer->SetCurrentLength(cbBuffer);
    }

    // Create a media sample and add the buffer to the sample.
    if (SUCCEEDED(hr))
    {
        hr = MFCreateSample(&sample);
    }
    if (SUCCEEDED(hr))
    {
        hr = sample->AddBuffer(buffer.Get());
    }

    // Set the time stamp and the duration.
    if (SUCCEEDED(hr))
    {
        hr = sample->SetSampleTime(rtStart);
    }
    if (SUCCEEDED(hr))
    {
        hr = sample->SetSampleDuration(rtDuration);
    }

    // Send the sample to the Sink Writer.
    if (SUCCEEDED(hr))
    {
        hr = sinkWriter->WriteSample(streamIndex, sample.Get());
    }

    return hr;
}

Going further

Some useful links to go further:

Media foundation doc: http://msdn.microsoft.com/en-us/library/windows/apps/ms694197.aspx
About Media foundation: http://msdn.microsoft.com/en-us/library/windows/apps/ms696274.aspx
Creating a WinRT component – part 1:http://blogs.msdn.com/b/eternalcoding/archive/2012/08/13/creating-a-winrt-component-using-c-cx-deform-a-direct2d-effect-toolkit.aspx
Creating a WinRT component – part 2: http://blogs.msdn.com/b/eternalcoding/archive/2012/08/28/creating-a-winrt-component-using-c-cx-part-2-adding-a-custom-direct2d-effect-to-deform.aspx

How to create an animated GIF with WinRT

David Catuhe — Thu, 28 Feb 2013 15:05:20 GMT

Why using GIF files?

For one of my project (Flipflop: http://apps.microsoft.com/windows/app/flipflop/99c01512-fe4f-4d1a-872e-eb9fd6638ff4), I needed to create an animated GIF based on a group of HTML5 canvas.

A GIF produced by Flipflop

The main goal of Flipflop is to allow users to create an animated flipbook. And to allow them to share it, I finally found that GIF files were a great deal.

Creating a GIF using WIC components

So I propose you to share the small class that I used to achieve this goal: GifMaker.

To work this class requires you to specify frames size and obviously all frames:

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices.WindowsRuntime;
using Windows.Foundation;
using Windows.Graphics.Imaging;
using Windows.Storage;

namespace Utilities
{
    public sealed class GifMaker
    {
        readonly List<byte[]> frames = new List<byte[]>();
        private readonly uint frameWidth;
        private readonly uint frameHeight;

        public GifMaker(uint width, uint height)
        {
            frameWidth = width;
            frameHeight = height;
        }

        public void AppendNewFrame([ReadOnlyArray]byte[] frame)
        {
            frames.Add(frame);
        }

This class mainly uses WIC or actually Windows.Graphics namespace. This namespace contains a class called BitmapEncoder that will allow us to generate files using specific encoders such as GifEncoder.

The point is that GIF files are composed of a list of frames. You just have to browse through them and send each of them to the encoder using GoToNextFrameAsync to register a new frame:

public IAsyncInfo GenerateAsync(StorageFile file)
{
    return AsyncInfo.Run(async ctx =>
        {
            var outStream = await file.OpenAsync(FileAccessMode.ReadWrite);

            var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.GifEncoderId, outStream);

            for (int i = 0; i < frames.Count; i++)
            {
                var pixels = frames[i];
                encoder.SetPixelData(BitmapPixelFormat.Rgba8, BitmapAlphaMode.Ignore,
                                     frameWidth, frameHeight,
                                     92.0, 92.0,
                                     pixels);

                if (i < frames.Count - 1)
                    await encoder.GoToNextFrameAsync();
            }

            await encoder.FlushAsync();
            outStream.Dispose();
        });
}

Easy, isn’t it ?

Adding delay between frames

To be complete, you also have to handle the delay between frames and for this point, I must admit it took me a long time to figure out how to do that.

The solution is handled by frame properties (encoder.BitmapProperties) and especially via the property named “/grctlext/Delay” (!!).

To the final code is like the following:

public IAsyncInfo GenerateAsync(StorageFile file, int delay)
{
    return AsyncInfo.Run(async ctx =>
        {
            var outStream = await file.OpenAsync(FileAccessMode.ReadWrite);

            var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.GifEncoderId, outStream);

            for (int i = 0; i < frames.Count; i++)
            {
                var pixels = frames[i];
                encoder.SetPixelData(BitmapPixelFormat.Rgba8, BitmapAlphaMode.Ignore,
                                     frameWidth, frameHeight,
                                     92.0, 92.0,
                                     pixels);

                if (i == 0)
                {
                    var properties = new BitmapPropertySet
                        {
                            {
                                "/grctlext/Delay",
                                new BitmapTypedValue(delay / 10, PropertyType.UInt16)
                            }
                        };

                    await encoder.BitmapProperties.SetPropertiesAsync(properties);
                }

                if (i < frames.Count - 1)
                    await encoder.GoToNextFrameAsync();
            }

            await encoder.FlushAsync();
            outStream.Dispose();
        });
}

Using it from other languages

For instance if you want to use it from JavaScript, you just have to do something like that:

var picker = new Windows.Storage.Pickers.FileSavePicker();
picker.fileTypeChoices.insert("Gif files", [".gif"]);

picker.pickSaveFileAsync().then(function(file) {
    if (!file) {
        return;
    }
    var gifMaker = new Utilities.GifMaker(800, 600);

    for (var commandsIndex = 0; commandsIndex < currentFlipflop.pages.length; commandsIndex++) {
        // This code is used to retrieve canvases bytes
        var bytes = Flipflop.Tools.GetBytesfromFlipflop(currentFlipflop.pages[commandsIndex],
            800, 600);

        gifMaker.appendNewFrame(bytes);
    }

    gifMaker.generateAsync(file, 200).done();
});

Obviously it works the same way from .NET!