Web Application Template (WAT) is a great tool to simply generate an app based on the content of a Web site. Yclip_image002ou can then add application bar, navigation bar, inject scripts or CSS and really customize and well integrate your application using a simple .json configuration file.

clip_image004

You can learn more about WAT on this site: http://wat-docs.azurewebsites.net/

You can download or even contribute to the project using the Codeplex site of WAT: https://wat.codeplex.com/

You want to discuss about this article: reach me on Twitter: @deltakosh

Filling the gap

One missing thing in this scenario is the support of offline mode. By nature, WAT is connected to a web site using a Webview embedded into a WinJS application. So when the network is down, the Webview is unable to get data and so the application will fallback to “offline content” inside the package.

WebView doesn’t support AppCache. Even if it did, it would not bring value as most websites don’t implement it yet. Feedback about AppCache from partners and community has been extremely negative.

WAP1_1_13A

This is why I decided to add a new module: the offline engine aka SuperCache! The main goal of this module is to save visited pages into a local cache in order to be able to reuse them when network is down. Mainly here, we want to create a complete HTTP caching layer.

Imagine an app that presents cooking recipes. You could continue to use it for your favorite recipes even without network !

Taking control over web requests

The first step to be able to cache visited pages is to know when a page is requested. We also need to get the content of the page. Finally we need to be able to return this content when network is down.

To do so, I used a wonderful feature of the Webview (introduced with Windows 8.1): navigateToLocalStreamUri.

Using this function, we can simulate local browsing and provide a C# class that will be called by the Webview each time it needs to download a new page or content (meaning .js, .css, etc.):

WAT.options.webView.navigateToLocalStreamUri(contentUri, uriResolver);

The C# class must implement a method called UriToStreamAsync:

public IAsyncOperation<IInputStream> UriToStreamAsync(Uri uri)
{
    var customUri = GetCustomUri(uri);

    return DownloadOnlineData(customUri).AsAsyncOperation();
}

The DownloadOnlineData is an async method that have a hashtable to determine a relation between and URI and a content saved on disk. When online, this method uses an HttpClient object to download content then it saves it locally and returns the result to the webview (this is a pretty complex method so I am going to discuss part of I right after):

async Task<IInputStream> DownloadOnlineData(Uri uri)
{
    bool needToDownload;

    if (hashTable == null)
    {
        hashTable = await Utilities.RestoreAsync<Dictionary<string, OfflineEntry>>(SerializationFile);
        if (hashTable == null)
        {
            hashTable = new Dictionary<string, OfflineEntry>();
        }
        else
        {
            // Set all files as ready
            foreach (var e in hashTable.Values)
            {
                e.ReadyEvent.Set();
            }
        }
    }

    var entry = GetLocalName(uri, out needToDownload);

    if (entry == null)
    {
        var offlineUri = new Uri(DefaultOfflineUri);
        var defaultFile = await StorageFile.GetFileFromApplicationUriAsync(offlineUri);
        return await defaultFile.OpenAsync(FileAccessMode.Read);
    }

    var workingFolder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(
"workingFolder", CreationCollisionOption.OpenIfExists); try { StorageFile file; if (needToDownload) // First time we see this file, so download it { if (!IsInternetAvailable) { return null; } using (var client = new HttpClient()) { client.DefaultRequestHeaders.Accept.TryParseAdd("text/html, application/xhtml+xml, */*"); client.DefaultRequestHeaders.UserAgent.TryParseAdd(UserAgent); var response = await client.GetAsync(uri); entry.SetExtension(response, uri); var dataBuffer = await response.Content.ReadAsBufferAsync(); file = await workingFolder.CreateFileAsync(entry.Path + entry.Extension, CreationCollisionOption.ReplaceExisting); if (entry.Extension == ".html" || entry.Extension == ".js") // need to fix global references for .html { DataReader dataReader = DataReader.FromBuffer(dataBuffer); var dataString = dataReader.ReadString(dataBuffer.Length); var content = Utilities.RemoveAllGlobalReferences(dataString, uri, entry.Extension == ".js"); if (entry.Extension == ".html") { content = Utilities.InjectScripts(content, uri, ScriptsToInject); } await FileIO.WriteTextAsync(file, content); } else { await FileIO.WriteBufferAsync(file, dataBuffer); } // Signal file is ready entry.ReadyEvent.Set(); } } else { await Task.Factory.StartNew(() => { entry.ReadyEvent.WaitOne(); // Wait for the file to be ready }); file = await workingFolder.GetFileAsync(entry.Path + entry.Extension); } return await file.OpenAsync(FileAccessMode.Read); } catch (Exception ex) { Debug.WriteLine(ex.Message); throw; } }

Saving the cache metadata

You can note that the first step involves a hashtable used to get the cache metadata. A cache entry is defined by this object:

[DataContract]
public sealed class OfflineEntry
{
    private ManualResetEvent readyEvent;

    [DataMember]
    public string Path { get; set; }
    
    [DataMember]
    public string Key { get; set; }

    [DataMember]
    public string Headers { get; set; }

    [DataMember]
    public string Extension { get; set; }

    internal ManualResetEvent ReadyEvent
    {
        get { return readyEvent ?? (readyEvent = new ManualResetEvent(false)); }
    }
 }

Nothing special except the ManualResetEvent. We will come back to this point later.

Downloading the content

This part is pretty straightforward because I use an HttpClient object:

using (var client = new HttpClient())
{
    client.DefaultRequestHeaders.Accept.TryParseAdd("text/html, application/xhtml+xml, */*");
    client.DefaultRequestHeaders.UserAgent.TryParseAdd(UserAgent);
    
    var response = await client.GetAsync(uri);
    entry.SetExtension(response, uri);

    var dataBuffer = await response.Content.ReadAsBufferAsync();
}

Moving for global to local references

The next step we have to cover is the removal of global references. Actually if the Webview finds a global reference (http:// or https:// or //) it will not ask our C# class. So we need to take care of that point and remove all global references.

This job is done by the Utilities.RemoveAllGlobalReferences method which using regular expressions try to replace all global references by a kind of local reference. This means that a http://www.bing.com will become http____www.bing.com

And then thanks to GetLocalName method the reverse operation is done during loading:

private Uri GetCustomUri(Uri uri)
{
    string host = uri.Host;
    int delimiter = host.LastIndexOf('_');
    string encodedContentId = host.Substring(delimiter + 1);
    IBuffer buffer = CryptographicBuffer.DecodeFromHexString(encodedContentId);

    string contentIdentifier = CryptographicBuffer.ConvertBinaryToString(BinaryStringEncoding.Utf8, buffer);
    string relativePath = System.Net.WebUtility.UrlDecode(uri.PathAndQuery);

    if (relativePath.Contains("http___"))
    {
        var index = relativePath.LastIndexOf("http___", StringComparison.Ordinal);

        relativePath = relativePath.Substring(index);

        return new Uri(relativePath.Replace("http___", "http://"));
    }

    if (relativePath.Contains("https___"))
    {
        var index = relativePath.LastIndexOf("https___", StringComparison.Ordinal);

        relativePath = relativePath.Substring(index);

        return new Uri(relativePath.Replace("https___", "https://"));
    }

    if (contentIdentifier.EndsWith("/") && relativePath.StartsWith("/"))
    {
        contentIdentifier = contentIdentifier.Substring(0, contentIdentifier.Length - 1);
    }

    if (Root == null)
    {
        Root = contentIdentifier;

    }

    if (relativePath.StartsWith("/") && relativePath.Length > 1)
    {
        return new Uri(Root + relativePath);
    }

    return new Uri(contentIdentifier + relativePath);
}

Handling multi-threading

We saw that OfflineEntry comes with a reset event. This event is used to signal that an operation is in progress on this specific entry. Then the Webview should wait if it requests this specific entry:

await Task.Factory.StartNew(() =>
{
    entry.ReadyEvent.WaitOne(); // Wait for the file to be ready
});
file = await workingFolder.GetFileAsync(entry.Path + entry.Extension);

The case of XHR

Once all the web requests done by the Webview are handled, we still need to take in account XmlHttpRequest (XHR). This object allows JavaScript to make a web request directly so this specific request will not be transmitted to our C# class.

The idea to handle this point is simple: let’s write our own XHR object!

JavaScript can communicate with the Webview’s host using a simple asynchronous messaging system. The JavaScript code calls window.external.notify to pass a string to the host. The host can then respond using webView.invokeScriptAsync(functionToCall, responseString)

So here is the plan:

  • We create a JavaScript wrapper for XHR
  • Inside this wrapper, we transmit all requests to the host
  • The host effectively does the request and save the result if online. If offline it uses the saved results to return a value to the caller

First of all, we need to load a script file and inject it into the content return to Webview. This is done by the Utilities.InjectScripts method. This method looks for the <head> tag and injects our script just before (To be sure to be launched before ANY others scripts)

static public string InjectScripts(string source, Uri uri, string scriptsToInject)
{
    // Inject script
    if (source.IndexOf("<head") != -1)
    {
        source = source.Replace("<head", "\r\n<script>\r\n" + scriptsToInject + "\r\n</script><head");
    }
    else if (source.IndexOf("<HEAD") != -1)
    {
        source = source.Replace("<HEAD", "\r\n<script>\r\n" + scriptsToInject + "\r\n</script><head");
    }

    source = source.Replace("####URL####", uri.Scheme + "://" + uri.Host);
    return source;
}

The scripts file are loaded on the WinJS side:

readScript("ms-appx:///template/js/xhr.js").then(function (xhrScript) {
    var page = "";

    if (WAT.config.offline.baseDomainURL && root.indexOf(WAT.config.offline.baseDomainURL) !== -1) {
        page = root.replace(WAT.config.offline.baseDomainURL, "");
        root = WAT.config.offline.baseDomainURL;
    }

    var contentUri = WAT.options.webView.buildLocalStreamUri(root, page);

    uriResolver.scriptsToInject = xhrScript;

    WAT.options.webView.navigateToLocalStreamUri(contentUri, uriResolver);

The injected script for XHR is the following:

(function () {

    if (XMLHttpRequest.__alreadyPatched === undefined) {
        window.XMLHttpRequest = function () {
            this._headers = {};
            this._responseHeaders = "";
            this.withCredentials = false;
        };

        window.XMLHttpRequest.__alreadyPatched = true;

        var root = "####URL####";

        // Storage space
        var globalId = 0;
        var waitingRequests = [];

        // Receptor: function to handle response from host
        window.getXhrCallbacks = function (responseString) {
            var response = JSON.parse(responseString);
            var xhr;

            for (var index = 0; index < waitingRequests.length; index++) {
                if (waitingRequests[index].callbackId == response.callbackId) {
                    xhr = waitingRequests[index];
                    waitingRequests.splice(index, 1);
                    break;
                }
            }

            if (responseString === "error") {
                xhr.readyState = 0; //READYSTATE_UNINITIALIZED
                xhr.status = 408; // request timeout
                xhr.statusText = "error";

                if (xhr.onreadystatechange) {
                    xhr.onreadystatechange();
                } else if (xhr.onerror) {
                    xhr.onerror();
                }
                return;
            }

            xhr.response = response.response;
            xhr.responseText = response.responseText;
            xhr.responseXML = response.responseXML;
            xhr.readyState = 4;
            xhr.status = 200;
            xhr.statusText = response.statusText;
            xhr.responseType = response.responseType;
            xhr._responseHeaders = response.responseHeaders;

            if (xhr.onreadystatechange) {
                xhr.onreadystatechange();
            } else if (xhr.onload) {
                xhr.onload();
            }
        };

        // Just block calls
        window.XMLHttpRequest.prototype.abort = function () {
        };

        window.XMLHttpRequest.prototype.send = function (body) {
            waitingRequests.push(this);

            this.callbackId = globalId;

            var order = {
                type: "XHR",
                url: this.__dataUrl,
                method: this.__dataMethod,
                callbackId: globalId++,
                headers: this._headers,
                body: body
}; window.external.notify(JSON.stringify(order)); }; // Get all required information window.XMLHttpRequest.prototype.open = function (method, url, async, user, password) { if (url.indexOf("https://") == -1 && url.indexOf("http://") == -1) { if (url.indexOf("ms-local-stream") != -1) { url = url.replace("ms-local-stream://", ""); var position = url.indexOf("/"); url = url.substring(position); } console.log(url + " was redirected to " + root + url); if (url[0] === '/') { url = root + url; } else { url = root + "/" + url; } } this.__dataUrl = url; this.__dataMethod = method; }; // Headers - Nothing to do right now with headers window.XMLHttpRequest.prototype.setRequestHeader = function (index, header) { this._headers[index] = header; }; window.XMLHttpRequest.prototype.getAllResponseHeaders = function () { return this._responseHeaders; }; window.XMLHttpRequest.prototype.getResponseHeader = function (index) { return this._responseHeaders[index]; }; console.log("XHR hooked"); } })();

The main idea is here: We catch the request on the Webview side, transmit a string version of the request to the host and wait for a response.

The host on its side waits for a request and process it.

Intercept: function (order, webView, uriResolver) {
    order.url = order.url.replace("http___", "http://");
    order.url = order.url.replace("https___", "https://");

    var index = order.url.lastIndexOf("http");
    if (index > -1) {
        order.url = order.url.substring(index);
    }

    var cacheKey = order.method + order.url + order.body; // generate unique key

    if (uriResolver.isInternetAvailable) { // Strategy here is to always get the latest version when online
        var oReq = new XMLHttpRequest;

        oReq.open(order.method, order.url, true); // Need to be saved for offline access
        oReq.onload = function() {
            var responseHeaders = oReq.getAllResponseHeaders(); // Get headers back

            var response = {
                response: oReq.response,
                responseText: oReq.responseText,
                responseType: oReq.responseType,
                responseXML: oReq.responseXML,
                statusText: oReq.statusText,
                callbackId: order.callbackId,
                responseHeaders: responseHeaders
            };

            var responseString = JSON.stringify(response);

            // Save to cache
            uriResolver.saveToCacheAsync(cacheKey, responseString).done();

            // Send back to webview
            sendDataBackToWebview(webView, "getXhrCallbacks", responseString);
        };

        // Transmit headers
        for (var key in order.headers) {
            oReq.setRequestHeader(key, order.headers[key]);
        }

        oReq.onerror = function(err) {

        };

        oReq.send(order.body);
    } else {
        uriResolver.getFromCacheAsync(cacheKey).then(function(responseString) {
            // Send back to webview
            sendDataBackToWebview(webView, "getXhrCallbacks", responseString);
        });
    }
}

Please note that the WinJS code here uses the cache storage of the C# class to factorize code

Handling unmanageable cases

The code that detect global references cannot be smart enough to detect all kind of JavaScript that generate an URI (for instance, let’s imagine a code that create an URI from another strings: one for the moniker, one for the domain, etc…). So I added another script for pictures: the ImageGuardBand. Its role is simple:

  • It monitors every new element added to the DOM
  • If a picture (IMG) is created it changes its addEventListener function in order to catch the onerror before any other JavaScript code
  • If an onerror event is raised, the code tries to fix the URL
  • If an onerror event is still raised then the code calls the previous onerror handler

The code itself is like this:

(function () {
    var interceptAddEventListener = function (root) {
        var current = root.prototype ? root.prototype.addEventListener : root.addEventListener;

        var customAddEventListener = function (name, func, capture) {

            switch (name) {
                case "errorCustom":
                    current.call(this, "error", func, capture);
                    break;
                case "error":
                    if (!this._savedErrorHandlers) {
                        this._savedErrorHandlers = [];
                    }

                    this._savedErrorHandlers.push(func);
                    break;
                default:
                    current.call(this, name, func, capture);
                    break;
            }

        };

        if (root.prototype) {
            root.prototype.addEventListener = customAddEventListener;
        } else {
            root.addEventListener = customAddEventListener;
        }
    };

    interceptAddEventListener(HTMLImageElement);

    var secureImage = function(img) {
        if (!img._secure) {
            img._secure = true;

            img.addEventListener("errorCustom", function (evt) {
                var src = this.src;

                if (this._alreadyFixed) {
                    window.external.notify(JSON.stringify({ type: "LOG", message: "Image error: " + src }));

                    if (this._savedErrorHandlers) { // We were unable to fix src then call all registered error handlers
                        for (var index = 0; index < this._savedErrorHandlers.length; index++) {
                            this._savedErrorHandlers[index].call(this, evt);
                        }
                    }
                    return; // Prevents infinite loop
                }

                // Check common problems
                src = src.replace("ms-local-stream:http___", "ms-local-stream://");

                this._alreadyFixed = true;
                this.src = src;
            });
        }
    };

    document.addEventListener("DOMNodeInserted", function (evt) {
        var target = evt.target;
        if (target.nodeName == "IMG") {
            secureImage(target);
        }
    });


    document.addEventListener("DOMContentLoaded", function () {
        var imgs = document.querySelectorAll("img");

        for (var index = 0; index < imgs.length;index++){
            secureImage(imgs[index]);
        };
    });
})();

The big picture

As you can see, this is a rather complex process, so here is a picture to try to summarize things up:

WAP1_1_13

Conclusion

Now you can create application using Web Application Template that can works even when offline. WAT 1.1 comes with this feature enabled. So feel free to use it and send us feedbacks!