Announcing 0.2.0-alpha2 preview of Windows Azure WebJobs SDK

Announcing 0.2.0-alpha2 preview of Windows Azure WebJobs SDK

Rate This
  • Comments 17

We are releasing an update to Windows Azure WebJobs SDK introduced by Scott Hanselman here.

Download this release

You can download WebJobs SDK in a console application project from the NuGet gallery. You can install or update to these packages through NuGet gallery using the NuGet Package Manager Console, like this:

What is WebJobs SDK?

The WebJobs feature of Windows Azure Web Sites provides an easy way for you to run programs such as services or background tasks in a Web Site. You can upload and run an executable file such as an .exe, .cmd, or .bat file to your web site. You can run these as triggered or continuous WebJobs. Without WebJobs SDK, connecting and running background task requires a lot of complex programming. The SDK provides a framework that lets you write a minimum amount of code to get common tasks done.

The WebJobs SDK has a binding and trigger system which works with Windows Azure Storage Blobs, Queues and Tables. The binding system makes it easy to write code that reads or writes Windows Azure Storage objects. The trigger system calls a function in your code whenever any new data is received in a queue or blob.

The WebJobs SDK includes the following components:

    • NuGet packages. The NuGet packages enable your code to use the WebJobs SDK binding and trigger system with Windows Azure Storage tables, blobs and queues.
    • Dashboard. Part of the WebJobs SDK is already installed in Windows Azure Web Sites and provides rich monitoring and diagnostics for the programs that you write by using the NuGet packages. You don't have to write code to use these monitoring and diagnostics features.

Scenarios for WebJobs SDK

Here are some typical scenarios you can handle more easily with the Windows Azure WebJobs SDK:

    • Image processing or other CPU-intensive work. A common feature of web sites is the ability to upload images or videos. Often you want to manipulate the content after it's uploaded
    • Other long-running tasks that you want to run in a background thread, such as sending emails. Until now you couldn't do this in ASP.NET because IIS would recycle your app if your app was idle for some time. Now with AlwaysOn in Windows Azure Web Sites you can keep the web site from being recycled when the app is idle. AlwaysOn ensures that the site does not go to sleep, which means you can run long-running tasks or services using WebJobs and the WebJobs SDK.
    • Queue processing. A common way for a web frontend to communicate with a backend service is to use queues. When the web site needs to get work done, it pushes a message onto a queue. A backend service pulls messages from the queue and does the work. This is a common producer – consumer pattern.
    • RSS aggregation. If you have a site that maintains a list of RSS feeds, you could pull in all of the articles from the feeds in a background process.
    • File maintenance, such as aggregating or cleaning up log files.  You might have log files being created by several sites and you want to do analysis on them. Or you might want to schedule a task to run weekly to clean up old log files.

Goals of the SDK

    • Provide a way to make it easier to use Windows Azure Storage when doing any background processing work.
      • The SDK makes it easier to consume Azure Storage within your application. You do not have to deal with writing code to read/ write from storage.
    • Provider a rich diagnostics and monitoring experience without having the user write any diagnostics and logging code.

Features of the SDK

Azure Storage

The SDK works with Azure Blobs, Queues and Tables.

Triggers

Functions get executed when a new input is detected on a Queue or a Blob. For example. In the following code ProcessQueue function will be triggered when a new message comes on a queue called “longqueue”. For more details on triggers please see this post.

Code Snippet
  1. publicstaticvoid ProcessQueue([QueueInput("longqueue")] string output)
  2. {
  3.     Console.WriteLine(output);
  4.  
  5. }

 

Bindings

The SDK supports binding to provides model binding between C# primitive types and Azure storage like Blobs, Tables, and Queues. This makes it easy for a developer to read/ write from Blobs, Tables and Queues as they do not have to learn about the code around reading/ writing from Azure Storage.

  • Convenience. You can pick the type that’s most useful for you to consume and the WebJobs SDK will take care of the glue code. If you’re doing string operations on a blob, you can bind directly to TextReader/TextWriter, rather than worry about how to convert to a TextWriter.
  • Flushing and Closing: The WebJobs SDK will automatically flush and close outstanding outputs.
  • Unit testability. The SDK makes it possible to unit test your code since you can mock primitive types like TextWriter rather than ICloudBlob.
  • Diagnostics.  Model binding works with the dashboard to give you real time diagnostics on your parameter usage.

The following Bindings are currently supported: Stream, TextReader/Writer, and String. You can add support for binding to your custom types and other types from the Storage SDK as well.

For more details on how Bindings work against Azure Storage, please read Blobs, Queues and Tables

Hosting

A JobHost is an execution container which knows what all functions do you have in your program. A JobHost object (which lives in Microsoft.WindowsAzure.Jobs.Host ) reads the bindings, listens on the triggers, and invokes the functions. In the following example, you create an instance of JobHost and call RunAndBlock(), which will cause the JobHost to listen for any triggers on any functions that you define in this Host.

Code Snippet
  1. staticvoid Main(string[] args)
  2.         {
  3.             JobHost host = newJobHost();
  4.             host.RunAndBlock();
  5.  
  6.         }

Dashboard for monitoring WebJobs.

As WebJobs (written in any language and of any type) execute, you can monitor them in real time. You can see their state (Running, Stopped, Successfully completed), last run time and the logs of a particular execution. The following screenshot shows you a view of all WebJobs running in your Website.

image

When you write a WebJob using the SDK, you get diagnostics and monitoring experience for the functions in your program. For example, let’s say that you have an Image processing WebJob called “ImageResizeAndWaterMark” that has the following flow.

When a user uploads an image to a Blob container called “images1-input”, the SDK will trigger WaterMark function. Watermark will process the image and write to “images2-input” container which will trigger the Resize function. Resize function will resize the image and write it to “images2-output” Blob container. The following code shows the WebJob described above. For a full working sample, please see the sample here 

image

When you run the WebJob in Azure, you can view the WebJobs Dashboard by clicking the logs link of the “ImageResizeAndWaterMark” in the WEBJOBS tab of Windows Azure Websites portal.

image

Since the Dashboard is a SiteExtension you can access it by going to the url:  https://mysite.scm.azurewebsites.net/azurejobs.  You will need your deployment credentials to access the SiteExtension. For more information on accessing Site Extension, see the documentation on the Kudu project https://github.com/projectkudu/kudu/wiki/Accessing-the-kudu-service

Function execution details

When you are monitoring a particular execution of this “ImageResizeAndWaterMark” WebJob, you can view invocation details about the functions in the program such as:

    • What are the parameters of this function?
    • How much time did it take for the function to execute.
    • How much time did it take to read from Blob and how many bytes were read/ written.

image

Invoke & Replay

In the above example if the WaterMark function fails for some reason, you can upload a new image and Replay WaterMark function, which will trigger the execution chain and call Resize function as well. This is useful to diagnose and debug an issue when you have a complicated graph for chaining functions together. You can also Invoke a function from the dashboard.

Causality of functions

In the above example, we know that when the WaterMark function writes to a Blob, it will trigger the Resize function. The dashboard will show this causality between functions. If you have chained lots of functions which will get triggered as new inputs are detected then it can be useful to see this causality graph.

Search Blobs

You can click on Search for a Blob and get information on what happened to that Blob. For example, in the case of the ImageResizeAndWaterMark, the Blob was written because the WaterMark function got executed. For more details on Search Blobs see this post.

Samples

Samples for WebJobs SDK can be found at https://aspnet.codeplex.com/SourceControl/latest#Samples/AzureWebJobs/ReadMe.txt

    • You can find samples on how to use triggers and bindings for Blobs, Tables and Queues.
    • There is a sample called PhluffySuffy which is an Image processing Website where a customer can upload pictures which will trigger a function to process those pictures from Blob storage.

Documentation

Deploying WebJobs with SDK

If you don't want to use the WebJobs portal page to upload your scripts, you can use FTP, git, or Web Deploy. For more information, see How to deploy Windows Azure WebJobs and Git deploying a .NET console app to Azure using WebJobs

If you want to deploy your WebJobs along with your Websites, check out the following Visual Studio extension.

Known Issues from 0.1.0-alpha1 to 0.2.0-alpha2

Dashboard will only work for WebJobs deployed with 0.2.0-alpha2

If you had a WebJob deployed with 0.1.0-alpha1 of SDK and, if you access the dashboard to see the logs for the WebJob, then you will see a warning about “Host not running”. This happens because as part of this release a newer version of the dashboard gets deployed to all Azure Websites. The new dashboard has some protocol changes which are not compatible with 0.1.0-alpha1. To work around this error, please update your WebJob to use 0.2.0-alpha2 NuGet package and redeploy your WebJob.

 

image

Give feedback and get help

The WebJobs feature of Windows Azure Web Sites and the Windows Azure WebJobs SDK are in preview and not formally supported. Feedback will be considered in changes we make to future versions.

If you have questions that are not directly related to the tutorial, you can post them to the Windows Azure forum, the ASP.NET forum, or StackOverflow.com. Use #AzureWebJobs for Twitter and the tag Azure-WebJobsSDK for StackOverflow.

Leave a Comment
  • Please add 8 and 2 and type the answer here:
  • Post
  • Is there a way to ensure only one copy of a web job runs if there are multiple web site instances?

  • @Brian, you cannot restrict a job to have only one running instance. However, if you have multiple instances of the same job and you are listening for queue messages, only one instance of the job will pick a specific queue message. The same applies to blobs.

  • Hello all, great job with WebJobs. There's any possibility a WebJobs on premise? Tks!

  • @Brian there is some support for it. Please see github.com/.../Web-jobs

    There is no User Interface to set it yet in the Azure portal. As Victor mentioned if you are using Queues, then only instance of the WebJob will pick up the Queue message

  • @Luiz I can take this feedback back to the team.

  • Hi.

    I understand the purpose of the sdk, but I would like to know if something like the following is in line with the design -- especially the scheduler bit:

    class Program

    {

    static void Main(string[] args)

    {

    var host = new JobHost();

    // Will run the queue worker trigger on background thread

    host.RunOnBackgroundThread();

    // Will run the scheduler in the main thread

    host.Call(typeof(Workers).GetMethod("Scheduler"));

    }

    }

    public class Workers

    {

    [NoAutomaticTrigger]

    public static void Scheduler(IBinder binder)

    {

    while (true)

    {

    // Do something on a regular basis

    //

    Thread.Sleep(5000);

    }

    }

    public static void QueueWorker(

    [QueueInput("workerqueue")] WorkItem workItem,

    IBinder binder)

    {

    // Do something when triggered

    //

    }

    }

    If there are better ways to do it, please advise.

    Thx

  • @lnaie: A better approach is to use "RunAndBlock" instead of "RunOnBackgroundThread". That way, you don't need the "Scheduler" function because RunAndBlock will keep the process alive.

  • I've been playing with this a solution. I really like webjobs so far. One behavior I noticed though: webjobs seem to start back up if they were interrupted mid process.

    It really became evident when debugging it on a local machine and stopping several debug sessions in the middle of the job. The next time I ran a debug session, the webjob started back up saying it received a message in the queue, even though the queue was empty.

    Is there a way to see if a webjob is pending and will restart on next run via .net library?

  • You can check the status of the WebJob in the dashboard. In your case since you were debugging, you might have stopped the WebJob midway through the processing so the SDK would have put the message back on the Queue. The next time when you started debugging since the Queue now has a message it will trigger the function again.

  • If you are running tests, you can call CloudQueue clear method to clear the queue before you start the test.

  • Ah yes, it appears that the SDK puts the message back on the queue after a while to retry them. Is there a way to control this behavior?

  • Currently no. the reason the SDK puts the message back in the Queue is because if the WebJob stopped in the middle or processing, the message is put back on the queue since the processing did not finish so when the webjobs comes back up next time, the function will be triggered and the processing can happen again.

  • Try this: blog.smarx.com/.../deleting-windows-azure-queue-messages-handling-exceptions

  • If there is a binding error, and the job fails then the message is deleted from the queue. Clearly it shouldn't be dequeued if the job fails, the invocation log shows the job failing, but the message is deleted from the queue.

    I am getting this error and the message is deleted, how do I report a fault?

    Error while binding parameter #0 'TheClassLibrary.Certificate Certificate':Binding parameters to complex objects (such as 'Certificate') uses Json.NET serialization.

    1. Bind the parameter type as 'string' instead of 'Certificate' to get the raw values and avoid JSON deserialization, or

    2. Change the queue payload to be valid json. The JSON parser failed: Error parsing boolean value. Path '', line 0, position 0.

  • @John M, Thank you for reporting this issue. We have opened an issue to track this.

Page 1 of 2 (17 items) 12