Luca Bolognese's WebLog : .NET Futures

My PDC session is online - “Future directions for C# and Visual Basic”

lucabol — Mon, 23 Nov 2009 21:17:44 GMT

In case you are training to play the part of a first generation Italian immigrant in a Broadway show or if you want to understand why I never short msft stock: http://microsoftpdc.com/Sessions/FT11

Luca at PDC 2009 next week

lucabol — Fri, 13 Nov 2009 18:49:18 GMT

I’ll be in Los Angeles next week for PDC 2009. My session is called “Future Directions for C# and Visual Basic” and it comes on Tuesday immediately after the first keynote.

I’m planning on spending the first half of the session talking about the biggest trends influencing C# and VB. The second part will be about more future looking features we are playing with these days.

My deck has just two slides, as always. It’s all about watching me typing code on stage, hoping that I make crazy errors …

There are several other interesting language related sessions at PDC. Here is a list of them (please notice that the times might change). Also feel free to drop by the languages booth and chat. My team and I will be there most of the time.

Day	Time	Title	Presenter	Room
Tues 11/17	11:00 - 12:00	Future Directions for C# and Visual Basic	Luca Bolognese	Hall F
Tues 11/17	11:00 - 12:00	Accelerated Windows Application Development with Microsoft Visual C++ 2010	Boris Jabes	408B
Tues 11/17	12:30 - 13:15	Dynamic Binding in C# 4	Mads Torgersen	Hall F
Tues 11/17	12:30 - 13:15	Using Dynamic Languages to Build Scriptable Applications	Dino Viehland	403AB
Tues 11/17	13:30 - 14:30	C++ Forever: Interactive Applications in the Age of Manycore	Rick Molloy	515B
Tues 11/17	16:30 - 17:30	Manycore and the Microsoft .NET Framework 4: A Match Made in Microsoft Visual Studio 2010	Stephen Toub	502A
Tues 11/17	16:30 - 17:30	Code Contracts and Pex: Power Charge Your Assertions and Unit Tests	Mike Barnett, Nikolai Tillmann	408A
Wed 11/18	11:30 - 12:30	Microsoft Perspectives on the Future of Programming	Panel	Petree Hall C
Wed 11/18	12:00 - 13:00	F# for Architects – Hitting the Sweet Spot	Chris Smith	VS and .NET Pavillion
Wed 11/18	13:00 - 13:45	Code Like the Wind with Microsoft Visual Basic 2010	Lucian Wischik	Petree Hall D
Wed 11/18	13:00 - 13:45	Future of Garbage Collection	Patrick Dussud	Petree Hall C
Wed 11/18	13:00 - 13:45	Microsoft Project Code Name “M”: The Data and Modeling Language	Don Box, Jeff Pinkston	408A
Thu 11/19	08:30 - 09:30	PLINQ: LINQ, but Faster!	Ed Essey, Igor Ostrovsky	515A
Thu 11/19	10:00 - 11:00	A Lap around Microsoft Visual Studio 2010 for the Visual Basic Developer	Lisa Feigenbaum	VS and .NET Pavillion
Thu 11/19	10:00 - 11:00	Axum: A .NET Language for Safe and Scalable Concurrency	Niklas Gustafsson	515A
Thu 11/19	11:30 - 12:30	F# for Parallel and Asynchronous Programming	Luke Hoban	515A
Thu 11/19	12:45 - 13:30	Microsoft Visual C# IDE Tips and Tricks	DJ Park	Petree Hall D
Thu 11/19	12:45 - 13:30	Microsoft Visual Basic IDE Tips and Tricks	Dustin Campbell	Petree Hall C

An Async Html cache – part II – Testing the cache

lucabol — Fri, 08 May 2009 18:34:00 GMT

Other posts:

Part I – Writing the cache

Let’s try out our little cache. First I want to write a synchronous version of it as a baseline.

    Private Shared Sub TestSync(ByVal sites() As String, ByVal sitesToDownload As Integer, ByVal howLong As Integer)
        Dim syncCache As New Dictionary(Of String, String)
        Dim count = sites.Count()
        Dim url1 = "http://moneycentral.msn.com/investor/invsub/results/statemnt.aspx?Symbol="

        For i = 0 To sitesToDownload - 1
            Dim html As String = ""
            Dim url = url1 & sites(i Mod count)
            If Not syncCache.TryGetValue(url, html) Then
                html = LoadWebPage(url)
                syncCache(url) = html
            End If
            DoWork(html, howLong)
        Next
    End Sub

This is a loop that loads webpages in the cache if they are not already there. sites is a list of tickers used to compose the urls; sitesToDownload is the total number of sites to download, so that a single url can be loaded multiple times; howLong represents the work to be done on each loaded page.

In this version the cache is simply a Dictionary and there is no parallelism. The two bold lines is where the cache is managed.

DoWork is this.

    Public Shared Sub DoWork(ByVal html As String, ByVal howLong As Integer)
        Thread.Sleep(howLong)
    End Sub

Let’s take a look at the asynchronous version.

    Private Shared Sub TestAsync(ByVal sites() As String, ByVal sitesToDownload As Integer, ByVal howLong As Integer)
        Dim htmlCache As New HtmlCache
        Dim count = sites.Count()
        Dim url = "http://moneycentral.msn.com/investor/invsub/results/statemnt.aspx?Symbol="
        Using ce = New CountdownEvent(sitesToDownload)
            For i = 1 To sitesToDownload
                htmlCache.GetHtmlAsync(
                    url & sites(i Mod count),
                    Sub(s)
                        DoWork(s, howLong)
                        ce.Signal()
                    End Sub)
            Next
            ce.Wait()
        End Using

There are several points worth making on this:

The lambda used as second parameter for GetHtmlAsync is invoked on a different thread whenever the html has been retrieved (which could be immediately if the cache has downloaded the url before)
CountDownEvent allows a thread to wait for a certain number of signals to be sent. The waiting happens on the main thread in the ce.Wait() instruction. The triggering of the event happens in the lambda described in the point above (the ce.Signal() instruction)

This is the driver for the overall testing.

    Private Shared Sub TestPerf(ByVal s As String, ByVal a As Action, ByVal iterations As Integer)
        Dim clock As New Stopwatch

        clock.Start()
        For i = 1 To iterations
            a()
        Next
        clock.Stop()
        Dim ts = clock.Elapsed
        Dim elapsedTime = String.Format(s & ": {0:00}:{1:00}:{2:00}.{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10)
        Console.WriteLine(elapsedTime, "RunTime")
    End Sub

There is not much to say about it. Start the clock, perform a bunch of iterations of the passed lambda, stop the clock, print out performance.

And finally the main method. Note that all the adjustable parameters are factored out before the calls to TestPerf.

    Public Shared Sub Main()
        Dim tickers = New String() {"mmm", "aos", "shlm", "cas", "abt", "anf", "abm", "akr", "acet", "afl", "agl", "adc", "apd",
                                           "ayr", "alsk", "ain", "axb", "are", "ale", "ab", "all"}

        Dim sitesToDownload = 50
        Dim workToDoOnEachUrlInMilliSec = 20
        Dim perfIterations = 5

        TestPerf("Async", Sub() TestAsync(tickers, sitesToDownload, workToDoOnEachUrlInMilliSec), perfIterations)
        TestPerf("Sync", Sub() TestSync(tickers, sitesToDownload, workToDoOnEachUrlInMilliSec), perfIterations)
    End Sub

Feel free to change (tickers, sitesToDownload, workToDoOnEachUrlInMilliSec, perfIterations). Depending on the ratios between these parameters and the number of cores on your machine, you’re going to see different results. Which highlights the fact that parallelizing your algorithms can yield performance gains or not depending on both software and hardware considerations. I get ~3X improvement on my box. I attached the full source file for your amusement.

An Async Html cache – Part I - Writing the cache

lucabol — Tue, 28 Apr 2009 02:57:00 GMT

Other posts:

Part II - Testing the cache

In the process of converting a financial VBA Excel Addin to .NET (more on that in later posts), I found myself in dire need of a HTML cache that can be called from multiple threads without blocking them. Visualize it as a glorified dictionary where each entry is (url, cachedHtml). The only difference is that when you get the page, you pass a callback to be invoked when the html has been loaded (which could be immediately if the html had already been retrieved by someone else).

In essence, I want this:

    Public Sub GetHtmlAsync(ByVal url As String, ByVal callback As Action(Of String))

I’m not a big expert in the .Net Parallel Extensions, but I’ve got help. Stephen Toub helped so much with this that he could have blogged about it himself. And, by the way, this code runs on Visual Studio 2010, which we haven’t shipped yet. I believe with some modifications, it can be run in 2008 + .Net Parallel Extensions CTP, but you’ll have to change a bunch of names.

In any case, here it comes. First, let’s add some imports.

Imports System.Collections.Concurrent
Imports System.Threading.Tasks
Imports System.Threading
Imports System.Net

Then, let’s define an asynchronous cache.

Public Class AsyncCache(Of TKey, TValue)

This thing needs to store the (url, html) pairs somewhere and, luckily enough, there is an handy ConcurrentDictionary that I can use. Also the cache needs to know how to load a TValue given a TKey. In ‘programmingese’, that means.

    Private _loader As Func(Of TKey, TValue)
    Private _map As New ConcurrentDictionary(Of TKey, Task(Of TValue))

I’ll need a way to create it.

    Public Sub New(ByVal l As Func(Of TKey, TValue))
        _loader = l
    End Sub

Notice in the above code the use of the Task class for my dictionary instead of TValue. Task is a very good abstraction for “do some work asynchronously and call me when you are done”. It’s easy to initialize and it’s easy to attach callbacks to it. Indeed, this is what we’ll do next:

    Public Sub GetValueAsync(ByVal key As TKey, ByVal callback As Action(Of TValue))

        Dim task As Task(Of TValue) = Nothing
        If Not _map.TryGetValue(key, task) Then
            task = New Task(Of TValue)(Function() _loader(key), TaskCreationOptions.DetachedFromParent)
            If _map.TryAdd(key, task) Then
                task.Start()
            Else
                task.Cancel()
                _map.TryGetValue(key, task)
            End If
        End If

        task.ContinueWith(Sub(t) callback(t.Result))
    End Sub

Wow. Ok, let me explain. This method is divided in two parts. The first part is just a thread safe way to say “give me the task corresponding to this key or, if the task hasn’t been inserted in the cache yet, create it and insert it”. The second part just says “add callback to the list of functions to be called when the task has finished running”.

The first part needs some more explanation. What is TaskCreationOptions.DetachedFromParent? It essentially says that the created task is not going to prevent the parent task from terminating. In essence, the task that created the child task won’t wait for its conclusion. The rest is better explained in comments.

        If Not _map.TryGetValue(key, task) Then ' Is the task in the cache? (Loc. X)
            task = New Task(Of TValue)(Function() _loader(key), TaskCreationOptions.DetachedFromParent) ' No, create it
            If _map.TryAdd(key, task) Then ' Try to add it
                task.Start() ' I succeeded. I’m the one who added this task. I can safely start it.
            Else
                task.Cancel() ' I failed, someone inserted the task after I checked in (Loc. X). Cancel it.
                _map.TryGetValue(key, task) ' And get the one that someone inserted
            End If
        End If

Got it? Well, I admit I trust Stephen that this is what I should do …

I can then create my little HTML Cache by using the above class as in:

Public Class HtmlCache

    Public Sub GetHtmlAsync(ByVal url As String, ByVal callback As Action(Of String))
        _asyncCache.GetValueAsync(url, callback)
    End Sub

    Private Function LoadWebPage(ByVal url As String) As String
        Using client As New WebClient()
            'Test.PrintThread("Downloading on thread {0} ...")
            Return client.DownloadString(url)
        End Using
    End Function

    Private _asyncCache As New AsyncCache(Of String, String)(AddressOf LoadWebPage)

End Class

I have no idea why coloring got disabled when I copy/paste. It doesn’t matter, this is trivial. I just create an AsyncCache and initialize it with a method that knows how to load a web page. I then simply implement GetHtmlAsync by delegating to the underlying GetValueAsync on AsyncCache.

It is somehow bizarre to call Webclient.DownloadString, when the design could be revised to take advantage of its asynchronous version. Maybe I’ll do it in another post. Next time, I’ll write code to use this thing.

Luca presenting F# at PDC 2008

lucabol — Fri, 24 Oct 2008 19:21:14 GMT

Next Thursday at 1.45pm I will present "An introduction to Microsoft F#" at PDC 2008. If you are there, please stop by or drop by the lounge area to chat.

There are a bunch of other great sessions from my team. They are all good (I swear I tried to format John's session correctly, but failed).

TL16: The Future of C#
Speaker: Anders Hejlsberg
Mon 10/27 | 1:45 PM-3:00 PM | Petree Hall CD
Repeated: Wed 10/29 | 3:00 PM-4:15 PM | 502A
TL10: Deep Dive: Dynamic Languages in .NET
Speaker: Jim Hugunin
Mon 10/27 | 3:30PM | 502A
TL11: An Introduction to Microsoft F#
Speaker: Luca Bolognese
Thu 10/30 | 1:45 PM-3:00 PM | 502A
TL12: Future Directions for Microsoft Visual Basic
Speaker: Paul Vick
Tue 10/28 | 5:15 PM-6:30 PM | 406A
TL54: Natural Interop with Silverlight and Office
Alex Turner
Tue 10/28 | 12:45 PM-1:30 PM | 404A
TL57: A Panel on the Future of Programming Languages
G. Bracha, D.Crockford, A. Hejlsberg, E. Meijer, W.Schulte, J. Siek
Wed 10/29 | 10:30 AM-11:45 AM | 403AB

TL44 IronRuby: The Right Language for the Right Job

l Speaker: John Lam
l Mon 10/27, 5:15PM, 515B

ObjectSpaces early days ...

lucabol — Thu, 07 Jun 2007 17:40:00 GMT

Matt has a very good post on the history of object relational framework inside Microsoft. He and I started the whole ObjectSpaces thing together very many years ago (about six and a half). I thought I should add my two cents.

You might be wondering how a project starts inside Microsoft (or you might not). In this case, I was hired in the ADO.NET team to build an object relational framework. That was my assignment. The problem was that I had no dev or tester to work with. I also had just arrived in the US from Italy and my English was terrible (that hasn’t changed much). My only qualifications for the job were that I was incredibly passionate about the topic; I had built a couple of small ones in my spare time and have used many more in various projects. But still I had no idea on how to make things happen in this big company. Hell, I could barely understand what people were talking about at lunch.

Anyhow, my boss told me that there was a guy who was relatively free, but I had to come up with an idea, convince him that the idea was a good one and that he should go ahead and prototype it. That guy was Matt Warren, already one of the best developers in the SQL team and the dev lead of plenty of our .NET stack.

The first time I entered his office and started talking, he immediately told me: “Not now, I’m debugging”. A good start … But after that, we hit it off immediately. We talked about what an OR framework is, what it should be, how we could innovate in the space and so on. After a while, Matt started working on the very first ObjectSpaces prototype. Our modus operandi was peculiar. I would go to Matt’s office and we would discuss a particular feature or scenario. He would tell me: “Ok, we are on the same page, I’ll see you tomorrow”. The next day the feature would be implemented. If he told me: “It is going to take me 5 days to code it” I knew he didn’t like the feature and I had some more convincing to do. Sometime we would agree, sometime we wouldn’t. Sometime I came up with staff to implement, sometimes Matt did. We figured out a lot of stuff in those days.

Also I learned how to work with different categories of programmers. If you work with a bad programmer you have to tell him how to implement something. If you work with a good programmer you have to tell him what you want the feature to look like. If you work with an excellent programmer, you just have to tell him what your final goal is. I quickly realized that the latter was the most productive strategy with Matt. I had just to convince him of the business need for something. Implementing it ended up to be a detail.

That initial prototype grew to become ObjectSpaces (it was called Cheops initially). The team grew to be about 25 people, growing and stretching me personally. We went through two painful rounds of unifications with bigger products. In the end ObjectSpaces was cut. Matt moved to the C# team to work on LINQ. Dinesh and I followed after a short while. The whole ObjectSpaces team scattered in various places around the company. The object relational thing started again in the C# team as part of the LINQ project. This time around we also had a compiler to play with. Plus Anders was on board to sprinkle his design magic over the whole thing.

As Roland would say, the world has moved on. Matt is now a big shot architect and I lead a team of amazingly smart individuals (they are forced to act as if my words make sense, imagine that …). We are shipping a game changing product in LINQ and a wonderful object relational framework in LINQ to SQL. Things turned out for the best (even if five years too late).

Still it is a pleasure to think back at those early days (and nights) of ‘figuring out stuff’ six years ago. A lot of that ‘stuff’ is inside our products today, and that is something to be proud of.

My LINQ Overview session at TechEd will be live on the Internet ...

lucabol — Mon, 04 Jun 2007 19:45:00 GMT

... at least they told me so. It is at 8.30am Eastern time. I don't know the exact link, but it will be somewhere here: http://www.microsoft.com/events/teched2007/default.mspx.