Dynamic Language Runtime 0.9 and IronPython 2.0

nhodge — Tue, 16 Dec 2008 05:01:00 GMT

Over on codeplex, the smart team behind the DLR (Dynamic Language Runtime) and IronPython have released code. IronPython 2.0 is built on top of the DLR. This means that IronPython can run in Silverlight. Which tickles me.

DLR 0.9:

implements a language on .NET using the DLR
adds dynamic features to their existing language like C#’s ‘dynamic’ (http://code.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=csharpfuture&DownloadId=3550 )
adds scripting to their applications.
creatse .NET libraries with dynamic objects

IronPython 2.0:

An MSI installer for Windows platforms which includes parts of the CPython 2.5 standard library
IronPython assemblies targeting Silverlight and tools such as Chiron to improve the Silverlight dynamic development experience
The addition of more C-based standard modules such as cmath and _winreg
Significant improvements in importing compatibility and features
Distribution of IronPython under the Microsoft Public License which has been approved by OSI
Performance improvements. On that note, a new Wiki page has been created for IronPython performance reports - see http://www.codeplex.com/IronPython/Wiki/View.aspx?title=IronPython%20Performance
Over 500 bugs have been closed in 2.0. 453 of these were reported on CodePlex
Support for precompilation of Python source files into a single dll

IronPython + F# + Parallel + Async = A Kittehz Brekfst

nhodge — Wed, 12 Nov 2008 13:19:00 GMT

more animals

As per the image above, cats and threads do no mix. Python and threads may not mix, either.

Whilst thread-safety, dead-locks, queues and other nasties are nasty, Microsoft sees the world of n-cores as a software problem to be solved, not ignored. Both at a low-level and a high-level, well constructed and debugged Parallel libraries are appearing to make the splitting of tasks easier.

IronPython implements the Python language on the DLR and subsequently on the CLR, but this does not automagically provide IronPython with threading and parallelism. Nor could I suggest that IronPython is the silver-bullet for clean parallelism with Python. Various projects such as ParallelPython (including cluster support), Stackless Python and the recent python-multiprocessing package are appearing to move CPython into today’s world of n-cores.

IronPython does have kissing-cousin languages like F#. From F#, and the Microsoft Parallel Extensions Jun08 CTP – a parallel library is a .dll away. Step 1: create a .dll from Luca’s demonstration from PDC2008 specifically the MathLibrary/StockAnalyzer. Combining both the async/parallel versions of the F# code with the single-threaded version was easy.

From IronPython:

   1:  import clr

   2:  clr.AddReferenceToFile("MathLibrary.dll")

   3:  import StockLibrary as StockLibrary

   4:  import sys, time

   5:  tickers = ['msft','adbe','aapl','ebay']

Line 1 imports the CLR so your code can import a .NET DLL (clr.AddReferenceToFile). The namespace is StockLibrary. Now methods within this DLL can be called. The fifth line sets up the four test Nasdaq codes we are analyzing.

   6:  analysersA = StockLibrary.StockAnalyzerParallel.GetAnalyzers (tickers, 365)

   7:  print ">> starting async"

   8:  start = time.clock()

   9:  for a in analysersA:

   10:      print a.Return

   11:  print "> seconds taken: ", time.clock() - start

   12:  print ">> completed"

The analysersA instantiates the async/parallel version of the F# methods. Here I have wrapped the code with a simple time.clock() timer

   13:  analysersP = StockLibrary.StockAnalyzer.GetAnalyzers (tickers, 365)

   14:  print ">> starting normal"

   15:  start = time.clock() # get current time in seconds

   16:  for a in analysersP:

   17:      print a.Return

   18:  print "> seconds taken: ", time.clock() - start

   19:  print ">> completed"

The analysersP (p for plain) uses the single-threaded version in the library.

Results:

On General Melchett, my Intel Quadcore, I was able to execute the parallel version in 0.5s vs 11.0s in single-threaded mode. I suggest the 22x speed improvement comes from the Asyc calls to the network methods more than the calculations involved. More tests would be required to confirm the dramatic difference.

In my example I am treating the F# DLL as a simple black box: a list of parameters in and single results out. In the real-world, careful consideration would need to be taken on the immutability of any Python objects/variables passed in; and the resulting factoring of code in F# vs. IronPython.

(this was a simple 1 minute demonstration at the Edge of the Web Conference in Perth expanded into a blog post)

For reference: My version of Luca’s F#: MathLibrary.fs

#light

// ASYNC version
// note: you need to ensure the project has a reference to FSharp.PowerPack to work
// async is difficult, here we change 4 lines, add some curly braces and we have async+parallel

open System.Net
open System.IO

let internal loadPricesAsync ticker = async { // now returns async token
    let url = "http://ichart.finance.yahoo.com/table.csv?s=" + ticker + "&d=10&e=3&f=2008&g=d&a=2&b=13&c=1986&ignore=.csv"

    let req = WebRequest.Create(url)
    let! resp = req.AsyncGetResponse()                      // holds the thread waiting for the network, blocked: so parallelize + asynch
    let stream = resp.GetResponseStream()
    let reader = new StreamReader(stream)
    let! csv = reader.AsyncReadToEnd()                      // holds the thread waiting for the network, blocked: so parallelize + asynch

    let prices =                                        
        csv.Split([|'\n'|])                             
        |> Seq.skip 1
        |> Seq.map ( fun line -> line.Split([|','|]) )    
        |> Seq.filter (fun values -> values |> Seq.length = 7 ) 
        |> Seq.map (fun values -> 
            System.DateTime.Parse(values.[0]),              
            float values.[6] )
    return prices
    }                                                       // added async to let, add a return, look at places for release control, add bang and call Async versions
    
type StockAnalyzerParallel (lprices, days) =
    let prices =
        lprices
        |> Seq.map snd
        |> Seq.take days                                    // chop sequence to days
    static member GetAnalyzers (tickers, days) =
        tickers
        |> Seq.map loadPricesAsync
        |> Async.Parallel                                   // now that loadPrices returns async, parallelize and run; then map
        |> Async.Run                                        // now run in parallel
        |> Seq.map (fun prices -> new StockAnalyzerParallel (prices, days))
    member s.Return =
        let lastPrice = prices |> Seq.nth 0
        let startPrice = prices |> Seq.nth ( days - 1 )
        lastPrice / startPrice - 1.0
    member s.StdDev =
        let logRets =
            prices
            |> Seq.pairwise
            |> Seq.map ( fun (x, y) -> log (x / y))
        let mean = logRets |> Seq.average
        let sqr x = x * x
        let var = logRets |> Seq.average_by (fun r -> sqr (r - mean))
        sqrt var
        
        
        
let internal loadPrices ticker =
    let url = "http://ichart.finance.yahoo.com/table.csv?s=" + ticker + "&d=10&e=3&f=2008&g=d&a=2&b=13&c=1986&ignore=.csv"

    // code is identical to C# version as we are using underlying .Net libraries
    let req = WebRequest.Create(url)
    let resp = req.GetResponse()
    let stream = resp.GetResponseStream()
    let reader = new StreamReader(stream)
    let csv = reader.ReadToEnd()

    let prices =                                        // returns a tuple based on comma
        csv.Split([|'\n'|])                             //note [| syntax passes in .Net array to the string.Split method
        |> Seq.skip 1
        |> Seq.map ( fun line -> line.Split([|','|]) )    // fun define anonymous function/lambda expression
        |> Seq.filter (fun values -> values |> Seq.length = 7 )  // filter out where less than 7 values
        |> Seq.map (fun values -> 
            System.DateTime.Parse(values.[0]),              // get the 0th and 6th column
            float values.[6] )
    prices
    
type StockAnalyzer (lprices, days) =
    let prices =
        lprices
        |> Seq.map snd
        |> Seq.take days // chop sequence to days
    static member GetAnalyzers (tickers, days) =
        tickers
        |> Seq.map loadPrices
        |> Seq.map (fun prices -> new StockAnalyzer (prices, days))
    member s.Return =
        let lastPrice = prices |> Seq.nth 0
        let startPrice = prices |> Seq.nth ( days - 1 )
        lastPrice / startPrice - 1.0
    member s.StdDev =
        let logRets =
            prices
            |> Seq.pairwise // sequence of things, first + second tuple; second + third etc
            |> Seq.map ( fun (x, y) -> log (x / y))
        let mean = logRets |> Seq.average
        let sqr x = x * x
        let var = logRets |> Seq.average_by (fun r -> sqr (r - mean))
        sqrt var

Demos and Links from Edge of the Web, Nov’08 Perth

nhodge — Tue, 11 Nov 2008 06:08:11 GMT

During the 50 minutes of presentation (including at least 9 ICHC images), I demonstrated the following bits and pieces of open source and or free Microsoft bits.

Demo set 1: Web Standards

Live Mesh Mobile : take photo to mesh on laptop

MacOS X 10.5 client available too [not demo’d]

DataModelBrowser to apis.mesh.com; showing REST/JSON/APP [part of Live Framework SDK]
Oomph: Microformats toolkit (source on codeplex.com) incorporating:

LiveWriter; hCard plugin; publish; show IE plugin, same site with Firefox Tail

Demo set 2: PHP, SQLServer 2008 & IIS7

Demo set 3: IronPython

IronPython; ipy hello world winforms
using as aspx as "code behind" and ASP.NET
F# integration with via .NET compiled .dll (calling from IronPython)

Demo set 4: IronRuby

IronRuby; flickr.net example from John Lam’s PDC2008 session
Photoviewer; using local Chiron, browser, ruby in browser doing HTML/DOM

Demo set 5: Silverlight

Silverlight 2 tools for Visual Studio now support Visual Studio Express

Silverlight 2 also supporting Eclipse (see, competition is a good thing)

Silverlight Toolkit, run from chrome-less Google Chrome (contains Microsoft Windows Template Lib)

basketweaving for the mind : ironpython