• Kirill Osenkov

    Get most duplicated strings from a heap dump using ClrMD

    • 0 Comments

    ClrMD is an awesome managed API to inspect managed processes and dump files. To use it, just add a NuGet reference to Microsoft.Diagnostics.Runtime. When loading a dump, be sure to have the mscordacwks.dll from the machine where the dump was taken. Also make sure that your program that uses ClrMD is the same platform (32/64-bit) as the process/dump that you’re inspecting.

    Here’s a sample of getting most duplicated strings out of a dump, an indication that you might need a string cache somewhere. Remember that if creating a dump of a 32-bit process on a 64-bit OS you need to use the 32-bit Task Manager, otherwise the dump will be useless.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using Microsoft.Diagnostics.Runtime;
     
    namespace DumpTools
    {
        class DumpHeapStrings
        {
            static void Main(string[] args)
            {
                using (var dataTarget = DataTarget.LoadCrashDump(@"app.dmp"))
                {
                    var dacLocation = dataTarget.ClrVersions[0].TryGetDacLocation();
                    var runtime = dataTarget.CreateRuntime(dacLocation);
                    var heap = runtime.GetHeap();
                    var objects = heap.EnumerateObjects();
     
                    var stringUsages = new Dictionary<string, long>();
     
                    foreach (var instance in objects.Take(1000000))
                    {
                        var type = heap.GetObjectType(instance);
                        if (type != null && type.IsString)
                        {
                            var size = type.GetSize(instance);
                            var value = (string)type.GetValue(instance);
                            long usages = 0L;
                            if (stringUsages.TryGetValue(value, out usages))
                            {
                                stringUsages[value] = usages + 1;
                            }
                            else
                            {
                                stringUsages[value] = 1;
                            }
                        }
                    }
     
                    var sorted = stringUsages.OrderByDescending(kvp => kvp.Value).Take(100);
                    foreach (var kvp in sorted)
                    {
                        Console.WriteLine(kvp.Value + "\t\t" + kvp.Key);
                    }
                }
            }
        }
    }
    
  • Kirill Osenkov

    sn.exe cheat sheet

    • 1 Comments

    I don’t find sn.exe command line arguments particularly intuitive, and the fact they’re case sensitive doesn’t exactly put you in the pit of success either.

    Also, it turns out there are two kinds of .snk files. One kind is 596 bytes by default and contains a key pair (private and public key). The other kind is 160 bytes typically and only contains the public key.

    I’ve looked at sn.exe /? for long enough to compose this cheat sheet. Hopefully it'll be useful to others too.

    Create a key pair sn –k private.snk
    Extract public key sn –p private.snk public.snk
    View public key and public key token sn –tp public.snk
    View public key and public key token of assembly sn –Tp assembly.dll
    Extract public key from assembly sn –e assembly.dll public.snk
    Verify if assembly is signed and with what key sn –vf assembly.dll
    Show list of assemblies on machine registered for verification skipping sn –Vl
    Register assemblies for verification skipping sn –Vr * (must elevate)
  • Kirill Osenkov

    Creating an instance of an open generic type without specifying a type argument

    • 5 Comments

    My fellow Roslyn tester @vreshetnikov has found not one, but two ways to do the impossible:

    1:

    using System;
     
    class Program
    {
        static void Main()
        {
            var open = Enum.ToObject(typeof(C<>.E), 0);
            Console.WriteLine(open.GetType());
        }
    }
     
    class C<T>
    {
        public enum E { }
    }

    2:

    using System;
     
    class Program
    {
        static void Main()
        {
            Action<C<int>.E> a = M;
            var open = a.Method.GetGenericMethodDefinition().GetParameters()[0].DefaultValue;
            Console.WriteLine(open.GetType());
        }
     
        static void M<T>(C<T>.E e = 0) { }
    }
     
    class C<T>
    {
        public enum E { }
    }
  • Kirill Osenkov

    Saving Regedit Favorites in a .reg file

    • 2 Comments

    Regedit.exe has a useful feature – Favorites menu, you can add an often used registry location to favorites to quickly expand to it in the tree view.

    But did you know that favorites themselves are stored in registry and so you can have a RegeditFavorites.reg that you run to quickly fill the Favorites menu with your most-used defaults? This is really helpful when configuring a new machine.

    To create this file:

    1. Go to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Applets\Regedit\Favorites
    2. File –> Export –> Selected branch, save the .reg file.

    It is very meta that the actual favorites location in the registry is also one of my favorites! Here’s my favorites file:

    Windows Registry Editor Version 5.00

    [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Applets\Regedit\Favorites]
    "HKCU\\Software\\Microsoft\\VisualStudio\\12.0"="Computer\\HKEY_CURRENT_USER\\Software\\Microsoft\\VisualStudio\\12.0"
    "AeDebug"="Computer\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug"
    "DebugApplications"="Computer\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\Windows Error Reporting\\DebugApplications"
    "HKCU\\S\\M\\VS\\12.0\\ReferenceManager"="Computer\\HKEY_CURRENT_USER\\Software\\Microsoft\\VisualStudio\\12.0\\ReferenceManager"
    "EnableVSIPLogging"="Computer\\HKEY_CURRENT_USER\\Software\\Microsoft\\VisualStudio\\12.0\\General"
    "Favorites"="Computer\\HKEY_CURRENT_USER\\Software\\Microsoft\\Windows\\CurrentVersion\\Applets\\Regedit\\Favorites"
    "VSIP InstallDir"="Computer\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\VSIP\\12.0"
    "Open with Notepad2"="Computer\\HKEY_CLASSES_ROOT\\*\\shell\\Notepad2"

  • Kirill Osenkov

    Registry keys to configure Watson and automatically save full dumps locally

    • 1 Comments

    You might have noticed that after installing Visual Studio the behavior of crashing programs changes – instead of the standard crash dialog you get the Visual Studio Just-In-Time Debugger:

    image

    AeDebug

    The behavior of handling crashing programs can be controlled in the registry:

    Windows Registry Editor Version 5.00

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]
    "Debugger"="\"C:\\windows\\system32\\vsjitdebugger.exe\" -p %ld -e %ld"
    "Auto"="1"

    http://technet.microsoft.com/en-us/library/cc939483.aspx

    If Auto is not set or is set to 0, you get to choose whether to attach a debugger or not:

    image

    If Auto is set to 1, the JIT debugger dialog starts without the prompt above. Visual Studio sets the debugger to vsjitdebugger, which in turn lets you start a Visual Studio instance and attach to the crashing process.

    Windows Error Reporting

    Another useful registry branch that deals with watsons is:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting]

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\DebugApplications]
    "w3wp.exe"=dword:00000001
    "aspnet_wp.exe"=dword:00000001
    "devenv.exe"=dword:00000001

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]
    "DumpFolder"=hex(2):25,00,4c,00,4f,00,43,00,41,00,4c,00,41,00,50,00,50,00,44,\
      00,41,00,54,00,41,00,25,00,5c,00,43,00,72,00,61,00,73,00,68,00,44,00,75,00,\
      6d,00,70,00,73,00,00,00
    "DumpCount"=dword:00000005
    "DumpType"=dword:00000001

    Here’s a good MSDN article that describes these options:

    http://msdn.microsoft.com/en-us/library/bb513638.aspx

    To turn it off, set DontShowUI to 1 and set Disabled to 1 to stop sending reports to Microsoft.

    Local Dumps

    LocalDumps is interesting. It configures Watson to automatically save a dump of the crashing process in the folder you specify and keep N latest dumps in that folder:

    image

    This is really useful if the process doesn’t show the Watson dialog, you can still get the dump for it. Just open your %LOCALAPPDATA%\CrashDumps folder and you’ll see the latest N dumps (I’ve set it to 32). DumpType is 1 for minimal, 2 for full.

  • Kirill Osenkov

    Circular assembly references in the .NET framework

    • 3 Comments

    It was a surprise to me when I discovered that a lot of assemblies in the .NET BCL reference each other in a circular manner:

    image

    Cycles in the reference graph are in red. Apparently there is special MSBuild magic used when compiling these assemblies. My guess would be that proper factoring of the assemblies into a DAG without cycles would create too many assemblies and the .NET team opted against that.

    I’ve generated this .dgml file to play with the assembly reference graph. You can open it in Visual Studio, and use the Layout –> Analyzers –> Circular References Analyzer on the graph to highlight the cycles in red like I did in the image above. Find Hubs and Unreferenced Nodes are also interesting.

    Here’s a fragment of the file with just the 4 assemblies:

    <DirectedGraph xmlns="http://schemas.microsoft.com/vs/2009/dgml">
      <Nodes>
        <Node Id="mscorlib" />
        <Node Id="System" />
        <Node Id="System.Core" />
        <Node Id="System.Xml" />
      </Nodes>
      <Links>
        <Link Source="System" Target="mscorlib" />
        <Link Source="System" Target="System.Xml" />
        <Link Source="System.Core" Target="mscorlib" />
        <Link Source="System.Core" Target="System" />
        <Link Source="System.Core" Target="System.Xml" />
        <Link Source="System.Xml" Target="mscorlib" />
        <Link Source="System.Xml" Target="System.Core" />
        <Link Source="System.Xml" Target="System" />
      </Links>
    </DirectedGraph>
  • Kirill Osenkov

    Dump MSBuild properties and items for a project

    • 2 Comments

    Sometimes when you’re reading a .csproj file you encounter properties like $(TargetDir) and wonder what directory does this property evaluate to when the project is compiled. Of course you can use heavy machinery like the MSBuild debugger, but sometimes a simpler tool can suffice. Here’s what I wrote in literally 5 minutes:

    using System;
    using System.IO;
    using Microsoft.Build.Evaluation;
     
    class MSBuildDumper
    {
        static void Main(string[] args)
        {
            if (args.Length != 1)
            {
                Console.WriteLine("Usage: MSBuildDumper <path-to-project.csproj>");
                return;
            }
     
            var projectFilePath = Path.GetFullPath(args[0]);
            var project = new Project(projectFilePath); // add reference to Microsoft.Build.dll to compile
     
            foreach (var property in project.AllEvaluatedProperties)
            {
                Console.WriteLine(" {0}={1}", property.Name, property.EvaluatedValue);
            }
     
            foreach (var item in project.AllEvaluatedItems)
            {
                Console.WriteLine(" {0}: {1}", item.ItemType, item.EvaluatedInclude);
            }
        }
    }

    The API I’m using is called MSBuild evaluation API, it reads MSBuild project files and makes sense of their content. Try running this tool on one of your .csproj project files and be amazed at how much stuff is happening during build evaluation.

  • Kirill Osenkov

    Naming Roslyn concepts

    • 28 Comments

    We have a rather central component in Roslyn and we’re looking how to name it best. I’d like to gather some advice and opinions that could potentially help us find a good name.

    I am intentionally not mentioning what is it called right now. Instead, I’m going to describe it. If you recognize this component from the Roslyn CTPs, please don’t name it in the comments so as not to accidentally drive others towards its current name, which we believe is overused in our domain already.

    Roslyn analyzes code. Code is most often organized into solutions, projects and documents. Primarily we use three types to model source code: Solution, Project and Document. A Solution has a set of Projects, a Project has a set of Documents. These form an immutable tree that captures a snapshot of a source code tree at a moment in time. This data structure only captures the information interesting to the compiler, it’s not an MSBuild or a project system concept. When the user edits the text of a document in the editor, adds a new project in Visual Studio or otherwise modifies code, a new immutable snapshot of the entire Solution is created. This is implemented using efficient immutable data structures that reuse most of the data between the snapshots. This way if an analysis was in progress while the user changed code, the analysis can continue on it’s own snapshot safely knowing that the snapshot will never change. Also we can utilize multiple cores for fast, safe, lock-free analysis.

    Now, the concept we’re looking to name is something that has a CurrentSolution property that points to the latest snapshot of the solution data structure. It can listen to changes in user’s source code, build a new Solution snapshot to reflect those changes, and finally update the CurrentSolution property to point to the latest snapshot. This concept also has an event that our features can listen to to update themselves when the current solution has changed.

    An environment that is hosting Roslyn (say the Visual Studio project system) is responsible for forwarding changes into this component such that it can recalculate the most up-to-date Solution and notify all observers.

    If a Roslyn feature, such as a refactoring, wants to modify the users code (e.g. generate a method), it doesn’t talk directly to the environment. Instead, it describes the changes it wants to apply to the current solution and forms a new, delta (diff) Solution, and hands it off to our component. Our component then takes this delta (changes) and sends it back to the host, asking the host to do whatever is necessary to actually apply these changes (edit a file, checkout a file from TFS, add a reference, etc.) The host then, after applying these changes will pipe the notifications back into our component to complete the circle.

    How do we name this component that is responsible for keeping the CurrentSolution up-to-date and ensures bidirectional communication of changes with the host environment?

    Thanks!

  • Kirill Osenkov

    Collection initializers and Add method with more than one parameter

    • 7 Comments

    I didn’t know this about C# until today. This code compiles:

    using System;
    using System.Collections.Generic;
    
     
    class List : List<Tuple<string, int, bool>>
    {
        public void Add(string s, int i, bool b)
        {
            Add(Tuple.Create(s, i, b));
        }
    
     
        static void Main(string[] args)
        {
            var list = new List
            {
                { "zero", 0, true },
                { "one", 1, false }
            };
     
            foreach (var item in list)
            {
                Console.WriteLine(item.ToString());
            }
        }
    }

    and outputs:

    (zero, 0, True)
    (one, 1, False)

    It turns out, the way collection initializers work they pattern match to expect the Add method on the list object, and it can have more than one or two parameters. This is extremely handy for writing declarative, data-driven unit-tests where you only specify the data for your scenario and the imperative control-flow of the test is abstracted away into a helper method. This allows you to shape the relevant data in a way convenient to be able to clearly read the test scenario.

    P.S. For a second, I admit to having a thought “I bet even Jon Skeet didn’t know about this one”, but then of course I opened C# in Depth 2nd Edition, page 216 and was once again in awe of Jon’s superpowers. Sorry for ever having the slightest of doubt, Jon, won’t happen again!

  • Kirill Osenkov

    Tips on building a modern *silent* desktop PC?

    • 6 Comments

    For many years I’ve been a happy user of completely silent desktop PCs at home. I hate noise from the fans and spinning rust aka HDDs. In fact, I think it’s a shame that most desktop computers still have mechanically moving parts. I guess the majority of users don’t care much about the noise, hence insufficient demand and lack of pressure on OEMs and manufacturers to build silent PCs. Sad, sad, sad.

    Anyway, I’ve previously used Deltatronic GmbH in Germany (http://www.deltatronic.de/en) and they’re pretty good. Check out their tech at http://www.deltatronic.de/en/technology. Fanless power supplies from massive solid metal, high quality CPU heatpipes, graphic cards with passive cooling, special casing for HDDs (now a moot point – viva SSDs!) – a dream come true (well, at least for me).

    The last machine I ordered from them (the one I’m using right now at home to write this blog) is from 2006, and from this date alone you can probably deduce that:

    • their boxes are quite reliable,
    • with their pricing I can’t afford upgrading all too often, and
    • I do really need a modern box (this one is a dual CPU @ 2 GHz, 2 GB RAM, 4.9 Windows Experience Index). I mean it’s been perfectly OK for checking e-mail from home, remoting into work, web browsing, and I totally love the fact that I can’t even tell whether it’s on or not by listening alone. It’s just showing it’s age.

    Unfortunately at work we don’t get silent machines, but the HPs and Dells we’re getting have been pretty quiet as a rule, and so I was forced to get used to them. However at home it’s pretty quiet and I really wouldn’t want to introduce fan noise to spoil the idyll...

    And now we finally come to the point of this blog post: I’ve looked at Deltatronic’s website and the prices for a modern i7 box are... a little steep (€1980 plus I’m guessing shipping it to the States won’t be free either). I’m used to paying this much for the quality and satisfaction I’m getting, but I really haven’t looked around and done any research for the past five years. I’m hopelessly out-of-date. Has there been any progress? Has anyone built or investigated recent silent configurations? Are there good reliable manufacturers in the US that I could try out? Or nobody cares about the amount of noise their machines make?

    Any experience reports or advice from you, dear readers, will be greatly appreciated!

  • Kirill Osenkov

    Roslyn Code Quoter updated to work with September 2012 CTP

    • 1 Comments

    I’ve blogged before about the tool called Quoter that shows how to use the Roslyn Syntax APIs to construct syntax trees manually:

    http://blogs.msdn.com/b/kirillosenkov/archive/2012/07/22/roslyn-code-quoter-tool-generating-syntax-tree-api-calls-for-any-c-program.aspx

    I’ve updated the source to work with the latest (September 2012) Roslyn CTP:

    http://code.msdn.microsoft.com/Roslyn-Code-Quoter-f724259e

    Apart from several API changes I’ve also made it remove redundant calls, such as calls to insert { and } into new type declarations (they’re added by default already). Thus, the generated code is now shorter. For the source code “class C { }” we now generate:

    Syntax.CompilationUnit()
      .WithMembers(
        Syntax.List<MemberDeclarationSyntax>
          Syntax.ClassDeclaration(
            @"C")))
      .NormalizeWhitespace()

    which is way shorter than it used to be. I also made it not preserve whitespace/formatting by default (it will now generate trees without whitespace and then apply default formatting). If you’d like to preserve the exact whitespace of your source program, you need to set UseDefaultFormatting = false.

    Enjoy!

  • Kirill Osenkov

    Split an imagestrip into individual images/icons

    • 3 Comments

    If you have a large horizontal image Nx16 and you’d like to split it into individual 16x16 icons, here’s how to do it in WPF. I had to look it up recently and nothing showed up, so it was faster to write it myself. I’m posting it here so that it’s easy to find in the future.

    using System.IO;
    using System.Windows;
    using System.Windows.Media.Imaging;
     
    class Program
    {
        static void Main(string[] args)
        {
            var file = @"Imagestrip1600x16.png";
            var output = @"IconsFolder";
            using (Stream imageStreamSource = new FileStream(
                file, FileMode.Open, FileAccess.Read, FileShare.Read))
            {
                PngBitmapDecoder decoder = new PngBitmapDecoder(
                    imageStreamSource,
                    BitmapCreateOptions.PreservePixelFormat,
                    BitmapCacheOption.Default);
                BitmapSource bitmapSource = decoder.Frames[0];
     
                for (int i = 0; i < bitmapSource.PixelWidth / 16; i++)
                {
                    CroppedBitmap croppedBitmap = new CroppedBitmap(
                        bitmapSource,
                        new Int32Rect(i * 16, 0, 16, 16));
     
                    PngBitmapEncoder encoder = new PngBitmapEncoder();
                    var frame = BitmapFrame.Create(croppedBitmap);
                    encoder.Frames.Add(frame);
                    var fileName = Path.Combine(output, i.ToString() + ".png");
                    using (var stream = new FileStream(fileName, FileMode.Create))
                    {
                        encoder.Save(stream);
                    }
                }
            }
        }
    }
  • Kirill Osenkov

    Roslyn September 2012 CTP is out!

    • 0 Comments

    You can download it here. Just like the previous CTPs, the setup is fairly non-intrusive and should be safe to install on your work machine. This release will only install into Visual Studio 2012 RTM (meaning you can still have VS 2010 on the machine of course, it’s just this release of Roslyn won’t work with it). We have also updated the Roslyn NuGet packages, however if you have the CTP you shouldn’t need them because the CTP installer GACs all Roslyn binaries.

    This release has support for almost all C# 3.0 and VB 9 language features. In fact, it is already powerful enough to compile itself (it means than the Microsoft C# and VB compilers are finally bootstrapped!).

    In terms of samples, documentation and public API we didn’t update much since our June CTP, though you will still notice minor breaking changes here and there (such as SyntaxTree.ParseCompilationUnit() renamed to SyntaxTree.ParseText()).

    As always, you’re welcome to play with it and let us know what you think! Dust off that #RoslynCTP hashtag :)

  • Kirill Osenkov

    Roslyn Code Quoter tool – generating syntax API calls to generate any C# program

    • 3 Comments

    Whether you’ve played with the Roslyn CTP or are planning to do so in the future, chances are that sooner or later you’ll run into a need to generate code.

    One approach is to just use a StringBuilder, concatenate the source together, and then just call any of the several parse methods (SyntaxTree.ParseCompilationUnit, Syntax.ParseExpression, Syntax.ParseStatement, etc.) on the source text to get a syntax node or the whole syntax tree:

    var root = Syntax.ParseCompilationUnit("class C{}");
    

    Another way is to manually call the fluent API on the Syntax class (factory methods to create nodes, tokens and trivia). For those who don’t know, nodes are non-terminals, tokens are terminals, and trivia are whitespace, comments, preprocessor directives, etc. To create the syntax tree for this program: “class C{}”, you’d need to make a series of syntax API calls similar to this:

    var root = Syntax.CompilationUnit()
    .WithMembers(
        Syntax.List<MemberDeclarationSyntax>(
            Syntax.ClassDeclaration(
                Syntax.Identifier(
                    @"C"))
            .WithKeyword(
                Syntax.Token(
                    SyntaxKind.ClassKeyword,
                    Syntax.TriviaList(
                        Syntax.Space)))
            .WithOpenBraceToken(
                Syntax.Token(
                    SyntaxKind.OpenBraceToken))
            .WithCloseBraceToken(
                Syntax.Token(
                    SyntaxKind.CloseBraceToken))))
    .WithEndOfFileToken(
        Syntax.Token(
            SyntaxKind.EndOfFileToken));
    

    The first thing you notice is that the second approach is slightly more verbose (I’ve calculated that it explodes ~50-100 times in code length on an average program). So using this to serialize syntax trees is (though absolutely possible) quite impractical (might as well use Charles Petzold’s CSAML). It just so happens that the C# syntax is the most terse human-readable format to serialize C# syntax trees.

    However the second approach still has its advantages:

    1. Not always you need to construct the whole tree from scratch – more likely you already have most of the nodes you need, you just need to recombine them in a certain configuration using just a few glue syntax nodes. In this case the factory methods are really handy.
    2. With a StringBuilder it quickly gets hairy when you generate code based on some non-trivial logic (e.g. open curly, repeat this part N times, after each time (but not after the last one) insert a comma, close curly, indent this much, etc). You need to worry about creating a syntactically valid tree, keep track of closing braces, brackets and parentheses, indentation, etc.
    3. It’s faster than parsing text – the parser will eventually call the same APIs to construct the tree, but it first needs to parse and that takes time and memory. By specifying the structure of the tree in our API calls, we eliminate the need for the parser.

    So sometimes there’s a need to manually write calls to those Syntax.* APIs to construct syntax nodes. And it can get tedious. Also, sometimes it’s not obvious which API need to be called to construct syntax nodes of the desired shape.

    So I decided to write a sample demo tool that would automate generating syntax API calls to construct any given program. The tool is called Quoter, because what it does is basically quasi-quotation: given a source program, generate a program that, when run, will generate the source program. Among programming languages that natively support quasi-quotation are F#, Nemerle, and, beside others, … C#! Surprised? Yes, C# has at least two features where the compiler is quoting your code – generating code that describes how to generate your code at runtime. The first one is, of course, expression trees, where you just write an expression like you would otherwise, and the compiler emits calls to Expression.* factory methods for you. The second one is more subtle: when you have calls on a variable or expression typed as dynamic, the compiler bakes in the information about the calls you made into a call site, which is basically a description of the syntax tree that you had in the source program, that is interpreted (and cached!) at runtime. I guess you could say that expression trees do “early bound quoting” (type resolution occurs at compile time), whereas dynamic does “late bound quoting” (it produces unbound trees and binding occurs at runtime).

    How would one write a quoting tool? A good approach would be to use a parser to obtain the syntax tree, and then apply a syntax visitor to that tree – the classic Visitor design pattern where you declare a VisitXXX method for each type of syntax node, token and trivia. You get a class declaration syntax node and you emit “Syntax.ClassDeclaration(“ + ... + “)”. Simple, but tedious. We have lots and lots of kinds of nodes!

    Fortunately for me, a lot of the Roslyn classes that model nodes, tokens and trivia, as well as most factory methods, are themselves generated (we didn’t write all of them by hand). Due to this nature they all look uniform and adhere to simple, predictable rules. In fact, to generate a call to Syntax.ClassDeclaration() for a node of type ClassDeclarationNode, you could use Reflection to inspect all public static methods on the type Roslyn.Compilers.CSharp.Syntax, and select the one that has ClassDeclarationSyntax type as its ReturnType.

    Moreover, for the properties of nodes that aren’t initialized from the factory method, you could enumerate them, put them in a bag, and then for each property pick out a With*** modification method that accepts the type of the property as parameter. Using this simple approach, the main recursive method of the tool can look surprisingly simple:

        /// <summary>
        /// The main recursive method that given a SyntaxNode recursively quotes the entire subtree.
        /// </summary>
        private ApiCall QuoteNode(SyntaxNode node, string name)
        {
            List<ApiCall> quotedPropertyValues = QuotePropertyValues(node);
            MethodInfo factoryMethod = PickFactoryMethodToCreateNode(node);
     
            var factoryMethodCall = new MethodCall()
            {
                Name = factoryMethod.DeclaringType.Name + "." + factoryMethod.Name
            };
     
            var codeBlock = new ApiCall(name, factoryMethodCall);
     
            AddFactoryMethodArguments(factoryMethod, factoryMethodCall, quotedPropertyValues);
            AddModifyingCalls(node, codeBlock, quotedPropertyValues);
     
            return codeBlock;
        }
    

    You will immediately notice that I myself am using simple string concatenation and basically, the StringBuilder approach to generate the code that will generate the code. Using the Roslyn syntax APIs here to generate calls to the Roslyn syntax APIs that will generate the target source code is left as an exercise to the reader. I almost wish that the tool would quote it’s own source, find a fixed point and eventually converge to write itself, however for obvious reasons this is not going to happen (quoting is not a contraction mapping on the space of C# programs).

    For now, visiting the tree creates a simple data structure I defined (ApiCall), which is basically a tree of strings. This is the simplest representation I could find to represent the simple method calls of the form Syntax.A(b, c).WithD(e).WithF(g). If we later want to actually use the generated code to construct the syntax tree, we could copy-paste the generated program into our own project. To verify that my Quoter tool does the right thing, I wrote a simple Evaluator based on Roslyn scripting that executes the generated code, produces a syntax tree, gets its text and compares it to the original source:

                var sourceText = "class C{}";
                var generatedCode = new Quoter()
                {
                    OpenParenthesisOnNewLine = false,
                    ClosingParenthesisOnNewLine = false
                }.Quote(sourceText);
     
                var evaluator = new Evaluator();
                var generatedNode = evaluator.Evaluate(generatedCode) as CompilationUnitSyntax;
                var resultText = generatedNode.GetFullText();
                if (sourceText != resultText)
                {
                    throw new Exception();
                }
     
                Console.WriteLine(generatedCode);
    

    And here’s the source code of the Evaluator:

    using Roslyn.Compilers.Common;
    using Roslyn.Compilers.CSharp;
    using Roslyn.Scripting;
    using Roslyn.Scripting.CSharp;
     
    public class Evaluator
    {
        private ScriptEngine engine;
        private Session session;
     
        public Evaluator()
        {
            engine = new ScriptEngine(
                importedNamespaces: new[] { "Roslyn.Compilers", "Roslyn.Compilers.CSharp" });
            session = Session.Create();
            session.AddReference(typeof(CommonSyntaxNode).Assembly);
            session.AddReference(typeof(SyntaxNode).Assembly);
        }
     
        public object Evaluate(string code)
        {
            var result = engine.Execute(code, session);
            return result;
        }
    }

    I tested it on round-tripping this little program and it seems to me that we’ve got all the C# 4.0 syntax covered.

    A couple of tips. You can adjust OpenParenthesisOnNewLine and ClosingParenthesisOnNewLine boolean properties to configure how the generated code is formatted. I personally prefer this more verbose format because the nesting of the blocks is clearly visible:

    Syntax.CompilationUnit()
    .WithMembers
    (
        Syntax.List<MemberDeclarationSyntax>
        (
            Syntax.ClassDeclaration
            (
                Syntax.Identifier
                (
                    @"C"
                )
            )
            .WithKeyword
            (
                Syntax.Token
                (
                    SyntaxKind.ClassKeyword,
                    Syntax.TriviaList
                    (
                        Syntax.Space
                    )
                )
            )
            .WithOpenBraceToken
            (
                Syntax.Token
                (
                    SyntaxKind.OpenBraceToken
                )
            )
            .WithCloseBraceToken
            (
                Syntax.Token
                (
                    SyntaxKind.CloseBraceToken
                )
            )
        )
    )
    .WithEndOfFileToken
    (
        Syntax.Token
        (
            SyntaxKind.EndOfFileToken
        )
    )

    Another tip: you generally don’t need to generate whitespace trivia yourself. Just generate a tree without any whitespace trivia and then call SyntaxNode.NormalizeWhitespace() method that will automatically insert whitespace using the common C# formatting rules. NormalizeWhitespace() is quite a simple formatter. To use the full-blown feature-rich formatter used by Roslyn Services and Visual Studio to format code, you’ll need to additionally reference Roslyn.Services.dll and Roslyn.Services.CSharp.dll and then call the Format() extension method on the node.

    The full source (Quoter.cs, 833 lines) is published over at http://code.msdn.microsoft.com/Roslyn-Code-Quoter-f724259e. I’ll be happy to answer any questions.

  • Kirill Osenkov

    Windows Explorer Preview Pane for .vb files

    • 3 Comments

    I love using the Preview Pane in Windows Explorer to quickly preview file contents:

    image

    Usually, to enable this preview for a file extension I just open regedit, go to HKEY_CLASSES_ROOT –> .xxx (where xxx is the extension you’re interested in), and add a string key called PerceivedType with value of “text”: http://blog.wpfwonderland.com/2011/01/15/customize-windows-7-preview-pane-for-xaml-files/

    However for some reason it didn’t work for .vb files. After some web searching, one solution I found that worked for me is to add the preview handler GUID explicitly under a “shellex” key: http://www.howtogeek.com/howto/windows-vista/make-windows-vista-explorer-preview-pane-work-for-more-file-types/

    image

    Windows Registry Editor Version 5.00

    [HKEY_CLASSES_ROOT\.vb\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]
    @="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

  • Kirill Osenkov

    High accuracy DateTime.UtcNow

    • 5 Comments

    Update: in the original version of this post I’ve jumped to an incorrect conclusion that the timer resolution depends on the operating system. However thanks to a comment from James Manning who has corrected me, I realized that this is likely not related to the OS version (because we’re seeing high resolution DateTime on both Win7 and Windows Server 2008 R2).

    I’ve just discovered an interesting thing: apparently one of my machines has timer resolution 1 millisecond or better (as opposed to the typical 15ms that I’m used to seeing elsewhere).

    I ran these tests on one machine:

            var dt1 = DateTime.UtcNow;
            Thread.Sleep(11);
            var dt2 = DateTime.UtcNow;
            Console.WriteLine(dt2 - dt1);

    It outputs 00:00:00.0156006 (common knowledge – 15-16 milliseconds)

    However on my other machine, it outputs: 00:00:00.0110011

    Also, I ran this test:

            var sw = Stopwatch.StartNew();
            var start = DateTime.UtcNow;
            int changes = 0;
            int idle = 0;
            while (sw.ElapsedMilliseconds < 50)
            {
                var now = DateTime.UtcNow;
                if (now != start)
                {
                    start = now;
                    changes++;
                }
                else
                {
                    idle++;
                }
            }
     
            var elapsed = sw.Elapsed.ToString();
            Console.WriteLine(elapsed);
            Console.WriteLine(changes);
            Console.WriteLine(idle);
    

    and it printed:

    00:00:00.0500000
    3
    449714

    Which means it only updated 3 times within a 50 millisecond interval. On my new machine, this program printed:

    00:00:00.0500001
    50
    471655

    Which means that it updated at least every millisecond.

  • Kirill Osenkov

    A list of common HRESULT error codes

    • 3 Comments

    I was looking for error code –2146232797 (hex 0x80131623, which turned out to be what is thrown by Environment.FailFast) and I’ve stumbled upon this treasure:

    http://blogs.msdn.com/b/eldar/archive/2007/04/03/a-lot-of-hresult-codes.aspx

    Also, here’s a great blog about deciphering an HRESULT:

    http://blogs.msdn.com/b/heaths/archive/2005/07/21/441391.aspx

    And here's another good list from MSDN:

    http://msdn.microsoft.com/en-us/library/cc704587.aspx

    I sincerely wish you to never ever need this knowledge...

  • Kirill Osenkov

    A Google Group for Augmented Programming

    • 0 Comments

    Josh Marinacci (from http://joshondesign.com) recently sparked a discussion about structured editing: http://joshondesign.com/2012/03/14/sacred-cows-2

    Jonathan Edwards has also blogged about this: http://alarmingdevelopment.org/?p=680#comment-59782

    It seems like there is continuous interest in structured editing.

    Jake Brownson had an idea to create a Google Group for us, structured editing and languages enthusiasts, and here it is:

    https://groups.google.com/forum/#!forum/augmented-programming

    The name “Augmented Programming” is neutral and temporary on purpose (to avoid preconceptions).

    I don’t know how much participation we’ll have, but it��s good to at least have a single central forum with the right folks involved. If you’re interested in structured editing, join in!

  • Kirill Osenkov

    Your Feedback about the Roslyn CTP NuGet Package?

    • 2 Comments

    As part of the Roslyn CTP in October 2011 we’ve published the core compiler and services binaries as a NuGet package:

    http://nuget.org/packages/roslyn

    Right now the Roslyn package adds references to the six core assemblies:

    1. Roslyn.Compilers
    2. Roslyn.Compilers.CSharp
    3. Roslyn.Compilers.VisualBasic
    4. Roslyn.Services
    5. Roslyn.Services.CSharp
    6. Roslyn.Services.VisualBasic

    I’d like to gather suggestions and other feedback about this package. How can we improve?

    The main piece of feedback we’ve heard so far is more fine-grained definition of what layers are needed:

    1. consumers don’t always need both C# and Visual Basic support
    2. consumers don’t always need the Services layer (Compiler layer is enough)

    Hence it would probably make sense to split the main package into several dependent packages, but then it gets a little complicated. We’d have to publish 8 packages and then the consumer could pick one “top” package which would then pull down all the dependencies it needs and nothing else):

    image

    It looks like we’re not the first to ask these questions:

    The consensus on this one seems to be that fine-granular is OK and as long as we publish updates to all subpackages simultaneously there shouldn’t be a problem.

    Your feedback and ideas about this issue and other suggestions are welcome!

  • Kirill Osenkov

    My Alma mater – BTU Cottbus in Germany

    • 1 Comments

    For my regular readers – this is a non-technical, non-work related post about my university BTU Cottbus and some current events related to it. I consider it important to publicly share my personal opinion on the matters to support my former professors, teachers, classmates and friends. This post is not endorsed by Microsoft and represents my private opinion only.   -- Kirill

    I consider myself fortunate to have studied at and graduated from the Brandenburg University of Technology (hereafter affectionally referred to as BTU) in Cottbus, Germany.

    imageimageimage

    It is a great university that welcomed me as an international student from Ukraine and provided me with a solid, high quality education.

    I studied computer science. The courses were great, modern, deeply technical and providing with the necessary foundations as well as practical applications. I’ve learned about structural programming (C++), object oriented programming (Java, .NET), functional programming (Haskell), pure computer science (algorithms, data structures, computability, turing machines, automata, languages), compilers (grammars, parsers, optimizers, semantics, lambda calculus), graphics, databases, OS fundamentals (threading, synchronization) and many many other things. I wrote an operating system (a toy one, but it booted and had threads!), a 3D fractal landscape generator in Haskell with OpenGL (anybody know what the $= operator means?), an AI (alpha-beta pruning) game engine in Prolog, a VPRO processor model in VHDL with an assembler and an interpreter, a garbage pickup schedule database in PHP and IBM DB2, and last, but not least, a structured editor for a subset of C#.

    The teachers and professors were very appproachable, loved their work, stayed late to help explain things, and often enjoyed a beer or two together with the students after lectures. The atmosphere was great.

    The facilities were good too, new buildings, projectors, computers, free internet, free web hosting, free source control, etc. Check out the new library for example:

     

    The university also provides students with valuable connections to start their future career. Through such a university-industry connection I found two wonderful summer internships with Microsoft in Redmond. After I’ve finished my thesis and graduated, I started fulltime here at Microsoft, which I consider is a great beginning that opens wide opportunities in life. I was interested in programming languages, editors and developer tools and found the best place to work on it!

    At the university, I’ve gained a lot of valuable professional and personal connections, found friends, learned new things, experienced life. The excellent German course for international students helped me learn German reasonably well in 6 months.

    To me, BTU is a university that provides deep and solid foundations to students of many disciplines – math, physics, computer science. It creates professionals who one day will grow up to become system changers and improve the world around them by creative thinking and applying what they’ve learned.

    And now they want to close this university as it exists today.

    There are plans and proposals from the ministry to dissolve BTU and the neighboring Hochschule Lausitz (another college in Brandenburg), and create a new university/college that is more targeted for environment/energy. As part of this, there are recommendations to close or severely trim the Department of Mathematics, Physics and join/trim the Departments of Computer Science and otherwise deprioritize math/computer science.

    Many of my friends among computer science teachers, professors and former classmates oppose this initiative, instead voting for preserving both colleges, and improving ties and collaboration between them, without sacrificing our deep specialization in math, physics and computer science. There is a lot of existing value that has accumulated in the current BTU structure over the years, it would be unwise to destroy all of it, throw away all the experience, fire (some? most?) teachers and professors and start from scratch.

    I would like to join and support this movement because I deeply believe that universities should not deprioritize fundamental things like math and computer science – we need to grow new generations of professionals who didn’t learn to code by “Teach yourself ThisYearsTechnologyX in 24 hours” kind of books.

    It is also especially important to attract and support women in math, physics and computer science, the way the ratio is skewed towards men nowadays is I think a real problem. And how do we attract more girls and boys to computer science, by closing well-established computer science departments and universities? Why fix what aint broken? Why be destructive before being constructive?

    Universities are a cradle of hope for the progress of our civilization, this is where we grow our best and brightest who will grow up and solve problems. We should do everything we can to foster and support them.

    image

    -----

    An meine Freunde, Alumni, Professoren, Mitarbeiter und Studierende der BTU, sowohl auch andere, die sich beteiligt fühlen – bitte äußern auch Sie Ihre Meinung zu diesem wichtigen Thema. Dieser Link http://studiy.tu-cottbus.de/projektwiki/gruppen:mittelbau:alumni_berichten erhält schon viele Berichte und Statements, aber nur wenn es viel mehr Stimmen gibt, die sich zur BTU bekennen, wird das Ministerium hoffentlich zuhören.

    -- Dipl.-Inf. Kirill Osenkov, Microsoft (from Redmond, WA, USA)

  • Kirill Osenkov

    New Tabula video

    • 0 Comments

    For those of you who follow me because of my interest in geometry, NumeracyWorks has released a new video about Tabula: http://www.youtube.com/watch?v=ZgkWtyfpi8Y

    The reason I’m so excited about it is that it’s a great showcase of the open-source geometry framework we’ve been developing at http://livegeometry.codeplex.com. I had no idea that one could build such amazing things with it. That’s the beauty of “seeding” an open-source project – it lives on without you!

    Since starting on Roslyn full time, I had to unfortunately de-prioritize my hobby work on Live Geometry for various reasons. First, I have to focus on one thing at a time to go deep and be productive. Second, Roslyn is a project of utmost importance, it’s The Right Thing To Do and I feel I need to give it my 100%.

    So it’s amazing to see that thanks to my contributors (mainly David) the framework lives on and brings fruition to such awesome products!

  • Kirill Osenkov

    Found my 1000th bug in Roslyn!

    • 3 Comments

    This might not seem like a big deal, but...

    1000BugsDev11

    Knowing that you guys won’t be seeing any of these bugs in production helps me sleep better at night :)

    So far, my favorite bug (nothing to be proud of, but I think it’s a fun one) is this:

    ContractFailure in FormattingContext.DebugCheckIntervalTree when formatting "War and Peace" by Leo Tolstoy

    Yes, Roslyn can parse, format and analyze large programs with errors. And believe me, the C# program War and Peace contains quite a few errors!

     

    Note: please don’t interpret this post as “Roslyn has 1000 bugs” :) On the contrary, I’m pretty happy with our rolling day-to-day quality, and folks who know me can appreciate what it says about the codebase.

  • Kirill Osenkov

    How to change the default command prompt color on Vista and above

    • 7 Comments

    If you like me don’t like the default green on black in the command prompt, you can change the default via registry:

    image

    Tip taken from here: http://www.pctools.com/guides/registry/detail/12/

    Update: and here’s a list of color values http://technet.microsoft.com/en-us/library/cc978716.aspx

  • Kirill Osenkov

    VS hangs for 1 minute on start debugging? Check for dead symbol paths!

    • 0 Comments

    I’ve just hit this again so I decided to share the solution in case anyone else hits this.

    Symptom: Visual Studio freezes for about a minute after you hit F5 (but before the app actually starts), then suddenly unfreezes and happily proceeds to launch the app.

    Cause: VS calls the Windows CreateFile function on one of the paths specified in Tools –> Options –> Debugging –> Symbols –> Symbol file (.pdb) locations and that location either doesn’t exist or can’t be accessed (e.g. due to misconfigured DNS settings).

    Solution: Check that all network paths in the list above can be accessed normally.

    If you launch ProcMon.exe before starting debugging and configure the filter to only monitor “devenv.exe”, Result = “BAD NETWORK PATH”, it should show you which CreateFile call causes a network timeout.

    Sources:

    * http://support.microsoft.com/kb/951336
    * http://social.msdn.microsoft.com/forums/en/vsdebug/thread/1ede20ce-b236-4361-8dcb-04189b607152

  • Kirill Osenkov

    DateTime.UtcNow is generally preferable to DateTime.Now

    • 9 Comments

    This seems to be commonly known and accepted best practice to use DateTime.UtcNow for non-user facing scenarios such as time interval and timeout measurement.

    I’ve just done an audit of the Roslyn codebase and replaced most DateTime.Now calls with DateTime.UtcNow. I thought it’d be useful to post my changeset description here (although none of it is new – I just summarize some common knowledge readily available in the sources linked below).

    ====

    Replacing DateTime.Now with DateTime.UtcNow in most cases.

    We should be using DateTime.Now only in user-facing scenarios. It respects the timezone and Daylight Savings Time (DST), and the user feels comfortable when we show them the string that matches their wall clock time.

    In all other scenarios though DateTime.UtcNow is preferable.

    First, UtcNow usually is a couple of orders of magnitude faster, since Now is basically first calling UtcNow and then doing a very expensive call to figure out the time zone and daylight savings time information. Here’s a great chart from Keyvan’s blog (modified to avoid allocations of the Stopwatch object):

    image

    Second, Now can be a big problem because of a sudden 1-hour jump during DST adjustments twice a year. Imagine a waiting loop with a 5-sec timeout that happens to occur exactly at 2am during DST transition. The operation that you expect to timeout after 5 sec will run for 1 hour and 5 seconds instead! That might be a surprise.

    Hence, I'm replacing DateTime.Now with DateTime.UtcNow in most situations, especially polling/timeout/waiting and time interval measurement. Also, everywhere where we persist DateTime (file system/database) we should definitely be using UtcNow because by moving the storage into a different timezone, all sorts of confusion can occur (even "time travel", where a persisted file can appear with a future date). Granted, we don't often fly our storage with a supersonic jet across timezones, but hey.

    For precise time interval measurements we should be using System.Diagnostics.StopWatch which uses the high resolution timer (QueryPerformanceCounter). The resolution of both DateTime.Now and DateTime.UtcNow is very low – about 10 ms – this is a huge time interval!

    For a cheap timestamp, we should be generally using Environment.TickCount - it's even faster than DateTime.UtcNow.Ticks.

    I'm only leaving DateTime.Now usages in test code (where it's never executed), in the compiler Version parsing code (where it's used to generate a minor version) and in a couple of places (primarily test logging and perf test reports) where we want to output a string in a local timezone.

    Sources:

    * http://www.keyvan.ms/the-darkness-behind-datetime-now
    * http://stackoverflow.com/questions/62151/datetime-now-vs-datetime-utcnow
    * http://stackoverflow.com/questions/28637/is-datetime-now-the-best-way-to-measure-a-functions-performance

Page 1 of 7 (153 items) 12345»