Welcome to MSDN Blogs Sign in | Join | Help

While I haven't been posting much here, I have been posting pretty regularly over on my main development blog code-o-rama.   The main reason is that I'm running a newer version of Community Server over there, which has better editing, etc..  Here are some articles you might want to check out:

Stand alone unit test programs

VS macro to insert GUIDs

Real Team Foundation command line help

Playing with VS macros: Getting the PrimaryOutput

Changing console fonts

Subtle visible whitespace in VS2005

Visual Studio column guides redux

Reading custom configuration sections in .NET 2.0

SxS problems

Turning off Vista problem reporting

Reader/writer lock in .NET 3.5

Putting SxS errors in your face

Running MSBuild scripts programmatically

VS macro to insert TODO:

Listing all solution project properties

Getting accurate per thread timing on Windows

Merry Christmas, etc., etc..   Hope you are all enjoying your XBOX 360's.  New year, new blog name.  I'm renaming by blog and refocusing it for 2007.

Recently we decided to give everyone on the Visual Studio Diagnostics team a Director's title (minus the actual promotion).  Mine was "Director of Random Technologies" probably on account of the fact that I have a tendency to act a bit like Hammy in "Over the Hedge" when it comes to software and computers.  It's all good.

The realization hit me this year that the blogs I read the most are the ones that are focused in some way in their scope.  They have a theme.  Sadly my blog has thus far had no theme.  It also helps when trying to write blog entries if you have a theme, otherwise it's a bit like a book without a plot. 

So I've decided refocus, and to use my MSDN blog to write solely about Microsoft diagnostic tools and technologies.  We'll see how it works out.

Another thing I'm going to try and do is keep my entries short and sweet.

We were having a discussion today about Microsoft Points and that it would be useful to be able to transfer them to others as payment for, oh I don't know let's say, um, gambling debts.  Hey, that would be just like real money!  Couldn't find anything about that, but I did notice that there is a handy dandy Microsoft Points Converter.

As you know if you are a Visual Studio Team System user, we provide two types of profilers with the product; sampling and trace.  If you are not familiar with this feature of VSTS, one place to start are the excellent TechNotes under the Development heading on this page

The trace profiler works by rewriting the method bodies of your program to include probes.  A probe is a small piece of code, usually a function call, that calls out to the profilers collection logic so that it can record the current location of the program execution.

The subject of this blog is how those probes get into your methods.  This process is called instrumentation.  I'll discuss two approaches, and then I'll discuss the pros and cons of each.

Static Instrumentation

In this approach the instrumentation occurs before your program even runs.  Essentially the probes are baked in to you binary.  There are two ways this can be done.  One is to have the compiler put the probes in.  Visual C++ takes this approach to implent Profile Guided Optimizations (PGO).  Another is to do what is called post link instrumentation.  With this approach you take the binary that has been emitted by the compiler, crack it open to find all the methods, instrument and then rewrite the binary like nothing ever happened. 

Pros & Cons:
This approach yields much faster start-up times.  The images that is loaded in to memory is the image that runs.

The rewrite process destroys .NET strong naming, and thus images must be resigned after instrumentation, or verification skipping must be enabled.  Statically instrumented images can also be placed in the native image cache.

Rewriting the images means you have to deal with the whole question of where to put the files that you instrument.  This can be quite a housekeeping chore for the user.

Rewriting PE files correctly is relatively straightforward for .NET applications.  It's really hard for native applications, particularly for x64 images, where instrumenting methods can turn them from leaf methods to non-leaf methods.

Dynamic Instrumentation. 

In this approach the instrumentation occurs as the program is running.  In the case of a just in time compilation environment such as .NET, this is acheived by using the .NET profiling API to get notified when methods are about to be compiled and executed for the first time.  Your profiler gets the opportunity to rewrite the method that is about to be run, by adding probes or whatever you want.  Then the program continues on.  Dynamic instrumentation can also be performed on native images too, but doing this is hard and rarely done.  The approaches you could take to pull this involve injecting a DLL into the process, suspending it and then using debug information to patch and rewrite methods.

Pros & Cons

Yields significantly slower start-up times and increased memory usage because each method body must be reallocated, as well as the the some CPU load to instrument on the fly. 

If you want to do instrumentation at the line level, this approach requires loading additional data from the .pdb files in at runtime too.  If you load the actual .pdb's that has a huge runtime cost. 

Because the original file was not modified, .NET strong naming is not broken.  However, profiling of images in the native image cache must be disabled, as these cannot be dynamically instrumented because they are already compiled to native code

Conclusion 

Hopefully this entry has given you an overview of the two approaches.  Please comment with any question you may have.

One of the things that you often want to do when creating a Windows Installer using WiX is to uninstall the existing product before installing the new one.  This is referred to as doing a "major upgrade".  The way to pull this off is to add the following WiX snippet to your .wxs file:

<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi" xmlns:netfx="http://schemas.microsoft.com/wix/NetFxExtension">
  <Product 
    Id="*" 
    ...
    UpgradeCode="35ED4D41-BA5F-4598-8266-A1B9EB512345">
    ...
    <Upgrade Id="35ED4D41-BA5F-4598-8266-A1B9EB512345">
      <UpgradeVersion Minimum="$(var.ProductVersion)" OnlyDetect="yes" Property="NEWERVERSIONDETECTED"/>
      <UpgradeVersion Minimum="1.0.0" IncludeMinimum="yes" Maximum="$(var.ProductVersion)" IncludeMaximum="no" Property="OLDERVERSIONBEINGUPGRADED"/>
    </Upgrade>
    ...
    <InstallExecuteSequence>
      <RemoveExistingProducts After="InstallInitialize" />
    </InstallExecuteSequence>

Note the product's Id attribute set to "*" so you don't forget to change it each time, but make sure the UpgradeCode stays the same forever.  You could change the IncludeMaximum attribute to "yes" on the second UpgradeVersion element for debug builds so that you can continuously reinstall during development, even though your version number may not be changing.  However I don't recommend you ship that way, because you want repair existing installations by default not uninstall/install them.

My thanks to Bob Arnson on the WiX team for providing this tip.

I just rediscovered the Ethereal network protocol analyzer.  I was trying to track down a problem using the tcptrace proxy which is a nice simple little tool when it works.  I'm not quite sure what the problem was, but I was seeing different behavior when I was using the proxy than when I wasn't, so I went for the full blown driver level sniffing that Ethereal provides.  It's based on the work of the WinPCap organization.  I'll let you know why I was sniffing network packets in my next post.

I've always loved editors with built in macro languages. Once you have macro control of every facet of the editor you can pull off some really cool and time saving tricks. I gave up macros back in the early 90's for the graphical environment of Microsoft Quick C, Developer Studio and now Visual Studio. So I was overjoyed to see macros make a real comeback with the release of Visual Studio 2002/2003. I'm still a little bit miffed that the macro language is Visual Basic. Not that I have anything against VB, it's just not a language that I'm super fluent in, so it makes the task of writing macros that much more challenging.I was writing a macro the other day, and every time I ran it I noticed this little balloon pop up on the task bar. I couldn't read what the balloon said, it came and went so fast. Finally, I cranked up the macro debugger and while stopped on a breakpoint I got a good look at the little bugger.

It's hard to click on something that flashes up and disappears in the blink of an eye. There's two reasons that I can think of for showing this balloon at all. One is as an informational message to educate you about macro usage. The other is a security feature so that you know that a macro is running. Neither of which I find that compelling.

A few minutes work with Regmon and one can discover that setting the following value in the registry to one will disable this balloon for ever more.

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\8.0]
"DontShowMacrosBalloon"=dword:00000006

Craig Skibo, one of the Microsoft experts on the VSSDK say that this security balloon was the subject of much debate, and that it is in fact coded to not show up after 5 or 6 times of being clicked on. Nice touch, if you could actually click on it. Hence the the value of 6 above.

Some platform features are like the stuff hiding under rocks at the seashore. No, I don't mean the one's that are slimey and stinky. I'm talking about the one's that are interesting but hard to spot. But, all you need to do is catch a glimpse of them and lift the rock...

Sometime last year, I was reading a page in the MSDN documentation when a certain line caught my attention. The page was "Locating the Assembly through Codebases or Probing" and the line was this one, at the very bottom of the page:

"If Assembly2 is not found at either of those locations, the runtime queries the Windows Installer."

For some reason, this line struck me as interesting – a hitherto unseen mystery in the .NET runtime. OK, it's not like I was finding some hidden clue in the Hypernotomachia Poliphili or anything. But this is my world, and I take my kicks where I can get 'em.

I'd read this line many times before, but I'd never really thought anything of it. Now I was suddenly realizing in all the times I'd seen assembly probing fail, I'd never ever seen it report that it was asking the Windows Installer to find the assembly. Was this just a documentation error, or was this a real feature of the .NET runtime?

To test things out, I created a simple application. It basically consisted of a managed executable (A) that loaded a library (B) using Assembly.Load(AssemblyName). Trying to do things the fastest way possible, I created a Windows installer by creating a Setup Project in VS 2003. I installed the application, deleted library B and ran A, and sure enough there was an assembly load exception, but there was no message in the Fusion log about the Windows Installer being invoked to find the assembly.

I sent of a couple of e-mails to Microsoft folks, and got a response back from the .NET team that indicated that if all was well with the installation, a call to MsiProvideAssembly with the correct parameters should cause the missing assembly to reappear. OK, so .NET must be relying on this call internally to find the assembly, right? I ran the installed application under a debugger, and I did not see msi.dll, the Windows Installer library, being loaded by the .NET runtime. Very strange. A quick peek confirmed that any Windows Installer probing code had been stripped out of the Rotor sources, so no help there.

So I logged a bug at the Feedback center. It even got a mention in Junfeng Zhangs blog, one of the CLR loader wizards. The problem, it would appear, was with the installer built by the VS 2003 Setup project. It was not doing the right magic when it installed the application to allow .NET to find the assembly. What that magic was, and why .NET never loaded msi.dll remained a mystery.Well, I did eventually figure out what was going on, but I never got around to blogging about it.

So for the other pathologically curious folks out there, here's the answer to the mystery. Firstly, as we've already established, .NET is not really at fault. It is however up to some of its usual sneakiness in the never ending quest for performance, which is what made it hard for me to point the finger at the actual cause of the problem in the first place. The fact is, the .NET runtime doesn't even bother loading msi.dll unless it can be certain that the cost will be worthwhile. In order to do that, it actually does some of the checks that MsiProvideAssembly would do. That's why I never saw msi.dll being loaded. To discover this I ran Regmon on the installed application (with the library deleted), on the assumption that that was the most likely place for further clues, and searched for the word "Installer" in the output. It turns out that the runtime scans the following list of registry keys for a record of the missing assembly:

HKLM\SOFTWARE\Microsoft\Windows CurrentVersion\Installer\Managed\<SID>  Installer\Assemblies\Global\<Strong Name>
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer Managed\<SID>\Installer\Assemblies\<File Path>HKCU\Software\Microsoft\Installer\Assemblies Global\<Strong Name>
HKCU\Software\Microsoft\Installer\Assemblies\<File Path>HKLM\SOFTWARE\Classes\Installer\Assemblies\Global\<Strong Name>
HKLM\SOFTWARE\Classes\Installer\Assemblies\<File Path>\

If any record of the assembly being installed is found, the runtime loads msi.dll to bring back the missing assembly. <SID> appears to be the security identifier of the user that the process is running as. <File Path> is the full path to the file, but any '\' is substituted with a ''. <Strong Name> is the full assembly strong name. "Global" refers to the Global Assembly Cache (GAC).

Unfortunately, in the case that the assembly is found the Windows Installer obviously scans the same registry keys again looking for the assembly. I won't hide the fact that I don't like these sorts of shenanigans – by making assumptions about the behavior of the Windows Installer, the Windows Installer team is now essentially prevented from making changes to the way that MsiProvideAssemby works without a corresponding update to the .NET runtime. But what's done is done.

So how do you create an installer that will cause assembly probing to query the Windows Installer and bring back missing assemblies? Well, I'm sure that one of the commercial installers will do the right thing, but my preferred solution is just to use WiX, given that it's free.

So, let's build another sample from scratch. Firstly, we'll create an assembly called Binding.exe. Here's the code:

using System;
using System.Reflection;
using System.Globalization;
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyKeyFile("binding.snk")]
namespace Binding
{
 class MainClass
 {
  static void Main(string[] args)
  {
   AssemblyName myAssemblyName = new AssemblyName();
   myAssemblyName.Name = "ClassLibrary1";
   myAssemblyName.Version = new Version("1.0.0.0");
   myAssemblyName.SetPublicKeyToken(new byte[]
    {0x32, 0xea, 0x68, 0xe6, 0x16, 0x01, 0x2a, 0xe3});
   myAssemblyName.CultureInfo = new CultureInfo("");
   Assembly ass = Assembly.Load(myAssemblyName);
   MethodInfo method =
    ass.GetType("ClassLibrary1.Class1").GetMethod("Method1");
   method.Invoke(null, null);
  }
 }
}

Just to make things interesting this code builds an AssemblyName and loads an assembly explicitly. Note that everything should be strongly named in order for things to work properly. Don't forget to set the CultureInfo. The code for ClassLibrary1.dll is just:

using System;
using System.Reflection;
using ClassLibrary2;

[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyKeyFile("binding.snk")]

namespace ClassLibrary1
{
  public class Class1
  {
    public Class1()
    {
    }
   
    public static void Method1()
    {
      Console.WriteLine("ClassLibrary1.Class1.Method1");
      Class2.Method2();
    }
  }
}

ClassLibrary1 has an implicit dependency on the assembly ClassLibrary2.dll. The code for ClassLibrary2.dll is:

using System;
using System.Reflection;
using ClassLibrary2;[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyKeyFile("binding.snk")]namespace ClassLibrary1
{
  public class Class1
  {
    public Class1()
    {
    }
    
    public static void Method1()
    {
      Console.WriteLine("ClassLibrary1.Class1.Method1");
      Class2.Method2();
    }
  }
}

OK, now for the installer code:

<?xml version="1.0" encoding="utf-8"?>
  <Wix xmlns=http://schemas.microsoft.com/wix/2003/01/wi>
  <Product Id="DEB8151D-1EEB-4549-A5AD-405C4CD28085" Name="BindingProduct" Language="1033" Version="0.0.0.0" Manufacturer="Microsoft Corporation">
  <Package Id="511F5323-27B5-47C1-A488-8F92ED38EFEB" Description="Binding Test Product" Comments="Used for testing assembly binding" InstallerVersion="200" Compressed="yes" />
  <Media Id="1" Cabinet="PRODUCT.CAB" EmbedCab="yes" />
  <Directory Id="TARGETDIR" Name="SourceDir">
    <Directory Id="ProgramFilesFolder" Name="PFiles">
      <Directory Id="BindingProductDirectory" Name="ProdDir" LongName="BindingTest">
        <Component Id="BindingComponent" Guid="3A505CEA-C035-4F4C-A7CD-C0CAE01F492F">
          <File Id="BindingExeConfigFile" Name="BINDIN01.CON" LongName="Binding.exe.config" DiskId="1" src="Binding.exe.config" mce_src="Binding.exe.config"/>
          <File Id="BindingExeFile" Name="BINDING.EXE" LongName="Binding.exe" DiskId="1" src="Binding.exe" mce_src="Binding.exe" Assembly=".net" KeyPath="yes" AssemblyApplication="BindingExeConfigFile" AssemblyManifest="BindingExeFile" Vital="yes"/>
        </Component>
        <Component Id="ClassLibrary1Component" Guid="F6E5ECF8-EBD8-42C1-8BC0-9A3E7CAF985C">
          <File Id="ClassLibrary1DllFile" Name="CLASSL01.DLL" LongName="ClassLibrary1.dll" DiskId="1" src="ClassLibrary1.dll" mce_src="ClassLibrary1.dll" Assembly=".net" KeyPath="yes" AssemblyApplication="BindingExeConfigFile" AssemblyManifest="BindingExeFile" Vital="yes"/>
        </Component>
        <Component Id="ClassLibrary2Component" Guid="FCEAC462-3710-4A01-ACAB-B41F522ACD41">
          <File Id="ClassLibrary2DllFile" Name="CLASSL02.DLL" LongName="ClassLibrary2.dll" DiskId="1" src="ClassLibrary2.dll" mce_src="ClassLibrary2.dll" Assembly=".net" KeyPath="yes" AssemblyApplication="BindingExeConfigFile" AssemblyManifest="BindingExeFile" Vital="yes"/>
        </Component>
      </Directory>
    </Directory>
  </Directory>
  <Feature Id="BindingProductFeature" Title="Binding Test Product Feature" Level="1">
    <ComponentRef Id="BindingComponent" />
    <ComponentRef Id="ClassLibrary1Component"/>
    <ComponentRef Id="ClassLibrary2Component"/>
  </Feature>
  </Product>
</Wix>

The key component of the installer is the use of the AssemblyApplication attribute on the File element. This attribute points to the Binding.exe.config file for the application. This is the file that .NET will look for to determine if your application is installed before calling the Windows Installer bring back any missing assemblies.

Compile the three assemblies, then the installer. Run the installer. No navigate to the folder C:\Program Files\BindingTest\. Run Binding.exe. It outputs the two method names. Now for the magic. Delete ClassLibrary1.dll and ClassLibrary2.dll. Go on, don't be shy. Done that? Now run Binding.exe again. Instead of an AssemblyLoadException the program should run as if nothing had happened. Check again and the two DLL's are magically back. A self repairing application. Pretty cool, eh?

Lastly, I built a small program to test things out called CheckInstalled.cs, as follows:

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Reflection;
using System.Text;

// After running msiexec /i WixSetup.msi the command:
//
// CheckInstalled ClassLibrary1 "C:\Program Files\BindingTest\Binding.exe"
//
// Should return the path to ClassLibrary1.dll. See one of the registry keys:
//
// HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Managed\
//   <SID>\Installer\Assemblies\Global\<Strong Name>
// HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Managed\
//   <SID>\Installer\Assemblies\<File Path>
// HKCU\Software\Microsoft\Installer\Assemblies\Global\<Strong Name>
// HKCU\Software\Microsoft\Installer\Assemblies\<File Path>
// HKLM\SOFTWARE\Classes\Installer\Assemblies\Global\<Strong Name>
// HKLM\SOFTWARE\Classes\Installer\Assemblies\<File Path>

class MainClass
{
  [DllImport("msi.dll", CharSet=CharSet.Auto, SetLastError=true)]
  static extern uint MsiProvideAssembly(
    string assembly, string context, int mode, int info,
    System.Text.StringBuilder path, ref int pathSize); 
 
  public static void Main(String[] args)
  {
    if (args.Length < 1)
    {
      Console.WriteLine(
        "USAGE: {0} <assemblyName> [<installPath>]",
        Path.GetFileName(Assembly.GetExecutingAssembly().Location));
      Console.WriteLine(
        "\nAssembly name is just a full or partial assembly name. " +
        "The <installPath> is case sensitive");
      return;
    }
    
    if (args.Length > 1)
      Console.WriteLine(
        "Checking for installation of assembly '{0}' at install path '{1}'",
        args[0], args[1]);
    else
      Console.WriteLine("Checking for installation of assembly '{0}'",
        args[0]);
  
    try
    {
      Console.WriteLine(EnsureAssemblyIsInstalled(
        args[0], args.Length < 2 ? null : args[1]));
    }
    catch (SystemException e)
    {
      Console.WriteLine("ERROR: " + e.Message);
    }
  }
 
  public static string EnsureAssemblyIsInstalled(
    string assemblyName, string installPath)
  {
    int buffer = 1024;
    StringBuilder path = new StringBuilder(1024);  
    uint code = MsiProvideAssembly(assemblyName, installPath,
      0, 0, path, ref buffer);    
    if (code != 0)
      throw new SystemException("Windows Installer error " +
        code.ToString());   
      return "Found '" + path.ToString() + "'";
  }
}

Run this program as follows:

CheckInstalled ClassLibrary1 "C:\Program Files\BindingTest\Binding.exe.config"

It should report that ClassLibrary1 was found. Delete it and run the application again and you'll see Windows Installer bring the file back, this time with a user interface. This UI is suppressed by the .NET runtime.

So next time you see something hiding underneath a rock, go and check it out. It might be nothingit could be a nasty smelly bug, or it could just be a hidden gem.

If you have been using NUnit to write your unit tests, and you have access to Visual Studio 2005 Team System you'll want to consider upgrading your NUnit tests to the VS Unit Test Framework, which I'll call VSUnit for short. There is a walkthrough that you should read to get an idea of the VSUnit product features.

I recently upgraded some old unit tests to see what was involved, and it was really straightforward. First I'll make some assumptions. You have a VS 2003 solution that you've recently upgraded to VS 2005 and which contains some NUnit test projects. I'll also assume you are using C# as your development language.

Here's a rundown of what I did:

  1. Add a new Test Project to your solution (Test menu -> New Test -> Unit Test). This is going to add a Solution Items folder containing .vsmdi and .testrunconfig files. It should look something like the image on the below in your Solution Explorer.

  2. I usually delete the AuthoringTests.txt and ManualTest.mht files, the Additional Test Attributes #region, and all the // TODO: and /// comments.
  3. Drag all your .cs files from your old NUnit project. This will copy them into the VSUnit project.
  4. Back-up your old NUnit project somewhere, then remove the project from the solution and delete or rename the project directory on disk.
  5. At this point you should also rename your new VSUnit project. If you right click Rename it, bear in mind that that it sadly won't update the directory name. You'll have to close the solution, rename the directory in Explorer and open the solution .sln file in Notepad and change the directory before TestProject1.csproj.
  6. When you're done with the tidy-up you are ready to start converting the test code. The diagram below shows the basic conversions. Just follow the conversions, 1 through 7. li>When you're done with the tidy-up you are ready to start converting the test code. The diagram below shows the basic conversions. Just follow the conversions, 1 through 7.


    Note that VSUnit also has support for assembly wide setup and cleanup which you might want to make use of. But just focus on the conversion for now. Note the change to a static method for the class setup and cleanup, and the addition of the TestContext parameter.
  7. When you're done with step 6, try to compile your project. If it compiles, you are ready to run it and see what happens! Select the test then Test -> Start Selected Test Project Without Debugger.

Note that VSUnit also has support for assembly wide setup and cleanup which you might want to make use of. But just focus on the conversion for now. Note the change to a static method for the class setup and cleanup, and the addition of the TestContext parameter.

When you're done with step 6, try to compile your project. If it compiles, you are ready to run it and see what happens! Select the test then Test -> Start Selected Test Project Without Debugger.

You're probably wondering about all those Asserts in your NUnit tests. The really good news is that the Assert class exists in VSUnit and the methods signatures and behavior are by in large the same. The compiler will tell you where there are problems.

Other things to note. You can debug your unit tests just like with NUnit by setting breakpoints in the unit test and then selecting Test -> Start Selected Test Project With Debugger. The TestContext parameter also lets you do some really cool stuff when setting up and configuring your tests.

You can take an existing set of unit tests and easily run the methods in specific orders by adding one or more Ordered Tests to your unit test project. The only way to do this in NUnit was to alphabetize your test method names!

There's lots of other fun stuff available too. My favorite feature by far is to turn on coverage profiling for the tests, by going to Test -> Edit Test Configurations then select Coverage and select the assemblies you want to check the coverage for. I think you'll soon agree that these two tools were made for each other.

Boy, I seem to spend all my time in the deep dark recesses of the CLR. OK, this post is a little bit out there; I hope someone finds it useful.

There's a problem with doing instrumented profiling of applications in Visual Studio 2005 Team System that comes up when working with strongly named assemblies. The problem is quite simply that instrumentation breaks the strong name signature of your assemblies. And in working around this problem, I'll show you how you can actually be hiding start-up performance problems in your .NET applications.

Let us recap how assembly signing works in .NET. First, you create a public/private encryption key pair using sn.exe -k or similar. Then you take your assemblies files and you sign them. This signing can be done with the sn.exe tool or more transparently by using the /keyfile or /keycontainer command line options of the C# or VB compilers.

Basically what the signing process actually does is to compute a cryptographic hash from the metadata, intermedate language (IL) and other parts of your assembly file. It then takes this hash and encrypts it using the private key from your key pair. Finally this signature blob and your public key are embedded in the assembly. Now you can send your assembly out into the world safe in the knowledge that no one can mess with it.

Any CLR that your assembly gets presented to can now validate that your assembly has not been tampered with. This is done by first calculating the same hash that the signing process calculated. Then, the CLR takes the signature blob from the assembly and uses the public key to reverse the encryption performed by your private key. This will only work if the public key is truly the partner of the private key used to encrypt the signature. Hence the importance of keeping the private key, uh, private. If all is well, the decrypted signature and the calculated hash will match, and Bob's your Uncle!

Note: The 16 character public key token that you see as part of the full name of a strongly named assembly is not the signature hash referred to above. It is just a hash of your assemblies public key which makes the key a little easier to work with. For example, the public key token

7e16da7b2e1bcfa6

is merely a hash of the public key

0024000004800000940000000602000000240000525341310004000001000100e91eb19e092741
e85e7eda996ba566d491be9a300b2c26a8ce015398c87529f76075385c960bbc4dcf24ece0d1ad
a9f312dd0a42538d490be9228148645ecf5f45aadc12404af4c07d4b8060056b4724174ecc7d65
c65ac455a7321f71adfa8be981b02e9c5bd721d588395fa212c4a79cc007cf3a60738a0777614e
15b990a5

Which would you rather type?

So signing allows tampering to be caught, but it doesn't prevent it. And there's a back door into the signing process using the sn tool that allows signature verification to be turned off for assemblies containing a specific public key. In VSTS performance (and coverage) instrumentation works by inserting additional calls at the basic block boundaries in your applications IL and then rewriting the application binaries to disk. We call this static instrumentation because it's done before the program runs. Thus it breaks strong names signatures by changing the hash that will be computed when the assembly is verified by the CLR.

OK, big deal right? What's all this got to do with performance profiling? Well, the issue is that validating assembly signatures takes time. All that cryptographic stuff is computationally expensive. So, if you turn off verification for an assembly using something like:

sn -Vr *,b5f241dcf0cb58c4

Then you are potentially hiding a performance problem that might occur when assemblies are fully signed and signing is not disabled. Now, before we go running screaming into the hills, there's one caveat that has to do with the the global assembly cache (GAC). Putting an assembly in the GAC requires that the assembly be strongly named. The full signature verification for assemblies in the GAC is actually only performed when the assembly is inserted into the GAC, and not on subsequent loads of the assembly. So in that sense, there's actually a small performance boost to be had by putting assemblies in the GAC. But all other strongly named assemblies will incur a verification check the first time they are loaded into an AppDomain.

For what types of applications might turning off strong naming incur an artificial performance boost? Well, in practice this is only likely to be a problem for applications that depend on very large numbers of strongly named private assemblies, and even then it's only going to skew the applications start-up time. Applications that load and unload assemblies via additional AppDomains might also incur some penalty, but then I suspect that other factors such as memory fragmentation and disk I/O are much more likely to cause performance noise that will hide this issue.

So, what is test signing? Test signing is a halfway point between fully signed/fully verified and unsigned/unverified. It's a new feature in .NET 2.0. With test signing you can create a throw away test key that you can use to sign your assemblies. The CLR verification process then goes through the full set of computations to verify the assembly hash against the test public key, resulting in the same performance hit as if the assembly were fully signed. However, the assembly doesn't actually contain the public key for the test key pair, but the real public key for the assembly. In this way the strong name of the assembly is not broken. The public key for the test key pair actually lives in the registry.

OK, time for an example. Let's say you just generated a new project called ConsoleApplication1. Let's say that it starts of fully signed using your companies real key pair. Now we instrument:

>vsinstr ConsoleApplication2.exe
Microsoft (R) VSInstr Post-Link Instrumentation 8.0.50727.0
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   C:\Projects\ConsoleApplication2\bin\Debug\ConsoleApplication2.exe
Original file backed up to C:\Projects\ConsoleApplication2\bin\Debug\ConsoleApplication2.exe.orig

Warning VSP2001 : C:\Projects\ConsoleApplication2\bin\Debug\ConsoleApplication2.exe is a strongly
named assembly.  It will need to be re-signed before it can be executed.
Successfully instrumented file C:\Projects\ConsoleApplication2\bin\Debug\ConsoleApplication2.exe.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header
flags have been updated to reflect this.

Note that the message about having to sign the assembly before it can be executed, should really say something like "before it can be strong name verified". Now you create your test key and test sign the assembly.

>sn -k TestKey.snk
Microsoft (R) .NET Framework Strong Name Utility  Version 2.0.50727.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Key pair written to TestKey.snk

And you'll need just the public key too, so lets get that now:

>sn -p TestKey.snk TestPublicKey.snk.
Microsoft (R) .NET Framework Strong Name Utility.  Version 2.0.50727.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key written to TestPublicKey.snk

You need the public key token that's currently in the assembly, which remember is from the real public key pair and is part of the real strong name of the assembly.

>sn -T ConsoleApplication2.exe
Microsoft (R) .NET Framework Strong Name Utility  Version 2.0.50727.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key token is b5f241dcf0cb58c4

Now you do the test signing:

>sn -TS ConsoleApplication2.exe TestKey.snk
Microsoft (R) .NET Framework Strong Name Utility  Version 2.0.50727.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Assembly 'ConsoleApplication2.exe' successfully re-signed

And finally you register the test signing public key:

>sn -Vr *,b5f241dcf0cb58c4 TestPublicKey.snk
Microsoft (R) .NET Framework Strong Name Utility  Version 2.0.50727.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Verification entry added for assembly '*,b5f241dcf0cb58c4'

Sadly the message that the last step gives you is identical to the one you get for regular verification skipping registration. Go to the registry and look under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\StrongName\Verification\*,b5f241dcf0cb58c4

(or whatever your public key token is). You'll see a TestPublicKey value containing the public key token.

Now you can run your performance tests and measure the real cost of checking all those assembly strong name signatures.

I seem to have come across this several times in the past week or two, so it's worth making a note.  In .NET 2.0 XmlValidatingReader is obsolete.  You need to do something like the following in order to create a validating reader now:

XmlReaderSettings settings = new XmlReaderSettings();

settings.ValidationType = ValidationType.Schema;
                
XmlReader reader = XmlReader.Create(new XmlTextReader(pathToYourXmlFile), settings);

settings.Schemas.Add(null, 
  new XmlTextReader(Assembly.GetExecutingAssembly().GetManifestResourceStream("NameOfYourSchema.xsd")));

// Do your XML thing here...
                
reader.Close();

All the interesting configuration for the validation now hangs off the XmlReaderSettings.  Stuff like the ValidationEventHandler event. 

By the way, don't forget that XmlReader supports line and column information via the IXmlLineInfo interface.  If you catch an XmlSchemaException you first use the C# as operator to cast the reader to an IXmlLineInfo (just to be safe - in an exception handler you might not be sure where the reader came from), check for null, and then if you have an interface call HasLineInfo to see if you got far enough along in the reading to have any line information.

One thing that I tend to do with validating readers is to process the XML in two phases, like some compilers. In the first phase I focus on validation, and spit out as many errors as possible by catching the validation event.  Then, if all went well you can re-read the XML with a non-validating reader and you only have to focus on the remaining symantic errors that are not or cannot be caught by your schema.   Obviously, not a good idea if you are reading gobs and gobs of XML.  Your call. 

Lastly, don't forget to "manually" check your XML's default namespace is correctly referring to one of your validation schemas, because if its not your validating reader won't load the correct schema from your schema set and therefore won't give you any errors.   Something like this:

while (!validatingReader.EOF)
{
  validatingReader.Read();
  
  if (validatingReader.NodeType == XmlNodeType.Element &&
    validatingReader.Name == "RootElement")
  {
    string defaultNamespace = validatingReader.LookupNamespace("");
    bool found = false;
     
    foreach (string namespaceUri in validNamespaceUris)
      if (defaultNamespace == namespaceUri)
      {
        found = true;
        break;
      }
              
    if (!found)
    {
      IXmlLineInfo lineInfo = (IXmlLineInfo)validatingReader.Reader;
      
      throw new XmlSchemaException(String.Format("{0}({1},{2}): error: The default namespace '{3}' is invalid", 
        validatingReader.BaseURI, lineInfo.LineNumber, lineInfo.LinePosition, defaultNamespace), null);
    }
  
    // Now validate the rest of the file, without the check
    while (validatingReader.Read());
  }
}

I was over at Building 20 on the Microsoft campus the other day and I noticed that all the lab machines were running a really cool 3D screen saver.  I just had to have it!  And here it is.

A little more clicking leads me to a collection of Really Slick Screen Savers.

You find yourself facing a seemingly broken logon screen.  Every character you type seems to come out as some other character.  Perhaps you know JohnL-S had something to do with configuring that system.  You type ASDF and it comes out AOEU.   What’s going on?

It was brought to my attention this morning that many of you cannot type on a Dvorak keyboard.   It’s OK – don’t feel bad.  You should not think any less of yourselves because of this.   I’m here to help.  Plus I want JoC to stop yelling at me.  So I bring you this handy dandy Dvorak Keyboard Self Help Guide.

Even though I have been programming computers since I was about 12, I was a late touch typist (like 20-something).  I could not kick the bad habits that I had developed as a hunt-and-peck typist.   Then I stumbled upon the Dvorak layout in a copy of Mavis Beacon Teaches Touch Typing.  Because it was completely different, it forced me to actually stop looking at the keyboard, and in about 6 weeks I had surpassed my previous two fingered typing speed.   It was tough, but I got some encouragement from Randy Cassingham of This is True fame, who played a part in getting the Dvorak drivers built into Windows in the first place.

OK then.  Now, even though I type on a Dvorak keyboard layout, I do it on standard QWERTY keyboards.   It’s hard to find Dvorak keyboards, and the stickers are too expensive.  Plus, you’re not supposed to be looking at the keyboard as you type anyway.  If you know what you want to type, but you just don’t know what keys to press, the following command line program can help.  Just pass it the string you are trying to type and it will spit out the exact QWERTY keys to press to get what you want.

using System;

namespace DvorakOnQwerty
{
  class Program
  {
    static String dvorak = " `1234567890[]',.pyfgcrl/=\\aoeuidhtns-;qjkxbmwvz~!@#$%^&*(){}\"<>PYFGCRL?+|AOEUIDHTNS_:QJKXBMWVZ";
    static String qwerty = " `1234567890-=qwertyuiop[]\\asdfghjkl;'zxcvbnm,./~!@#$%^&*(){}QWERTYUIOP{}|ASDFGHJKL:\"ZXCVBNM<>?";
        
    static void Main(string[] args)
    {
      String[] temp = Environment.CommandLine.Split(new char[] {' '}, 2);
      
      if (temp.Length == 2)
      {
        String cmdLine = temp[1];
                    
        for (int i = 0; i < cmdLine.Length; i++)
          Console.Write(qwerty[dvorak.IndexOf(cmdLine[i].ToString(), StringComparison.Ordinal)]);
      }
    }
  }
}

So if you type:

> DvorakOnQwerty Dvorak Rules!
H.soav Ofpd;!

Then you would just type the encoded sequence of letters exactly as shown on a QWERTY keyboard, paying attention to case of course.  There’s also a good picture here so you can see what’s actually going on.  As you can see, the number keys are the same, which helps.

Here are some other things about the way I use Dvorak.  I always set up my machines with both the QWERTY and Dvorak drivers installed, and I set the keyboard shortcuts so that CTRL+SHIFT+1 switches to Dvorak and CTRL+SHIFT+2 switches to QWERTY.   Remember that since around Windows 2000 the decision was made to make the keyboard mapping input thread specific, so you have to change it for every process that you use.  That’s right.  For some reason it doesn’t change the keyboard for every program on the desktop.  I guess that’s a feature so that you and I can sit in front of a computer alternately wrestling the keyboard out of each others hand, whilst still being able to touch type at full speed.   This is rather nutty, and I don’t get it either, but there you go.  

I also set always set things up so that the language bar icon appears on the Taskbar.   Click on the icon and select the input keyboard layout you want, like so.

Last but not least, I make life easy on all you non-Dvorak users and don’t change the default user keyboard layout, so the logon desktop is in the QWERTY layout.  BUT, occasionally Windows Terminal Services loses its mind and switches this user/desktop to Dvorak at random times.  I don’t know why this happens.  It’s could be a bug or it could be some strange configuration problem.  I’ve also seen Internet Explorer switch keyboard layouts at random times too.  But that’s OK, because now you have the handy-dandy program above, a helpful link and some other stuff, so you can rescue yourself.

I’m really happy that I learned Dvorak, but it seems to get me no end of grief wherever I go, so I leave you with this final request.   Nd lgid ks H.soav f;do;!

Ahhhhh.  There's nothing quite like pasting a big chunk of code into your blog.  It makes the prose writing part so much easier!  My last entry was about counting bits.  Today I put that in the context of the original problem.

Todays entry comes from an internal support alias issues from week or so ago.  First a bit of background.  By off roading (using the command line) with the VSTS profiling tools, it's perfectly feasible to instrument an assembly, resign it or turn on verification skipping for that assemblies public key, and place the assembly in the GAC and/or in the native image cache.  Our client was trying to instrument an assembly and place it in the native image cache.  So they did:

> vsinstr xyz.dll
> ngen xyz.dll

to update the native image cache.  They then proceeded to get nasty AV’s when they ran their application.  Richard did some investigation and the reason turned out to be that the native image in the cache no longer matched the instrumented image.  Why?  Because in this case ngen was using the Mvid of the assembly to determine if the assembly that needed recompiling, and vsinstr doesn’t change the Mvid.

The CLI Specification – Partition II – Metadata states that, “While the [runtime] itself makes no use of the Mvid, other tools (such as debuggers, which are outside the scope of this standard) rely on the fact that the Mvid almost always differs from one module to another.”  So the ngen use of the Mvid makes sense, and the fact that trace instrumentation doesn’t change this value is probably something we need to look at in future.  But that’s how it’s going to be for Whidbey.

One way around the problem is to force full update of the native image cache using:

> ngen update

Which is a good fix in many cases.  But this can be very time consuming if there are lots of other out of date assemblies in the cache.

So in the meantime, for those of you that run across this problem in our tools, or any other tool that you feel should be changing the Mvid and doesn’t, I put together the following code to help you out.   Remember that doing this will invalidate any strongly named signed assemblies.  Cut and paste it into a file called chgmvid.cpp and compile it into a tool call chgmvid.exe:

#include <windows.h>
#include <tchar.h>
#include <stdio.h>
#include <crtdbg.h>
#include <corhdr.h>
#include <math.h>
#include <rpc.h>

// This utility allows you to change the MVID on an assembly. It's solves the problem that occurs because VSINSTR
// doesn't change the MVID after instrumenting. This screws up NGEN and other tools like Reflector for .NET that
// use the MVID to try and figure out if the assembly has been modified.
//
// So for example, you VSINSTR System.Data.dll, do a GACUTIL -i System.Data.dll then run NGEN System.Data.dll.
// NGEN does not refresh the image. It does if you do NGEN UPDATE, but that might end up recompiling a whole bunch
// of stuff. So, just run CHGMVID on the assembly first and the former NGEN will refresh the .NI image.

#pragma pack(8)
#include <dbghelp.h>

#pragma comment(lib, "dbghelp.lib")
#pragma comment(lib, "rpcrt4.lib")

#define Align(a, x) ((a) * ((((x) - 1) / (a)) + 1))

typedef struct
{
HANDLE hFile;
HANDLE hMap;
PVOID pBase;
PIMAGE_NT_HEADERS pNtHeader;
} LOADED_IMAGE_DATA, *PLOADED_IMAGE_DATA;

#define RVA2VA(pdata, rva) ImageRvaToVa((pdata)->pNtHeader, (pdata)->pBase, (rva), NULL)

// See http://blogs.msdn.com/johnls/archive/2005/08/09/449655.aspx
int BitCount(UINT64 n)
{
n = (n & 0x5555555555555555) + ((n >> 1) & 0x555555555555555);
n = (n & 0x3333333333333333) + ((n >> 2) & 0x3333333333333333);
n = (n & 0x0F0F0F0F0F0F0F0F) + ((n >> 4) & 0x0F0F0F0F0F0F0F0F);
return (int)(n % 0xFF);
}

BOOL IsBitSet(UINT64 n, int b)
{
return (b < 0x40) && (n & ((UINT64)1 << b));
}

void UnmapImage(PLOADED_IMAGE_DATA pImageData)
{
if (NULL != pImageData->pBase)
UnmapViewOfFile(pImageData->pBase);

if (NULL != pImageData->hMap)
CloseHandle(pImageData->hMap);

if (INVALID_HANDLE_VALUE != pImageData->hFile)
CloseHandle(pImageData->hFile);
}


HRESULT MapImage(wchar_t* pImage, PLOADED_IMAGE_DATA pImageData)
{
pImageData->hFile = NULL; // A bit redundant perhaps
pImageData->hMap = NULL;
pImageData->pBase = NULL;
pImageData->pNtHeader = NULL;

HRESULT hr = S_OK;

if (INVALID_HANDLE_VALUE != (pImageData->hFile = CreateFile(pImage, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL)))
{
if (NULL != (pImageData->hMap = CreateFileMapping(pImageData->hFile, NULL, PAGE_READWRITE, 0, 0, NULL)))
{
if (NULL != (pImageData->pBase = MapViewOfFile(pImageData->hMap, FILE_MAP_ALL_ACCESS, 0, 0, 0)))
{
if (NULL != (pImageData->pNtHeader = ImageNtHeader(pImageData->pBase)))
return S_OK;
else
hr = HRESULT_FROM_WIN32(ERROR_BAD_FORMAT);
}
}
}

if (hr == S_OK)
hr = HRESULT_FROM_WIN32(GetLastError());

UnmapImage(pImageData);
return hr;
}

int _cdecl wmain(int argc, wchar_t* argv[])
{
if (argc < 2)
{
wprintf(
L"usage: chgmvid <image-file> [/c]\n\n"
L"/c If not specified, MVID will be displayed but not changed.\n");
return 0;
}

bool changeMvid = false;

if (argc == 3)
{
if (argv[2][0] == '/' || argv[2][0] == '-')
{
if (argv[2][1] == 'c' || argv[2][1] == 'C')
changeMvid = true;
}
}

LOADED_IMAGE_DATA imageData;
HRESULT hr = MapImage(argv[1], &imageData);

if (FAILED(hr))
{
wprintf(L"ERROR: Unable to load image (%u)\n", HRESULT_CODE(hr));
return 1;
}

PIMAGE_DATA_DIRECTORY pDataDirectories = NULL;

switch (imageData.pNtHeader->OptionalHeader.Magic)
{
case IMAGE_NT_OPTIONAL_HDR32_MAGIC:
pDataDirectories = reinterpret_cast<PIMAGE_NT_HEADERS32>(
imageData.pNtHeader)->OptionalHeader.DataDirectory;
break;

case IMAGE_NT_OPTIONAL_HDR64_MAGIC:
pDataDirectories = reinterpret_cast<PIMAGE_NT_HEADERS64>(
imageData.pNtHeader)->OptionalHeader.DataDirectory;
break;

default:
wprintf(L"ERROR: Not a PE/PE+ file\n");
return 1;
}

// Make sure there is a CLR header
_ASSERTE(pDataDirectories != NULL);

if (0 == pDataDirectories[IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR].VirtualAddress ||
0 == pDataDirectories[IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR].Size)
{
wprintf(L"ERROR: No CLR header\n");
return 1;
}

// Get the CLR header
PIMAGE_COR20_HEADER pCorHeader = reinterpret_cast<PIMAGE_COR20_HEADER>(
RVA2VA(&imageData, pDataDirectories[IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR].VirtualAddress));

if (NULL == pCorHeader)
{
wprintf(L"ERROR: Could not get CLR header\n");
return 1;
}

BYTE* pMetadataRoot = (BYTE*)RVA2VA(&imageData, pCorHeader->MetaData.VirtualAddress);

if (*(UINT32*)pMetadataRoot != 0x424A5342)
{
wprintf(L"ERROR: Could not identify metadata root\n");
return 1;
}

wprintf(L"\nCLR Target: %S\n", (CHAR*)(pMetadataRoot + 16));

UINT32 paddedVersionLength = Align(4, *(UINT32*)(pMetadataRoot + 12));
UINT16 numStreams = *(UINT16*)(pMetadataRoot + 16 + paddedVersionLength + 2);
BYTE* pStreamHeader = (pMetadataRoot + 16 + paddedVersionLength + 4);
BYTE* pTildeStream = NULL;
BYTE* pGuidStream = NULL;

for (UINT16 i = 0; i < numStreams; i++)
{
BYTE* pStreamData = pMetadataRoot + *(UINT32*)(pStreamHeader);
char* pStreamName = (char*)(pStreamHeader + 8);

if (strcmp(pStreamName, "#~") == 0)
pTildeStream = pStreamData;
else if (strcmp(pStreamName, "#GUID") == 0)
pGuidStream = pStreamData;

pStreamHeader += 8 + Align(4, strlen((char*)(pStreamHeader + 8)) + 1);
}

if (NULL == pTildeStream || NULL == pGuidStream)
{
wprintf(L"ERROR: Cannot find #~ and/or #GUID streams\n");
return 1;
}

if (*(pTildeStream + 4) != 2 && *(pTildeStream + 5) != 0)
{
wprintf(L"ERROR: Expected version number 2.0 in metadata stream\n");
return 1;
}

BOOL bigStringHeap = (*(pTildeStream + 6) & 0x01) != 0;
BOOL bigGuidHeap = (*(pTildeStream + 6) & 0x02) != 0;

// Check that the module table is present. It's table 0x00. The CLI spec. is not clear about the format
// of these bit arrays. They are actually stored as little endian 64-bit integers.
if (!IsBitSet(*(UINT64*)(pTildeStream + 8), 0x00))
{
wprintf(L"ERROR: No module table present\n");
return 1;
}

unsigned presentTables = BitCount(*(UINT64*)(pTildeStream + 8));

// Remember, the module table is number 0x00, so it's easy to find.
BYTE* pModuleTable = pTildeStream + 24 + 4 * presentTables;

// Indexes into the GUID table are 1 based
UINT32 guidIndex = bigGuidHeap ?
(*(UINT16*)(pModuleTable + 2 + (bigStringHeap ? 4 : 2)) - 1) :
(*(UINT32*)(pModuleTable + 2 + (bigStringHeap ? 4 : 2)) - 1);
BYTE* pMvid = pGuidStream + guidIndex * 16;
GUID mvid;

memcpy(&mvid, pMvid, sizeof(mvid));

wprintf(L"Current MVID: {%08X-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X}\n",
mvid.Data1, mvid.Data2, mvid.Data3,
(int)mvid.Data4[0], (int)mvid.Data4[1], (int)mvid.Data4[2], (int)mvid.Data4[3],
(int)mvid.Data4[4], (int)mvid.Data4[5], (int)mvid.Data4[6], (int)mvid.Data4[7]);

if (changeMvid)
{
UuidCreate(&mvid);

memcpy(pMvid, &mvid, sizeof(mvid));

wprintf(L"New MVID: {%08X-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X}\n",
mvid.Data1, mvid.Data2, mvid.Data3,
(int)mvid.Data4[0], (int)mvid.Data4[1], (int)mvid.Data4[2], (int)mvid.Data4[3],
(int)mvid.Data4[4], (int)mvid.Data4[5], (int)mvid.Data4[6], (int)mvid.Data4[7]);
}

// Clean-up
UnmapImage(&imageData);

return 0;
}
 
Page view tracker