Random ramblings about Technology

SQL Server–HOW-TO: quickly retrieve accurate row count for table

martijnh1 — Thu, 15 Jul 2010 20:24:00 GMT

Recently, I’ve been involved in a very interesting project in which we need to perform operations on a table containing 3,000,000,000+ rows. For some tooling, I needed a quick and reliable way to count the number of rows contained within this table. Performing a simple

SELECT COUNT(*) FROM Transactions

operation would do the trick on small tables with low IO, but what’s the ‘best’ way (quick and reliable) to perform this operation on large tables?

I searched and found different answers, which I note here so it might be of use to someone… (My table was called ‘Transactions’)

Index	Query	Comment
1	SELECT COUNT(*) FROM Transactions	Performs a full table scan. Slow on large tables.
2	SELECT CONVERT(bigint, rows) FROM sysindexes WHERE id = OBJECT_ID('Transactions') AND indid < 2	Fast way to retrieve row count. Depends on statistics and is inaccurate. Run DBCC UPDATEUSAGE(Database) WITH COUNT_ROWS, which can take significant time for large tables.
3	SELECT CAST(p.rows AS float) FROM sys.tables AS tbl INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2 INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int) AND p.index_id=idx.index_id WHERE ((tbl.name=N'Transactions' AND SCHEMA_NAME(tbl.schema_id)='dbo'))	The way the SQL management studio counts rows (look at table properties, storage, row count). Very fast, but still an approximate number of rows.
4	SELECT SUM (row_count) FROM sys.dm_db_partition_stats WHERE object_id=OBJECT_ID('Transactions') AND (index_id=0 or index_id=1);	Quick (although not as fast as method 2) operation and equally important, reliable.

SQL Server 2008 R2 - generating INSERT statements for a table

martijnh1 — Thu, 24 Jun 2010 12:19:46 GMT

Learned something new today. Maybe it’s well-known, but new to me, you can just run “C:\Program Files (x86)\Microsoft SQL Server\90\Tools\Publishing\1.4\SqlPubWiz.exe”. This nifty little tool can generate INSERT script very easily. Pictures tell the story :)

Reading MSMQ from within SSIS packages

martijnh1 — Thu, 10 Jun 2010 19:54:30 GMT

Recently, I had to implement an SSIS package which imports some data into SQL Server from MSMQ. It turns out reading MSMQ from within an SSIS package is more difficult than it seems at first. I’ve documented how to get things working because I could find little other documentation.

When reading data from MSMQ, there’s multiple steps to take:

Setting up a queue, if not yet available
Set up a connection with MSMQ by adding a connection manager to the SSIS package
Adding and configuring a Message Queue Task
Putting data onto the queue
Running the package to validate the data is read from the queue

Setting up a queue

In Windows Server (2008 R2 in my case), create a new queue. It doesn’t really matter whether this queue is transactional or not.

Setting up a connection with MSMQ from within an SSIS package

First things first, let’s start with setting up a connection manager for MSMQ. When creating a new SSIS package, this is the view you get:

In the “Connection Managers” area, right-click and choose “new connection”. From the list that is displayed, select the “MSMQ” type and click “Add”.

A new dialog pops up, in which you can configure the connection to be made. I choose “.\Private$\mh” (a local queue called “mh”). Click “Test” to ensure the connection can be made and click “OK”.

The connection manager is set up.

Adding a Message Queue Task

Adding a Message Queue task to the Control Flow is easy, drag and drop it from the Toolbox on the left (press CTRL-ALT-X if it’s not displayed) to the Control Flow surface, like displayed below.

To configure the Message Queue Task, double-click the Message Queue task’s icon on the left of the Task in the Control Flow. A new dialog pops up in which we’ll configure the task. Select the connection manager we set up previously for the “MSMQConnection” property within the dialog and change the default “Send message” value for the “Message” property to “Receive message”.

Change to the “Receive” configuration page by click “Receive” on the left-hand side of the dialog. Choose whether the message should be removed from the queue after it is read by SSIS. I’ve chosen not to timeout reading the queue. Set the “MessageType” property to “String message to variable”.

Click on the drop-down for the “Variable” property and choose “New variable". This will show a new dialog, in which a new variable can be defined. For this example, let’s use the information shown below and click “OK”.

After closing the dialog, we’ll be returned to the “Message Queue Task editor” dialog, which shows the newly created variable.

The Message Queue Task is configured to read messages from the “.\Private$\mh” queue.

Putting data onto the queue

For SSIS to be able to read the queue, ensure you label everything you put onto the queue with the label ‘String message’. If you don’t the message will not be picked up by SSIS.

Validation the package

Pressing F-5 (Debug) from within the Business Intelligence Development Studio will run the package, effectively picking up a message from the queue if there is any.

HTH, happy trails!

TFS 2010 beta 2: connecting from Visual Studio 2008

martijnh1 — Mon, 11 Jan 2010 10:52:59 GMT

Connecting to Team Foundation Server 2010 beta 2 from Visual Studio 2008 is easy: from Team Explorer, type the full URL in the "add server" dialog, e.g.:

http://mytfsserver:8080/tfs/defaultcollection

Please ensure you've (re-)installed Visual Studio 2008 SP1. If you installed Team Foundation Client after applying SP1, reinstall SP1.

Active Directory - Adding a user to a group from a non domain-joined computer throws PrincipalException

martijnh1 — Wed, 06 Jan 2010 11:30:23 GMT

When using the new System.DirectoryServices.AccountManagement namespace, one might be inclined (as was I) to use the following code to add a user to a group (exception and comments handling removed):

using (GroupPrincipal groupPrincipal = 
    GroupPrincipal.FindByIdentity(
        domainContext, 
        System.DirectoryServices.AccountManagement.IdentityType.Name, 
        groupName))
{
    if (groupPrincipal != null)
    {
        using (UserPrincipal userPrincipal = 
            UserPrincipal.FindByIdentity(
                domainContext, 
                System.DirectoryServices.AccountManagement.IdentityType.UserPrincipalName, 
                UPN))
        {
            if (userPrincipal != null)
            {
                groupPrincipal.GroupScope = GroupScope.Global;
                groupPrincipal.Members.Add(userPrincipal);
                groupPrincipal.Save();
            }
        }
    }
}

This works just fine if your machine is joined to the domain you're trying to provision. However, if your machine is not joined, the groupPrincipal.Save(); call throws a PrincipalException with an error code 1355 ("Information about the domain could not be retrieved (1355)").

Joining the domain solves this issue.

So what if joining the domain isn't an option?

In this case I found resorting to good old System.DirectoryServices fixes the issue, using the following code fragment:

using (UserPrincipal userPrincipal =
    UserPrincipal.FindByIdentity(domainContext, UPN))
{
    if (userPrincipal != null)
    {
        using (GroupPrincipal groupPrincipal =
            GroupPrincipal.FindByIdentity(domainContext, groupName))
        {
            if (groupPrincipal != null)
            {
                if (!userPrincipal.IsMemberOf(groupPrincipal))
                {
                    string userSid = string.Format("<SID={0}>", userPrincipal.ToSidString());
                    DirectoryEntry groupDirectoryEntry =
                        (DirectoryEntry)groupPrincipal.GetUnderlyingObject();
                    groupDirectoryEntry.Properties["member"].Add(userSid);
                    groupDirectoryEntry.CommitChanges();
                }
            }
        }
    }
}

ToSidString is an extension method which translates the "objectSid" property. Thanks to Richard for the way better implementation! (ignore "Enforce")

public static string ToSidString(this Principal principal)
{
    Enforce.IsNotNull<Principal>(principal, "principal");

    SecurityIdentifier sid = principal.Sid;
    if (sid == null || sid.BinaryLength == 0)
    {
        return null;
    }

    byte[] buffer = new byte[sid.BinaryLength];
    sid.GetBinaryForm(buffer, 0);

    string[] hexBytes = Array.ConvertAll(buffer, b => b.ToString("X2", NumberFormatInfo.InvariantInfo));

    return string.Concat(hexBytes);
}

HTH!

Unity v1.2 - using enum values from configuration; writing and configuring a custom type converter

martijnh1 — Wed, 06 Jan 2010 08:34:24 GMT

Recently, I used Unity v1.2, which in my opinion is a great product. I had difficulty, however, with the configuration for a scenario I encountered. I needed an enum value to be passed to a constructor. Trying as I might, there was no way Unity was accepting my typeAlias, as such:

<unity>
  <typeAliases>
    <typeAlias 
      alias="contextType" 
      type="System.DirectoryServices.AccountManagement.ContextType, 
            System.DirectoryServices.AccountManagement, Version=3.5.0.0, 
            Culture=neutral, PublicKeyToken=b77a5c561934e089" />
  </typeAliases>
  <containers>
    <container name="identitystores">
      <types>
        <type name="primary" type="IIdentityStore" mapTo="ActiveDirectoryIdentityStore" >
          <lifetime type="singleton" />
          <typeConfig 
            extensionType="Microsoft.Practices.Unity.Configuration.TypeInjectionElement, 
                           Microsoft.Practices.Unity.Configuration">
            <constructor>
              <param name="context" parameterType="contextType">
                <value
                  value="Domain"
                  type="contextType"  />
              </param>
            </constructor>
          </typeConfig>
        </type>
      </types>
    </container>
  </containers>
</unity>

For the life of me, I couldn't figure out what I did wrong. I asked in an internal discussion list, and someone mentioned writing a plain old TypeConverter... Right he was, getting my enum value to work was pretty easy. The TypeConverter (as always, comments removed for clarity and provided AS IS, etc.):

public class ContextTypeTypeConverter : TypeConverter
{
    public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
    {
        return sourceType.GetType() == typeof(string);
    }

    public override bool CanConvertTo(ITypeDescriptorContext context, Type destinationType)
    {
        return destinationType == typeof(ContextType);
    }
    
    public override object ConvertFrom(
        ITypeDescriptorContext context, 
        CultureInfo culture, 
        object value)
    {
        return Enum.Parse(typeof(ContextType), (string)value);
    }

    public override object ConvertTo(
        ITypeDescriptorContext context, 
        CultureInfo culture, 
        object value, 
        Type destinationType)
    {
        return Enum.GetName(typeof(ContextType), value);
    }
}

and modified configuration file:

<unity>
  <typeAliases>
    <typeAlias 
     alias="contextType" 
      type="System.DirectoryServices.AccountManagement.ContextType, 
            System.DirectoryServices.AccountManagement, Version=3.5.0.0, 
            Culture=neutral, PublicKeyToken=b77a5c561934e089" />
  </typeAliases>
  <containers>
    <containername="identitystores">
      <types>
        <typename="primary" type="IIdentityStore" mapTo="ActiveDirectoryIdentityStore" >
          <lifetimetype="singleton" />
          <typeConfig 
           extensionType="Microsoft.Practices.Unity.Configuration.TypeInjectionElement, 
                           Microsoft.Practices.Unity.Configuration">
            <constructor>
              <param name="context" parameterType="string">
                <value
                 value="Domain"
                  type="contextType"
                  typeConverter="Microsoft.AccountManagement.Extensions.ContextTypeTypeConverter, 
                                 Microsoft.AccountManagement.Extensions, Version=1.0.0.0, 
                                 Culture=neutral, PublicKeyToken=4fd564a94067c21"/>
              </param>
            </constructor>
          </typeConfig>
        </type>
      </types>
    </container>
  </containers>
</unity>

That's it, now everything works just fine. HTH someone out there :)

WCF extensions: The type 'x' registered for extension 'y' could not be loaded.

martijnh1 — Mon, 04 Jan 2010 12:26:14 GMT

Quick note: if you configure a custom extension in WCF, don't forget to add it as a reference to your project. If you don't configuration will run just fine, but at runtime, the following exception occurs:

System.Configuration.ConfigurationErrorsException: The type 'x' registered for extension 'y' could not be loaded. (config information, line z)

   at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult)
   at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
   at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName)
   at System.Configuration.ConfigurationManager.GetSection(String sectionName)
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetSectionFromConfigurationManager(String sectionPath)
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetAssociatedSection(ContextInformation evalContext, String sectionPath)
   at System.ServiceModel.Configuration.ServicesSection.UnsafeGetSection()
   at System.ServiceModel.Description.ConfigLoader.LookupService(String serviceConfigurationName)
   at System.ServiceModel.ServiceHostBase.ApplyConfiguration()
   at System.ServiceModel.ServiceHostBase.InitializeDescription(UriSchemeKeyedCollection baseAddresses)
   at System.ServiceModel.ServiceHost..ctor(Type serviceType, Uri[] baseAddresses)
   at Microsoft.Tools.SvcHost.ServiceHostHelper.CreateServiceHost(Type type, ServiceKind kind)
   at Microsoft.Tools.SvcHost.ServiceHostHelper.OpenService(ServiceInfo info)

Unit testing: COM object that has been separated from its underlying RCW cannot be used

martijnh1 — Thu, 31 Dec 2009 10:20:39 GMT

Let's say you're writing unit tests for code which uses COM objects. You want some initialization and some cleanup to be done. So you set up your test:

private static MyStore store;

[ClassInitialize()]
public static void MyClassInitialize(TestContext testContext)
{
    store = new MyStore("some value");
}

[ClassCleanup]
public static void MyClassCleanup()
{
    store.DoWork(someArgument);
}

However, as your 'store' instance uses a COM object, you will get a test run error, saying: "System.Runtime.InteropServices.InvalidComObjectException: COM object that has been separated from its underlying RCW cannot be used..".

The way to solve this issue is to open up your testrunconfig file using the XML editor and adding the highlighted element to it:

<?xml version="1.0" encoding="UTF-8"?>
<TestRunConfiguration name="Local Test Run" id="fbbfe81f-97fa-4e33-bde1-00b3827b933e" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2006">
  <ExecutionThread apartmentState="MTA" />
  <Description>This is a default test run configuration for a local test run.</Description>
  <TestTypeSpecific />
</TestRunConfiguration>

The ExecutionThread element will instruct the test to use the MTA threading model. This way, the static instance of "store" can be shared across tests.

Office 2010 Development – Outlook main window ribbon ID

martijnh1 — Tue, 01 Sep 2009 09:51:00 GMT

I’m currently writing an Outlook add-in and I’m looking into getting this compatible with both Office 2007 and Office 2010. One of the new features in Outlook 2010 is the main window Ribbon.

Using VSTO, you can create your own Ribbon, which can be automatically merged with default Ribbons by setting the correct OfficeId property for your Ribbon. Office 2007 has the list of available OfficeId values available here, but I has some trouble locating a list for the Office 2010 Technical Preview.

After some trial and error, I figures out there’s an easy way to find out what the different IDs for a specific Ribbon and it’s tabs is.

Office 2010 let’s users customize the Ribbon from within Office.

Clicking ‘Customize the Ribbon’ brings up the customization dialog:

On the right-hand side of the dialog, right-click the ‘RSS’ tab within the expanded ‘Home’ tab and choose ‘Rename’. Name it e.g. ‘testRSS’. After customizing, press OK on the dialog and re-open the customization dialog once more (if you don’t close and re-open the dialog, the following steps will not lead to the desired result).

Now press the ‘Import/Export’ button and select ‘Export all Ribbon and Quick Access Toolbar customizations’. Save the file.

The exported Office UI file has a .exportedUI extension. Rename this to .xml and open up the file. You’ll see something like this:

<mso:cmd app="MSOutlook" dt="0" />

<mso:customUI xmlns:mso="http://schemas.microsoft.com/office/2007/10/customui">

<mso:ribbon>

<mso:qat/>

<mso:tabs>

<mso:tab idQ="mso:TabMail">

<mso:group idQ="mso:GroupRss" label="testRSS"/>

</mso:tab>

</mso:tabs>

</mso:ribbon>

</mso:customUI>

Note the ‘TabMail’ ID. This is the current (it’s a technical preview, things will probably change in the future) identifier for the Ribbon Tab you’ve just modified.

OK, we’ve got the Tab OfficeID. However, for a Ribbon to show up, you also need to specify RibbonType property. Normally, using VSTO, you get a drop down with all possible values for this property within Visual Studio. However, the main window Ribbon in Outlook is new, so there’s no value for this Ribbon within the list. Luckily, you can also type in your own value. Now all that’s left is finding out the name of the main Ribbon and we’d be set.

I looked around, but the value for the main Ribbon was a mystery to me. Finally, I decided Outlook probably had the string in memory during runtime, so I could use Process Explorer to find the correct value. As you probably know, Process Explorer has a tab called ‘strings’ on the properties dialog of a process. This tab enumerates all strings within the image of the process and lists them here.

I searched for ‘Microsoft.Outlook.’ (all values in the regular drop down start with this) and found ‘Microsoft.Outlook.Explorer’. Searching further, the other values we all know and love came up. However, ‘Microsoft.Outlook.Explorer’ was the only truly new value.

I tried using this value for the RibbonType and low and behold, my add-in appeared on the ‘RSS’ tab of the main Ribbon.

Key take-away: Using a RibbonType of ‘Microsoft.Outlook.Explorer’ and ‘TabMail’ as your ControlId, you can place your add-in on the main ribbon of the Outlook window.

UPDATE: in Visual Studio 2010 beta 2, you can just select the RibbonType mentioned ;-)

Disclaimer: This has been tested on the Technical Preview of Office 2010. There are no guarantees whether these IDs will be stable as we move forward with the product.

PrintMonitor – A C# print spooler monitor

martijnh1 — Wed, 05 Aug 2009 12:40:34 GMT

Recently, a colleague of mine had a requirement of getting information on the number of pages being printed by users. Monitoring a printer is done by calling FindFirstPrinterChangeNotification, waiting for the returned WaitHandle to be signaled and then calling

FindNextPrinterChangeNotification to see why the handle got signaled and act accordingly. Sadly, there don’t seem to be any samples of this in C# out there, so I created one in my spare time. The attached application monitors all printer notification fields possible. It still has some issues like not disposing things correctly, but that’s left as an exercise to the reader, as I don’t have more time to spent.

Upon starting the application, the Load event of the main form hooks up the event and handles (it currently takes the first printer it can find):

printerWaitHandle = new ManualResetEvent(false);

PRINTER_INFO_2[] printers = enumPrinters(PrinterEnumFlags.PRINTER_ENUM_NAME);
string printerName = printers[0].pPrinterName;
OpenPrinter(printerName, out printerHandle, 0);

printerChangeHandle = FindFirstPrinterChangeNotification(printerHandle, (int)PRINTER_CHANGES.PRINTER_CHANGE_DELETE_JOB, 0, notifyOptions);
printerWaitHandle.SafeWaitHandle = new SafeWaitHandle(printerChangeHandle, true);

printerChangeNotificationHandle = ThreadPool.RegisterWaitForSingleObject(

printerWaitHandle,

new WaitOrTimerCallback(PrinterNotifyWaitCallback), printerWaitHandle, -1, true);

The callback method (PrinterNotifyWaitCallback) interprets the change:

public void PrinterNotifyWaitCallback(object state, bool timedOut)
{
    // BUG: if we Thread.Sleep here for the amount of time it takes to spool the print job, we only get the right notification for number of pages.

    int changeReason = 0;
    IntPtr pNotifyInfo = IntPtr.Zero;

    bool nxt = FindNextPrinterChangeNotification(printerChangeHandle, out changeReason, null, out pNotifyInfo);

...

This works quite well. There is a bug in there somewhere which causes the following behavior: the total number of pages is first set to 0 when spooling the print job and only on the *subsequent* print job being printed is the correct number of pages signaled. If you put a Thread.Sleep (don’t ;-)) in the callback for the amount of time it takes the spooler to spool the print job, the correct number of printed pages is displayed directly (without the ‘0’ entry and within the actual print job it belongs to). If anyone can find the bug, please let me know!

Sourcecode attached, as always, no warranties of any kind, express, implied, etc.

Sourcecode

Outlook Add-in: Replying to people who voted

martijnh1 — Tue, 04 Aug 2009 22:19:21 GMT

Recently, I asked my colleagues if they were interested in beta testing some software I’m building for internal use. I used voting buttons to enable easy tracking of replies. When I wanted to reply to all people who voted “Yes” on my request, I discovered there’s no real way of just replying to a set of people who voted a certain way. Meet “ReplyToVoteTrackingType” (what’s in a name?); it’s an add-in which allows you to easily reply to anyone who casted a certain vote by selecting that vote:

The code for the add-in is pretty straight-forward, when loading the Ribbon we fill the dropdown menu with the items from the original vote and hookup the same event handler (we don’t discriminate) to each added button:

// Retrieve the MailItem we're currently viewing.
currentItem = ((InspectorClass)this.Context).CurrentItem as MailItem;
// Retrieve all possible voting options from when we sent out the e-mail.
string[] votingOptions = currentItem.VotingOptions.Split(new char[] { ';' });

// Set up a button for each of the options, labeled <option> ('yes', 'no', etc.).
foreach (string votingOption in votingOptions)
{
    RibbonButton btn = new RibbonButton() { Label = votingOption, SuperTip = string.Format(Resources.SuperTip, votingOption) };
    btn.Click += clickEventHandler;
    mnuReplyToTracking.Items.Add(btn);
}

Then, when the end user clicks on any of the voting options (e.g. “Yes” as the picture above shows), we check who responded in this fashion and add them to a newly created e-mail:

StringBuilder recipients = new StringBuilder();

// For each recipient,
foreach (Recipient recipient in currentItem.Recipients)
{
    // if they responded, and the response is what the users choose for a target audience,
    if (recipient.AutoResponse != null && string.Compare(recipient.AutoResponse, ((RibbonButton)sender).Label, true, CultureInfo.InvariantCulture) == 0)
    {
        // add the recipient to the addressee list.
        recipients.AppendFormat("{0};", recipient.Name);
    }
}

// Set up a new e-mail to the addressees.
MailItem mail = ((InspectorClass)this.Context).Application.CreateItem(OlItemType.olMailItem) as MailItem;
mail.To = recipients.ToString();
// Convenience: set the mail subject to be "RE: <subject of voting e-mail>".
mail.Subject = string.Format(Resources.ReplyToVoteEmail, currentItem.Subject);
// Display the e-mail, non-modal.
mail.Display(false);

That’s all; now, when the end users wants to reach his/her audience given a certain vote, it’s as easy as click the right voting reply!

Source code is attached, as is an installer. As always, NO WARRANTIES.

Source code
Installer

Feedback requested: Planning on Enterprise Library 5 has started!

martijnh1 — Thu, 29 Jan 2009 09:50:00 GMT

Planning on Enterprise Library 5 has started! See Gregori's blog for more information if you want your EntLib needs to be in the next release: http://blogs.msdn.com/agile/archive/2009/01/26/enterprise-library-5-what-s-on-your-wish-list.aspx!

Reflector add-in: Favorites

martijnh1 — Thu, 15 Jan 2009 14:47:00 GMT

Every developer knows about and uses Reflector ;-) I use it a lot to learn about the inner workings of products. When analyzing what goes where and why, it's easy to lose track (jumping from method to method to base class, etc.). Bookmarks are quick and easy, but don't allow for a description to be added. If I need to get back to a certain point within my analysis two weeks (ok, ok, two minutes ;-)) later, the Type declaration from the bookmarks doesn't help me.

Meet Reflector Favorites!

I wrote Reflector Favorites. It allows for developers to:

Easily tag a location within Reflector, adding a description that works for you.
Import/Export favorites between computers.
Share favorites (they work as long as you have the right assemblies opened in Reflector).

Reflector Favorites in action.

I hope someone finds this useful. As always, use at your own risk, no warranties of any kind.

Suppressing Personal Identifiable Information (PII) in WCF log files

martijnh1 — Mon, 18 Aug 2008 23:00:00 GMT

Given the vast amount of information transported between systems in today's enterprise environments, a lot of which is considered either private (addresses, social security numbers, credit card information, mortgage and pension plans, etc) or of high business value, information security is key.

From the release of WCF, it has brought us tremendous flexibility and allowed development teams to be almost completely agnostic of the way information is transported between point A to point B, focusing on adding business value instead.

Another great feature WCF brings us is runtime troubleshooting. In WCF, an administrator can simply enable logging to troubleshoot an environment which is in production.

Although a great feature for troubleshooting, Personal Identifiable Information (PII) you care about deeply while on the wire (using transport and message-level security to protect it) is available in plain text if logging is turned on at the server side.

WCF's logging features currently filter out specific PII by default (only if the message is not malformed). However, this functionality is not exposed to the developer.

Consider the following scenario:

You want the administrator to be able to monitor messages for auditing, logging or troubleshooting purposes;
certain information is considered PII and should not show up in the log files.

You could write your own logging subsystem and hook it in to your services. This takes a lot of work, is error prone, needs custom maintenance, documentation, etc.

My proposal for the scenario:

Use a behavior which
- introspects the published ServiceEndPoints for published OperationDescriptions and the used DataContracts,
- finds public properties on the DataContract marked as PII;
- instructs the WCF logging runtime to ignore the contents of the specified message elements.
configure the behavior through code (this is not generally considered a best practice, but the security needs of the service outweighs the flexibility needs of configuration);

Let's have a look at how the solution works:

Take a DataContract, e.g. PurchaseOrder and apply an attribute (PII) on each property which represents PII.

Next, add a PIIProtectionBehavior attribute to your service implementation class

You're done. The PIIProtectionBehavior attribute is also an IServiceBehavior implementation which performs the introspection on the EndPoints.

The result when logging is turned on:

As you can see, 'CreditCardnumber' is no longer logged, it just displays '<-- Removed -->' .

I've attached the sample code, hope it's useful!

Ofcourse, the normal disclaimers apply, the accompanied sample is for educational purposes only, it has not gone through rigorous testing, might contain bugs due to limited testing (me ;-)), etc. Please see the sourcecode for further remarks.

Adding a new BizTalk Server Isolated Host needs a new Application Pool

martijnh1 — Wed, 23 Jul 2008 12:29:00 GMT

Recently, we tried creating a secondary Isolated Host to separate receive and send processing for WCF. After reconfiguring the servers, we were faced with the "Receive location for address '' not found. (The BizTalk receive location may be disabled)". Puzzled, we looked around for configuration issues. After some reading (http://technet.microsoft.com/en-us/library/aa559328.aspx), we remembered a new isolated host needs a new application pool. Adding another application pool to the mix the the new host fixed our trouble... Jotted down for posterity ;-)

Running Virtual Machines under alternate user credentials on a Domain Controller

martijnh1 — Fri, 02 May 2008 16:43:51 GMT

Introduction

Recently, I was experimenting with Team Foundation Server 2008, setting it up with SSL and running it within a test domain. For the Team Foundation Server, I couldn't use the host installation as it runs Windows Server 2003 R2 x64 edition and TFS doesn't support (see the TFS installation guide, under 'Overview of Team Foundation Architecture', '64-bit Support in Team Foundation') this setup in a single server scenario. So, I installed Virtual Server 2005 R2 SP1, installed TFS on top, configure SSL and voila, a working TFS setup.

When I shut down the host computer, I want the virtual machine of TFS to save state and come back up again when the host is turned on again. This can be done in Virtual Server using an alternative use running the actual Virtual Machine instance. Trying to set this up in a least privilege way proved not to be obvious from documentation, so this blog entry documents what I did for posterity (and myself ;-)).

Outline

Here's what needs to be done in order to circumvent obscure error messages:

1. We need a new user group so the account we'll use doesn't belong to default domain users and inherits no permissions.

2. We need a new user to run the Virtual Machine instance, the user should belong to this group only.

3. The user needs to be given 'Local on locally' rights.

4. Permissions need to be set on the folder containing network configuration information for Virtual Server.

5. Permissions need to be set on the folder containing the Virtual Machine and the actual files (*.vhd, .vmc) making up the Virtual Machine.

6. The Virtual Machine needs to be configured to use the new user.

Instuctions

Let's configure the necessary elements.

1. Create a new group

Create a new group within the Active Directory Users and Computers MMC snapin (found under Administration Tools):

2. Create a new user

Create the user which is to run the specific Virtual Machine (done from the same MMC snap-in), add it to the 'empty' group, set the 'empty' group as it's Primary Group and remove the 'Domain Users' group from the list. After this, your user overview should resemble this image:

3. Assign 'Log on locally' rights to the user

This step is critical in getting the Virtual Machine running under the new user context. Steps to achieve the appropriate right setting are described here. If the user doesn't receive the 'Log on locally' right, Virtual Server will display an error: 'The account name and password could not be set. The virtual machine account could not be set. The account has not been granted the requested logon type.'. Make sure the security policy is updated before using the account.

4. Set permissions for the used virtual network interface

Now that we have the user and it's group configured, let's set the appropriate permissions for the user to make use of the configured network. Mind you, these instructions will only allow the user to use the network it's given access to from the instructions, the 'local network only' et al will not work as the user has no rights on the files used for those configurations.

The virtual network configuration files for Virtual Server are stored in %SystemDrive%\Documents and Settings\All Users\Application Data\Microsoft\Virtual Server\Virtual Networks. The user needs permissions as specified in order to use the network. If permissions are set incorrectly, the Virtual Machine will not have network access.

5. Set permissions on Virtual Machine folders and files

In order to start up the Virtual Machine, save state, etc, the new user needs access rights on the folder storing the actual files making up the Virtual Machine as well as specific rights on the Virtual Machine files. The folder structure containing my Virtual Machine files is:

%SystemDrive%\vms\<Virtual Machine>

First, let's set the appropriate rights on the folder hosting all Virtual Machines:

Now, let's set the permissions for the appropriate Virtual Machine ('tfs') folder:

Lastly, set up permissions for the Virtual Machine files (my TFS has 3: tfs.vmc, tfs.vhd and sql.vhd):

6. Configure the Virtual Machine

All permissions are set, we're ready to configure Virtual Server to run the Virtual Machine under the new user context:

Wrap up

That's it! We've configured the Virtual Machine to run under a user context which has the least amount of privileges it needs to function correctly. The Virtual Machine will save it's state when the host is shut down and will automatically turn back on when the host comes back online.

HTH

Fixing exception messages on the .NET Compact Framework 3.5

martijnh1 — Thu, 03 Jan 2008 19:04:00 GMT

FWIW: I recently ran into trouble while developing for the .NET Compact Framework, v3.5. Upon all exceptions, the framework would tell me:

"An error message is available for this exception but cannot be displayed because these messages are optional and are not currently installed on this device. Please install ‘NETCFv35.Messages.EN.wm.cab’ for Windows Mobile 5.0 and above or ‘NETCFv35.Messages.EN.cab’ for other platforms. Restart the application to see the message."

Interpreting the message as 'I have to run the NETCFv35.Messages.EN.wm.cab (I'm developing an application for WM6) on my device', I copied the file from C:\Program Files (x86)\Microsoft.NET\SDK\CompactFramework\v3.5\WindowsCE\Diagnostics onto my mobile device and ran the installation.

To no avail, nothing changed and I was stuck. After installing the 'Power Toys for .NET Compact Framework 3.5', however, I configured logging using the '.NETCF Logging Configuration' application and found the following error for the loader log:

'Attempt to load [\Program Files\SomeApp\System.SR.dll] has failed (err 0x80001000).'

Renaming the file 'SYCCFA~1.001', included in the aforementioned CAB file into System.SR.dll and adding it as a reference in my application proved to fix the issue. I now had readable exception messages.

Just jotting this down as there's nothing to be found on the web regarding the issue that I could find.

Testing WCF service implementations

martijnh1 — Thu, 17 May 2007 14:21:00 GMT

When writing WCF implementations, or any code for that matter, testing is an important part of the process. This week, I was building tests for implemented WCF services and had difficulty aligning all the external dependencies of the tests (running hosts, configuration settings, behavior component references, etc). As a result, I set up my WCF services to be self-hosted services, run from the context of the test itself. This has several advantages, and some pitfalls. The advantages:

· Isolation, the test has external dependencies only to the actual service implementation.

· No manual setup running hosts to ensure they are available, the test ‘just works’.

The pitfall might be:

· Configuration settings could become out-of-sync. Because the host of the service implementation runs within the context of the test and thereby it’s AppDomain, you need to configure the service ABCs within the test configuration file. This could mean over time you’d get configuration settings for the tests no longer in sync with the actual host.

However, because the scope of the tests is under your control and so is it’s configuration, the described pitfall might not pose an issue.

In order for the test to host it’s own instance of the services to be tested I created a generic container. It’s a simple, but effective, class which can easily host different service types (all tracing and the IDisposable implementation removed for clarity):

/// <summary>

/// Defines a generic implementation of a runnable background host.

/// </summary>

/// <typeparam name="T">The service to run within the host.</typeparam>

public class RunnableBackgroundHost<T> : IDisposable where T : class, new()

{

internal AutoResetEvent stopEvent = new AutoResetEvent(false);

/// <summary>

/// Starts the host on a new thread.

/// </summary>

public void Run()

{

ThreadPool.QueueUserWorkItem(

new WaitCallback(delegate(object state)

{

using (ServiceHost host = new ServiceHost(typeof(T)))

{

host.Open();

stopEvent.WaitOne();

host.Close();

}

}));

}

/// <summary>

/// Stops the running host.

/// </summary>

public void Stop()

{

stopEvent.Set();

}

Using this class, testing services become quite easy. Set up the host before starting the test:

[System.Diagnostics.CodeAnalysis.SuppressMessage(

"Microsoft.Performance",

"CA1812:AvoidUninstantiatedInternalClasses"), TestClass]

class HostInit

{

protected static RunnableBackgroundHost<Service1> service1;

protected static RunnableBackgroundHost<Service2> service2;

[AssemblyInitialize]

public static void AssemblyInitalize(TestContext context)

{

service1 = new RunnableBackgroundHost<Service1>();

service2 = new RunnableBackgroundHost<Service2>();

service1.Run();

service2.Run();

}

[AssemblyCleanup]

public static void AssemblyCleanup()

{

service1.Stop();

service2.Stop();

}

And set up your test, using a generated proxy (svcutil) and assuming the service is alive:

[TestClass]

public class Service1Tests : IDisposable

{

private Service1Client service1Client;

[TestInitialize]

public void TestInitialize()

{

service1Client = new Service1Client();

}

[TestMethod]

public void TestService1Method1()

{

Service1Request request = new Service1Request();

request.Id = "1";

Service1Response response = service1Client.Method1(request);

Assert.IsNotNull(response, ErrorMessages.NullResponse);

Assert.AreEqual<int>(

response.Field,

ErrorMessages.NonMatchingId);

}

ErrorMessages is a simple resource file containing the messages the test would display in case of failure.

What the test framework does in this case:

· Call the static method HostInit.AssemblyInitalize marked with [AssemblyInitialize].

· For every class within the test marked with [TestClass]

o Call any instance method marked with [TestInitialize].

o Run any method marked with [TestMethod].

o ...

· Call the static method HostInit.AssemblyCleanup marked with [AssemblyCleanup].

Within the test configuration file, configure both the service host and the client:

<system.serviceModel>

<endpoint

address="http://localhost:8889/Services/Service1"

binding="basicHttpBinding"

contract="IService1" />

</client>

<endpoint

address="http://localhost:8889/Services/Service1"

binding="basicHttpBinding"

contract="IService1" />

</service>

</services>

</system.serviceModel>

That’s the basic setup. You can now easily test different services by defining the tests for it in it’s own class using it’s own generated client proxy and adding the service to the HostInit class. You could test behaviors by adding them to the configuration file and writing a test for them if they can be tested from the outside.

Just press SHIFT+ALT+X to run the test in the debugger. No console windows popping up, your test just executes :).

WCF Message validation (1) – WCF samples to the rescue

martijnh1 — Sun, 13 May 2007 14:12:00 GMT

Recently, I needed to validatie messages going into a WCF service. It turned out the WCF samples are a rich source of knowledge and I found a sample which performs XMLSchema-based validation in \TechnologySamples\Extensibility\MessageInspectors.

The sample contains a few classes:

ClientValidationException.cs

Defines exceptions where validation fails clientside

SchemaValidationBehavior.cs

Defines an implementation of IEndpointBehavior in order to be an extender for defined endpoints

SchemaValidationBehaviorExtensionElement.cs

Inherits from BehaviorExtensionElement, used for configuration-time binding

SchemaValidationMessageInspector.cs

Contains the actual validation code, implementing IClientMessageInspector and IDispatchMessageInspector.

ValidationFault.cs

Defines FaultExceptions used within the code.

The sample uses a property called Schemas within it’s configuration, allowing us to define which schemas should be taken into account when validating.

</schemas>

</schemaValidator>

That’s great, but as you all probably know, WCF services have the ABC’s: Address, Binding and Contract. What I actually wanted the behavior to do was validate my message against the actual Contract of the endpoint the validator was working on.

In order for this to work, I did two thing:

1. I removed all “Schemas” configuration and arguments from the code.

2. I modified the SchemaValidationBehavior class to validate the actual contract.

The first steps should be self-explanatory, but the second I want to elaborate on. What I actually did was modify the methods ApplyClientBehavior and ApplyDispatchBehavior. The sample took the configured Schemas property to create the actual SchemaValidationMessageInspector:

SchemaValidationMessageInspector inspector =

new SchemaValidationMessageInspector(

schemaSet, validateRequest, validateReply, false);

I wanted to dynamically use the endpoint contract:

WsdlExporter exporter = new WsdlExporter();

exporter.ExportContract(endpoint.Contract);

SchemaValidationMessageInspector inspector =

new SchemaValidationMessageInspector(

exporter.GeneratedXmlSchemas,

validateRequest, validateReply, false);

After modifying both methods to perform this kind of setup, it was easy to configure validation based on the endpoint contract (some elements abbreviated for clarity):

<system.serviceModel>

<endpoint

address="SomeAddress"

binding="wsHttpBinding"

contract=" ISomeService"

behaviorConfiguration="SomeServiceEndpointBehavior" />

</service>

</services>

<SchemaValidator

validateRequest="True"

validateReply="True" />

</behavior>

</endpointBehaviors>

</behaviors>

<add

name="SchemaValidator"

type="SchemaValidationBehaviorExtensionElement, ..."/>

</behaviorExtensions>

</extensions>

</system.serviceModel>

That’s it, now we have a service endpoint validated based on it’s WSDL contract.

The sample code is attached for your enjoyment.

Shockwave Flash preview handler for Windows Vista and Outlook 2007.

martijnh1 — Wed, 02 May 2007 14:27:00 GMT

Based on the excellent article from Stephen Toub, I was inspired to write a Shockwave Flash preview handler. Basically, it's just a plain UserControl which hosts the ActiveX wrapper for the player. It works from both Outlook 2007 and Windows Vista. The installer is attached.

Programmatic BizTalk Server Orchestrations - code repost

martijnh1 — Fri, 06 Apr 2007 17:30:00 GMT

A previous post of mine had some sample code on creating BizTalk Server Orchestrations programmatically, opening up template based guidance packages. The sourcecode for the sample can now be found here.

BizTalk Server Pipeline Component Wizard moved

martijnh1 — Mon, 12 Mar 2007 11:17:00 GMT

As we are phasing out GotDotNet, projects need a new location to be hosted. I've moved the BizTalk Server Pipeline Component Wizard to it's new location. Please find it here.

CAB WPF XBAP implementation available (again)

martijnh1 — Tue, 06 Mar 2007 13:40:00 GMT

In the time to come, I'll be uploading packages available from my previous blog in the past because they got lost during my ISP transfer. First off, here's the CompositeUI Application Block XBAP WPF implementation, as I blogged about here.

Please note the passwords for both protected signing keys are empty within the solution.

This posting is provided "AS IS" with no warranties, and confers no rights.

Integration Masterclass Redelivery

martijnh1 — Wed, 14 Feb 2007 16:40:00 GMT

Microsoft Consulting Services in The Netherlands has developed a great integration masterclass, in which integration specialists compare Microsoft integration technologies and deliver real-world field experience in BizTalk Server implementations including best practices, tips and tricks. Check out the webpage at http://www.microsoft.com/netherlands/evenementen/integration.aspx for more information (Dutch content).

Ping!

martijnh1 — Mon, 08 Jan 2007 13:55:00 GMT

Welcome to my blog. My name is Martijn Hoogendoorn, I work for the Dutch Services dvision of Microsoft and will be posting my technical adventures on here. I've been blogging for some time now; previous posts can be found here (archive only).