Reading Twitter Data with C# and LINQ

I wanted to read Twitter.com search results (tweets) using C#.  I started by deciding that a tweet looks something like this:

    public class Tweet
    {
        public string Id { get; set; }
        public DateTime Published { get; set; }
        public string Link { get; set; }
        public string Title { get; set; }
        public Author Author { get; set; }
    }
    public class Author
    {
        public string Name { get; set; }
        public string Uri { get; set; }
    }

Next I needed a way to get the data from the web and into a collection of Tweet’s.  I defined a TweetStream type that would handle the following:

1. Downloading the data from Twitter.com
2. Deserializing the Tweet data
3. Remembering the high-watermark so that we always only get new tweets.

I stubbed out this:

    public class TweetStream
    {
        private string m_refreshUri;
        List<Tweet> m_tweets;
        public TweetStream(string queryUri)
        {
            m_refreshUri = queryUri;
            m_tweets = new List<Tweet>();
        }
        public List<Tweet> Tweets
        {
            get
            {
                return m_tweets;
            }
        }
        public void Refresh()
        {
            // TODO - download tweet information from Twitter.com,
            // populate the tweet collection with the new
            // tweet data, and store the new high-watermark
        }
    }

Now I just need to fill in Refresh.

Download the results from Twitter

The LINQ XDocument type makes this very easy.  What’s not clear here is that I am loading ATOM data using the Twitter ATOM search API.  The result stream will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:google="http://base.google.com/ns/1.0" xml:lang="en-US" xmlns:openSearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom" xmlns:twitter="http://api.twitter.com/">
  <id>tag:search.twitter.com,2005:search/TFS</id>
  <link type="text/html" rel="alternate" href="http://search.twitter.com/search?q=TFS"/>
  <link type="application/atom+xml" rel="self" href="http://search.twitter.com/search.atom?q=TFS"/>
  <title>TFS - Twitter Search</title>
  <link type="application/opensearchdescription+xml" rel="search" href="http://search.twitter.com/opensearch.xml"/>
  <link type="application/atom+xml" rel="refresh" href="http://search.twitter.com/search.atom?q=TFS&amp;since_id=1152416895"/>
  <twitter:warning>adjusted since_id, it was older than allowed</twitter:warning>
  <updated>2009-01-27T16:13:14Z</updated>
  <openSearch:itemsPerPage>15</openSearch:itemsPerPage>
  <openSearch:language>en</openSearch:language>
  <link type="application/atom+xml" rel="next" href="http://search.twitter.com/search.atom?max_id=1152416895&amp;page=2&amp;q=TFS"/>
  <entry>
    <id>tag:search.twitter.com,2005:1152416895</id>
    <published>2009-01-27T16:13:14Z</published>
    <link type="text/html" rel="alternate" href="http://twitter.com/bubbafat/statuses/1152416895"/>
    <title>All day TFS training.</title>
    <content type="html">All day &lt;b&gt;TFS&lt;/b&gt; training.</content>
    <updated>2009-01-27T16:13:14Z</updated>
    <link type="image/png" rel="image" href="http://static.twitter.com/images/default_profile_normal.png"/>
    <author>
      <name>bubbafat (Robert Horvick)</name>
      <uri>http://twitter.com/bubbafat</uri>
    </author>
  </entry>
  <entry>...</entry>
  <entry>...</entry>
  <entry>...</entry>
</feed>
And we load it like this: 

XDocument feed = XDocument.Load(m_refreshUri);

Populate the Tweet Collection

With the document loaded we can now start pulling out the data we want.  First we need to define the ATOM namespace:

XNamespace atomNS = "http://www.w3.org/2005/Atom";

Next we need to iterate over every “entry” element in the feed and extract the data we care about.  LINQ makes this a breeze (though perhaps a bit tough to debug and grok at first).

m_tweets = (from tweet in feed.Descendants(atomNS + "entry")
    select new Tweet
    {
        Title = (string)tweet.Element(atomNS + "title"),
        Published = DateTime.Parse((string)tweet.Element(atomNS + "published")),
        Id = (string)tweet.Element(atomNS + "id"),
        Link = tweet.Elements(atomNS + "link")
            .Where(link => (string)link.Attribute("rel") == "alternate")
            .Select(link => (string)link.Attribute("href"))
            .First(),
        Author = (from author in tweet.Descendants(atomNS + "author")
            select new Author
            {
                Name = (string)author.Element(atomNS + "name"),
                Uri = (string)author.Element(atomNS + "uri"),
            }).First(),
    }).ToList<Tweet>();

Store the High-watermark

Finally we want to load the high watermark so we don’t ask for data twice.  This is just a matter of loading the “refresh” link from the top-level feed elements.

m_refreshUri = feed.Descendants(atomNS + "link")
    .Where(link => link.Attribute("rel").Value == "refresh")
    .Select(link => link.Attribute("href").Value)
    .First();

Putting it all Together

Not to put it all together you can create a collection of TweetStream’s and refresh them in a loop – printing out the tweets as you find them.

using System;
using System.Threading;
using TwitterLib;
namespace TestHost
{
    class Program
    {
        static void Main(string[] args)
        {
            int totalTweets = 0;
            TweetStream[] tweetStreams = new TweetStream[] {
                new TweetStream("http://search.twitter.com/search.atom?q=TFS"),
                new TweetStream("http://search.twitter.com/search.atom?q=Team+Foundation+Server"),
                new TweetStream("http://search.twitter.com/search.atom?q=TFS2008"),
                new TweetStream("http://search.twitter.com/search.atom?q=TFS2005"),
            };
            while (true)
            {
                int currentTweets = 0;
                bool newTweets = false;
                foreach (TweetStream stream in tweetStreams)
                {
                    stream.Refresh();
                    foreach(Tweet tweet in stream.Tweets)
                    {
                        Console.WriteLine("{0}: {1}",
                            tweet.Author.TwitterId,
                            tweet.Title);
                        newTweets = true;
                        totalTweets++;
                        currentTweets++;
                    }
                }
                if (newTweets)
                {
                    Console.WriteLine("Loaded {0} more tweets", currentTweets);
                    Console.WriteLine("Loaded {0} total tweets", totalTweets);
                }
                else
                {
                    Console.WriteLine("No new tweets.");
                }
                DateTime nextCheck = DateTime.Now.AddMinutes(5);
                Console.WriteLine("Will check again at {0}", nextCheck.ToShortTimeString());
                Thread.Sleep(TimeSpan.FromMinutes(5));
            }
        }
    }
}

You may have noticed I used a property (Author.TwitterId) that is not in the code – it’s just a simple Regex:

public string TwitterId
{
    get
    {
        return s_idParser.Match(Name).Groups["twitterid"].Value;
    }
}
private static Regex s_idParser = 
    new Regex(@"^(?<twitterid>.*)\s+\((?<displayname>.*)\)");

Whats’ Next?

To be clear this is not production-ready code.  It is very optimistic, quite inefficient and a bit ugly.  Also it doesn’t do anything terribly interesting.

But in less than 100 lines you have an ugly, slow, optimistic Twitter feed reader (which could read any ATOM stream with just a few additional lines).

How I Use Hyper-V to Configure my Development Environment

Prior to using Hyper-V as part of my daily development and test efforts I wasted man weeks every year.  I lost days at a time configuring machines and recovering from updated NDP builds.  I wasted hours every week repeating the same steps over and over.

I will be bold enough to say that Hyper-V allows me to get at least 15% more work done each week on less than a third of the hardware I had before and I never have long downtimes due to my machine being down.  Even a total machine failure is not going to put me offline for longer than it takes to reinstall Windows and Office.

So how did I setup my environment?

Hardware

• Dell OptiPlex 755
• Intel Core 2 Quad (Q6600) @ 2.4 Ghz
• 8 GB RAM
• 250 GB HD (Primary)
• C: – 25GB (boot drive)
• F: – 225GB (data drive)
• 500 GB HD (Secondary)
• E: (data drive)
• 150 GB HD
• Marked Offline in Windows (you can do this through disk management – it must be a real drive, not just a logical partition)
• 500 GB Western Digital MyBook HD (external USB drive)
• Monitors
• Samsung SyncMaster 206BW (20” LCD)
• Philips 170B (17” LCD)
• MidiLand Speakers (el-cheapo junk speakers I found in a box somewhere)

The overall hardware investment is about $1300 (and that’s retail pricing that anyone can get).

To create my development environment I do the following (applying the latest patches at every step and CA eTrust is installed on every machine [physical and virtual])

Configure Primary Box

1. Install Windows 2008 (64bit) on the C: drive
2. Install Office 2007 on the C: drive
3. Install the Hyper-V services (RTM)
4. Create the folder F:\VM
5. Plug my Zune directly into the speakers – I don’t want to give up CPU/RAM for music to play.
6. Configure Windows Backup to backup my C: and F: drive to the external hard-drive (that’s why it does not have a drive letter – Windows takes over the drive).  I do nightly backups of my VMs.

Configure Development Box

My goal with the dev box is to optimize for quick resets when we take new builds of Visual Studio or the CLR.  Since I work on the VS team I take new builds every few weeks.  To do this I pre-install a lot of stuff, create some baseline snapshots and then mount the physical disk for my source enlistment.  Mounting physical disk is REALLY IMPORTANT.  You do NOT want your source code on a virtual drive because when you revert to an older snapshot you will lose it’s contents and that will both screw up your workspace contents (requiring a force sync) and may result in losing data (files that were added/edited but not checked in).  Also the physical drive means build performance is not impacted by a virtualized drive.  The virtual drives in Hyper-V (even the expandable ones) have great performance – but moving the build to a physical drive really helps keep things rolling.

One caveat – you cannot take snapshots with a physical drive mounted so you do not get live snapshots and you must unmount whenever you apply a previous snapshot (this is trivial, but for some reason the Apply operation won’t do it for you behind the scenes).

I can recover from an VS/NDP update in about 2 hours with this model.  On physical hardware it would mean formatting and starting over – costing about a day.

1. Create a new (127GB - Expandable) virtual disk at F:\VM\Disks
2. Create a new virtual machine (at F:\VM\Machines) called “HORVICKVM-DEV” with
1. Four Virtual Processors
2. 2 GB of RAM
3. Legacy Network Adaptor (So I can use RIS for network installs)Boot to RIS network boot
4. The drive you created in step #1
3. Install Windows 2003 (32 bit) [Choose whatever OS you want]
4. When it is finished installing shut the machine down and take a snapshot called “OS Installation Complete (OFF)”
1. I always put “OFF” or “LIVE” in my snapshot names so it’s clear to me what the running state of the snapshot is without having to look at the details or preview window.
5. In the VM configuration settings for the dev box
1. Change the network adaptor from Legacy Network Adaptor to “Network Adaptor”
6. Boot the Development VM
7. Install NDP 2.0, and 3.5 + SP1
8. Install Source Insight
1. I always want to have an editor on the box in the event I run into a VS bug I can’t work around.
9. Install WinDbg and configure it for use with our symbol and source server plus my favorite configuration tweaks.
1. I always want to have a debugger installed in case I need to debug the VS install (and frankly WinDbg + SoS is pretty damn powerful).
10. Shut Down the VM
11. Snapshot named “Editor and NDP 3.5+ Patched (OFF)”
1. Now – Once I’m here I never need to do any of the above steps again.  Ever.  This is my persistent baseline snapshot that I will love forever.  In the future I will come back to this snapshot, apply patches and make a new baseline (deleting this snapshot and renaming the new one to this name)
12. Boot the VM
13. Install the build of Visual Studio we are using at the moment (this includes NDP4, etc)
14. Start VS and setup my configuration settings (C# environment, debugger and symbol options, editor options, etc)
15. Shut down the VM
16. Snapshot the VM as “Installed VSTS and NDP4 build XXXXX (OFF)”
1. When we take a new version of VS I will need to abandon this snapshot tree so it was important that I did all the OS and non-CLR/VS related changes prior to this snapshot.
17. In the Hyper-V MMC bring up the settings window for this VM
1. Select “IDE Controller 1”
2. Select “Hard Drive” and click the Add button
3. Choose the “Physical hard disk” radio button and select the drive marked offline in the drop down list (it should be your only option since only offline drives will show up here).
4. Click OK to save the drive in the settings.
18. Boot the VM
19. On the new drive I create a folder named “HORVICKVM-DEV” (the machine name) and under it enlist in the source code for my branch.
1. I create the folder with the machine name because I sometimes mount the same drive under multiple VMs (not at the same time) and this helps me isolate changes.
20. Build, Test – Ready!

When I take a new VS I will revert to step 12 and start over (I do export my configuration settings to make configuring VS easier).

Configure Test Box 1 (2 and 3)

My goal with the test boxes is, like the dev box, to be able to quickly recover from NDP updates but also to have a stable SQL and IIS environment to start from.  I will create multiple identical test boxes that differ only in OS (2003 or 2008) and data width (32 or 64).

I can recover from an NDP update in about 30 minutes with this model.  On physical hardware this would also be about a day.

1. Create a new (127GB - Expandable) virtual disk at F:\VM\Disks
2. Create a new virtual machine (at F:\VM\Machines) called “HORVICKVM-TEST1” with
1. Two Virtual Processors
2. 1 GB of RAM
3. Legacy Network Adaptor (So I can use RIS for network installs)Boot to RIS network boot
4. The drive you created in step #1
3. Install Windows 2003 (32 bit), Windows 2008 (32bit) or Windows 2008 (64 bit)
1. Since I work on the server component I don’t install a client OS on my test boxes
2. When it is finished installing shut the machine down and take a snapshot called “OS Installation Complete (OFF)”
4. In the VM configuration settings for test box 1
1. Change the network adaptor from Legacy Network Adaptor to “Network Adaptor”
5. Boot the Test VM
1. Install NDP 2.0, and 3.5 + SP1
2. Install WinDbg and configure it for use with our symbol and source server plus my favorite configuration tweaks.
6. Shut Down the VM
7. Snapshot named “NDP 3.5+ Patched (OFF)”
8. Boot the VM
9. Install SQL server 2008
10. Install IIS 6 or 7 (depending on OS)
11. Copy my test scripts from my persistent network share to a well known location on the virtual drive (I always want these in the baseline – they will update themselves if newer versions exist after a snapshot revert)
12. Shut down the VM
13. Snapshot named “SQL 2008 + IIS (OFF)”
1. Now – Once I’m here I never need to do any of the above steps again.  Ever.  This is my persistent baseline snapshot that I will love forever.  In the future I will come back to this snapshot, apply patches and make a new baseline (deleting this snapshot and renaming the new one to this name)
14. Boot the VM
15. Snapshot the VM as “Before NDP4 (LIVE)”
1. In case the NDP4 install has problems I want a live clean snapshot I can quickly get back to
16. Install the version of the NDP4 that matches the version installed on my dev box
17. Snapshot the VM as “NDP4 XXXXX installed (LIVE)”
18. Done!

The machine is now ready for testing.  Did you notice that WSS was not installed?  If I ever want it to be persistently installed I will create a new branch off the snapshot “NDP4 XXXXX installed (LIVE)” with WSS installed.  Typically I will either install TFS choosing to let it install WSS or I will have it not configure WSS.

Fun Things I Can Do Now …

• After test runs complete I can revert back to the snapshot “NDP4 XXXXX installed (LIVE)” and start over (or just re-run them if they are non-destructive).  When I take patches I start with the “NDP4 XXXXX installed (LIVE)” snapshot, apply them there and then make that the new baseline.
• Say I want to test the MSI out with four or five different configurations.  Say it takes 8 minutes to get the MSI to the point where I want to test.  I can save over 30 minutes by doing a live snapshot of the test machine at the point where I want to branch the testing and then after each attempt re-apply that live snapshot.  As long as I don’t muck with network resources (e.g. external SQL or WSS servers) this works great.
• Hardware/OS configurations.  On a single box I can test several OS and data width options.  This saves the company thousands of dollars and me hours of time.  Better still my code is better tested and that saves our customers countless hours of frustration and pain.
• Limited memory testing.  I can test deploying TFS on a single tier (SQL + WSS + RS + TFS) with, say, 512 MB of RAM (or less).  I don’t need to pull sticks or disable things in the BIOS.  This saves me hours and lets me cover scenarios I wouldn’t otherwise.
• Single/Multi-Proc configurations.  I have a quad core machine but say I want to test a bug that only repros on single core machines – well – I can!  Or 2.  Or 4.  It’s a nice balance.
• Using 1GB footprints I can run several test VMs at once to test multi-tier configurations (SQL on one box, RS on another, WSS on another, TFS on another) that before would have been too difficult to coordinate easily and nearly impossible to repeat numerous times per hour (at my desk – our test team has tools that can help them do this in a lab environment).

Yeah – I know a lot of you are reading this going “Well … DUH!  I’ve been using VMWare/Xen/Whatever to do that for years …”  I’m not saying this is new.  What I’m saying is that for me, as someone who didn’t have the option to use those before, this is a game-changing way of development.

I will never go back to piles of boxes under my desk.

Credit is due to James Manning for pointing out several of the configuration choices I’ve described above.

Windows 7 Feature I Like – Programmer’s Calculator

I just installed Windows 7 beta on a Hyper-V VM (1GB RAM, single proc, 127 expandable drive) and found that the calculator has changed and I like it.

Beyond Standard and Scientific there are two new modes, Programmer and Statistics.

Programmer supports Binary, Hex, Octal and Decimal and includes a slick variable length (byte to qword) binary view.

Since I’m currently working with quite a lot of bit twiddling this is a handy mode that I will be using frequently.

Statistics mode is far less interesting and has features to do things I no longer recall the usefulness of.  It’s all very important, I’m sure.

New VSTS and TFS Virtual PC/Server and Hyper-V images are available!

Brian Randell has been hard at work putting together a holiday surprise we all will enjoy.  New Virtual PC/Server and Hyper-V images with VSTS and TFS and a 12/31/2009 expiration date.

Please see his blog post for the full details - I will re-post a little here.

 http://www.pluralsight.com/community/blogs/brian/archive/2008/12/24/happy-holidays-and-look-what-santa-s-brought.aspx

  --

As part of the refresh and extending the expiration date, I’ve updated the images to include the following (naturally some items only appear in the “all-up” images):

• the latest virtual machine additions or integration services components
• all of the latest Windows Updates as of December 1, 2008
• Team Foundation Server 2008 SP1
• Visual Studio 2008 Team Suite SP1
• Team System Web Access 2008 SP1
• Team Foundation Power Tools, October 2008 update
• the latest MSSCCI provider
• Team Explorer 2005 (this allows you to work with TFS from Visual Studio 2005 which is installed so that you can edit and customize TFS reports)
• the GDR for the product formally known as Visual Studio 2008 Database Edition (aka Data Dude)
• current Process Explorer, Process Monitor, and Background Info

Naturally at this point, you’re wondering, where are the goods? Here they are:

• VSTS “all-up” Virtual PC/Virtual Server image (6 GB download, expands to 15 GB)
• TFS “only” Virtual PC/Virtual Server image (3 GB download, expands to 8 GB)
• VSTS “all-up” Hyper-V image (6 GB download, expands to 15 GB)
• TFS “only” Hyper-V image (3 GB download, expands to 8 GB)

The Virtual PC 2007/Virtual Server 2005 R2 compatible images provide VMC files for Virtual PC 2007 SP1 and Virtual Server 2005 R2 SP1 Update. The VMC file that ends in –V7 is for Virtual PC and the one that ends in –R2 is for, well, Virtual Server.

The Hyper-V downloads only contain the VHD. You’ll need to configure them yourself. I wanted to make it as easy as possible.

TFS Branching Guidance Version 2 has been Released

 The VSTS Ranger team has released TFS Branching Guidance II.  This document describes pragmatic branching practices that are based on the real world experience of TFS professionals inside and outside of Microsoft.

What's new?  I'l just repeat their information:

• TFS Branching Guide - Main 2.0
• This is the main article which briefly explains branching concepts and introduces 3 levels of the most common branching scenario
• TFS Branching Guide - Scenarios 2.0
• A collection of less common branching scenarios
• TFS Branching Guide - Q&A 2.0
• A set of most frequently asked questions with answers
• TFS Branching Guide - Drawings 2.0
• A set of branching drawings in different formats including a large branching poster
• TFS Branching Guide - Labs 2.0
• A couple of examples for hands on labs with step by step instruction for practicing the branching scenarios

 Check it out!

 http://www.codeplex.com/TFSBranchingGuideII

Using a little Ruby to automate mstest code coverage runs

I hate repetitive tasks - especially ones that are easily automated.  Generating code coverage runs using mstest is an example of that.

The goal was to create a script that could do that following:

1)     Validate that the input binaries were available

2)     Validate that the necessary runtime components were available

3)     Instrument the input binaries

4)     Build up the test sandbox

5)     Start the coverage service

6)     Perform the tests using the instrumented binaries

7)     Stop the coverage service

Being able to start the process at any point in there was bonus points.

I choose to use ruby, and specifically Rake, to automate this process.  Rake is perfect for this type of task.  I can easily break the tasks down into their core parts and build up the automation one bit at a time.  Creating dependencies between the tasks is trivial.  It reads well (even with my poor ruby skills) and has a very nice command line experience.

What I ended up with was a process where I could do this:

C:\ruby\coverage>rake --tasks

(in C:/ruby/coverage)

coverage coverage:all       # Perform a complete code coverage pass...

coverage coverage:clean     # Clean the coverage temp directory and sandbox

coverage coverage:cover     # Instrument the assemblies for code coverage

coverage coverage:merge     # Merge the test results

coverage coverage:populate  # Populate the sandbox

coverage coverage:restore   # Restores the backed up files if they exist

coverage coverage:run       # Run the test suite

coverage coverage:start     # Start the coverage service

coverage coverage:stop      # Stop the coverage service

I can do all the things I set out to do (using “coverage:all”) or onesy-twosy as I need them (e.g. if rake aborts midway through the “run” step the coverage service will need to be manually stopped before the next run – I could modify a task to handle this more gracefully but I haven’t yet since the payoff isn’t there yet).

But you came for code ...

Rakefile

require 'rake'

require "win32/dir"

require 'fileutils'

require 'yaml'

def assert_config(config)

  ['vsperfcmd',

   'prodbin',

   'root',

   'sandbox',

   'covtemp',

   'covorig',

   'vsinstr',

   'suitebin']. each { |setting|

      assert_config_setting(config, setting)

  } 

end

def assert_config_setting(config, setting)

  fail "Setting \"#{setting}\" not found in coverage.yml" if config[setting].nil?

end

Rake.application.init('coverage')

config = YAML::load(File.open('coverage.yml'))

assert_config(config)

covfiles = FileList.new(

'Microsoft.TeamFoundation.Admin.dll',

'Microsoft.TeamFoundation.Management.Core.dll',

'Microsoft.TeamFoundation.Management.SnapIn.dll',

'Microsoft.TeamFoundation.Management.Controls.dll',

'tfsconfig.exe',

'tfsmgmt.exe'

)

namespace :coverage do

  desc "Instrument the assemblies for code coverage"

  task :cover => [:clean_coverage, :clean_orig ] do

    Dir.chdir(config['prodbin']) do

      mkdir config['covtemp']

      covfiles.each do |f|

        fail "Input file not found: #{f}" if not File.exists?(f)

        puts "Instrumenting #{f}"

        cmd = "\"#{config['vsinstr']}\" \"#{f}\" /COVERAGE"

        sh cmd

        assert_file_exists(File.join(config['covtemp'], f), cmd)

        copy_coverage(f, config['covtemp'])

      end

    end

  end

  desc "Restores the backed up files if they exist"

  task :restore do

    Dir.chdir(config['prodbin']) do

      covfiles.each do |f|

        origfile = "#{f}.orig"

        if File.exists?(origfile)

          rm f if File.exists?(f)

          mv origfile, f

          instrpdb = f.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

          rm instrpdb if File.exists?(instrpdb)

        end

      end

    end

  end

  task :clean_backup do

    Dir.chdir(config['prodbin']) do

      covfiles.each do |f|

        origfile = "#{f}.orig"

        rm origfile if File.exists?(origfile)

        instrpdb = f.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

        rm instrpdb if File.exists?(instrpdb)

      end

    end

  end

  task :clean_coverage do

    FileUtils.rm_rf(config['covtemp'])

  end

  task :clean_orig do

    FileUtils.rm_rf(config['covorig'])

  end

  task :clean_sandbox do

    FileUtils.rm_rf(config['sandbox'])

  end

  desc "Start the coverage service"

  task :start do

    mkdir config['sandbox'] unless File.exists?(config['sandbox'])

    sh "\"#{config['vsperfcmd']}\" /start:coverage \"/output:#{File.join(config['sandbox'], 'adminops.coverage')}\" /user:redmond\\vseqa1 /user:redmond\\tfssvc /user:#{ENV['USERDOMAIN']}\\#{ENV['USERNAME']} /waitstart"

  end

  desc "Stop the coverage service"

  task :stop do

    sh "#{config['vsperfcmd']} /shutdown"

  end

  desc "Run the test suite"

  task :run do

        Dir.chdir(config['sandbox']) do

          puts "Running Unit Tests"

          sh "aotest.exe /assembly:AdminOps.UnitTests.dll /test * /out:logs"

    end

  end

  desc "Merge the test results"

  task :merge do

  end

  desc "Populate the sandbox"

  task :populate=> [:clean_sandbox] do

    mkdir config['sandbox']

    puts "Copying Product binaries..."

    FileUtils.cp_r(Dir.glob(File.join(config['prodbin'], 'Microsoft.TeamFoundation.*')), config['sandbox'])

    cp File.join(config['prodbin'], 'NetFwTypeLib.dll'), config['sandbox']

    puts "Copying Test binaries..."

    FileUtils.cp_r(File.join(config['suitebin'], '.'), config['sandbox'])

    puts "Copying Instrumented binaries..."

    FileUtils.cp_r(Dir.glob(File.join(config['covtemp'], '.')), config['sandbox'])

  end

  desc "Perform a complete code coverage pass from scratch"

  task :all => [:clean, :cover, :populate, :start, :run, :stop, :merge]

  desc "Clean the coverage temp directory and sandbox"

  task :clean => [ :clean_coverage, :clean_sandbox, :restore, :clean_backup ]

end

def assert_file_exists(file, cmd)

  if not File.exists?(file)

    puts "COVERAGE ERROR: file not found: #{file}"

    puts "Executed: #{cmd}"

  end

end

def copy_coverage(file, dir) 

  if File.exists?(file)

    cp file, dir

  else

    die "Assembly not found for assembly #{file}"

  end

  pdbfile = file.gsub(/(.*)\.(dll|exe)$/, '\1.pdb')

  if File.exists?(pdbfile)

    cp pdbfile, dir

  else

    puts "WARNING: PDB file not found for assembly #{pdbfile}"

  end

  instrpdbfile = file.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

  if File.exists?(instrpdbfile)

    cp instrpdbfile, dir

  else

    puts "WARNING: INSTR PDB file not found for assembly #{instrpdbfile}"

  end

end

coverage.yml

vsperfcmd: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsperfcmd.exe

prodbin: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386

root: C:/VMDEV/TfsArch1/binaries/x86chk

sandbox: C:/SANDBOX

covtemp: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage

covorig: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage/orig

vsinstr: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsinstr.exe

suitebin: C:/VMDEV/TfsArch1/binaries/x86chk/SuiteBin/i386/tfs/AdminOps/bin

So there you have it…

I go to my ruby command line (which is just a normal command line with the ruby bin directory in the path  - e.g. “set PATH=%PATH%;c:\ruby\bin”) and perform:

C:\ruby\coverage>rake coverage:all

When the tests are done the coverage file I created is in c:\sandbox ready to be imported into VS or merged with other files or dumped to excel or whatever.

And one note about the rakefile – you may notice that I call “aotest.exe” not “mstest.exe” – aotest is just a little wrapper around mstest that one of our test devs wrote.  I could have just as easily used mstest.

PDC Hours - Come and Talk Team Foundation Server Setup, Configuration and Administration

I'll be available to talk about TFS admin scenarios at PDC - my hours are:

Monday  - 12:30 - 6:00 in the VSTS Lounge

Tuesday  - 12:30 - 6:00 in the VSTS Lounge 

Wednesday  10:00-12:30 in the VSTS Lounge and Ask The Experts from 5:30-9:00PM

I won't be available Thursday so please find me before then!

Lets Talk TFS Setup, Configuration and Administration at PDC!

Coming to PDC?  Interested in Team Foundation Server Setup, Configuration and Administration?  Stop by the PDC Lounge and let's chat!

I'll post the hours I'll be available tomorrow.

A link to my Ruby and Rails blog

Since life can't be all Team Foundation Server and rainbows I've started getting serious about learning more about Ruby and the Rails framework.  To keep the TFS focus on this blog I've started a new blog dedicated to non-TFS tech topics http://www.GhostOnThird.com.

What to do when TfsQuiesce fails with "No associated service account for role TFSEXECROLE" error

When installing a service path on Team Foundation Server one of the first things that needs to happen is that TFS is "quiesced" - which basically means "to make quiet" - i.e. it's running but it's not accepting new client connections.  The tool that does this during the SP installation is called 'TfsQuiesce.exe". 

This blog post is to explain what you can do when you see the following in your log file after a failed upgrade:

TFSQuiesce - Team Foundation Server Maintenance Tool
Copyright (c) Microsoft Corporation.  All rights reserved.

Using workflow file from location exe.
Executing workflow 'Quiesce DT'...
Disabling SQL Jobs for databases TFSActivityLogging,TFSBuild,TFSIntegration,TFSVersionControl,TFSWorkItemTracking,TFSWorkItemTrackingAttachments,TFSWarehouse
Blocking service account from accessing database TFSActivityLogging
No associated service account for role TFSEXECROLE.
Executing workflow 'Unquiesce DT'...
Enabling SQL Jobs.
Unblocking service account from accessing database TFSActivityLogging
Unblocking service account from accessing database TFSBuild
Unblocking service account from accessing database TFSIntegration
Unblocking service account from accessing database TFSVersionControl
Unblocking service account from accessing database TFSWorkItemTracking
Unblocking service account from accessing database TFSWorkItemTrackingAttachments
Unblocking service account from accessing database TFSWarehouse

Workflow 'Quiesce DT' failed! ExitCode = 8000.
10/31/07 12:17:15 DDSet_Status: Process returned 8000
10/31/07 12:17:15 DDSet_Status: Found the matching error code  for return value '8000' and it is: '29206'
10/31/07 12:17:15 DDSet_Error:  8000
10/31/07 12:17:15 DDSet_CARetVal: 29206
10/31/07 12:17:15 DDSet_Status: QuietExec returned 29206

Notice the bolded line "No associated service account for role TFSEXECROLE" - this is the key line.

This error is indicating is that one or more of the databases have no members in the TFS exec role.  The reason we are running this check at all is to ensure that we are not running the upgrade as the service account user.  Let’s start by figuring out if this is a problem with all seven TFS databases or just one of them.

Run the following SQL statement 7 times – each time changing the “use TfsVersionControl” to one of the other TFS databases:

use TfsVersionControl

SELECT  dpMember.name
FROM    sys.database_principals dp
JOIN    sys.database_role_members drm
ON      drm.role_principal_id = dp.principal_id
JOIN    sys.database_principals dpMember
ON      dpMember.principal_id = drm.member_principal_id
WHERE   dpMember.Type = 'U'
        AND dp.name IN ('TFSEXECROLE')

I expect one or more of the result sets to be empty. 

Ultimately the role can’t be empty.  When you find the database(s) with the empty TFSEXECROLE role you should add the appropriate service account user to the role for that database(s).

Re-run the SQL snippet to ensure the fix was made properly and now running the installation should past this error.

Migration Toolkit Pre-release nears 100 downloads ...

At this time 98 people have downloaded the TFS Migration and Synchronization Toolkit.  Ok - some of you are saying "100?  What's the big deal?  A picture of a rabbit with a pancake on it's head gets more downloads every hour!"

And they would be right.

But think big picture - what if 50% of the downloaders actually open the zip file?

And what it 50% of them actually compile and run the product?

And what if 50% of them start looking into the source code and trying to write their own converter? 

And what if 50% of them actually succeed?

And what if just one of those people decides that their converter is worth sharing with the world?  How many people will that help?  Hunderds?  Thousands?  More?  Or maybe they decide to use the tool for consulting engagements - how many successful careers could be made using that tool?

In the migration space it does not require big numbers to make a big impact.

We've already impacted 100 people.  That is exciting to me.

How many more will download it tomorrow?  And the next day? 

Have you downloaded it yet?

Migration Toolkit Pre-Release published on CodePlex

Interested in writing a migration tool targeting (or synchronizing with) Team Foundation Server version control or workitem tracking?

Then check out the pre-release of the TFS Migration and Synchronization Toolkit on CodePlex!

http://www.codeplex.com/MigrationSyncToolkit

This drop is the complete source code for the migration toolkit and sample converters.  The VC sample is an application that will synchronize WSS document libraries with TFS version control.  Developers can check in doc changes with their code and managers can see the updated docs on the WSS server after the mirroring is done (and it works the other way too).

Oh - and this is still under active development (and has known issues) so if you have any feedback it's still early enough for it to be considered for inclusion in the final drop.

And to quote Matt ...

"Please note that this is a prerelease version of the Migration and Synchronization Toolkit. Several features are not complete and as testing has not completed, the code has the potential to contain bugs and overwrite data stored in TFS. Please be careful to use this only in a testing environment and not on live production data."

TFS Migration Toolkit Spec released for public review

As Matt noted on the migration blog a working copy of the toolkit specification has been released.  You can download it and leave feedback from this link:

http://msdn2.microsoft.com/en-us/vstudio/aa948851.aspx

 If you have feedback on migration scenarios for version control, work item tracking or other products (test, office, web, file systems, etc) please check it out and let us know.  We are still actively developing the toolkit and your feedback is very important to us!

How to handle VSSConverter UserMap.xml files for users who no longer exist

When running the VSS to TFS converter (VSSConverter.exe) one of the most common questions I hear is how to create a UserMap.xml file for users who are no longer with the company or have had their accounts renamed.

My first suggestion is that any customer who has to deal with this issue (or any user mapping issues – or users in a workgroup configuration) should start by getting the hotfix provided in KB 928828.  This can be gotten through your CSS (support) contact.

This hotfix changes the way that user authentication is done and is an important fix.  It is compatible with both RTM and SP1 however it was not included in SP1 (it was made after the SP1 cut-off date and does include all the VSSconverter fixes already in SP1). 

That said - what I’m going to describe is the same in both the RTM and SP1 versions as well as in KB 928828.

The correct way to deal with the users may be as simple as doing nothing.  Let me give you an example.

To test this out I created a sample VSS repository and added two users “User1” and “User2”.  These were in addition to the default users of “rhorvick” (me) and “Admin”.

I connected as “User1” and created some content.  Then as “User2” and created more.  And finally as “rhorvick” and created even more.

I then ran the converter in analyze mode.  This created the following UserMap.xml file:

<?xml version="1.0" encoding="utf-8"?>

<UserMappings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

  <UserMap From="USER1" To="" />

  <UserMap From="RHORVICK" To="" />

  <UserMap From="USER2" To="" />

  <UserMap From="ADMIN" To="" />

</UserMappings>

At this point I just ran the conversion – I did not change the mapping file in any way.  Please note that User1 and User2 are NOT existing accounts on my TFS server.

With the QFE the conversion created the following history in TFS:

Changeset User          Date       Comment

--------- ------------- ---------- --------------------------------------------

10837     rhorvick      1/4/2007   {1/4/2007 3:47:26 PM}added by rhorvick---- D

10836     rhorvick      1/4/2007   {1/4/2007 3:47:22 PM}added by rhorvick

10835     rhorvick      1/4/2007   {1/4/2007 3:47:14 PM}

10834     User2         1/4/2007   {1/4/2007 3:47:00 PM}added by user2

10833     User2         1/4/2007   {1/4/2007 3:46:18 PM}

10832     User1         1/4/2007   {1/4/2007 3:46:00 PM}user 1

10831     User1         1/4/2007   {1/4/2007 3:45:12 PM}added by user1

10830     User1         1/4/2007   {1/4/2007 3:43:52 PM}created by user1

10829     rhorvick      1/4/2007   ----------------VSSConverter Project Mapping

So the migration did properly migrate the changes and attributed them to the proper authors.  I don’t know that I’m explaining this correctly – perhaps someone from Admin/Ops can explains it better – but when a checkin occurs and the author the change is being attributed to (which is not necessarily the account making the checkin call) is a non-existing user and the account is not a domain name or a TFS account/group name then a special identity is created which can only be used for the purpose of attributing changesets to (i.e. an audit-only account, not one that can be logged in as).

Now – had I wanted to attribute the changes to a real user I could have altered the mapping file like so:

<?xml version="1.0" encoding="utf-8"?>

<UserMappings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

  <UserMap From="USER1" To="northamerica\rhorvick" />

  <UserMap From="RHORVICK" To="northamerica\rhorvick" />

  <UserMap From="USER2" To="northamerica\rhorvick" />

  <UserMap From="ADMIN" To="northamerica\rhorvick" />

</UserMappings>

And this time when I migrated I see the following history:

Changeset User          Date       Comment

--------- ------------- ---------- --------------------------------------------

10846     rhorvick      1/4/2007   {1/4/2007 3:47:26 PM}added by rhorvick---- D

10845     rhorvick      1/4/2007   {1/4/2007 3:47:22 PM}added by rhorvick

10844     rhorvick      1/4/2007   {1/4/2007 3:47:14 PM}

10843     rhorvick      1/4/2007   {1/4/2007 3:47:00 PM}added by user2

10842     rhorvick      1/4/2007   {1/4/2007 3:46:18 PM}

10841     rhorvick      1/4/2007   {1/4/2007 3:46:00 PM}user 1

10840     rhorvick      1/4/2007   {1/4/2007 3:45:12 PM}added by user1

10839     rhorvick      1/4/2007   {1/4/2007 3:43:52 PM}created by user1

10838     rhorvick      1/4/2007   ----------------VSSConverter Project Mapping

WSS Rant – Linking to the latest version of a sharepoint document considered harmful. (Lessons 6 and 7)

SharePoint does not have persistent hyperlinks for all document revisions.

More specifically – SharePoint has two methods for linking to documents: the canonical path and the revision paths. 

The canonical path is the latest copy of the document at the time of the download request.  Every time a new revision is created the contents of the canonical path change.  Canonical paths look like this:

http://<servername>/Shared%20Documents/document.doc

Revision paths are paths to historical revisions of the document.  More specifically they are paths to the non-tip (latest) version of the document.  The revision path exists under the _vti_history location and have the version number in the path.  They look like this:

http://<servername>/_vti_history/7/Shared%20Documents/document.doc

That link is for revision 7 of the document.

When I first saw this I assumed that all documents were accessible via revision paths, and the canonical path was just an abstraction over the latest revision document.  I was incorrect.

You can prove this to yourself by taking any document in a WSS repository and trying to request the latest version by the revision path that would contain its content.  For example if you add a new file it has only one revision.  So its revision path (if it could have one) would be:

http://<servername>/_vti_history/1/Shared%20Documents/document.doc

Requesting this fails with a HTTP 400 response (HTTP BAD REQUEST).  Requesting the canonical path works as expected.  If a new revision of the document is created the revision path for version 1 becomes valid and the canonical path now refers to version 2 (and as before – the revision path for version 2 is not valid).

Why is this a problem?

Enumeration of the document library is slow.  The time between when we analyze the changes to migrate and the point where we actually perform the migration could be seconds, minutes or hours apart.  During that time the document may have been updated.  When that happens the version we cared about is demoted from the canonical path down to a revision path.  If we downloaded the canonical path we would end up getting the wrong content.

At first my plan was to store the canonical path and then to check during migration to make sure that canonical path is still the version we want before downloading.  But that just narrows the window on the race condition – it does not eliminate it.

I ended up doing the following:

When analyzing an item we know if we are looking at the latest version (canonical path) or a historical revision (revision path).  We store this fact as a boolean along with the path necessary to download the document (the canonical or revision path) and the version number of the item.  During migration we download the item from the stored download path.  Next, if the document was the latest version (i.e. a canonical path) we now double check that the canonical path refers to the document version we needed.  If it does we are done.  If it does not then we enumerate the revision history to find the revision we want.  If it exists we download the specific revision and continue.  If it does not then the revision was deleted and we handle that condition.

I believe this approach guarantees that the content we end up is always the proper content and does so in the minimum number of round-trips to the server.

All because canonical paths are not just an abstraction over revision history.

Lesson 6: The conceptually simplest operation can end up being incredibly complex to perform correctly.

Lesson 7: When representing data (such as document revisions) make all the data available through a uniform mechanism.  Shortcuts to the latest revision of the data should be an abstraction over that mechanism.