granth's blog : Other

Multi-threaded robocopy for faster copies

grantholliday — Tue, 08 Dec 2009 06:27:09 GMT

Windows 7 and Windows Server 2008 R2 include an updated version of Robocopy that includes a multi-threaded copy feature.

/MT[:n] :: Do multi-threaded copies with n threads (default 8).

I’ve been using this for a while now and I’ve been seeing excellent results. Be careful with who you share a switch with though – I managed to saturate my 100Mbit switch so badly that my fellow switch neighbors started getting timeouts. My favorite usage is:

robocopy \\server\share\folder C:\Temp\folder /S /MT /Z

This copies the contents of the remote folder, with 8 threads in restartable mode – great for when you’re on a slow or unreliable link. Also note that the progress monitors don’t seem accurate in multi-threaded mode. It starts copying 8 files at once, but only shows you the progress of 1 of them.

Replying inline in Outlook with [Name]:

grantholliday — Wed, 06 May 2009 03:27:27 GMT

Did you know that there’s a feature in Outlook that prefixes your name automatically when you reply inline like this?

When you reply to somebody and start replying inline, it automatically changes the font to a new color and prefixes it with [GrantH] or similar.

Open Outlook | Tools | Options | Mail Format | Signatures & Stationary..

Hosted Team Foundation Server

grantholliday — Wed, 03 Dec 2008 05:56:47 GMT

Since the Readify TFSNow! hosted TFS offering that I helped to build is no longer in operation, people have asked if there is anybody else doing it.

There are three partners that I know of that offer to run and maintain Team Foundation Server in the cloud for you:

Phase2
TeamDevCentral
Saas Made Easy
CodePlex – your project must be open source

I don’t have any experience with any of them, so please let me know what you think.

OzTFS, OzMOSS and OzSilverlight are moving to a new mail server

grantholliday — Tue, 02 Dec 2008 10:58:10 GMT

Status update: OzTFS, OzMOSS and OzSilverlight have been migrated. Please note the new addresses that you need to send mail to.

These three mailing lists are a valuable resource to the Australian community as well as the global community. Please bear with us during this transition, we’ll trying to make it as smooth as possible.

What do you need to do?

You don’t need to re-subscribe. The subscriptions have been bulk imported to the new server.
You may need to modify your filtering rules. The best way to do that is to filter on the “list-id:” in the headers.
The email address that you post to changes from listserver AT oztfs.com –> oztfs AT oztfs.com

Background

As you are probably aware, we’ve been suffering a few problems on the OzTFS, OzMOSS and OzSilverlight mailing lists lately. (List Posting Errors, etc)

Additionally, the server that runs the lists is near the end of its life and it’s time to retire it. Codify currently host the busy & popular ausdotnet mailing list and were successfully plied with offers of beer to host the other Australian MS community lists on the same infrastructure.

Where/What/etc is it?

The list is hosted in a data centre in Brisbane in a dedicated 42RU rack belonging to Codify. It is served from Windows Server 2008, IIS 7, and GFI MailEssentials running in a Hyper-V guest VM on one of Codify's production Dell 2950 servers. The server has 8 x 2.4Ghz XEON cores, 16 GB of RAM and 2TB of RAID 5 disk. The VM is shipped periodically over a private 100 meg link to another server at Pipe Network's DC2 in Creek Street, Brisbane for backups. The server has 100 meg of full-duplex Ethernet connectivity and BGP routing to diverse upstream providers. That should be sufficient for a mailing list :).

www.oztfs.com, www.ozmoss.com and www.ozsilverlight.com will be setup as a redirects to http://www.codify.com/lists/oztfs, /ozmoss and /ozsilverlight which includes information on how to subscribe and unsubscribe from the lists.

If you are having problems with your subscription or you have questions about the list, please contact me or Codify.

Asus Eee PC Netbook

grantholliday — Fri, 28 Nov 2008 06:56:55 GMT

In my recent trip to Australia, I had some time to walk through the shops. One of the things that really caught my attention was the Asus Eee PC. (It’s pronounced “e p c”). I’d heard about these low-cost, low-power netbooks before but I’d never considered how useful they might be. The thing that struck me about these is that they were appearing on retail shelves – which means that they’re no longer a gadget-geek, early-adopter piece of technology.

Here’s what I like about them:

They’re small, yet still useable.
They’re power-efficient. That means longer times between charging.
They’re cheap.
They come configured. They come pre-loaded with Windows XP, Skype, Live services and everything ready to go. No bloatware, no turds.

The current models are also able to run the next version of Windows – Windows 7. That means that they’re not going to be obsolete too soon.

My prediction is that they’re going to be the Next Big Thing and you’re going to see them as popular as iPods with young people in the next 12 months. To add some weight to my theory, I’ve been reading and asking what other people think of them:

First of all came the email responses from other softies:

“I use the unit now as my everyday – domain joined – Microsoft laptop and love it”

“Highly recommended – the portability and battery life can’t be beat.”

The next example was from the girl at the counter from where we ended up buying 1000H model preloaded with XP, 1Gb RAM and 160Gb Hard drive:

“They’re really cool. Everybody at uni has them, they fit on the lecture desk”

The next example was when we got home and my wife tried it out:

“It fits in my bag!”

Then there’s these comments from a New York Times article – Mini Laptops Fuel PC Sales Growth:

“The UMPC is not going to be a fad. It is going to evolve into something that Generation Y will find even more useful as they become the increasingly affluent, on-the-go, always connected members of society. This generation already looks at small screens, type on keyboards that are accessible only by their thumbs, and type fewer and fewer documents, in favor of blogs, twitters, metatagging photos, videos and creating presentations vice point-papers.
Desktop/Tabletop laptops will replace the current mid-tower PC, as the increasing power fits into the smaller form factor, but the UMPC - in whatever its evolved state - will be their second device; indeed their true mobile device that will enable them to stay in communication with their world and in touch with their community.”

“For us ladies, you can slip an Eee PC right into a handbag. Really can’t say the same thing about any regular laptop. It’s really different when you can take a full PC with you wherever you want whenever you want.”

Let me know what you think. What are you doing with your Eee PC?

TFS and the OWASP Top 10 Threats

grantholliday — Thu, 27 Nov 2008 06:06:44 GMT

My first MSDN document has been published. Team Foundation Server (TFS) and the Open Web Application Security Project (OWASP) Top Ten describes how TFS complies with the ten most common threats for web applications.

Recently, our sales guys were working with a customer that has an IT policy which states that any web applications they deploy internally must be compliant with the Open Web Application Security Project’s (OWASP) Top Ten Threats. This applies to custom developed software and off the shelf software.

The Open Web Application Security Project (OWASP) is a worldwide free and open community focused on improving the security of application software. The OWASP Top Ten provides a powerful awareness document for Web application security. The OWASP Top Ten represents a broad consensus about what the most critical Web application security flaws are. The current version is the OWASP Top Ten 2007.

Even though the OWASP Top Ten shouldn’t be used as a policy or standard, we wasn’t in a position to argue this point and were tasked to prove that TFS itself is compliant with the threats.

It was an interesting process learning about the different projects and initiatives. In the end it turns out that Microsoft employs a few of the founders of the OWASP project and they now work on the SDL team.

The core of it comes down to the fact that the engineering teams at Microsoft practice the Security Development Lifecycle which covers these threats.

The Microsoft Security Development Lifecycle (SDL) is the industry-leading software security assurance process. A Microsoft-wide initiative and a mandatory policy since 2004, SDL has played a critical role in embedding security and privacy into Microsoft software and culture. Combining a holistic and practical approach, SDL introduces security and privacy early and throughout the development process.

Every shipping Microsoft product must be approved by the Secure Windows Initiative (SWI) team and go through a process of review and registration in a central repository. Visual Studio Team System 2008 Team Foundation Server SP1 has achieved compliance with Microsoft’s Security Development Lifecycle (SDL).

Remotely connecting to a Hyper-V virtual machine with networking disabled

grantholliday — Mon, 10 Nov 2008 21:59:13 GMT

In a previous blog post, I explained the steps to convert the VS2010 CTP VPC from Virtual PC to Hyper-V. When using virtual machines, sometimes you don’t want them to be connected to any network. This is especially true with the Team Foundation Server 2010 CTP VPC – if you connect it to the network, it probably will stop functioning.

This post explains how you can use the Hyper-V Virtual Machine Connection tool to connect to a VM running on a remote Hyper-V server. The way this works is that it connects to the Hyper-V instance, then directly to the virtual machine’s console on that instance. It doesn’t require the virtual machine to have networking support enabled.

If your workstation is running Windows Server 2008, then you just need to install the Hyper-V tools and use them. If you are running Windows Vista on your remote workstation, you will need to install one of these packages:

You may also need to enable remote WMI between your computers.

Once you have the VMConnect.exe tool installed at "C:\Program Files\Hyper-V\vmconnect.exe", you can create a shortcut to connect directly to the virtual machine. The syntax is this:

So for example, to connect to the VM by the name of “VS2010CTP” running on the remote machine “MyHyperVServer”, create a shortcut like this:

"C:\Program Files\Hyper-V\vmconnect.exe" MyHyperVServer VS2010CTP

Querying Perfmon data from SQL

grantholliday — Sat, 08 Nov 2008 03:41:33 GMT

In a previous post I talked about Relogging Perfmon binary log files to SQL. The next obvious step is to query the data from the database.

There are two ways to do this:

Use Perfmon itself to connect to the SQL data source
Use another tool to query the SQL data source directly

1. Using Performance Monitor

Once you open Performance Monitor, press the View Log Data button (Ctrl-L):

Then on the Source tab, if you have the ODBC System DSN configured to connect to your performance database, it will show up.

This allows you to view the log files directly from SQL, just as if you had loaded the binary log files.

2. Querying SQL directly

At first glance, the schema doesn’t seem very intuitive. But with a little digging, it’s not too bad.

Whenever you use PerfMon or relog to log to a database, it automatically sets up this schema for you. There are three tables:

DisplayToID – This lists all the log sets that are in the database
CounterData – This is where the actual counter values are stored
CounterDetails – This is where the metadata about each counter / object / instance is stored

Here are some queries that I’ve found useful:

Get all computers that have counter data logged:

SELECT DISTINCT MachineName
FROM CounterDetails
ORDER BY MachineName

Get available object names for a particular computer:

SELECT DISTINCT ObjectName
FROM CounterDetails
WHERE MachineName = ‘\\MYSERVER’
ORDER BY ObjectName

Get counter names for a particular computer and object:

SELECT DISTINCT CounterName
FROM CounterDetails
WHERE MachineName = ‘\\MYSERVER’
AND ObjectName = ‘Processor’
ORDER BY CounterName

Get instance names for a particular computer, object and counter:

SELECT DISTINCT InstanceName
FROM CounterDetails
WHERE MachineName = ‘\\MYSERVER’
AND ObjectName = ‘Processor’
AND CounterName = ‘% Processor Time’
ORDER BY InstanceName

Get counter values for a particular computer, object, counter and instance. Name the column appropriately:

SELECT
    CAST(LEFT(CounterDateTime, 16) as smalldatetime) AS CounterDateTime,
    REPLACE(CounterDetails.MachineName,'\\','') AS ComputerName,
    CounterDetails.ObjectName + ISNULL('(' + CounterDetails.InstanceName + ')','') + '\' + CounterDetails.CounterName AS [Counter],
    CounterData.CounterValue
FROM CounterData
    INNER JOIN CounterDetails ON CounterData.CounterID = CounterDetails.CounterID
    INNER JOIN DisplayToID ON CounterData.GUID = DisplayToID.GUID
WHERE CounterDetails.ObjectName = 'Processor'
    AND    CounterDetails.CounterName = '% Processor Time'
    AND    CounterDetails.MachineName = '\\MYSERVER’
    AND CounterDetails.InstanceName = '_Total'
ORDER BY CounterData.CounterDateTime

There’s a few nuances in the schema which creates complexity in this query. Let me try and explain:

CounterDateTime is a char(24) instead of a datetime. By trimming it to 16 characters, this allows it to be converted to a smalldatetime at the ‘minutes’ resolution – not seconds.
MachineName includes ‘\\’, we want to trim it to make it look pretty.
InstanceName can be null. An example is the Memory\Pages/sec object & counter. The instance value for this combination is null. So we only show the instance name if it’s not null.

Of course now that you have the data being collected, post-processed and queryable in SQL – the next logical step is to make it reportable. Here is a report that I’ve built to display this data. I’ll talk about it more in a future blog post.

Script to configure SQL Server Maximum Memory

grantholliday — Sat, 04 Oct 2008 03:38:14 GMT

I’ve often got SQL Server running on my workstation, laptop or VPC and find that it just soaks up all my memory. This script sets the maximum limit:

USE [master]

GO

-- Set max server memory limit

EXEC sp_configure 'show advanced options', 1

RECONFIGURE WITH OVERRIDE

EXEC sp_configure 'max server memory (MB)', 484

RECONFIGURE WITH OVERRIDE

-- Check the setting

EXEC sp_configure 'max server memory (MB)'

Relogging Perfmon binary log files to SQL

grantholliday — Wed, 24 Sep 2008 05:05:33 GMT

Lately I’ve been doing quite a bit of work with Performance Monitor (or PerfMon for short) to monitor some performance counters on a collection of servers.

PerfMon has the ability to log it’s counter collections directly to a SQL ODBC data store. The downside of this is that if the store is remote and it is unavailable for some reason, then you’ve lost that data. Another alternative is to log the data to a *.blg (binary) format on the local disk of the machine that you’re monitoring, then post-process it later.

That’s what I’m doing. Every 24 hours I start a new .blg logfile (see screenshot below). Then at a convenient time I copy the directory log files onto another machine for post-processing using robocopy.

One of the very useful tools that comes with Windows that I didn’t know about is “relog”.

Microsoft r Relog.exe (6.0.6000.16386)

Relog creates new performance logs from data in existing performance logs by changing the sampling rate and/or
converting the file format. Supports all performance log formats, including Windows NT 4.0 compressed logs.

Usage:
relog <filename [filename ...]> [options]

Parameters:
<filename [filename ...]>     Performance file to relog.

Options:
-?                            Displays context sensitive help.
-a                            Append output to the existing binary file.
-c <path [path ...]>          Counters to filter from the input log.
-cf <filename>                File listing performance counters to filter from the input log. Default is all
                                counters in the original log file.
-f <CSV|TSV|BIN|SQL>          Output file format.
-t <value>                    Only write every nth record into the output file. Default is to write every record.
-o                            Output file path or SQL database.
-b <M/d/yyyy h:mm:ss[AM|PM]> Begin time for the first record to write into the output file.
-e <M/d/yyyy h:mm:ss[AM|PM]> End time for the last record to write into the output file.
-config <filename>            Settings file containing command options.
-q                            List performance counters in the input file.
-y                            Answer yes to all questions without prompting.

Examples:
relog logfile.csv -c "\Processor(_Total)\% Processor Time" -o logfile.blg
relog logfile.blg -cf counters.txt -f bin
relog logfile.blg -f csv -o logfile.csv -t 2
relog logfile.blg -q -o counters.txt

Relog gives you the ability to extract out the counters you care about and put them into another file. It also allows you to import the binary logfile format into a SQL store. You have to create a SQL Server ODBC System DSN for this to work.

One of the things relog can do is summarize the data. If you collect performance counters every 1 minute and you want to summarize that data to the hour, use “-t 60” as a parameter.

The command to relog from a *.blg logfile to a SQL database is:

relog logfile.blg -cf CounterList.txt -f SQL -o SQL:PDB!LogFileIdentifierHere

CounterList.txt is a text file that contains a list of counters that you care about, for example:

\\SERVERNAME\LogicalDisk(C:)\Avg. Disk sec/Transfer
\\SERVERNAME\LogicalDisk(C:)\Disk Bytes/sec
\\SERVERNAME\LogicalDisk(C:)\Disk Transfers/sec
\\SERVERNAME\Memory\Pages/sec
\\SERVERNAME\Processor(_Total)\% Processor Time
\\SERVERNAME\System\Processor Queue Length

And here’s my latest one-liner. This will take a directory of *.blg files, check to see if they’ve been relogged into the SQL database and if not, it will relog them. It uses the filename as the key for the log file identifier.

for %a in (*.blg) DO osql -o NUL -b -E -S (LOCAL) -d PDB -Q "EXIT(SELECT COUNT([GUID]) FROM DisplayToID WHERE DisplayString = '%a')" & IF NOT ERRORLEVEL 1 relog %a -cf CounterList.txt -f SQL -o SQL:PDB!%a

It makes use of FOR, osql.exe, ERRORLEVEL and relog. All of which are available from the standard windows command prompt.