Speaking of which...

Investigating Locks

2009-10-09T16:49:56Z

Consider the following scenario:

You have an ASP.NET application which intermittently responds sluggishly. As the problem occurs memory usage is about average, as is CPU usage, but still certain pages respond slower and slower. The machine acts just as if it is under heavy load, but judging from the CPU it isn't. In fact CPU usage might even decrease rather than increase. When trying to access non .NET content such as plain text/html or images you usually find that response times are just as fast as ever.

When you find that your application has hung in a low CPU state it is usually because of one of the following reasons (in order of likelihood based on my experience) :

External resources responding slowly. E.g. a database query or AD lookup is taking a long time to complete and so the application is simply waiting for another server to respond.
One of the threads of the application has entered a critical section and now several other threads are waiting to enter the same critical section.
A classic deadlock. Thread A waits for thread B which waits for thread C which waits for thread A.

Quite often you'll see a mixture of items 1 and 2. A thread enters a critical section, calls a remote database, or similar, and all other threads attempting to enter the same critical section will have to nicely stay in line until the struggling database has responded. Still, it is quite possible that the critical section is waiting for a local event to finish, but then you'd usually see a higher CPU load on the machine. (Though not necessarily in the w3wp.exe process.)

What are Critical Sections?

In case you've forgotten or simply not worked that much with critical sections it might be necessary to refresh your memory on this.

A Critical Section is a segment of code that you do not want to be executed more than one at a time. In order to ensure this you use the lock keyword to ensure that a certain block of code is able to complete with no interruption from any other threads. To illustrate this I've created the following code sample:

public class FileIO
{
    private static object myLock = new object();

	public FileIO()
	{
	}

    public string performLockOperation()
    {
        string strResult = "The Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        lock (myLock)
        {
            // Let the thread sleep for 10 seconds
            // This way we simulate a File IO operation that takes some time to complete.
            System.Threading.Thread.Sleep(10000);
        }
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

    public string performNoLockOperation()
    {
        string strResult = "The No Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        System.Threading.Thread.Sleep(10000);
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

}

Let's take a quick look at the parts of this class that I've written, called FileIO:

myLock

The class has a static variable called myLock that I will be using to identify the lock I'm about to create. This is why the variable is static. Otherwise I'd be creating new locks for each instance of the FileIO class leaving the lock operation completely redundant. Now that I use a static variable the lock spans all instances of the FileIO class.

constructor

The constructor is empty. Move along. Nothing to see here.

performLockOperation

This is the most interesting part of the class. It is the function we'll be using to actually reproduce the potential problems. The function returns a string which will contain information about when the call to the function was made and when it completed. We store this data in a string called strResult which we will eventually be returning as the result of the function call.

Having declared this variable and put the initial time of the function call in it we then create our lock. This is simply done with the following syntax:

lock (object)
{
}

Any code within this block will first make sure that there is not already a lock taken on this object. If there is, then they will pause execution and wait for the lock to become available before continuing execution.

Inside our lock we then simulate a slow File IO operation by letting the thread sleep for 10 seconds.

After that we append the current time to the result and return it.

performNoLockOperation

This function is identical to the previous one, except this time we do not have a critical section. This function is included just for comparison.

The potential problem

Now, let's get down to business. If we have a web page that makes a call to the performLockOperation-function and that page is requested at the same time by four different clients. What would be the response times?

The answer is 10, 20, 30 & 40 seconds.

Here's why:

The thread handling the first request enters the critical section, effectively locking it for everyone else. When it is complete 10 seconds have passed due to the Sleep-call.
One of the other threads enters the critical section once the first request is finished. It has already waited for 10 seconds, and the thread will now sleep for an additional 10 seconds.
As the second thread finishes one of the remaining two enters the critical section. It has now waited for 20 seconds and will sleep for an additional 10.
The final request is able to enter the critical section after having waited for 30 seconds. Once it is complete 40 seconds have passed.

If we'd been using the performNoLockOperation instead then all requests would have responded after 10 seconds.

Now, I know that 10 seconds is an exceptionally long time to let the thread sleep, but this was all for illustration purposes. Imagine instead that the sleeping time was 1 second and the number of requests was 200...

Troubleshooting this issue

Okay, so what would it look like when this problem occurs? Let's have a look. If you haven't looked at my posts on getting started with Windbg, then I suggest reading them before continuing. (Part I & Part II)

First of all I fire up WinDbg, then I attach to the w3wp.exe. I then use "~*e!clrstack" to look at the managed threads of the w3wp.exe application. Here is a snippet of what I get:

OS Thread Id: 0xce0 (24)
Child-SP         RetAddr          Call Site
000000000447e0f0 000007ff001e13d2 FileIO.performLockOperation()
000000000447e170 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000447e1b0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000447e1e0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000447e210 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000447e250 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000447e2a0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000447e370 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000447e3d0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000447e430 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000447e490 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000447e4c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000447e570 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000447e610 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000447e7a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000447e7f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000447e910 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447ea80 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447eae0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000447f310 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000447f3f0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447f560 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447f5c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x15ec (25)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0x1958 (26)
Child-SP         RetAddr          Call Site
0000000004c1dd10 000007ff001e13d2 FileIO.performLockOperation()
0000000004c1dd90 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004c1ddd0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004c1de00 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004c1de30 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004c1de70 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004c1dec0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004c1df90 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004c1dff0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004c1e050 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004c1e0b0 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004c1e0e0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004c1e190 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004c1e230 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004c1e3c0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004c1e410 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004c1e530 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e6a0 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e700 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004c1ef30 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004c1f010 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f180 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f1e0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1e28 (27)
Child-SP         RetAddr          Call Site
000000000492dcf0 000007ff001e13d2 FileIO.performLockOperation()
000000000492dd70 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000492ddb0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000492dde0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000492de10 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000492de50 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000492dea0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000492df70 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000492dfd0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000492e030 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000492e090 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000492e0c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000492e170 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000492e210 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000492e3a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000492e3f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000492e510 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492e680 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492e6e0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000492ef10 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000492eff0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492f160 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492f1c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1c38 (28)
Child-SP         RetAddr          Call Site
0000000004d0e0b0 000007ff001e13d2 FileIO.performLockOperation()
0000000004d0e130 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004d0e170 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004d0e1a0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004d0e1d0 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004d0e210 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004d0e260 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004d0e330 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004d0e390 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004d0e3f0 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004d0e450 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004d0e480 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004d0e530 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004d0e5d0 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004d0e760 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004d0e7b0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004d0e8d0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0ea40 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0eaa0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004d0f2d0 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004d0f3b0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f520 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f580 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32

So, what does this tell us?

Well, obviously we have four threads that call my function FileIO.performLockOperation() from Page_Load.

Please note that we don't have any additional callstack, and since the performLockOperation-function was named by me it could just as well have been named "foo". In order to find out that these threads are actually waiting for a lock we need to look at the native callstack using the kb-command. Let's take a look at thread 24:

0:029> ~24kb
ntdll!NtDelayExecution+0xa
kernel32!SleepEx+0x84
mscorwks!EESleepEx+0x2d
mscorwks!Thread::UserSleep+0x71
mscorwks!ThreadNative::Sleep+0xf9
App_Code_tvh3usvo!FileIO.performLockOperation()+0x8a
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
App_Web_uywrbugj!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x9
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x501
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72

Apparently we entered FileIO.performLockOperation, and then called sleep, (which is what we're doing in this repro in order to simulate a slow IO operation.) So, what do the other threads look like? Here's thread 26:

0:029> ~26kb
ntdll!ZwWaitForMultipleObjects+0xa
kernel32!WaitForMultipleObjectsEx+0x10b
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
mscorwks!Thread::DoAppropriateAptStateWait+0x41
mscorwks!Thread::DoAppropriateWaitWorker+0x191
mscorwks!Thread::DoAppropriateWait+0x5c
mscorwks!CLREvent::WaitEx+0xbe
mscorwks!AwareLock::EnterEpilog+0xc9
mscorwks!AwareLock::Enter+0x72
mscorwks!AwareLock::Contention+0x1fb
mscorwks!JITutil_MonContention+0xdf
App_Code_tvh3usvo!FileIO.performLockOperation()+0x7f
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b

This is one of the threads waiting for the critical section. It then found that it was unable to get a lock and so began waiting for the critical section on thread 24 to end.

Resolving the problem

This is the tricky part. There is no quick-fix for this. We can use the debugger to identify the bottleneck, but after that we'll need to review our design in order to get this working smoothly. A couple of generic things to try would be the following:

Review what you're doing inside the critical section. Is it really necessary to perform all of the operations you're doing, or would it be possible to do some/all of them outside the scope of the critical section?
Identify the reason for the lock taking so much time. Can it be resolved by load-balancing or clustering the resource we're waiting for?
If we're waiting for another server, make sure the physical infrastructure is okay. Are network response times within the expected parameters or do you need to fix your wiring, router, etc?
Are all calls to the critical section taking the same amount of time, or are there certain calls that take an exceptional amount of time? Perhaps most calls to the function are just fine, but intermittently a call comes along that tries to get everything from the AD, database, etc. In order to investigate this I'd use the commands outlined in Getting started with windbg -part II. Identify the thread owning the lock and investigate the callstack and stack objects using !clrstack and !dso. Use !dso and !do to query the stack objects and try to find the SQL query or AD query that is causing the thread to take such a long time to finish. Is the query significantly different from the ones waiting to execute, then this is most likely our culprit.

Post Script

A quick thought on locks and their limits:

Locks do not scale well. A lock can only be made within a specific process, so if you're using a web farm, or in any other way more than one w3wp.exe, then you will not be able to lock all processes. In other words. Server 1 can get a lock, and so can Server 2. Both will happily execute their code independent of each other. For products such as SQL server you'd deal with this in your SQL query, but if you're performing other operations, such as File IO you might have to work around this by using the same approach as Ms Word. Create a lock file in a shared space, and before you enter your critical section you simply check if the file exists:

You are entering your critical section. Look for lock file in shared location
If file exists, then wait
If/when file doesn't/no longer exist, create file and enter critical section
Make sure you have solid exception handling within the critical section so that you under no circumstances leave the critical section without removing the lock file.
When done, remove lock file

/ Johan

Problems with Flash-content in the WebBrowser control

2009-08-06T17:59:34Z

A recent case I worked on involved the following scenario.

Problem description

The client was building a kiosk-like application. I.e. a WinForms application with the WebBrowser control slapped onto it. The application worked fine for the most part, but they'd found that when viewing video via facebook they were only able to view the video the first time. Any additional attempt to view the video would fail with a custom error from the facebook application stating that the video was either restricted or no longer available.

This only happened when viewing the page via the WinForms application. Using IE directly worked just fine. We also found that this problem occurred with both IE7 and IE8 installed, so it seemed not to be related to the browser version.

Other sites using the Adobe Flash player to display video, such as MSN or Youtube worked just fine.

Troubleshooting

Using the Developer tools included in IE8 I found that the facebook application displays a placeholder image with a play-button. Once that button is clicked the .swf player is loaded via client-side script. The .swf player then downloads the .mp4 that is to be displayed and divides it into suitable chunks.

Fiddler traces revealed that the mp4 was downloaded when using IE but not when using the WinForms application. No errors were reported, instead the request was simply never made.

We removed all code behind. All that remained was the On_Load event that set a starting page for the WebBrowser control, still we were unable to successfully load the movie.

Adding facebook to the trusted zone changed nothing.

Using the COM-version of the WebBrowser control still reproduced the problem.

I then suspected restrictions in cross-domain access or similar, (facebook uses a lot of different domains,) but couldn't find anything conclusive.

Finally I stumbled upon the root cause.

Cause

The problem is due to a bug within the Adobe Flash-player: http://bugs.adobe.com/jira/browse/FP-256

The problem occurs when dynamically trying to display a cached .swf application with an external interface through script. So, another quick repro would be the following (Copied from the Adobe bug report):

Create a C# application with an embedded WebBrowser control.
Load a HTML page with no content.
Via JavaScript, add a SWF with external interface to the page via an object tag. (At this point I see server traffic, requesting the SWF.)
From JavaScript, access the external interface of the SWF. Works OK.
Browse to a second web page.
Browse back to the empty one.
Via JavaScript, add the SWF again. Cannot contact the external interface. (There is no server traffic, indicating that the SWF is already being cached.)

Solution

While waiting for a fix from Adobe the only way to circumvent this is to manually remove the cached .swf. Article 262110 describes in detail how you can do this. Unfortunately this can be a bit tricky some times. In the sample repro above you'd resolve it by simply removing all .swf items from the cache upon page-load, but this wouldn't work in the facebook situation, since it might use the same control repeatedly without refreshing the page in between. Depending on your level of control over the target platform it might be easier to simply turn of caching.

/ Johan

Adding Server ROles to an IIS installation on an RODC

2009-05-05T17:10:15Z

In a recent case I was involved with I stumbled upon something quite interesting. The customer had set up a brand new Windows 2008 Server running IIS, and made it into a Read Only Domain Controller (RODC). Having done this they now wanted to add the FTP-server Role to the existing IIS installation and found that they couldn't.

The reason for this is that the RODC is quite picky about what server roles and features it will allow you to install. I recommend looking at the following articles, but they still don't cover everything.

Applications That Are Known to Work with RODCs
http://technet.microsoft.com/en-us/library/cc732790.aspx

Planning for Application Compatibility with RODCs
http://technet.microsoft.com/en-us/library/cc731746.aspx

The main lesson to be learned is that although the application is listed as known to work with RODCs that doesn't necessarily mean it installs well once the server has been promoted to RODC. When it comes to FTP you have two valid options to get this working:

Install the FTP server before promoting the server to RODC
Install the FTP Service 7.5 for IIS7. This is a stand-alone installer and works just fine.

/ Johan

Creating a game for the Zune

2009-03-31T17:34:00Z

I've finally had the time to sit down and create a simple game for my Zune. Rather than running off and creating a complex game requiring a lot of AI, etc I wanted to create a simple application so that I could focus on the XNA-specifics. I decided to make a Code Breaker game, also known as Master Mind. It's a pretty straight-forward game. There is no increasing difficulty curve and no fancy AI to be written, so it seemed like an excellent first attempt.

Prerequisites

Before I could start I needed to install the necessary software. I already had Visual Studio 2008 and the Zune Software. Actually you don't need the full Visual Studio 2008. You can use the completely free Visual C# 2008 Express Edition instead.

You do need to sign up as a creator in the XNA Creators Club. Having done that, download the XNA Game Studio and install it. Now you're set to go.

Creating content

I gathered a few sound effects and whipped up some graphics for my project. I decided that this is what I wanted the game to look like:

The background would be customizable. I wanted to be able to cycle through the available pictures on your device.

The architecture

I decided on a fairly simple architecture for my game. In my main class (Game1.cs) I deal with the necessities. E.g. I load the sound effects that will be used throughout the application, and I create the GraphicsDeviceManager that will be used to draw to the screen. Apart from that I have two classes that I use. MainGame and MainMenu. As their names imply they either show a menu or the game screen. User input is handled in the main class and it then calls methods in the relevant classes. E.g. if the menu is shown and the user clicks on the Zune "squircle" then a menu item has been selected and the event should bubble to the MainMenu class and not to the MainGame class.

I do more or less the same thing in the Draw-method of the main class. Depending on what is shown on screen individual methods of the two classes are called. And the screen is updated accordingly.

Zune features

I made sure to add at least some of the Zune-specific features.

Pictures

Like I've already mentioned I added support for getting backgrounds from the pictures on the device.

Music

I also let the player choose a playlist to be running in the background. (Anything already playing when the game is started will continue to play until the player actually changes this setting) I could have given full access to each individual song, but building an interface for that would have been quite ridiculous. Playlists only seemed like a good idea.

Touch interface

After some usability testing I found that in order to get this game working properly I needed to decide if I should incorporate the touch/sweep interface of the Zune 2.0 devices, or if I should settle for "proper" clicks only. I decided on the latter. Mainly because I still needed to rely on the center click, and it was easy to accidentally brush away to the side when trying to click. It also ment that this application would work with all Zunes.

Other design choices

In the beginning I let the player change colors by clicking up/down and going to the next "peg" by clicking right/left.

This turned out to be a bit complicated. It was quite easy to accidentally click up or down as well which would mean that you not only moved to the next peg, but also changed the previous pegs color. I ended up using the following key settings.

Click Right - Move Right
Click Left - Move Left
Center Click - Change color / Click button
Play / Pause - Change color "back"
Back - Menu

Installing

If you want to install and run this little game you need to go through the motions filed under Prerequisites above. Open the project in Visual Studio 2008 and dock your Zune. You should then be able to deploy the release version or debug the application directly on the device.

Download

You can download the project here

Additional screenshots

Here are some additional screenshots of the game:

I hope you enjoy it!

/ Johan

Internet Explorer 8 Released today

2009-03-19T15:57:00Z

Today we release Internet Explorer 8. You'll be able to download it later today from http://www.microsoft.se/ie8

/ Johan

Glimpsing 10 years into the future

2009-03-09T13:58:27Z

Stephen Elop, the president of Microsoft's Business Division recently held a presentation where he allowed the viewers to glimpse 10 years into the future. It's amazing to look at a "real" interpretation of the future interfaces and technologies rather than the standard hollywood stuff. - You know the holographic projections that for some strange reason still use slow-typing green text on a black background. The video is really inspirational, and what I really liked was how Stephen, after showing the video, went through all the technologies show in the clip and discussed how they could be accomplished.

If you're in a rush you can skip the first part of the presentation and jump to directly to the video at about 14½ minutes. But I really recommend watching the whole thing, especially the final rundown where he discusses the underlying technologies.

/ Johan

Running a scheduled PowerShell script on a pre-Vista OS

2009-03-05T13:00:50Z

It's been fun to see the reception that PowerShell has gotten. People are using it for tons of different stuff and the ingenuity in some of the scenarios is quite impressive. There is one thing, however, that you should be aware of. PowerShell is not an ideal candidate for execution through the Task Scheduler.

Problem

A long-running PowerShell script is being executed using the Task Scheduler. At apparently random intervals the script will fail with a System.Management.Automation.PipelineStoppedException.

Cause

Looking up the exception on MSDN might not shed any light on the situation unless you know what is happening.

The exception thrown when a cmdlet or a Windows PowerShell provider attempts to write to the pipeline or perform a number of other Windows PowerShell operations after the pipeline has been terminated. The pipeline could have been terminated before the call was made or during the call.

When the Task Scheduler executes a PowerShell script it needs to use the .NET Framework. When running the .NET Framework it uses the console session (Session 0). This problem occurs when someone is logged on to the root console (either physically or using Terminal Services) and logs off while the script is executing.

Repro

You can easily reproduce this by creating a small script that sleeps for a while. Schedule the script to run immediately and then log off. To illustrate I've used the following script:

function Main()
{
     Start-Transcript "C:\Test\Psjob.txt"
     'Start-Sleep'
     Start-Sleep (2*60)
     'Ready'
     Stop-Transcript
}

Main

When correctly executed this script will generate a transcript that looks like this:

**********************
Windows PowerShell Transcript Start
Start time: 20090305111600
Username : DOMAIN\SYSTEM
Machine : MACHINENAME (Microsoft Windows NT 5.2.3790 Service Pack 2)
**********************
Transcript started, output file is C:\Test\Psjob.txt
Start-Sleep
Ready
**********************
Windows PowerShell Transcript End
End time: 20090305111800
**********************

However, if I log off the console while the script is executing I get the following transcript:

**********************
Windows PowerShell Transcript Start
Start time: 20090305114300
Username : DOMAIN\SYSTEM
Machine : MACHINENAME (Microsoft Windows NT 5.2.3790 Service Pack 2)
**********************
Transcript started, output file is C:\Test\Psjob.txt
Start-Sleep
**********************
Windows PowerShell Transcript End
End time: 20090305114340
**********************

As you can see the transcript is properly ended as PowerShell shuts down, but if you look at the output in the middle you'll see that we never write "Ready" to the file. This is because the script was prematurely terminated.

Resolution

The quick and easy resolution is to upgrade to Vista, Windows Server 2008 or later, since as of Windows Vista the console session is no longer running in Session 0.

PowerShell - Managing Feature Delegation in IIS7

2008-12-16T16:51:54Z

Perhaps you've read my earlier post on advanced feature delegation using PowerShell and the IIS 7.0 PowerShell Provider. This could actually be called Part II.

The previous post was based on a quick question from a premier customer. He is a never-ending source of intriguing questions and I always enjoy his "just one thing"-type of inquiries. They often force you to think a little extra. Anyway, the other day he came with a question, which (as always) seemed simple enough at a first glance. The goal was to set Feature Delegation for SMTP E-mail And Forms Authentication to "Read Only".

Now, most basic configuration topics have been covered in countless posts, and at first I thought that this shouldn't be so hard. There is, however, a major difference. The SMTP E-mail and Forms Authentication settings are set in the root web.config. Not in applicationhost.config.

If we open up and compare web.config before, and after we change the setting, we'll notice that the change made by the IIS Manager is adding the following into the root <configuration>-block:

Okay, so using what we already know from the old post we should put together a call that looks something like this:

Set-WebConfiguration "/System.Net/mailSettings/smtp" -value @{<something... overrideMode="Deny", maybe?>} -PSPath IIS:\

Please note that this is NOT the correct syntax. It is close, but no cigar. What we need to do is:

Correctly set the overrideMode. The above sample is not the way to do it.
Make sure the changes are saved to web.config

So, the final, (and correct,) call to the Set-WebConfiguration cmdlet would look like this:

Set-WebConfiguration //System.Net/mailSettings/smtp -metadata overrideMode -value Deny -PSPath MACHINE/WEBROOT

The Forms Authentication call would look like this:

Set-WebConfiguration //System.Web/authentication -metadata overrideMode -value Deny -PSPath MACHINE/WEBROOT

I've searched quite extensively for samples on this and so far I've found none. So I thought someone should write it down. :)

Happy Holidays! / Johan

Office Automation

2008-10-14T16:15:30Z

A very common scenario that keeps sprouting new heads like a hydra is Office Automation. Let me start by saying that this is not supported.

There is a KB-article number 257757 discusses this, and clearly states the following:

Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.

Why is this not supported?

The crucial thing to consider is the fact that Microsoft Office is designed to be an end-user, single client product. Automating Microsoft Office in a client application, using the identity and security context of the logged on client is supported, but unattended execution is not.

There are a number of things that can go wrong and article 257757 lists most of them. For arguments sake, let's consider the following scenario:

Your server application starts up and uses CreateObject to create an instance of MS Word. Word tries to read the settings of the current client and since your application is using the Network Service account this presents an immediate problem. There are, for example, certain methods that rely on a default printer being installed. All this can lead to serious problems. You then need to monitor the server, since Office might show a modal dialogue for some reason. The "install on first use" feature of MSI might also kick in, prompting the client to install additional features. All this would hang the current thread. Also, running one instance of Word, might be okay, but what happens when you get 100 more or less concurrent requests and each request starts up an instance of Word? MS Word is not a server, it's an excellent piece of single-client software.

So what should I do instead?

Actually, most of your problems can be resolved using HTML. I've seen applications where lines and lines of complex code were used for generating an Excel document, when it could just as well have been made using a standard HTML-table. Excel would have no problems whatsoever reading a table like the one below:

<table>
    <tr>
        <td><b>Person</b></td>
        <td><b>Age</b></td>
    </tr>
    <tr>
        <td>Pete</td>
        <td>30</td>
    </tr>
    <tr>
        <td>Claire</td>
        <td>40</td>
    </tr>
    <tr>
        <td>Average age</td>
        <td>=AVERAGE(B2:B3)</td>
    </tr>
</table>

As you can see all formatting, and even the functions, would be interpreted correctly.

Obviously Word is great with HTML as well, so most formatting issues can quite easily be resolved using this approach.

So how do I do this?

The solution is quite easy. In order to generate an Excel document all you need to do is to create a table with the data you want, and add the following two lines to the Page_Load event of your page:

Response.ContentType = "application/vnd.ms-excel";
Response.AddHeader("Content-Disposition", "attachment; filename=Data.xls;");

The first line, sets the returned ContentType to Excel, and the second changes the filename of the returned file. This means that the client will be prompted to Open/Save the file, and the filename will be set to "Data.xls" rather than "Default.aspx" (or whatever your original document may be called.)

If you wanted to return a word document you'd set the ContentType to "application/msword" instead.

One little thing to consider is the fact that by default you're probably adding a lot of redundant information to your webpage. You might want to remove all excessive HTML from the page so that you remove all unnecessary headers, stylesheets, viewstate, etc. For an example, please consider the code below:

Response.Clear();
Response.Buffer = true;
Response.ContentType = "application/vnd.ms-excel";
Response.AddHeader("Content-Disposition", "attachment; filename=Data.xls;");
this.EnableViewState = false;
System.IO.StringWriter oSw = new System.IO.StringWriter();
System.Web.UI.HtmlTextWriter oHtml = new System.Web.UI.HtmlTextWriter(oSw);
Table oTable = new Table();
for (int ctr = 1; ctr <= 10; ctr++)
{
    TableRow oRow = new TableRow();
    TableCell oCell1 = new TableCell();
    oCell1.Text = ctr.ToString();
    TableCell oCell2 = new TableCell();
    oCell2.Text = (ctr*10).ToString();
    oRow.Cells.Add(oCell1);
    oRow.Cells.Add(oCell2);
    oTable.Rows.Add(oRow);
}
oTable.RenderControl(oHtml);
Response.Write(oSw.ToString());
Response.End();

Security issues in Office 2007

If you're running Office 2007 you might come across the following error message:

"The file you are trying to open, 'Data.xls', is in a different format than specified by the file extension. Verify that the file is not corrupted and is from a trusted source before opening the file. Do you want to open the file now?

This is completely by design. In this example Excel has noticed that though the file is named .xls it does in fact contain html, so it's warning us about this inconsistency. We might bypass this by actually changing the file extension to htm or, perhaps saving the data as comma separated values (csv) which Excel also supports, but both options would be opened directly in the browser rather than passed on to Excel, so this is a design consideration we have to take into account.

There is really no way of bypassing this security feature other than manually disabling it, client-side, using a registry hack. (The key is HKCU\Software\Microsoft\Office\12.0\Excel\Security\ Add a DWORD named xtensionHardening and set it to 0.) For obvious reasons this is not something that is generally recommended.

Other options

There are also third-party products available for generating MS Office-compatible documents, but there are also some articles in the knowledge base on the subject. For example this little gem which can easily be adopted to ASP.NET as well:

How to use ASP to generate a Rich Text Format (RTF) document to stream to Microsoft Word

/ Johan

PowerShell - Advanced configuration editing in IIS7

2008-10-02T16:28:18Z

I've written a lot of PowerShell posts lately and here's another one. :-)

I got a question from one of the account managers if it was possible to alter the FTP Authorization Rules for a specific folder on his IIS.

The appcmd for the operation was

appcmd.exe set config "FTPFolder" -section:system.ftpServer/security/authorization /+"[accessType='Allow',users='*',roles='*',permissions='Read, Write']" /commit:apphost

He wanted to know if there was a PowerShell equivalent. Sure, you could use the appcmd directly from powershell, but if you're going to use the command line for everything, then what's the use of PowerShell?

The section in applicationhost.config that we want to edit is the following:

System Specs

As you might have noticed from the screenshots already he was Running Windows 2008 with FTP 7 installed. He also had the IIS 7.0 PowerShell Provider installed.

Troubleshooting

There are a lot of cool things you can do with the WebSites using the IIS7 PowerShell provider. Below is a sample copied from iis.net. To use it properly, go to an IIS directory such as IIS:\DemoSite\DemoApp:

$winAuth = Get-WebConfiguration -filter /system.webServer/security/authentication/windowsAuthentication
$winAuth.enabled = $false
$winAuth | set-Webconfiguration -filter /system.webServer/security/authentication/windowsAuthentication -PSPath IIS:\ -location "DemoSite/DemoApp"

So judging from the sample above we should simply have to set anonymousAuthentication.enabled = $true or something like that. Unfortunately this isn't the case. If you try to access the section of applicationhost.config that we're attempting to edit (see above) you will see that we don't have any applicable properties for the object. $ftpAuth = Get-WebConfiguration -filter /system.ftpServer/security/authorization will not throw any exceptions, but using TAB to cycle through the properties of $ftpAuth will show us no immediate properties or methods of use.

Solution

The solution in this scenario is to use the Add-WebConfiguration cmdlet. The applicationhost.config can easily be translated to a valid Add-WebConfiguration call:

Add-WebConfiguration "/system.ftpServer/security/authorization" -value @{accessType="Allow";users="?";permissions=3} -PSPath IIS:\ -location FTPFolder

The only thing that sticks out as being out of the ordinary is the permissions=3 setting. Why isn't it "Read, Write"? Actually, passing "Read, Write" as a parameter will not work. It will leave the permissions setting blank. "Read" or "Write" only will work, but not both together. I've tried figuring out if there's a valid way of passing both arguments, but so far I've drawn blanks. I've tried putting them in an array {"Read";"Write"}, passing them as "ReadWrite", "Read;Write", "Read+Write" etc. but the only way I've found so far is to pass the value 3 which is the obvious sum of the two enum values for Read and Write.

Later! / Johan

PowerShell - Editing permissions on a file or folder

2008-10-01T15:52:49Z

I got the following question from a reader the other day:

I've been trying to figure out how to change permissions on a folder in PowerShell. I've looked at the Get-Acl and Set-Acl, but I can only use them to copy the settings from a pre-existing object. How do I manually configure permissions?

This is actually a quite common question, so I thought I'd write a quick post on the subject.

Get-Acl and Set-Acl

To quote the PowerShell documentation "Get-Acl gets the security descriptor for a resource, such as a file or registry key." while "Set-Acl changes the security descriptor of a specified resource, such as a file or a registry key." In other words; if you want Folder_A to have the exact same permissions as Folder_B, then you simply copy the Access Control List (ACL) of Folder_B and "paste" it onto Folder_A.

$Acl = Get-Acl "C:\Folder_B"
Set-Acl "C:\Folder_A" $Acl

So far, so good.

Changing the ACL

Okay, so you want to change the ACL. Here's some sample code for how to do that:

New-Item -type directory -path C:\MyFolder
$Acl = Get-Acl "C:\MyFolder"
$Ar = New-Object  system.security.accesscontrol.filesystemaccessrule("username","FullControl","Allow")
$Acl.SetAccessRule($Ar)
Set-Acl "C:\MyFolder" $Acl

So, first we create a new folder. We then copy the ACL of that folder. We then create a new AccessRule that gives "username" full control. We then add this AccessRule to the ACL, and finally we reapply the new, altered ACL to the folder.

If we wanted to we could also have used $Acl.RemoveAccessRule($Ar) or possibly $Acl.RemoveAccessRuleAll() as well.

/ Johan

PowerShell - Automatically organizing your mp3-collection

2008-09-23T14:53:23Z

Is your mp3-collection perfectly sorted? I know mine wasn't. I thought I'd address that while creating a pretty good demo-script to show some basic filtering, script structures, etc. Since I also wanted the script to extract the metadata from every file in my collection I had to use the Shell.Application Com-object as well. All in all I think it turned out pretty well.

NOTE:

This script is provided completely as is. It is not an official product in any way and I take no responsibility for any harm it may cause.

Please note that certain metadata editors that integrate directly into the Shell may cause some severe confusion in the metadata, so I'd recommend disabling them before trying. Off course I would also recommend making a backup copy of the music library before surrendering it to the mercy of PowerShell.

If you're new to PowerShell and haven't done so already I'd recommend looking at my earlier posts on the subject:
PowerShell - An introduction, Part I
PowerShell - An introduction, Part II

The script

Param([String] $Folder)
$INVALIDCHARS = [System.IO.Path]::GetInvalidPathChars() + "/", "\", "*", "?", ":"
$MUSICATTRIBS = "*.m4a", "*.m4b", "*.mp3", "*.mp4", "*.wma", "*.flc"
$PLAYLISTATTRIBS = "*.m3u", "*.zpl"
$ALLATTRIBS = $MUSICATTRIBS + $PLAYLISTATTRIBS
$PLAYLISTSFOLDER = "Playlists"
$objShell = New-Object -ComObject Shell.Application
$iTotalFiles = 0
$iCurrentFile = 0

function checkFolderName($FolderName)
{
	# Make sure the folder doesn't contain any invalid characters

	if(!$FolderName) { $FolderName = "Unknown" }
	$INVALIDCHARS | % {$FolderName = $FolderName.replace($_, "")}
	return $FolderName
}



function MoveFiles($startDir)
{
	Write-Host "Indexing playlists..."
	
	# Do a recursive DIR in the starting directory, include everything with the attributes of a playlist.
	# Filter the result by excluding everything that is a Container (folder)
	$dirResult = get-childitem -force -path $startDir -recurse -include $PLAYLISTATTRIBS | where{! $_.PSIsContainer}
	if($dirResult)
	{
		Write-Host "Moving playlists..."

		foreach($dirItem in $dirResult)
		{
			move-item $dirItem.FullName ($PLAYLISTSFOLDER + "\" + $dirItem.Name)
		}
	}
	Write-Host "Indexing music..."

	# Do a recursive DIR in the starting directory, include everything with the attributes of a music file.
	# Again we filter the result by excluding everything that is a Container.
	$dirResult = get-childitem -force -path $startDir -recurse -include $MUSICATTRIBS | where{! $_.PSIsContainer}
	
	# Make a note of how many hits we got, so we can show the progress to the client.
	$iTotalFiles = $dirResult.Count
	if($dirResult)
	{
		foreach($dirItem in $dirResult)
		{
			# Up the counter for how many files we've processed so that we can show progress
			$iCurrentFile +=1

			# Get the metadata for the file
			$fileData = getMP3MetaData($dirItem.FullName)
			
			# Find the path where we the song should be stored
			$ArtistPath = $fileData["Album Artist"]
			if (!$ArtistPath) { $ArtistPath = $fileData["Artists"] }
			if (!$ArtistPath) { $ArtistName = "Unknown" }

			# Make shure it's a valid path
			$ArtistPath = checkFolderName($ArtistPath)
			$AlbumPath = checkFolderName($fileData.Album)
			$ArtistPath = join-path $startDir $ArtistPath 
			$AlbumPath = join-path $ArtistPath $AlbumPath

			# Check if the file should be moved
			if($dirItem.DirectoryName -ne $AlbumPath)
			{
				if(!(test-path $ArtistPath)) # If the Artist folder doesn't exist
				{
					MKDIR $ArtistPath | out-null
				}
				if(!(test-path $AlbumPath)) # If the Album folder doesn't exist
				{
					MKDIR $AlbumPath | out-null
				}
				move-item $dirItem.FullName ($AlbumPath + "\" + $dirItem.Name)
			}

			# Show progress
			$percentage = ([int](($iCurrentFile / $iTotalFiles)*100))
			cls
			Write-Host "$percentage% ($iCurrentFile files of $iTotalFiles) complete"
		}
	}
}


function removeEmptyFolders($startDir)
{
	# Get all folders and subfolders
	$dirResult = Get-childitem $startDir -recurse | where{$_.PSIsContainer}
	if($dirResult)
	{
		foreach($dirItem in $dirResult)
		{
			# If the folder is empty (Doesn't contain any music or playlists) it should be deleted
			if(isfolderempty($dirItem.FullName))
			{
				# Check if the path is still valid, we may have deleted the folder already recursively.
				if(Test-Path $dirItem.FullName)
				{
					remove-item -path $dirItem.FullName -force -recurse
				}
			}
		}
	}
}

function removeOtherFiles($startDir)
{
	# Get all non-music files (except *.jpg-files) and remove them 
	$dirResult = Get-childitem $startDir -recurse -exclude ($ALLATTRIBS + "*.jpg") | where{! $_.PSIsContainer}
	if($dirResult)
	{
		foreach($dirItem in $dirResult)
		{
			if(Test-Path $dirItem.FullName)
			{
				# Make sure the file still exists, and hasn't been deleted already
				remove-item -path $dirItem.FullName -force
			}
		}
	}
}

function isFolderEmpty($folderPath)
{
	# Search for any remaining music items in the folder.
	if(Test-Path $folderPath)
	{
		$dirChildren = get-childitem -force -path $folderPath -recurse -include $ALLATTRIBS | where{! $_.PSIsContainer}
		if($dirChildren -ne $null)
		{
			return $false
		}
		else
		{
			return $true
		}
	}
	else
	{
		# No use trying to delete the folder if it doesn't exist
		return $false
	}
}

function getMP3MetaData($path)
{
	# Get the file name, and the folder it exists in
	$file = split-path $path -leaf
	$path = split-path $path
	$objFolder = $objShell.namespace($path)
	$objFile = $objFolder.parsename($file)
	$result = @{}
	0..266 | % {
		if ($objFolder.getDetailsOf($objFile, $_))
		{
			$result[$($objFolder.getDetailsOf($objFolder.items, $_))] = $objFolder.getDetailsOf($objFile, $_)
		}
	}
	return $result
}

# MAIN FUNCTION

# If no argument was passed, see if anything was piped to the function instead.
if (!$Folder)
{
	$input | % { $Folder = $_ }
}

# Check if the path is valid, if not reset it to ""
if ($Folder) { if (!(Test-Path -path $Folder)) { $Folder = "" }}

# Since no valid path was given, prompt the user for a proper path.
while (!$Folder)
{
	"Please enter the full path to the folder where you wish to rearrange your MP3's"
	$Folder = Read-Host
	if ($Folder) { if (!(Test-Path -path $Folder)) { $Folder = "" }}
}

# Make sure we have a folder to store the playlists in.
$PLAYLISTSFOLDER = join-path $Folder $PLAYLISTSFOLDER
if(!(test-path $PLAYLISTSFOLDER)) # If the Playlists folder doesn't exist
{
	MKDIR $PLAYLISTSFOLDER | out-null
}

# Now that everything is ready, begin rearranging the files.
MoveFiles($Folder)
removeEmptyFolders($Folder)
removeOtherFiles($Folder)

Running the script

So, how do you run this script?

It's quite easy. The first step would be to copy it into notepad save it with a .ps1 extension and open up PowerShell

Execution policy

If this is the first time you try to execute a script from PowerShell you will most likely have to change the execution policy for PowerShell. I'd recommend setting it to RemoteSigned. This (obviously) means that all remote scripts must be signed or PowerShell will refuse to execute them. To change the execution policy simply type:

Set-ExecutionPolicy RemoteSigned

That's it.

If you're uncertain what your execution policy setting is you simply type:

Get-ExecutionPolicy

Okay, having changed that we can now execute the script. If you saved the script in a folder that is in your system path you only need to type the filename of the .ps1 file. (The .ps1 is optional) So, if you saved it as RearrangeMP3s.ps1 you'd write RearrangeMP3s and press enter

Breakdown

Okay, so let's take a look at the separate parts of the script.

Initial variables and constants

First we declare the parameters we're going to use. In this case it's just one: $Folder.

After that we have some constants. You may want to change $MUSICATTRIBS or $PLAYLISTATTRIBS to include, or remove, extensions based on your needs. $PLAYLISTSFOLDER is a constant which indicates in which subfolder you wish to store your playlists.

Finally we have some variables. $objShell is used for file system operations, and rather than creating a new object all the time I'd rather have a global one. $iTotalFiles and $iCurrentFile are two integers used to create a progress indicator.

Functions

After the initial variables come the sub functions called by the main function. To see what each individual function does I'd recommend looking at the comments. I've prioritized readability over minimum number of keystrokes, so yes, instead of writing:

foreach($dirItem in $dirResult)
{
	move-item $dirItem.FullName ($PLAYLISTSFOLDER + "\" + $dirItem.Name)
}

I could have written

$dirResult | % { mv $_.FullName ($PLAYLISTSFOLDER + "\" + $_.Name) }

But in this case I think that would have been counterproductive.

The main function

Finally we have the main function. It is written last in the script. Why is that? Well, actually PowerShell is the most sequential language you'll ever come across. It will not bother to check the entire script for function declarations. Instead it will go through the script one line at a time. This means that the following script will not work:

SayHello

function SayHello
{
	Write-Host "Hello World"
}

When you call SayHello in the sample above the function has not yet been declared, so PowerShell will throw an exception in your face saying that SayHello is unrecognized.

Footnote

There is actually a supported way of achieving the very same thing, but where's the fun in that?

Open up Windows Media Player
Go to Options -> Library and tick the checkbox for "Rearrange music in rip folder"
Go to the "Rip Music"-tab
Make sure the path is correct in the "Rip music to this location"-section
Click OK
Select all files in the library
Right-click and choose the "Advanced Tag Editor"-option
Do not change anything. Simply click Apply
Windows Media Player will now rearrange all the files using your Rip settings.

/ Johan

PowerShell - An introduction, Part II

2008-08-25T14:22:12Z

This is a continuation of part I. If you haven't read it I suggest at least going through the summary at the end of the post.

Working with Drives

In your basic command prompt you have the normal file system drives that you use to access your hard-drive, floppy, DVD, USB-stick, network shares etc. PowerShell uses this established interface to access other data sources as well. The following providers are accessible by default through drives in PowerShell 1.0:

Registry (HKLM: or HKCU:)
Certificate store (Cert:)
Environment variables (Env:)
Aliases (Alias:)
Functions (Function:)
Variables (Variable:)

When you install extensions for like the one fore IIS, etc. you usually get another drive accessible where you can look at the current webapplications and check their properties.

Navigating

Selecting one of the PowerShell drives is easy. It is almost like in MS-DOS.

PS C:\>CD HKLM:
PS HKLM:\> DIR

You can use the same commands as you'd do in MS-DOS, such as CD, DIR, MKDIR, etc.

To get a certain property for a registry key, (or a regular folder for that matter) you use the Get-ItemProperty cmdlet.

PS HKLM:\> CD system
PS HKLM:\system> CD CurrentControlSet\Control
PS HKLM:\system\CurrentControlSet\Control> Get-ItemProperty CrashControl

PSPath           : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\system\CurrentControlSet\Control\CrashControl
PSParentPath     : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\system\CurrentControlSet\Control
PSChildName      : CrashControl
PSDrive          : HKLM
PSProvider       : Microsoft.PowerShell.Core\Registry
AutoReboot       : 1
CrashDumpEnabled : 2
Overwrite        : 1
LogEvent         : 1
DumpFile         : C:\Windows\MEMORY.DMP
MinidumpDir      : C:\Windows\Minidump

All in all it's more or less the same working with Aliases or Environment variables. Note the following difference in output, though:

PS Env:\> Dir Env:PROCESSOR_ARCHITECTURE

Name                           Value
----                           -----
PROCESSOR_ARCHITECTURE         AMD64

PS Env:\> $Env:PROCESSOR_ARCHITECTURE
AMD64
PS Env:\>

Did you note how putting a $-sign before the variable name gave us the value of the variable instead of a complete object? This brings us quite nicely to the subject of variables.

Variables

Variables aren't that tricky to work with. They do not need to be pre-initiated. They do, however, require a $ to be identified as variables. Otherwise you get the following error message:

PS C:\> $Name = "Johan"
PS C:\> Name = "Johan"
The term 'Name' is not recognized as a cmdlet, function, operable program, or script file. Verify the term and try again.
At line:1 char:5
+ Name <<<< = "Johan"
PS C:\>

To display the value of a variable you simply type the name of the variable and PowerShell kindly returns the contents

PS C:\> $Name
Johan

When it comes to strings, you can choose for yourself if you want to use double quotes or single quotes, but there is one minor difference:

PS C:\> $str1 = "Hello"
PS C:\> $str2 = "World!"
PS C:\> $strCombo1 = "$str1 $str2"
PS C:\> $strCombo2 = '$str1 $str2'
PS C:\> $strCombo1
Hello World!
PS C:\> $strCombo2
$str1 $str2
PS C:\>

As you can see, a variable within double quotes will be evaluated, while within single quotes it will be interpreted literally.

Apart from strings you can also store integers and dates. Integers do not need delimiters and dates are most easily created using the Get-Date function

PS C:\> $birthday = 11/15/1973

Arrays

Declaring Arrays is a breeze. PowerShell will automatically Split any comma-separated sequence of values so creating a arrays is as easy as this:

PS C:\> $Colors = "Red", "Green", "Blue"

The arrays are also dynamic by default, so adding values to them is a simple matter as well.

PS C:\> $Colors += "Cyan", "Magenta", "Yellow"

Creating two-dimensional arrays are done by declaring each row of the array as a one-dimensional array and then adding them to another array:

PS C:\> $arr1 = 1, 2, 3
PS C:\> $arr2 = "A", "B", "C"
PS C:\> $arr3 = $arr1, $arr2
PS C:\> $arr3[0][0]
1
PS C:\> $arr3[1][1]
B
PS C:\>

User input

Finally, to get dynamic user input you can use the Read-Host cmdlet.

$Name = Read-Host "What is your name?"

Summary

Okay, so to summarize we have now covered the following:

PowerShell uses system drives to access the registry, environment variables, etc.
Variables are preceded with a $
To access the value of a variable, simply type it's name.
Arrays are comma-separated lists of variables
User input can be handled by the Read-Host cmdlet

Next post will probably cover some script logic such as if-clauses, looping, etc. After that I think it will be time to post some sample scripts.

/ Johan

Back in business

2008-08-22T18:00:00Z

Okay, after a summer break that no doubt got a little longer than expected I am now I'm now back in the saddle again.

Next post will most likely be a follow-up on the PowerShell post I wrote a while back. Stay tuned.

/ Johan

The first thing you should do after setting up an IIS7-server

2008-05-29T15:14:58Z

As you're probably well aware IIS7 relies heavily on a master configuration file named ApplicationHost.config. Now, instead of using a history file like IIS6 did, IIS7 will check this file every 2 minutes to see if it's changed. If the file has been updated, then a backup copy of the old .config will be stored in the inetpub\history-directory.

Configuration history

This is really cool and a very useful feature. There's only one little caveat in my opinion and that's the fact that by default IIS7 will only save up to 10 backups. All the other backups will be discarded. This means that if you're spending 20 minutes fiddling with the settings in ApplicationHost.config you will quite likely have overwritten all your old backups. Now, if something goes wrong at that point and you decide to spend another 20 minutes trying to fix this problem by further editing the ApplicationHost.config you will surely have overwritten the backups.

In a worst case scenario you might even have gone from Healthy Server to Unstable Server to Server Down and you now find that you aren't even able to restore it to the Unstable state.

My suggestion is to increase the default number of backups to 100, or why not 1000? You can then feel certain that any backups taken aren't overwritten when you try to undo the very same changes. Here's how you do it:

Go to %windir%\system32\inetsrv\config\
Open applicationHost.config in notepad
Locate the system.applicationHost section group
Add the following item to the section group:
<configHistory maxHistories="100" period="00:02:00" />
(Naturally the maxHistories value can be whichever value you prefer)
Save

To restore one of the previous configurations, simply go to inetpub\history\cfgHistory_[nnnnnnnnnn] and copy the config from there, or use appcmd. (See below)

Backup using appcmd

Now that we've done this there is one more thing I'd strongly recommend doing:

Open up a command-line with administrative priviliges
Go to %windir%\system32\inetsrv
Run "appcmd add backup "[Name of your backup]""

What you've done now is to create a manual backup of your current config. It's always good to have a decent (and persistent) restore point to return to.

If you wish to restore this backup you simply use the following syntax: "appcmd restore backup "[Name of your backup]"". You can also delete a backup using the "delete" keyword, or you can list all backups, including the ones generated by the Configuration history, like this: "appcmd list backup"

Okay, so now we have made sure we have some extensive automatic backups in place which might save us a lot of headaches later on.

Later! / Johan