CodeDOM is used by several other library components including WinForms, ASP.NET, XSLT and Web Services.  It can be used to automatically generate code at design time, such as from a visual form, and it can be used to generate code at runtime, "on the fly", for performance.  It provides fast performance because the code is compiled into an assembly, loaded by the CLR and then JIT compiled into native code.  It's been used successfully in many contexts since it was first released in .NET 1.0.

One aspect of this that may catch some customers offguard is that assemblies are never unloaded by the CLR, aside from entire AppDomain unloads.  Therefore each assembly created by CodeDOM will remain in the address space even if it is no longer used.  For most applications, this doesn't present a problem - there is a bound to the number of assemblies the applications loads, and when they are no longer needed, the pages can be swapped out to disk by the Windows Virtual Memory (VM) subsystem, so they don't affect the rest of the application.  We have seen a few cases where the application uses CodeDOM repeatedly without bound, however, leading to out-of-memory and related exceptions.  When we examined these situations, we noticed that there was actually lots of free memory, but it was so fragmented that it couldn't be used.  We were puzzled until we examined how many assemblies were loaded into the process.  We found nearly 40000!  The fragmentation arose because each assembly under 64 KB in size is loaded into a segment of 64 KB.  So a 16 KB assembly would cause 48 KB of address space to be wasted.  Repeat thousands of time and it's not hard to exhaust the available address space of a 32-bit CPU. 

The best way to avoid this situation is to limit the number of times your application generates and loads a new assembly.  Cache the assembly in memory rather than regenerating it.  For example, if you use the XSLTransform type, cache the result of the Load method rather than calling it repeatedly for the same XSL document.  Assuming you have less than 40000 XSL documents in use, that will completely solve your problem, and reduce the number of costly compilations to boot.

Here are some details of our investigation.  The exception we were seeing actually happened when we didn't have enough contiguous memory to start up an external process. 

Exception type: System.Runtime.InteropServices.ExternalException

Message: Cannot execute a program. The command being executed was "C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\csc.exe" /noconfig /fullpaths @"C:\WINDOWS\TEMP\rvnypb_j.cmdline".

InnerException: <none>

StackTrace (generated):<none>

StackTraceString: <none>

HResult: 8

Here's some memory information from Windbg.  The last line indicates large amounts of fragmentation, leaving only individual 64KB segments available.

-------------------- State SUMMARY --------------------------

    TotSize (      KB)   Pct(Tots)  Usage

   31b0d000 (  814132) : 25.88%   : MEM_COMMIT

   7c5b9000 ( 2037476) : 64.77%   : MEM_FREE

   11f2a000 (  294056) : 09.35%   : MEM_RESERVE

 

Largest free region: Base 00000000 - Size 00010000 (64 KB)

Howard van Rooijen is doing interesting things with Team Foundation Server events, and sharing his work.  That's great to see.  There are a few things I'd like to add to his post.  There are some important events not listed on his page.  One is the CheckinEvent.  It is raised with each version control checkin.  Another is the ProjectCreatedEvent.  It is fired at the very end of new project creation.  The CheckinEvent type can be found in the Microsoft.TeamFoundation.VersionControl.Common.Integration.dll assembly, while the ProjectCreatedEvent (and ProjectDeletedEvent too) may be found in the Microsoft.TeamFoundation.dll assembly.  Other events in that assembly are: NodeCreatedEvent, NodeRenamedEvent and NodesDeletedEvent

Also, a few of the events are listed in his post but are no longer raised, or raised inconsistently.  They are the IdentityCreatedEvent, the IdentityDeletedEvent, the MembershipChangedEvent and the AclChangedEvent.  It's best not to use these events.

Thanks Howard.

We tried to be generally case insensitive in Tfs, in alignment with the default settings in Sql Server and Windows.  There's one place where we messed up though.  If you create a subscription to a "CheckInEvent" through the command-line tool, you won't get any alerts.  That's because the event type is really "CheckinEvent" -- small 'i'.  We do the type matching based on a hashtable lookup, and the hashcode for "CheckinEvent" is different from the hashcode for "CheckInEvent".  The unfortunate result is that you won't receive any notifications.  We will certainly fix it in the next version.  Until then, be careful.

I'm posting this for a colleague - Naren Datha.  It's a document he wrote for developers writing web service/application Team System add-ons that rely on the Work Item Tracking object model. 

If you've written any web services that listen for Team Foundation events for any previous beta releases, you'll want to update your web method for RC.  We made one (hopefully last) change to the Action and RequestNamespace.  The "02" became an "03".  If you don't change this, you will not get notified and you will see an error about an "unrecognized Soap Action".  Here is one valid way to write the webmethod:

[SoapDocumentMethod(Action = "http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Notification/03/Notify", 
  RequestNamespace="http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Notification/03"
)]
[WebMethod]
public void Notify(string eventXml)
{
  ...
}

You’ve read Inside Sql Server 2000 cover to cover, but how well do you really know the database locking behavior?  Here's a 10 question quiz (attached) to find out.

If you want to get notified for all new workitems, regardless of who they get assigned to, you will want a custom subscription.  Here is an example of how to create one with the BisSubscribe tool that is on the server.  There is a ChangeType with each WorkItemChangedEvent, set to either "Change" or "New".  In the filter expression below, only new changes will generate emails.

BisSubscribe.exe /userId <domain\username> /eventType WorkItemChangedEvent /deliveryType EmailHtml /domain <server name> /address <email address> /filter "PortfolioProject = '<project>' AND ChangeType = 'New' "

<domain\username> - user getting the notification
<server name> - your server
<email address> - full email address to receive the notification
<project> - project title of interest

Note, if you are using a build later that Beta3, the syntax should be modified slightly to the following:

BisSubscribe.exe /eventType WorkItemChangedEvent /deliveryType EmailHtml /server <server name> /address <email address> /filter "PortfolioProject = '<project>' AND ChangeType = 'New' "

Where there is no "/userId" and "/server" replaces "/domain"

I did some experiments interacting with Tfs through IronPython, and it's been enjoyable.  They've done a great job making .NET available in a dynamic way through the Python language.  IronPython can run as an interactive toplevel, where it responds to your commands as you give them.  That's good because you get immediate feedback.  Instead of a lengthy cycle of code - compile - launch - navigate to the location of the change - see result cycle, it's a quick try method - see result cycle.  You can also quickly ask the system for a list of available methods of an object.  I was able to test a few WorkItem queries directly to the API.  See the result below.

I installed IronPython on my client machine and entered the following at the IronPythonConsole.exe prompt (”>>>”):

servername = "myServerName"

privateAssembliesPath = "c:\\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies"

import clr

clr.AddReferenceByPartialName("Microsoft.TeamFoundation")

clr.AddReferenceByPartialName("Microsoft.TeamFoundation.Common")

clr.AddReferenceByPartialName("Microsoft.TeamFoundation.Client")

clr.AddReferenceToFile(privateAssembliesPath + "\\Microsoft.TeamFoundation.WorkItemTracking.Client.dll")

from Microsoft.TeamFoundation.Server import *

from Microsoft.TeamFoundation.Proxy import *

from Microsoft.TeamFoundation.Client import *

from Microsoft.TeamFoundation.WorkItemTracking.Client import *

tfs = TeamFoundationServerFactory.GetServer(servername)

wis = tfs.GetService(WorkItemStore)

for p in wis.Projects:

  print p.Name

query = "SELECT [System.Id],[System.State], [System.Title] FROM WorkItems \

WHERE [System.State] <> 'Closed' ORDER BY \

[System.WorkItemType], [System.Id]"

workitems = wis.Query(query)

for wi in workitems:

  print wi.Title

    <system.diagnostics>
        <assert assertuienabled="false" />
        <trace autoflush="false" indentsize="4" />
        <!--  Trace Switches
            Each of the trace switches should be set to a value between 0 and 4, inclusive.
              0: No trace output
              1-4: Increasing levels of trace output; see Systems.Diagnostics.TraceLevel
        -->
        <switches>
            <add name="API" value="0" />
            <add name="Authentication" value="0" />
            <add name="Authorization" value="0" />
            <add name="Database" value="0" />
            <add name="General" value="3" />
            <!-- WorkItem trace switches -->
            <add name="traceLevel" value="1" />
        </switches>        
    </system.diagnostics>

        <listeners>
           <add name="myListener" type="System.Diagnostics.TextWriterTraceListener"
               initializeData="c:\windows\temp\myListener.log" />
        </listeners>

[Info, PID 3456, TID 3040, 22:59:25.296] Calling notification filter: Microsoft.TeamFoundation.VersionControl.Server.CheckinEventFilter 
[Info, PID 3456, TID 3040, 22:59:27.374] Found restrictions: Microsoft.TeamFoundation.VersionControl.Server.CheckinEventFilter 
[Info, PID 3456, TID 3040, 22:59:27.390] Activating a new thread to send events. 
[Info, PID 3456, TID 3060, 22:59:27.390] Entering Send loop 
[Info, PID 3456, TID 3060, 22:59:27.640] Done with notification. 
[Info, PID 3456, TID 3060, 22:59:27.656] Exiting Send loop

[Info, PID 3456, TID 3040, 22:55:39.870] Calling notification filter: Microsoft.TeamFoundation.VersionControl.Server.CheckinEventFilter 
[Info, PID 3456, TID 3040, 22:55:41.995] Found restrictions: Microsoft.TeamFoundation.VersionControl.Server.CheckinEventFilter 
[Info, PID 3456, TID 3040, 22:55:42.026] Activating a new thread to send events. 
[Info, PID 3456, TID 304, 22:55:42.026] Entering Send loop 
[Info, PID 3456, TID 304, 22:55:42.636] Exiting Send loop

    TF50282: Could not connect to the following e-mail server: smtpHost 
    Error message: System.Net.Mail.SmtpException: Client does not have permission to submit mail to this server. The server response was: 5.7.3 Client does not have permission to Send As this sender.

If you are having trouble getting Tfs to send emails, one thing to test is connectivity and permission to send.  Sometimes the mail server will refuse to accept email if the sending account doesn't itself have a mailbox, or if the "from" address doesn't match the account that is connecting to the server. Log in as the service account, create a console application, copy and paste the following code, replace the first three strings with the appropriate values, compile and run it.  If it is a problem connecting to the mail server, you should get a good error message printed out.  Otherwise you should see the email appear in your inbox.

using System;
using System.Net;
using System.Net.Mail;

namespace MailSender
{

class Sender
{
    public static void Main(string[] args)
    {
        try
        {
            string Host = "myEmailServer";
            string FromAddress = "serviceAccount@somewhere.com";
            string ToAddress = "me@somewhere.com";
           
            SmtpClient client = new SmtpClient(Host);
            MailMessage mm = new MailMessage(FromAddress, ToAddress);
            mm.Body="TestBody";
            mm.Subject="TestSubject";
           
            client.UseDefaultCredentials = true;           
            mm.BodyEncoding = System.Text.Encoding.UTF8;
           
            client.Send(mm);
        }
        catch (Exception e)
        {
            Console.WriteLine(e.ToString());
        }
        Console.ReadLine();
    }
   
}

}

I’ve been meaning to write more about Team Foundation Server (TFS) events and notifications for some time now.  These are the hooks for being notified of, for example, checkins or workitem (bug) changes.  Now that our beta3 is out, a lot more people are testing the features.  Many customers want to enable email notification.  Some partners are looking at using them to integrate other tools.  So I’d like to write more about them in some upcoming posts.   First I’ll focus on email notifications, since that’s a mainstream feature.  Later I’ll write more on using events to integrate with other software, an advanced topic that will be of interest to a more limited audience.  Underlying both is the same event processing engine, so naturally there is considerable overlap.  Comments and questions are welcome.

The documentation for Visual Studio beta 2 gives the following example for receiving TFS events (work item change, checkin, build completed, etc.).

[SoapDocumentMethod(Action = "http://Microsoft.VisualStudio.Bis/Notify")]
[WebMethod]
public void Notify(string bisEvent)
{
  ...
}

[SoapDocumentMethod(Action = "http://Microsoft.VisualStudio.Bis/Notify", 
RequestNamespace = "http://Microsoft.VisualStudio.Bis")]
[WebMethod]
public void Notify(string eventXml)
{ 
  ...
} 

Note that the parameter name must be "eventXml" exactly, since webmethods depend on reflection to match SOAP arguments with methods. Post beta 2, it will be slightly different still:

[SoapDocumentMethod(Action = "http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Notification/02/Notify" , 
   RequestNamespace="http://schemas.microsoft.com/TeamFoundation/2005/06/Services/Notification/02"
)]
[WebMethod]
public void Notify(string eventXml) 
{
  ...
}

If you don't have the attributes or the parameter name correct, you are likely to see the strange behavior of the method being called, but the parameter being null.

As promised, here are the details on how we limit the number of threads being used to perform a background task.  The code below was written by Brian Harry. With this class, we can queue up as many requests as we want, but only maxThreads number of them will be active at any given time.  So long as maxThreads is smaller than the size of the system thread pool, threads remain available for high-priority tasks such as serving ASP.NET pages or web requests.  Given that there is only one thread pool per application in .NET, this class provides a good substitute for isolated thread pools. 

using System;
using System.Collections.Generic;
using System.Threading;

namespace ThreadPoolThrottle
{
    class ThreadPoolThrottle
    {
        // Initialize the throttler to the specified # of max threads.
        public ThreadPoolThrottle(int maxThreads)
        {
            m_maxThreads = maxThreads;
            m_callback = new WaitCallback(ProcessWorkItem);
        }

        /// <summary>
        /// The thread priority at which thread pool threads dispatched to handle work items
        /// for this throttle class will run.
        /// </summary>
        public ThreadPriority Priority
        {
            get { return m_priority; }
            set { m_priority = value; }
        }

        /// <summary>
        /// Queue a work item to the threadpool while obeying the thread limit passed in the
        /// constructor.  If this instance of the ThreadPoolThrottle already has submitted
        /// its limit of simultaneous tasks to the threadpool, wait for an existing work
        /// item to complete and submit this request when there is room.
        /// </summary>
        /// <param name="callBack">The callback to be called on the threadpool thread.</param>
        /// <param name="state">A state object to be supplied on the callback.</param>
        /// <returns>True if the request was successfully queued and false if not.</returns>
        public bool QueueUserWorkItemWithWait(WaitCallback callBack, Object state)
        {
            // Loop in the case that a race condition happens that causes us to not see space in
            // the thread pool without the lock and then find space with the lock.
            for (;;)
            {
                // Lightweight check to see if there is room in our limit to start this work item.
                if (m_currentThreads < m_maxThreads)
                {
                    // Allocate a slot.
                    if (Interlocked.Increment(ref m_currentThreads) <= m_maxThreads)
                    {
                        // If we succeeded in getting a slot under the limit, queue the item to
                        // the thread pool.
                        return ThreadPool.UnsafeQueueUserWorkItem(m_callback, new ThrottleCallbackState(callBack, state));
                    }

                    // We failed to get a slot so return it and try with a lock.
                    Interlocked.Decrement(ref m_currentThreads);
                }

                lock (m_queue)
                {
                    // Check under the lock if the thread limit is still full.
                    if (m_currentThreads >= m_maxThreads)
                    {
                        // If it is, mark us as waiting and go to sleep until we are awoken.
                        ++m_waiting;
                        Monitor.Wait(m_queue);
                        --m_waiting;

                        // Once we have been awoken, run our work item - the thread that woke us
                        // did not decrement m_currentThreads so we don't need to increment it.
                        return ThreadPool.UnsafeQueueUserWorkItem(m_callback, new ThrottleCallbackState(callBack, state));
                    }
                }
            }
        }

        /// <summary>
        /// Queue a work item to the threadpool while obeying the thread limit passed in the
        /// constructor.  If this instance of the ThreadPoolThrottle already has submitted
        /// its limit of simultaneous tasks to the threadpool, queue this work item and return.
        /// </summary>
        /// <param name="callBack">The callback to be called on the threadpool thread.</param>
        /// <param name="state">A state object to be supplied on the callback.</param>
        /// <returns>True if the request was successfully queued and false if not.</returns>
        public bool QueueUserWorkItem(WaitCallback callBack, Object state)
        {
            for (;;)
            {
                // Lightweight check to see if there is room in our limit to start this work item.
                if (m_currentThreads < m_maxThreads)
                {
                    // Allocate a slot.
                    if (Interlocked.Increment(ref m_currentThreads) <= m_maxThreads)
                    {
                        // If we succeeded in getting a slot under the limit, queue the item to
                        // the thread pool.
                        return ThreadPool.UnsafeQueueUserWorkItem(m_callback, new ThrottleCallbackState(callBack, state));
                    }

                    // We failed to get a slot so return it and try with a lock.
                    Interlocked.Decrement(ref m_currentThreads);
                }

                lock (m_queue)
                {
                    // Check under the lock if the thread limit is still full.
                    if (m_currentThreads >= m_maxThreads)
                    {
                        try
                        {
                            // Put the work item on the queue.
                            m_queue.Enqueue(new ThrottleCallbackState(callBack, state));
                            return true;
                        }
                        catch
                        {
                            return false;
                        }
                    }
                }
            }
        }

        /// <summary>
        /// An approximate count of the number of queued work items that are being processed or
        /// waiting to be processed.
        /// </summary>
        public int UncompletedWorkItems
        {
            get { return m_currentThreads + m_waiting + m_queue.Count; }
        }

        // Callback funtion from the thread pool to process a work item.  We need this so that
        // we can keep track of the number that are currently being processed and limit it to
        // the requested number.
        private void ProcessWorkItem(Object state)
        {
            ThrottleCallbackState callbackState = (ThrottleCallbackState)state;

            // Call the queued callback and pass its state.
            CouldntQueue:
            try
            {
                if (m_priority != ThreadPriority.Normal)
                {
                    Thread.CurrentThread.Priority = m_priority;
                }
                callbackState.m_callback(callbackState.m_state);
            }
            finally
            {
                if (Thread.CurrentThread.Priority != ThreadPriority.Normal)
                {
                    Thread.CurrentThread.Priority = ThreadPriority.Normal;
                }
            }

            lock (m_queue)
            {
                if (m_waiting > 0)
                {
                    // This will wake up a thread that will take our spot in the "current"
                    // threads list so we don't need to decrement m_currentThreads.
                    Monitor.Pulse(m_queue);
                }
                else if (m_queue.Count > 0)
                {
                    // Dequeue a waiting item and queue it back to the thread pool.  Since
                    // we are queuing it, we don't need to decrement m_currentThreads.
                    callbackState = m_queue.Dequeue();
                    if (!ThreadPool.UnsafeQueueUserWorkItem(m_callback, callbackState))
                    {
                        // If we weren't able to queue the next item, rather than drop it on
                        // the floor or let threads drain out when they shouldn't, run it
                        // manually.
                        goto CouldntQueue;
                    }
                }
                else
                {
                    // There are no work items waiting to process, so reduce the number of
                    // working threads.
                    Interlocked.Decrement(ref m_currentThreads);
                }
            }
        }

        int m_maxThreads;
        int m_currentThreads;
        ThreadPriority m_priority = ThreadPriority.Normal;
        WaitCallback m_callback;
        int m_waiting;
        Queue<ThrottleCallbackState> m_queue = new Queue<ThrottleCallbackState>(32);

        // In internal class to track the user's callback and state.
        class ThrottleCallbackState
        {
            public ThrottleCallbackState(WaitCallback callback, Object state)
            {
                m_callback = callback;
                m_state = state;
            }

            internal WaitCallback m_callback;
            internal Object m_state;
        }
    }
}

You’ve written your web application using all the latest features of .NET.  Because some tasks should be done in the background, you invoke them by calling BeginInvoke on a delegate, asking the thread pool to run the task asynchronously.  That should have a minute effect on the rest of the system, right?

Not necessarily.  You may see, like we have, a different kind of contention.  It’s not threads competing for CPU time.  It’s requests competing for the thread pool.  The thread pool has a fixed maximum number of threads, to minimize the amount of time the system spends context switching.  The default maximum is 25 threads per CPU.

There is one shared thread pool per process.  ASP.NET uses threads from the pool to service web requests. Requests to the thread pool all have equal priority – they are granted in a first-come first-served way.  So if you have made 25 requests for background tasks, the system won’t begin the 26^th request until one of them completes.  It can’t – the threads are all occupied.  If that 26^th request is an ASP.NET web request, too bad, it will have to wait its turn.

“So why is the CPU only at 50%” you may ask.  Those threads won’t necessarily be using the CPU.   Most enterprise applications today are I/O-bound, spending most of their time waiting for the disk or the network.  It’s not uncommon to see 25 threads all waiting for I/O at the same time. 

So be on the watch for this situation if you use the thread pool in .NET.  Look to see if the number of threads in your process has maxed out. 

Later I’ll give some details on how we solved the problem for our system.

One more excellent free tool is WinDirStat.  It shows both where the disk space is going, as well as what it is going toward, based on file type, in a clean graphical way.  In just a few clicks you can clean up gigabytes of unneeded data.  I found many .pdb files on my machine I didn't need.  Even if you have a huge hard drive, fewer files means faster defragmentation and faster virus scans.  Highly recommended.