Insert witty blog title here

How to diagnose 500 server errors

philipbeber — Tue, 06 Feb 2007 23:21:00 GMT

Step One: Turn off custom errors in your ASP.Net app. Usually this means editing web.config and settings <customErrors mod="Off" />. E.g.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <system.web>
         <customErrors mode="Off" />
    </system.web>
</configuration>

Step Two: Configure TAS to tell you more about the error.

To do this you need to edit the file TasWorker.exe.config. If it exists under %Program Files%\Microsoft Speech Server\TAS then add this line:

If the file doesn't exist then create it and make it look like this:

</appSettings>

</configuration>

Now restart TAS. Now when it tries to download a web page and gets a 500 error, it will include the body of the HTML page in a Trace statement (these go to the ETL file). Read it as it is or cut and paste it into and html file for readability.

Welcome TAPs!

philipbeber — Mon, 24 Oct 2005 20:45:00 GMT

Today we have some specially selected partners in town to try out our next release of MSS. Welcome! I hope I can be useful in enabling you to use Microsoft Speech Server 2007 (TAP Release). More likely I'll learn as much as you!

Dynamic Script Part I - Overview

philipbeber — Wed, 19 Oct 2005 00:09:00 GMT

The time has come to say some words about dynamic script and how it can kill the performance of your MSS deployment. First an overview of the general problem:

JScript.NET is at the heart of MSS 2004 applications. Some of it is generated by ASP.Net and tons of it is generated by MSS Speech Controls too. You've probably had to write loads for your prompt functions and client-side behavior. You just can't get away from it. (Fortunately you can in MSS 2006.)

MSS deals with this script in a wonderfully fast and efficient manner - it compiles each web page of script to a .Net assembly. This has the following advantages:

Security: MSS can sandbox applications from each other and limit what they can do using .Net Code Access Security.
Scalability: Compiled JScript consumes less CPU than interpreted JScript, allowing one box to support more ports.
Efficiency: As long as the script on a web page doesn't change, the assembly only needs to be compiled once and can be shared amongst applications.

Ah, but what happens when the script does change each time MSS downloads the page? I'm no ASP.Net expert but I believe you can do stuff like this:

This might be okay for Internet Explorer, which does no caching and interprets everything. This is not so good for MSS, for the simple reason that the .Net Framework does not allow you to unload an assembly once it's loaded. So if the script is subtlely different each time it is downloaded MSS will compile it to a new assembly and then be stuck with that assembly until the AppDomain or TasWorker is recycled. The first sign of trouble is usually unexpected process recycles - since these are automatically triggered by memory consumption over a certain threshold. You will also see increased latencies since you are incurring the cost of compilation for each page navigation - something which usually seems to happen when the end user is waiting for a response. Note that MSS 2004 R2 will detect dynamic script problems and write an event to the NT Event Log.

The good news is that there are ways and means to avoid this issue. The number one fact to take advantage of is that MSS does not cache the Document Object Model (i.e. everything on the page that isn't script.) Hence the above fragment can be rewritten as:

Since the script is now static different instances of the page will share the same assembly. At run time the application gets the value to display from the DOM.

[Side note: What is and what isn't script? Clearly everything within <script> blocks is script, as well as files referenced by <script src=.../>. For the purposes of this discussion, stuff inside 'id' attributes is also considered script, i.e. if you change the id of an element you are changing the assembly that the page compiles to. Event handlers are also script, e.g. <body onload="thisIsScript()"> <salt:dtmf onreco="moreScript()">]

That's probably enough of an overview. Next I'm going to talk about some general strategies for dealing with dynamic script and some specific gotchas to do with the Speech Controls.

Mysterious logging outages

philipbeber — Tue, 18 Oct 2005 21:44:00 GMT

At last! Something to write about!

There have been various reports of the logging in MSS 2004 going mysteriously silent. I got to investigate one of these issues for the first time yesterday. The symptoms were that only TimLogger was writing anything to the ETL file. (I.e. the only events showing up were CallStartedEvent, AnswerCallLatencyEvent, etc.) Footling with MssLogConfig.vbs made no difference. The logging config file TasInstrumentation.config seemed correct. (SES was running on different machines, so only TAS events were expected.) No errors or warnings were reported in the NT Event Log.

It turns out the issue was that the customer was not running TAS as NetworkService, but as a special user MSS_USER. When I tried this myself I found I got this warning in the Event Log:

MssTrace: Unexpected exception EventProvider.TraceControlCallback

exception = System.ComponentModel.Win32Exception: Access is denied

The first thing I did was give MSS_USER write access to the directory with the ETL logs in it. In theory this is all you need to do but no dice. Next I tried adding MSS_USER to the "Performance Log Users" Group. Success! The warning diappeared and TAS events started appearing in the ETL file. So what is going one here? According to MSDN you only need to be in that group to "control" ETW sessions (i.e. start and stop collection to an ETL file.) Writing events to a session requires no special privilege.

Looking at the code I found that MSS logging calls QueryTrace to get the size of the ETW buffer and as far as I can see it only does this so it can print a nice warning if you set it too low, or if someone tries to log an event which is bigger than one buffer. Since 99% of users will enable logging from the MSS Management Console, and therefore will be using a buffer size of 60k, this seems fairly pointless.

I will definitely fix this in MSS 2006. I'm glad there is an easy workaround for MSS 2004 users.

[Update 10/19/05: It turns out this is actually documented in the MSS help. Whoever discovered this the first time must have thought it was a feature of ETW rather than a bug in MSS.]

An improvement (?) on the TextWriterTraceListener class

philipbeber — Thu, 07 Jul 2005 00:51:00 GMT

I often use System.Diagnostics.Trace.WriteLine() debugging applications. In the past, if I needed to persist these statements to a log file programmatically I was using a TextWriterTraceListener object to do this. However, it turns out that TraceListeners only work for the current appdomain, which is a bit limiting.

I thought it would be cool to write a version of TextWriterTraceListener that worked more like DbgView.exe from SysInternals, i.e. that listened to Trace.WriteLine()s from all appdomains in all processes. Here is my first draft, which is basically a managed version of the DbMon sample code from MSDN. It's a console app that listens for calls to OutputDebugString() and echoes them to stdout.

Note: I've only tested this with .Net Framework 2.0 (Whidbey Beta 2). You'll probably need to tweak it significantly to get it to work with 1.0 or 1.1.

using System;
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Security.AccessControl;
using System.Security.Principal;
using System.Text;
using System.Threading;

namespace DebugMonitor
{
    unsafe static class NativeMethods
    {
        [StructLayout(LayoutKind.Sequential)]
        public struct SECURITY_ATTRIBUTES
        {
            public UInt32 nLength;
            public void* lpSecurityDescriptor;
            public UInt32 bInheritHandle;
        }

        internal const uint PAGE_READWRITE = 0x04;
        internal const uint FILE_MAP_READ = 0x0004;

        [DllImport("kernel32.dll", SetLastError = true)]
        internal static extern IntPtr CreateFileMapping(IntPtr hFile,
          SECURITY_ATTRIBUTES* lpFileMappingAttributes,
          UInt32 flProtect, UInt32 dwMaximumSizeHigh,
          uint dwMaximumSizeLow, string lpName);

        [DllImport("kernel32.dll", SetLastError = true)]
        internal static extern byte* MapViewOfFile(IntPtr hFileMappingObject, UInt32
          dwDesiredAccess, UInt32 dwFileOffsetHigh, UInt32 dwFileOffsetLow,
          UIntPtr dwNumberOfBytesToMap);

        [DllImport("kernel32.dll", SetLastError = true)]
        internal static extern bool UnmapViewOfFile(byte* lpBaseAddress);

        [DllImport("kernel32.dll", SetLastError = true)]
        internal static extern bool CloseHandle(IntPtr hHandle);
    }

    unsafe class Program
    {
        const int SharedMemorySize = 512;

        static void Main(string[] args)
        {
            EventWaitHandle ackEvent = null;
            EventWaitHandle readyEvent = null;
            IntPtr sharedFile = IntPtr.Zero;
            byte* sharedMemory = null;

            uint lastPid;
            uint* thisPid;
            byte* theString;
            bool didCR;

            bool wasCreated;

            try
            {
                EventWaitHandleSecurity ewhSec = new EventWaitHandleSecurity();
                //Note: CLR doesn't recognize S-1-0 (bug?) so I'm using S-1-1-0 instead. Seems to work.
                EventWaitHandleAccessRule rule = new EventWaitHandleAccessRule(new SecurityIdentifier("S-1-1-0"), EventWaitHandleRights.FullControl, AccessControlType.Allow);
                ewhSec.AddAccessRule(rule);

                FileSecurity fileSecurity = new FileSecurity();
                FileSystemAccessRule fileRule = new FileSystemAccessRule(new SecurityIdentifier("S-1-1-0"), FileSystemRights.FullControl, AccessControlType.Allow);
                fileSecurity.AddAccessRule(fileRule);

                byte[] fileSecurityBytes = fileSecurity.GetSecurityDescriptorBinaryForm();

                ackEvent = new EventWaitHandle(false,
                    EventResetMode.AutoReset,
                    "DBWIN_BUFFER_READY",
                    out wasCreated,
                    ewhSec);

                if (!wasCreated)
                {
                    Console.WriteLine("There is already an instance of this program or a similar monitor program (e.g. DebugView, DbMon) running.");
                    return;
                }

                readyEvent = new EventWaitHandle(false,
                    EventResetMode.AutoReset,
                    "DBWIN_DATA_READY",
                    out wasCreated,
                    ewhSec);

                fixed (byte* fileSecurityPtr = fileSecurityBytes)
                {
                    NativeMethods.SECURITY_ATTRIBUTES sa;
                    sa.bInheritHandle = 1;
                    sa.lpSecurityDescriptor = fileSecurityPtr;
                    sa.nLength = (uint)fileSecurityBytes.Length;

                    sharedFile = NativeMethods.CreateFileMapping(
                        new IntPtr(-1),
                        &sa,
                        NativeMethods.PAGE_READWRITE,
                        0,
                        4096,
                        "DBWIN_BUFFER");
                }

                if (sharedFile == IntPtr.Zero)
                {
                    Exception innerException = new Win32Exception();
                    throw new ApplicationException("CreateFileMapping failed", innerException);
                }

                sharedMemory = NativeMethods.MapViewOfFile(sharedFile, NativeMethods.FILE_MAP_READ, 0, 0, new UIntPtr(SharedMemorySize));

                if (sharedMemory == null)
                {
                    Exception innerException = new Win32Exception();
                    throw new ApplicationException("MapViewOfFile failed", innerException);
                }

                thisPid = (uint*)sharedMemory;
                theString = sharedMemory + sizeof(uint);

                lastPid = 0xffffffff;
                didCR = true;

                ackEvent.Set();

                char[] text = new char[SharedMemorySize];

                while (true)
                {
                    readyEvent.WaitOne();

                    if (lastPid != *thisPid)
                    {
                        lastPid = *thisPid;

                        if (!didCR)
                        {
                            Console.WriteLine();
                            didCR = true;
                        }
                    }

                    int length = 0;

                    fixed (char* buffer = text)
                    {
                        for (byte* p = theString; p < (sharedMemory + SharedMemorySize); ++p, ++length)
                        {
                            if (*p == 0)
                            {
                                break;
                            }
                        }
                        length = Encoding.ASCII.GetChars(theString, length, buffer, SharedMemorySize);
                    }

                    Console.Write(text, 0, length);
                    didCR = text.Length > 0 && text[text.Length - 1] == '\n';

                    ackEvent.Set();
                }
            }
            catch (Exception exn)
            {
                Console.WriteLine(exn.ToString());
            }
            finally
            {
                if (ackEvent != null)
                {
                    ackEvent.Close();
                }

                if (readyEvent != null)
                {
                    readyEvent.Close();
                }

                if (sharedMemory != null)
                {
                    NativeMethods.UnmapViewOfFile(sharedMemory);
                }

                if (sharedFile != IntPtr.Zero)
                {
                    NativeMethods.CloseHandle(sharedFile);
                }
            }
        }
    }
}

The first post

philipbeber — Thu, 16 Jun 2005 07:29:00 GMT

Welcome one and all to the official blog of Philip Beber, a Software Design Engineer for Microsoft Speech Server. On my blog I plan to post interesting things I learn about C# and stuff about version 2 of Speech Server too. Maybe even stuff about version 1, if there's any interest.

Here are some facts about me:

I take the bus to work.
I like to play soccer. (Microsoft have an especially good field for this on campus.)
I've never met Bill Gates. I thought working at Microsoft would give me this right but not so far.

You also might like to know that my primary area of work right now is Logging. So if you have any strong opinions about Logging in Speech Server you should definitely bug me. Also, I didn't have anything to do with Logging in version 1 of Speech Server, so you can harrass me all you like and I won't take it personally.

Let's hope this works out for both of us.