WER Services

The only thing constant is change – Part 1

Gr8Gatsby — Mon, 08 Aug 2011 15:52:05 GMT

Hello it’s Kevin Hill here. I want to provide some insight into where we are heading with the WER datasets for partners.

We surveyed users of WER data on https://winqual.microsoft.com to get feedback for the direction of features, the below two themes surfaced quickly as a result:

“Reduce noise” – we had a lot of feedback around filtering out low priority “events” as these end up being time-wasters.
“More automation & notifications” – Don’t make me always come to the website to get information push it to me.

We have made feature investments to help make progress in these areas:

“Reduce noise”

For the user mode datasets in the past we had shown “event” (a.k.a. Bucket) based information. We are moving this dataset to be based on Failures which are the result of the !Analyze debugging extension. Below I scratch the surface with the difference between these two datasets, but watching the Channel 9 video really helps to understand what logic goes into analyzing failures.

Buckets classify issues by: Application Name, Application Version, Application Build Date, Module Name, Module Version, Module Build Date, OS Exception Code, and Module Code Offset. On the surface this looks like a very precise way to classify issues; however it can be misleading at times since we generate this bucket on the Windows OS client without performing any symbol analysis on the memory dump. The module that is picked by the Windows Error Reporting client is the module at the top of the stack. We have found that investigating these issues as raw buckets can cause many investigations that result in a faulting module that is different from the original bucket determination. Ryan Kivett (a developer lead from the Windows Reliability team) has a great blog post titled: WER History 101

Failures, however, are the result of processing memory dumps collected from buckets in order to more precisely assess blame to the appropriate module. I have included the graphic below to show how buckets can result in multiple different failures. While this is a typically scenario once private symbol analysis has been applied, there can also be cases where multiple buckets get aggregated into a !Unknown symbol as well. If you download cabs and run !Analyze with your own private symbol paths you will break down the !Unknown failures into a very precise set of issues.

Here is another example considering multiple buckets:

Since we are performing automated analysis of issues using private symbols we are able to filter out issues where we know we will likely be able to provide a fix, in order to provide fewer issues that will result in investigations that point to issues in modules that don’t belong to you. This is one of the ways we are reducing the noise in the dataset. This doesn’t mean that there will be fewer number of issues, but the issues we share are more likely to be actionable for your software.

The below video contains a detailed explanation of the logic behind !Analyze and what goes into the failure dataset.

“More automation & notifications”

We are providing email updates for all the new failure based reports that are going to be published based on the new user mode failure based dataset and the kernel mode failure based dataset as well.

Stay tuned for more updates on planned changes, Part 2 will provide more information…

Why You Should Use XMLHttpRequest Asynchronously

Dom R — Wed, 03 Aug 2011 20:20:00 GMT

8.4% of all hangs in IE9 in the past month are caused by XMLHttpRequest objects blocking the UI thread with a synchronous request. That’s a huge number! With some manageable code changes, these hangs can be avoided, and developers can give their users a better experience across their websites. We’ll get into what’s happening here, what you can do about it, and we’ll give a little demonstration to see firsthand what can happen when a synchronous request hangs the browser.

Synchronous XMLHttpRequest Blocks the UI Thread

We already know performing a blocking operation on a UI thread is asking for trouble. We’ve blogged about this before. It might not be apparent, but this is exactly what can happen when calling the XMLHttpRequest.send() method synchronously (i.e. passing false for bAsync when calling the open method). The UI thread has to wait until it gets a response from the server, or until the request times out. While waiting, it’s not pumping messages.

Why Use Synchronous XMLHttpRequest At All?

Developers tend to use synchronous over asynchronous is because it’s easier and can many times lead to less complex code. This StackOverflow thread states some potentially valid opinions on why you’d use one over the other, but nothing compelling enough that makes it worth the risk.

Fixing It…

There are at least two ways you can write your code to avoid these hangs. The first method is preferred, because you won’t block on the UI thread. The second is less than optimal, because you’ll still block, but for less time than you’re probably doing now.

Write your code to call open asynchronously.
You need to pass true for the bAsync parameter to the open method, and you need to write an event handler for the onreadystatechange event. There are many examples of how to do this out one the web, like this one.
Set the timeout property.
Because the Desktop Window Manager detects a hang after 5 seconds of unresponsiveness, and IE9 Hang Resistance after 8, I would set the timeout property to something under 5 seconds. You may also want to consider creating a function to handle the ontimeout event.
(How did I get these numbers? Refer to here and here respectively – also already mentioned above)

How About an Example?

Let’s dig into the details by creating a couple of scenarios:

We’ll create some code that will make a synchronous request and hang the browser, and then
change the code to be asynchronous, and observe the acute change in browser behavior.

The goal of these scenarios is to demonstrate how switching from synchronous to asynchronous requests will improve the user experience of your web applications.

DISCLAIMER: This will hang your browser and may result in unintended, perhaps even undesirable behavior on both the client and the server. In other words, don’t run this on your production server, or any other machine that can’t tolerate running instable and potentially broken code.

The Setup

You’ll need a web server capable of running ASP.NET if you want to follow the example directly. I realize some readers may not be using ASP.NET, but the code should be straight-forward enough that you can adapt it to other environments without too much trouble.

Also, I will assume you are familiar with your web server environment and know how to render a page server-side.

Copy this code into your favorite text editor and save it as hangme.aspx somewhere on your web server.

 <!-- code starts after this line -->
 <%@ Page Language="C#" %> 
 <html> 
 <head> 
 <title>XmlHttpRequest open hang test</title> 
 <script runat="server">
    protected void Page_Load(object sender, EventArgs e) { 
       if(Request.QueryString["hang"] == "1") {
          int seconds = 0; 
          Int32.TryParse(Request.QueryString["seconds"], out seconds); 
          System.Threading.Thread.Sleep(seconds * 1000); 
       } 
    } 
 </script> 
 <script type="text/javascript"> 
    function call_hangme() { 
       var oReq;       if (window.XMLHttpRequest) { 
          oReq = new XMLHttpRequest(); 
       } 
       if (oReq != null) { 
          var sUrl = "http://localhost/hangme.aspx?hang=1&seconds=360"; 

          <!-- change localhost to your server name if applicable --> 
          document.getElementById("txt1").innerHTML = "Request Sent..."; 

          <!-- pass false for the bAsync parameter for a synchronous request --> 
             oReq.open("GET", sUrl, false); 

          oReq.send(); 
          document.getElementById("txt1").innerHTML = "Response Received!"; 
       } 
       else { 
          window.alert("Error creating XmlHttpRequest object."); 
       }
    }
 </script> 
 </head> 
 <body> 
    <p>Click this button to hang the browser</p> 
    <form name="form1" action="" method="get"> 
       <input type="button" name="btn1" value="hang me" onClick="call_hangme()"> 
    </form>
    <p id="txt1"/> 
 </body> 
 </html> 
 <!-- code ends on the line before this one -->

Make It Hang

Now,

Open Internet Explorer and browse to the hangme.aspx page.
Click the “hang me” button.

Uh-oh! We’re hung! Because we told the thread to sleep for 6 minutes, we’re going to be here awhile. If you’re using IE9, you’ve probably noticed the gold band at the bottom offering to “Recover webpage”. If you’re using another browser, you’ve probably got a ghosted window.

Calling Open Asynchronously

Copy this code new file and save it as wont_hangme.aspx.

 <!-- code starts after this line -->  
 <%@ Page Language="C#" %>
 <html>
 <head>
 <title>XmlHttpRequest open hang test</title>
 <script runat="server">
    protected void Page_Load(object sender, EventArgs e) {
       if(Request.QueryString["hang"] == "1") {
          int seconds = 0;
          Int32.TryParse(Request.QueryString["seconds"], out seconds);
          System.Threading.Thread.Sleep(seconds * 1000);
       }
    }
 </script>
 <script type="text/javascript"> 
    function hangme() {
       var oReq;
       if (window.XMLHttpRequest) {
       oReq = new XMLHttpRequest();
       }

       if (oReq != null) {
 
          <!-- change localhost to your server name if applicable -->
          var sUrl = "http://localhost/wont_hangme.aspx?hang=1&seconds=500"; 
          document.getElementById("txt1").innerHTML = "Request Sent...";

          <!-- pass true for the bAsync parameter -->
          oReq.open("GET", sUrl, true); 

          <!-- Here we define an anonymous function for the 
           onreadystatechange event.  We check if we received all the data, 
 >           and the request was successful before changing the text
          -->

          oReq.onreadystatechange = function() {
             if (oReq.readyState == 4 && oReq.status == 200) {
                document.getElementById("txt1").innerHTML = "Response Received!";
             }
          }

          oReq.send();
       }
       else {
          window.alert("Error creating XmlHttpRequest object.");
       }
    }
 </script>
 </head>
 <body>
    <p>Click this button to hang the browser</p>
    <form name="form1" action="" method="get">
       <input type="button" name="btn1" value="hang me" onClick="hangme()">
    </form>
    <p id="txt1"/>
 </body>
 </html>
 <!-- code ends on the line before this one -->

The new code is highlighted. Here, we’re now calling open with bAsync = true, which means we’re making an asynchronous call. This will free up the UI thread to do other things, instead of it being blocked waiting for the response or timeout. If we’re not blocked, we can pump messages, and if we can do that, we’re not hung!

Now, when the response eventually comes back from the server, we’ll need a function to handle it. That is what the highlighted code does. It’s our event handler for onreadystatechange event. As you can see, I’m only checking that there response was successful (status == 200) and that all the data has been received (readyState == 4).

Can We Make It Hang Now?

Again,

Open Internet Explorer and browse to the cant_hangme.aspx page.
Click the “hang me” button.

Voilà! You’ll see the request is sent, but the browser is not hung. When the response finally makes its way back, the text will change; all the while you’re still able to interact with the page.

In Closing

As stated from the outset, you want to implement your XMLHttpRequest calls asynchronously. This example clearly demonstrates why; you’ll improve the responsiveness of your pages, avoid hangs, and most importantly, give your users a better experience.

Using the Winqual Software Web Services with Live ID authentication

HardikPanchal — Wed, 09 Feb 2011 23:38:00 GMT

In our early February release we switched from custom authentication to Live ID authentication. We had posted sample power-shell scripts to demonstrate how to download cabs, get event hit data etc.

I want to share simple C# sample code snippet that demonstrates how to interact with the Windows Error Reporting web services at the most basic level. This example is provided as-is with no warranties: This script shows how to download a cab for a bucket/event.

This code snippet is divided into two parts: The first part is how to do authentication, and the second part is a helper method to download cabs for the event.

NOTE : You need to install sample application from http://wer.codeplex.com/ . Live ID authentication DLL Microsoft.Whos.Shared.Client.LiveID will be installed to your GAC for authentication once you install sample application. You need to add DLL mentioned to compile your code.

Code snippet to authenticate user,

 1: string userName = @"your user name";

 2: string password = @"your user password";

 3: string eventId = @"Event Id you want";

 4: string eventType = @"Type of the Event you want";

5:

 6: // Add dll reference to Microsoft.Whos.Shared.Client.LiveID for below

 7: LiveIdAuthentication liveAuth = new LiveIdAuthentication(

 8:     "",

 9:     new Guid(),

 10:     "Sample Code",

 11:     "winqual.microsoft.com",

 12:     "MBI_SSL",

 13:     true);

14:

 15: System.Security.SecureString ss = new System.Security.SecureString();

 16: foreach (char c in password)

 17:     ss.AppendChar(c);

18:

 19: //Authenticate user

 20: if (!liveAuth.Authenticate(userName, ss))

 21:     throw new Exception("User or Password invalid");

22:

 23: //Check Cab downloaded Cabs for the events

 24: if (DownloadCabs(eventId, eventType, liveAuth))

 25:     Console.WriteLine("Cab downloaded successfully");

 26: else

 27:     Console.WriteLine("Cab downloaded failed");

Code snippet of Helper method to download cab for the event,

 1: public static bool DownloadCabs(string eventId

 2:     , string eventtype

 3:     , LiveIdAuthentication liveAuth)

 4: {

 5:     //User specific Variables

 6:     string cabPath = "D:\\Temp\\"; https";winqual.microsoft.com";%%PROTOCOL%%://%%HOSTNAME%%/services/wer/user/cabs.aspx?eventid=%%EVENTID%%&eventtypename=%%EVENTTYPE%%";

7:

 8:     //Other Variables

 9:     string protocol = "

 10:     string website = "

 11:     WebResponse response = null;

 12:     CookieContainer defaultCookieContainer = new CookieContainer();

 13:     string feeddata = null;

 14:     string challenge = null;

 15:     cabServiceUrl = @"

16:

 17:     if (String.IsNullOrEmpty(eventId)

 18:         || String.IsNullOrEmpty(eventtype)

 19:         || null == liveAuth)

 20:     {

 21:         throw new Exception("Invalid argument passed to DownloadCabs");

 22:     }

23:

 24:     //Build URL

 25:     string cabDownloadURL = (new StringBuilder(cabServiceUrl)

 26:                                         .Replace(@"%%PROTOCOL%%", protocol)

 27:                                         .Replace(@"%%HOSTNAME%%", website)

 28:                                         .Replace(@"%%EVENTID%%", eventId)

 29:                                         .Replace(@"%%EVENTTYPE%%", eventtype)).ToString();

30:

 31:     // generate challenge on first call

 32:     challenge = LiveIdAuthentication.GetChallengeFromServiceRequest(cabDownloadURL);

33:

 34:     //Create new web request to get a list of cab for eventid

 35:     HttpWebRequest request = (HttpWebRequest)WebRequest.Create(cabDownloadURL);

36:

 37:     request.CookieContainer = defaultCookieContainer;

38:

 39:     // generate response from cached challenge and add as authorization header

 40:     //- you MUST do this for each service call

 41:     LiveIdAuthentication.AddAuthorizationHeaderToRequest(request,

 42:         liveAuth.GetResponseForHttpChallenge(challenge));

43:

 44:     response = request.GetResponse();

45:

 46:     using (StreamReader responseReader = new StreamReader(response.GetResponseStream()))

 47:     {

 48:         feeddata = responseReader.ReadToEnd();

 49:     }

50:

 51:     //Download cabs

 52:     System.Net.WebClient client = new System.Net.WebClient();

 53:     client.Encoding = System.Text.Encoding.UTF8;

54:

 55:     LiveIdAuthentication.AddAuthorizationHeaderToRequestHeaders(client.Headers

 56:         , liveAuth.GetResponseForHttpChallenge(challenge));

57:

 58:     XNamespace ns = XNamespace.Get("http://www.w3.org/2005/Atom");

 59:     XElement xmlFeed = XElement.Parse(feeddata);

60:

 61:     foreach (var feed in xmlFeed.Elements(ns + "entry").Elements(ns + "link"))

 62:     {

 63:         client.DownloadFile(feed.Attribute("href").Value,

 64:             cabPath + eventId + "-" +

 65:             Regex.Match(feed.Attribute("href").Value, @"cabID=\d+") + ".cab");

 66:     }

67:

 68:     return true;

 69: }

A checklist to help you create clear and accurate responses

kentwalter — Tue, 09 Nov 2010 17:49:00 GMT

We’ve had some questions and comments from our partners recently about the process of creating and publishing responses for Windows Error Reporting, so I want to give you some information that will help you create quality responses that get published quickly and really help your customers.

I work on the team that approves and publishes responses, and if you’ve created more than a few responses, there’s a good chance that we’ve worked with you to clarify your response content at some point. Since clarifications add time to the approval and publishing process, I wanted to give you a checklist of things to look at before you submit a response, so that your responses will sail through approval and publishing.

· Is the landing page straightforward and uncomplicated? Can you link to a better page, or better yet, a direct download? (The number one reason we clarify responses is that it links to a top-level page, like www.microsoft.com.)

· Does the experience described in the template you used match the one that users will actually see? For example, if the template says a direct download is available, the linked URL should point to that download, not to a landing page that requires further action from the customer. Note that the templates you see on Winqual are only a subset of what is available.

· Does a landing page link provide what is promised? For example, if the template selected promises an automated update tool, but the landing page for the response describes a manual update process that doesn’t use a tool, the message could be clearer.

· Is the solution within easy reach of the user? The guideline we try to hit with every response is three clicks, including the main URL in the response itself.

· Is your download as directly and easily usable as it could possibly be? Users have an easier time with an MSI than they do with a self-extractor or .zip file, for example, and the smaller the download, the better.

· Does your response require More Info links? These should not be used unless the response does not work without them. There are a couple of reasons for this:

Any More Info link text will need to be localized, which means that the entire response be available only in English for 4-6 weeks
We found through our research that most customers just don’t see More Info links. Customers look for links in the body of the response.

These points cover the majority of response clarifications that we see. Of course, you might need input or have questions before you even create a response. We’d be glad to help you out. If you have any questions or any specific needs, let us know. Send email to WERCMS@microsoft.com, and we’ll give you a hand. Thanks for using Windows Error Reporting and letting us help you.

Kent Walter
Windows Error Reporting

XPROC Application Hang Cabs in Windows 7

Dom R — Mon, 16 Aug 2010 23:18:00 GMT

In Windows 7, we've added some techniques and tools that assist with troubleshooting hangs involving multiple processes (if you're interested in the evolutionary history, check out Ryan's "Let There Be Hangs" series, where he describes how hang reporting has changed from Windows XP on.) With these improvements come some new concepts, terms, and techniques, which I will attempt to convey so that, after reading this, you can fire up the debugger and understand a bit more about what’s going on.

Defined Terms and Concepts

XPROC cab

The “X” stand for cross and the “PROC” stands for process. So in other words, we’re talking about a cross process dump. And what is that exactly? Well, when an app is hung due to cross-process communication, we capture a dump of the hung process and, when possible, the process determined to be at the end of the “wait chain” in the inter-process communication. We include both dumps, as well as other analysis related information, in the cab submitted with the error report. XPROC dumps were introduced in Vista, but there were cases where the state of the hang would be lost before the heap dump was captured.

Process Reflection

At its core, Process Reflection is used to make a clone of an existing process. This clone, also referred to as the reflected process, contains a single thread and a copy of the original process’ address space. Mark Russinovich talks more about it here (31:00)

In the case of hang reporting, when a chain of processes is involved in the hang, we use Process Reflection to create a clone of the process determined to be at the end of the wait chain by virtue of the GetThreadWaitChain API. A heap dump is taken of the clone instead of the original. Cloning takes way less time than generating a heap dump file, so using this technique, we increase our chances of capturing the hung state, than if we were just create a dump of the original process. Also, it’s worth pointing out that our single thread in the clone does not execute during hang reporting, so it is simply parked and left idle. Here’s what its stack looks like today:

ChildEBP RetAddr
0028fdbc 77a97b8c ntdll!KiFastSystemCallRet
0028fdc0 77af02ef ntdll!ZwWaitForSingleObject+0xc
0028fe38 75ef0239 ntdll!RtlpProcessReflectionStartup+0x1bf
0028fe44 77ab1ec2 kernel32!BaseThreadInitThunk+0xe
0028fe84 77ab1e95 ntdll!__RtlUserThreadStart+0x70
0028fe9c 00000000 ntdll!_RtlUserThreadStart+0x1b

When it comes to debugging analysis, some things aren’t readily apparent with the reflected process. For instance, what about the original threads? Does the handle data conflict with that of the original process? After all, the reflected process is a process in its own right. The short answer is the debugger handles all of this for you. As they say, “the devil is in the details” so we’ll get into how the debugger does this shortly. But first, it’s important to understand the files contained in an AppHangXProcB1 bucket’s cab.

On to the Fun Stuff...

AppHangXProcB1 Cab Contents

So to recap:

A hang has occurred. We get a heap dump of the hung process. Using GetThreadWaitChain(), we discover the process at the end of the wait chain and clone it using ProcessRelection.
Next we grab a heap dump of the clone. But, we also have to deal with the fact that the thread and handle data from the original aren’t copied to the clone. So, in addition to this heap dump, we also capture a mini dump of the original process that has the thread and handle data. The debugger knows how to “stitch” both files together into a single, coherent target.

Let’s take a look at an example cab:

File name                      File size     Date      Time   Attrs
----------------------------- ---------- ---------- -------- -----
   iexplore.exe.xml                  5496 2010/07/14 16:25:22 ----
   iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini      224 2010/07/14 16:25:22 ----
   WER57FB.tmp.ref                1596362 2010/07/14 16:25:22 ----
   iexplore.exe.6e79.wxhu.dmp    150380775 2010/07/14 16:25:22 ----
   memory.hdmp                   40343241 2010/07/14 16:26:18 ----
   WERInternalMetadata.xml           3226 2010/07/14 16:26:30 ----
   AppCompat.txt                    47354 2010/07/14 16:26:30 ----

Look at the contents of the dbgcfg.ini file and you'll see something like this:

[UserModeDump]
default=WER57FB.tmp.ref
memory=iexplore.exe.6e79.wxhu.dmp
module=iexplore.exe.6e79.wxhu.dmp

You may be able to discern what’s going on here. The *.tmp.ref file is the default target, which is the original process mini dump that I described earlier; the one with the thread and handle data. Using the debugger, you can view the flags passed to MiniDumpWriteDump used to create this file and thus the data contained within:

0:004> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 11
Flags           1104
                0004 MiniDumpWithHandleData
                0100 MiniDumpWithProcessThreadData
                1000 MiniDumpWithThreadInfo

The *.wxhu.dmp file is the reflected process dump. Its naming convention hasn’t changed from Vista. So, those of you that have been working with XPROC dumps in Vista may have been a little confused when you analyzed this file from a Windows 7 machine, only to see one thread waiting in RtlpProcessReflectionStartup. Open the file directly in the debugger and you’ll see that it has full data, thread and memory info, etc…

0:000> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 14
Flags           51B25
                0001 MiniDumpWithDataSegs
                0004 MiniDumpWithHandleData
                0020 MiniDumpWithUnloadedModules
                0100 MiniDumpWithProcessThreadData
                0200 MiniDumpWithPrivateReadWriteMemory
                0800 MiniDumpWithFullMemoryInfo
                1000 MiniDumpWithThreadInfo
                10000 MiniDumpWithPrivateWriteCopyMemory
                40000 MiniDumpWithTokenInformation

You’ll also probably notice that this dump also has handle data. But again, hang reporting does not duplicate the handle data from the original to the reflected process, so the only handle you should see is a process handle from the original process:

Stream 10: type HandleDataStream (12), size 00000038, RVA 08F35E0F
1 descriptors, header size is 16, descriptor size is 40
Handle(0000000000000008,"Process","")

Viewing the "Stitched" Target

There are at least two ways you can view the stitched target (using the same cab as an example):

Extract the files from the cab and open the *.dbgcfg.ini file instead of the individual dump files

C:\debuggers\cdb -z C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini

Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
Loading User-mode Dump Configuration File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini,C:,\temp\]
        default dump: [WER57FB.tmp.ref]
        memory dump: [iexplore.exe.6e79.wxhu.dmp]
        module list dump: [iexplore.exe.6e79.wxhu.dmp]

Loading Dump File [C:\temp\xproc\WER57FB.tmp.ref]
User Mini Dump File: Only registers, stack and portions of memory are available

Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Open the cab file directly in the debugger. Then open the dbgcfg.ini file from there

C:\debuggers\cdb –z C:\temp\xproc\sample.cab

Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Extracted C:\temp\xproc\000011a00_memory.hdmp from C:\temp\xproc\sample.cab

Loading Dump File [C:\temp\xproc\000011a00_memory.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

0:016> .opendump /c "iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini";g
Extracted C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini from C:\temp\xproc\sample.cab

Loading Dump File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]

Loading User-mode Dump Configuration File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
        default dump: [WER57FB.tmp.ref]
        memory dump: [iexplore.exe.6e79.wxhu.dmp]
        module list dump: [iexplore.exe.6e79.wxhu.dmp]

Loading Dump File [C:\temp\xproc\000011a00_WER57FB.tmp.ref]

Loading Dump File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp]
Just run !analyze. It uses the same procedure outlined in option 2, and it leaves the targets opened once it’s finished.

Last But Not Least!

Make sure you have the latest debugger or else it may not recognize this type of target. You can download it from here.

StackHash Beta – A client application that uses Winqual web services

Gr8Gatsby — Wed, 02 Jun 2010 03:17:39 GMT

You might remember a project that the Windows Error Reporting team published on CodePlex. This project contains a wrapper for our ATOM base web services that allow for automatically downloading Windows Error Reporting data.

There is a new project going on at www.stackhash.com where they have released a Beta 3 build of their application that builds on the Windows Error Reporting web services to locally cache and analyze crash dumps.

You can read about and sign up for their beta here: http://www.stackhash.com/blog/post/StackHash-Beta-1-Preview.aspx.

Scripting in this tool is really useful. You can create a script to run a sequence of WinDbg commands and then StackHash will automatically run the script each time a new cab is downloaded. Included scripts pull common information out of minidumps including process uptime, architecture and CLR version (if applicable).

You can also search and filter events. Filters can be saved, so for instance you could show events with more than ten hits and at least one cab for a specific product.

The beta looks like a promising tool.

Check it out and provide the StackHash team with some feedback.

MDOP error reporting for small, midsize, and large companies

Gr8Gatsby — Sun, 30 May 2010 03:45:34 GMT

It is time to do a blog post that points to some resources about the Microsoft Desktop Optimization Pack (MDOP), since we get a lot of questions about how Error Reporting can work better for small, mid-size, and large corporations. This is part of the Microsoft Software Assurance Program and provide many of the benefits of Windows Error Reporting that IT Professionals can customize. Part of the customizations enable IT Professionals to better analyze and respond to common problems within their organization.

A SpringBoard Series is available here: http://technet.microsoft.com/en-us/windows/bb899442.aspx

A great starting place to learn about all MDOP has to offer is here: http://technet.microsoft.com/en-us/windows/ff383366.aspx

For the Error Reporting Component (DEM) here are two links to videos that are worth watching:

The Official MDOP Blog

New response interface

Gr8Gatsby — Fri, 19 Feb 2010 00:04:35 GMT

Today we are deploying an update the the response tools available on https://winqual.microsoft.com

1. The life of a response – Although the lifecycle of a response can vary depending on the scenarios below the general goal of a response is to lead to a solution for the problem that was identified by Windows Error Reporting (WER).

2. Selecting the appropriate response template – we provide templates that provide guidance for the most common scenarios when communicating with customers who have experienced a crash or hang on Windows. Below you will find a description of common scenarios and the corresponding template(s) that support these scenarios.

We have a template based system for choosing messages to show to end-users. The most common templates are shown and support the most common scenarios. If you have another scenario please comment on this blog post and we will review additional scenarios as they arise.

Common Response Scenarios and templates:

Scenario	Template
You no longer support a product – If there is a product no longer supported by your company these templates best communicate this to your customers.	Go to website for support info or uninstall [240] Beta software expired [37]
You have more information for your customer or need more information from a customer – Company has additional information for customers like KB articles for advanced users, or survey’s asking for feedback.	Link to KB [6582] Send info / feedback - link to co. survey page [230]
You have a specific fix – If you know that an issues is fixed in a future version or specific update these templates best communicate the fix to your customers.	Download the solution [217] Use included automatic update tool, link to instr. [92] Install Latest Version - link to co. [74]

Windows Error Reporting Terms of Use Update

Gr8Gatsby — Thu, 14 Jan 2010 21:51:23 GMT

I wanted to give an update on things we are doing to make things a little easier for ISV’s, after receiving a lot of feedback and looking closely at our Terms of Use (TOU) agreement for Window Error Reporting we are making some updates. Once to document is live on https://winqual.microsoft.com you can review the differences but I have provided a high-level description below:

We are updating the Windows Error Reporting Terms of Use (TOU) to be less restrictive on some data retention guidelines for non Personally Identifiable Information (PII) data, and enabling a scenario where ISV’s can submit public symbols in order to get more pinpointed problem identification through windows error reporting.

The new TOU is scheduled to be available for signing on Feb 15^th 2010 through April 15^th 2010. The effective start date of this new agreement will be March 15^th 2010. There is a one month overlap to allow time for companies to review the changes.

Happy Debugging!

Advanced .NET Debugging

Gr8Gatsby — Tue, 05 Jan 2010 21:56:08 GMT

If you have ever wondered where a good place to start with .NET debugging this book is it! Where as the previous Advanced Windows Debugging book had a single chapter on .NET debugging, this is an entire book dedicated to .NET debugging!

Advanced .NET Debugging is authored by Mario Hewardt, who also has authored: Advanced Windows Debugging.

The Advanced .NET Debugging book is focused on Managed code debugging for Windows. Chapter 8: Postmortem Debugging talks about how to leverage Windows Error Reporting (WER) (pages 409-438).

This book provides very good details on how to sign-up for an account to access WER data, as well as a good walk through of the existing Website and Web service available at https://winqual.microsoft.com.

Here is what some other people are saying about the book:

http://blogs.msdn.com/debuggingtoolbox/archive/2009/05/16/advanced-net-debugging-book-preview.aspx

http://dotnetdebug.net/2009/11/30/advanced-net-debugging-the-book-by-mario-hewardt/

http://www.informit.com/store/product.aspx?isbn=0321578899

http://blogs.msdn.com/tess/archive/2009/12/15/advanced-net-debugging-book.aspx

Bucketing parameters but no dump. What is the symbol?

david.grant — Wed, 09 Dec 2009 17:40:00 GMT

There may be times when you have a set of bucketing parameters possibly from an end users event log, the Windows Error Reporting Portal, an online forum entry, or other reasons, but you don't have a cab or dump of this exception. All is not lost, we can use the debugger, and a local copy of the binary and pdb (symbol) to get the function name.

These bucketing parameters will include the module name where the crash occurred and the offset, for example:

Problem signature

Problem Event Name:      APPCRASH
Application Name:           MyApp.exe
Application Version:         6.0.7600.0
Application Timestamp:   4507980b
Fault Module Name:        MyMod.DLL
Fault Module Version:      6.0.7600.0
Fault Module Timestamp:                45079fe0
Exception Code:                c0000005
Exception Offset:              00011710
OS Version:        6.0.7600.2.0.0.0.1
Locale ID:           1033

Using the debugging tools for Windows which are available from:

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

You can lookup the offset by loading the binary referenced in “Fault Module Name” into the debugger, checking the base address, and calculating the function based on the base address and the offset.

From a command prompt, open the image with the debugger.

C:\Debuggers>windbg -z MyMod.dll

This assumes that either your symbol path is set properly, or that the pdb file (symbol) is in the same directory as the image.

Then from in the debugger using the debugger command prompt:

Get the base address of the module:

0:000> lmvm MyMod
start end module name
74100000 74200000 MyMod

<output truncated>

Add the base address and the offset from the bucketing parameters:

0:000> ln 74100000+00011710
(7411171c) MyMod!CSample::Foo

From there you have the function, and you can continue to investigate as necessary, for example:

0:000> u MyMod!CSample::Foo L1
MyMod!CSample::Foo
74111710 8b4008 mov eax,dword ptr [eax+8]

Microsoft PDC 2009

Gr8Gatsby — Thu, 19 Nov 2009 18:21:49 GMT

Technorati Tags: pdc09,PDC,Microsoft,Microsoft PDC,PDC 2009,Windows Error Reporting,Watson,WER,Developer,Quality,Software,Development,Windows 7,VS 2010,.NET 4.0

I had a chance to present a lunch session at Microsoft PDC 2009 this year!

I wanted to call out some special thanks to Rajeev Acharya for helping to make this presentation and demo possible. Rajeev is an SDET on our team that developed this demo application in his “extra” time.

http://microsoftpdc.com/Sessions/CL33 – This talk starts with creating a better customer experience around software failures using the Application Restart and Recovery (ARR) API, to downloading and debugging mini-dumps send in from customers, to creating customer responses pointing to updates and fixes.

Steven Sinofsky highlighted the importance of Windows Error Reporting and how Microsoft uses this technology to build Windows in his keynote at PDC 2009

Some of his talk was summarized in this CNET article:

“Sinofsky is talking about the different mechanisms Microsoft uses from Windows Error Reporting, or Watson, to its Software Quality Monitor. Sinofsky notes that the monitoring tools require the user's permission in the final versions of Windows.”

http://news.cnet.com/8301-13860_3-10400476-56.html?part=rss&subj=news&tag=2547-1_3-0-5

Here is Sinofsky’s keynote: http://microsoftpdc.com/Sessions/KEY02 (00:15:32, starts with Telemetry)

Other blog entries:

http://blogs.pcmag.com/miller/2009/11/pdc09_sinofsky_talks_about_cre.php

http://blogs.msdn.com/e7/archive/2009/05/11/OurNextEngineeringMilestone2.aspx

!Analyze – Automatic Root Cause Analysis

Gr8Gatsby — Tue, 29 Sep 2009 20:19:19 GMT

Meet the two engineers behind the !Analyze windows debugger extension!

http://channel9.msdn.com/posts/Charles/David-Grant-and-Ryan-Kivett-Analyze-Automatic-Root-Cause-Analysis/

!Analyze is an automatic root cause analysis tool for software failures. For years, it has provided insight to engineers both inside and outside of Microsoft. It is a key enabling technology behind numerous higher-level feedback systems, including Windows Error Reporting and Watson.
!Analyze runs millions of times each day, producing actionable results from reliability telemetry data sent to Microsoft. Ordinary debugging tools report the file and function where a failure ended. !Analyze pinpoints where the failure started.
How does it work, exactly? What's the story behind !Analyze?
Meet two of the Software Developers behind !Analyze, David Grant and Ryan Kivett. They share with us how !Analyze works, it's history and provide a glimpse into it's potential future.Tune in.

Using the web services to delete product mappings

sunilab — Mon, 24 Aug 2009 22:18:00 GMT

In our June release we added web services for deleting mapped product(s), deleting mapped files from a product and deleting a mapped file from multiple products.

The following code sample shows how to delete mapped products.

IMPORTANT: Please use and/or test this code very carefully as this delete cannot be undone.

The first part of the code is used to login to the web service and get the encrypted token. The second part of the code is for deleting mapped products. The web service for deleting mapped products should be called with a post parameter named "mappedproductid" with a comma separated list of mapped product id's.

Get the mapped product id's from the web service for getting the list of products.

   1:  string baseUrl = "https://winqual.microsoft.com";

   2:  string userName = "your winqual username";

   3:  string password = "your winqual password";

4:

   5:  //

   6:  // login

   7:  //

   8:  string loginUrl = string.Format(

   9:      "{0}/services/Authentication/Authentication.svc/BasicTicket"

  10:      , baseUrl);

  11:  string request = string.Format(

  12:      "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><GetBasicTicket xmlns=\"https://winqual.microsoft.com/Services/Authentication/\"><userName>{0}</userName><password>{1}</password></GetBasicTicket></soap:Body></soap:Envelope>"

  13:      , userName

  14:      , password);

  15:  WebClient loginClient = new WebClient();

  16:  loginClient.Headers.Add(HttpRequestHeader.ContentType, "text/xml");

  17:  loginClient.Headers.Add(

  18:      "SOAPAction",

  19:      "https://winqual.microsoft.com/Services/Authentication/IBasicTicket/GetBasicTicket");

  20:  string loginResponse = loginClient.UploadString(loginUrl, request);

21:

  22:  //

  23:  // TODO: Handle condition where the ticket is null in case of bad pwd, bad username or some other error

  24:  //

25:

  26:  //

  27:  // parse the response to get the encrypted ticket out

  28:  //

  29:  int startingIndex = loginResponse.IndexOf("<GetBasicTicketResult>") + "<GetBasicTicketResult>".Length;

  30:  int endingIndex = loginResponse.IndexOf("</GetBasicTicketResult>");

  31:  string encryptedTicket = loginResponse.Substring(startingIndex, (endingIndex - startingIndex));

32:

  33:  //

  34:  // comma separated list of mapped product id’s (integers) to delete.

  35:  // IMPORTANT: This delete cannot be undone.

  36:  //

37:

  38:  string mappedProductIDs = "list of comma separated mapped product id's";

  39:  NameValueCollection nameValueCollection = new NameValueCollection(1);

  40:  nameValueCollection.Add("mappedproductid", mappedProductIDs);

  41:  WebClient webClient = new WebClient();

  42:  webClient.Encoding = Encoding.UTF8;

  43:  webClient.Headers.Add("encryptedTicket", encryptedTicket);

  44:  webClient.UploadValues(baseUrl + "/services/wer/user/deletemappedproducts.aspx", nameValueCollection);

This code will delete the mapped products and these products will look grayed out in the mapped product list at https://winqual.microsoft.com/member/wer/user/ManageProductMappings.aspx.

The code for deleting mapped files for a product and deleting mapped file from multiple products is similar to the code above.

The relative URL for deleting multiple mapped files from a single product is “/member/wer/user/DeleteMappedFilesForProduct.aspx”. This form requires two POST parameters. One is the “mappedproductid” parameter with the id of the mapped product and the second is “mappedfileid” which is the comma seperated list of mapped file id’s to delete.

The relative URL for deleting a single mapped file from multiple products is “/member/wer/user/DeleteMappedFileFromProducts.aspx”. This form requires two POST parameters. One is the “mappedfileid” parameter with the id of the mapped file and the second is “mappedproductid” which is the comma seperated list of mapped product id’s to delete the mapped file id.

The Three Cs of Response Satisfaction

MSDNArchive — Wed, 12 Aug 2009 21:00:00 GMT

The Windows Error Reporting (WER) platform offers software and hardware companies a way to provide helpful information to customers. When an application stops working the WER client application that runs on Windows (since XP) detects these events and checks to see if a solution exists. Software and hardware companies are able to create WER responses through the Windows Quality web portal (https://winqual.microsoft.com)

Windows end-users also have the ability to provide feedback on the quality of the solutions. Through analyzing the feedback provided over the years and performing extensive usability studies we have found some simple rules that help to create a pleasant experience for end-users. The 3 C’s are comprised of keeping Customer focus, providing Clear information, and keeping your information Current.

Customer focus – Keep the customer’s experience paramount in creating a response for your customers. Understanding that the customer sees messages first on the Windows operating system is important in creating a seamless experience.

Do:

· Create custom content that guides the customer

· Acknowledge that the customer was referred by Windows Error Reporting

· Use clear language and common terms

Don’t:

· Put the customer on your main homepage

· Send the customer to a general support forum

Clear information – Ensure that information provides clear and concise steps for the end-user.

Do:

· Provide a direct download link to an update from the response message when possible

· Provide a customized and dedicated landing page on your company’s website that contains an easily identifiable link to download a fix

· Provide concise steps if user action is needed during an install

· Provide clear instructions if a work around is available

Don’t:

· Provide misleading links

· Make the user click on three or more options before they reach an actionable solution

· Provide non-actionable information

Current information – Keep your responses and landing pages up-to-date! We regularly review response quality and make decisions to un-publish responses based on current information and customer ratings.

Do:

· Check links on the landing pages after website updates

· Monitor the Response Satisfaction report available on the Developer Portal

Don’t:

· Let content sit for months unchanged

If you have questions about creating a good response, you are welcome to use the contact information available on the WER portal. We are always happy to help.

Using the Product Mapping File Upload Web Service

sunilab — Fri, 12 Jun 2009 23:08:00 GMT

Yesterday we released updates to our web services offerings. These updates include the ability to automate the upload of the product mapping XML, getting a list of products mapped, getting the list of mapped files for a product, deleting a mapped file from a product, deleting mapped products and deleting a mapped file from multiple products.

This blog post lists the code (in C#) required to access our web services for uploading a product mapping file.

The code does 2 things:

Login to the Winqual portal, since the authentication web service uses SOAP the code for accessing the authentication web service without a proxy is a bit cumbersome.
Use the .NET WebClient class (you can also use the HttpWebRequest and HttpWebResponse classes for this) to upload the product mapping file and get the response.

Code for using the product mapping file upload service (also attached as a CS file):

   1:  string filePath = "path to the product mapping XML file e.g. c:\upload\CoolApplication.xml";

   2:  string baseUrl = "https://winqual.microsoft.com";

   3:  string userName = "your winqual username";

   4:  string password = "your winqual password";

   5:  //

   6:  // login

   7:  //

   8:  string loginUrl = string.Format(

   9:      "{0}/services/Authentication/Authentication.svc/BasicTicket"

  10:      , baseUrl);

  11:  string request = string.Format(

  12:      "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope

xmlns:soap=\http://schemas.xmlsoap.org/soap/envelope/\

xmlns:xsi=\http://www.w3.org/2001/XMLSchema-instance\

xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body>

<GetBasicTicket

xmlns=\"https://winqual.microsoft.com/Services/Authentication/\">

<userName>{0}</userName><password>{1}</password>

</GetBasicTicket></soap:Body></soap:Envelope>"

  13:      , userName

  14:      , password);

  15:  WebClient loginClient = new WebClient();

  16:  loginClient.Headers.Add(HttpRequestHeader.ContentType, "text/xml");

  17:  loginClient.Headers.Add(

  18:      "SOAPAction",

  19:      "https://winqual.microsoft.com/Services

/Authentication/IBasicTicket/GetBasicTicket");

  20:  string loginResponse = loginClient.UploadString(loginUrl, request);

21:

  22:  //

  23:  // parse the response to get the encrypted ticket out

  24:  // TODO: Handle condition where the ticket is null in case of bad pwd,

bad username or some other error

  25:  //

26:

  27:  int startingIndex = loginResponse.IndexOf("<GetBasicTicketResult>")

+ "<GetBasicTicketResult>".Length;

  28:  int endingIndex = loginResponse.IndexOf("</GetBasicTicketResult>");

  29:  string encryptedTicket = loginResponse.Substring(startingIndex

, (endingIndex - startingIndex));

30:

  31:  //

  32:  // upload file

  33:  //

  34:  WebClient webClient = new WebClient();

  35:  webClient.Headers.Add("encryptedTicket", encryptedTicket);

  36:  byte[] responseBytes = webClient.UploadFile(baseUrl

+ "/services/wer/user/fileupload.aspx", filePath);

37:

  38:  //

  39:  // the response that the WebClient gets may contain an updated ticket, so get that

  40:  //

  41:  if (webClient.ResponseHeaders["encryptedTicket"] != null)

  42:  {

  43:      encryptedTicket = webClient.ResponseHeaders["encryptedTicket"];

  44:  }

45:

  46:  //

  47:  // get the response

  48:  //

  49:  UTF8Encoding encoding = new UTF8Encoding();

  50:  string fileUploadResponse = encoding.GetString(responseBytes);

51:

  52:  //

  53:  // TODO: Check the response XML for success or error message

  54:  //

55:

  56:  return fileUploadResponse;

The XML response from the authentication service will be similar to the following when the login is successful:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
        <GetBasicTicketResponse

xmlns="https://winqual.microsoft.com/Services/Authentication/">
                     <GetBasicTicketResult>

encrypted ticket string</GetBasicTicketResult> 
              </GetBasicTicketResponse>
      </s:Body>
</s:Envelope>

The XML response from the authentication service in case of a login failure will be similar to the following:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
        <GetBasicTicketResponse

xmlns="https://winqual.microsoft.com/Services/Authentication/">
            <GetBasicTicketResult a:nil="true"

xmlns:a="http://www.w3.org/2001/XMLSchema-instance" /> 
        </GetBasicTicketResponse>
    </s:Body>
</s:Envelope>

The XML response when the upload is successful is similar to the following ATOM output:

<?xml version="1.0" encoding="utf-8" ?>
<feed xmlns=http://www.w3.org/2005/Atom

xmlns:wer=http://schemas.microsoft.com/windowserrorreporting

 wer:status="ok">
    <title>Feed Success</title> 
    <link rel="alternate" type="text/html"

href="https://winqual.microsoft.com/services/wer/user/fileupload.aspx" /> 
    <updated>2009-06-12 19:28:28Z</updated> 
    <id>Success</id> 
    <entry>
        <updated>2009-06-12 19:28:28Z</updated> 
        <published>2009-06-12 19:28:28Z</published> 
        <title>File uploaded successfully</title> 
        <id>File uploaded successfully</id> 
        <wer:additionalInformation /> 
    </entry>
</feed>

You can check the file upload by going to the website:

https://winqual.microsoft.com/member/wer/user/ManageProductMappings.aspx

Once you sort the table by the Map Date column in descending order, you should see your uploaded product.

You can see the files for the product uploaded by clicking the icon under the File Mappings column.

Windows Error Reporting – Vote for a fix!

Gr8Gatsby — Thu, 14 May 2009 00:38:53 GMT

If you are a developer and reading this blog you may already know of the direct benefits provided by users submitting error reports, if you are an end-user of an application, on Windows, developers are relying on you to help identify solutions to issues that are difficult to reproduce.

I want to take this short blog entry on what good Windows Error Reporting brings to the Windows Operating System and eco-system. I am including hyperlinks that give a view from 3rd parties on the good things that Windows Error Reporting enables for developers on the Windows OS.

The title of this blog posting was inspired by the WindowsITPro.com article below where each error report sent to Windows Error Reporting (WER) is a vote for a fix, although the article was written a few years ago, the theme of voting remains the same. Although not like texting a vote into American Idol, I argue that sending a Windows Error Report is easier and can end up fixing the issue you reported in the end!

Kate details how using Windows Error Reporting to solve problems is helpful on Vista.

http://www.gregcons.com/KateBlog/WindowsErrorReportingIsYourFriend.aspx

O’Reilly gives an overview:

http://windowsdevcenter.com/pub/a/windows/2004/03/16/wer.html

Bob Kelly has a post talking about his recommendations:

http://www.realtime-vista.com/administration/2007/03/windows_error_reporting_option.htm

WindowsITPro.com article highlights how each error report you send in is like a vote for a fix:

http://windowsitpro.com/article/articleid/46982/windows-error-reporting-elementary-my-dear-watson.html

An article talking about how WER has helped to build Windows 7:

http://news.softpedia.com/news/Windows-7-Windows-Error-Reporting-Infrastructure-103252.shtml

Technorati Tags: WER,Windows,Error,Reports,Windows Error Reporting,Fix,Developer

Vista WER Settings UI Deciphered

varunb — Thu, 16 Apr 2009 00:53:38 GMT

This post is a follow up to the original post I did about WER Settings & UI for Windows 7. While that post outlined the settings UI in Windows 7, this post covers the WER settings UI for Windows Vista.

WER Consent Settings: The various WER consent levels are identical across Windows Vista and Windows 7. Please review this post to get an understanding of the various consent levels first. The rest of this blog post explains how these consent options are exposed in the Windows Vista UI.

The WER setting in Windows Vista is exposed in the ‘Problem Reports & Solutions’ (aka PRS) Control Panel.

Clicking the ‘Change settings’ link in PRS, opens up the basic settings page shown below. These settings allow configuration of WER consent settings for that specific logged in user only.

Clicking on the ‘Advanced Settings’ link on this page opens the, well, advanced settings page shown below.

Clicking on the lower ‘Change Settings’ button, brings up a dialog which allows setting consent for all users on the machine. Here is how that dialog looks:

Hope this clarifies the various WER settings and how they can be configured on Windows Vista.

Problem Steps Recorder (PSR.exe) + Windows Error Reporting = Another tool to help find solutions to software defects

Gr8Gatsby — Mon, 30 Mar 2009 22:24:00 GMT

Technorati Tags: PSR,WER,WER Services,Problem Steps Recorder,Repro Steps

There is a bunch of information on the web about PSR and one of the best is this video on CNET: http://cnettv.cnet.com/2001-1_53-50005144.html as you can see a demo of the tool working for an end-user perspective, it behaves differently when invoked by WER as described below. The key difference is that WER doesn’t collect screenshots.

From an error reporting perspective we plan on exposing this via the WER Services. The concept is that if you aren’t able to use standard mini-dumps or heap dumps to find the source of the problem, WER Services will allow the PSR tool to be run (with user consent)

Example of screen-shot shown to user:

When we enable an error Bucket to collect PSR data, here is the workflow:

User prompted with a request for additional data
PSR waits until next time the same application is launched
PSR runs silently in the background while user is using the application
PSR captures descriptions of steps user has taken (without screenshots)
IF the same unhandled exception occurs (BucketID)

The Memory dump, Additional Files, and the PSR output is sent in a CAB file through WER
Developer’s can now download CAB files associated with a particular event that contain these steps a user took leading up to the crash
Developer’s can try to reproduce the error locally following these steps.

After PSR steps are sent with CAB the PSR tool is disabled for user

IMPORTANT NOTES:

PSR.exe when invoked by WER does NOT capture screenshots, see example output below.
Only CAB files that contain PSR steps will be submitted to WER, this means that possibly fewer cabs will be sent for a particular Bucket.
When Windows 7 firsts launches this feature will only be manually enabled by the team behind wer@microsoft.com.

Other links and blog posts:

http://technet.microsoft.com/en-us/magazine/dd464813.aspx

http://blogs.msdn.com/cjacks/archive/2009/02/25/deciphering-the-command-line-configuration-of-the-windows-7-problem-steps-recorder.aspx

http://blogs.msdn.com/tims/archive/2009/01/12/the-bumper-list-of-windows-7-secrets.aspx

Sample output:

Problem Step 1: (3/27/2009 7:53:51 PM) User right click on "user consent (editable text)" in "Problem Steps Recorder (PSR.exe) + Windows Error Reporting = Another tool to help find solutions to software defects - Windows Live Writer"

Problem Step 2: (3/27/2009 7:53:53 PM) User left click on "Hyperlink... (menu item)"

Problem Step 3: (3/27/2009 7:53:55 PM) User keyboard input in "Insert Hyperlink" [... Enter]

Let There Be Hangs: Part 4 – Hashes and Type Codes and XProc, Oh My!

ryankiv — Thu, 19 Mar 2009 22:51:00 GMT

Hang Bucketing, A Better Way

In the previous post I gave a brief introduction of how the first version of hang reporting was implemented using the existing crash reporting infrastructure.

Eventually (after Windows XP shipped) a new general purpose event reporting and bucketing mechanism was built. In a nutshell, this mechanism provides a very flexible way to report incidents to the WER pipeline. It supports a custom named event (internally at Microsoft dubbed “generic events”) and up to 10 custom bucketing parameters P1...Pn...P10.

During the development of Windows Vista, we decided to do something about the problematic hang bucketing and leverage the new general purpose reporting to gain a much better client-side classification of hang issues. Before I explain what bucketing parameters these have, let me discuss a few other new things in Vista:

Another problem with Windows XP hang reporting was that the application process was often hung waiting on an external process. The report would only include a memory dump of the hung application and developers were often forced to give up not able to debug the other process. (Now, granted, you very often don't need to debug the other process to fix a hang, but that is a discussion for a later blog post.)

To solve this problem, the GetThreadWaitChain API was created. This API allows the caller to discover the blocking graph (or “wait chain”) for a given thread. A trivial example: Thread A is waiting to enter a critical section owned by Thread B which is making a blocking SendMessage call to Thread C which is waiting on a mutex owned by Thread D which is running in another process. The output from the API provides the caller all of this information.

It's this mechanism that hang reporting uses in Vista to discover not only the wait chain information but to collect memory dumps of external processes.

Back to the bucketing parameters – the hang reporting infrastructure actually reports hangs to two different generic events: AppHangB1 and AppHangXProcB1 (yes, B1 stands for Beta 1. Don’t ask…). On Winqual, AppHangB1 is shown as Event Type "Hang" and AppHangXProcB1 is shown as "Hang XProc". AppHangB1 has 5 parameters (by convention P1 & P2 are typically Application Name & Version):

P1 - Application Name
P2 - Application Version
P3 - Application Timestamp
P4 - Stack Hash
P5 - Type Code

AppHangXProcB1 has those 5 and adds 2 more:

P6 - Waiting On Application Name
P7 - Waiting On Application Version

Most of these parameters are self explanatory. The two that might not be are P4 and P5:

Stack Hash (P4) - hang reporting traverses the wait chain and for the final thread in the chain before it jumps to a different process, we generate a hash based on the thread’s stack back trace. P4 is a restricted hash which means we chop the MD5 128-bit hash down to 16 bits. This prevents any wild spraying of buckets and when we studied it we were still achieving ~85% uniqueness.

Type Code (P5) - this is a bit field based in part on the WCT_OBJECT_TYPEs found during wait chain traversal with GetThreadWaitChain (e.g., mutex, COM, etc.) but also on a few other items too - like if there's a deadlock or if the hang report came from End Task in Task Manager (more blog fodder).

As rich and wonderful as hang bucketing has become in Windows Vista, there are still edge cases (just as there are in crash bucketing) where a bucket does not uniquely identify a single bug. In a future post I will discuss bucketing more generally and how we REALLY determine and quantify bugs using WER data at Microsoft.

Let There Be Hangs: Part 2 – WER History 101

ryankiv — Thu, 19 Mar 2009 22:50:00 GMT

Crashes Suck

In the beginning, we needed a way to close the loop with our customers in order to ease the pain felt from software defects (bugs) that caused crashes. A simple client service was produced that collected crash dumps from Windows desktops and sent them back to Microsoft for analysis. The crash dumps were debugged, bugs exposed, fixes were made, problems went away. Lather, rinse, repeat. Note: This is how user mode error reporting began. Interestingly, kernel mode error reporting (bluescreens) had its origin in a roughly parallel timeframe (actually a little earlier) but it began in a completely different group at Microsoft. For the rest of this series of posts I’m talking about the user mode side of WER.

Buckets

An early design assumption was that this client had zero analysis ability and so we needed a simple and deterministic way to organize the data that would allow both the client and the backend servers at Microsoft to speak to each other in a common language. Oh, and the solution needed to be (*ahem*) scalable. That common language is the unique combination of event parameters which give us the Event ID. Internally at Microsoft these are actually known as bucket IDs – or just buckets. It’s more natural for me to talk about buckets and I’ll try to be consistent, but just in case, know that “bucket” and “event” are mostly interchangeable in the context of error reporting.

Originally, plain old crashes were bucketed using 5 parameters:

The application’s name
The application’s version
The name of the module containing the instruction that causes the crash exception
The version of that module
The byte offset into that module where that instruction resides

The idea was that crashes with the same set of parameters were caused by the same bug. It turns out that this isn’t always true (We’ll save that for a later post) but in general it works pretty well. Today, this “client” is built as a core component of Windows and became generally referred to Windows Error Reporting (WER) Services. In Vista we added 3 more parameters to plain old crash reporting: file link timestamps for both the application and modules (to try and avoid some issues of naming collisions) and exception code. And by the way, the client itself has become smarter and in fact does some cursory analysis when deriving bucketing parameters.

Notice that you can tell the explorer crash to the right is from Windows XP because those extra 3 parameters don’t have data.

< Read Part 1 [Part 2] Read Part 3 >

Let There Be Hangs: Part 3 – The ‘hungapp’ module

ryankiv — Thu, 19 Mar 2009 22:50:00 GMT

Hang Bucketing, v1

On Windows XP, hangs have it rough. Like a younger sibling, error reporting for hangs has to wear the hand-me-down clothes of crash reporting – it piggybacks on the same 5 fixed bucketing parameters used by crash reporting. However with a hang there is no exception context and so there is no faulting instruction, therefore there is no module name, module version or module offset. So on XP, hangs really only have 2 effective bucketing parameters (application name and application version). The other parameters are set where module name is "hungapp", module version is "0.0.0.0", and the offset is 0.

This means all of particular version of an application’s hangs ended up in a single bucket. Knowing this, you might correctly guess that these buckets have very high hit counts when compared with crash buckets for the same app name/version (though some corruption buckets are not far behind). When someone runs a generalized query similar to, "what are my highest hitting XP buckets this week?", the top end of the results are often littered with hang buckets. A naive conclusion would be "Oh my! Hangs are a huge problem and we must focus all our effort to eradicate them!" While hangs are indeed a big problem, arriving at this conclusion based on bucket hit volume is incorrect. Also, when the actual failures contained in these hang buckets are studied, it is quickly apparent they are composing many bugs – not just the ideal 1 (or few). This is a big reason development teams historically struggled to make progress with hang bugs - the data was difficult to sort and measure.

< Read Part 2 [Part 3] Read Part 4 >

Let There Be Hangs: Part 1 (Not Responding)

ryankiv — Thu, 19 Mar 2009 22:43:00 GMT

Technorati Tags: application,hang,application hang,windows,hung,xproc,hang XProc

Well hang reports anyway…

Those of you signed up on the Winqual site are likely familiar with the "Event ID" and what it (ideally) represents – that is, a basic demarcation of unique software defects. Additionally you’re probably more familiar with crash-related Event IDs like “Crash 32-bit” and “Crash 64-bit”. But what about the other event types? Specifically “Hang” and “Hang XProc”? And by the way – what’s a crash in module “hungapp”?

In this and later posts I’m going to explain a little bit about error reporting and a little bit more about hangs – what they are, how they’re detected, reported, organized, counted, etc. Before getting into all that, however, we need a (very) brief history lesson.

Ghost Windows

Some of you might remember something like the following scene from the days of Windows NT 4 (and maybe you’ve seen similar scenes in more recent times):

This is a screenshot of CALC.EXE moved across an intentionally hung instance of NOTEPAD.EXE.

Without getting into too much nitty gritty, a thread that creates GUI elements, namely windows, has an implicit contract with the desktop window manager that it will service messages that arrive in its message queue... in a timely fashion. “Servicing” means retrieving and dispatching the messages (aka pumping messages). There is plenty of content on MSDN that covers the specific APIs and mechanisms used for this and plenty of content elsewhere about the interesting peculiarities of this area. What "timely fashion" means is somewhat a matter of debate among developers and users. To the window manager, it means 5 seconds by default. Our user research shows that this is actually quite generous.

If a thread stops pumping messages from its message queue, bad things start to happen. In such cases, the thread is often off busy working on whatever task the user just “asked” it to do (e.g. opening a file or recalculating some total or talking to a web service over the internet) and there’s a good chance that it didn’t update its UI to indicate that it’s off doing this work; it almost certainly hasn’t shown any UI resulting from that work. But there is a worse and more fundamental problem – while it’s away, the messages it isn’t pumping could be mouse, keyboard, or touch input along with paint messages. Not pumping those messages would result in a window stuck on the user’s screen – the user wouldn’t be able to move, minimize, or close it and if they moved something in front of it, they’d get something like the jaggy mess in the screen shot above.

I say would because beginning with Windows 2000, we added functionality to the window manager to handle these cases. The concept is actually quite simple – the window manager watches for when a thread has pending input in its queue but hasn’t serviced it for more than 5 seconds. When the situation is detected (aka IsHungAppWindow), the window manager does a presto chango maneuver whereby it hides the window whos thread isn’t pumping input messages and seamlessly shows a replacement window (known as a “ghost” window) with its client area filled with the real window’s last known good client area bits.

While the original thread isn’t responding, the window manager manages both windows in parallel, to the extent that the application doesn’t know this is happening. For example, IsWindowVisible() will actually return true for the hidden/unresponsive window. The only visual difference (initially) to the user is that the text “(Not Responding)” is appended to the ghost window’s title. Using the ghost window, the user can effectively move, minimize, and even close the unresponsive application window. When (if) the thread starts pumping messages again, the original window is re-shown and the ghost window goes away. By the way, this window manager feature is called (obviously) window ghosting - and naturally, we offer an API you should almost never use: DisableProcessWindowsGhosting.

While we were working on Windows XP, we realized that it might be nice to know about these unresponsive applications. We had already started to collect data about crashes, which were clearly annoying to customers, and some schools of thought (and some research) indicated that unresponsive UI was even more annoying -some might even use words like “infuriating” (I know I do). So, we decided to wire up a ghost window’s close button to the infrastructure that sent crash reports back to Microsoft… and voila!, hang reporting was born.

[Part 1] Read part 2 >

FAQ

MSDNArchive — Mon, 16 Mar 2009 19:54:00 GMT

How do I get started with Windows Error Reporting (WER)?

How do I find Windows Error Reporting information on a PC?

How do we match up crash data (Windows Error Reporting signatures) with companies?

What files should I map?

How do we detect applications hangs in Windows?

What is a bucket?

What is a “special” exception?

What are the different types of memory dumps?

When will I see data?

Q: How do I get started with Windows Error Reporting (WER)?

A: As an Independent Software Vendor (ISV) who develops software applications for the Windows Operating Systems, you are able to register to receive Software Quality information from the Windows Quality website.

Please follow this link to a page with directions on how to get started! (http://www.microsoft.com/whdc/maintain/StartWER.mspx).

Q: How Do I find Windows Error Reporting information on a PC?

A: Here are the solutions for finding WER data on XP, Vista, and Windows 7.

OS	WER Information
Windows XP	Event Log
Windows Server 2003	Event Log
Windows Vista	Event Log ”wercon”
Windows Server 2008	Event Log ”wercon”
Windows 7	Event Log Action Center

Windows XP/ Server 2003

On XP / Server 2003 you can find information about error reports in the Application Log.
When a crash occurs the bucketing parameters are written to the Application Event log with the event source of “Application Error” and event ID 1000.
When the crash is submitted the bucket ID is written to the Application Log with the same event source and has an event ID 1001.

Windows Vista / Server 2008

On Vista you can use the “wercon” control panel accessible via:
Start >Control Panel >System and Maintenance >Problem Reports and Solutions >View Problem History
Or
By typing “wercon” in the Search Box at the base of the Start Menu.

Click on the “View problem history” link

Double click on on a line item below the Application Name to view details:

Notice the bucketID and the Copy to Clipboard feature.

Windows 7

Using the Action Center:

Open the Action Center by clicking on the flag icon in the system tray

Select the Action Icon from the system tray, and click Open Action Center.

By expanding the Maintenance toggle button, the customer can see the View System History link.

After selecting the View System History link, the customer can click the View problem reports and responses link.

Or, you can type error into the Search bar on the Start Menu and click the “View problem History”

Select View Problem History, and then select View problem reports and responses.

Q: How do we match up crash data?

A: Here is the general mapping logic we used based on crash data that is gathered by Windows Error Reporting (WER) client. Notice the difference between Pre Windows Vista, and Post Windows Vista.

How do we match up crash signatures with Files that are mapped?

We use 4-6 basic parameters to match mapped files in the developer portal, Windows XP has fewer and matches on File Name and File Version for either Applications and Modules. With this in mind if you are still developing for Windows XP it is a best practice to come up with a unique file name and file version. This best practice translates to any other development environment, but is important with Windows XP as you can see the parameters below that the Windows XP WER Client sends to our service.

Windows XP (Includes Windows Server 2003)

WER client sends these parameters we use to match crashes:

Application Name
Application Version

Module Name
Module Version

With Windows XP/Windows Server 2003 crashes we have less data to use for matching crash signatures with files mapped on the portal.

Windows Vista++ (includes Windows Server 2008)

WER client sends these parameters we use to match crashes:

Application Name
Application Version
Application File Link Date

Module Name
Module Version
Module File Link Date

We use the link date to get a more accurate match on the files mapped on the portal.

Q: What files should I map

A: Given the way we have implemented mapping logic the best policy is to map only the files where you own or maintain the source code. If you ship 3rd party or open source modules with your software where you do not own or maintain source then you will not be able to take action on these error reports, and this information will just be added noise and complexity.

Even if you don’t map 3rd party modules you will still see crash reports where any module crashed your application.

NOTE: We have mapping collision detection that will preclude you from mapping certain files.

Example:

Product Name: FabrikamReportingApplication

Product Binary files:

File Name	File Version	Description	Should I map?
FabrikamReportGUI.exe	1.0.0.0	User interface for my reporting product	Yes
FabrikamWebserviceDataImport.dll	1.0.0.1	Library used to import and transform data from web services.	Yes
ThirdPartyChartControl.dll	5.6.1.3	3rd party DLL I’ve licensed to ship with my report product	No
OpenSourceDataVisualizer.exe	4.5.3.1	An open source application that helps with visualizing multiple data sources (I don’t maintain code for this)	No
OpenSourceGridControl.dll	2.3.1.4	An open source library where I’ve incorporated the source code into my build and maintain changes and updates for this source.	Yes

With the mapping scenario above you will still see crashes where ThirdPartyChartControl.dll caused a crash in the FabrikamReportGUI.exe application.

Q: What is a bucket?

A: A bucket uses a set of parameters gathered by the WER client to describe a WER Event. For crash events the bucketing parameters are Application Name, Application Version, Application Build Date, Module Name, Module Version, Module Build Date, Exception Code, and Code Offset.

There are also Generic Buckets that are made up of up to 10 parameters.

Q: What is “special” exception?

A: A special exception is a grouping of two WER event types: Buffer Overruns (BEX) where the exception code = C0000409, and Data Execution Prevention (DEP) where the exception code = C0000005. These two event types are related to security of software and can also be viewed in the Security Alerts section of the website. In order for the WER client to capture BEX events you need to use the /GS command when compiling your software.

Q: What are the different types of memory dumps?

A: Below is a table that contains brief descriptions of the different types of dump files that can appear in .CAB file.

Extension	Description
.mdmp	minidump (Default dump collection type, stacks, and loaded modules)
.hdmp	.mdmp += heap data
.kdmp	on-demand kernel mode minidump (snapshot of the kernel mode portion of a given user mode process) only sent if WCT (Wait Chain Traversal) added it, or was explicitly requested for a given ibucket by a user.
memory.dmp	Old style (<=XPSP2) heap dump. This is not actually a debugging dump file, instead its just a blob of read only pages
atk.dmp	application termination kernel (taken when an exited process takes more than 10 seconds to actually terminate)
hu.dmp	heap user
mu.dmp	mini user
odk.dmp	on demand kernel
axhu.dmp	arbitrary x-process heap user
axmu.dmp	arbitrary x-process mini user
wxhu.dmp	wct x-process heap user (wct is wait chain traversal)
wxmu.dmp	wct x-process mini user

Q: When will I see data?

A: After mapping files we will start to pull any existing data we have for crashes. Generally you will see Hit data within a day or so. Users who setup Vista and Windows 7 with recommended settings (~80%) will automatically send us the hit data without any prompting, so you can see hits without cabs.

By default we collect 10 cab (minidump) files per event*, since a user still has to click on the “send additional information” button for us to get the crash dump it can take some time to get cab files.

Once we receive cab files for an event you will generally be able to see these cabs within a few hours of us receiving them.

NOTE: For newly detected crashes it can take more than 4 days to get the crashes processed and up on the site. Commonly if you create an app to test our system you will see a longer latency in getting WER data. Since WER is collecting unhandled exceptions on Windows all the time, we recommend mapping existing released software.

*We do not set collection rules for files that are un-versioned, and automatic collection rules depend on the time you map your files. If you are mapping old files, all the buckets have been created and we will not automatically collect files for old buckets. For newly created buckets we will set the collection rules for the most common file extensions.

An overview of WER consent settings and corresponding UI behavior

varunb — Wed, 11 Mar 2009 21:52:00 GMT

This post gives a high level overview of the various consent settings WER supports and the corresponding experience a user has when opted into any of those setting. Before going into further details, I want to clarify a couple of terms used here:

WER Client – The binaries that ship with the Windows OS and report problems to Microsoft.
WER Service – The backend service hosted by Microsoft to which the WER client reports problems.

Windows users have three different levels to choose from when opting into reporting errors to Microsoft. these levels are:

Level 1 – If a user chooses this option, Windows does not report problems to Microsoft automatically. Instead, every time a fault (like a crash or hang) occurs on the user’s machine, a dialog is presented to the user, to get explicit consent to report the fault to Microsoft.
Level 2 (Is the default recommended by Windows - more details about defaults later in the post) – For users choosing this option, Windows automatically reports the problem signature (aka WER event parameters) to the Microsoft WER Service. However, if the backend WER Service requests for additional information from the machine (like mini dumps, log files), a dialog is presented to the user to get explicit consent before uploading the additional information.
Level 3 – For users choosing this option, Windows automatically reports the WER event parameters to the backend WER Service, and if the WER Service requests for additional information that does not contain any personally identifiable information (PII), then that information is uploaded to the service automatically. An example of such information is a mini dump, which has an extremely low probability of containing PII. However, if the information requested by the WER Service has the potential to contain PII (for example, a heap dump), a dialog is presented to the user to get explicit consent before uploading the additional information.

Note that the three consent levels only come into play if WER is enabled for the user/machine. If WER is disabled, then the user is never even prompted to report problems.

Following is a summary of the typical Windows 7 user experience for each of the consent levels (Note: Windows 7 is still in Beta and the UI is still susceptible to changes):

User experience when WER is disabled: Error reporting does not kick in. A simple dialog informing the user about the crash is shown.

User experience when WER consent level is set to 1:

     Step 1: On detecting a crash, WER client presents UI to get explicit user consent to report WER event parameters to Microsoft’s WER backend service



      Step 2: On getting consent in step 1, WER client starts reporting parameters to Microsoft



      Step 3: If the WER Service requests additional data (like heap dumps, log files etc.), WER client starts collecting the information.



      Step 4: After completing data collection, WER client asks user for explicit consent to upload data to Microsoft. If user agrees, WER client uploads the data. If user clicks ‘Cancel’, the data is discarded and not sent to Microsoft.

User experience when consent level is set to 2:

Step 1: On detecting a crash, WER client automatically starts reporting WER event parameters to Microsoft’s WER Service.

Step 2: If the WER Service requests additional data (like mini dump, heap dumps, log files etc.), WER client starts collecting the information.

Step 3: After completing data collection, WER client asks user for explicit consent to upload data to Microsoft. If user agrees, WER client uploads the data.

User experience when consent level is set to 3:

Step 1: On detecting a crash, WER client automatically starts reporting WER event parameters to Microsoft’s WER backend service.

Step 2: If the WER Service requests additional data (like mini dump, heap dumps, log files etc.), WER client starts collecting the information

Step 3: If the data collected is considered safe (i.e. not containing PII; like a minidump) WER client uploads that data automatically without asking for explicit user consent. If however, the data isn't considered safe (like heap dumps, log files etc.), WER client asks user for explicit consent to upload data to the WER Service. If user clicks ‘Send Information’, WER uploads the data.

Note: It is noteworthy that WER client never uploads data containing personally identifiable information (like heap dumps) automatically. Explicit user consent it always needed to upload such information. This design decision has been made to respect the privacy of Windows users.

So how does a user configure/change WER settings?

The first place where a user chooses to opt into WER is the Windows Out of the Box Experience (aka Windows OOBE). The first time a new Windows 7 machine boots up, an OOBE screen is presented to the user which opts the machine into one of two WER consent levels depending on the user’s selection.

If the user does not choose “recommended settings” in OOBE, another “up-sell” dialog is presented to the user in the Windows Troubleshooting Control Panel

In the above dialog, if the user clicks ‘Yes’, his consent is bumped up to level 2. Note however, if the user has already set his consent level to higher than 2, it is not brought down.

The OOBE and the up-sell dialog are presented to the user only once in the entire lifetime of his Windows 7 installation. If the user wants to change WER settings there after, he can do so by accessing the settings page via the Action Center Control Panel

This page allows setting the consent only for the specific user interacting with this page. However, if an administrator wants to configure settings for all users on a machine, he can do so by clicking on “Change report settings for all users” and making the same choice in that dialog.

Hope this information helps you understand the various WER settings, the corresponding Windows UI behavior for each of the settings, and the exactly how to configure/edit these settings.

Happy Error Reporting!