WER Services

Windows Dev Center Web Services Update

DarrenMcKenzie — Tue, 09 Oct 2012 23:59:03 GMT

Hello again. With an update coming soon to the Dev Center reports, we have made corresponding updates to the Web Service based on the reports. All of the Dev Center Reports are exposed as feeds. This allows your company to pull down all report data, including CABs, programmatically without directly connecting to the Dev Center site. For more information on the feeds in general, including supporting sample code, please see the MSDN Dev Center Web Services overview.

Here are the list of changes in the feed. These updates will start for some companies at the end of this week and will be available for all companies with current reports by next week.

1) Product / Product Version split in the Product Detail Report. The feed splits the Product Version from the Product Name into separate fields.

Report Name: ProductDetailNonSuiteReport, ProductDetailReport.rdl

Feed Name: ATOMFeed

Example Before:

Product and Product Version: <d:Product>CrashMeForBuild Version 1.0.0.0</d:Product>

Example After:

Product: <d:Product>CrashMeForBuild</d:Product>

Product Version: <d:ProductVersion>1.0.0.0</d:ProductVersion>

2) Offset breakout of CABs: The Offset is the place in the code where the failure is happening. It helps with determining the root cause of a set of issues. Offset values are added in two feeds in the reports.

a) Offset addition to Product reports

Reports: CompanySummaryReport.rdl, ProductSummaryReport.rdl, ProductDetailReport.rdl, ProductSuiteSummaryReport.rdl, ProductSuiteDetailReport.rdl

Feed Name: ATOMFeed

Example new field:

Offset Value: <d:Failure_Offset>11A7</d:Failure_Offset>

b) Offset update to CAB details

Reports: CompanySummaryReport.rdl, ProductSummaryReport.rdl, ProductDetailReport.rdl, ProductSuiteSummaryReport.rdl, ProductSuiteDetailReport.rdl

Feed Name: CABFeed

Example new fields:

Offset Value: <d:Offset>106D</d:Offset>

Offset as percentage of CABs: <d:OffsetPc m:type="Edm.Double">1</d:OffsetPc>

Offset as percentage for other CABs: <d:OffsetAllOtherPercentage m:type="Edm.Double">0</d:OffsetAllOtherPercentage>

3) Addition of Operating System/Service Pack and file version associated with the CAB (can be Kernel or User Mode)

Reports: CompanySummaryReport.rdl, ProductSummaryReport.rdl, ProductDetailReport.rdl, ProductSuiteSummaryReport.rdl, ProductSuiteDetailReport.rdl

Feed Name: CABFeed

Example New Fields:

CAB OS and Service Pack: <d:CabOS>Windows 7</d:CabOS>

Is a File or Driver associated with CAB: <d:CabsFileDriverVersionHeader>File Version</d:CabsFileDriverVersionHeader>

Version of file associated with CAB: <d:CabVersion>1.0.0.1</d:CabVersion>

4) Addition of Operating System/Service Pack and file version associated with CAB (Kernel Mode or driver specific)

Reports: DriverDetailReport.rdl

Feed Name: CABFeed

Example New Fields:

CAB OS and Service Pack: <d:CabOSVersion>Windows 7 (SP1)</d:CabOSVersion>

Version of driver file associated with CAB<d:CabDriverVersion>6.1.7600.16385</d:CabDriverVersion>

5) Additions to ATOM feed to support pulling multiple driver versions in a report

Report: DriverDetailReport.rdl

Feed: ATOMFeed

Example New Fields:

CAB Date<d:Cab_Date m:type="Edm.DateTime">2012-08-02T00:00:00</d:Cab_Date>

CAB Type<d:Cab_Type>Kernel Mini</d:Cab_Type>

Link to CAB<d:Cab_Download_Link>CAB URL</d:Cab_Download_Link>

Windows Dev Center Reports Update

DarrenMcKenzie — Thu, 27 Sep 2012 08:20:00 GMT

Hello. I’d like to provide some information on some changes available next month with the Dev Center Reports.

As mentioned in “The only thing constant is change” blog, user mode reports use the Failure based approach. When symbols are not available (the most common scenario) this can sometimes make it difficult to determine the root cause of an issue. In these cases, Failures with the same exception code, problem class and module are all grouped into failure with !unknown instead of the function name.

To help mitigate this issue the CABs will be broken down by an Offset grouping, with the Offset value and the percentage of overall CABs that have that value. The Offset is the place in the code where the failure occurred and it helps determine the root cause of a set of issues. One item to keep in mind is that a file update can cause the Offset to move, so symbols are almost always the best way to identify issues.

Up to 10 CABs per Offset will be available and each Failure can show up to 10 Offsets. This means that up to 100 CABs could be provided per Failure. Up to now, only 10 CABs are available per Failure. With the combination of more CABs and an Offset breakout, more debugging information will be available per Failure.

Each Failure will show an Offset breakout per CAB set, such as the below:

Each user mode report type will provide an Offset breakout per Failure including:

1) Company Hotlist – The top 10 Failures are provided across all mappings

2) Product Group summary – Provides top 10 Failures across all product mappings in a product group. Each of the product groups provide a per product drill down

3) Product Summary – Provides top 10 Failures for product not in a product group. There is also a per product drill down providing more Failures

Based on feedback – we’re are also making the following changes with the CAB section of the report:

For both user and kernel Failures, Full or Heap dumps will be prioritized over Mini dumps
The OS version will be provided. This will be at an OS and Service Pack level, e.g. Windows 7 (SP1)
For User Mode reports the File Version will be displayed, for Kernel Mode reports, the Driver Version
For the Driver Summary Report, the Version Grouping will use a version sort rather than a string sort

Example of Kernel Mode Changes. The OS Version and Driver Version are new fields in the CAB section:

Example of User Mode Changes. CAB are broken down by an Offset group. OS Version and File Version are new fields in the CAB section:

With the report changes, this means some changes to the feed. The next blog will go over the feed changes in detail……

Windows Dev Center – Reports FAQ

DarrenMcKenzie — Wed, 28 Mar 2012 19:00:58 GMT

How do I map files?

How can I see Driver Reports?

How can I see reports in the Product group summary?

How do I make sure I see data for my Product based mappings?

Where can I find information on report automation?

How do I automate generating product mappings?

How do I use PowerShell to automate product mappings?

What is the Windows 8 Reliability Mappings Product and why do I see this within my Product summary list?

Where can I go if I can’t find an answer to my question?

Q: How do I map files?

A: The Product Mapping Tool (also known as the Microsoft Ecosystem Metadata Exchange client) is the correct tool for mapping a new product, product group or driver file. A prerequisite for the Product Mapping Tool is you will need to install at least one item from the Windows Live Essentials. This is to support Windows Live ID authentication on the Developer Dashboard. The Product Mapping Tool itself is available at the bottom of the Reports main link off the dashboard. You will need to log-in to see this view

Choose the top level Reports link:

The Product Mapping tool is at the bottom of the page under Downloads (Failure data is blurred out in this view):

Once you have authenticated using your Windows Live ID and finished synchronizing with the Microsoft Metadata Exchange Server, you will see your existing drivers and products if previously

provided. The mapping inventory is divided into three areas:

1) Drivers – Provides driver mappings on a file name basis. These are files that inform the Driver summary report, which show a version breakout. Drivers can be mapped through choosing Drivers New Driver. When a duplicate mapping already exists, the file will need to be investigated for ownership. Once signoff exists, reports can be generated. In most cases the Signoff Status will be “Not needed.”

2) Products – Provides data based on specific files or file versions from specified directory. These files can be either kernel and/or user mode; however, the kernel files must be already mapped as a file in the Drivers section in order to be seen in Product based reports. The new product cycle view is a way of determining where in the development stage a product falls. This can help companies evaluate where to invest their development resources on fixes.

Products can be mapped through Products, New Product

Choose a Name and Version

And add the appropriate file or directory of files

Like driver files, when a duplicate mapping already exists, the product file will need to be investigated for ownership. Once signoff exists, reports can be generated. In most cases the Signoff Status will be “Not needed”

3) Product Groups – Groups products for reporting. These mappings make up the Product group summary report. Products not part of a Product group are available in the Product summary report.

Product Groups are mapped by choosing the Product Group item

and associating the Product Group with existing Products

Once you’ve made your updates, you can publish your changes. Published mappings will update the report data once CABs are collected for your selected Drivers, Product and Product Groups

Previously provided metadata around Drivers, Products and Product Groups can be saved through Tools Import metadata and Export Metadata

Q: How can I see Driver Reports

A: Make sure the Product Mapping Tool has entries in the Drivers area. The Driver Summary report will show error reporting data across all versions

Q: How can I see reports in the Product group summary?

A: Product Group must be mapped by default in the Product Mapping Tool. Product Groups are made up of different Products. This is a new concept that allows for more flexible ways of organizing data by concepts that makes sense to your organization (e.g. by Suite, by Division.) Once these mappings are created, the next generated set of reports will show a populated Product summary report

Q: How do I make sure I see data for my Product based mappings?

A: In order for data to be provided to the reports the following conditions must be met:

1) Product data is based on the file version and link date granularity. If a file version is used across multiple link dates, make sure mappings exist for all applicable files

2) CABs are collected within the last 30 days and a Failure (see the blog “the only thing constant is change – Part 1” for more information) has at least 4 hits

3) You company has logged into the Windows Dev Center within the last 60 days

Q: Where can I find information on report automation?

A: You can download sample code and description of the feed for each core report from the Microsoft Download Center

Q: How do I automate generating product mappings?

A: Automation is done through PowerShell. PowerShell uses the MicrosoftExchange module.This module can be directly initiated through the client

or directly though the PowerShell command

PS C:\Users\testuser> import-module MetaDataExchange

in both cases the Product Mapping Tool (Metadata Exchange client) must be present on the local system. You can get a list of commands through the below:

PS C:\Users\testuser> get-command -module metadataexchange

Q: How do I use PowerShell to automate product mappings?

A: Much like directly using the Product Mapping client, a product and associated files need to be provided.

Create a new product

This action uses the New-Product commandlet and pipes the output into Add-Product, which adds the product to the current Metadata Object

NAME New-Product

SYNTAX

New-Product [-Name] <string> [-Version] <string> [[-Lifecycle] <Lifecycle> {Prerelease | CurrentRelease |

Supported}] [<CommonParameters>]

NAME Add-Product

SYNTAX

Add-Product [-Product] <Product[]> [-IgnoreDuplicates] [-WhatIf] [-Confirm] [<CommonParameters>]

Add-Product [-ProductGroup] <ProductGroup> [-Product] <Product[]> [-IgnoreDuplicates] [-WhatIf] [-Confirm]

[<CommonParameters>]

Console commands:

PS C:\Users\TestUser> $p1 = New-Product -Name "CrashMe" -Version "1.0.0.0" -Lifecycle Supported | Add-Product

PS C:\Users\TestUser> $p1

Console output:

Name : CrashMe

Version : 1.0.0.0

Lifecycle : Supported

CreatedDate : 1/1/0001 12:00:00 AM

ModifiedDate : 1/1/0001 12:00:00 AM

ProductFiles : {}

ProductGroups : {}

This process can be repeated to create multiple products. In the below case a second variable is being assigned to the additional product

Console commands for second product:

PS C:\Users\TestUser> $p2 = New-Product -Name "CrashMe" -Version "1.0.0.0" -Lifecycle Supported | Add-Product

PS C:\Users\TestUser> $p2

Add files to the product

We are going to be using the New-ProductFile commandlet and pipe the output into Add-ProductFile commandlet passing in the Product variable ($p) we created in the previous command

NAME New-ProductFile

SYNTAX

New-ProductFile [-Path] <string[]> [-IgnoreErrors] [-RecursionDepth <int>] [-WhatIf] [-Confirm]

[<CommonParameters>]

New-ProductFile [-LiteralPath] <string[]> [-IgnoreErrors] [-RecursionDepth <int>] [-WhatIf] [-Confirm]

[<CommonParameters>]

NAME Add-ProductFile

SYNTAX

Add-ProductFile [-Product] <Product> [-ProductFile] <ProductFile[]> [-IgnoreDuplicates] [-WhatIf] [-Confirm]

[<CommonParameters>]

Console commands:

PS C:\Users\TestUser> New-ProductFile "C:\Program Files (x86)\App\CrashMeForBuild\Release1" | add-ProductFile $p1

Console output:

FileName       : CrashMeForBuild.exe
FileVersion    : 1.0.0.1
LinkDate       : 8/24/2011 8:02:14 PM
IsThirdParty   : False
CompanyName    : SpiralOrbit
ProductName    : CrashMeForBuild
ProductVersion : 1.0.0.1
ProgramId      : 00000f85cc37222304c06205aa6a3b7b755600000904
FileId         : 000006cea01916adcf2526f7607a9d3ee1e806d6d6b0
SignoffStatus : Requested
CreatedDate    : 1/1/0001 12:00:00 AM
ModifiedDate   : 1/1/0001 12:00:00 AM

this process can be repeated for additional Product Versions

Console command for second product:

PS C:\Users\TestUser> New-ProductFile "C:\Program Files (x86)\App\CrashMeForBuild\Release2" | add-ProductFile $p2

Once the mappings have been generated, the output can be consumed by the Product Mapping Client.

Export Metadata to file

We are going to use the Export-Metadata commandlet to save a file that can be consumed by the Ecosystem Metadata Exchange Client tool for upload. You need to save the file extension as .emx for it to be recognized by the EMX.exe client tool.

NAME Export-Metadata

SYNTAX

Export-Metadata [-Path] <string> [-CreateNew] [<CommonParameters>]

Console command:

PS C:\Users\TestUser> Export-MetaData "C:\CrashMeForBuild\Filemappings\EMXMetadataTest.emx"

Console output:

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         4/20/2012   3:35 PM       2550 EMXMetadataTest.emx

Import the .emx file to the client tool

From the Tools Menu select Import Metadata

point to the directory with the saved .emx file

Select the file and open

You will see the changes reflected in the Publish Changes section of the tool. Click on Publish to submit the products for processing

Q: What is the Windows 8 Reliability Mappings Product and why do I see this within my Product summary list?

A: The Windows 8 Reliability Mappings Product is to support file mappings for the Windows 8 Reliability report. This report looks at the top issues affecting the reliability of Window 8. In order to provide CAB links within the report, mappings are required at the file version and link date granularity. If these mappings are not currently available through your organization’s current Product based mappings, they will be added to the Windows 8 Reliability Mappings Product. This is to ensure your organization continues to get CABs for the Windows 8 Reliability Report

Q: Where can I go if I can’t find an answer to my question?

A: Contact the Windows Dev Center Support Team

The only thing constant is change – Part 1

Kevin_Hill — Mon, 08 Aug 2011 15:52:05 GMT

Hello it’s Kevin Hill here. I want to provide some insight into where we are heading with the WER datasets for partners.

We surveyed users of WER data on https://winqual.microsoft.com to get feedback for the direction of features, the below two themes surfaced quickly as a result:

“Reduce noise” – we had a lot of feedback around filtering out low priority “events” as these end up being time-wasters.
“More automation & notifications” – Don’t make me always come to the website to get information push it to me.

We have made feature investments to help make progress in these areas:

“Reduce noise”

For the user mode datasets in the past we had shown “event” (a.k.a. Bucket) based information. We are moving this dataset to be based on Failures which are the result of the !Analyze debugging extension. Below I scratch the surface with the difference between these two datasets, but watching the Channel 9 video really helps to understand what logic goes into analyzing failures.

Buckets classify issues by: Application Name, Application Version, Application Build Date, Module Name, Module Version, Module Build Date, OS Exception Code, and Module Code Offset. On the surface this looks like a very precise way to classify issues; however it can be misleading at times since we generate this bucket on the Windows OS client without performing any symbol analysis on the memory dump. The module that is picked by the Windows Error Reporting client is the module at the top of the stack. We have found that investigating these issues as raw buckets can cause many investigations that result in a faulting module that is different from the original bucket determination. Ryan Kivett (a developer lead from the Windows Reliability team) has a great blog post titled: WER History 101

Failures, however, are the result of processing memory dumps collected from buckets in order to more precisely assess blame to the appropriate module. I have included the graphic below to show how buckets can result in multiple different failures. While this is a typically scenario once private symbol analysis has been applied, there can also be cases where multiple buckets get aggregated into a !Unknown symbol as well. If you download cabs and run !Analyze with your own private symbol paths you will break down the !Unknown failures into a very precise set of issues.

Here is another example considering multiple buckets:

Since we are performing automated analysis of issues using private symbols we are able to filter out issues where we know we will likely be able to provide a fix, in order to provide fewer issues that will result in investigations that point to issues in modules that don’t belong to you. This is one of the ways we are reducing the noise in the dataset. This doesn’t mean that there will be fewer number of issues, but the issues we share are more likely to be actionable for your software.

The below video contains a detailed explanation of the logic behind !Analyze and what goes into the failure dataset.

“More automation & notifications”

We are providing email updates for all the new failure based reports that are going to be published based on the new user mode failure based dataset and the kernel mode failure based dataset as well.

Stay tuned for more updates on planned changes, Part 2 will provide more information…

Why You Should Use XMLHttpRequest Asynchronously

Dom R — Wed, 03 Aug 2011 20:20:00 GMT

8.4% of all hangs in IE9 in the past month are caused by XMLHttpRequest objects blocking the UI thread with a synchronous request. That’s a huge number! With some manageable code changes, these hangs can be avoided, and developers can give their users a better experience across their websites. We’ll get into what’s happening here, what you can do about it, and we’ll give a little demonstration to see firsthand what can happen when a synchronous request hangs the browser.

Synchronous XMLHttpRequest Blocks the UI Thread

We already know performing a blocking operation on a UI thread is asking for trouble. We’ve blogged about this before. It might not be apparent, but this is exactly what can happen when calling the XMLHttpRequest.send() method synchronously (i.e. passing false for bAsync when calling the open method). The UI thread has to wait until it gets a response from the server, or until the request times out. While waiting, it’s not pumping messages.

Why Use Synchronous XMLHttpRequest At All?

Developers tend to use synchronous over asynchronous is because it’s easier and can many times lead to less complex code. This StackOverflow thread states some potentially valid opinions on why you’d use one over the other, but nothing compelling enough that makes it worth the risk.

Fixing It…

There are at least two ways you can write your code to avoid these hangs. The first method is preferred, because you won’t block on the UI thread. The second is less than optimal, because you’ll still block, but for less time than you’re probably doing now.

Write your code to call open asynchronously.
You need to pass true for the bAsync parameter to the open method, and you need to write an event handler for the onreadystatechange event. There are many examples of how to do this out one the web, like this one.
Set the timeout property.
Because the Desktop Window Manager detects a hang after 5 seconds of unresponsiveness, and IE9 Hang Resistance after 8, I would set the timeout property to something under 5 seconds. You may also want to consider creating a function to handle the ontimeout event.
(How did I get these numbers? Refer to here and here respectively – also already mentioned above)

How About an Example?

Let’s dig into the details by creating a couple of scenarios:

We’ll create some code that will make a synchronous request and hang the browser, and then
change the code to be asynchronous, and observe the acute change in browser behavior.

The goal of these scenarios is to demonstrate how switching from synchronous to asynchronous requests will improve the user experience of your web applications.

DISCLAIMER: This will hang your browser and may result in unintended, perhaps even undesirable behavior on both the client and the server. In other words, don’t run this on your production server, or any other machine that can’t tolerate running instable and potentially broken code.

The Setup

You’ll need a web server capable of running ASP.NET if you want to follow the example directly. I realize some readers may not be using ASP.NET, but the code should be straight-forward enough that you can adapt it to other environments without too much trouble.

Also, I will assume you are familiar with your web server environment and know how to render a page server-side.

Copy this code into your favorite text editor and save it as hangme.aspx somewhere on your web server.

 <!-- code starts after this line -->
 <%@ Page Language="C#" %> 
 <html> 
 <head> 
 <title>XmlHttpRequest open hang test</title> 
 <script runat="server">
    protected void Page_Load(object sender, EventArgs e) { 
       if(Request.QueryString["hang"] == "1") {
          int seconds = 0; 
          Int32.TryParse(Request.QueryString["seconds"], out seconds); 
          System.Threading.Thread.Sleep(seconds * 1000); 
       } 
    } 
 </script> 
 <script type="text/javascript"> 
    function call_hangme() { 
       var oReq;       if (window.XMLHttpRequest) { 
          oReq = new XMLHttpRequest(); 
       } 
       if (oReq != null) { 
          var sUrl = "http://localhost/hangme.aspx?hang=1&seconds=360"; 

          <!-- change localhost to your server name if applicable --> 
          document.getElementById("txt1").innerHTML = "Request Sent..."; 

          <!-- pass false for the bAsync parameter for a synchronous request --> 
             oReq.open("GET", sUrl, false); 

          oReq.send(); 
          document.getElementById("txt1").innerHTML = "Response Received!"; 
       } 
       else { 
          window.alert("Error creating XmlHttpRequest object."); 
       }
    }
 </script> 
 </head> 
 <body> 
    <p>Click this button to hang the browser</p> 
    <form name="form1" action="" method="get"> 
       <input type="button" name="btn1" value="hang me" onClick="call_hangme()"> 
    </form>
    <p id="txt1"/> 
 </body> 
 </html> 
 <!-- code ends on the line before this one -->

Make It Hang

Now,

Open Internet Explorer and browse to the hangme.aspx page.
Click the “hang me” button.

Uh-oh! We’re hung! Because we told the thread to sleep for 6 minutes, we’re going to be here awhile. If you’re using IE9, you’ve probably noticed the gold band at the bottom offering to “Recover webpage”. If you’re using another browser, you’ve probably got a ghosted window.

Calling Open Asynchronously

Copy this code new file and save it as wont_hangme.aspx.

 <!-- code starts after this line -->  
 <%@ Page Language="C#" %>
 <html>
 <head>
 <title>XmlHttpRequest open hang test</title>
 <script runat="server">
    protected void Page_Load(object sender, EventArgs e) {
       if(Request.QueryString["hang"] == "1") {
          int seconds = 0;
          Int32.TryParse(Request.QueryString["seconds"], out seconds);
          System.Threading.Thread.Sleep(seconds * 1000);
       }
    }
 </script>
 <script type="text/javascript"> 
    function hangme() {
       var oReq;
       if (window.XMLHttpRequest) {
       oReq = new XMLHttpRequest();
       }

       if (oReq != null) {
 
          <!-- change localhost to your server name if applicable -->
          var sUrl = "http://localhost/wont_hangme.aspx?hang=1&seconds=500"; 
          document.getElementById("txt1").innerHTML = "Request Sent...";

          <!-- pass true for the bAsync parameter -->
          oReq.open("GET", sUrl, true); 

          <!-- Here we define an anonymous function for the 
           onreadystatechange event.  We check if we received all the data, 
 >           and the request was successful before changing the text
          -->

          oReq.onreadystatechange = function() {
             if (oReq.readyState == 4 && oReq.status == 200) {
                document.getElementById("txt1").innerHTML = "Response Received!";
             }
          }

          oReq.send();
       }
       else {
          window.alert("Error creating XmlHttpRequest object.");
       }
    }
 </script>
 </head>
 <body>
    <p>Click this button to hang the browser</p>
    <form name="form1" action="" method="get">
       <input type="button" name="btn1" value="hang me" onClick="hangme()">
    </form>
    <p id="txt1"/>
 </body>
 </html>
 <!-- code ends on the line before this one -->

The new code is highlighted. Here, we’re now calling open with bAsync = true, which means we’re making an asynchronous call. This will free up the UI thread to do other things, instead of it being blocked waiting for the response or timeout. If we’re not blocked, we can pump messages, and if we can do that, we’re not hung!

Now, when the response eventually comes back from the server, we’ll need a function to handle it. That is what the highlighted code does. It’s our event handler for onreadystatechange event. As you can see, I’m only checking that there response was successful (status == 200) and that all the data has been received (readyState == 4).

Can We Make It Hang Now?

Again,

Open Internet Explorer and browse to the cant_hangme.aspx page.
Click the “hang me” button.

Voilà! You’ll see the request is sent, but the browser is not hung. When the response finally makes its way back, the text will change; all the while you’re still able to interact with the page.

In Closing

As stated from the outset, you want to implement your XMLHttpRequest calls asynchronously. This example clearly demonstrates why; you’ll improve the responsiveness of your pages, avoid hangs, and most importantly, give your users a better experience.

A checklist to help you create clear and accurate responses

kentwalter — Tue, 09 Nov 2010 17:49:00 GMT

We’ve had some questions and comments from our partners recently about the process of creating and publishing responses for Windows Error Reporting, so I want to give you some information that will help you create quality responses that get published quickly and really help your customers.

I work on the team that approves and publishes responses, and if you’ve created more than a few responses, there’s a good chance that we’ve worked with you to clarify your response content at some point. Since clarifications add time to the approval and publishing process, I wanted to give you a checklist of things to look at before you submit a response, so that your responses will sail through approval and publishing.

· Is the landing page straightforward and uncomplicated? Can you link to a better page, or better yet, a direct download? (The number one reason we clarify responses is that it links to a top-level page, like www.microsoft.com.)

· Does the experience described in the template you used match the one that users will actually see? For example, if the template says a direct download is available, the linked URL should point to that download, not to a landing page that requires further action from the customer. Note that the templates you see on Winqual are only a subset of what is available.

· Does a landing page link provide what is promised? For example, if the template selected promises an automated update tool, but the landing page for the response describes a manual update process that doesn’t use a tool, the message could be clearer.

· Is the solution within easy reach of the user? The guideline we try to hit with every response is three clicks, including the main URL in the response itself.

· Is your download as directly and easily usable as it could possibly be? Users have an easier time with an MSI than they do with a self-extractor or .zip file, for example, and the smaller the download, the better.

· Does your response require More Info links? These should not be used unless the response does not work without them. There are a couple of reasons for this:

Any More Info link text will need to be localized, which means that the entire response be available only in English for 4-6 weeks
We found through our research that most customers just don’t see More Info links. Customers look for links in the body of the response.

These points cover the majority of response clarifications that we see. Of course, you might need input or have questions before you even create a response. We’d be glad to help you out. If you have any questions or any specific needs, let us know. Send email to WERCMS@microsoft.com, and we’ll give you a hand. Thanks for using Windows Error Reporting and letting us help you.

Kent Walter
Windows Error Reporting

XPROC Application Hang Cabs in Windows 7

Dom R — Mon, 16 Aug 2010 23:18:00 GMT

In Windows 7, we've added some techniques and tools that assist with troubleshooting hangs involving multiple processes (if you're interested in the evolutionary history, check out Ryan's "Let There Be Hangs" series, where he describes how hang reporting has changed from Windows XP on.) With these improvements come some new concepts, terms, and techniques, which I will attempt to convey so that, after reading this, you can fire up the debugger and understand a bit more about what’s going on.

Defined Terms and Concepts

XPROC cab

The “X” stand for cross and the “PROC” stands for process. So in other words, we’re talking about a cross process dump. And what is that exactly? Well, when an app is hung due to cross-process communication, we capture a dump of the hung process and, when possible, the process determined to be at the end of the “wait chain” in the inter-process communication. We include both dumps, as well as other analysis related information, in the cab submitted with the error report. XPROC dumps were introduced in Vista, but there were cases where the state of the hang would be lost before the heap dump was captured.

Process Reflection

At its core, Process Reflection is used to make a clone of an existing process. This clone, also referred to as the reflected process, contains a single thread and a copy of the original process’ address space. Mark Russinovich talks more about it here (31:00)

In the case of hang reporting, when a chain of processes is involved in the hang, we use Process Reflection to create a clone of the process determined to be at the end of the wait chain by virtue of the GetThreadWaitChain API. A heap dump is taken of the clone instead of the original. Cloning takes way less time than generating a heap dump file, so using this technique, we increase our chances of capturing the hung state, than if we were just create a dump of the original process. Also, it’s worth pointing out that our single thread in the clone does not execute during hang reporting, so it is simply parked and left idle. Here’s what its stack looks like today:

ChildEBP RetAddr
0028fdbc 77a97b8c ntdll!KiFastSystemCallRet
0028fdc0 77af02ef ntdll!ZwWaitForSingleObject+0xc
0028fe38 75ef0239 ntdll!RtlpProcessReflectionStartup+0x1bf
0028fe44 77ab1ec2 kernel32!BaseThreadInitThunk+0xe
0028fe84 77ab1e95 ntdll!__RtlUserThreadStart+0x70
0028fe9c 00000000 ntdll!_RtlUserThreadStart+0x1b

When it comes to debugging analysis, some things aren’t readily apparent with the reflected process. For instance, what about the original threads? Does the handle data conflict with that of the original process? After all, the reflected process is a process in its own right. The short answer is the debugger handles all of this for you. As they say, “the devil is in the details” so we’ll get into how the debugger does this shortly. But first, it’s important to understand the files contained in an AppHangXProcB1 bucket’s cab.

On to the Fun Stuff...

AppHangXProcB1 Cab Contents

So to recap:

A hang has occurred. We get a heap dump of the hung process. Using GetThreadWaitChain(), we discover the process at the end of the wait chain and clone it using ProcessRelection.
Next we grab a heap dump of the clone. But, we also have to deal with the fact that the thread and handle data from the original aren’t copied to the clone. So, in addition to this heap dump, we also capture a mini dump of the original process that has the thread and handle data. The debugger knows how to “stitch” both files together into a single, coherent target.

Let’s take a look at an example cab:

File name                      File size     Date      Time   Attrs
----------------------------- ---------- ---------- -------- -----
   iexplore.exe.xml                  5496 2010/07/14 16:25:22 ----
   iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini      224 2010/07/14 16:25:22 ----
   WER57FB.tmp.ref                1596362 2010/07/14 16:25:22 ----
   iexplore.exe.6e79.wxhu.dmp    150380775 2010/07/14 16:25:22 ----
   memory.hdmp                   40343241 2010/07/14 16:26:18 ----
   WERInternalMetadata.xml           3226 2010/07/14 16:26:30 ----
   AppCompat.txt                    47354 2010/07/14 16:26:30 ----

Look at the contents of the dbgcfg.ini file and you'll see something like this:

[UserModeDump]
default=WER57FB.tmp.ref
memory=iexplore.exe.6e79.wxhu.dmp
module=iexplore.exe.6e79.wxhu.dmp

You may be able to discern what’s going on here. The *.tmp.ref file is the default target, which is the original process mini dump that I described earlier; the one with the thread and handle data. Using the debugger, you can view the flags passed to MiniDumpWriteDump used to create this file and thus the data contained within:

0:004> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 11
Flags           1104
                0004 MiniDumpWithHandleData
                0100 MiniDumpWithProcessThreadData
                1000 MiniDumpWithThreadInfo

The *.wxhu.dmp file is the reflected process dump. Its naming convention hasn’t changed from Vista. So, those of you that have been working with XPROC dumps in Vista may have been a little confused when you analyzed this file from a Windows 7 machine, only to see one thread waiting in RtlpProcessReflectionStartup. Open the file directly in the debugger and you’ll see that it has full data, thread and memory info, etc…

0:000> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 14
Flags           51B25
                0001 MiniDumpWithDataSegs
                0004 MiniDumpWithHandleData
                0020 MiniDumpWithUnloadedModules
                0100 MiniDumpWithProcessThreadData
                0200 MiniDumpWithPrivateReadWriteMemory
                0800 MiniDumpWithFullMemoryInfo
                1000 MiniDumpWithThreadInfo
                10000 MiniDumpWithPrivateWriteCopyMemory
                40000 MiniDumpWithTokenInformation

You’ll also probably notice that this dump also has handle data. But again, hang reporting does not duplicate the handle data from the original to the reflected process, so the only handle you should see is a process handle from the original process:

Stream 10: type HandleDataStream (12), size 00000038, RVA 08F35E0F
1 descriptors, header size is 16, descriptor size is 40
Handle(0000000000000008,"Process","")

Viewing the "Stitched" Target

There are at least two ways you can view the stitched target (using the same cab as an example):

Extract the files from the cab and open the *.dbgcfg.ini file instead of the individual dump files

C:\debuggers\cdb -z C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini

Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
Loading User-mode Dump Configuration File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini,C:,\temp\]
        default dump: [WER57FB.tmp.ref]
        memory dump: [iexplore.exe.6e79.wxhu.dmp]
        module list dump: [iexplore.exe.6e79.wxhu.dmp]

Loading Dump File [C:\temp\xproc\WER57FB.tmp.ref]
User Mini Dump File: Only registers, stack and portions of memory are available

Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Open the cab file directly in the debugger. Then open the dbgcfg.ini file from there

C:\debuggers\cdb –z C:\temp\xproc\sample.cab

Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Extracted C:\temp\xproc\000011a00_memory.hdmp from C:\temp\xproc\sample.cab

Loading Dump File [C:\temp\xproc\000011a00_memory.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

0:016> .opendump /c "iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini";g
Extracted C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini from C:\temp\xproc\sample.cab

Loading Dump File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]

Loading User-mode Dump Configuration File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
        default dump: [WER57FB.tmp.ref]
        memory dump: [iexplore.exe.6e79.wxhu.dmp]
        module list dump: [iexplore.exe.6e79.wxhu.dmp]

Loading Dump File [C:\temp\xproc\000011a00_WER57FB.tmp.ref]

Loading Dump File [C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp]
Just run !analyze. It uses the same procedure outlined in option 2, and it leaves the targets opened once it’s finished.

Last But Not Least!

Make sure you have the latest debugger or else it may not recognize this type of target. You can download it from here.

MDOP error reporting for small, midsize, and large companies

Kevin_Hill — Sun, 30 May 2010 03:45:34 GMT

It is time to do a blog post that points to some resources about the Microsoft Desktop Optimization Pack (MDOP), since we get a lot of questions about how Error Reporting can work better for small, mid-size, and large corporations. This is part of the Microsoft Software Assurance Program and provide many of the benefits of Windows Error Reporting that IT Professionals can customize. Part of the customizations enable IT Professionals to better analyze and respond to common problems within their organization.

A SpringBoard Series is available here: http://technet.microsoft.com/en-us/windows/bb899442.aspx

A great starting place to learn about all MDOP has to offer is here: http://technet.microsoft.com/en-us/windows/ff383366.aspx

For the Error Reporting Component (DEM) here are two links to videos that are worth watching:

The Official MDOP Blog

New response interface

Kevin_Hill — Fri, 19 Feb 2010 00:04:35 GMT

Today we are deploying an update the the response tools available on https://winqual.microsoft.com

1. The life of a response – Although the lifecycle of a response can vary depending on the scenarios below the general goal of a response is to lead to a solution for the problem that was identified by Windows Error Reporting (WER).

2. Selecting the appropriate response template – we provide templates that provide guidance for the most common scenarios when communicating with customers who have experienced a crash or hang on Windows. Below you will find a description of common scenarios and the corresponding template(s) that support these scenarios.

We have a template based system for choosing messages to show to end-users. The most common templates are shown and support the most common scenarios. If you have another scenario please comment on this blog post and we will review additional scenarios as they arise.

Common Response Scenarios and templates:

Scenario	Template
You no longer support a product – If there is a product no longer supported by your company these templates best communicate this to your customers.	Go to website for support info or uninstall [240] Beta software expired [37]
You have more information for your customer or need more information from a customer – Company has additional information for customers like KB articles for advanced users, or survey’s asking for feedback.	Link to KB [6582] Send info / feedback - link to co. survey page [230]
You have a specific fix – If you know that an issues is fixed in a future version or specific update these templates best communicate the fix to your customers.	Download the solution [217] Use included automatic update tool, link to instr. [92] Install Latest Version - link to co. [74]

Windows Error Reporting Terms of Use Update

Kevin_Hill — Thu, 14 Jan 2010 21:51:23 GMT

I wanted to give an update on things we are doing to make things a little easier for ISV’s, after receiving a lot of feedback and looking closely at our Terms of Use (TOU) agreement for Window Error Reporting we are making some updates. Once to document is live on https://winqual.microsoft.com you can review the differences but I have provided a high-level description below:

We are updating the Windows Error Reporting Terms of Use (TOU) to be less restrictive on some data retention guidelines for non Personally Identifiable Information (PII) data, and enabling a scenario where ISV’s can submit public symbols in order to get more pinpointed problem identification through windows error reporting.

The new TOU is scheduled to be available for signing on Feb 15^th 2010 through April 15^th 2010. The effective start date of this new agreement will be March 15^th 2010. There is a one month overlap to allow time for companies to review the changes.

Happy Debugging!

Advanced .NET Debugging

Kevin_Hill — Tue, 05 Jan 2010 21:56:08 GMT

If you have ever wondered where a good place to start with .NET debugging this book is it! Where as the previous Advanced Windows Debugging book had a single chapter on .NET debugging, this is an entire book dedicated to .NET debugging!

Advanced .NET Debugging is authored by Mario Hewardt, who also has authored: Advanced Windows Debugging.

The Advanced .NET Debugging book is focused on Managed code debugging for Windows. Chapter 8: Postmortem Debugging talks about how to leverage Windows Error Reporting (WER) (pages 409-438).

This book provides very good details on how to sign-up for an account to access WER data, as well as a good walk through of the existing Website and Web service available at https://winqual.microsoft.com.

Here is what some other people are saying about the book:

http://blogs.msdn.com/debuggingtoolbox/archive/2009/05/16/advanced-net-debugging-book-preview.aspx

http://dotnetdebug.net/2009/11/30/advanced-net-debugging-the-book-by-mario-hewardt/

http://www.informit.com/store/product.aspx?isbn=0321578899

http://blogs.msdn.com/tess/archive/2009/12/15/advanced-net-debugging-book.aspx

Microsoft PDC 2009

Kevin_Hill — Thu, 19 Nov 2009 18:21:49 GMT

Technorati Tags: pdc09,PDC,Microsoft,Microsoft PDC,PDC 2009,Windows Error Reporting,Watson,WER,Developer,Quality,Software,Development,Windows 7,VS 2010,.NET 4.0

I had a chance to present a lunch session at Microsoft PDC 2009 this year!

I wanted to call out some special thanks to Rajeev Acharya for helping to make this presentation and demo possible. Rajeev is an SDET on our team that developed this demo application in his “extra” time.

http://microsoftpdc.com/Sessions/CL33 – This talk starts with creating a better customer experience around software failures using the Application Restart and Recovery (ARR) API, to downloading and debugging mini-dumps send in from customers, to creating customer responses pointing to updates and fixes.

Steven Sinofsky highlighted the importance of Windows Error Reporting and how Microsoft uses this technology to build Windows in his keynote at PDC 2009

Some of his talk was summarized in this CNET article:

“Sinofsky is talking about the different mechanisms Microsoft uses from Windows Error Reporting, or Watson, to its Software Quality Monitor. Sinofsky notes that the monitoring tools require the user's permission in the final versions of Windows.”

http://news.cnet.com/8301-13860_3-10400476-56.html?part=rss&subj=news&tag=2547-1_3-0-5

Here is Sinofsky’s keynote: http://microsoftpdc.com/Sessions/KEY02 (00:15:32, starts with Telemetry)

Other blog entries:

http://blogs.pcmag.com/miller/2009/11/pdc09_sinofsky_talks_about_cre.php

http://blogs.msdn.com/e7/archive/2009/05/11/OurNextEngineeringMilestone2.aspx

!Analyze – Automatic Root Cause Analysis

Kevin_Hill — Tue, 29 Sep 2009 20:19:19 GMT

Meet the two engineers behind the !Analyze windows debugger extension!

http://channel9.msdn.com/posts/Charles/David-Grant-and-Ryan-Kivett-Analyze-Automatic-Root-Cause-Analysis/

!Analyze is an automatic root cause analysis tool for software failures. For years, it has provided insight to engineers both inside and outside of Microsoft. It is a key enabling technology behind numerous higher-level feedback systems, including Windows Error Reporting and Watson.
!Analyze runs millions of times each day, producing actionable results from reliability telemetry data sent to Microsoft. Ordinary debugging tools report the file and function where a failure ended. !Analyze pinpoints where the failure started.
How does it work, exactly? What's the story behind !Analyze?
Meet two of the Software Developers behind !Analyze, David Grant and Ryan Kivett. They share with us how !Analyze works, it's history and provide a glimpse into it's potential future.Tune in.

The Three Cs of Response Satisfaction

MSDNArchive — Wed, 12 Aug 2009 21:00:00 GMT

The Windows Error Reporting (WER) platform offers software and hardware companies a way to provide helpful information to customers. When an application stops working the WER client application that runs on Windows (since XP) detects these events and checks to see if a solution exists. Software and hardware companies are able to create WER responses through the Windows Quality web portal (https://winqual.microsoft.com)

Windows end-users also have the ability to provide feedback on the quality of the solutions. Through analyzing the feedback provided over the years and performing extensive usability studies we have found some simple rules that help to create a pleasant experience for end-users. The 3 C’s are comprised of keeping Customer focus, providing Clear information, and keeping your information Current.

Customer focus – Keep the customer’s experience paramount in creating a response for your customers. Understanding that the customer sees messages first on the Windows operating system is important in creating a seamless experience.

Do:

· Create custom content that guides the customer

· Acknowledge that the customer was referred by Windows Error Reporting

· Use clear language and common terms

Don’t:

· Put the customer on your main homepage

· Send the customer to a general support forum

Clear information – Ensure that information provides clear and concise steps for the end-user.

Do:

· Provide a direct download link to an update from the response message when possible

· Provide a customized and dedicated landing page on your company’s website that contains an easily identifiable link to download a fix

· Provide concise steps if user action is needed during an install

· Provide clear instructions if a work around is available

Don’t:

· Provide misleading links

· Make the user click on three or more options before they reach an actionable solution

· Provide non-actionable information

Current information – Keep your responses and landing pages up-to-date! We regularly review response quality and make decisions to un-publish responses based on current information and customer ratings.

Do:

· Check links on the landing pages after website updates

· Monitor the Response Satisfaction report available on the Developer Portal

Don’t:

· Let content sit for months unchanged

If you have questions about creating a good response, you are welcome to use the contact information available on the WER portal. We are always happy to help.

Windows Error Reporting – Vote for a fix!

Kevin_Hill — Thu, 14 May 2009 00:38:53 GMT

If you are a developer and reading this blog you may already know of the direct benefits provided by users submitting error reports, if you are an end-user of an application, on Windows, developers are relying on you to help identify solutions to issues that are difficult to reproduce.

I want to take this short blog entry on what good Windows Error Reporting brings to the Windows Operating System and eco-system. I am including hyperlinks that give a view from 3rd parties on the good things that Windows Error Reporting enables for developers on the Windows OS.

The title of this blog posting was inspired by the WindowsITPro.com article below where each error report sent to Windows Error Reporting (WER) is a vote for a fix, although the article was written a few years ago, the theme of voting remains the same. Although not like texting a vote into American Idol, I argue that sending a Windows Error Report is easier and can end up fixing the issue you reported in the end!

Kate details how using Windows Error Reporting to solve problems is helpful on Vista.

http://www.gregcons.com/KateBlog/WindowsErrorReportingIsYourFriend.aspx

O’Reilly gives an overview:

http://windowsdevcenter.com/pub/a/windows/2004/03/16/wer.html

Bob Kelly has a post talking about his recommendations:

http://www.realtime-vista.com/administration/2007/03/windows_error_reporting_option.htm

WindowsITPro.com article highlights how each error report you send in is like a vote for a fix:

http://windowsitpro.com/article/articleid/46982/windows-error-reporting-elementary-my-dear-watson.html

An article talking about how WER has helped to build Windows 7:

http://news.softpedia.com/news/Windows-7-Windows-Error-Reporting-Infrastructure-103252.shtml

Technorati Tags: WER,Windows,Error,Reports,Windows Error Reporting,Fix,Developer

Vista WER Settings UI Deciphered

varunb — Thu, 16 Apr 2009 00:53:38 GMT

This post is a follow up to the original post I did about WER Settings & UI for Windows 7. While that post outlined the settings UI in Windows 7, this post covers the WER settings UI for Windows Vista.

WER Consent Settings: The various WER consent levels are identical across Windows Vista and Windows 7. Please review this post to get an understanding of the various consent levels first. The rest of this blog post explains how these consent options are exposed in the Windows Vista UI.

The WER setting in Windows Vista is exposed in the ‘Problem Reports & Solutions’ (aka PRS) Control Panel.

Clicking the ‘Change settings’ link in PRS, opens up the basic settings page shown below. These settings allow configuration of WER consent settings for that specific logged in user only.

Clicking on the ‘Advanced Settings’ link on this page opens the, well, advanced settings page shown below.

Clicking on the lower ‘Change Settings’ button, brings up a dialog which allows setting consent for all users on the machine. Here is how that dialog looks:

Hope this clarifies the various WER settings and how they can be configured on Windows Vista.

Problem Steps Recorder (PSR.exe) + Windows Error Reporting = Another tool to help find solutions to software defects

Kevin_Hill — Mon, 30 Mar 2009 22:24:00 GMT

Technorati Tags: PSR,WER,WER Services,Problem Steps Recorder,Repro Steps

There is a bunch of information on the web about PSR and one of the best is this video on CNET: http://cnettv.cnet.com/2001-1_53-50005144.html as you can see a demo of the tool working for an end-user perspective, it behaves differently when invoked by WER as described below. The key difference is that WER doesn’t collect screenshots.

From an error reporting perspective we plan on exposing this via the WER Services. The concept is that if you aren’t able to use standard mini-dumps or heap dumps to find the source of the problem, WER Services will allow the PSR tool to be run (with user consent)

Example of screen-shot shown to user:

When we enable an error Bucket to collect PSR data, here is the workflow:

User prompted with a request for additional data
PSR waits until next time the same application is launched
PSR runs silently in the background while user is using the application
PSR captures descriptions of steps user has taken (without screenshots)
IF the same unhandled exception occurs (BucketID)

The Memory dump, Additional Files, and the PSR output is sent in a CAB file through WER
Developer’s can now download CAB files associated with a particular event that contain these steps a user took leading up to the crash
Developer’s can try to reproduce the error locally following these steps.

After PSR steps are sent with CAB the PSR tool is disabled for user

IMPORTANT NOTES:

PSR.exe when invoked by WER does NOT capture screenshots, see example output below.
Only CAB files that contain PSR steps will be submitted to WER, this means that possibly fewer cabs will be sent for a particular Bucket.
When Windows 7 firsts launches this feature will only be manually enabled by the team behind wer@microsoft.com.

Other links and blog posts:

http://technet.microsoft.com/en-us/magazine/dd464813.aspx

http://blogs.msdn.com/cjacks/archive/2009/02/25/deciphering-the-command-line-configuration-of-the-windows-7-problem-steps-recorder.aspx

http://blogs.msdn.com/tims/archive/2009/01/12/the-bumper-list-of-windows-7-secrets.aspx

Sample output:

Problem Step 1: (3/27/2009 7:53:51 PM) User right click on "user consent (editable text)" in "Problem Steps Recorder (PSR.exe) + Windows Error Reporting = Another tool to help find solutions to software defects - Windows Live Writer"

Problem Step 2: (3/27/2009 7:53:53 PM) User left click on "Hyperlink... (menu item)"

Problem Step 3: (3/27/2009 7:53:55 PM) User keyboard input in "Insert Hyperlink" [... Enter]

Let There Be Hangs: Part 4 – Hashes and Type Codes and XProc, Oh My!

ryankiv — Thu, 19 Mar 2009 22:51:00 GMT

Hang Bucketing, A Better Way

In the previous post I gave a brief introduction of how the first version of hang reporting was implemented using the existing crash reporting infrastructure.

Eventually (after Windows XP shipped) a new general purpose event reporting and bucketing mechanism was built. In a nutshell, this mechanism provides a very flexible way to report incidents to the WER pipeline. It supports a custom named event (internally at Microsoft dubbed “generic events”) and up to 10 custom bucketing parameters P1...Pn...P10.

During the development of Windows Vista, we decided to do something about the problematic hang bucketing and leverage the new general purpose reporting to gain a much better client-side classification of hang issues. Before I explain what bucketing parameters these have, let me discuss a few other new things in Vista:

Another problem with Windows XP hang reporting was that the application process was often hung waiting on an external process. The report would only include a memory dump of the hung application and developers were often forced to give up not able to debug the other process. (Now, granted, you very often don't need to debug the other process to fix a hang, but that is a discussion for a later blog post.)

To solve this problem, the GetThreadWaitChain API was created. This API allows the caller to discover the blocking graph (or “wait chain”) for a given thread. A trivial example: Thread A is waiting to enter a critical section owned by Thread B which is making a blocking SendMessage call to Thread C which is waiting on a mutex owned by Thread D which is running in another process. The output from the API provides the caller all of this information.

It's this mechanism that hang reporting uses in Vista to discover not only the wait chain information but to collect memory dumps of external processes.

Back to the bucketing parameters – the hang reporting infrastructure actually reports hangs to two different generic events: AppHangB1 and AppHangXProcB1 (yes, B1 stands for Beta 1. Don’t ask…). On Winqual, AppHangB1 is shown as Event Type "Hang" and AppHangXProcB1 is shown as "Hang XProc". AppHangB1 has 5 parameters (by convention P1 & P2 are typically Application Name & Version):

P1 - Application Name
P2 - Application Version
P3 - Application Timestamp
P4 - Stack Hash
P5 - Type Code

AppHangXProcB1 has those 5 and adds 2 more:

P6 - Waiting On Application Name
P7 - Waiting On Application Version

Most of these parameters are self explanatory. The two that might not be are P4 and P5:

Stack Hash (P4) - hang reporting traverses the wait chain and for the final thread in the chain before it jumps to a different process, we generate a hash based on the thread’s stack back trace. P4 is a restricted hash which means we chop the MD5 128-bit hash down to 16 bits. This prevents any wild spraying of buckets and when we studied it we were still achieving ~85% uniqueness.

Type Code (P5) - this is a bit field based in part on the WCT_OBJECT_TYPEs found during wait chain traversal with GetThreadWaitChain (e.g., mutex, COM, etc.) but also on a few other items too - like if there's a deadlock or if the hang report came from End Task in Task Manager (more blog fodder).

As rich and wonderful as hang bucketing has become in Windows Vista, there are still edge cases (just as there are in crash bucketing) where a bucket does not uniquely identify a single bug. In a future post I will discuss bucketing more generally and how we REALLY determine and quantify bugs using WER data at Microsoft.

Let There Be Hangs: Part 2 – WER History 101

ryankiv — Thu, 19 Mar 2009 22:50:00 GMT

Crashes Suck

In the beginning, we needed a way to close the loop with our customers in order to ease the pain felt from software defects (bugs) that caused crashes. A simple client service was produced that collected crash dumps from Windows desktops and sent them back to Microsoft for analysis. The crash dumps were debugged, bugs exposed, fixes were made, problems went away. Lather, rinse, repeat. Note: This is how user mode error reporting began. Interestingly, kernel mode error reporting (bluescreens) had its origin in a roughly parallel timeframe (actually a little earlier) but it began in a completely different group at Microsoft. For the rest of this series of posts I’m talking about the user mode side of WER.

Buckets

An early design assumption was that this client had zero analysis ability and so we needed a simple and deterministic way to organize the data that would allow both the client and the backend servers at Microsoft to speak to each other in a common language. Oh, and the solution needed to be (*ahem*) scalable. That common language is the unique combination of event parameters which give us the Event ID. Internally at Microsoft these are actually known as bucket IDs – or just buckets. It’s more natural for me to talk about buckets and I’ll try to be consistent, but just in case, know that “bucket” and “event” are mostly interchangeable in the context of error reporting.

Originally, plain old crashes were bucketed using 5 parameters:

The application’s name
The application’s version
The name of the module containing the instruction that causes the crash exception
The version of that module
The byte offset into that module where that instruction resides

The idea was that crashes with the same set of parameters were caused by the same bug. It turns out that this isn’t always true (We’ll save that for a later post) but in general it works pretty well. Today, this “client” is built as a core component of Windows and became generally referred to Windows Error Reporting (WER) Services. In Vista we added 3 more parameters to plain old crash reporting: file link timestamps for both the application and modules (to try and avoid some issues of naming collisions) and exception code. And by the way, the client itself has become smarter and in fact does some cursory analysis when deriving bucketing parameters.

Notice that you can tell the explorer crash to the right is from Windows XP because those extra 3 parameters don’t have data.

< Read Part 1 [Part 2] Read Part 3 >

Let There Be Hangs: Part 3 – The ‘hungapp’ module

ryankiv — Thu, 19 Mar 2009 22:50:00 GMT

Hang Bucketing, v1

On Windows XP, hangs have it rough. Like a younger sibling, error reporting for hangs has to wear the hand-me-down clothes of crash reporting – it piggybacks on the same 5 fixed bucketing parameters used by crash reporting. However with a hang there is no exception context and so there is no faulting instruction, therefore there is no module name, module version or module offset. So on XP, hangs really only have 2 effective bucketing parameters (application name and application version). The other parameters are set where module name is "hungapp", module version is "0.0.0.0", and the offset is 0.

This means all of particular version of an application’s hangs ended up in a single bucket. Knowing this, you might correctly guess that these buckets have very high hit counts when compared with crash buckets for the same app name/version (though some corruption buckets are not far behind). When someone runs a generalized query similar to, "what are my highest hitting XP buckets this week?", the top end of the results are often littered with hang buckets. A naive conclusion would be "Oh my! Hangs are a huge problem and we must focus all our effort to eradicate them!" While hangs are indeed a big problem, arriving at this conclusion based on bucket hit volume is incorrect. Also, when the actual failures contained in these hang buckets are studied, it is quickly apparent they are composing many bugs – not just the ideal 1 (or few). This is a big reason development teams historically struggled to make progress with hang bugs - the data was difficult to sort and measure.

< Read Part 2 [Part 3] Read Part 4 >

Let There Be Hangs: Part 1 (Not Responding)

ryankiv — Thu, 19 Mar 2009 22:43:00 GMT

Technorati Tags: application,hang,application hang,windows,hung,xproc,hang XProc

Well hang reports anyway…

Those of you signed up on the Winqual site are likely familiar with the "Event ID" and what it (ideally) represents – that is, a basic demarcation of unique software defects. Additionally you’re probably more familiar with crash-related Event IDs like “Crash 32-bit” and “Crash 64-bit”. But what about the other event types? Specifically “Hang” and “Hang XProc”? And by the way – what’s a crash in module “hungapp”?

In this and later posts I’m going to explain a little bit about error reporting and a little bit more about hangs – what they are, how they’re detected, reported, organized, counted, etc. Before getting into all that, however, we need a (very) brief history lesson.

Ghost Windows

Some of you might remember something like the following scene from the days of Windows NT 4 (and maybe you’ve seen similar scenes in more recent times):

This is a screenshot of CALC.EXE moved across an intentionally hung instance of NOTEPAD.EXE.

Without getting into too much nitty gritty, a thread that creates GUI elements, namely windows, has an implicit contract with the desktop window manager that it will service messages that arrive in its message queue... in a timely fashion. “Servicing” means retrieving and dispatching the messages (aka pumping messages). There is plenty of content on MSDN that covers the specific APIs and mechanisms used for this and plenty of content elsewhere about the interesting peculiarities of this area. What "timely fashion" means is somewhat a matter of debate among developers and users. To the window manager, it means 5 seconds by default. Our user research shows that this is actually quite generous.

If a thread stops pumping messages from its message queue, bad things start to happen. In such cases, the thread is often off busy working on whatever task the user just “asked” it to do (e.g. opening a file or recalculating some total or talking to a web service over the internet) and there’s a good chance that it didn’t update its UI to indicate that it’s off doing this work; it almost certainly hasn’t shown any UI resulting from that work. But there is a worse and more fundamental problem – while it’s away, the messages it isn’t pumping could be mouse, keyboard, or touch input along with paint messages. Not pumping those messages would result in a window stuck on the user’s screen – the user wouldn’t be able to move, minimize, or close it and if they moved something in front of it, they’d get something like the jaggy mess in the screen shot above.

I say would because beginning with Windows 2000, we added functionality to the window manager to handle these cases. The concept is actually quite simple – the window manager watches for when a thread has pending input in its queue but hasn’t serviced it for more than 5 seconds. When the situation is detected (aka IsHungAppWindow), the window manager does a presto chango maneuver whereby it hides the window whos thread isn’t pumping input messages and seamlessly shows a replacement window (known as a “ghost” window) with its client area filled with the real window’s last known good client area bits.

While the original thread isn’t responding, the window manager manages both windows in parallel, to the extent that the application doesn’t know this is happening. For example, IsWindowVisible() will actually return true for the hidden/unresponsive window. The only visual difference (initially) to the user is that the text “(Not Responding)” is appended to the ghost window’s title. Using the ghost window, the user can effectively move, minimize, and even close the unresponsive application window. When (if) the thread starts pumping messages again, the original window is re-shown and the ghost window goes away. By the way, this window manager feature is called (obviously) window ghosting - and naturally, we offer an API you should almost never use: DisableProcessWindowsGhosting.

While we were working on Windows XP, we realized that it might be nice to know about these unresponsive applications. We had already started to collect data about crashes, which were clearly annoying to customers, and some schools of thought (and some research) indicated that unresponsive UI was even more annoying -some might even use words like “infuriating” (I know I do). So, we decided to wire up a ghost window’s close button to the infrastructure that sent crash reports back to Microsoft… and voila!, hang reporting was born.

[Part 1] Read part 2 >

FAQ

MSDNArchive — Mon, 16 Mar 2009 19:54:00 GMT

How do I get started with Windows Error Reporting (WER)?

How do I find Windows Error Reporting information on a PC?

How do we match up crash data (Windows Error Reporting signatures) with companies?

What files should I map?

How do we detect applications hangs in Windows?

What is a bucket?

What is a “special” exception?

What are the different types of memory dumps?

When will I see data?

Q: How do I get started with Windows Error Reporting (WER)?

A: As an Independent Software Vendor (ISV) who develops software applications for the Windows Operating Systems, you are able to register to receive Software Quality information from the Windows Quality website.

Please follow this link to a page with directions on how to get started! (http://www.microsoft.com/whdc/maintain/StartWER.mspx).

Q: How Do I find Windows Error Reporting information on a PC?

A: Here are the solutions for finding WER data on XP, Vista, and Windows 7.

OS	WER Information
Windows XP	Event Log
Windows Server 2003	Event Log
Windows Vista	Event Log ”wercon”
Windows Server 2008	Event Log ”wercon”
Windows 7	Event Log Action Center

Windows XP/ Server 2003

On XP / Server 2003 you can find information about error reports in the Application Log.
When a crash occurs the bucketing parameters are written to the Application Event log with the event source of “Application Error” and event ID 1000.
When the crash is submitted the bucket ID is written to the Application Log with the same event source and has an event ID 1001.

Windows Vista / Server 2008

On Vista you can use the “wercon” control panel accessible via:
Start >Control Panel >System and Maintenance >Problem Reports and Solutions >View Problem History
Or
By typing “wercon” in the Search Box at the base of the Start Menu.

Click on the “View problem history” link

Double click on on a line item below the Application Name to view details:

Notice the bucketID and the Copy to Clipboard feature.

Windows 7

Using the Action Center:

Open the Action Center by clicking on the flag icon in the system tray

Select the Action Icon from the system tray, and click Open Action Center.

By expanding the Maintenance toggle button, the customer can see the View System History link.

After selecting the View System History link, the customer can click the View problem reports and responses link.

Or, you can type error into the Search bar on the Start Menu and click the “View problem History”

Select View Problem History, and then select View problem reports and responses.

Q: How do we match up crash data?

A: Here is the general mapping logic we used based on crash data that is gathered by Windows Error Reporting (WER) client. Notice the difference between Pre Windows Vista, and Post Windows Vista.

How do we match up crash signatures with Files that are mapped?

We use 4-6 basic parameters to match mapped files in the developer portal, Windows XP has fewer and matches on File Name and File Version for either Applications and Modules. With this in mind if you are still developing for Windows XP it is a best practice to come up with a unique file name and file version. This best practice translates to any other development environment, but is important with Windows XP as you can see the parameters below that the Windows XP WER Client sends to our service.

Windows XP (Includes Windows Server 2003)

WER client sends these parameters we use to match crashes:

Application Name
Application Version

Module Name
Module Version

With Windows XP/Windows Server 2003 crashes we have less data to use for matching crash signatures with files mapped on the portal.

Windows Vista++ (includes Windows Server 2008)

WER client sends these parameters we use to match crashes:

Application Name
Application Version
Application File Link Date

Module Name
Module Version
Module File Link Date

We use the link date to get a more accurate match on the files mapped on the portal.

Q: What files should I map

A: Given the way we have implemented mapping logic the best policy is to map only the files where you own or maintain the source code. If you ship 3rd party or open source modules with your software where you do not own or maintain source then you will not be able to take action on these error reports, and this information will just be added noise and complexity.

Even if you don’t map 3rd party modules you will still see crash reports where any module crashed your application.

NOTE: We have mapping collision detection that will preclude you from mapping certain files.

Example:

Product Name: FabrikamReportingApplication

Product Binary files:

File Name	File Version	Description	Should I map?
FabrikamReportGUI.exe	1.0.0.0	User interface for my reporting product	Yes
FabrikamWebserviceDataImport.dll	1.0.0.1	Library used to import and transform data from web services.	Yes
ThirdPartyChartControl.dll	5.6.1.3	3rd party DLL I’ve licensed to ship with my report product	No
OpenSourceDataVisualizer.exe	4.5.3.1	An open source application that helps with visualizing multiple data sources (I don’t maintain code for this)	No
OpenSourceGridControl.dll	2.3.1.4	An open source library where I’ve incorporated the source code into my build and maintain changes and updates for this source.	Yes

With the mapping scenario above you will still see crashes where ThirdPartyChartControl.dll caused a crash in the FabrikamReportGUI.exe application.

Q: What is a bucket?

A: A bucket uses a set of parameters gathered by the WER client to describe a WER Event. For crash events the bucketing parameters are Application Name, Application Version, Application Build Date, Module Name, Module Version, Module Build Date, Exception Code, and Code Offset.

There are also Generic Buckets that are made up of up to 10 parameters.

Q: What is “special” exception?

A: A special exception is a grouping of two WER event types: Buffer Overruns (BEX) where the exception code = C0000409, and Data Execution Prevention (DEP) where the exception code = C0000005. These two event types are related to security of software and can also be viewed in the Security Alerts section of the website. In order for the WER client to capture BEX events you need to use the /GS command when compiling your software.

Q: What are the different types of memory dumps?

A: Below is a table that contains brief descriptions of the different types of dump files that can appear in .CAB file.

Extension	Description
.mdmp	minidump (Default dump collection type, stacks, and loaded modules)
.hdmp	.mdmp += heap data
.kdmp	on-demand kernel mode minidump (snapshot of the kernel mode portion of a given user mode process) only sent if WCT (Wait Chain Traversal) added it, or was explicitly requested for a given ibucket by a user.
memory.dmp	Old style (<=XPSP2) heap dump. This is not actually a debugging dump file, instead its just a blob of read only pages
atk.dmp	application termination kernel (taken when an exited process takes more than 10 seconds to actually terminate)
hu.dmp	heap user
mu.dmp	mini user
odk.dmp	on demand kernel
axhu.dmp	arbitrary x-process heap user
axmu.dmp	arbitrary x-process mini user
wxhu.dmp	wct x-process heap user (wct is wait chain traversal)
wxmu.dmp	wct x-process mini user

Q: When will I see data?

A: After mapping files we will start to pull any existing data we have for crashes. Generally you will see Hit data within a day or so. Users who setup Vista and Windows 7 with recommended settings (~80%) will automatically send us the hit data without any prompting, so you can see hits without cabs.

By default we collect 10 cab (minidump) files per event*, since a user still has to click on the “send additional information” button for us to get the crash dump it can take some time to get cab files.

Once we receive cab files for an event you will generally be able to see these cabs within a few hours of us receiving them.

NOTE: For newly detected crashes it can take more than 4 days to get the crashes processed and up on the site. Commonly if you create an app to test our system you will see a longer latency in getting WER data. Since WER is collecting unhandled exceptions on Windows all the time, we recommend mapping existing released software.

*We do not set collection rules for files that are un-versioned, and automatic collection rules depend on the time you map your files. If you are mapping old files, all the buckets have been created and we will not automatically collect files for old buckets. For newly created buckets we will set the collection rules for the most common file extensions.

An overview of WER consent settings and corresponding UI behavior

varunb — Wed, 11 Mar 2009 21:52:00 GMT

This post gives a high level overview of the various consent settings WER supports and the corresponding experience a user has when opted into any of those setting. Before going into further details, I want to clarify a couple of terms used here:

WER Client – The binaries that ship with the Windows OS and report problems to Microsoft.
WER Service – The backend service hosted by Microsoft to which the WER client reports problems.

Windows users have three different levels to choose from when opting into reporting errors to Microsoft. these levels are:

Level 1 – If a user chooses this option, Windows does not report problems to Microsoft automatically. Instead, every time a fault (like a crash or hang) occurs on the user’s machine, a dialog is presented to the user, to get explicit consent to report the fault to Microsoft.
Level 2 (Is the default recommended by Windows - more details about defaults later in the post) – For users choosing this option, Windows automatically reports the problem signature (aka WER event parameters) to the Microsoft WER Service. However, if the backend WER Service requests for additional information from the machine (like mini dumps, log files), a dialog is presented to the user to get explicit consent before uploading the additional information.
Level 3 – For users choosing this option, Windows automatically reports the WER event parameters to the backend WER Service, and if the WER Service requests for additional information that does not contain any personally identifiable information (PII), then that information is uploaded to the service automatically. An example of such information is a mini dump, which has an extremely low probability of containing PII. However, if the information requested by the WER Service has the potential to contain PII (for example, a heap dump), a dialog is presented to the user to get explicit consent before uploading the additional information.

Note that the three consent levels only come into play if WER is enabled for the user/machine. If WER is disabled, then the user is never even prompted to report problems.

Following is a summary of the typical Windows 7 user experience for each of the consent levels (Note: Windows 7 is still in Beta and the UI is still susceptible to changes):

User experience when WER is disabled: Error reporting does not kick in. A simple dialog informing the user about the crash is shown.

User experience when WER consent level is set to 1:

     Step 1: On detecting a crash, WER client presents UI to get explicit user consent to report WER event parameters to Microsoft’s WER backend service



      Step 2: On getting consent in step 1, WER client starts reporting parameters to Microsoft



      Step 3: If the WER Service requests additional data (like heap dumps, log files etc.), WER client starts collecting the information.



      Step 4: After completing data collection, WER client asks user for explicit consent to upload data to Microsoft. If user agrees, WER client uploads the data. If user clicks ‘Cancel’, the data is discarded and not sent to Microsoft.

User experience when consent level is set to 2:

Step 1: On detecting a crash, WER client automatically starts reporting WER event parameters to Microsoft’s WER Service.

Step 2: If the WER Service requests additional data (like mini dump, heap dumps, log files etc.), WER client starts collecting the information.

Step 3: After completing data collection, WER client asks user for explicit consent to upload data to Microsoft. If user agrees, WER client uploads the data.

User experience when consent level is set to 3:

Step 1: On detecting a crash, WER client automatically starts reporting WER event parameters to Microsoft’s WER backend service.

Step 2: If the WER Service requests additional data (like mini dump, heap dumps, log files etc.), WER client starts collecting the information

Step 3: If the data collected is considered safe (i.e. not containing PII; like a minidump) WER client uploads that data automatically without asking for explicit user consent. If however, the data isn't considered safe (like heap dumps, log files etc.), WER client asks user for explicit consent to upload data to the WER Service. If user clicks ‘Send Information’, WER uploads the data.

Note: It is noteworthy that WER client never uploads data containing personally identifiable information (like heap dumps) automatically. Explicit user consent it always needed to upload such information. This design decision has been made to respect the privacy of Windows users.

So how does a user configure/change WER settings?

The first place where a user chooses to opt into WER is the Windows Out of the Box Experience (aka Windows OOBE). The first time a new Windows 7 machine boots up, an OOBE screen is presented to the user which opts the machine into one of two WER consent levels depending on the user’s selection.

If the user does not choose “recommended settings” in OOBE, another “up-sell” dialog is presented to the user in the Windows Troubleshooting Control Panel

In the above dialog, if the user clicks ‘Yes’, his consent is bumped up to level 2. Note however, if the user has already set his consent level to higher than 2, it is not brought down.

The OOBE and the up-sell dialog are presented to the user only once in the entire lifetime of his Windows 7 installation. If the user wants to change WER settings there after, he can do so by accessing the settings page via the Action Center Control Panel

This page allows setting the consent only for the specific user interacting with this page. However, if an administrator wants to configure settings for all users on a machine, he can do so by clicking on “Change report settings for all users” and making the same choice in that dialog.

Hope this information helps you understand the various WER settings, the corresponding Windows UI behavior for each of the settings, and the exactly how to configure/edit these settings.

Happy Error Reporting!

Windows 7 Beta Data – Windows 7 Release Candidate Data

Kevin_Hill — Fri, 13 Feb 2009 18:27:14 GMT

Technorati Tags: Windows 7 Beta,Windows 7 RC,Windows 7 Release Candidate,Windows 7,Crash Data

Today we are releasing a report showing the companies Windows 7 beta crash data, and now the Windows 7 Release Candidate crash data! This reports’ focus is to help companies see how well their applications are performing on the Windows 7 beta. The report has two section, one highlights the top user-mode issues and the other highlights the top kernel-mode issues. The data between the two of these sections are slightly different I will explain what each data point is trying to communicate.

Navigate to: Windows Error Reporting > Software > My Reports

And select the Windows7BetaDataFailures Report. (Being retired soon).

Select the Windows7_RC_Failures report

Note: Download !analyze to help in debugging the problems shown in this report.

User-mode: What is a user-mode failure?

A failure is chosen based on analysis of the cab by !analyze, a debugger extension that is part of the Debugging Tools For Windows. A Failure is a unique combination of Symbol, Problem Class and Exception code. The failure may aggregate multiple Buckets (EventID’s). The results of the analysis can be dependent on the availability of symbols (PDB files) for the modules related to the failure.

Note: Since we Microsoft does not have private symbols for debugging memory dump files failures are based on the best available analysis of memory dump files. In order to get the most accurate depiction of what the failure is using private symbols in debugging will quickly show where the precise origin of a failure occurs.

Rank	Shows the overall rank of this particular failure across all of Windows 7 within the specific Event Type listed below.
Product Name	Shows the name of the product detected in this failure.
Symbol	Shows the symbol (Module/Function) that caused the failure detected via !analyze.
Problem Class	Shows a general description of the nature of the problem.
Exception Code	Shows the OS Exception Code.
Regression Percentage	Shows the percentage of time this failure has occurred on Windows 7 verses previous versions of Windows. (100% Regression percentage = Detected only on Windows 7)
Module Version Range	Shows the versions of a file where this failure was detected via !analyze.
View Cabs	A list of cabs that contain memory dumps related to this specific failure.

Kernel-mode: What is a kernel-mode bucket?

In kernel mode, the debugger walks the stack of the crashing thread and determines where to place the blame for the crash. Typically, a crash bucket name is derived from the debugger's choice of bugcheck stopcode + the blamed driver + the function name (if any, symbols are needed for this) + the function offset. All crashes assigned to a given bucket name can usually be considered as failing for the same reason.

Rank	Shows where the crash bucket ranks in the general Win7 crash population
Product Name	Shows Product detected in this hardware bucket, where available
Bucket Name	Combination of bugcheck stopcode, failing driver, failing function, and function offset. For example: 0xD1_tdx!TdxEventReceiveConnection+2d3
File Name	Name of the driver chosen as the cause of the crash via !analyze
Version	Shows the version of the file that was detected via !analyze
Hits	Shows the number of hits the crash bucket has seen in the past 30 days
Regression	Shows if this bucket exists only in Windows 7 or also on Previous versions (True = Detected only on Windows 7)
Driver Version Range	Shows the range of versions where this particular bucket has been detected.
View Cabs	Shows a list of cabs that contain memory dumps related to this specific kernel bucket.

Kernel dump storage and clean up behavior in Windows 7

varunb — Tue, 10 Feb 2009 01:25:00 GMT

Technorati Tags: WER,windows,error,reporting,kernel,dump,collection,cleanup,minidump,mini,heap,memory,windows7

Lately we have been getting several questions about the kernel memory dump storage and clean up behavior in Windows 7, so I figured it will be a good idea to evangelize the design.

A kernel dump gets created every time a machine has a kernel fault. This dump gets stored in the %systemroot% folder as a memory.dmp file. Several options relating to collection of kernel dumps are configurable by going into

Control Panel > System and Security > System > Advanced system settings > Advanced tab > Startup and Recovery Settings.

The file size of a kernel dump is typically several hundred mega bytes. This brings us to the feedback we received regarding the Vista dump storage and clean up behavior. Customer feedback provided by home PC users indicates kernel dumps, although valuable in debugging crashes, takes up too much disk space (eating up to 200MB), which is not always appreciated by users. Thus, we have to make a judicious tradeoff between keeping a kernel dump on the machine so that the user could provide it to her support professional for debugging vs. freeing up valuable memory space on her disk. We realize that the scenarios are very different for corporate vs. home users.

IT departments usually manage corporate machines. IT departments know best about what they want to keep on their users’ machines. They typically store the memory.dmp files to debug problems and provide fixes or workarounds. Thus for the corporate scenario (or machines that are joined to a domain), Windows 7 will continue to store the memory.dmp file on disk, unless the IT department feels otherwise and overrides the default settings via group policy.

Home users are much less likely to contact product support for a system crash that occurs once in a blue moon. Thus, the value of the dump files is relatively low for them. If given a choice between having their disk space back vs. keeping the dump file on disk, they would rather have their disk space back. For these users, we do not want to keep the dump file if they are running low on disk. Needless to say they should be allowed to override this behavior and always keep the dump if they choose to.

Given the above thought process, we have come up with the following logic to save/delete the memory.dmp file on a machine:

Algorithm to decide whether to store/delete memory.dmp file generated after a system crash:

First report the kernel fault to the Online Crash Analysis Service.
Then, if machine has a registry setting AlwaysKeepMemoryDump set to 1, store the dump file on disk.
Else, if machine is a Windows Server SKU, store the dump file on disk.
Else, if the machine is joined to a domain (i.e. this is a corporate machine), store the dump file on disk.
Else, if machine is not on a domain (i.e. this is a home user’s machine),
- If free disk space is >= 25GB, store the dump file on disk.
- Else (free disk space is < 25 GB), delete the dump file.

The exact location of the AlwaysKeepMemoryDump setting in the registry is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl ! AlwaysKeepMemoryDump Type: REG_DWORD.

Note that any time the memory.dmp file is deleted, a system event is also logged to make sure the deletion is tracked. This can be useful for product support professional who want to confirm that the memory.dmp was deleted as per design, or is just mysteriously missing. Here is a screenshot of how this event (Event ID: 1018, Source: BugCheck) looks in the event viewer:

While we are clearing up the algorithm for kernel dumps, let us also provide you the new logic used to store kernel minidumps on disk:

Algorithm to store kernel minidumps in the %windir%\Minidump location:

WER will keep the last ‘x’ minidumps in the %windir%\Minidump location. ‘x’ can be configured by the following registry setting:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl ! MinidumpsCount Type: REG_DWORD, default value is 50

Hope this clears up any questions or concerns about dump storage and cleanup behavior in Windows 7. Note that the behavior in the Beta build does not match this. The RC and RTM builds will have this behavior though.

Happy debugging!