Premier Field Engineering Developer Blog

Top 5 Reasons Your Windows Store App Fails Store Certification

2013-05-06T13:50:00Z

Summary: Whether you're trying to get your Windows Store App certified in the Windows Store and running into blockers, or simply writing a new app for Windows 8 that you expect to publish via the Windows Store, check out the great insight provided by Robert Evans and Christophe Nasarre (two Microsoft Premier Field Engineers) in this article on the primary culprits you might run into when it comes to the certification process. They are technical leads for Windows Store App Labs and they provide a multitude of tips and pointers to some great resources. If that’s not enough, they also have the following video to walk you through some of the details. Watch the video, read the post, be informed!

When submitting your app to the Windows Store (http://www.windowsstore.com) you may encounter issues with Store Certification. These are the top 5 most common Store Certification blockers that we’ve seen in our Windows Store App Labs – and how you can fix them. Let’s count-down in reverse order:

Number 5: Your App Should be Fully Functional

Certification requirement 1.2: “App must be fully functional” (also similar failure is related to certification requirement 3.2 “Your app must not stop responding, end unexpectedly, or contain programming errors”).

If you are failing on this certification requirement there are some things you should check:

Have you tested your App Snapped? Are there overlapping icons in your app bar while snapped?

Does the charms menu cover up any critical commands?

If you’ve declared your app as supporting portrait and landscape – does it?

Does your submitted description talk about features you haven’t implemented yet? Does your app crash at any point?

If the app plays background audio does it implement play, pause, and play/pause events to enable users to control audio playback?

If your app has “login” functionality did you create a test user account and provide the username/password in the “Notes to Testers” field when you submit your application?

Does the app work on all the architectures such as ARM that you checked off and claim to support when you submitted your app?

Have you run the Windows App Cert Kit against your app – and from an ARM device such as Surface?

Number 4: Localize Appropriately

Certification requirement 6.5: “You must localize your app for all languages that it supports.”

This is different than specifying Markets. Your supported language may differ from the Markets, and Markets are specified when submitting your app under “Selling Details.” For example, if you open Package.apppxmanifest and under your <Resources> specify <Resource Language="x-generate" /> then Visual Studio will automatically add the languages your app supports based on the discovered resources when generating your final manifest. The important thing with regards to certification is that you support at least one of the allowed 37 certification languages. You may support any number of the 66 total allowed languages, but at least one language must be one of the allowed certification languages for testing purposes. Be sure that all of your images are also localized – if you have any images that contain text they should be localized.

Number 3: Performance

Certification requirement 3.8: “Your app must meet the basic performance criteria on a low-power computer - the app must launch in 5 seconds or less and the app must suspend in 2 seconds or less.”

The Runtime broker suspends all threads of any Windows Store App that is not in the forefront. If your app takes more than the allotted time to save state while suspending or takes too long while loading then the your app will be terminated. Typically this failure is uncovered by testing with the latest version of the Windows App Cert Kit which does automated tests of this. Failing on this requirement results in a technical compliance failure. In some cases people run the Windows App Cert Kit on their developer machine but when they submit to the store they still fail with this issue. That is why we recommend running the Windows App Cert Kit against your app on different hardware – including on an ARM device such as Surface.

Then the question is what are you doing in your app when you are loading and when you are suspending? If you are talking to webservices perhaps it works on your local machine but not when you upload as performance may vary based on different network capability. You should avoid any long operations when loading and when suspending and save state only locally. LocalSettings is a keyvalue pair that can be used that saves locally. RoamingSettings saves locally first and then synchs at a later date so that too is a great way to save data – and it is easy to change the default backing store of the SuspensionManager to RoamingSettings (see http://aka.ms/vrrplb).

If you find your initial loading is taking a long time due to heavy operations such as xml serialization then check out the MSDN Splashcreen Sample. They load the app and then create a fake splashscreen while they continue heavy loading operations – that’s one way to get around the time limitations.

Another common issue related to performance is when navigating pages within your app – you should only use simple types for your navigation parameters – such as Strings or Ints.

Number 2: Primary Experiences Stay Within The App

Certification requirement 2.4: “The primary experiences your app provides must take place within the app.”

This was a very common failure reason particularly in the first wave of Windows Store Apps and it is still common today. You should not redirect the user to the web browser to complete one of your primary scenarios. In our labs we recommend developers document their five primary scenarios for testing and include them in their tester notes with each step and a “scenario complete” at the end.

It’s okay to redirect the user to the browser or even embedded browser for other scenarios in your app such as submitting help requests or tutorials and such – but it’s not okay for your primary scenarios. You should not take them out to a browser nor should you use an embedded browser.

The reason is simple – if you are using a browser to complete your primary scenario – then it is arguable that your app should be a website and not an app.

Number 1: Privacy-Related Requirements

Drum roll please… the number one top failure reason for Windows Store Certification is…

Certification requirement 4.1: “Your app must comply with the following privacy-related requirements”

Why do so many people fail on this? By default in Visual Studio, Windows Store App templates have the Internet (Client) capability declared. When this is declared – even if you aren’t using it – you must have a Privacy Statement and it must be available from the Settings Flyout. That Privacy Statement can be a link to your online Privacy Statement or another page within your app that lists the Privacy Statement. If you are not using the network you can simply disable that capability to avoid this requirement. Otherwise it is easy enough to add a settings flyout. There are examples on MSDN or see Tim Heuer’s excellent free Callisto controls also available as a Nuget package.

In that similar category of failure reasons – if you are selling your app – you must also have a help and support section. There are three acceptable help and support options. A telephone, an email address (and they will test if it is a working address) or a webform. The help and support can be a separate link from your Settings flyout or it can be within your About section (also linked in your Settings flyout).

Finally…

If you’ve followed these pointers and you’ve checked the Windows Store Apps Forums and you are still blocked on Store Certification – please don’t sit there in frustration depriving the world of your amazing app!

Please either visit one of our free Windows Store App Labs (www.windowsstore.com/applabs) or follow the instructions below to get fast assistance. There’s even a “chat” option!

1)    Visit the URL: http://aka.ms/storesupport

2)    Towards the bottom of the page, there is a link that says “…contact us immediately.”

3)    Click that link and then it will ask you to choose the support type. To reach the App Certification team choose the following:

Problem Type: “App Submission and Certification”

Category: “Submitting an app”

Once you do that chose an option for support such as “Request a Call”

Windows Store App Development, with Christophe

2013-04-05T01:21:34Z

Christophe Nasarre-Soulier is a development-focused PFE I’ve worked via the Windows Store App Labs program.

As part of that program, we provide design and development assistance to Windows Store App developers across the world, and gained a lot of exposure to common questions, issues, and techniques.

Christophe’s been busy blogging some of his learnings on Windows Store Apps – including (naughty!) testing-only features that are fun to play with in local development environments but that won’t pass Store certification requirements – and has a veritable smorgasbord of posts for you to consume.

First up, his series on inter-process communication with Windows Store Apps:

Since the pre-release of Windows 8 at the Microsoft BUILD conference in 2011, I’ve been investigating the different ways for a Windows Store App (WSA in the rest of the post) to interact with either other WSAs or Desktop Apps (DA in the rest of the post). Unlike .NET that was built from the beginning with interoperability in mind, WinRT is another beast.

A Microsoft collegue has been working on a WSA that makes it simple to deploy and launch applications in an enterprise environment (visit http://companystore.codeplex.com/ for more details). However, some actions are not allowed from a WSA such as looking for installed WSAs and launch them. This was a great opportunity to turn my research into real code. I’m sharing the outcome in a few blog posts starting with this one.

The first post (linked above) covers the basics of possible interaction mechanisms between Windows Store apps (file associations and protocol handlers), and it’s a quick, helpful read.

The second post delves deeply into using file associations to allow app activation outside the bounds of a Windows Store App into a desktop app – with the requisite warning that you can’t do this in a Store-published app:

WARNING: all these techniques are violating the point 3.1 from the Windows Store Requirements that states: “Your app may only depend on software listed in the Windows Store“. Therefore, you should not use them for a WSA that you plan to publish into the Windows Store because it will be rejected.

It’s a fascinating look into the internals of how file association activation works.

Christophe’s back-catalogue covers lots of other WinRT-related goodness, from Windows Store checklists to make it easier to make great Windows Store apps, to a tool that identifies BCL classes available vs excluded from WinRT and the .Net Framework 4.5. Enjoy!

Posted by Tristan Kington, MSPFE Editor that just got to use the word “smorgasbord” in a serious context

Taming XML query performance using persisted computed columns

2013-03-26T09:58:55Z

Storing data in XML format has presented a conundrum to SQL developers from time to time. While storing it is easy, how does one efficiently extract those pieces of data which are needed from the XML field?

My friend and PFE colleague from Dubai, Batuhan Yildiz, had faced this situation recently. He shares his experiences on the performance impact of retrieving specific data from the XML field, and presents an interesting way of solving the issue by using persisted computed columns. Read the details at his blog here!

Linked content authored by Batuhan Yildiz; posted by Arvind Shyamsundar

The case of the broken AppFabric

2013-03-22T09:36:44Z

This article has been contributed by Vasco Preto, a Premier Field Engineer with Microsoft Portugal. Vasco is no stranger to MSPFE, having previously contributed a very interesting case study on our TechNet blog. This time he shares a captivating walkthrough on how to troubleshoot AppFabric caching services.

I’ve been recently involved in troubleshooting a production problem reported by a customer which revolved around a generic error that was preventing client applications from successfully establishing a connection to the AppFabric – Caching Services layer. Here’s a quick recap of what the exact symptoms were and the analysis and diagnostic that resulted from the troubleshooting effort.

Problem description

The client presented a specific case where a set of recently deployed client applications were unable to connect to the AppFabric – Caching Services layer. This specific error was generating the following message (extracted from the event logging system).

{Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified Cache servers are unavailable, which could be caused by busy network or servers. Ensure that security permission has been granted for this client account on the cluster and that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Retry later.)

Furthermore, the client stated that this error was only affecting a subset of client applications and these specific client applications had functioned correctly in the past and, supposedly, they had not been subjected to any change recently.

Analysis

Step 1: Check for known issues

Considering the starting point, which was little more than a fairly generic message, I did a quick search through existing KB articles in an attempt to clarify which conditions could actually lead to this type of error. Unfortunately, this quick research did not produce a more defined and limited scope to investigate, as there are many different conditions that can lead to problems which are reported through error code ERRCA0017 and sub-status code ES0006.

Step 2: Check if the AppFabric cluster is running

Considering that the client was describing the problem as a connection error between client applications and AppFabric server, I started off my direct troubleshooting effort by verifying that the AppFabric cluster was in fact up and running. Using some administrative PowerShell commands I concluded that the AppFabric server was in fact working properly (commands used include Get-CacheClusterHealth, Get-Cache and Get-CacheStatistics).

Afterwards, I verified if I could in fact establish a TCP connection between client and server machines through the AppFabric port, which is port 22233. By using a simple telnet client utility I did in fact verify that the client application was able to connect to the server through port 22233. However, when that connection was established through AppFabric’s client libraries the previously showed error would appear.

Step 3: Gathering network traces

At this point, I was able to get a few network traces supplied by the customer’s operations team. As shown in the following image, which depicts the filtered frames that pertain to the action of establishing the AppFabric connection (filtered by client and server IP addresses), we can clearly see two things:

The initial TCP/IP handshake between client and server was correctly established.
A few frames after the handshake, the server reset the connection.

Step 4: Drilling down into the network trace

Continuing within the network trace, and following the TCP stream regarding these frames, I could drill down on a bit more information. As seen in the next image, the information contained within the network trace showed a few more interesting pieces of information:

Information regarding the client and its credentials.
Identification of the server to which the communication was being established.
References to authentication protocols.

Step 5: Checking which accounts have access to AppFabric

At this point, I discussed the situation with the customer and we both agreed that the problem seemed to be related with client credentials and inherent authentication process. Going back to the PowerShell administration console, we analyzed which accounts had been granted access to AppFabric. As shown in the next output, the existing configuration was apparently allowing a wide open access to all client credentials.

PS C:\Windows\system32> Get-CacheAllowedClientAccounts
EVERYONE
ANONYMOUS LOGON

Despite this configuration, I queried the customer about the exact credentials that the client was sending (which were also identifiable in the previously shown TCP stream – please note reference to local machine) and we concluded that the client application was using a local account. As such, the customer agreed to change the client credentials and the client application was reconfigured to use a domain account.

Step 6: AppFabric logging and ETW

Unfortunately this change did not fix the problem so we analyzed other available troubleshooting options and decided that we needed to get more logging data from AppFabric. By the way, this avenue has impact as it requires restarting the cache service, which implies temporarily restarting one of the cache cluster nodes, and hence temporarily affecting performance (other cache nodes are not directly affected, but regions within this node will ultimately need to be re-populated from ground up afterwards). After some debate, the client accepted this route and we performed two actions.

Firstly, we changed the logging level of the caching service by adding the following configuration elements to its configuration file (file DistributedCacheService.exe.config).

<system.diagnostics>
    <sources>
      <source name="System.ServiceModel"
              switchValue="Information, ActivityTracing"
              propagateActivity="true">
        <listeners>
          <add name="traceListener"
              type="System.Diagnostics.XmlWriterTraceListener"
              initializeData= "c:\Temp\DistributedCacheService.svclog" />
        </listeners>
      </source>
    </sources>
    <trace autoflush="true" />
</system.diagnostics>

Secondly, we activated ETW for AppFabric through the following command line:

tracelog.exe -start debugtrace -f "c:\Temp\cachedebugtrace.etl" -guid "c:\Windows\System32\AppFabric\Manifests\ProviderGUID.txt" -level 5 -cir 512

Step 7: Analyzing ETW

After acquiring the ETW log we transformed it into a CSV file through the command:

tracerpt "c:\Temp\cachedebugtrace.etl" -o "c:\Temp\cachedebugtrace.csv" -of CSV

And by analyzing its contents we were able to find some interesting information such as:

"Recoverable exception raised - ChannelID = 29180324 System.ServiceModel.ProtocolException: Stream Security is required at http://www.w3.org/2005/08/addressing/anonymous, but no security context was negotiated. This is likely caused by the remote endpoint missing a StreamSecurityBindingElement from its binding. at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ServerFramingDuplexSessionChannel.SetupSecurityIfNecessary() at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ServerFramingDuplexSessionChannel.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ApplicationServer.Caching.WcfServerChannel.Listen(IAsyncResult result)"

Interestingly, this message hints that there is something definitely wrong with the security context, but not necessarily implying problems with credentials. It also suggests that there are some issues with the underlying WCF transport that is used between client application and AppFabric server whenever the AppFabric API is used.

The error message is almost clear to the point that it identifies the lack of a required security context, which results in a binding error to an existing WCF endpoint. To confirm this we opened the log file (the svclog file generated by the changes introduced in the distributed cache service configuration file) which confirmed what had been established so far (as shown in the following picture).

In possession of all this information we went back to the configuration files of both client application and AppFabric server and concluded that there was indeed a mismatch, as the client configuration (contrary to what was defined in the server side configuration) was using None as the configuration value for both securityMode and protectionLevel (more information on these options can be found here.

Finally, the solution!

So, finally, to fix the problem we changed the securityMode to Transport and the protectionLevel to EncryptAndSign and, having matched client and server configuration attributes, the solution was finally up and running.

To wrap up, it is almost inevitable to report that, as it turns out, someone had, inadvertently, copied a configuration file from development environment to production, thinking that they were an exact match, without verifying that the security configuration elements were in fact distinct.

I guess this is another example that supports the prohibition of direct copying from development and quality environments into production environments, although a more clear error message from AppFabric would also be much appreciated .

Written by Vasco Preto; posted by MSPFE Editor Arvind Shyamsundar

More Efficiently Debug the Managed Heap Using SOS 4.5

2013-03-13T22:27:00Z

More often than not, when we investigate the managed heap using SOS we use the all powerful !DumpHeap command. It's an incredibly useful command to get in depth detail on what currently resides on the heap. Let's look at an example of the abbreviated output from !DumpHeap -stat: MT Count TotalSize ...read more...(read more)

Video series: .Net Debugging for the Production Environment

2013-01-30T10:38:59Z

Brad Linscott is a very experienced Premier Field Engineer from the .Net development side of the house (Ed: who originally trained me in .Net Debugging techniques, oh, eight? years ago?) and he’s published a video series on .Net Debugging on Channel 9, which runs you from the basics to some quite advanced techniques in .Net debugging, as performed by the experts!

This series was created based off of a thread on the Channel 9 Site Feedback forum. Read the original post here. We heard what you wanted, so we created the first .NET Debugging Starter Kit series. With the help of Brad Linscott, Premier Field Engineer, this series will teach you how to use various tools, including the Debugging Tools for Windows package (windbg), to resolve your .NET application issues in a production and pre-production environment, including post-mortem analysis of dump files.

It’s a series of shortish videos (given the subject depth), which is nice and lunch-break friendly. Or you could replace your mandatory afternoon snooze with learnings!

Posted by Tristan Kington, MSPFE Editor, and a table as well.

Free Expert Advice for Building Your Windows 8 Apps. Windows Store App Labs!

2013-01-25T15:59:00Z

Summary: Robert Evans, a Senior Microsoft Premier Field Engineer focused on development, informs us of some awesome resources for Windows 8 developers to get advice from experts and access to a myriad of Win8 devices. Take advantage of it if you can.

Windows Store App Labs (WSALs) are here! A place where you can access Windows 8 devices, get design help, and meet with Windows 8 Experts to make your app great! All for free!

Get technical guidance from Windows 8 Experts

Our Windows Store app developer experts are on-hand to help you get started, to test your app and offer any technical advice you need! We can help you with coding tips, debugging, and prepping your app to submit to the Windows Store.

Check out and test on the newest Windows 8 devices

Test your app on a variety of screen sizes, input methods, and chipsets with the latest Windows 8 devices including Windows RT tablets, Ultrabooks, All-in-Ones, and more. Designers will review your app mock-ups or your complete app and provide feedback on how you can improve your app’s layout, navigation, live tile, and other UI elements.

As you get started - check out 5 tips to help you get your app certified quickly, top 10 app secrets of success, Win8 Forms and more helpful information on our lab resources page.

All for free!

Space is limited. Please register for a lab in your location:

Country	City	Registration Link
Canada	Vancouver	https://applabs.msregistration.com/LabList.aspx?drop=52&userid=null#list
US	San Francisco/Santa Clara	https://applabs.msregistration.com/LabList.aspx?drop=139&userid=null#list
Canada	Toronto	https://applabs.msregistration.com/LabList.aspx?drop=40&userid=null#list
US	New York	https://applabs.msregistration.com/LabList.aspx?drop=140&userid=null#list
England	London	https://applabs.msregistration.com/LabList.aspx?drop=9&userid=null#list
France	Paris	https://applabs.msregistration.com/LabList.aspx?drop=13&userid=null#list
Belgium	Brussels	https://applabs.msregistration.com/LabList.aspx?drop=25&userid=null#list
Czech Republic	Prague	https://applabs.msregistration.com/LabList.aspx?drop=25&userid=null#list
Sweden	Kista	https://applabs.msregistration.com/LabList.aspx?drop=22&userid=null#list
Turkey	Istanbul	https://applabs.msregistration.com/LabList.aspx?drop=22&userid=null#list
India	Mumbai	https://applabs.msregistration.com/LabList.aspx?drop=22&userid=null#list
India	Bangalore	https://applabs.msregistration.com/LabList.aspx?drop=3&userid=null#list
China	Beijing	https://applabs.msregistration.com/LabList.aspx?drop=11&userid=null#list

Debugging STL containers using WinDbg

2013-01-15T05:02:02Z

My good friend and colleague, Ambrose Wong, is a Principal PFE in Hong Kong. He has been debugging and troubleshooting production issues for over a decade now. More importantly, he mentors our customers to be their own hero and debug issues on their own.

One of the conversations we have with customers who are using WinDbg for the first time is obvious comparisons with Visual Studio, and how easy it is to ‘Watch’ things in VS.NET. The expression evaluation (a.k.a. Watch) capability in VS.NET is very easy to use, but what if you have to do that with WinDbg in a production environment?

While it may be easy to dump scalar values, how about when you are dealing with STL vector classes, such as vector<>? It may not be very apparent how to do that in WinDbg. Ambrose has started a series on how YOU can do that yourself – read the introduction and part 1 of his series. Highly recommended for WinDbg buffs!

BTW, if you have never heard about the Standard Template Library – STL – you can hear Stephan Lavavej (Mr. STL himself ) introduce it here.

Posted by Arvind Shyamsundar, MSPFE Editor

SQL Server Upgrade Advisor: Considerations when upgrading from SQL 2000 to SQL 2012

2012-12-06T02:29:00Z

This article was contributed by Tony O’Grady, a Premier Field Engineer from the UK.

Are you currently running old SQL Server 2000 instances in your estate?

Are you planning to upgrade these to SQL Server 2012?

A direct upgrade from SQL 2000 to SQL 2012 is not supported, so chances are you’ll need to go via an intermediate version. Identifying issues we may need to fix - either before or after the upgrade - is one of the challenges with this technique.

SQL Server Upgrade Advisor:

To prepare for upgrades, Microsoft provides a tool called the SQL Server Upgrade Advisor. This tool analyses a SQL Server and creates a report which identifies issues we may need to fix when upgrading to a later version.

The Upgrade Advisor can be used for supported upgrade scenarios:

· The SQL Server 2012 version of the Upgrade Advisor can examine instances of SQL Server 2005, SQL Server 2008, and SQL Server 2008 R2.

· The older SQL Server 2008 R2 Upgrade Advisor can be used for older SQL 2000 instances.

Since a direct upgrade from SQL 2000 to 2012 is not a supported scenario, we cannot use the SQL 2012 Upgrade Advisor to examine SQL 2000 instances. This is explained in the SQL Server 2012 Upgrade Technical Reference Guide which offers the following guidance on Upgrading SQL Server 2000 to SQL Server2012:

· You cannot upgrade a SQL Server 2000 instance or database to SQL Server 2012.

· For a side-by-side upgrade, first restore the SQL Server 2000 databases to SQL Server 2005, 2008, or 2008 R2, and then restore the resulting database to SQL Server 2012.

Do we need to run multiple versions of the Upgrade Advisor?

Is it really necessary to run both versions of the Upgrade Advisor, or will it be sufficient to run the 2012 Upgrade Advisor to identify all potential issues?

To answer this question we have to consider whether the SQL Server 2012 upgrade Advisor will identifySQL 2000 features discontinued in SQL 2005.

One example of such a feature is the non-ANSI outer join operator “*=” which is only supported on SQL Server 2005 when the database is running in the older SQL Server 2000 (80) compatibility mode. For forward compatibility, it should be replaced with “LEFT OUTER JOIN” syntax.

Using this as an example on a development SQL Server 2005 instance, we created a test stored procedure which uses the “*=” join operator. We then evaluated the instance with both the SQL 2008 R2 and SQL 2012 Upgrade Advisor tools. A summary of this test is presented below.

Creating our test scenario:

On a development instance of SQL Server 2005 we created a test database called TestComp and set the compatibility of the database to 80. Setting the compatibility of the database to 80 ensured that the non-ANSI outer join operator “*=”would execute successfully. To facilitate testing, we also imported in a couple of tables from the AdventureWorks database.

CREATE DATABASE[TESTComp]

EXECUTEdbo.sp_dbcmptlevel @dbname=N'TESTComp', @new_cmptlevel=80

Next, we created a stored procedure called testJoin that used the deprecated “*=” join operator:

CreatePROCEDURE [dbo].[testJoin]

BEGIN

SELECT*

FROM Employee LEFTOUTERJOIN

EmployeeAddress ON Employee.EmployeeID = EmployeeAddress.EmployeeID

SELECT*

FROM Employee, EmployeeAddress

WHERE Employee.EmployeeID *= EmployeeAddress.EmployeeID

END

Since the database compatibility mode was set to 80, the stored procedure executed successfully:

EXECUTE[dbo].[testJoin]

(290 row(s) affected)

SQL Server 2008 Upgrade Advisor results

We then ran the SQL Server 2008 R2 Upgrade Advisor against the TESTComp database, and looked at the results:

As we can see, the report highlights the use of the outer join operator “*=”.

SQL Server 2012 Upgrade Advisor results

We tried the SQL 2012 Upgrade Advisor against the same TESTComp database, and got the following error:

The SQL 2012 Upgrade Advisor requires the database to be at a compatibility level of 90 or greater. Since this was a test database on my laptop, we went ahead and changed the compatibility level:

EXECUTE dbo.sp_dbcmptlevel @dbname=N'TESTComp', @new_cmptlevel=90

EXECUTE[dbo].[testJoin]

Msg 4147, Level 15, State 1, Procedure testAnsci, Line 12

The query uses non-ANSI outer join operators ("*=" or "=*"). To run this query without modification, please set the compatibility level for current database to 80 or lower, using stored procedure sp_dbcmptlevel. It is strongly recommended to rewrite the query using ANSI outer join operators (LEFT OUTER JOIN, RIGHT OUTER JOIN). In the future versions of SQL Server, non-ANSI join operators will not be supported even in backward-compatibility modes.

Changing the compatibility level to 90 caused the execution of the [dbo].[testJoin]stored procedure to fail. This was expected, so we continued on and ran the 2012 Upgrade Advisor:

As you can see from the report above, the 2012 Upgrade Advisor did not give a specific warning on the outer join operator “*=”. If you simply tried to upgrade the compatibility level and relied on the report alone, you might not have known that your database was no longer functioning correctly.

Conclusion:

The SQL Server 2012 Upgrade Advisor will analyse instances of SQL Server 2005, SQL Server 2008, and SQL Server 2008 R2 when the database is in a compatibility mode greater than 80. In our example above, we used a feature which is not supported when the compatibility level of the database is set to 90. The resulting report generated from the SQL 2012 Upgrade Advisor did not highlight this as something to fix, but this issue was clearly reported on the older SQL 2008 R2 Upgrade Advisor tool.

So, when upgrading from SQL Server 2000 to 2012, consider using both the older 2008 R2 Upgrade Advisor and the new SQL 2012 Upgrade Advisor, as this will provide a more complete picture of issues you may need to fix.

For additional information on upgrading to SQL 2012 see the MSDN library and the SQL Server 2012 Upgrade Technical Reference Guide

Posted by Tristan “Mad King” Kington, MSPFE Editor. With fire.

Heap Corruption – a W3WP case study

2012-12-04T15:06:32Z

This blog post has been authored by Yawei Wang, a Premier Field Engineer with Microsoft Singapore. He describes a very common and important type of issue – heap corruption – and how you can successfully debug it using the Debugging Tools for Windows.

Background

Heap corruption is by nature a complicated issue to troubleshoot and in some cases luck is as important as debugging knowledge as well. It normally happened intermittently and we cannot reproduce it within a short time frame. The best way to understand how it works is through a real case analysis. Today we are going to unveil the secret a W3WP heap corruption case together.

Symptoms

The background is that a customer reported an intermittent crash of W3WP process on IIS server when the server was under heavy load. Sometimes it would respond with HTTP 500 (IIS 5.0 / 6.0, IIS 7.x) but sometimes the process was just terminated unexpectedly. It’s a generic second chance access violation (0xC0000005) exception (see What is a First Chance Exception?) and we were able to get the crash dump with DebugDiag.

Troubleshooting

From the dump, we can find the thread was crashed on CUSTOM_ERROR_TABLE::FindCustomError. FindCustomError finds the applicable custom error entry for a given status / subcode and return it back to client. Here is the call stack:

0:033> knL
# ChildEBP RetAddr
00 06cee2cc 5a49fb48 w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x18
01 06cee428 5a42392b w3core!ISAPI_REQUEST::GetCustomError+0x8e

…

1b 06ceffb8 7c82482f msvcrt!_threadstartex+0x74
1c 06ceffec 00000000 kernel32!BaseThreadStart+0x34

The calls stack was quite clean without any third-party components to suspect. It’s also very rare that the crash happened within IIS module and as you may not know w3core.dll is the core component in IIS 6 (same as iiscore.dll in IIS 7) there can’t be an issue in it as IIS 6 has been released for years and the number of calls to FindCustomError can be numerous. If there should be a issue, it wouldn’t survive for a minute.

So, how did it happen? Let’s first take a look at the register status: ESI is null, which seems to be the direct culprit.

eax=06ce0000 ebx=06ceedb4 ecx=017c4774 edx=000006e2 esi=00000000 edi=06ceedb4
eip=5a49fbd2 esp=06cee2c4 ebp=06cee2cc iopl=0 nv up ei ng nz ac po cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010293
w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x18:
5a49fbd2 668b4608 mov ax,word ptr [esi+8] ds:0023:00000008=????

To understand why ESI was null we have to analyze the below assembly. We see the address stored in ECX was null which caused the C++ exception.

0:033> u
w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x18
5a49fbd2 668b4608 mov ax,word ptr [esi+8] <======esi is null
5a49fbd6 663b4508 cmp ax,word ptr [ebp+8]
5a49fbda 7404 je w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x22 (5a49fbe0)
5a49fbdc 8b36 mov esi,dword ptr [esi]
5a49fbde ebea jmp w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x34 (5a49fbca)
5a49fbe0 668b460a mov ax,word ptr [esi+0Ah]
5a49fbe4 663b450c cmp ax,word ptr [ebp+0Ch]
5a49fbe8 0f841c780000 je w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x3f (5a4a740a)

Here is the disassembly of the function w3core!CUSTOM_ERROR_TABLE::FindCustomError:

5a49fbac 8bff mov edi,edi
5a49fbae 55 push ebp
5a49fbaf 8bec mov ebp,esp
5a49fbb1 56 push esi
5a49fbb2 57 push edi
5a49fbb3 8b7d10 mov edi,dword ptr [ebp+10h]
5a49fbb6 85ff test edi,edi
5a49fbb8 0f8467fdffff je w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x59 5a49f925)
5a49fbbe 837d1400 cmp dword ptr [ebp+14h],0
5a49fbc2 0f845dfdffff je w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x59 (5a49f925)
5a49fbc8 8b31 mov esi,dword ptr [ecx]<=======poi(ecx) -> esi ; poi(ecx) = null
5a49fbca 3bf1 cmp esi,ecx
5a49fbcc 0f8448fdffff je w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x38 (5a49f91a)

Why ECX was NULL? We know “this call” calling convention (used for calling C++ non-static member functions) will pass “this” pointer in ECX. As you see, The ‘dt’ command will show the structure name. Although the debugger says “this” is stored in EDX, it's wrong! It should be in ECX which is null.

0:033> dt this

Local var @ edx Type CUSTOM_ERROR_TABLE*

0:033> dd poi(ecx) L1

01dc7540 00000000

Then we dumped the CUSTOM_ERROR_ENTRY and would like to see why the object CUSTOM_ERROR_TABLE was null.

0:033> !list "-t ntdll!_LIST_ENTRY.Flink -e -x /"dt w3core!CUSTOM_ERROR_ENTRY @$extret/" 017c4774"
dt w3core!CUSTOM_ERROR_ENTRY @$extret
+0x000 _listEntry : _LIST_ENTRY [ 0x1dc7540 - 0x1d60620 ]
+0x008 _StatusCode : 0xbda0
+0x00a _SubError : 0x1db
+0x00c _strError : STRU
+0x03c _fIsFile : 0n0

dt w3core!CUSTOM_ERROR_ENTRY @$extret
+0x000 _listEntry : _LIST_ENTRY [ 0x0 - 0x0 ]
+0x008 _StatusCode : 0
+0x00a _SubError : 0
+0x00c _strError : STRU
+0x03c _fIsFile : 0n0

0:033> ? 0xbda0
Evaluate expression: 48544 = 0000bda0

Obviously, the custom error list is completely corrupted so we can't get actual custom error 500.100 (Internal Server Error - ASP error).

0:033> .frame 0

00 06cee2cc 5a49fb48 w3core!CUSTOM_ERROR_TABLE::FindCustomError+0x18

0:033> dv
this = 0x000006e2
StatusCode = 0x1f4<=====500
SubError = 0x64<======100
pfIsFile = 0x06ceedb4
pstrError = 0x06cee334

When we check the list entry address 0x1d60620, we can see that it belongs to heap 0x2b0000, which is the MSVCRT (C Runtime) heap.

0:033> !address 0x1d60620
Usage: Heap
Allocation Base: 01d50000
Base Address: 01d50000
End Address: 01e50000
Region Size: 00100000
Type: 00020000 MEM_PRIVATE
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
More info: heap containing the address: !heap 0x2b0000
More info: heap entry containing the address: !heap -x 0x1d60620
Heap 3 - 0x002b0000
Heap Name msvcrt!_crtheap
Heap Description This heap is used by msvcrt
Reserved memory 3.13 MBytes
Committed memory 1.64 MBytes (52.38% of reserved)
Uncommitted memory 1.49 MBytes (47.63% of reserved)
Number of heap segments 3 segments
Number of uncommitted ranges 1 range(s)
Size of largest uncommitted range 1.43 MBytes
Calculated heap fragmentation 3.94%

At the current stage, we believe that it is a typical heap corruption. While debugging heap corruption issues is not an easy task because the thread that caused the exception was not usually the thread that caused the corruption (FindCustomError is the victim in this case), we still can use the full PageHeap feature to capture another round of IIS crash dump.

With the assistance of pageheap.exe, we managed to capture another round of IIS crash dump and this time we found the culprit:

FAILED_INSTRUCTION_ADDRESS:
safeagent+1ae10
1001ae10 ?? ???
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: safeagent!unloaded+1ae10
FOLLOWUP_NAME: wintriag
MODULE_NAME: safeagent

IMAGE_NAME: safeagent.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 45f51b03

FAILURE_BUCKET_ID: BAD_INSTRUCTION_PTR_c0000005_safeagent.dll!unloaded

BUCKET_ID: APPLICATION_FAULT_BAD_INSTRUCTION_PTR_INVALID_POINTER_READ_BAD_IP_safeagent!unloaded+1ae10

Followup: wintriag

The call stack was truncated, due to the module being unloaded:

0:007> knL
# ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 088bc188 00000000 <Unloaded_safeagent.dll>+0x1ae10

Conclusion

In this case, we can conclude that the third party module (safeagent.dll) was involved in this issue. The next step was to contact the provider of this module for further investigation.

Edited and posted by Arvind Shyamsundar, MSPFE Editor