Mahjayar's WebLog. : Debugging

DbSyncProvider: Improving memory performance involving execissive conflicts.

Mahjayar — Thu, 15 Jan 2009 23:17:50 GMT

This is the second post aimed at providing tips for improving runtime memory footprint when using the P2P DbSyncProvier class in V2 of SyncServices for ADO.NET.

First post link: DbSyncProvider- Improving Memory Performance In WCF Based Synchronization

We have had customers report memory usage surging when DbSyncProvider is applying changes from a peer and encounters conflicts. If the number of conflicts increases the memory usage spikes and pretty soon will run out of memory.

After debugging, we narrowed down the issue down to a property in DbSyncTableProgress class: Conflicts. This property was being used by the runtime to “log” conflicts that were skipped during that Sync session. DbSyncTableProgress holds the running count of progress made during a sync session for each sync adapter configured. Whenever the system encounters a Conflict while applying a row it will try to invoke the optional conflict event handler. The runtime will then decide what to do with the row based on the ApplyAction enum returned by that event handler. The default action is ApplyAction.Continue which means the runtime will assume “LocalWins” resolution and just update the local metadata for that row and move on. That is the best case scenario. Now lets see the worst case scenarios: we have two.

1. No event handler registered but row application failed due to Database error.

2. User registered event handler returned an ApplyAction of RetryNextSync.

The first case can happen for a variety of reasons such as errors in user configured TSQL command, typos in user configured Stored proc names or params, error in executing stored proc or database timeouts or other database related errors. The second is just a normal way of saying “dont worry about this row for this sync and we will try this again in the next sync”. Since the P2P provider is based off of Microsoft Sync Framework choosing RetryNextSync adds an exception for that particular row in the sync knowledge and moves on. This exception will tell the source to resend the row during the next sync.

Whenever the above two scenarios happen the runtime will “log” the conflict in the corresponding DbSyncTableProgress for that table. Logging includes cloning the source DataRow and the local DataRow and storing it in DbSyncTableProgress.Conflicts property. So for each conflict the runtime caches the source and destination row. If the number of conflicts increases or the size of each DataRow is large the system will gobble up available memory eventually leading to OutOfMemory exceptions.

This property was carried over from SyncTableProgress type in SyncServices V1. SyncServices V1 was an anchor based hub-spoke synchronization. V1 too had conflict resolution handling and one such option was to skip applying rows. Since the runtime had no way of storing anchors for those skipped rows, those rows would never ever be synchronized again (until a future changed to the same row happens). So, we had to aggregate and log all such conflicts so users can view it at the end of a sync session and take appropriate action. As you can see with the advent of SyncFramework this property is no longer needed as this bookkeeping is automatically done by the SyncFramework API’s.

Workaround:

So there is a simple workaround to avoid this issue. Workaround is to implement the SyncProgress event handler on DbSyncProvider and clear the Conflicts collection on the TableProgress property.

this.peerProvider.SyncProgress += new EventHandler<DbSyncProgressEventArgs>(peerProvider_SyncProgress);

void peerProvider_SyncProgress(object sender, DbSyncProgressEventArgs e)
{
e.TableProgress.Conflicts.Clear();
}

This will ensure that the collection is always cleared and no extra memory will be used.

Ps: This bug is on track to be fixed for the next release.

Maheshwar Jayaraman

Microsoft releasing .NET framework source code

Mahjayar — Thu, 04 Oct 2007 01:06:00 GMT

Microsoft is releasing the source code for the framework libraries. Read more about it here. Its not like you couldnt do this earlier. You could always have used .Net Reflector but having the ability to step in to the framework classes while debugging is neat. Looks like plans are on for releasing the source code for WCF as well!

Debugging the mystery of the crashing desktop

Mahjayar — Mon, 02 Apr 2007 05:51:00 GMT

My home desktop running Vista would crash with a blue screen everytime I logged in the main console if I had remotely logged in via TerminalServices prior to that. This was happening every time I connected remotely to the box and couple of weeks earlier I found another way of crashing it. I was able to consistently crash the system if I logged in as Guest and then login with main account. Prior to today, I never bothered to check why it crashed as I attributed the crash to some faulty driver not updated for vista and I had a simple workaround of restarting the computer before disconnecting from my remote session (I know its not the ideal way but it worked for me). My desktop is a Gateway T3306, a "cheap" desktop that I had purchased during 2005 thanksgiving. The desktop was bare bones and I had beefed it up by adding extra memory/disk.

Last week I attended David Solomon's 5 day course on "Windows OS Internals" and part of that talked about debugging a system crash dump. After the exercise I decided to come home and check the dumps to see which driver was actually causing the crash.

I started up Windbg and loaded the latest dump from \windows directory and the autoanalyze gave me the following output.

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: dd05221e, memory referenced.
Arg2: 00000000, value 0 = read operation, 1 = write operation.
Arg3: 90ea9301, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 00000002, (reserved)

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: win32k!SearchIconCache+20

FAILURE_BUCKET_ID: 0x50_win32k!SearchIconCache+20

BUCKET_ID: 0x50_win32k!SearchIconCache+20

Followup: MachineOwner

The followup usually points to a driver if it finds one in the thread callstack that caused the KiTrap0E error (crash). MachineName usually means that it didnt find a driver in the stack. Walking the stack using the k command produced the following output.

kd> k
ChildEBP RetAddr
a0c9fc30 81c8fa74 nt!MmAccessFault+0x106
a0c9fc30 90ea9301 nt!KiTrap0E+0xdc
a0c9fcc4 90ea93bc win32k!SearchIconCache+0x20
a0c9fce4 90ea94ab win32k!_FindExistingCursorIcon+0x4a
a0c9fd50 81c8c96a win32k!NtUserFindExistingCursorIcon+0xe5
a0c9fd50 77ce0f34 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
001be900 00000000 0x77ce0f34

So the stack points to a Win32k thread trying to call a SearchIconCache method. Tried searching on the method name but didnt hit anything. So tried to see what it was trying to do with memory location dd05221e mentioned in the argument so the bubcheck. Did a unassemble at the return address for the Trap0E function and got this.

kd> u 90ea9301
win32k!SearchIconCache+0x20:
90ea9301 663b461c        cmp     ax,word ptr [esi+1Ch]
90ea9305 754e            jne     win32k!SearchIconCache+0x74 (90ea9355)
90ea9307 f6462004        test    byte ptr [esi+20h],4
90ea930b 7448            je      win32k!SearchIconCache+0x74 (90ea9355)
90ea930d 668b461e        mov     ax,word ptr [esi+1Eh]
90ea9311 663b4704        cmp     ax,word ptr [edi+4]
90ea9315 753e            jne     win32k!SearchIconCache+0x74 (90ea9355)
90ea9317 8d4614          lea     eax,[esi+14h]

The first statement compares a value pointed by pointer (esi + 1ch). ESI register had the value dd052202 and adding 1CH points to memory location dd05221e which is the same as the one pointed by the bugcheck argument. The memory at dd05221e pointed to nothing/garbage.

kd> dd dd05221e
dd05221e ???????? ???????? ???????? ????????
dd05222e ???????? ???????? ???????? ????????
dd05223e ???????? ???????? ???????? ????????
dd05224e ???????? ???????? ???????? ????????
dd05225e ???????? ???????? ???????? ????????
dd05226e ???????? ???????? ???????? ????????
dd05227e ???????? ???????? ???????? ????????
dd05228e ???????? ???????? ???????? ????????

the address dd05221e is kernel mode address space and hence this meant some kernel mode code was pointing to a wrong address or had its address space overwritten by a rogue driver as the memory could have been corrupted long before poor Win32K tried to do what it was doing. The process that was running this thread during the crash was mobsync.exe and from running procmon on my live system I found that it was the Microsoft Mobile Sync service running within the "Plug n Play" svchost service. Apparently this kicks in when I open the "Sync Center" or when you plugin a Windows mobile device which I had done occasionally on this PC. To confirm whether this process was the culprit each time my system crashed I opened all Minidumps from \Windows\Minidumps folder. Vista by default stores minidumps of all crashes in \Windows\MiniDump folder but keeps overwriting the full kernel dump at \Windows\Memory.dmp. I was surprised to find that each minidump showed the same mobsync process as the culprit with identical callstack as the kernel dump. I was almost convinced to fill a bug with the sync team and add the kernel dump for reference but then I decided to check one more detail before filing the bug. I read up on the BugCheck code 0x50 and here is what the documentation had to say about it.

Cause
Bug check 0x50 usually occurs after the installation of faulty hardware or in the event of failure of installed hardware (usually related to defective RAM, be it main memory, L2 RAM cache, or video RAM).
Another common cause is the installation of a faulty system service.
Antivirus software can also trigger this error, as can a corrupted NTFS volume.

Great, the cause could be

faulty RAM (Which I ruled out as I just had a faulty RAM crash at work and the blue screen was completely different for that. I also confirmed this by looking at the BIOS log which usually mentions which DIMM faulted)
Video RAM
System Service (Which mobsync.exe is)
AntiVirus (I had to rule out service and video ram before trying this out)

My Video card is an integrated S3/Via UniChrome Pro card from S3Graphics.com and looking up their site for updated Vista drivers yielded no results. That leaves me with two options, spend $$ on a video card and hope the problem goes away or try to confirm the issue with the mobsync team before spending the money. I think I am going to opt for the second option. I plan on turning on Driver Verification for the video card and then trying to reproduce the crash to see if DriverVerifier catches it and if that doesnt help then file the bug and wait. I will update later on how my DriverVerifier experiment went along.

Bottom line is I got more out of "Windows OS Internals" class that what I had initially hoped and I would strongly recommend it esp for people working on drivers. Mark Russionovich made a guest appearance at the class talking about UAC and impressed everyone with his brilliance.

Maheshwar Jayaraman

Mahjayar's WebLog. : Debugging

DbSyncProvider: Improving memory performance involving execissive conflicts.

Microsoft releasing .NET framework source code

Debugging the mystery of the crashing desktop

Cause