Dgoldman's WebLog

How to change the temp location for the OAB Generation process on Exchange 2007 sp2 and Exchange 2010

In all version of Exchange the OAB Generation process will store the data is reads from the active directory in the %temp% directory, on the server that is generating the OAB. Once the OAB Generation process completes the temp files will be removed.

For some customers that have several million users in their organization or large hosting environments this can be a problem, as most systems are never upgraded to meet the current demands. Administrators from time to time often find themselves running out of drive space on the c:\ as this part of the system never gets upgraded.

In Exchange 2007 SP2 we added a registry key called OALGEN_REG_V4_ALT_TEMP_FILES_LOCATION which will allow you to change the temporary storage location for the OAB Version 4 temporary files. This registry key will *ONLY* work for OAB Version 4. OAB versions 2 and 3’s temporary files will still go to the %temp% directory. If this registry key is not present on the OAB’s generating system we will revert back to the %temp% directory.

Being that OAB Version 4 is generated slightly different and uses different compression algorithms we can allow for a larger number of users far exceeded what we could generate on legacy systems.

We have also added two new event log events for troubleshooting:

Event Type:    Informational
Category: OAL Generator
Source: MSExchangeSA
Event ID:    9400
Date:        2/6/2010
Time:        7:13:01 PM
User:        N/A
Computer:    OABGenServer
Description:

OALGen will use 'd:\storage\OabTempFiles' as the alternative temporary directory for generating the offline address book version 4 files.

and

Event Type:    Error
Category: OAL Generator
Source: MSExchangeSA
Event ID:    9401
Date:        2/6/2010
Time:        7:13:01 PM
User:        N/A
Computer:    OABGenServer
Description:

OALGen failed to open directory 'd:\storage\OabTempFiles' as the alternative location for intermediate files. The default location ‘c:\windows\temp’ will be used.

Before using this registry in a production environment I suggest that you test this out in your lab first.

Dave

Exchange 2007 and 2010 Address List Segregation changes

We’re in the process of validating the existing Exchange Address List Segmentation whitepaper against Exchange2010.  The whitepaper was written for Exchange 2007 and may need to be modified to support changes in Exchange 2010.  The current plan is for us *to provide support* for Exchange 2010 address list segregation once I have completed the white paper.

When I wrote the white paper for Exchange 2007 it was tested for Exchange 2007. Anyone that tries to use this white paper on an Exchange 2010 will run in to problems that we have no yet documented and or tested and might have to tear down their exchange infrastructure to correct the problem.

If you are going to remove Exchange 2010 you should not have any problems as long as you do not remove any older versions of Exchange. If you do choose to remove Exchange 2010 you may notice some left over (new 2010) containers in the active directory. These containers will not cause any issues and are now only removed if you removed all of the Exchange versions in the organization.

I am currently working on the white paper and working with development to make the necessary changes required due to changes in how Outlook makes directory connections.  Once the white paper is complete this will be supported.  Please be patient and please do not try to contact me with regards to when this white paper will be ready. I have a ton of debugging and architectural documentation to do, and I will post updates as they pop up.

Dave

PLEASE READ –> Events 9320 and 9359 on new installation of Exchange 2010

When you attempt to generate your OAB you will see the following event id’s logged in your application log:

Log Name:    Application
Source:          MSExchangeSA
Date:              12/1/2009 13:01:03 AM
Event ID:        9359
Level:             Warning
User:              N/A
Computer:     OABGenServer
Description:   OALGen truncated or dropped properties for entry ‘Dgoldman’ in address list '\Global Address List' because they exceeded the configured size limits for the version 4 offline address list.  The affected MAPI ids are:  8008. - \Default Offline Address Book

Here is what are MAPI ids 8cd8 and 8008 are?

• 0x8CD8000D = PR_EMS_AB_AUTH_ORIG                           PT_OBJECT
• 0x8008101F = PR_EMS_AB_IS_MEMBER_OF_DL_W         PT_MV_UNICODE

This event will be generated for each user read in the oab that is a member of a DL, and for all DL’s that have members.

The Problem

The problem is that the “member” and “memberOf” attributes are not a part of the truncated property list for regular groups. So when we are generating the OAB we will read this property, and we will not see it in the list and log the 9359 event.

The Fix

This is by DESIGN. These warnings are 100% expected and can be ignored. This has no ramifications on your oab generation process.

We will be taking a chance in the future to have the following behavior:

1. The 9320 Event ID that is related to truncation moved become a Medium level event log item
2. The 9359 Event ID will be changed from the Highest category to Expert

NOTE: There is no time frame on when this change will take place. Please understand that the above two items are also subject to change without notice.  I will keep everyone posted via the blog.

Dave

IP_V6 is a must if you are trying to install Exchange 2010

I am getting ready to write the whitepaper for Exchange 2010 address list segregation and I ran in to a problem where my setup was failing with no errors. I decided to open up the application log and MSEXCHANGEADTOPOLOGYSERVICE.EXE was nice enough to throw the following error for me:

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: Topology
Event ID: 2114
User: N/A
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1936). Topology discovery failed, error 0x80040a02 (DSC_E_NO_SUITABLE_CDC). Look up the Lightweight Directory Access
Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Like everyone else I see on the internet I also went to the following KB article, and to my surprise the article was outdated and of little use. I will see what I can do to get this fixed for everyone.

From here I went to check my network card settings to make sure that my DNS servers were set properly and I found that my IPV6 settings were unchecked. This is a huge problem. The Exchange 2010 services need to have IPV6 enabled for them to start. Once I enabled the IPV6 the transport service started and then I was able to reinstall all of Exchange 2010.

NOTE: If you need to disable IP_V6 for any reason the correct way to do this would be to do the following:

1. Uncheck IPV6 in the network card properties on both the DC and the Exchange server. 
2. You must apply a registry key and reboot each machine to completely remove IPv6:

Key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters
Value Name: DisabledComponents
Value Type: DWORD
Value: FF

3. Once this key is set reboot both the your domain controller(s) and the Exchange server.
4. run an Ipconfig /all
5. On each machine and ensure you see no IPv6 addressing in the output.
6. Once this is complete, elevate diagnostic logging for ADAccess and NSPI, use the following cmdlets:

• get-eventloglevel –id “MSExchange ADAccess\*" | set-eventloglevel –level:medium
• Get-eventloglevel –id “MSExchangeSA\*" | set-eventloglevel –level:medium
• Get-eventloglevel –id “MSExchangeSA\NSPI Proxy" –Level:high

7. Recycle the ADAccess and System Attendant services, let the machine run for 5-10 minutes, and send me the resulting application log.

Dave

 

A new version of FindSearchFolder.exe has been posted to the internet and is available for download

Back on Tuesday, July 01, 2008 I wrote the following blog on Microsoft Exchange and Search Folders. For this blog I also wrote a tool that would parse the isinteg.pri file to help identify exchange mailboxes that have high item counts or a high number of search folders, which lead to massive exchange server performance issues. I have re-written this tool in managed code using Visual Studio 2010 and it has a ton more functionality.

I have left the old version of the tool on the internet, and have also added a new version that will run on Windows 2008, Vista and Windows 7 machines. The new filename is  FindSearchFolder09.7z and it can be found here: http://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=FindSearchFolders&ReleaseId=1456.

To run this version of the tool all you need to do is put the isinteg.pri file in the same directory as the tool and run it.

This is what the tools UI looks like. As you can see from the screen shot below, once a scan has run the tool will report back the different stages that it is in and how many mailboxes were found in the log.

Once the tool runs it will create the following directories:

All log files can be located in the Logs directory. As you can see from the following screen shot each time you run the tool it will create a new output file based on a time stamp. This version will no longer merge the results.

The output of the tool has changed as well. The new output looks like this:

 

10-28-2009  -  Find Search Folders Output - 10-28-2009-14.53.35.txt created. Program Logging - Enabled.
***************************************************************************************
Dave Goldman - Exchange Escalation Services
NOTE: There is no support for this tool and will be used as is!
Purpose: Scan an isinteg.pri file and find the highest search folder offenders
10-28-2009  -  Trace time: 10-28-2009-14.53.35
***************************************************************************************
10-28-2009  -  Program Initialization - Complete
10-28-2009  -  Create Program Directories - Start
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Logs\
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Backup\
10-28-2009  -  Create Program Directories - Complete
10-28-2009  -  Creating default mail handler - Complete
10-28-2009  -  Initializing Isinteg parser  - Complete
10-28-2009  -  Getting current application run path: D:\Tools\FindSearchFolders\Release
10-28-2009  -  Starting Isinteg parse - Start
10-28-2009  -  Search size set to: 75
10-28-2009  -  Isinteg is from: Server name: INCLCW02A.corp.statestr.com
10-28-2009  -  Isinteg was run against database: Database name: First Storage Group\Mailbox Store5 (INEMAL16)
10-28-2009  -  Processing Mailbox Table
10-28-2009  -  Display Name = Not Found - MBX Table dump only               

10-28-2009  -  Display Name = Goldman, Dave
                      Parent FID=0001-000001AB62E5
                      Root FID=0001-000001AB62E4
                      Restriction=
                      Name=Contacts
                      Folder FID=0001-000001AB1F4E               
                      Folder Type=1
                      Search Folder Count = 357  <- Is over 75 search folders and this is a BIG problem!
                      Backlink Folder Count = 357

10-28-2009  -  Processing Mailbox Table
10-28-2009  -  ==========================================================
10-28-2009  -  Mailbox Table
10-28-2009  -  ==========================================================
10-28-2009  -  Mailbox - Goldman, Dave - RootFID=0015-000012D75D62
10-28-2009  -  ==========================================================
10-28-2009  -  Total number of mailboxes with over 75 search folders: 1
10-28-2009  -  Highest number of search folders found in a single mailbox: 357
10-28-2009  -  Parent Folder: Parent FID=0001-000001AB62E5
10-28-2009  -  Highest offender is: Root 0001-000001AB62E4
10-28-2009  -  Total folders that do not contain search folders: 0
10-28-2009  -  Total mailboxes found on this information store: 1554
10-28-2009  -  ==========================================================
10-28-2009  -  ==========================================================
10-28-2009  -  These are the possible folder types
10-28-2009  -  ==========================================================
10-28-2009  -  Folder_Root = Folder Type = 0
10-28-2009  -  Normal_Folder = Folder Type = 1
10-28-2009  -  Search_Folder = Folder Type = 2
10-28-2009  -  MDB_Attachment_Folder = Folder Type = 4
10-28-2009  -  Repl_Folder = Folder Type = 5
10-28-2009  -  Categorization_Folder = Folder Type = 7
10-28-2009  -  ==========================================================
10-28-2009  -  Stat totals
10-28-2009  -  ==========================================================
10-28-2009  -  Total type Root Folders found: 1554
10-28-2009  -  Total type Normal folders found: 55580
10-28-2009  -  Total type Search Folders found: 1
10-28-2009  -  Total type MDB Attachment Folders found: 0
10-28-2009  -  Total type Repl Folder found: 0
10-28-2009  -  Total type Categorization (Views) Folders found: 4885
10-28-2009  -  ==========================================================

10-28-2009  -  NOTE: Folders with a type of 7 are cached views in the information store. For more information please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx To expire cached views from the information store, please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx

10-28-2009  -  Who has the highest search folder count in an isinteg dump
10-28-2009  -  ==========================================================
10-28-2009  -  1. Take the highest offenders RootFID and search the isinteg.pri file for this
10-28-2009  -  2. Look in the ESM and match these items counts and you should have your offending mailbox.
10-28-2009  -  Isinteg parse - completed

Dave

How can we try to control the network consumption by using OAB Throttling?

The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.

Example 1: If a public store that supports 10,000 users receives 3,000 requests in one hour and the offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data. This traffic could potentially overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This traffic could potentially overload a gigabit-per-second LAN for longer than one hour.

How do we calculate the largest OAB message size?
1.To find out the biggest  message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your oab files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state.

Here is an example of the Oabfldcheck summary:

Scan Completed
+------------------+
Message Class Normal found: 2
Message Class Differential found: 175
Message Class Unknown found: 0
Message Attachments found: 503
Messages found but unable to read the properties: 0
System folders found: 3
Highest sequence number found: 231
Lowest sequence number found: 16
Biggest attachment found: 16700 Bytes
Smallest attachment found: 102 Bytes
Biggest message found: 60234440 Bytes 
Smallest message found: 106 Bytes

You can then take the size of the largest message (here it is 60MB) x total users downloading the OAB x request per hour = number of data to be delivered by the server.

To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool

How to enable OAB Throttling
Use one of the following methods to determine the OAB Bandwidth Threshold data value to use:

Method 1 – Less Accurate
1. Set the value preemptively. *Based on your knowledge of the network topology* and how full offline Address Book downloads from multiple servers can add up to overload the LAN, set the thresholds on the individual servers to the largest values that are consistent with not overloading the LAN.

Bandwitdh Example: Type 5000 to configure the server to use 5000 kilobytes per second (KBps) as the bandwidth threshold for offline Address Book download throttling. 5000 KBps is approximately 40,960 kilobits per second (Kbps), or 40.96 megabits per second (Mbps).

Method 2 – Most Accurate
1. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that overloads the LAN. This can be a short period of time.
2. Make note of the bandwidth used by offline Address Book downloads on each server.
3. Set the threshold values to a percentage of those values. For example, set the threshold values to 60% of those values.

To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.

Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.

***Use Registry Editor at your own risk***
1. Click Start, click Run, type regedit in the Open box, and then click OK. 
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\ 
3. On the Edit menu, point to New, and then click DWORD Value. 
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify. 
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.

Things you need to Understand about OAB Throttling

1. To return to the default behavior, delete the registry value.

2. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.

3. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.

4. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:

• If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.

• If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.

5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.

Things not to do when using OAB Throttling

1. You will need to monitor this every 2 to 3 hours to recalculate the numbers. This is due to the clients succeeding in downloading the OAB.
2. Do not try to change the exchange servers network cards to some different lan speed settings. This will only cause more problems.
3. Do not restart the exchange server as this will just rebuild the OAB and start the process over again.
4. Do not move the OAB to a different server as this will just rebuild the OAB and start the process over again.
5. Do not try to stop the process. The damange has already happened and you need to ride it out!

Dave

Event ID 9373 is logged in the Application log when you update the offline address book in Exchange Server 2007

This was an article that I wrote about two years ago. I have high lighted a section in red which I think is very cool. You can rebuild your web distribution off of your public folder OAB without creating a new OAB.

When you update the offline address book by using the Update-OfflineAddressBook command or by using a scheduled task, the following event (Event ID 9373) is logged in the Application log on a computer that is running Microsoft Exchange Server 2007:

Event Type: Error
Event Source: MSExchangeSA
Event Category: OAL Generator
Event ID: 9373
Description: OALGen detected that the file '\\<SERVER name>\ExchangeOAB\<folder name>\a58e388b-7b23-4944-b042-58a7d0c6590f-data-1.lzx' is corrupted or missing. This indicates data tampering or disk problems. Restore files in this folder from the recent backup or clean up folder content and force a full OAB generation. - Default Offline Address Book
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Cause -This issue occurs when one or more OAB Version 4 offline address book files are corrupted or missing. In this case, the Microsoft Outlook clients cannot download the updated files until the issue is resolved.

RESOLUTION

To resolve this issue, use one of the following methods.

Method 1: Restore the contents of the OABGen folder

If daily backups are available, restore the contents of the OABGen folder to the following folder:

ExchangeInstallDir \ExchangeOAB\ OAB GUID

Note ExchangeInstallDir is a placeholder for the name of the Exchange Server 2007 installation directory. OAB GUID is a placeholder for the name of the folder.

Method 2: Rebuild the offline address book files by using Exchange Management Console or the Exchange Management Shell

If the corrupted offline address book item is enabled for public folders distribution, and if the data was published in public folders before the offline address book item was corrupted, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following method highlighted in red below:

• Use Exchange Management Console to rebuild the offline address book files. To do this, follow these steps:

1. Open Exchange Management Console.
2. Expand Organization Configuration, and then expand Mailbox.
3. On the details panel, click Offline Address Book, and then select the offline address book item that you want to update.
4. On the Action panel, click Update, and then click Yes.

•Use the Update-OfflineAddressBook command to rebuild the offline address book files.
To do this, run the following command in the Exchange Management Shell.

Update-OfflineAddressBook <OAB GUID> 

Note You can run the Update-OfflineAddressBook command on a computer that has the Mailbox, Client Access, or Hub Transport server role installed. To do this, you must log on by using an account that is a member of the Exchange Organization Administrators group. The account must also be a member of the local Administrators group on that computer.

 

Method 3: Rebuild the offline address book files by using a command

If neither daily backups nor public folders distribution is available, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following commands.

Update-OfflineAddressBook -Identity MyOAB 

Note In this example, the Update-OfflineAddressBook command is used to update the offline address book file that is named MyOAB.

Update-OfflineAddressBook <OAB GUID> 

Note When you use this command, the offline address book will be rebuilt. This will cause all clients to process a full offline address book download. For OAB Version 2 and OAB Version 3a, a full offline address book download can cause massive network traffic between client access servers and clients.

Dave

The Case of the Mysterious Exchange Server Hang

I have been working an interesting case for a large enterprise customer with a few other people from my group. We were having a problem where outlook clients were unable to connect to the exchange server, however the exchange server was operating like normal. After looking at the ESE internal structures I could see that the information store was not backed up on any I/O requests. As I started to debug a number of store dumps, I saw two interesting things that stood out. Below is a RPC fault, which is very rare in a store dump.

0:261> kbnL
# ChildEBP RetAddr  Args to Child             
00 61d9fc18 7c827d29 77e61d1e 000089dc 00000000 ntdll!KiFastSystemCallRet
01 61d9fc1c 77e61d1e 000089dc 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
02 61d9fc8c 77cdbedf 000089dc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
03 61d9fca8 77cdbf39 61d9fd20 000089dc 00000000 rpcrt4!UTIL_WaitForSyncHTTP2IO+0x18
04 61d9fcd0 77cccb39 67d1f18c 61d9fd20 000089dc rpcrt4!UTIL_GetOverlappedHTTP2ResultEx+0x20
05 61d9fcf8 77ccf2e8 67d1f18c 61d9fd1c 67d1f18c rpcrt4!WaitForSyncSend+0x26
06 61d9fd5c 77cd05ac 00000020 61d9fdd0 00000001 rpcrt4!HTTP2VirtualConnection::SyncSend+0xdd
07 61d9fd7c 77ccc1f6 00000020 61d9fdd0 00000001 rpcrt4!HTTP2ServerVirtualConnection::SyncSend+0x2a
08 61d9fd98 77c689e8 67d1f18c 00000020 61d9fdd0 rpcrt4!HTTP_SyncSend+0x2a
09 61d9fdc0 77ca9db9 61d9fdd0 00000020 03030005 rpcrt4!OSF_SCONNECTION::TransSend+0x74
0a 61d9fe08 77caa3cf 00000005 00000000 00000e1d rpcrt4!OSF_SCONNECTION::SendFault+0xb5
0b 61d9fe30 77c93e4e 00000005 00000000 67c57af8 rpcrt4!OSF_SCALL::CleanupCallAndSendFault+0x46
0c 61d9fe60 77c68b5e 67c57af8 030001f0 0000000a rpcrt4!OSF_SCALL::ProcessReceivedPDU+0x4a6
0d 61d9fe80 77c6e8db 00000000 000001f0 00000000 rpcrt4!OSF_SCALL::BeginRpcCall+0x194
0e 61d9fee0 77c6e7b4 00000000 67c57af8 000001f0 rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x435
0f 61d9fef4 77cae695 00091918 0000001c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
10 61d9ff18 77c7b799 00091918 0000001c 00000000 rpcrt4!ProcessComplexTReceive+0x54
11 61d9ff84 77c7b9b5 61d9ffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
12 61d9ff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
13 61d9ffac 77c7b110 0008f5f8 61d9ffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
14 61d9ffb8 77e6482f 6a084fa0 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
15 61d9ffec 00000000 77c7b0f5 6a084fa0 00000000 kernel32!BaseThreadStart+0x34

and a ton of threads calling in to “secur32!CallSPM”.

0:329> kbn
# ChildEBP RetAddr  Args to Child             
00 6ccafbe4 7c827899 76f52135 000006f4 6ccafc24 ntdll!KiFastSystemCallRet
01 6ccafbe8 76f52135 000006f4 6ccafc24 6ccafc24 ntdll!NtRequestWaitReplyPort+0xc
02 6ccafc04 76f55173 0009ad70 6ccafc24 6ccafc24 secur32!CallSPM+0x23
03 6ccafd30 76f54eb5 00000000 0009ad70 6ccafdac secur32!SecpAcceptSecurityContext+0x23c
04 6ccafdc0 76f54d63 00000000 6ccafe00 6ccafef0 secur32!LsaAcceptSecurityContext+0x15d
05 6ccafe10 77c8b90e 0010a090 67b31e68 6ccafef0 secur32!AcceptSecurityContext+0xa8
06 6ccafe58 77c8b8b8 00000010 6ccafef0 00000000 rpcrt4!SECURITY_CONTEXT::AcceptThirdLeg+0x42
07 6ccafea0 77c8b1c5 00000010 6ccafef0 00000000 rpcrt4!OSF_SCONNECTION::AcceptThirdLeg+0x61
08 6ccaff04 77c6e7b4 00000000 69ca4c68 000000ce rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x594
09 6ccaff18 77c7b799 00091918 0000000c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
0a 6ccaff84 77c7b9b5 6ccaffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
0b 6ccaff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
0c 6ccaffac 77c7b110 0008f5f8 6ccaffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
0d 6ccaffb8 77e6482f 5ac85d28 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
0e 6ccaffec 00000000 77c7b0f5 5ac85d28 00000000 kernel32!BaseThreadStart+0x34

Based on the “secur32!CallSPM” call stack, this tells me that we are processing authentication requests. These requests need to go to the domain controller for processing, and could potentially be proxied to another domain controller for processing. Being there could be a latency here I contacted my good friend Tate Calhoun up in the Platforms group so he could take a look at the LSASS dumps for confirmation. Tate is a Platforms EE debugging god, so I know I was in good hands here :) From the LSASS dumps Tate was able to tell me that we have 199 call stacks found containing : "netlogon!NLPUSERVALIDATEHIGHER"

0:061> kbn fff
#   ChildEBP RetAddr  Args to Child             
00  02e3ec98 7c827d29 77e61d1e 000005b4 00000000 ntdll!KiFastSystemCallRet
01  02e3ec9c 77e61d1e 000005b4 00000000 02e3ece0 ntdll!NtWaitForSingleObject+0xc
02  02e3ed0c 77e61c8d 000005b4 0000afc8 00000000 kernel32!WaitForSingleObjectEx+0xac
03  02e3ed20 74266947 000005b4 0000afc8 000e0270 kernel32!WaitForSingleObject+0x12
04  02e3ed3c 74266e4c 000a9f70 0000afc8 000e0270 netlogon!NlAllocateClientApi+0x53
05  02e3edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x5d
06  02e3ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
07  02e3eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
08  02e3eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
09  02e3f720 76c9a815 ffffffff 00000003 03b61f40 msv1_0!LsaApLogonUserEx2+0x1106
0a  02e3fa80 76c95302 00607a20 01b47508 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
0b  02e3fc98 4ab85ebb 00607a20 00b48f80 02e3fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
0c  02e3fd0c 4ab860a8 01b474c0 01b474c8 02e3fe40 lsasrv!WLsaAcceptContext+0x139
0d  02e3fe84 4ab7ae1b 01b47498 00000000 005ed200 lsasrv!LpcAcceptContext+0x13b
0e  02e3fe9c 4ab7ad1e 01b47498 4ac22738 05924808 lsasrv!DispatchAPI+0x46
0f  02e3ff54 4ab7a769 01b47498 02e3ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
10  02e3ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
11  02e3ffb8 77e6482f 006bee48 00000000 00000000 lsasrv!LsapThreadBase+0x91
12  02e3ffec 00000000 4ab8f40e 006bee48 00000000 kernel32!BaseThreadStart+0x34

So, we are looking at authentication requests going to the LSASS process. LSASS has two threads that process authentication requests for the operating system. Here is one of the two allowed to proceed….

0:093> kbn fff
#   ChildEBP RetAddr  Args to Child             
00  0363e618 7c827d29 77e61d1e 00001541 00000001 ntdll!KiFastSystemCallRet
01  0363e61c 77e61d1e 00001541 00000001 0363e660 ntdll!NtWaitForSingleObject+0xc
02  0363e68c 77c6a85f 00001541 00057e40 00000001 kernel32!WaitForSingleObjectEx+0xac
03  0363e6a8 77c69bf7 0167b15c 00000001 00057e40 rpcrt4!UTIL_WaitForSyncIO+0x20
04  0363e6cc 77c59546 0167b124 0167b15c 0363e6fc rpcrt4!UTIL_GetOverlappedResultEx+0x1d
05  0363e700 77c6af6a 00000001 0363e84c 0363e848 rpcrt4!WS_SyncRecv+0xbe
06  0363e728 77c69667 00000000 00ab8f70 000001a0 rpcrt4!OSF_CCONNECTION::TransSendReceive+0xb8
07  0363e7b0 77c695d4 10000000 01688228 00000001 rpcrt4!OSF_CCONNECTION::SendFragment+0x2ae
08  0363e808 77c6977a 00000000 00000160 0363e84c rpcrt4!OSF_CCALL::SendNextFragment+0x1e2
09  0363e850 77c699f2 0363e8d8 0363e894 00000001 rpcrt4!OSF_CCALL::FastSendReceive+0x148
0a  0363e86c 77c69975 0363e8d8 0363e894 0363e8d8 rpcrt4!OSF_CCALL::SendReceiveHelper+0x5b
0b  0363e89c 77c7fcf0 00ab8f88 0363e8bc 77c80673 rpcrt4!OSF_CCALL::SendReceive+0x41
0c  0363e8a8 77c80673 0363e8d8 71c43dd0 0363ecc4 rpcrt4!I_RpcSendReceive+0x24
0d  0363e8bc 77ce315a 0363e904 00ab90e8 00000000 rpcrt4!NdrSendReceive+0x2b
0e  0363eca4 71c4e665 71c43dd0 71c4b2e4 0363ecc4 rpcrt4!NdrClientCall2+0x22e
0f  0363ecbc 71c4e620 01b2ee68 03985a38 000e0300 netapi32!NetrLogonSamLogonEx+0x1c
10  0363ed1c 74266fdc 01b2ee68 03985a38 000e0300 netapi32!I_NetLogonSamLogonEx+0x41
11  0363edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x2d5
12  0363ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
13  0363eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
14  0363eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
15  0363f720 76c9a815 ffffffff 00000003 00b55030 msv1_0!LsaApLogonUserEx2+0x1106
16  0363fa80 76c95302 00607708 01b05188 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
17  0363fc98 4ab85ebb 00607708 017920d0 0363fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
18  0363fd0c 4ab860a8 01b05140 01b05148 0363fe40 lsasrv!WLsaAcceptContext+0x139
19  0363fe84 4ab7ae1b 01b05118 00000000 005ed050 lsasrv!LpcAcceptContext+0x13b
1a  0363fe9c 4ab7ad1e 01b05118 4ac22738 01b420b8 lsasrv!DispatchAPI+0x46
1b  0363ff54 4ab7a769 01b05118 0363ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
1c  0363ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
1d  0363ffb8 77e6482f 006bf2c8 00000000 00000000 lsasrv!LsapThreadBase+0x91
1e  0363ffec 00000000 4ab8f40e 006bf2c8 00000000 kernel32!BaseThreadStart+0x34

Now the question remains, why are we having a backlog. Mike wrote a detailed blog describing these findings and how to fix. Please see Mike Lagase’s blog: http://blogs.technet.com/mikelag/archive/2009/08/04/the-case-of-the-mysterious-exchange-server-hang.aspx

Dave

When downloading the OAB from an Exchange 2007 server running on Windows Server 2008 it can take forever, or you can be prompted repeatedly for credentials

We released an update for Rollup 8 for Exchange Server 2007 Service Pack 1 (KB 968012) back on May 26th, 2009. When Exchange 2007 is running on a Windows Server 2008, the Exchange 2007 clients may be repeatedly prompted for their credentials during Outlook Anywhere session, or an OAB download will be perceived to hang forever.

This issue occurs when NTLM Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box for the Outlook profile on the client computer. This issue does not occur if Basic Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box.

By default, Kernel Mode Authentication is enabled in Internet Information Services (IIS) 7.0 on the Client Access server. To resolve this issue, disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008.

The Fix

To disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008

• At a command prompt, type the following command, and then press ENTER:

  %systemroot%\system32\inetsrv\AppCmd.exe set config /section:system.webServer/security/authentication/windowsAuthentication /useKernelMode:false

Running the following command will verify that the changes have taken place:

C:\Windows\System32\inetsrv>appcmd.exe list config /section:WindowsAuthentication

<system.webServer>
   <security>
     <authentication>
       <windowsAuthentication enabled="false" useKernelMode="false">
         <providers>
           <add value="Negotiate" />
           <add value="NTLM" />
         </providers>
       </windowsAuthentication>
     </authentication>
   </security>
</system.webServer>

NOTE: Installing an Exchange Rollup or IU should never change a web.config file on a server. Running this command is a manual step that must be preformed, and is outlined in the following TechNet documentation:

http://technet.microsoft.com/en-us/library/bb123889.aspx

Dave

Global Address list shows the legacyExchangeDN instead of the email address

There are certain situations where the Global Address list display will show the legacyExchangeDN instead of the SMTP email address. The one condition where this is true is when you are viewing the Global Address List through Outlook 2003. The default behavior for Outlook 2003 is to use the EX address type for the Email Address column.

 

The default behavior for Outlook 2007 is to use the SMTP email address.

The best thing you can do is look at the users attribute in the active directory if you are not sure, because the mail attribute should contain a SMTP proxy address. If it contains a legacyExchangeDN, then this is incorrect. Here is where you can run in to problems. The OAB Generation process will turf an object when the mail attribute and primary proxy address for a user does not match. Running OABInteg can help you figure this out. You can run OABInteg /s:srvName /t:proxytest /v:2 /l. At the end of the output you will see something that looks like this:

Scan Finished
Total number of entries processed in the address book: 241004
Total number of entries skipped: 0
Total number of contacts: 12868
Total number of mailboxes: 95214
Total number of distribution lists: 131193
Total number of groups: 0
Total number of folders: 282
Total number of Address Book Container objects found: 0
Total number of temp legacyExchangeDN's found: 0
Total number of objects that are missing some main attributes: 0
Total number of objects that mail and proxy attribute don't match: 1
Total number of objects that do not have a domain value: 0
Total number of objects that do not have a valid unicode domain value: 0
Total number of objects that do not have a valid SMTP Domain because first character is not greater than '/': 0
Total number of objects that do not start with /o= or /O=: 1
Total user objects that are missing the Primary Proxy address attribute: 0
Total user objects with proxy addresses equal to or over 64 characters: 0
Total objects with the '@' character in the legacyExchangeDN: 0
Total objects with bad Active Directory Backlinks: 0
Total objects that have a legacyExchangeDN of ADCDisableMail: 0
Total objects that have a legacyExchangeDN of ADCDisableMailByADC: 0

I highlighted in bold red what is the important information.

Here is a good object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
Primary Proxy Address and mail attribute match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dgoldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

Here is a bad object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
ERROR - Primary Proxy Address and mail attribute do not match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dave.goldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

For more information on the proxytest command: http://blogs.msdn.com/dgoldman/archive/2007/03/08/how-to-use-oabinteg-s-oabfldcheck-and-proxytest-to-find-oab-issues.aspx

So to recap, if you are seeing the legacyExchangeDN in the Global Address List display, you should verify it the mail attribute is correct. If not, you will want to get this fixed. This can be done via logon script, using tools like ADModify, etc.

Dave

Unable to delete the OAB system folders in Exchange 2007?

Often we will get question that have to deal with deleting system folders in Exchange 2007. When deleting an OAB public folder from Exchange Management console, you receive the following error: 0x80070005. This error means "Access Denied." You will still receive this error even if you use an account that has Exchange Organization administrator permissions.

 

Logging will also show the following:

Action 'Remove' could not be performed on object - ‘/o=Organization/cn=addrlists/cn=oabs/cn=OAB'.

Error: The folder '\NON_IPM_SUBTREE\OFFLINE ADDRESS BOOK\ /o=Organization/cn=addrlists/cn=oabs/cn=OAB' or some of its subfolder(s) encountered errors and could not be deleted. MapiExceptionPartialCompletion: Unable to delete folder. (hr=0x40680, ec=0)

Diagnostic context:
    Lid: 18969   EcDoRpcExt2 called [length=54]
    Lid: 27161   EcDoRpcExt2 returned [ec=0x0][length=85][latency=0]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropDeleteFolder [29]
    Lid: 17082   ROP Error: 0x80070005
    Lid: 19297 
    Lid: 21921   StoreEc: 0x80070005
    Lid: 27962   ROP: ropNone [0]
    Lid: 26881 
    Lid: 21817   ROP Failure: 0x80070005
    Lid: 24721 
    Lid: 20625   StoreEc: 0x80070005

Cause

This is by design. You cannot manually create or manually delete an OAB root folder. These system folders are handled by the information store. When you create an OAB object in the directory via powershell or the Exchange Management Console, the owning information store will create the OAB root folder when site folder maintenance is run. When you delete an OAB object from the active directory, the information store will mark the OAB folder as unused during site folder maintenance. If it goes unused for 7 days, it is deleted by the store during site folder maintenance.

Dave

Full OAB downloads caused by mis-configured anti virus software settings

Over the last few months I have seen a rash of calls come in where clients were being forced to download a full.  Full OAB downloads can cause very high network usage and cause more havoc than anything.

Typically in these situations it is pretty easy to tell what is going on from the event errors that are logged. The two following events are as follows:

Event ID     : 9116
Raw Event ID : 9116
Record Nr.   : 860555
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error while generating the binpatch.oab file for differential downloads of address list '\Global Address List'.  Clients will not be able to incrementally update to the new version of the offline address list, they will perform a full download instead.  This is normal if this is the first time this offline address list has been generated.  Check other logged events to see if this is a serious error.
- Default Offline Address List

NOTE: This is the first indication that you have a problem! You should never see a 9116 event any other time then when you generate your OAB for the first time.

Event ID     : 9109
Raw Event ID : 9109
Record Nr.   : 860554
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error ffffffff (internal ID 50303f6) while generating address list '\Global Address List'.  Check other logged events to see if this is a serious error.
- Default Offline Address List

The key here is to check your anti-virus software to make sure that none of the OAB files have been quarantined. Being that the details.oab file is the biggest OAB file, it has the potential to get looked at more by anti-virus software than the other OAB files due to it’s size and usage. More evidence that the anti virus software had foul play here is to look at the data.oab file. In doing so you will see that it might contain one attachment called Deleted attachment.txt.

When this happens the client can not apply the difference file which results in the 9116 which most of the time is a normal event if you just built your OAB for the first time.

The Fix

1. Make sure your anti virus software has the proper exclusions set to not monitor the exchange OAB directories on your exchange server.
2. Make sure on the client you are excluding the *.oab files.

Dave

Forward to Internet recipients without internal mail-enabled contacts no longer works on Exchange 2007 – Part 2

I was recently asked by a HMC customer to see if this solution [http://blogs.msdn.com/dgoldman/archive/2008/04/09/how-to-forward-to-internet-recipients-without-creating-an-internal-mail-enabled-contact.aspx] would work for Exchange 2007 as it did work on Exchange 2003. The answer to this is *no*, this will not work. Being that the Exchange 2007 transport is a rewrite from 2003 we do not process the forwardingAddress of a user object, and this is by design.

 

I decided to set up a repro for this using the following steps:

1. Installed 3 Exchange 2007 servers. (1 Client Access Server, 1 Mailbox Server and 1 Hub Server.
2. In the Exchange System Manager I clicked on the Organization Configuration | Remote Domains Tab | Format of original message sent as attachment to journal report tab | click "Allow automatic forward"
3. Created a new mailbox called mailForward.
4. Using ADSI Edit connect to Schema, then navigate to Schema attribute ms-Exch-Forwarding-Address properties and set isMemberOfPartialAttributeSet to True 5. Now using ADSiEdit and navigate to the user account in the domain naming context and set forwardingAddress attribute on mailbox user’s properties. I changed the external recipient address to which you want the mail forwarded (for example dgoldman@hotmail.com)

Did my test on RTM and it was a no go. I then installed the latest services packs and hot fixes to confirm there were no changes between RTM and the latest Rollup.

Sending the message to the user will produce the following results if you look at the dump out the queues using Get-MessageTrackingLog

… I cut the rest out for readability

RECEIVE  STORE... mailForward@tpworld.l... {mailForward@domain.com... test
DELIVER  STORE... mailForward@tpworld.l... {mailForward@domain.com... test

As you can see the information store received and delivered the message which is correct, however you do not see an email being set to dgoldman@hotmail.com.

Checking the queues with the Exchange Management Console shows the following:

[PS] C:\Documents and Settings\exadmin>Get-Queue | fl

Identity         : E2K7-HUB\Submission
DeliveryType     : Undefined
NextHopDomain    : Submission
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 0
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

If the message was bifurcated you would expect to see an Unreachable queue for the message that was destined for hotmail.

I created a separate message to a false user: badmail@hotmail.com and sent this from the mailbox and you then see the following:

Identity         : E2K7-HUB\Submission
DeliveryType     : Undefined
NextHopDomain    : Submission
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 0
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

Identity         : E2K7-HUB\Unreachable
DeliveryType     : Unreachable
NextHopDomain    : Unreachable Domain
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 1
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

As a follow up I checked with the transport team and they in fact agree that this will no longer work and is by design.

Dave

A new build of OABInteg has been posted to the web

I complied a new build of OABInteg that has a fix for a problem when someone runs the tool with an LDAP test and the customer has DN’s that exceed 64 characters. The following error returned is: ERROR_DS_REFERRAL.

Command line arguments: OABInteg /s:dc01 /t:oaltest /v:2 /l

Program started at: 12:40:26 PM

Running OABInteg on: OABServer1\Administrator

Trying to connect to: GC://dc01.company.com object found: Hosting

Checking to see if OABInteg is running on an Exchange Server

Checking version of Exchange Server

Exchange 2003 Version: 6.5.6944.00

Exchange 2003 Service Pack Version: 6.5.7638.2.

Failure: ADsOpenObject

ADSI Error: hr = 0x8007202b - LDAP_REFERRAL - ERROR_DS_REFERRAL: Cannot resolve referral.

Function: OfflineAddressListTest

Line number: 646

The new build has been posted to http://code.msdn.microsoft.com/oabinteg/Release/ProjectReleases.aspx?ReleaseId=726

Dave

The OAB Generation process fails to generate an OAB with error MAPI_E_FAILONEPROVIDER

I just finished troubleshooting an error where we were generating multiple OAB's on a server with 6 OAB's and one of them would always fail with error Event 9175 (MSExchangeSA). I want to provide a small tidbit on how the system attendant obtains it's connection objects.

Here is a little information on the connection logic.

1. When the system attendant starts up it will attempt to make a connection to the information store so it can get a session object.
2. This session object is for the public folder store that this system attendant is associated with. This session object will be used for connecting to the public folder store when we need to access it for OAB, Free/Busy, etc.
3. Each time we need to connect to the information store we use the same session object and we will log the information about how many times we connect and how long they take to connect.

Now I needed to figure out why only the last one would fail. Through creating debug builds of store and system attendant so I could output maximum logging that was relevant to the problem I was able to obtain some information. Looking at the traces I could see that we would always return error: 0x478.

Dumping out this error I can see that we get ecWrongServer.

C:\err 0x478
# for hex 0x478 / decimal 1144
  ecWrongServer   ec.h
# 1 matches found for "0x478"

Now the question is why are we returning ecWrongServer. The information store has logic built in to it where we get redirected to another information store we will detect this change and try to connect where we are connecting too. This can happen for multiple reasons, for one too many connections in a short time. The system attendant will connect to the information store via the RPC interface and call the following function emsmdb!CNCT::EcForceRpc(). Through debugging I was able to see that we always got a return value of [0x47f = ECSERVERPAUSE].

Through code review I was able to see that when a client connects to a server it isn't configured for we will get an error code of ecWrongServer. This error code tells us that this is the wrong server and we will do a query to  determine the correct server. Once we have this information we give the client the correct server to connect too so we can attempt to connect again. In certain situations this could lead to a looping connection condition and cause RPC problems. To correct this behavior we added some logic we that keeps track of how many ecWrongServer errors have been sent to a particular client.

In addition to this if a client tries to make RPC calls more than 5 times under 10 seconds we will automatically return the error code of ecServerPaused error. This error will bubble back up the stack and be mapped back to the following error code: MAPI_E_FAILONEPROVIDER(0x8004011d). In this case these observations show that there is nothing wrong with the OAB itself and has to do with too many connection attempts.

One solution to this problem is to stagger your OAB's so they are built at different times so we do not hit this logic, which is  a safety mechanism to protect the store from RPC DOS type attacks (ligitement or not).

Dave