Dgoldman's WebLog

IP_V6 is a must if you are trying to install Exchange 2010

dgoldman — Tue, 03 Nov 2009 21:25:56 GMT

I am getting ready to write the whitepaper for Exchange 2010 address list segregation and I ran in to a problem where my setup was failing with no errors. I decided to open up the application log and MSEXCHANGEADTOPOLOGYSERVICE.EXE was nice enough to throw the following error for me:

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: Topology
Event ID: 2114
User: N/A
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1936). Topology discovery failed, error 0x80040a02 (DSC_E_NO_SUITABLE_CDC). Look up the Lightweight Directory Access
Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Like everyone else I see on the internet I also went to the following KB article, and to my surprise the article was outdated and of little use. I will see what I can do to get this fixed for everyone.

From here I went to check my network card settings to make sure that my DNS servers were set properly and I found that my IPV6 settings were unchecked. This is a huge problem. The Exchange 2010 services need to have IPV6 enabled for them to start. Once I enabled the IPV6 the transport service started and then I was able to reinstall all of Exchange 2010.

NOTE: If you need to disable IP_V6 for any reason the correct way to do this would be to do the following:

1. Uncheck IPV6 in the network card properties on both the DC and the Exchange server.
2. You must apply a registry key and reboot each machine to completely remove IPv6:

Key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters
Value Name: DisabledComponents
Value Type: DWORD
Value: FF

3. Once this key is set reboot both the your domain controller(s) and the Exchange server.
4. run an Ipconfig /all
5. On each machine and ensure you see no IPv6 addressing in the output.
6. Once this is complete, elevate diagnostic logging for ADAccess and NSPI, use the following cmdlets:

get-eventloglevel –id “MSExchange ADAccess\*" | set-eventloglevel –level:medium
Get-eventloglevel –id “MSExchangeSA\*" | set-eventloglevel –level:medium
Get-eventloglevel –id “MSExchangeSA\NSPI Proxy" –Level:high

7. Recycle the ADAccess and System Attendant services, let the machine run for 5-10 minutes, and send me the resulting application log.

Dave

A new version of FindSearchFolder.exe has been posted to the internet and is available for download

dgoldman — Wed, 28 Oct 2009 19:50:33 GMT

Back on Tuesday, July 01, 2008 I wrote the following blog on Microsoft Exchange and Search Folders. For this blog I also wrote a tool that would parse the isinteg.pri file to help identify exchange mailboxes that have high item counts or a high number of search folders, which lead to massive exchange server performance issues. I have re-written this tool in managed code using Visual Studio 2010 and it has a ton more functionality.

I have left the old version of the tool on the internet, and have also added a new version that will run on Windows 2008, Vista and Windows 7 machines. The new filename is FindSearchFolder09.7z and it can be found here: http://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=FindSearchFolders&ReleaseId=1456.

To run this version of the tool all you need to do is put the isinteg.pri file in the same directory as the tool and run it.

This is what the tools UI looks like. As you can see from the screen shot below, once a scan has run the tool will report back the different stages that it is in and how many mailboxes were found in the log.

Once the tool runs it will create the following directories:

All log files can be located in the Logs directory. As you can see from the following screen shot each time you run the tool it will create a new output file based on a time stamp. This version will no longer merge the results.

The output of the tool has changed as well. The new output looks like this:

10-28-2009 - Find Search Folders Output - 10-28-2009-14.53.35.txt created. Program Logging - Enabled.
***************************************************************************************
Dave Goldman - Exchange Escalation Services
NOTE: There is no support for this tool and will be used as is!
Purpose: Scan an isinteg.pri file and find the highest search folder offenders
10-28-2009 - Trace time: 10-28-2009-14.53.35
***************************************************************************************
10-28-2009 - Program Initialization - Complete
10-28-2009 - Create Program Directories - Start
10-28-2009 - Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder
10-28-2009 - Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Logs\
10-28-2009 - Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Backup\
10-28-2009 - Create Program Directories - Complete
10-28-2009 - Creating default mail handler - Complete
10-28-2009 - Initializing Isinteg parser - Complete
10-28-2009 - Getting current application run path: D:\Tools\FindSearchFolders\Release
10-28-2009 - Starting Isinteg parse - Start
10-28-2009 - Search size set to: 75
10-28-2009 - Isinteg is from: Server name: INCLCW02A.corp.statestr.com
10-28-2009 - Isinteg was run against database: Database name: First Storage Group\Mailbox Store5 (INEMAL16)
10-28-2009 - Processing Mailbox Table
10-28-2009 - Display Name = Not Found - MBX Table dump only

10-28-2009 - Display Name = Goldman, Dave
                      Parent FID=0001-000001AB62E5
                      Root FID=0001-000001AB62E4
                      Restriction=
                      Name=Contacts
                      Folder FID=0001-000001AB1F4E
                      Folder Type=1
                      Search Folder Count = 357 <- Is over 75 search folders and this is a BIG problem!
                      Backlink Folder Count = 357

10-28-2009 - Processing Mailbox Table
10-28-2009 - ==========================================================
10-28-2009 - Mailbox Table
10-28-2009 - ==========================================================
10-28-2009 - Mailbox - Goldman, Dave - RootFID=0015-000012D75D62
10-28-2009 - ==========================================================
10-28-2009 - Total number of mailboxes with over 75 search folders: 1
10-28-2009 - Highest number of search folders found in a single mailbox: 357
10-28-2009 - Parent Folder: Parent FID=0001-000001AB62E5
10-28-2009 - Highest offender is: Root 0001-000001AB62E4
10-28-2009 - Total folders that do not contain search folders: 0
10-28-2009 - Total mailboxes found on this information store: 1554
10-28-2009 - ==========================================================
10-28-2009 - ==========================================================
10-28-2009 - These are the possible folder types
10-28-2009 - ==========================================================
10-28-2009 - Folder_Root = Folder Type = 0
10-28-2009 - Normal_Folder = Folder Type = 1
10-28-2009 - Search_Folder = Folder Type = 2
10-28-2009 - MDB_Attachment_Folder = Folder Type = 4
10-28-2009 - Repl_Folder = Folder Type = 5
10-28-2009 - Categorization_Folder = Folder Type = 7
10-28-2009 - ==========================================================
10-28-2009 - Stat totals
10-28-2009 - ==========================================================
10-28-2009 - Total type Root Folders found: 1554
10-28-2009 - Total type Normal folders found: 55580
10-28-2009 - Total type Search Folders found: 1
10-28-2009 - Total type MDB Attachment Folders found: 0
10-28-2009 - Total type Repl Folder found: 0
10-28-2009 - Total type Categorization (Views) Folders found: 4885
10-28-2009 - ==========================================================

10-28-2009 - NOTE: Folders with a type of 7 are cached views in the information store. For more information please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx To expire cached views from the information store, please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx

10-28-2009 - Who has the highest search folder count in an isinteg dump
10-28-2009 - ==========================================================
10-28-2009 - 1. Take the highest offenders RootFID and search the isinteg.pri file for this
10-28-2009 - 2. Look in the ESM and match these items counts and you should have your offending mailbox.
10-28-2009 - Isinteg parse - completed

Dave

How can we try to control the network consumption by using OAB Throttling?

dgoldman — Wed, 21 Oct 2009 15:47:00 GMT

The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.

Example 1: If a public store that supports 10,000 users receives 3,000 requests in one hour and the offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data. This traffic could potentially overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This traffic could potentially overload a gigabit-per-second LAN for longer than one hour.

How do we calculate the largest OAB message size?
1.To find out the biggest message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your oab files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state.

Here is an example of the Oabfldcheck summary:

Scan Completed
+------------------+
Message Class Normal found: 2
Message Class Differential found: 175
Message Class Unknown found: 0
Message Attachments found: 503
Messages found but unable to read the properties: 0
System folders found: 3
Highest sequence number found: 231
Lowest sequence number found: 16
Biggest attachment found: 16700 Bytes
Smallest attachment found: 102 Bytes
Biggest message found: 60234440 Bytes
Smallest message found: 106 Bytes

You can then take the size of the largest message (here it is 60MB) x total users downloading the OAB x request per hour = number of data to be delivered by the server.

To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool

How to enable OAB Throttling
Use one of the following methods to determine the OAB Bandwidth Threshold data value to use:

Method 1 – Less Accurate
1. Set the value preemptively. *Based on your knowledge of the network topology* and how full offline Address Book downloads from multiple servers can add up to overload the LAN, set the thresholds on the individual servers to the largest values that are consistent with not overloading the LAN.

Bandwitdh Example: Type 5000 to configure the server to use 5000 kilobytes per second (KBps) as the bandwidth threshold for offline Address Book download throttling. 5000 KBps is approximately 40,960 kilobits per second (Kbps), or 40.96 megabits per second (Mbps).

Method 2 – Most Accurate
1. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that overloads the LAN. This can be a short period of time.
2. Make note of the bandwidth used by offline Address Book downloads on each server.
3. Set the threshold values to a percentage of those values. For example, set the threshold values to 60% of those values.

To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.

Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.

***Use Registry Editor at your own risk***
1. Click Start, click Run, type regedit in the Open box, and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\
3. On the Edit menu, point to New, and then click DWORD Value.
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify.
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.

Things you need to Understand about OAB Throttling

1. To return to the default behavior, delete the registry value.

2. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.

3. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.

4. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:

If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.
If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.

5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.

Things not to do when using OAB Throttling

You will need to monitor this every 2 to 3 hours to recalculate the numbers. This is due to the clients succeeding in downloading the OAB.
Do not try to change the exchange servers network cards to some different lan speed settings. This will only cause more problems.
Do not restart the exchange server as this will just rebuild the OAB and start the process over again.
Do not move the OAB to a different server as this will just rebuild the OAB and start the process over again.
Do not try to stop the process. The damange has already happened and you need to ride it out!

Dave

Event ID 9373 is logged in the Application log when you update the offline address book in Exchange Server 2007

dgoldman — Fri, 28 Aug 2009 22:52:46 GMT

This was an article that I wrote about two years ago. I have high lighted a section in red which I think is very cool. You can rebuild your web distribution off of your public folder OAB without creating a new OAB.

When you update the offline address book by using the Update-OfflineAddressBook command or by using a scheduled task, the following event (Event ID 9373) is logged in the Application log on a computer that is running Microsoft Exchange Server 2007:

Event Type: Error
Event Source: MSExchangeSA
Event Category: OAL Generator
Event ID: 9373
Description: OALGen detected that the file '\\<SERVER name>\ExchangeOAB\<folder name>\a58e388b-7b23-4944-b042-58a7d0c6590f-data-1.lzx' is corrupted or missing. This indicates data tampering or disk problems. Restore files in this folder from the recent backup or clean up folder content and force a full OAB generation. - Default Offline Address Book
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Cause -This issue occurs when one or more OAB Version 4 offline address book files are corrupted or missing. In this case, the Microsoft Outlook clients cannot download the updated files until the issue is resolved.

RESOLUTION

To resolve this issue, use one of the following methods.

Method 1: Restore the contents of the OABGen folder

If daily backups are available, restore the contents of the OABGen folder to the following folder:

ExchangeInstallDir \ExchangeOAB\ OAB GUID

Note ExchangeInstallDir is a placeholder for the name of the Exchange Server 2007 installation directory. OAB GUID is a placeholder for the name of the folder.

Method 2: Rebuild the offline address book files by using Exchange Management Console or the Exchange Management Shell

If the corrupted offline address book item is enabled for public folders distribution, and if the data was published in public folders before the offline address book item was corrupted, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following method highlighted in red below:

• Use Exchange Management Console to rebuild the offline address book files. To do this, follow these steps:

1. Open Exchange Management Console.
2. Expand Organization Configuration, and then expand Mailbox.
3. On the details panel, click Offline Address Book, and then select the offline address book item that you want to update.
4. On the Action panel, click Update, and then click Yes.

•Use the Update-OfflineAddressBook command to rebuild the offline address book files.
To do this, run the following command in the Exchange Management Shell.

Update-OfflineAddressBook <OAB GUID>

Note You can run the Update-OfflineAddressBook command on a computer that has the Mailbox, Client Access, or Hub Transport server role installed. To do this, you must log on by using an account that is a member of the Exchange Organization Administrators group. The account must also be a member of the local Administrators group on that computer.

Method 3: Rebuild the offline address book files by using a command

If neither daily backups nor public folders distribution is available, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following commands.

Update-OfflineAddressBook -Identity MyOAB

Note In this example, the Update-OfflineAddressBook command is used to update the offline address book file that is named MyOAB.

Update-OfflineAddressBook <OAB GUID>

Note When you use this command, the offline address book will be rebuilt. This will cause all clients to process a full offline address book download. For OAB Version 2 and OAB Version 3a, a full offline address book download can cause massive network traffic between client access servers and clients.

Dave

The Case of the Mysterious Exchange Server Hang

dgoldman — Tue, 04 Aug 2009 18:59:00 GMT

I have been working an interesting case for a large enterprise customer with a few other people from my group. We were having a problem where outlook clients were unable to connect to the exchange server, however the exchange server was operating like normal. After looking at the ESE internal structures I could see that the information store was not backed up on any I/O requests. As I started to debug a number of store dumps, I saw two interesting things that stood out. Below is a RPC fault, which is very rare in a store dump.

0:261> kbnL
# ChildEBP RetAddr Args to Child
00 61d9fc18 7c827d29 77e61d1e 000089dc 00000000 ntdll!KiFastSystemCallRet
01 61d9fc1c 77e61d1e 000089dc 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
02 61d9fc8c 77cdbedf 000089dc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
03 61d9fca8 77cdbf39 61d9fd20 000089dc 00000000 rpcrt4!UTIL_WaitForSyncHTTP2IO+0x18
04 61d9fcd0 77cccb39 67d1f18c 61d9fd20 000089dc rpcrt4!UTIL_GetOverlappedHTTP2ResultEx+0x20
05 61d9fcf8 77ccf2e8 67d1f18c 61d9fd1c 67d1f18c rpcrt4!WaitForSyncSend+0x26
06 61d9fd5c 77cd05ac 00000020 61d9fdd0 00000001 rpcrt4!HTTP2VirtualConnection::SyncSend+0xdd
07 61d9fd7c 77ccc1f6 00000020 61d9fdd0 00000001 rpcrt4!HTTP2ServerVirtualConnection::SyncSend+0x2a
08 61d9fd98 77c689e8 67d1f18c 00000020 61d9fdd0 rpcrt4!HTTP_SyncSend+0x2a
09 61d9fdc0 77ca9db9 61d9fdd0 00000020 03030005 rpcrt4!OSF_SCONNECTION::TransSend+0x74
0a 61d9fe08 77caa3cf 00000005 00000000 00000e1d rpcrt4!OSF_SCONNECTION::SendFault+0xb5
0b 61d9fe30 77c93e4e 00000005 00000000 67c57af8 rpcrt4!OSF_SCALL::CleanupCallAndSendFault+0x46
0c 61d9fe60 77c68b5e 67c57af8 030001f0 0000000a rpcrt4!OSF_SCALL::ProcessReceivedPDU+0x4a6
0d 61d9fe80 77c6e8db 00000000 000001f0 00000000 rpcrt4!OSF_SCALL::BeginRpcCall+0x194
0e 61d9fee0 77c6e7b4 00000000 67c57af8 000001f0 rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x435
0f 61d9fef4 77cae695 00091918 0000001c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
10 61d9ff18 77c7b799 00091918 0000001c 00000000 rpcrt4!ProcessComplexTReceive+0x54
11 61d9ff84 77c7b9b5 61d9ffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
12 61d9ff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
13 61d9ffac 77c7b110 0008f5f8 61d9ffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
14 61d9ffb8 77e6482f 6a084fa0 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
15 61d9ffec 00000000 77c7b0f5 6a084fa0 00000000 kernel32!BaseThreadStart+0x34

and a ton of threads calling in to “secur32!CallSPM”.

0:329> kbn
# ChildEBP RetAddr Args to Child
00 6ccafbe4 7c827899 76f52135 000006f4 6ccafc24 ntdll!KiFastSystemCallRet
01 6ccafbe8 76f52135 000006f4 6ccafc24 6ccafc24 ntdll!NtRequestWaitReplyPort+0xc
02 6ccafc04 76f55173 0009ad70 6ccafc24 6ccafc24 secur32!CallSPM+0x23
03 6ccafd30 76f54eb5 00000000 0009ad70 6ccafdac secur32!SecpAcceptSecurityContext+0x23c
04 6ccafdc0 76f54d63 00000000 6ccafe00 6ccafef0 secur32!LsaAcceptSecurityContext+0x15d
05 6ccafe10 77c8b90e 0010a090 67b31e68 6ccafef0 secur32!AcceptSecurityContext+0xa8
06 6ccafe58 77c8b8b8 00000010 6ccafef0 00000000 rpcrt4!SECURITY_CONTEXT::AcceptThirdLeg+0x42
07 6ccafea0 77c8b1c5 00000010 6ccafef0 00000000 rpcrt4!OSF_SCONNECTION::AcceptThirdLeg+0x61
08 6ccaff04 77c6e7b4 00000000 69ca4c68 000000ce rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x594
09 6ccaff18 77c7b799 00091918 0000000c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
0a 6ccaff84 77c7b9b5 6ccaffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
0b 6ccaff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
0c 6ccaffac 77c7b110 0008f5f8 6ccaffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
0d 6ccaffb8 77e6482f 5ac85d28 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
0e 6ccaffec 00000000 77c7b0f5 5ac85d28 00000000 kernel32!BaseThreadStart+0x34

Based on the “secur32!CallSPM” call stack, this tells me that we are processing authentication requests. These requests need to go to the domain controller for processing, and could potentially be proxied to another domain controller for processing. Being there could be a latency here I contacted my good friend Tate Calhoun up in the Platforms group so he could take a look at the LSASS dumps for confirmation. Tate is a Platforms EE debugging god, so I know I was in good hands here :) From the LSASS dumps Tate was able to tell me that we have 199 call stacks found containing : "netlogon!NLPUSERVALIDATEHIGHER"

0:061> kbn fff
# ChildEBP RetAddr Args to Child
00 02e3ec98 7c827d29 77e61d1e 000005b4 00000000 ntdll!KiFastSystemCallRet
01 02e3ec9c 77e61d1e 000005b4 00000000 02e3ece0 ntdll!NtWaitForSingleObject+0xc
02 02e3ed0c 77e61c8d 000005b4 0000afc8 00000000 kernel32!WaitForSingleObjectEx+0xac
03 02e3ed20 74266947 000005b4 0000afc8 000e0270 kernel32!WaitForSingleObject+0x12
04 02e3ed3c 74266e4c 000a9f70 0000afc8 000e0270 netlogon!NlAllocateClientApi+0x53
05 02e3edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x5d
06 02e3ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
07 02e3eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
08 02e3eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
09 02e3f720 76c9a815 ffffffff 00000003 03b61f40 msv1_0!LsaApLogonUserEx2+0x1106
0a 02e3fa80 76c95302 00607a20 01b47508 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
0b 02e3fc98 4ab85ebb 00607a20 00b48f80 02e3fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
0c 02e3fd0c 4ab860a8 01b474c0 01b474c8 02e3fe40 lsasrv!WLsaAcceptContext+0x139
0d 02e3fe84 4ab7ae1b 01b47498 00000000 005ed200 lsasrv!LpcAcceptContext+0x13b
0e 02e3fe9c 4ab7ad1e 01b47498 4ac22738 05924808 lsasrv!DispatchAPI+0x46
0f 02e3ff54 4ab7a769 01b47498 02e3ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
10 02e3ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
11 02e3ffb8 77e6482f 006bee48 00000000 00000000 lsasrv!LsapThreadBase+0x91
12 02e3ffec 00000000 4ab8f40e 006bee48 00000000 kernel32!BaseThreadStart+0x34

So, we are looking at authentication requests going to the LSASS process. LSASS has two threads that process authentication requests for the operating system. Here is one of the two allowed to proceed….

0:093> kbn fff
# ChildEBP RetAddr Args to Child
00 0363e618 7c827d29 77e61d1e 00001541 00000001 ntdll!KiFastSystemCallRet
01 0363e61c 77e61d1e 00001541 00000001 0363e660 ntdll!NtWaitForSingleObject+0xc
02 0363e68c 77c6a85f 00001541 00057e40 00000001 kernel32!WaitForSingleObjectEx+0xac
03 0363e6a8 77c69bf7 0167b15c 00000001 00057e40 rpcrt4!UTIL_WaitForSyncIO+0x20
04 0363e6cc 77c59546 0167b124 0167b15c 0363e6fc rpcrt4!UTIL_GetOverlappedResultEx+0x1d
05 0363e700 77c6af6a 00000001 0363e84c 0363e848 rpcrt4!WS_SyncRecv+0xbe
06 0363e728 77c69667 00000000 00ab8f70 000001a0 rpcrt4!OSF_CCONNECTION::TransSendReceive+0xb8
07 0363e7b0 77c695d4 10000000 01688228 00000001 rpcrt4!OSF_CCONNECTION::SendFragment+0x2ae
08 0363e808 77c6977a 00000000 00000160 0363e84c rpcrt4!OSF_CCALL::SendNextFragment+0x1e2
09 0363e850 77c699f2 0363e8d8 0363e894 00000001 rpcrt4!OSF_CCALL::FastSendReceive+0x148
0a 0363e86c 77c69975 0363e8d8 0363e894 0363e8d8 rpcrt4!OSF_CCALL::SendReceiveHelper+0x5b
0b 0363e89c 77c7fcf0 00ab8f88 0363e8bc 77c80673 rpcrt4!OSF_CCALL::SendReceive+0x41
0c 0363e8a8 77c80673 0363e8d8 71c43dd0 0363ecc4 rpcrt4!I_RpcSendReceive+0x24
0d 0363e8bc 77ce315a 0363e904 00ab90e8 00000000 rpcrt4!NdrSendReceive+0x2b
0e 0363eca4 71c4e665 71c43dd0 71c4b2e4 0363ecc4 rpcrt4!NdrClientCall2+0x22e
0f 0363ecbc 71c4e620 01b2ee68 03985a38 000e0300 netapi32!NetrLogonSamLogonEx+0x1c
10 0363ed1c 74266fdc 01b2ee68 03985a38 000e0300 netapi32!I_NetLogonSamLogonEx+0x41
11 0363edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x2d5
12 0363ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
13 0363eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
14 0363eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
15 0363f720 76c9a815 ffffffff 00000003 00b55030 msv1_0!LsaApLogonUserEx2+0x1106
16 0363fa80 76c95302 00607708 01b05188 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
17 0363fc98 4ab85ebb 00607708 017920d0 0363fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
18 0363fd0c 4ab860a8 01b05140 01b05148 0363fe40 lsasrv!WLsaAcceptContext+0x139
19 0363fe84 4ab7ae1b 01b05118 00000000 005ed050 lsasrv!LpcAcceptContext+0x13b
1a 0363fe9c 4ab7ad1e 01b05118 4ac22738 01b420b8 lsasrv!DispatchAPI+0x46
1b 0363ff54 4ab7a769 01b05118 0363ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
1c 0363ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
1d 0363ffb8 77e6482f 006bf2c8 00000000 00000000 lsasrv!LsapThreadBase+0x91
1e 0363ffec 00000000 4ab8f40e 006bf2c8 00000000 kernel32!BaseThreadStart+0x34

Now the question remains, why are we having a backlog. Mike wrote a detailed blog describing these findings and how to fix. Please see Mike Lagase’s blog: http://blogs.technet.com/mikelag/archive/2009/08/04/the-case-of-the-mysterious-exchange-server-hang.aspx

Dave

When downloading the OAB from an Exchange 2007 server running on Windows Server 2008 it can take forever, or you can be prompted repeatedly for credentials

dgoldman — Tue, 14 Jul 2009 23:38:00 GMT

We released an update for Rollup 8 for Exchange Server 2007 Service Pack 1 (KB 968012) back on May 26th, 2009. When Exchange 2007 is running on a Windows Server 2008, the Exchange 2007 clients may be repeatedly prompted for their credentials during Outlook Anywhere session, or an OAB download will be perceived to hang forever.

This issue occurs when NTLM Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box for the Outlook profile on the client computer. This issue does not occur if Basic Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box.

By default, Kernel Mode Authentication is enabled in Internet Information Services (IIS) 7.0 on the Client Access server. To resolve this issue, disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008.

The Fix

To disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008

At a command prompt, type the following command, and then press ENTER:

%systemroot%\system32\inetsrv\AppCmd.exe set config /section:system.webServer/security/authentication/windowsAuthentication /useKernelMode:false

Running the following command will verify that the changes have taken place:

C:\Windows\System32\inetsrv>appcmd.exe list config /section:WindowsAuthentication

<system.webServer>
   <security>
     <authentication>
       <windowsAuthentication enabled="false" useKernelMode="false">
         <providers>
           <add value="Negotiate" />
           <add value="NTLM" />
         </providers>
       </windowsAuthentication>
     </authentication>
   </security>
</system.webServer>

NOTE: Installing an Exchange Rollup or IU should never change a web.config file on a server. Running this command is a manual step that must be preformed, and is outlined in the following TechNet documentation:

http://technet.microsoft.com/en-us/library/bb123889.aspx

Dave

Global Address list shows the legacyExchangeDN instead of the email address

dgoldman — Wed, 01 Jul 2009 21:27:00 GMT

There are certain situations where the Global Address list display will show the legacyExchangeDN instead of the SMTP email address. The one condition where this is true is when you are viewing the Global Address List through Outlook 2003. The default behavior for Outlook 2003 is to use the EX address type for the Email Address column.

The default behavior for Outlook 2007 is to use the SMTP email address.

The best thing you can do is look at the users attribute in the active directory if you are not sure, because the mail attribute should contain a SMTP proxy address. If it contains a legacyExchangeDN, then this is incorrect. Here is where you can run in to problems. The OAB Generation process will turf an object when the mail attribute and primary proxy address for a user does not match. Running OABInteg can help you figure this out. You can run OABInteg /s:srvName /t:proxytest /v:2 /l. At the end of the output you will see something that looks like this:

Scan Finished
Total number of entries processed in the address book: 241004
Total number of entries skipped: 0
Total number of contacts: 12868
Total number of mailboxes: 95214
Total number of distribution lists: 131193
Total number of groups: 0
Total number of folders: 282
Total number of Address Book Container objects found: 0
Total number of temp legacyExchangeDN's found: 0
Total number of objects that are missing some main attributes: 0
Total number of objects that mail and proxy attribute don't match: 1
Total number of objects that do not have a domain value: 0
Total number of objects that do not have a valid unicode domain value: 0
Total number of objects that do not have a valid SMTP Domain because first character is not greater than '/': 0
Total number of objects that do not start with /o= or /O=: 1
Total user objects that are missing the Primary Proxy address attribute: 0
Total user objects with proxy addresses equal to or over 64 characters: 0
Total objects with the '@' character in the legacyExchangeDN: 0
Total objects with bad Active Directory Backlinks: 0
Total objects that have a legacyExchangeDN of ADCDisableMail: 0
Total objects that have a legacyExchangeDN of ADCDisableMailByADC: 0

I highlighted in bold red what is the important information.

Here is a good object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
Primary Proxy Address and mail attribute match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dgoldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

Here is a bad object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
ERROR - Primary Proxy Address and mail attribute do not match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dave.goldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

For more information on the proxytest command: http://blogs.msdn.com/dgoldman/archive/2007/03/08/how-to-use-oabinteg-s-oabfldcheck-and-proxytest-to-find-oab-issues.aspx

So to recap, if you are seeing the legacyExchangeDN in the Global Address List display, you should verify it the mail attribute is correct. If not, you will want to get this fixed. This can be done via logon script, using tools like ADModify, etc.

Dave

Unable to delete the OAB system folders in Exchange 2007?

dgoldman — Thu, 21 May 2009 01:54:23 GMT

Often we will get question that have to deal with deleting system folders in Exchange 2007. When deleting an OAB public folder from Exchange Management console, you receive the following error: 0x80070005. This error means "Access Denied." You will still receive this error even if you use an account that has Exchange Organization administrator permissions.

Logging will also show the following:

Action 'Remove' could not be performed on object - ‘/o=Organization/cn=addrlists/cn=oabs/cn=OAB'.

Error: The folder '\NON_IPM_SUBTREE\OFFLINE ADDRESS BOOK\ /o=Organization/cn=addrlists/cn=oabs/cn=OAB' or some of its subfolder(s) encountered errors and could not be deleted. MapiExceptionPartialCompletion: Unable to delete folder. (hr=0x40680, ec=0)

Diagnostic context:
    Lid: 18969   EcDoRpcExt2 called [length=54]
    Lid: 27161   EcDoRpcExt2 returned [ec=0x0][length=85][latency=0]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropDeleteFolder [29]
    Lid: 17082   ROP Error: 0x80070005
    Lid: 19297
    Lid: 21921   StoreEc: 0x80070005
    Lid: 27962   ROP: ropNone [0]
    Lid: 26881
    Lid: 21817   ROP Failure: 0x80070005
    Lid: 24721
    Lid: 20625   StoreEc: 0x80070005

Cause

This is by design. You cannot manually create or manually delete an OAB root folder. These system folders are handled by the information store. When you create an OAB object in the directory via powershell or the Exchange Management Console, the owning information store will create the OAB root folder when site folder maintenance is run. When you delete an OAB object from the active directory, the information store will mark the OAB folder as unused during site folder maintenance. If it goes unused for 7 days, it is deleted by the store during site folder maintenance.

Dave

Full OAB downloads caused by mis-configured anti virus software settings

dgoldman — Wed, 08 Apr 2009 20:16:38 GMT

Over the last few months I have seen a rash of calls come in where clients were being forced to download a full. Full OAB downloads can cause very high network usage and cause more havoc than anything.

Typically in these situations it is pretty easy to tell what is going on from the event errors that are logged. The two following events are as follows:

Event ID     : 9116
Raw Event ID : 9116
Record Nr.   : 860555
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error while generating the binpatch.oab file for differential downloads of address list '\Global Address List'. Clients will not be able to incrementally update to the new version of the offline address list, they will perform a full download instead. This is normal if this is the first time this offline address list has been generated. Check other logged events to see if this is a serious error.
- Default Offline Address List

NOTE: This is the first indication that you have a problem! You should never see a 9116 event any other time then when you generate your OAB for the first time.

Event ID     : 9109
Raw Event ID : 9109
Record Nr.   : 860554
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error ffffffff (internal ID 50303f6) while generating address list '\Global Address List'. Check other logged events to see if this is a serious error.
- Default Offline Address List

The key here is to check your anti-virus software to make sure that none of the OAB files have been quarantined. Being that the details.oab file is the biggest OAB file, it has the potential to get looked at more by anti-virus software than the other OAB files due to it’s size and usage. More evidence that the anti virus software had foul play here is to look at the data.oab file. In doing so you will see that it might contain one attachment called Deleted attachment.txt.

When this happens the client can not apply the difference file which results in the 9116 which most of the time is a normal event if you just built your OAB for the first time.

The Fix

Make sure your anti virus software has the proper exclusions set to not monitor the exchange OAB directories on your exchange server.
Make sure on the client you are excluding the *.oab files.

Dave

Forward to Internet recipients without internal mail-enabled contacts no longer works on Exchange 2007 – Part 2

dgoldman — Wed, 25 Mar 2009 17:58:25 GMT

I was recently asked by a HMC customer to see if this solution [http://blogs.msdn.com/dgoldman/archive/2008/04/09/how-to-forward-to-internet-recipients-without-creating-an-internal-mail-enabled-contact.aspx] would work for Exchange 2007 as it did work on Exchange 2003. The answer to this is *no*, this will not work. Being that the Exchange 2007 transport is a rewrite from 2003 we do not process the forwardingAddress of a user object, and this is by design.

I decided to set up a repro for this using the following steps:

1. Installed 3 Exchange 2007 servers. (1 Client Access Server, 1 Mailbox Server and 1 Hub Server.
2. In the Exchange System Manager I clicked on the Organization Configuration | Remote Domains Tab | Format of original message sent as attachment to journal report tab | click "Allow automatic forward"
3. Created a new mailbox called mailForward.
4. Using ADSI Edit connect to Schema, then navigate to Schema attribute ms-Exch-Forwarding-Address properties and set isMemberOfPartialAttributeSet to True 5. Now using ADSiEdit and navigate to the user account in the domain naming context and set forwardingAddress attribute on mailbox user’s properties. I changed the external recipient address to which you want the mail forwarded (for example dgoldman@hotmail.com)

Did my test on RTM and it was a no go. I then installed the latest services packs and hot fixes to confirm there were no changes between RTM and the latest Rollup.

Sending the message to the user will produce the following results if you look at the dump out the queues using Get-MessageTrackingLog

… I cut the rest out for readability

RECEIVE STORE... mailForward@tpworld.l... {mailForward@domain.com... test
DELIVER STORE... mailForward@tpworld.l... {mailForward@domain.com... test

As you can see the information store received and delivered the message which is correct, however you do not see an email being set to dgoldman@hotmail.com.

Checking the queues with the Exchange Management Console shows the following:

[PS] C:\Documents and Settings\exadmin>Get-Queue | fl

Identity         : E2K7-HUB\Submission
DeliveryType     : Undefined
NextHopDomain    : Submission
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 0
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

If the message was bifurcated you would expect to see an Unreachable queue for the message that was destined for hotmail.

I created a separate message to a false user: badmail@hotmail.com and sent this from the mailbox and you then see the following:

Identity         : E2K7-HUB\Unreachable
DeliveryType     : Unreachable
NextHopDomain    : Unreachable Domain
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 1
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

As a follow up I checked with the transport team and they in fact agree that this will no longer work and is by design.

Dave

A new build of OABInteg has been posted to the web

dgoldman — Mon, 23 Feb 2009 23:58:46 GMT

I complied a new build of OABInteg that has a fix for a problem when someone runs the tool with an LDAP test and the customer has DN’s that exceed 64 characters. The following error returned is: ERROR_DS_REFERRAL.

Command line arguments: OABInteg /s:dc01 /t:oaltest /v:2 /l

Program started at: 12:40:26 PM

Running OABInteg on: OABServer1\Administrator

Trying to connect to: GC://dc01.company.com object found: Hosting

Checking to see if OABInteg is running on an Exchange Server

Checking version of Exchange Server

Exchange 2003 Version: 6.5.6944.00

Exchange 2003 Service Pack Version: 6.5.7638.2.

Failure: ADsOpenObject

ADSI Error: hr = 0x8007202b - LDAP_REFERRAL - ERROR_DS_REFERRAL: Cannot resolve referral.

Function: OfflineAddressListTest

Line number: 646

The new build has been posted to http://code.msdn.microsoft.com/oabinteg/Release/ProjectReleases.aspx?ReleaseId=726

Dave

The OAB Generation process fails to generate an OAB with error MAPI_E_FAILONEPROVIDER

dgoldman — Wed, 04 Feb 2009 19:07:25 GMT

I just finished troubleshooting an error where we were generating multiple OAB's on a server with 6 OAB's and one of them would always fail with error Event 9175 (MSExchangeSA). I want to provide a small tidbit on how the system attendant obtains it's connection objects.

Here is a little information on the connection logic.

When the system attendant starts up it will attempt to make a connection to the information store so it can get a session object.
This session object is for the public folder store that this system attendant is associated with. This session object will be used for connecting to the public folder store when we need to access it for OAB, Free/Busy, etc.
Each time we need to connect to the information store we use the same session object and we will log the information about how many times we connect and how long they take to connect.

Now I needed to figure out why only the last one would fail. Through creating debug builds of store and system attendant so I could output maximum logging that was relevant to the problem I was able to obtain some information. Looking at the traces I could see that we would always return error: 0x478.

Dumping out this error I can see that we get ecWrongServer.

C:\err 0x478
# for hex 0x478 / decimal 1144
ecWrongServer ec.h
# 1 matches found for "0x478"

Now the question is why are we returning ecWrongServer. The information store has logic built in to it where we get redirected to another information store we will detect this change and try to connect where we are connecting too. This can happen for multiple reasons, for one too many connections in a short time. The system attendant will connect to the information store via the RPC interface and call the following function emsmdb!CNCT::EcForceRpc(). Through debugging I was able to see that we always got a return value of [0x47f = ECSERVERPAUSE].

Through code review I was able to see that when a client connects to a server it isn't configured for we will get an error code of ecWrongServer. This error code tells us that this is the wrong server and we will do a query to determine the correct server. Once we have this information we give the client the correct server to connect too so we can attempt to connect again. In certain situations this could lead to a looping connection condition and cause RPC problems. To correct this behavior we added some logic we that keeps track of how many ecWrongServer errors have been sent to a particular client.

In addition to this if a client tries to make RPC calls more than 5 times under 10 seconds we will automatically return the error code of ecServerPaused error. This error will bubble back up the stack and be mapped back to the following error code: MAPI_E_FAILONEPROVIDER(0x8004011d). In this case these observations show that there is nothing wrong with the OAB itself and has to do with too many connection attempts.

One solution to this problem is to stagger your OAB's so they are built at different times so we do not hit this logic, which is a safety mechanism to protect the store from RPC DOS type attacks (ligitement or not).

Dave

How to set the msExchQueryBaseDN attribute for users via powershell

dgoldman — Thu, 08 Jan 2009 20:15:00 GMT

I was working in the lab today trying to debug a reproduction of an OWA hosting issue and I had the need to set the msExchQueryBaseDN for 10,000 users. Typically you can use Poweshell with the get-user | set-user commands to access mailbox properties for stamping purposes, however the msExchQueryBaseDN attribute is a user based attribute. Here are two ways you can do it:

How to set an attribute for one user via powershell
$user = ([ADSI]"LDAP://OU=MyOU,DC=Company,DC=com").psbase;
$user.Properties["msExchQueryBaseDN"].Value = "CN=User,OU=MyOU,DC=Company,DC=com";
$user.CommitChanges();

How to set an attributes for a group of users in an OU via Powershell
get-mailbox userName* -resultsize unlimited | Foreach { $dn = "LDAP://" + $_.distinguishedname;$obj = [ADSI]$dn;$obj.msExchQueryBaseDN = "OU=YourOU,DC=company,DC=com";$obj.setinfo()}

This little bad boy stamped 10,000 users in less than 2 minutes.

Dave

OAB, Exchange Replication Service and the fix for OAB Generation failing on CCR and SCC clusters

dgoldman — Tue, 23 Dec 2008 22:06:00 GMT

Back on Thursday, December 11, 2008 i published a blog that contained information on the fix for OAB Generation failing on CCR and SCC clusters. This went in to detail as to what the problem was for both cluster types. Tim McMichael who is our Exchange Cluster guru has written a very indepth detailed blog on how the Exchange Replication Service (Exchange 2007 SP1) and Windows 2008 Clusters works. If you have time please take a look at his blog: http://blogs.technet.com/timmcmic/archive/2008/12/23/exchange-replication-service-exchange-2007-sp1-and-windows-2008-clusters.aspx

Dave

Exchange 2007 OAB Generation fails with the following errors: 9369, 9328 and 9328

dgoldman — Thu, 18 Dec 2008 00:18:38 GMT

I recently fixed an issue where the Default Offline Address List was on Exchange 2003, deleted and then recreated in Exchange 2007. In Exchange 2003 all Address lists create links in the active directory and in the public folder information stores under the System Folders | OFFLINE ADDRESS BOOK | /o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List.

When you delete an address list what you are doing is removing all references from that address list and the following things will happen:

The address list association on the private mailbox stores will be removed. This is the msExchUseOAB attribute that is stamped on the mailbox object. If this becomes blank clients will fail to download the OAB and receive the following error: 0X8004010F.
If you have a public folder store references to that OAB in the PF store might or might not get cleaned up based on how you have your mixed organization configured.
When you try to generate your newly created OAB, the OAB generation will fail with the following errors:

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9369
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered an error while publishing the OAB files to the distribution point 'Default Offline Address List'. Check other logged events to see more information about the problem. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9328
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered file error ffffffff (internal ID 505026d) while generating the offline address list for address list '\Global Address List'. Make sure there is enough disk space available. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9328
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered file error ffffffff (internal ID 5050236) while generating the offline address list for address list '\Global Address List'. Make sure there is enough disk space available. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The Problem

When you recreate that address list we are re-establishing the links to the objects that were previously deleted. When you try to rebuild the address list you can actually see in the application log that we make a connection to the public folder store even know the newly created address list has only be configured for Web Distribution:

Event Type:    Information
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9134
Date:        12/17/2008
Time:        12:26:38 PM
User:        N/A
Computer:    OABGen
Description:
OALGen successfully created or opened the sub folder 'OAB Version 4' under the offline address list folder '/o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List'.
- Default Offline Address List

From here we will try to create our distribution point, however we will fail because the old OAB was never configured this way.

How to fix

1. Create your new Offline Address List (the one that is failing).

2. Only configure it for 'Public Folder Distribution'.

3. Rebuild your Offline Address List - you should see this complete with a 9107 event.

4. Go back and reconfigure your OAB for 'Web Distribution'.

5. Rebuild your Offline Address List - you should see this complete with a 9107 event.

6. Check your (CAS) Client Access Server for the 'General' 1008 event to make sure that the OAB files were copied over.

and that is it!

Dave