Dgoldman's WebLog

Exchange 2010 Address List Segregation – Update

Hey guys, I think you will be happy to know that I am towards the end of wrapping up the Exchange 2010 Address List Segregation white paper. I will post another update as soon as we are finished with the editing and when it will be posted.

With that being said I wanted to post some information that is a *must read*. This information is pretty generic, however you must be aware of the changes in the Exchange product before you start playing with Address List Segregation on an Exchange 2010 platform.

Exchange 2010 now has an Address Book Service - In earlier versions of Microsoft Exchange prior to Exchange Server 2010, the Exchange server provided a referral service that told Outlook clients where they could find a domain controller that registered the NSPI service as a connection point. This referral would direct the Outlook client NSPI address book queries to a global catalog server. In addition, some Outlook Anywhere connections could also point the Outlook client back to the local server, and then the local server’s NSPI calls will be proxied to a global catalog server in the local site.

The Outlook client expects to find this referral service on the same server that's used for mailbox access. In Exchange 2010 now both mailbox access and directory access are handled by the Client Access server.

When an Outlook client contacts a Client Access server there are a few possible things that can happen.

1. If the user's mailbox is on an Exchange Server 2007 Mailbox server or an Exchange Server 2003 server, the directory request is referred to the user's mailbox server.
2. If the user's mailbox is on an Exchange 2010 Mailbox server, then one of two actions happens.
3. If the user's mailbox is in the same site as the Client Access server, the request is referred to the Client Access server.
4. If the user's mailbox is in a different site, the request is referred to a Client Access server in the remote site.

Now for Exchange 2010, the Client Access server will facilitate both the Referral Service and the NSPI endpoint. These two components are necessary for directory access to flow through the Client Access server.

IMPORTANT NOTE: If your Client Access server roll is installed on a domain controller the Outlook will communicate directly with the domain controller. This will bypass the Client Access server all together and revert the Exchange functionality back to as it is in Exchange 2007. 

The Exchange 2010 Address List Segregation white paper is being tested on a multiple machine setup with the domain controller NOT hosting ANY exchange roles.

If you are running Exchange 2010 on a domain controller clients will by pass MAPI and Domain on the Middle Tier. THIS IS NOT SUPPORTED FOR ADDRESS LIST SEGREGATION for the reason that if you switch your roles from the domain controller to a new server you will break your address list segregation functionality.

Thanks

Dave

GAL Photos in Exchange 2010 and Outlook 2010

Bharat Suneja has written an EXCELLENT article on how to use GAL Photos in Exchange 2010 and Outlook 2010.

Dave

How to set use OAB Throttling effectively

In this blog I will break down how to use OAB throttling. There are a few things that I need to explain about this as most of the administrators that use this never get it to work properly. Using OAB throttling is not a set it and forget it type of deal, the numbers *must* be recalculated every hour for it to work properly (this is based on your network).

How can we try to control the network consumption by using OAB Throttling?
The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.

For example: if you public folder store supports 10,000 users and receives 3,000 OAB download requests per hour (1 hour), and your offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data to your clients. This traffic could potentially and most of the time will overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This same traffic could also potentially overload a gigabit-per-second LAN for longer than one hour.

How do we calculate the largest OAB message size?

1. To find out the biggest  message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your OAB files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state. Here is an example of the Oabfldcheck summary:

Scan Completed
+------------------+
Message Class Normal found: 2
Message Class Differential found: 175
Message Class Unknown found: 0
Message Attachments found: 503
Messages found but unable to read the properties: 0
System folders found: 3
Highest sequence number found: 231
Lowest sequence number found: 16
Biggest attachment found: 16700 Bytes
Smallest attachment found: 102 Bytes
Biggest message found: 100,000,000 Bytes 
Smallest message found: 106 Bytes

2. Take the size of the largest message (here in my case it is 100 MB) x total number of users downloading the OAB x request per hour = number of data to be delivered by the server.

To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool

How to enable OAB Throttling

1. Run OABInteg and find out what the largest OAB message is.
2. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that is overloading your LAN. This should be done for about one hour (this is the most accurate way).
3. Make note of the bandwidth used by offline Address Book downloads on each server.
4. Set the threshold values to a percentage of those values. For example, set the threshold values to 45 - 60% of those values.
5. Monitor for the next hour and recalculate the numbers. As users download the OAB these numbers will no longer be affective. If we cross the threshold all users will start to ignore the values specified in the registry.

The formula = 100 MB x 10,000 (users)  x 3,000 (requests) = 300 GB (of data) / 45% – 60% (total size) = 135,000,000 ( is based on 45% ).

To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.

Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.

***Use Registry Editor at your own risk***

1. Click Start, click Run, type regedit in the Open box, and then click OK. 
2. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\ 
3. On the Edit menu, point to New, and then click DWORD Value. 
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify. 
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.

To return to the default behavior, delete the registry value.

Things you need to Understand about OAB Throttling

1. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.
2. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.
3. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:

• If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.

• If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.

5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.

Dave

OABGen with regards to public folders and backup’s

I thought that this would be interesting to post as I don’t think that many people are aware of this. So we all know that when the OAB is generated it will be dropped to the local OAB share on the OAB Generation server in the following location: [D:\Program Files\Microsoft\Exchange Server\ExchangeOAB\<OAB Guid>]. By default we will always look to this location for generating later versions of the OAB.

What about a fail safe mechanism?

Now if you happen to have a OAB Version 4 public folder we can find this to be a great resource. In the event of a problem (Examples: Files in the OAB share get plucked out by virus software, they get deleted, drive gets nuked, etc), we will fall back and resort to the public folder as a backup. Since we can no longer have access to use the files in the OAB share to construct the OAB, we will connect to the public folder store and download those files so we can reconstruct the new OAB files for the OAB share. This will stop us from having to make the initial query to the domain controller to re-query for the data, AND rebuild the OAB. This ensures that we retain the same Guid as before. This will stop the generation of a new OAB with a different Guid which causes full downloads.

So, before you get rid of all of your public folder servers, it might be a good idea to keep one around

Dave

How to change the temp location for the OAB Generation process on Exchange 2007 sp2 and Exchange 2010

In all version of Exchange the OAB Generation process will store the data is reads from the active directory in the %temp% directory, on the server that is generating the OAB. Once the OAB Generation process completes the temp files will be removed.

For some customers that have several million users in their organization or large hosting environments this can be a problem, as most systems are never upgraded to meet the current demands. Administrators from time to time often find themselves running out of drive space on the c:\ as this part of the system never gets upgraded.

In Exchange 2007 SP2 we added a registry key called OALGEN_REG_V4_ALT_TEMP_FILES_LOCATION which will allow you to change the temporary storage location for the OAB Version 4 temporary files. This registry key will *ONLY* work for OAB Version 4. OAB versions 2 and 3’s temporary files will still go to the %temp% directory. If this registry key is not present on the OAB’s generating system we will revert back to the %temp% directory.

Being that OAB Version 4 is generated slightly different and uses different compression algorithms we can allow for a larger number of users far exceeded what we could generate on legacy systems.

We have also added two new event log events for troubleshooting:

Event Type:    Informational
Category: OAL Generator
Source: MSExchangeSA
Event ID:    9400
Date:        2/6/2010
Time:        7:13:01 PM
User:        N/A
Computer:    OABGenServer
Description:

OALGen will use 'd:\storage\OabTempFiles' as the alternative temporary directory for generating the offline address book version 4 files.

and

Event Type:    Error
Category: OAL Generator
Source: MSExchangeSA
Event ID:    9401
Date:        2/6/2010
Time:        7:13:01 PM
User:        N/A
Computer:    OABGenServer
Description:

OALGen failed to open directory 'd:\storage\OabTempFiles' as the alternative location for intermediate files. The default location ‘c:\windows\temp’ will be used.

Before using this registry in a production environment I suggest that you test this out in your lab first.

Dave

Exchange 2007 and 2010 Address List Segregation changes

We’re in the process of validating the existing Exchange Address List Segmentation whitepaper against Exchange2010.  The whitepaper was written for Exchange 2007 and may need to be modified to support changes in Exchange 2010.  The current plan is for us *to provide support* for Exchange 2010 address list segregation once I have completed the white paper.

When I wrote the white paper for Exchange 2007 it was tested for Exchange 2007. Anyone that tries to use this white paper on an Exchange 2010 will run in to problems that we have no yet documented and or tested and might have to tear down their exchange infrastructure to correct the problem.

If you are going to remove Exchange 2010 you should not have any problems as long as you do not remove any older versions of Exchange. If you do choose to remove Exchange 2010 you may notice some left over (new 2010) containers in the active directory. These containers will not cause any issues and are now only removed if you removed all of the Exchange versions in the organization.

I am currently working on the white paper and working with development to make the necessary changes required due to changes in how Outlook makes directory connections.  Once the white paper is complete this will be supported.  Please be patient and please do not try to contact me with regards to when this white paper will be ready. I have a ton of debugging and architectural documentation to do, and I will post updates as they pop up.

Dave

PLEASE READ –> Events 9320 and 9359 on new installation of Exchange 2010

When you attempt to generate your OAB you will see the following event id’s logged in your application log:

Log Name:    Application
Source:          MSExchangeSA
Date:              12/1/2009 13:01:03 AM
Event ID:        9359
Level:             Warning
User:              N/A
Computer:     OABGenServer
Description:   OALGen truncated or dropped properties for entry ‘Dgoldman’ in address list '\Global Address List' because they exceeded the configured size limits for the version 4 offline address list.  The affected MAPI ids are:  8008. - \Default Offline Address Book

Here is what are MAPI ids 8cd8 and 8008 are?

• 0x8CD8000D = PR_EMS_AB_AUTH_ORIG                           PT_OBJECT
• 0x8008101F = PR_EMS_AB_IS_MEMBER_OF_DL_W         PT_MV_UNICODE

This event will be generated for each user read in the oab that is a member of a DL, and for all DL’s that have members.

The Problem

The problem is that the “member” and “memberOf” attributes are not a part of the truncated property list for regular groups. So when we are generating the OAB we will read this property, and we will not see it in the list and log the 9359 event.

The Fix

This is by DESIGN. These warnings are 100% expected and can be ignored. This has no ramifications on your oab generation process.

We will be taking a chance in the future to have the following behavior:

1. The 9320 Event ID that is related to truncation moved become a Medium level event log item
2. The 9359 Event ID will be changed from the Highest category to Expert

NOTE: There is no time frame on when this change will take place. Please understand that the above two items are also subject to change without notice.  I will keep everyone posted via the blog.

Dave

IP_V6 is a must if you are trying to install Exchange 2010

I am getting ready to write the whitepaper for Exchange 2010 address list segregation and I ran in to a problem where my setup was failing with no errors. I decided to open up the application log and MSEXCHANGEADTOPOLOGYSERVICE.EXE was nice enough to throw the following error for me:

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: Topology
Event ID: 2114
User: N/A
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1936). Topology discovery failed, error 0x80040a02 (DSC_E_NO_SUITABLE_CDC). Look up the Lightweight Directory Access
Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Like everyone else I see on the internet I also went to the following KB article, and to my surprise the article was outdated and of little use. I will see what I can do to get this fixed for everyone.

From here I went to check my network card settings to make sure that my DNS servers were set properly and I found that my IPV6 settings were unchecked. This is a huge problem. The Exchange 2010 services need to have IPV6 enabled for them to start. Once I enabled the IPV6 the transport service started and then I was able to reinstall all of Exchange 2010.

NOTE: If you need to disable IP_V6 for any reason the correct way to do this would be to do the following:

1. Uncheck IPV6 in the network card properties on both the DC and the Exchange server. 
2. You must apply a registry key and reboot each machine to completely remove IPv6:

Key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters
Value Name: DisabledComponents
Value Type: DWORD
Value: FF

3. Once this key is set reboot both the your domain controller(s) and the Exchange server.
4. run an Ipconfig /all
5. On each machine and ensure you see no IPv6 addressing in the output.
6. Once this is complete, elevate diagnostic logging for ADAccess and NSPI, use the following cmdlets:

• get-eventloglevel –id “MSExchange ADAccess\*" | set-eventloglevel –level:medium
• Get-eventloglevel –id “MSExchangeSA\*" | set-eventloglevel –level:medium
• Get-eventloglevel –id “MSExchangeSA\NSPI Proxy" –Level:high

7. Recycle the ADAccess and System Attendant services, let the machine run for 5-10 minutes, and send me the resulting application log.

Dave

 

A new version of FindSearchFolder.exe has been posted to the internet and is available for download

Back on Tuesday, July 01, 2008 I wrote the following blog on Microsoft Exchange and Search Folders. For this blog I also wrote a tool that would parse the isinteg.pri file to help identify exchange mailboxes that have high item counts or a high number of search folders, which lead to massive exchange server performance issues. I have re-written this tool in managed code using Visual Studio 2010 and it has a ton more functionality.

I have left the old version of the tool on the internet, and have also added a new version that will run on Windows 2008, Vista and Windows 7 machines. The new filename is  FindSearchFolder09.7z and it can be found here: http://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=FindSearchFolders&ReleaseId=1456.

To run this version of the tool all you need to do is put the isinteg.pri file in the same directory as the tool and run it.

This is what the tools UI looks like. As you can see from the screen shot below, once a scan has run the tool will report back the different stages that it is in and how many mailboxes were found in the log.

Once the tool runs it will create the following directories:

All log files can be located in the Logs directory. As you can see from the following screen shot each time you run the tool it will create a new output file based on a time stamp. This version will no longer merge the results.

The output of the tool has changed as well. The new output looks like this:

 

10-28-2009  -  Find Search Folders Output - 10-28-2009-14.53.35.txt created. Program Logging - Enabled.
***************************************************************************************
Dave Goldman - Exchange Escalation Services
NOTE: There is no support for this tool and will be used as is!
Purpose: Scan an isinteg.pri file and find the highest search folder offenders
10-28-2009  -  Trace time: 10-28-2009-14.53.35
***************************************************************************************
10-28-2009  -  Program Initialization - Complete
10-28-2009  -  Create Program Directories - Start
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Logs\
10-28-2009  -  Directory already exists: C:\Users\dgoldman.NORTHAMERICA\AppData\Roaming\FindSearchFolder\Backup\
10-28-2009  -  Create Program Directories - Complete
10-28-2009  -  Creating default mail handler - Complete
10-28-2009  -  Initializing Isinteg parser  - Complete
10-28-2009  -  Getting current application run path: D:\Tools\FindSearchFolders\Release
10-28-2009  -  Starting Isinteg parse - Start
10-28-2009  -  Search size set to: 75
10-28-2009  -  Isinteg is from: Server name: INCLCW02A.corp.statestr.com
10-28-2009  -  Isinteg was run against database: Database name: First Storage Group\Mailbox Store5 (INEMAL16)
10-28-2009  -  Processing Mailbox Table
10-28-2009  -  Display Name = Not Found - MBX Table dump only               

10-28-2009  -  Display Name = Goldman, Dave
                      Parent FID=0001-000001AB62E5
                      Root FID=0001-000001AB62E4
                      Restriction=
                      Name=Contacts
                      Folder FID=0001-000001AB1F4E               
                      Folder Type=1
                      Search Folder Count = 357  <- Is over 75 search folders and this is a BIG problem!
                      Backlink Folder Count = 357

10-28-2009  -  Processing Mailbox Table
10-28-2009  -  ==========================================================
10-28-2009  -  Mailbox Table
10-28-2009  -  ==========================================================
10-28-2009  -  Mailbox - Goldman, Dave - RootFID=0015-000012D75D62
10-28-2009  -  ==========================================================
10-28-2009  -  Total number of mailboxes with over 75 search folders: 1
10-28-2009  -  Highest number of search folders found in a single mailbox: 357
10-28-2009  -  Parent Folder: Parent FID=0001-000001AB62E5
10-28-2009  -  Highest offender is: Root 0001-000001AB62E4
10-28-2009  -  Total folders that do not contain search folders: 0
10-28-2009  -  Total mailboxes found on this information store: 1554
10-28-2009  -  ==========================================================
10-28-2009  -  ==========================================================
10-28-2009  -  These are the possible folder types
10-28-2009  -  ==========================================================
10-28-2009  -  Folder_Root = Folder Type = 0
10-28-2009  -  Normal_Folder = Folder Type = 1
10-28-2009  -  Search_Folder = Folder Type = 2
10-28-2009  -  MDB_Attachment_Folder = Folder Type = 4
10-28-2009  -  Repl_Folder = Folder Type = 5
10-28-2009  -  Categorization_Folder = Folder Type = 7
10-28-2009  -  ==========================================================
10-28-2009  -  Stat totals
10-28-2009  -  ==========================================================
10-28-2009  -  Total type Root Folders found: 1554
10-28-2009  -  Total type Normal folders found: 55580
10-28-2009  -  Total type Search Folders found: 1
10-28-2009  -  Total type MDB Attachment Folders found: 0
10-28-2009  -  Total type Repl Folder found: 0
10-28-2009  -  Total type Categorization (Views) Folders found: 4885
10-28-2009  -  ==========================================================

10-28-2009  -  NOTE: Folders with a type of 7 are cached views in the information store. For more information please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx To expire cached views from the information store, please see: http://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx

10-28-2009  -  Who has the highest search folder count in an isinteg dump
10-28-2009  -  ==========================================================
10-28-2009  -  1. Take the highest offenders RootFID and search the isinteg.pri file for this
10-28-2009  -  2. Look in the ESM and match these items counts and you should have your offending mailbox.
10-28-2009  -  Isinteg parse - completed

Dave

How can we try to control the network consumption by using OAB Throttling?

The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.

Example 1: If a public store that supports 10,000 users receives 3,000 requests in one hour and the offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data. This traffic could potentially overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This traffic could potentially overload a gigabit-per-second LAN for longer than one hour.

How do we calculate the largest OAB message size?
1.To find out the biggest  message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your oab files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state.

Here is an example of the Oabfldcheck summary:

Scan Completed
+------------------+
Message Class Normal found: 2
Message Class Differential found: 175
Message Class Unknown found: 0
Message Attachments found: 503
Messages found but unable to read the properties: 0
System folders found: 3
Highest sequence number found: 231
Lowest sequence number found: 16
Biggest attachment found: 16700 Bytes
Smallest attachment found: 102 Bytes
Biggest message found: 60234440 Bytes 
Smallest message found: 106 Bytes

You can then take the size of the largest message (here it is 60MB) x total users downloading the OAB x request per hour = number of data to be delivered by the server.

To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool

How to enable OAB Throttling
Use one of the following methods to determine the OAB Bandwidth Threshold data value to use:

Method 1 – Less Accurate
1. Set the value preemptively. *Based on your knowledge of the network topology* and how full offline Address Book downloads from multiple servers can add up to overload the LAN, set the thresholds on the individual servers to the largest values that are consistent with not overloading the LAN.

Bandwitdh Example: Type 5000 to configure the server to use 5000 kilobytes per second (KBps) as the bandwidth threshold for offline Address Book download throttling. 5000 KBps is approximately 40,960 kilobits per second (Kbps), or 40.96 megabits per second (Mbps).

Method 2 – Most Accurate
1. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that overloads the LAN. This can be a short period of time.
2. Make note of the bandwidth used by offline Address Book downloads on each server.
3. Set the threshold values to a percentage of those values. For example, set the threshold values to 60% of those values.

To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.

Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.

***Use Registry Editor at your own risk***
1. Click Start, click Run, type regedit in the Open box, and then click OK. 
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\ 
3. On the Edit menu, point to New, and then click DWORD Value. 
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify. 
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.

Things you need to Understand about OAB Throttling

1. To return to the default behavior, delete the registry value.

2. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.

3. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.

4. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:

• If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.

• If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.

5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.

Things not to do when using OAB Throttling

1. You will need to monitor this every 2 to 3 hours to recalculate the numbers. This is due to the clients succeeding in downloading the OAB.
2. Do not try to change the exchange servers network cards to some different lan speed settings. This will only cause more problems.
3. Do not restart the exchange server as this will just rebuild the OAB and start the process over again.
4. Do not move the OAB to a different server as this will just rebuild the OAB and start the process over again.
5. Do not try to stop the process. The damange has already happened and you need to ride it out!

Dave

Event ID 9373 is logged in the Application log when you update the offline address book in Exchange Server 2007

This was an article that I wrote about two years ago. I have high lighted a section in red which I think is very cool. You can rebuild your web distribution off of your public folder OAB without creating a new OAB.

When you update the offline address book by using the Update-OfflineAddressBook command or by using a scheduled task, the following event (Event ID 9373) is logged in the Application log on a computer that is running Microsoft Exchange Server 2007:

Event Type: Error
Event Source: MSExchangeSA
Event Category: OAL Generator
Event ID: 9373
Description: OALGen detected that the file '\\<SERVER name>\ExchangeOAB\<folder name>\a58e388b-7b23-4944-b042-58a7d0c6590f-data-1.lzx' is corrupted or missing. This indicates data tampering or disk problems. Restore files in this folder from the recent backup or clean up folder content and force a full OAB generation. - Default Offline Address Book
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Cause -This issue occurs when one or more OAB Version 4 offline address book files are corrupted or missing. In this case, the Microsoft Outlook clients cannot download the updated files until the issue is resolved.

RESOLUTION

To resolve this issue, use one of the following methods.

Method 1: Restore the contents of the OABGen folder

If daily backups are available, restore the contents of the OABGen folder to the following folder:

ExchangeInstallDir \ExchangeOAB\ OAB GUID

Note ExchangeInstallDir is a placeholder for the name of the Exchange Server 2007 installation directory. OAB GUID is a placeholder for the name of the folder.

Method 2: Rebuild the offline address book files by using Exchange Management Console or the Exchange Management Shell

If the corrupted offline address book item is enabled for public folders distribution, and if the data was published in public folders before the offline address book item was corrupted, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following method highlighted in red below:

• Use Exchange Management Console to rebuild the offline address book files. To do this, follow these steps:

1. Open Exchange Management Console.
2. Expand Organization Configuration, and then expand Mailbox.
3. On the details panel, click Offline Address Book, and then select the offline address book item that you want to update.
4. On the Action panel, click Update, and then click Yes.

•Use the Update-OfflineAddressBook command to rebuild the offline address book files.
To do this, run the following command in the Exchange Management Shell.

Update-OfflineAddressBook <OAB GUID> 

Note You can run the Update-OfflineAddressBook command on a computer that has the Mailbox, Client Access, or Hub Transport server role installed. To do this, you must log on by using an account that is a member of the Exchange Organization Administrators group. The account must also be a member of the local Administrators group on that computer.

 

Method 3: Rebuild the offline address book files by using a command

If neither daily backups nor public folders distribution is available, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following commands.

Update-OfflineAddressBook -Identity MyOAB 

Note In this example, the Update-OfflineAddressBook command is used to update the offline address book file that is named MyOAB.

Update-OfflineAddressBook <OAB GUID> 

Note When you use this command, the offline address book will be rebuilt. This will cause all clients to process a full offline address book download. For OAB Version 2 and OAB Version 3a, a full offline address book download can cause massive network traffic between client access servers and clients.

Dave

The Case of the Mysterious Exchange Server Hang

I have been working an interesting case for a large enterprise customer with a few other people from my group. We were having a problem where outlook clients were unable to connect to the exchange server, however the exchange server was operating like normal. After looking at the ESE internal structures I could see that the information store was not backed up on any I/O requests. As I started to debug a number of store dumps, I saw two interesting things that stood out. Below is a RPC fault, which is very rare in a store dump.

0:261> kbnL
# ChildEBP RetAddr  Args to Child             
00 61d9fc18 7c827d29 77e61d1e 000089dc 00000000 ntdll!KiFastSystemCallRet
01 61d9fc1c 77e61d1e 000089dc 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
02 61d9fc8c 77cdbedf 000089dc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
03 61d9fca8 77cdbf39 61d9fd20 000089dc 00000000 rpcrt4!UTIL_WaitForSyncHTTP2IO+0x18
04 61d9fcd0 77cccb39 67d1f18c 61d9fd20 000089dc rpcrt4!UTIL_GetOverlappedHTTP2ResultEx+0x20
05 61d9fcf8 77ccf2e8 67d1f18c 61d9fd1c 67d1f18c rpcrt4!WaitForSyncSend+0x26
06 61d9fd5c 77cd05ac 00000020 61d9fdd0 00000001 rpcrt4!HTTP2VirtualConnection::SyncSend+0xdd
07 61d9fd7c 77ccc1f6 00000020 61d9fdd0 00000001 rpcrt4!HTTP2ServerVirtualConnection::SyncSend+0x2a
08 61d9fd98 77c689e8 67d1f18c 00000020 61d9fdd0 rpcrt4!HTTP_SyncSend+0x2a
09 61d9fdc0 77ca9db9 61d9fdd0 00000020 03030005 rpcrt4!OSF_SCONNECTION::TransSend+0x74
0a 61d9fe08 77caa3cf 00000005 00000000 00000e1d rpcrt4!OSF_SCONNECTION::SendFault+0xb5
0b 61d9fe30 77c93e4e 00000005 00000000 67c57af8 rpcrt4!OSF_SCALL::CleanupCallAndSendFault+0x46
0c 61d9fe60 77c68b5e 67c57af8 030001f0 0000000a rpcrt4!OSF_SCALL::ProcessReceivedPDU+0x4a6
0d 61d9fe80 77c6e8db 00000000 000001f0 00000000 rpcrt4!OSF_SCALL::BeginRpcCall+0x194
0e 61d9fee0 77c6e7b4 00000000 67c57af8 000001f0 rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x435
0f 61d9fef4 77cae695 00091918 0000001c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
10 61d9ff18 77c7b799 00091918 0000001c 00000000 rpcrt4!ProcessComplexTReceive+0x54
11 61d9ff84 77c7b9b5 61d9ffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
12 61d9ff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
13 61d9ffac 77c7b110 0008f5f8 61d9ffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
14 61d9ffb8 77e6482f 6a084fa0 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
15 61d9ffec 00000000 77c7b0f5 6a084fa0 00000000 kernel32!BaseThreadStart+0x34

and a ton of threads calling in to “secur32!CallSPM”.

0:329> kbn
# ChildEBP RetAddr  Args to Child             
00 6ccafbe4 7c827899 76f52135 000006f4 6ccafc24 ntdll!KiFastSystemCallRet
01 6ccafbe8 76f52135 000006f4 6ccafc24 6ccafc24 ntdll!NtRequestWaitReplyPort+0xc
02 6ccafc04 76f55173 0009ad70 6ccafc24 6ccafc24 secur32!CallSPM+0x23
03 6ccafd30 76f54eb5 00000000 0009ad70 6ccafdac secur32!SecpAcceptSecurityContext+0x23c
04 6ccafdc0 76f54d63 00000000 6ccafe00 6ccafef0 secur32!LsaAcceptSecurityContext+0x15d
05 6ccafe10 77c8b90e 0010a090 67b31e68 6ccafef0 secur32!AcceptSecurityContext+0xa8
06 6ccafe58 77c8b8b8 00000010 6ccafef0 00000000 rpcrt4!SECURITY_CONTEXT::AcceptThirdLeg+0x42
07 6ccafea0 77c8b1c5 00000010 6ccafef0 00000000 rpcrt4!OSF_SCONNECTION::AcceptThirdLeg+0x61
08 6ccaff04 77c6e7b4 00000000 69ca4c68 000000ce rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x594
09 6ccaff18 77c7b799 00091918 0000000c 00000000 rpcrt4!ProcessConnectionServerReceivedEvent+0x21
0a 6ccaff84 77c7b9b5 6ccaffac 77c8872d 00091918 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8
0b 6ccaff8c 77c8872d 00091918 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
0c 6ccaffac 77c7b110 0008f5f8 6ccaffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d
0d 6ccaffb8 77e6482f 5ac85d28 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
0e 6ccaffec 00000000 77c7b0f5 5ac85d28 00000000 kernel32!BaseThreadStart+0x34

Based on the “secur32!CallSPM” call stack, this tells me that we are processing authentication requests. These requests need to go to the domain controller for processing, and could potentially be proxied to another domain controller for processing. Being there could be a latency here I contacted my good friend Tate Calhoun up in the Platforms group so he could take a look at the LSASS dumps for confirmation. Tate is a Platforms EE debugging god, so I know I was in good hands here :) From the LSASS dumps Tate was able to tell me that we have 199 call stacks found containing : "netlogon!NLPUSERVALIDATEHIGHER"

0:061> kbn fff
#   ChildEBP RetAddr  Args to Child             
00  02e3ec98 7c827d29 77e61d1e 000005b4 00000000 ntdll!KiFastSystemCallRet
01  02e3ec9c 77e61d1e 000005b4 00000000 02e3ece0 ntdll!NtWaitForSingleObject+0xc
02  02e3ed0c 77e61c8d 000005b4 0000afc8 00000000 kernel32!WaitForSingleObjectEx+0xac
03  02e3ed20 74266947 000005b4 0000afc8 000e0270 kernel32!WaitForSingleObject+0x12
04  02e3ed3c 74266e4c 000a9f70 0000afc8 000e0270 netlogon!NlAllocateClientApi+0x53
05  02e3edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x5d
06  02e3ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
07  02e3eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
08  02e3eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
09  02e3f720 76c9a815 ffffffff 00000003 03b61f40 msv1_0!LsaApLogonUserEx2+0x1106
0a  02e3fa80 76c95302 00607a20 01b47508 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
0b  02e3fc98 4ab85ebb 00607a20 00b48f80 02e3fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
0c  02e3fd0c 4ab860a8 01b474c0 01b474c8 02e3fe40 lsasrv!WLsaAcceptContext+0x139
0d  02e3fe84 4ab7ae1b 01b47498 00000000 005ed200 lsasrv!LpcAcceptContext+0x13b
0e  02e3fe9c 4ab7ad1e 01b47498 4ac22738 05924808 lsasrv!DispatchAPI+0x46
0f  02e3ff54 4ab7a769 01b47498 02e3ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
10  02e3ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
11  02e3ffb8 77e6482f 006bee48 00000000 00000000 lsasrv!LsapThreadBase+0x91
12  02e3ffec 00000000 4ab8f40e 006bee48 00000000 kernel32!BaseThreadStart+0x34

So, we are looking at authentication requests going to the LSASS process. LSASS has two threads that process authentication requests for the operating system. Here is one of the two allowed to proceed….

0:093> kbn fff
#   ChildEBP RetAddr  Args to Child             
00  0363e618 7c827d29 77e61d1e 00001541 00000001 ntdll!KiFastSystemCallRet
01  0363e61c 77e61d1e 00001541 00000001 0363e660 ntdll!NtWaitForSingleObject+0xc
02  0363e68c 77c6a85f 00001541 00057e40 00000001 kernel32!WaitForSingleObjectEx+0xac
03  0363e6a8 77c69bf7 0167b15c 00000001 00057e40 rpcrt4!UTIL_WaitForSyncIO+0x20
04  0363e6cc 77c59546 0167b124 0167b15c 0363e6fc rpcrt4!UTIL_GetOverlappedResultEx+0x1d
05  0363e700 77c6af6a 00000001 0363e84c 0363e848 rpcrt4!WS_SyncRecv+0xbe
06  0363e728 77c69667 00000000 00ab8f70 000001a0 rpcrt4!OSF_CCONNECTION::TransSendReceive+0xb8
07  0363e7b0 77c695d4 10000000 01688228 00000001 rpcrt4!OSF_CCONNECTION::SendFragment+0x2ae
08  0363e808 77c6977a 00000000 00000160 0363e84c rpcrt4!OSF_CCALL::SendNextFragment+0x1e2
09  0363e850 77c699f2 0363e8d8 0363e894 00000001 rpcrt4!OSF_CCALL::FastSendReceive+0x148
0a  0363e86c 77c69975 0363e8d8 0363e894 0363e8d8 rpcrt4!OSF_CCALL::SendReceiveHelper+0x5b
0b  0363e89c 77c7fcf0 00ab8f88 0363e8bc 77c80673 rpcrt4!OSF_CCALL::SendReceive+0x41
0c  0363e8a8 77c80673 0363e8d8 71c43dd0 0363ecc4 rpcrt4!I_RpcSendReceive+0x24
0d  0363e8bc 77ce315a 0363e904 00ab90e8 00000000 rpcrt4!NdrSendReceive+0x2b
0e  0363eca4 71c4e665 71c43dd0 71c4b2e4 0363ecc4 rpcrt4!NdrClientCall2+0x22e
0f  0363ecbc 71c4e620 01b2ee68 03985a38 000e0300 netapi32!NetrLogonSamLogonEx+0x1c
10  0363ed1c 74266fdc 01b2ee68 03985a38 000e0300 netapi32!I_NetLogonSamLogonEx+0x41
11  0363edbc 74266cf8 000a9f70 00000000 00000006 netlogon!NlpUserValidateHigher+0x2d5
12  0363ee1c 74267407 000e0270 00000001 00000000 netlogon!NlpUserValidate+0x309
13  0363eea4 7426728d 00000000 00000000 00000000 netlogon!NlpLogonSamLogon+0x3f5
14  0363eee0 76c9ce2f 00000000 00000000 00000000 netlogon!NetILogonSamLogon+0x32
15  0363f720 76c9a815 ffffffff 00000003 00b55030 msv1_0!LsaApLogonUserEx2+0x1106
16  0363fa80 76c95302 00607708 01b05188 00200a03 msv1_0!SsprHandleAuthenticateMessage+0xcca
17  0363fc98 4ab85ebb 00607708 017920d0 0363fe40 msv1_0!SpAcceptLsaModeContext+0x1f8
18  0363fd0c 4ab860a8 01b05140 01b05148 0363fe40 lsasrv!WLsaAcceptContext+0x139
19  0363fe84 4ab7ae1b 01b05118 00000000 005ed050 lsasrv!LpcAcceptContext+0x13b
1a  0363fe9c 4ab7ad1e 01b05118 4ac22738 01b420b8 lsasrv!DispatchAPI+0x46
1b  0363ff54 4ab7a769 01b05118 0363ff9c 77e5ba69 lsasrv!LpcHandler+0x1fe
1c  0363ff78 4ab8f465 0062c510 00000000 00000000 lsasrv!SpmPoolThreadBase+0xb9
1d  0363ffb8 77e6482f 006bf2c8 00000000 00000000 lsasrv!LsapThreadBase+0x91
1e  0363ffec 00000000 4ab8f40e 006bf2c8 00000000 kernel32!BaseThreadStart+0x34

Now the question remains, why are we having a backlog. Mike wrote a detailed blog describing these findings and how to fix. Please see Mike Lagase’s blog: http://blogs.technet.com/mikelag/archive/2009/08/04/the-case-of-the-mysterious-exchange-server-hang.aspx

Dave

When downloading the OAB from an Exchange 2007 server running on Windows Server 2008 it can take forever, or you can be prompted repeatedly for credentials

We released an update for Rollup 8 for Exchange Server 2007 Service Pack 1 (KB 968012) back on May 26th, 2009. When Exchange 2007 is running on a Windows Server 2008, the Exchange 2007 clients may be repeatedly prompted for their credentials during Outlook Anywhere session, or an OAB download will be perceived to hang forever.

This issue occurs when NTLM Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box for the Outlook profile on the client computer. This issue does not occur if Basic Authentication is selected as the authentication method in the Exchange Proxy Settings dialog box.

By default, Kernel Mode Authentication is enabled in Internet Information Services (IIS) 7.0 on the Client Access server. To resolve this issue, disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008.

The Fix

To disable Kernel Mode Authentication for Client Access servers that are running Windows Server 2008

• At a command prompt, type the following command, and then press ENTER:

  %systemroot%\system32\inetsrv\AppCmd.exe set config /section:system.webServer/security/authentication/windowsAuthentication /useKernelMode:false

Running the following command will verify that the changes have taken place:

C:\Windows\System32\inetsrv>appcmd.exe list config /section:WindowsAuthentication

<system.webServer>
   <security>
     <authentication>
       <windowsAuthentication enabled="false" useKernelMode="false">
         <providers>
           <add value="Negotiate" />
           <add value="NTLM" />
         </providers>
       </windowsAuthentication>
     </authentication>
   </security>
</system.webServer>

NOTE: Installing an Exchange Rollup or IU should never change a web.config file on a server. Running this command is a manual step that must be preformed, and is outlined in the following TechNet documentation:

http://technet.microsoft.com/en-us/library/bb123889.aspx

Dave

Global Address list shows the legacyExchangeDN instead of the email address

There are certain situations where the Global Address list display will show the legacyExchangeDN instead of the SMTP email address. The one condition where this is true is when you are viewing the Global Address List through Outlook 2003. The default behavior for Outlook 2003 is to use the EX address type for the Email Address column.

 

The default behavior for Outlook 2007 is to use the SMTP email address.

The best thing you can do is look at the users attribute in the active directory if you are not sure, because the mail attribute should contain a SMTP proxy address. If it contains a legacyExchangeDN, then this is incorrect. Here is where you can run in to problems. The OAB Generation process will turf an object when the mail attribute and primary proxy address for a user does not match. Running OABInteg can help you figure this out. You can run OABInteg /s:srvName /t:proxytest /v:2 /l. At the end of the output you will see something that looks like this:

Scan Finished
Total number of entries processed in the address book: 241004
Total number of entries skipped: 0
Total number of contacts: 12868
Total number of mailboxes: 95214
Total number of distribution lists: 131193
Total number of groups: 0
Total number of folders: 282
Total number of Address Book Container objects found: 0
Total number of temp legacyExchangeDN's found: 0
Total number of objects that are missing some main attributes: 0
Total number of objects that mail and proxy attribute don't match: 1
Total number of objects that do not have a domain value: 0
Total number of objects that do not have a valid unicode domain value: 0
Total number of objects that do not have a valid SMTP Domain because first character is not greater than '/': 0
Total number of objects that do not start with /o= or /O=: 1
Total user objects that are missing the Primary Proxy address attribute: 0
Total user objects with proxy addresses equal to or over 64 characters: 0
Total objects with the '@' character in the legacyExchangeDN: 0
Total objects with bad Active Directory Backlinks: 0
Total objects that have a legacyExchangeDN of ADCDisableMail: 0
Total objects that have a legacyExchangeDN of ADCDisableMailByADC: 0

I highlighted in bold red what is the important information.

Here is a good object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
Primary Proxy Address and mail attribute match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dgoldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

Here is a bad object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
ERROR - Primary Proxy Address and mail attribute do not match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dave.goldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

For more information on the proxytest command: http://blogs.msdn.com/dgoldman/archive/2007/03/08/how-to-use-oabinteg-s-oabfldcheck-and-proxytest-to-find-oab-issues.aspx

So to recap, if you are seeing the legacyExchangeDN in the Global Address List display, you should verify it the mail attribute is correct. If not, you will want to get this fixed. This can be done via logon script, using tools like ADModify, etc.

Dave

Unable to delete the OAB system folders in Exchange 2007?

Often we will get question that have to deal with deleting system folders in Exchange 2007. When deleting an OAB public folder from Exchange Management console, you receive the following error: 0x80070005. This error means "Access Denied." You will still receive this error even if you use an account that has Exchange Organization administrator permissions.

 

Logging will also show the following:

Action 'Remove' could not be performed on object - ‘/o=Organization/cn=addrlists/cn=oabs/cn=OAB'.

Error: The folder '\NON_IPM_SUBTREE\OFFLINE ADDRESS BOOK\ /o=Organization/cn=addrlists/cn=oabs/cn=OAB' or some of its subfolder(s) encountered errors and could not be deleted. MapiExceptionPartialCompletion: Unable to delete folder. (hr=0x40680, ec=0)

Diagnostic context:
    Lid: 18969   EcDoRpcExt2 called [length=54]
    Lid: 27161   EcDoRpcExt2 returned [ec=0x0][length=85][latency=0]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropDeleteFolder [29]
    Lid: 17082   ROP Error: 0x80070005
    Lid: 19297 
    Lid: 21921   StoreEc: 0x80070005
    Lid: 27962   ROP: ropNone [0]
    Lid: 26881 
    Lid: 21817   ROP Failure: 0x80070005
    Lid: 24721 
    Lid: 20625   StoreEc: 0x80070005

Cause

This is by design. You cannot manually create or manually delete an OAB root folder. These system folders are handled by the information store. When you create an OAB object in the directory via powershell or the Exchange Management Console, the owning information store will create the OAB root folder when site folder maintenance is run. When you delete an OAB object from the active directory, the information store will mark the OAB folder as unused during site folder maintenance. If it goes unused for 7 days, it is deleted by the store during site folder maintenance.

Dave