Dgoldman's WebLog

Global Address list shows the legacyExchangeDN instead of the email address

There are certain situations where the Global Address list display will show the legacyExchangeDN instead of the SMTP email address. The one condition where this is true is when you are viewing the Global Address List through Outlook 2003. The default behavior for Outlook 2003 is to use the EX address type for the Email Address column.

 

The default behavior for Outlook 2007 is to use the SMTP email address.

The best thing you can do is look at the users attribute in the active directory if you are not sure, because the mail attribute should contain a SMTP proxy address. If it contains a legacyExchangeDN, then this is incorrect. Here is where you can run in to problems. The OAB Generation process will turf an object when the mail attribute and primary proxy address for a user does not match. Running OABInteg can help you figure this out. You can run OABInteg /s:srvName /t:proxytest /v:2 /l. At the end of the output you will see something that looks like this:

Scan Finished
Total number of entries processed in the address book: 241004
Total number of entries skipped: 0
Total number of contacts: 12868
Total number of mailboxes: 95214
Total number of distribution lists: 131193
Total number of groups: 0
Total number of folders: 282
Total number of Address Book Container objects found: 0
Total number of temp legacyExchangeDN's found: 0
Total number of objects that are missing some main attributes: 0
Total number of objects that mail and proxy attribute don't match: 1
Total number of objects that do not have a domain value: 0
Total number of objects that do not have a valid unicode domain value: 0
Total number of objects that do not have a valid SMTP Domain because first character is not greater than '/': 0
Total number of objects that do not start with /o= or /O=: 1
Total user objects that are missing the Primary Proxy address attribute: 0
Total user objects with proxy addresses equal to or over 64 characters: 0
Total objects with the '@' character in the legacyExchangeDN: 0
Total objects with bad Active Directory Backlinks: 0
Total objects that have a legacyExchangeDN of ADCDisableMail: 0
Total objects that have a legacyExchangeDN of ADCDisableMailByADC: 0

I highlighted in bold red what is the important information.

Here is a good object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
Primary Proxy Address and mail attribute match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dgoldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

Here is a bad object:

Processing Address Book Entry #1 of 50.
Display Name = Dave Goldman
Object is a mailbox object
LegacyExchangeDN starts with '/o=' or '/O='. Value = /o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Primary Proxy Address found. Value = dgoldman@microsoft.com
Primary Proxy Address has a vaild unicode domain. Value = @microsoft.com
SMTP Domain is valid and contains '@'.
X500 Address is: X500:/o=microsoft/ou=northamerica/cn=Recipients/cn=dgoldman
Proxy Address SMTP:dgoldman@microsoft.com is 8 characters. (First 8 characters)
Primary Proxy Address found. Value = SMTP:dgoldman@microsoft.com.
ERROR - Primary Proxy Address and mail attribute do not match.
Primary Proxy Address = dgoldman@microsoft.com, mail attribute = dave.goldman@microsoft.com.
Primary Proxy Address has a valid domain. Value = @microsoft.com

For more information on the proxytest command: http://blogs.msdn.com/dgoldman/archive/2007/03/08/how-to-use-oabinteg-s-oabfldcheck-and-proxytest-to-find-oab-issues.aspx

So to recap, if you are seeing the legacyExchangeDN in the Global Address List display, you should verify it the mail attribute is correct. If not, you will want to get this fixed. This can be done via logon script, using tools like ADModify, etc.

Dave

Unable to delete the OAB system folders in Exchange 2007?

Often we will get question that have to deal with deleting system folders in Exchange 2007. When deleting an OAB public folder from Exchange Management console, you receive the following error: 0x80070005. This error means "Access Denied." You will still receive this error even if you use an account that has Exchange Organization administrator permissions.

 

Logging will also show the following:

Action 'Remove' could not be performed on object - ‘/o=Organization/cn=addrlists/cn=oabs/cn=OAB'.

Error: The folder '\NON_IPM_SUBTREE\OFFLINE ADDRESS BOOK\ /o=Organization/cn=addrlists/cn=oabs/cn=OAB' or some of its subfolder(s) encountered errors and could not be deleted. MapiExceptionPartialCompletion: Unable to delete folder. (hr=0x40680, ec=0)

Diagnostic context:
    Lid: 18969   EcDoRpcExt2 called [length=54]
    Lid: 27161   EcDoRpcExt2 returned [ec=0x0][length=85][latency=0]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropDeleteFolder [29]
    Lid: 17082   ROP Error: 0x80070005
    Lid: 19297 
    Lid: 21921   StoreEc: 0x80070005
    Lid: 27962   ROP: ropNone [0]
    Lid: 26881 
    Lid: 21817   ROP Failure: 0x80070005
    Lid: 24721 
    Lid: 20625   StoreEc: 0x80070005

Cause

This is by design. You cannot manually create or manually delete an OAB root folder. These system folders are handled by the information store. When you create an OAB object in the directory via powershell or the Exchange Management Console, the owning information store will create the OAB root folder when site folder maintenance is run. When you delete an OAB object from the active directory, the information store will mark the OAB folder as unused during site folder maintenance. If it goes unused for 7 days, it is deleted by the store during site folder maintenance.

Dave

Full OAB downloads caused by mis-configured anti virus software settings

Over the last few months I have seen a rash of calls come in where clients were being forced to download a full.  Full OAB downloads can cause very high network usage and cause more havoc than anything.

Typically in these situations it is pretty easy to tell what is going on from the event errors that are logged. The two following events are as follows:

Event ID     : 9116
Raw Event ID : 9116
Record Nr.   : 860555
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error while generating the binpatch.oab file for differential downloads of address list '\Global Address List'.  Clients will not be able to incrementally update to the new version of the offline address list, they will perform a full download instead.  This is normal if this is the first time this offline address list has been generated.  Check other logged events to see if this is a serious error.
- Default Offline Address List

NOTE: This is the first indication that you have a problem! You should never see a 9116 event any other time then when you generate your OAB for the first time.

Event ID     : 9109
Raw Event ID : 9109
Record Nr.   : 860554
Category     : OAL Generator
Source       : MSExchangeSA
Type         : Warning
Generated    : 4/7/2009 5:19:17 PM
Written      : 4/7/2009 5:19:17 PM
Machine      : OABServer
Message      : OALGen encountered an error ffffffff (internal ID 50303f6) while generating address list '\Global Address List'.  Check other logged events to see if this is a serious error.
- Default Offline Address List

The key here is to check your anti-virus software to make sure that none of the OAB files have been quarantined. Being that the details.oab file is the biggest OAB file, it has the potential to get looked at more by anti-virus software than the other OAB files due to it’s size and usage. More evidence that the anti virus software had foul play here is to look at the data.oab file. In doing so you will see that it might contain one attachment called Deleted attachment.txt.

When this happens the client can not apply the difference file which results in the 9116 which most of the time is a normal event if you just built your OAB for the first time.

The Fix

1. Make sure your anti virus software has the proper exclusions set to not monitor the exchange OAB directories on your exchange server.
2. Make sure on the client you are excluding the *.oab files.

Dave

Forward to Internet recipients without internal mail-enabled contacts no longer works on Exchange 2007 – Part 2

I was recently asked by a HMC customer to see if this solution [http://blogs.msdn.com/dgoldman/archive/2008/04/09/how-to-forward-to-internet-recipients-without-creating-an-internal-mail-enabled-contact.aspx] would work for Exchange 2007 as it did work on Exchange 2003. The answer to this is *no*, this will not work. Being that the Exchange 2007 transport is a rewrite from 2003 we do not process the forwardingAddress of a user object, and this is by design.

 

I decided to set up a repro for this using the following steps:

1. Installed 3 Exchange 2007 servers. (1 Client Access Server, 1 Mailbox Server and 1 Hub Server.
2. In the Exchange System Manager I clicked on the Organization Configuration | Remote Domains Tab | Format of original message sent as attachment to journal report tab | click "Allow automatic forward"
3. Created a new mailbox called mailForward.
4. Using ADSI Edit connect to Schema, then navigate to Schema attribute ms-Exch-Forwarding-Address properties and set isMemberOfPartialAttributeSet to True 5. Now using ADSiEdit and navigate to the user account in the domain naming context and set forwardingAddress attribute on mailbox user’s properties. I changed the external recipient address to which you want the mail forwarded (for example dgoldman@hotmail.com)

Did my test on RTM and it was a no go. I then installed the latest services packs and hot fixes to confirm there were no changes between RTM and the latest Rollup.

Sending the message to the user will produce the following results if you look at the dump out the queues using Get-MessageTrackingLog

… I cut the rest out for readability

RECEIVE  STORE... mailForward@tpworld.l... {mailForward@domain.com... test
DELIVER  STORE... mailForward@tpworld.l... {mailForward@domain.com... test

As you can see the information store received and delivered the message which is correct, however you do not see an email being set to dgoldman@hotmail.com.

Checking the queues with the Exchange Management Console shows the following:

[PS] C:\Documents and Settings\exadmin>Get-Queue | fl

Identity         : E2K7-HUB\Submission
DeliveryType     : Undefined
NextHopDomain    : Submission
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 0
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

If the message was bifurcated you would expect to see an Unreachable queue for the message that was destined for hotmail.

I created a separate message to a false user: badmail@hotmail.com and sent this from the mailbox and you then see the following:

Identity         : E2K7-HUB\Submission
DeliveryType     : Undefined
NextHopDomain    : Submission
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 0
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

Identity         : E2K7-HUB\Unreachable
DeliveryType     : Unreachable
NextHopDomain    : Unreachable Domain
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status           : Ready
MessageCount     : 1
LastError        :
LastRetryTime    :
NextRetryTime    :
IsValid          : True
ObjectState      : Unchanged

As a follow up I checked with the transport team and they in fact agree that this will no longer work and is by design.

Dave

A new build of OABInteg has been posted to the web

I complied a new build of OABInteg that has a fix for a problem when someone runs the tool with an LDAP test and the customer has DN’s that exceed 64 characters. The following error returned is: ERROR_DS_REFERRAL.

Command line arguments: OABInteg /s:dc01 /t:oaltest /v:2 /l

Program started at: 12:40:26 PM

Running OABInteg on: OABServer1\Administrator

Trying to connect to: GC://dc01.company.com object found: Hosting

Checking to see if OABInteg is running on an Exchange Server

Checking version of Exchange Server

Exchange 2003 Version: 6.5.6944.00

Exchange 2003 Service Pack Version: 6.5.7638.2.

Failure: ADsOpenObject

ADSI Error: hr = 0x8007202b - LDAP_REFERRAL - ERROR_DS_REFERRAL: Cannot resolve referral.

Function: OfflineAddressListTest

Line number: 646

The new build has been posted to http://code.msdn.microsoft.com/oabinteg/Release/ProjectReleases.aspx?ReleaseId=726

Dave

The OAB Generation process fails to generate an OAB with error MAPI_E_FAILONEPROVIDER

I just finished troubleshooting an error where we were generating multiple OAB's on a server with 6 OAB's and one of them would always fail with error Event 9175 (MSExchangeSA). I want to provide a small tidbit on how the system attendant obtains it's connection objects.

Here is a little information on the connection logic.

1. When the system attendant starts up it will attempt to make a connection to the information store so it can get a session object.
2. This session object is for the public folder store that this system attendant is associated with. This session object will be used for connecting to the public folder store when we need to access it for OAB, Free/Busy, etc.
3. Each time we need to connect to the information store we use the same session object and we will log the information about how many times we connect and how long they take to connect.

Now I needed to figure out why only the last one would fail. Through creating debug builds of store and system attendant so I could output maximum logging that was relevant to the problem I was able to obtain some information. Looking at the traces I could see that we would always return error: 0x478.

Dumping out this error I can see that we get ecWrongServer.

C:\err 0x478
# for hex 0x478 / decimal 1144
  ecWrongServer   ec.h
# 1 matches found for "0x478"

Now the question is why are we returning ecWrongServer. The information store has logic built in to it where we get redirected to another information store we will detect this change and try to connect where we are connecting too. This can happen for multiple reasons, for one too many connections in a short time. The system attendant will connect to the information store via the RPC interface and call the following function emsmdb!CNCT::EcForceRpc(). Through debugging I was able to see that we always got a return value of [0x47f = ECSERVERPAUSE].

Through code review I was able to see that when a client connects to a server it isn't configured for we will get an error code of ecWrongServer. This error code tells us that this is the wrong server and we will do a query to  determine the correct server. Once we have this information we give the client the correct server to connect too so we can attempt to connect again. In certain situations this could lead to a looping connection condition and cause RPC problems. To correct this behavior we added some logic we that keeps track of how many ecWrongServer errors have been sent to a particular client.

In addition to this if a client tries to make RPC calls more than 5 times under 10 seconds we will automatically return the error code of ecServerPaused error. This error will bubble back up the stack and be mapped back to the following error code: MAPI_E_FAILONEPROVIDER(0x8004011d). In this case these observations show that there is nothing wrong with the OAB itself and has to do with too many connection attempts.

One solution to this problem is to stagger your OAB's so they are built at different times so we do not hit this logic, which is  a safety mechanism to protect the store from RPC DOS type attacks (ligitement or not).

Dave

How to set the msExchQueryBaseDN attribute for users via powershell

I was working in the lab today trying to debug a reproduction of an OWA hosting issue and I had the need to set the msExchQueryBaseDN for 10,000 users. Typically you can use Poweshell with the get-user | set-user commands to access mailbox properties for stamping purposes, however the msExchQueryBaseDN attribute is a user based attribute. Here are two ways you can do it:

How to set an attribute for one user via powershell
$user = ([ADSI]"LDAP://OU=MyOU,DC=Company,DC=com").psbase;
$user.Properties["msExchQueryBaseDN"].Value = "CN=User,OU=MyOU,DC=Company,DC=com";
$user.CommitChanges();

How to set an attributes for a group of users in an OU via Powershell
get-mailbox userName* -resultsize unlimited | Foreach { $dn = "LDAP://" + $_.distinguishedname;$obj = [ADSI]$dn;$obj.msExchQueryBaseDN = "OU=YourOU,DC=company,DC=com";$obj.setinfo()}

This little bad boy stamped 10,000 users in less than 2 minutes.

Dave

OAB, Exchange Replication Service and the fix for OAB Generation failing on CCR and SCC clusters

Back on Thursday, December 11, 2008 i published a blog that contained information on the fix for OAB Generation failing on CCR and SCC clusters. This went in to detail as to what the problem was for both cluster types. Tim McMichael who is our Exchange Cluster guru has written a very indepth detailed blog on how the Exchange Replication Service (Exchange 2007 SP1) and Windows 2008 Clusters works. If you have time please take a look at his blog: http://blogs.technet.com/timmcmic/archive/2008/12/23/exchange-replication-service-exchange-2007-sp1-and-windows-2008-clusters.aspx

Dave

Exchange 2007 OAB Generation fails with the following errors: 9369, 9328 and 9328

I recently fixed an issue where the Default Offline Address List was on Exchange 2003, deleted and then recreated in Exchange 2007. In Exchange 2003 all Address lists create links in the active directory and in the public folder information stores under the System Folders | OFFLINE ADDRESS BOOK | /o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List.

When you delete an address list what you are doing is removing all references from that address list and the following things will happen:

1. The address list association on the private mailbox stores will be removed. This is the msExchUseOAB attribute that is stamped on the mailbox object. If this becomes blank clients will fail to download the OAB and receive the following error: 0X8004010F.
2. If you have a public folder store references to that OAB in the PF store might or might not get cleaned up based on how you have your mixed organization configured.
3. When you try to generate your newly created OAB, the OAB generation will fail with the following errors:

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9369
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered an error while publishing the OAB files to the distribution point 'Default Offline Address List'. Check other logged events to see more information about the problem. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9328
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered file error ffffffff (internal ID 505026d) while generating the offline address list for address list '\Global Address List'.  Make sure there is enough disk space available. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Error
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9328
Date:        12/17/2008
Time:        12:03:00 PM
User:        N/A
Computer:    OABGen
Description:
OALGen encountered file error ffffffff (internal ID 5050236) while generating the offline address list for address list '\Global Address List'.  Make sure there is enough disk space available. - Default Offline Address List

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The Problem

When you recreate that address list we are re-establishing the links to the objects that were previously deleted. When you try to rebuild the address list you can actually see in the application log that we make a connection to the public folder store even know the newly created address list has only be configured for Web Distribution:

Event Type:    Information
Event Source:    MSExchangeSA
Event Category:    OAL Generator
Event ID:    9134
Date:        12/17/2008
Time:        12:26:38 PM
User:        N/A
Computer:    OABGen
Description:
OALGen successfully created or opened the sub folder 'OAB Version 4' under the offline address list folder '/o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List'.
- Default Offline Address List

From here we will try to create our distribution point, however we will fail because the old OAB was never configured this way.

How to fix

1. Create your new Offline Address List (the one that is failing).

2. Only configure it for 'Public Folder Distribution'.

3. Rebuild your Offline Address List - you should see this complete with a 9107 event.

4. Go back and reconfigure your OAB for 'Web Distribution'.

5. Rebuild your Offline Address List - you should see this complete with a 9107 event.

6. Check your (CAS) Client Access Server for the 'General' 1008 event to make sure that the OAB files were copied over.

and that is it!

Dave

OAB Generation logs an Event ID 9373 when you update the offline address book in Exchange Server 2007

When you update the offline address book by using the Update-OfflineAddressBook command or by using a scheduled task, the following event (Event ID 9373) is logged in the Application log on a computer that is running Microsoft Exchange Server 2007:

Event Type: Error
Event Source: MSExchangeSA
Event Category: OAL Generator
Event ID: 9373
Description: OALGen detected that the file '\\<SERVER name>\ExchangeOAB\<folder name>\a58e388b-7b23-4944-b042-58a7d0c6590f-data-1.lzx' is corrupted or missing. This indicates data tampering or disk problems. Restore files in this folder from the recent backup or clean up folder content and force a full OAB generation. - Default Offline Address Book

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The reason this happens is that that when one or more OAB Version 4 offline address book files are corrupted or missing. In this case, the Microsoft Outlook clients cannot download the updated files until the issue is resolved. This can happen for quiet a few reasons:

• Virus scanning the OAB Virtual Directory on the (CAS) Client Access Server
• Disk problems. Some of the disk sectors that are containing the OAB files become corrupted
• Permissions problems on the files in the \\ExchangeInstallDir\ExchangeOAB\OAB GUID directory have changed
• By default, below accounts should have following Share and Security permissions on ExchangeOAB folder:
• SYSTEM” account should have “Full Control” permissions
• “Exchange Servers” group should have “Read Only”
• Local “Administrators” group should have “Full Control” permissions.
• All of the OAB files have *not* been replicated to the (CAS) Client Access Server. For more information on this please see the following blog: http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx

There are 3 fixes for this. To resolve this issue, use one of the following methods.

Method 1: Restore the contents of the OABGenation folder

If daily backups are available, restore the contents of the OABGen folder to the following folder:

ExchangeInstallDir \ExchangeOAB\ OAB GUID

Note ExchangeInstallDir is a placeholder for the name of the Exchange Server 2007 installation directory. OAB GUID is a placeholder for the name of the folder.

Method 2: Rebuild the OAB files by using Exchange Management Console or the Exchange Management Shell

If the corrupted offline address book item is enabled for public folders distribution, and if the data was published in public folders before the offline address book item was corrupted, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following methods:

• Use Exchange Management Console to rebuild the offline address book files. To do this, follow these steps:

1. Open Exchange Management Console.

2. Expand Organization Configuration, and then expand Mailbox.

3. On the details panel, click Offline Address Book, and then select the offline address book item that you want to update.

4. On the Action panel, click Update, and then click Yes.

• Use the Update-OfflineAddressBook command to rebuild the offline address book files.
To do this, run the following command in the Exchange Management Shell.

Update-OfflineAddressBook <OAB GUID> 

Note You can run the Update-OfflineAddressBook command on a computer that has the Mailbox, Client Access, or Hub Transport server role installed. To do this, you must log on by using an account that is a member of the Exchange Organization Administrators group. The account must also be a member of the local Administrators group on that computer.

 

Method 3: Rebuild the offline address book files by using a command

If neither daily backups nor public folders distribution is available, delete the contents of the OABGen folder from the ExchangeInstallDir \ExchangeOAB\ OAB GUID folder. Then, rebuild the offline address book files by using one of the following commands.

Update-OfflineAddressBook -Identity MyOAB 

Note In this example, the Update-OfflineAddressBook command is used to update the offline address book file that is named MyOAB.

Update-OfflineAddressBook <OAB GUID> 

Note When you use this command, the offline address book will be rebuilt. This will cause all clients to process a full offline address book download. For OAB Version 2 and OAB Version 3a, a full offline address book download can cause massive network traffic between client access servers and clients.

Dave

Fix for OAB Generation failing on CCR and SCC clusters

The failure of the offline address list to generate was the direct result of two separate issues.

 

The Windows 2008 shared storage clustering component has additional hooks into the operating system to manage shares created on shared storage. When programmatically creating shares on the cluster and then trying to generate the OAB would cause an exception. This was a result of cluster returning the full HResult for certain operations instead of the short code.

 

Now in Exchange 2007 SP1 RU5 we implement a code change to change the way that offline address lists are published on clustered servers.  Originally the OAB generation process would publish the OAB information to \\NodeName\ExchangeOAB.  The FDS service on the Client Access Servers would expect to copy the offline address list information from \\CMSName\ExchangeOAB.  Due to the integration of shared folders and cluster in Windows 2008, a share created on shared storage cannot either programmatically or manually be scoped to the NodeName.  The change implemented here now creates the share a \\CMSName\ExchangeOAB and then publishes the files to \\CMSName\ExchangeOAB.  Client Access Servers will continue to attempt to copy files from \\CMSName\ExchangeOAB.

 

Problem: OAB Generation on a Windows 2008 Shared Storage Cluster with Exchange 2007 SP1 fails with the following errors:

 

Log Name:      Application

Source:        MSExchangeSA

Date:          5/12/2008 9:23:50 AM

Event ID:      9334

Task Category: OAL Generator

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      Node-4.test.com

Description:

OALGen encountered error ffffffff while initializing the offline address list generation process. No offline address lists have been generated. Please check the event log for more information.

- Default Offline Address List

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="MSExchangeSA" />

    <EventID Qualifiers="49152">9334</EventID>

    <Level>2</Level>

    <Task>13</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2008-05-12T13:23:50.000Z" />

    <EventRecordID>16392</EventRecordID>

    <Channel>Application</Channel>

    <Computer>Node-4.test.com</Computer>

    <Security />

  </System>

  <EventData>

    <Data>ffffffff</Data>

    <Data>Default Offline Address List</Data>

  </EventData>

</Event>

 

Log Name:      Application

Source:        MSExchangeSA

Date:          5/12/2008 9:23:50 AM

Event ID:      9109

Task Category: OAL Generator

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      Node-4.test.com

Description:

OALGen encountered an error ffffffff (internal ID 505068d) while generating address list '/o=Test/cn=addrlists/cn=oabs/cn=Default Offline Address List'.  Check other logged events to see if this is a serious error.

- Default Offline Address List

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="MSExchangeSA" />

    <EventID Qualifiers="32768">9109</EventID>

    <Level>3</Level>

    <Task>13</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2008-05-12T13:23:50.000Z" />

    <EventRecordID>16391</EventRecordID>

    <Channel>Application</Channel>

    <Computer>Node-4.test.com</Computer>

    <Security />

  </System>

  <EventData>

    <Data>ffffffff</Data>

    <Data>505068d</Data>

    <Data>/o=E-McMichael Home/cn=addrlists/cn=oabs/cn=Default Offline Address List</Data>

    <Data>Default Offline Address List</Data>

  </EventData>

</Event>

 

This occurs whether the generation process is spawned through update-offlineaddresslist or through a scheduled update.  This also appears to stall both the file system publishing and public folder publishing of the offline address list and its updates.

 

Additional Information

The issue does not occur if the OAB is generated on local storage (disks not under control of cluster) or if the disk hosting the OAB is in maintenance mode.

 

The issue appears at this time to be related to how Windows 2008 clusters handle file share / file server resources.

 

The Fix

1. Install KB article - 955733 - Incorrect status codes that are returned in failover clusters may cause operations to fail on a Windows Server 2008-based computer: http://support.microsoft.com/default.aspx?scid=kb;EN-US;955733 This ensures that cluster returns the correct status codes where necessary.

2. Install Exchange Rollup 5 - http://www.microsoft.com/downloads/details.aspx?FamilyID=41ec35e7-79a0-4ecb-a644-1ece94178f9d&displaylang=en

Warning message when you start Outlook 2007 and then connect to a mailbox that is hosted on an Exchange 2007-based server: "The name of the security certificate is invalid or does not match the name of the site"

When you start Microsoft Office Outlook 2007 and then connect to a mailbox that is hosted on a mailbox server that is running Microsoft Exchange Server 2007, you receive the following security warning message:

The name of the security certificate is invalid or does not match the name of the site.

Note The scenario that is described in this article only applies to Outlook clients that connect to Exchange from inside the local network. The scenario that is described in this article does not apply to remote Outlook clients that connect to Exchange by using Outlook Anywhere.

CAUSE: This issue occurs if the following conditions are true:

•You replace the default self-signed Exchange 2007 certificate with a different certificate.
Note The Exchange 2007 Setup program creates a default self-signed certificate when Exchange 2007 is installed.

• The common name on the replacement certificate does not match the fully qualified domain name (FQDN) of the URL that is stored in the following objects:

• The Service Connection Point object for the Autodiscover service

• The InternalUrl attribute of Exchange 2007 Web Service (EWS)

• The InternalUrl attribute of the Offline Address Book Web service

• The InternalUrl attribute of the Exchange unified messaging (UM) Web service

By default, the URL that is stored in these objects references the NetBIOS name of the server. For example, a URL that resembles the following URL is stored:

https:// NetBIOS_name .contoso.com/autodiscover/autodiscover.xml

This may differ from the host name that is used in the FQDN of the replacement certificate. For example, the replacement certificate may have an FQDN that resembles the following FQDN:

mail.contoso.com

This issue causes a name mismatch error to occur. Therefore, you receive the security warning message when you try to connect Outlook 2007 to the mailbox.

To resolve this issue, modify the URLs for the appropriate Exchange 2007 components. To do this, follow these steps:

1. Start the Exchange Management Shell.

2. Modify the Autodiscover URL in the Service Connection Point. The Service Connection Point is stored in the Active Directory directory service. To modify this URL, type the following command, and then press ENTER:

Set-ClientAccessServer -Identity CAS_Server_Name -AutodiscoverServiceInternalUri https:// mail .contoso.com/autodiscover/autodiscover.xml

3. Modify the InternalUrl attribute of the EWS. To do this, type the following command, and then press ENTER:

Set-WebServicesVirtualDirectory -Identity " CAS_Server_Name \EWS (Default Web Site)" -InternalUrl https:// mail .contoso.com/ews/exchange.asmx

4. Modify the InternalUrl attribute for Web-based Offline Address Book distribution. To do this, type the following command, and then press ENTER:

Set-OABVirtualDirectory -Identity " CAS_Server_name \oab (Default Web Site)" -InternalUrl https:// mail .contoso.com/oab

5. Modify the InternalUrl attribute of the UM Web service. To do this, type the following command, and then press ENTER:

Set-UMVirtualDirectory -Identity " CAS_Server_Name \unifiedmessaging (Default Web Site)" -InternalUrl https:// mail .contoso.com/unifiedmessaging/service.asmx

6. Open IIS Manager.

7. Expand the local computer, and then expand Application Pools.

8. Right-click MSExchangeAutodiscoverAppPool, and then click Recycle.

Important These steps assume that a host record exists in the DNS to map the FQDN that you specify to the IP address of the CAS server. Consider the following sample scenario:

• The original internal URLs for the Exchange components point to the internal FQDN of the server. For example, one of these URLs points to the following:

https:// ServerName .contoso.com/ews/exchange.asmx

• The FQDN that is specified on the certificate points to the externally accessed host name of the server. For example, the certificate specifies an FQDN, such as "mail.contoso.com."

In this scenario, you must add a host record for the mail host name that is mapped to the internally accessed IP address of the CAS server to let internal clients access the server.

MORE INFORMATION

The URL for the Autodiscover service is stored in the Service Connection Point object. By default, this URL references the internal FQDN of the CAS that is present when Autodiscover is installed. For example, the following URL is set:

https:// servername .contoso.local/autodiscover/autodiscover.xml

In this example, the FQDN references the internal namespace. Generally, this namespace differs from the externally-accessible namespace, such as mail.contoso.com.

If the internal namespace differs from the external namespace, and if you cannot use a certificate that supports Subject Alternative Names, use the Set-ClientAccessServer task in Exchange Management Shell to modify the URL. In this scenario, you must modify the URL to point to the new location for Autodiscover. For example, use the following command to point to the new location for Autodiscover:

Set-ClientAccessServer –AutodiscoverServiceInternalUri https:// mail .contoso.com/autodiscover/autodiscover.xml

For more information about third-party certification authorities that provide certificates that support Subject Alternative Names, click the following article number to view the article in the Microsoft Knowledge Base:

929395 Unified Communications Certificate Partners for Exchange 2007 and for Communications Server 2007

Dave

OABInteg and certificate testing

I just have finished a new code path for OABInteg that will now allow you to do a proactive scan against your active directory to see what your mail enabled objects look like when it comes to certificates. This should help you to eliminate certificates from your active directory so you can reduce your OAB's overall size.

Currently there are three attributes that ship with Windows 2003 and Exchange 2003 that can store user certificates: userCert, userCertificate, and userSMIMECertificate.

Information on Certificates

• userCert - UserCert is a single valued attribute that stores the old Nortel style certificates used long ago with Key Management Server for Exchange (KMS).
• userCertificate -  Exchange and Outlook use it to store DER encoded X.509 e-mail certificates, and Windows uses it to store the public keys for logon, EFS and other such keys.
• userSMIMECertificate - UserSMIMECertificate is used only by Exchange and Outlook for PKCS-7 encoded e-mail certificates and all certificates stored here are supposed to be usable for e-mail.

For the full blog on OAB certificate filtering please see: http://msexchangeteam.com/archive/2005/07/25/408188.aspx. This can be ran from an Outlook client however the filtering settings will be set to the default and if you run this on the Exchange server it will detect the OAB certificate registry key and give you a more accurate test. To run the test you can run the following command: [OABInteg /s:srvname /t:certtest /v:2 /l] (/l will create a c:\oabinteg.txt file for logging purposes). Here is what the output looks like when you run this test:

Processing Address Book Entry #1 of 50.
Display Name = Test User
No certificate found.

Processing Address Book Entry #2 of 50.
Display Name = Test User
Certificate is enhanced for email.

Processing Address Book Entry #3 of 50.
Display Name = Test User
Certificate is ok for email.
Certificate is ok only for email.
Certificate is enhanced but not for email.
We are filtering certificates so this certificate was filtered out.

Processing Address Book Entry #4 of 50.
Display Name = Test User
Certificate is ok for email.
Certificate is ok only for email.
Certificate is enhanced but not for email.

NOTE: The following entry indicates that the object has multiple certificates in the active directory.

Processing Address Book Entry #5 of 50.
Display Name = Test
Certificate is ok for email.
Certificate is ok only for email.
Certificate is enhanced but not for email.
We are filtering certificates so this certificate was filtered out.
Certificate is ok for email.
Certificate is ok only for email.
Certificate is enhanced but not for email.
We are filtering certificates so this certificate was filtered out.
Certificate is ok for email.
Certificate is ok only for email.
Certificate is enhanced but not for email.
We are filtering certificates so this certificate was filtered out.

Rest of users removed for readability....

Scan Finished
Total certificates filtered - 1303
Total certificates included in the OAB - 57
Total invalid certificates found - 0
Total expired certificates found - 146
Total certificates found that are not signed - 0
Total certificates with invalid usage - 1
Total enhanced certificates found - 1227
Total email certificates found - 1227
Total email enhanced certificates found - 128
Total invalid enchanged certificates found - 1
Total expired certificates found - 71
Total certificates with invalid bahaviour found - 0
Total objects without certificates - 6358

OABInteg-Admin_03:55:25 PM profile was found and deleted from the Windows Messaging Subsystem.

Logoff of profile successful.
Closing MAPI session.
The MAPI subsystem was un-initialized successfully.

Performing cleanup.
Exiting application.

OABInteg-Admin_03:40:53 PM profile was found and deleted from the Windows Messaging Subsystem.

You can download the latest version of OABInteg from here: http://code.msdn.microsoft.com/oabinteg

Dave

Understanding why error code 0X8004010F is thrown when trying to download an OAB

When it comes to downloading an OAB the following error code [ 0X8004010F ] is the biggest pain for everyone, and the most misunderstood error. The number one thing is to understand is that this error means MAPI_E_NOT_FOUND:

C:\dgoldman>err 0X8004010F
# for hex 0X8004010F / decimal -2147221233
ecNotFound ec.h
ecAttachNotFound ec.h
ecUnknownRecip ec.h
ecPropNotExistent ec.h
MAPI_E_NOT_FOUND mapicode.h
# 5 matches found for "0X8004010F"

With regards to downloading an OAB, if you see this error it means your OAB was not found. This can also be throw for a few different reasons based on your exchange configuration:

1. A mixed organization that is going through a migration (2000- 2003), (2000-2007), etc
2. You had public folder problems like: (replication, lost a public folder server and rebuilt it, had public folders rehomed)
3. You just starting using Outlook 2007 and are trying to use HTTP/RPC (Outlook Anywhere).

Here is a high level overview on the client server interaction when downloading an OAB.

1. Every morning at 5am or based on a custom schedule the Exchange server that is hosting the OAB will start an OABGen rebuild task.
2. The System Attendant invokes oabgen.dll to start the rebuilt and pass along the OAB to be built.
3. OABGen.dll will query a domain controller for the data needed to build an oab. While the data is being processed it will be split the data to different temp files residing on the c:\windows\temp directory.
4. OABGen.dll will make a connection to a public folder store and download last nights public folder OAB files so they can be compared. This is where we will create the OAB differential files.
5. OABGen.dll will make another connection to the public folder store it is associated with and posted a system message with the new files to the OAB Version 2, OAB Version 3 or OAB Version 4 folders. If you have Exchange 2007 the data will be posted to the local \\ExchangeOAB directory and then the FDS service on the CAS (Client Access Server) will copy over the new OAB files so they can be download by the client.
6. The msExchUseOAB attribute. This attribute contains the DN of the offline address list to be downloaded. This attribute resides in two places (1. The user object in the active directory - Domain Naming Context Container, and 2. on the information store object). When the outlook client starts up it will look at its own user object in the active directory to see if there is anything in the msExchUseOAB attribute, and if there is this will override the information store settings. Having this set to an offline address list that no longer exists will cause the download to fail. If there is nothing on the user object then a query for the msExchUseOAB attribute on the server object where the mailbox resides. The MAPI property equivelant for this is called PR_ADDRBOOK_FOR_LOCAL_SITE_ENTRYID and contains an EntryID that looks like the following: 0000000001A447390AA6611CD9BC800AA002FC45A0300B4CF3F5BB03DC0499AB07A96F707B1AB0000000000080000
7. The Outlook client will call the OpenFolder API on that EntryID which will corrosponds to the correct OAB root folder and then will try to open the OAB folders in the following format: OAB Version 4, OAB Version 3 or OAB Version 2.
8. Based on the service pack, version of the Outlook cilent and version of the Exchange Information Store (store.exe) the client will connect to one of the OAB folders. In order for an Outlook client to connect to the OAB Version 4 folders it must be on Exchange 2003 SP2 and Outlook SP2 or Outlook 2007. Based on this service pack / hot fix combination the Outlook client will attempt to connect to the latest version is had knowledge about.

For Outlook 2003 (all service packs)

1. For any reason if the client cannot connect to that the OAB folder it is looking for it will then attempt to connect to the next lowest version. Example (Connect to OAB Version 4 = failed, now try to connect to OAB Version 3 = failed, now try to connect to OAB Version 2 = failed), and now we throw the error: 0X8004010F.

Now this can be for a few reasons:

1. The mailbox store your client is residing on does not have an OAB associated with it.
2. The msExchUseOAB attribute is populate with a DN of an OAB that doesnt exist.
3. The msExchUseOAB attribute on the information store is blank.  If you look in the ESM this is the OAB that is associated with that mailbox store and this can be reset.
4. Someone has deleted the Default Global Address List. In this case this will break the link between the Default Offline Address List and the Default Global Address List and remove it from the information store object.
5. You have multiple public folder stores and the OAB generation process has posted them to a different public folder store than you thought. Replication has not taken effect yet, thus there will be no new files or OAB files to download.
6. Make sure that your SMTP connectors do not have restrictions blocking replicas from being passed to different public folder servers.
7. The OAB generation process had a problem and terminated. In this case the files would not be posted to the public folder store.
8. An administrator decommissioned the last Exchange server in a site and never pushed the replicas to another Exchange server.
9. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything.
10. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download.
11. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox.
12. The OAB was never generated or some OAB folders are missing from the public folder store.
13. Multiple OAB Version folders exist of the same type.
14. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates.
15. The offline address book list object has a missing address list. 
16. The offline address book list object has an incorrect address list.
17. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox.

For Outlook 2007 (all service packs)

1. If you have installed Exchange 2007 in to an Exchange 2003 organization you need to make sure that you have added the replicas of OAB to the Exchange 2007 server.
2. Make sure public folder replication is working between servers. 
3. An Autodiscover DNS record has not be setup by your ISP therefor will break HTTP/RPC because you will not be able to connect.
4. You have set an invalid proxy setting in IE. Windows, Outlook and IE all use the same WinHTTP Proxy settings and if this proxy is incorrect all Outlook WinHTTP calls fill fail.
5. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 
6. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog:

http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and
http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx

7. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process.

8. The OAB may not exist is if you have a cluster and that node that is currently inactive, and also is the only one set to generate the OAB.  For more information please see: http://blogs.msdn.com/dgoldman/archive/2007/02/08/oab-generation-on-a-cluster-server-fails-with-event-id-9395-or-9396.aspx

  Dave

How to get HideFromAddressBook to work on Exchange 2007

Often times customer have a need to create bulk contacts for their organization, but have business requirements to hide these contacts from the global address list. I am just listing two simple ways that you can hide them from the address list. These ways can be as complex as you want them to be depending on your powershell scripting.

You can create your bulk contacts by running the following command in powershell:

New-MailContact -ExternalEmailAddress 'SMTP:TestUser@YourCompany.com' -Name 'Test' -OrganizationalUnit 'YourDomain.com/Contacts/YourCompany' -FirstName 'Test' -LastNane 'User'.

Now that the contacts have been created the second part of this is to hide them from the global address list. There are a few different ways to do this and I only listed two (by searching the entire gal or or by DN).

General Filter: get-mailcontact * | set-mailcontact -hiddenfromaddresslistsenabled $true

By DN: get-mailcontact -OrganizationalUnit "DC=YourDomain,DC=COM,OU=TargetOU" | set-mailcontact -hiddenfromaddresslistsenabled $true

Dave