Dgoldman's WebLog : Offline Address Book Related

Understanding why error code 0X8004010F is thrown when trying to download an OAB

dgoldman — Wed, 01 Oct 2008 19:15:00 GMT

When it comes to downloading an OAB the following error code [ 0X8004010F ] is the biggest pain for everyone, and the most misunderstood error. The number one thing is to understand is that this error means MAPI_E_NOT_FOUND:

C:\dgoldman>err 0X8004010F
# for hex 0X8004010F / decimal -2147221233
ecNotFound ec.h
ecAttachNotFound ec.h
ecUnknownRecip ec.h
ecPropNotExistent ec.h
MAPI_E_NOT_FOUND mapicode.h
# 5 matches found for "0X8004010F"

With regards to downloading an OAB, if you see this error it means your OAB was not found. This can also be throw for a few different reasons based on your exchange configuration:

A mixed organization that is going through a migration (2000- 2003), (2000-2007), etc
You had public folder problems like: (replication, lost a public folder server and rebuilt it, had public folders rehomed)
You just starting using Outlook 2007 and are trying to use HTTP/RPC (Outlook Anywhere).

Here is a high level overview on the client server interaction when downloading an OAB.

Every morning at 5am or based on a custom schedule the Exchange server that is hosting the OAB will start an OABGen rebuild task.
The System Attendant invokes oabgen.dll to start the rebuilt and pass along the OAB to be built.
OABGen.dll will query a domain controller for the data needed to build an oab. While the data is being processed it will be split the data to different temp files residing on the c:\windows\temp directory.
OABGen.dll will make a connection to a public folder store and download last nights public folder OAB files so they can be compared. This is where we will create the OAB differential files.
OABGen.dll will make another connection to the public folder store it is associated with and posted a system message with the new files to the OAB Version 2, OAB Version 3 or OAB Version 4 folders. If you have Exchange 2007 the data will be posted to the local \\ExchangeOAB directory and then the FDS service on the CAS (Client Access Server) will copy over the new OAB files so they can be download by the client.
The msExchUseOAB attribute. This attribute contains the DN of the offline address list to be downloaded. This attribute resides in two places (1. The user object in the active directory - Domain Naming Context Container, and 2. on the information store object). When the outlook client starts up it will look at its own user object in the active directory to see if there is anything in the msExchUseOAB attribute, and if there is this will override the information store settings. Having this set to an offline address list that no longer exists will cause the download to fail. If there is nothing on the user object then a query for the msExchUseOAB attribute on the server object where the mailbox resides. The MAPI property equivelant for this is called PR_ADDRBOOK_FOR_LOCAL_SITE_ENTRYID and contains an EntryID that looks like the following: 0000000001A447390AA6611CD9BC800AA002FC45A0300B4CF3F5BB03DC0499AB07A96F707B1AB0000000000080000
The Outlook client will call the OpenFolder API on that EntryID which will corrosponds to the correct OAB root folder and then will try to open the OAB folders in the following format: OAB Version 4, OAB Version 3 or OAB Version 2.
Based on the service pack, version of the Outlook cilent and version of the Exchange Information Store (store.exe) the client will connect to one of the OAB folders. In order for an Outlook client to connect to the OAB Version 4 folders it must be on Exchange 2003 SP2 and Outlook SP2 or Outlook 2007. Based on this service pack / hot fix combination the Outlook client will attempt to connect to the latest version is had knowledge about.

For Outlook 2003 (all service packs)

For any reason if the client cannot connect to that the OAB folder it is looking for it will then attempt to connect to the next lowest version. Example (Connect to OAB Version 4 = failed, now try to connect to OAB Version 3 = failed, now try to connect to OAB Version 2 = failed), and now we throw the error: 0X8004010F.

Now this can be for a few reasons:

The mailbox store your client is residing on does not have an OAB associated with it.
The msExchUseOAB attribute is populate with a DN of an OAB that doesnt exist.
The msExchUseOAB attribute on the information store is blank. If you look in the ESM this is the OAB that is associated with that mailbox store and this can be reset.
Someone has deleted the Default Global Address List. In this case this will break the link between the Default Offline Address List and the Default Global Address List and remove it from the information store object.
You have multiple public folder stores and the OAB generation process has posted them to a different public folder store than you thought. Replication has not taken effect yet, thus there will be no new files or OAB files to download.
Make sure that your SMTP connectors do not have restrictions blocking replicas from being passed to different public folder servers.
The OAB generation process had a problem and terminated. In this case the files would not be posted to the public folder store.
An administrator decommissioned the last Exchange server in a site and never pushed the replicas to another Exchange server.
A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything.
The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download.
An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox.
The OAB was never generated or some OAB folders are missing from the public folder store.
Multiple OAB Version folders exist of the same type.
Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates.
The offline address book list object has a missing address list.
The offline address book list object has an incorrect address list.
Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox.

For Outlook 2007 (all service packs)

If you have installed Exchange 2007 in to an Exchange 2003 organization you need to make sure that you have added the replicas of OAB to the Exchange 2007 server.
Make sure public folder replication is working between servers.
An Autodiscover DNS record has not be setup by your ISP therefor will break HTTP/RPC because you will not be able to connect.
You have set an invalid proxy setting in IE. Windows, Outlook and IE all use the same WinHTTP Proxy settings and if this proxy is incorrect all Outlook WinHTTP calls fill fail.
Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store.
Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog:

http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and
http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx

If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process.
The OAB may not exist is if you have a cluster and that node that is currently inactive, and also is the only one set to generate the OAB. For more information please see: http://blogs.msdn.com/dgoldman/archive/2007/02/08/oab-generation-on-a-cluster-server-fails-with-event-id-9395-or-9396.aspx

Dave

Error 0x80190194 when using an Outlook 2007 client to download a Web Distribution enabled OAB

dgoldman — Mon, 27 Nov 2006 08:21:00 GMT

You can receive the following error (0x80190194) in Outlook when trying to download a Web Distribution only enabled Offline Address Book. The error 0x80190194 is an HTTP 404 error:

C:\WINDOWS\system32>err 0x80190194
# for hex 0x80190194 / decimal -2145844844
BG_E_HTTP_ERROR_404 bitsmsg.h
# 1 matches found for "0x80190194"

The reason for this error message is that when an Outlook 2007 client downloads the OAB files they are not being found on the Client Access Server.

Here is the flow for downloading the OAB:

An Outlook client starts up.
The Outlook client talks to the autodiscover service and retrieves the autodiscover.xml file which contains the URLs for all of the Exchange Web Services.
The Outlook client parses this autodiscover.xml file and looks for the <EXCH> section which is for the internal corporate network, and <EXPR> for Outlook Anywhere (External).
You select Tools | Send/Receive | Download Address Book.
The Outlook client looks for the OABUrl which looks like this: <OABUrl>http://OABServer.domain.com/OAB/edb3e44f-c49b-4597-b996-cb1093405985/</OABUrl>.
Once the Outlook client locates the OABUrl the Outlook client makes a HTTPS connection to the OAB Web Virtual Directory specified in the <OABUrl>.
If the connection to the OAB Web Virtual Directory is successful the Outlook client it will look in the directory for the OAB you are download and retrieve the OAB.xml file.
The Outlook client will examine the OAB.xml manifest file and find out what OAB files are needed for updating the Outlook client's local copy of the OAB.

Now, the Default OAB Web Site on the Client Access Server will have a local directory associated with it that has the path of: <Drive Letter>:\Program Files\Microsoft\Exchange Server\ClientAccess\OAB. This is the location for the OAB files when they are copied over from the Mailbox server by the FDS (File Distribution Service).

If you are in a single server site these OAB files should be copied over immediately to this directory after the OAB has been generated. If you have a multi-server site where your Client Access Server that is separate from your Mailbox server that is generating the OAB, the OAB files will be copied over under a few situations (not all are listed here):

A new OAB is created
A change to the OAB has happened
The OAB Generation Server has changed
The Default Polling Interval is up (this is 8 hrs by default).
There are new OAB files (new binpatch.oab file has been created).

One way to tell if your OAB files have been replicated to the Default OAB Web Site is to look for the event id 1008 in the application log. This event will be logged by the FDS (File Distribution Service) on the Client Access Server.

Event Type: Information
Event Source: MSExchangeFDS
Event Category: FileReplication
Event ID: 1008
Date: 11/28/2006
Time: 1:55:46 PM
User: N/A
Computer: OABServer
Description:
Process MSExchangeFDS.exe (PID=2484). Offline Address Book data synchronization task has completed successfully. OAB name: "Default Offline Address Book", Guid: edb3e44f-c49b-4597-b996-cb1093405985

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

If the OAB files are missing you will need to find out why the FDS service is not replicating them.

Workaround 1

Run 'net stop MSExchangeFDS' and then 'net start MSExchangeFDS' to restart the Microsoft Exchange File Distribution Service on your (All in one Exchange Server) or on your Client Access Server.
Check for the 1008 event id after the services are restarted.

Workaround 2 - This is just a workaround ONLY until you get the FDS replication fixed

You can go your mailbox server (OAB Generation Server) and go to one of the following locations:

Local directory - C:\Program Files\Microsoft\Exchange Server\ExchangeOAB

Unc path - \\ExchangeServerName\OAB\<GUID of OAB>

Copy the OAB directories that are needed for your clients over to the client access server location:

c:\Program Files\Microsoft\Exchange Server\Client Access\OAB\

Once the files have been replicated to this directory the Outlook client will be able to download them and you will not receive this error.

Dave

Outlook client fails to download the OAB with error 0x8004011B

dgoldman — Mon, 23 Oct 2006 20:00:00 GMT

Earlier today I saw a post on one of the aliases where a customer was having issues downloading the OAB with their Outlook 2003 clients. The Outlook client was failing with the error code of 0x8004011B. I asked a few questions in regards to the customers exchange environment, and I found out that they have an Exchange 2003 / Exchange 2007 mixed organization.

The first thing I did was dump out the error code 0x8004011B so I could see what it was. I used err.exe to dump this out.

C:\WXP\system32>err 0x8004011B
# for hex 0x8004011b / decimal -2147221221
ecCorruptData ec.h
MAPI_E_CORRUPT_DATA mapicode.h
# 2 matches found for "0x8004011B"

Ok, from this error I can see that we are possibly looking at either some corrupted data or the data is actually missing from the public folder information store. From looking at the application log events I was able to pull out the Event ID 27.

Event Type: Warning
Event Source: Outlook
Event Category: None
Event ID: 27
Date: 10/20/2006
Time: 4:24:56 PM
User: N/A
Computer: Computer
Description: OAB Download Failed. (Result code in event data).
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:
0000: 01 00 00 00 1b 01 04 80 .......€
0008: 01 00 00 00 18 02 00 00 ........
0010: 01 00 00 00 00 00 00 00 ........
0018: 00 00 00 00 00 00 00 00 ........
0020: e9 fd 00 00 57 f5 c9 dc éý..WõÉÜ
0028: 80 04 00 00 00 00 00 00 €.......
0030: 00 00 00 00 00 00 00 00 ........
0038: 00 00 00 00 00 00 00 00 ........
0040: 00 00 00 00 00 00 00 00 ........
0048: 00 00 00 00 5c 00 00 00 ....\...
0050: 00 00 00 00 03 00 00 00 ........
0058: 02 00 00 00 6c 69 61 7a ....liaz
0060: 6c 69 61 7a 00 00 00 00 liaz....
0068: 00 00 00 00 00 00 00 00 ........
0070: 00 00 00 00 00 00 00 00 ........
0078: 00 00 00 00 00 00 00 00 ........
0080: 00 00 00 00 00 00 00 00 ........
0088: 00 00 00 00 00 00 00 00 ........
0090: 00 00 00 00 00 00 00 00 ........
0098: 00 00 00 00 ....

Looking at the data above I can see two very interesting things:

At offset 0000 the first byte is 01. Looking up this error code I know 01 means that you do not have any offline address book files on your client computer, or the offline address book files could not be opened.
At offset 0090 there is not address list displayed.

From looking at offset 0090 I know that the possibility of corrupted data is slim, however does the OAB data actually exist? From here there are two things that needed to be checked.

Was the OAB ever generated and do the OAB files exist in the form of a message and attachments in the information store?
Is there an OAB associated with the customers mailbox store?

From here I suggested that the customer get me an OABInteg output with the two following tests (oabfldcheck and storealtest). Both of these tests will let me know the following:

If the OAB has been generated and posted to the public information store.
If the mailbox information store has an OAB associated with it.
If there is corrupted data in the public folder information store OABInteg will fail to read the OAB messages and attachments, so this will allow us to rule out the possibility of data corruption.

The customer ran the following test c:\oabinteg.exe /s:SrvName /t:storealtest /v:2 /l (this creates a c:\OABInteg.txt file):

The output for this test showed the following:

OABInteg (Offline Address Book Integrity Checker)
Product Version 06.05.7839.0
OABInteg.exe
Microsoft Corporation, Copyright (C) 2006
Microsoft and Windows are registered trademarks of Microsoft Corporation.
=====================================================

c:\OABinteg.txt has been opened for writing.

Program started at: 08:53:12 AM
Running OABInteg on: Username\Server
LDAP has been initialized successfully.
Using ldap port: 3268
Trying to connect to: GC://DomainController
Initiating search for rootDSE attributes.
rootDSE attributes found and cached.

Starting Test 1 - Store Offline Address List Test

LDAP has been initialized successfully.
Using ldap port: 3268
Global Catalog Server Selected: DomainController
Path: GC://DomainController/CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Domain,DC=com
Binding to the active was directory successful.
Ldap Search Page Size: 64

Search filter to: (objectClass=msExchPrivateMDB)
Search attributes: cn, distinguishedName, msExchUseOAB

Search started at: 08:53:12 AM

Search ended at: 08:53:12 AM

Scan Completed
+--------------------+
Total information store address lists found: 0

So from this I can see that the information store on this server does not have an OAB associate with it. The customer then ran the following test c:\oabinteg.exe /s:SrvName /t:oabfldcheck /v:2 /l (this creates a c:\OABInteg.txt file):

The output for this test showed the following:

=====================================================
OABInteg (Offline Address Book Integrity Checker)
Product Version 06.05.7839.0
OABInteg.exe
Microsoft Corporation, Copyright (C) 2006
Microsoft and Windows are registered trademarks of Microsoft Corporation.
=====================================================

c:\OABinteg.txt has been opened for writing.

...<I cut the data out because we really care about what is below>...

Scan Completed
+--------------------+
Message Class Normal found: 3
Message Class Differential found: 80
Message Class Unknown found: 0
Message Attachments found: 0
Messages found but unable to read the properties: 0
System folders found: 4
Highest sequence number found: 538
Lowest sequence number found: 2
Biggest attachment found: 0 Bytes
Smallest attachment found: 36500 Bytes
Biggest message found: 2.2 MB
Smallest message found: 326 Bytes

From this output I can see that the OAB has been generated. If you look at the output above you can see that there are 3 normal message (OAB messages), 80 differential files and 4 system folders. So based on this output I know that the OAB was generated and posted to the public folder information store. Now we know the problem is there is no OAB associated with the mailbox information store.

The customer was able to verify and fix this by pulling up the properties of the mailbox information store and look to see what Offline Address List is assigned to this store. There wasn't one associated there. The customer then linked this mailbox information store to the OAB on the Exchange 2007 public folder store. The customer had an Outlook client login and they were able to download the OAB files.

Dave

The Exchange System Attendant fails to start with 9175 and MAPI Error 80040111-0286-00000000

dgoldman — Sun, 15 Oct 2006 22:03:00 GMT

Often people will ask me a question like "What do you do as an Escalation Engineer at Microsoft?" Being an EE is a really exciting job because I get to see and do a lot of things that most other engineers can't or will never get a chance to do. I felt that I would talk a little about what I do as an EE. A normal day at work for me starts by sending out emails to the 10 or so customer cases that I have in my work bin. Once that is done I will triage some of the other cases that the guys have and offer help if need be. If everything is ok there I will then start on my own cases. One of the cases that I got was going on for quite some time and needed to be solved. Now that I have this case I can do what I do best.

This case was another interesting OAB issue for me as I have not seen this type of behavior before. Basically the problem was the customer was having issues generating the OAB, however this was not the typical OAB generation failure that you would normally see on a server. For some reason the active node on this two node cluster would just stop generating the OAB. Ok cool since this was a cluster the customer failed over to the passive node, and the OAB would generate for a short while and then start failing as well. Failing back the other node *did not* fix it and now both nodes were broken. So now I am starting to wonder what is going on here.

I started up an easy assist session so I could get live access to the customer's server so I could see the logs, etc. I was able to find that her was unable to rebuild the OAB because of a provider initialization error, or so I thought at that time. The errors looked like this:

Exchange System Manager Error
The information store could not be opened. The logon to the Microsoft Exchange server computer failed.
MAPI 1.0 ID no: 80040111-0286-00000000
ID no: C1050000 Exchange System Manager

Application Log Error
Event ID              : 9175
Event Type         : Error
Event Source     : MSExchangeSA
Event Category : MAPI Session
Description        : The MAPI call 'OpenMsgStore' failed with the following error: The information store could not be opened. The logon to the Microsoft Exchange Server computer failed. MAPI 1.0 ID no: 80040111-0286-00000000

Now when troubleshooting issues like this I check to see if the customer is using the DS Servers registry value to load balance connections to a global catalog server. As this is really not a recommended practice I need to rule it out. For more information on this registry key, see this blog: The DS Server registry key and rebuilding an Offline Address List. What was interesting here is that the customer was not using the DS Servers registry, however he was failing with the same errors as listed in the blog and above.

I set up to do a live debug on the customers server and while rebuilding the OAB I was able to get to the following call stack in the debugger:

1647ecc4 616a9e94 emsabp32!NetBind+0x54
1647ece8 616ad524 emsabp32!BindServer+0x6e
1647ed44 616addb4 emsabp32!DsNetLogon+0x295
1647eea8 616acb5e emsabp32!GetABServer+0x2c7
1647f304 61e4b6b1 emsabp32!ABP_Logon+0x310
1647f358 61e4b477 mapi32!ABCustomLoad+0xde
1647f3f0 61e4bae4 mapi32!HrCommonLoadProviders+0x3d4
1647f494 61e4d34a mapi32!HrIntOpenAddressBook+0x57
1647f50c 61e4ce8b mapi32!MAPILogonX+0x494
1647f54c 0041b6e5 mapi32!MAPILogonEx+0x1b8

What is interesting to me about this call stack is that we end up in emsabp32!NetBind. Now in order for OABGen.dll to generate an Offline Address List, the System Attendant must first do the following things:

1. Initalize the MAPI subsystem and create a system attendant profile.
2. Log on to that profile.
3. Open and initialize the Address Book Provider (loads emsabp32.dll in to the mad.exe process space).
4. Load other customer providers that are needed.
5. Locate a domain controller that we can connect to.
6. Get the binding order from the registry on the local server so we can bind to the domain controller we are going to use.
7. Perform the actual Directory Service bind to the domain controller so we can perform our NSPI queries and request the data we want.

So while I was debugging the emsabp32!Netbind function I noticed that one of the bind operations failed, and the EAX debugger register which contains our return data from the called function contained this error code:

0:029> p
eax=80040111 ebx=80004005 ecx=77ce21ce edx=173c7bf0 esi=149b49e4 edi=00000000
eip=616a9d13 esp=1647ec6c ebp=1647ecc4 iopl=0 nv up ei pl nz ac pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000216

emsabp32!NetBind+0x54:
616a9d13 8bd8 mov ebx,eax

Interestingly enough this is the same error that is being displayed in the dialog box from the ESM, and is also in the application log. If you run 80040111 through the err.exe tool, and you will get the following output:

C:\WINXP\system32>err 80040111
# for hex 0x80040111 / decimal -2147221231
ecLoginFailure ec.h
MAPI_E_LOGON_FAILED mapicode.h

Well, from looking at this error this is partially true because we did fail to logon to the domain controller, but this wasn’t the root cause. Being that I did not have my root cause yet I needed to keep researching. I dumped out the data structures that contained the domain controller information and now I could see where we were connecting to:

0:029> dt pServerGuid
Local var @ 0x1647ecd8 Type __MIDL_nspi_0001*
0x14a3ad10
+0x000 ab : [16] "Domain Controller Name"

Ok, now that I have the domain controller we are using I can dump out the local variables. If you look at the scode, it is -2147221231, which is the same error as 80040111.

0:029> dv
hRpc = 0x173c7b78
pConnect = 0x149b49e4
pServerGuid = 0x1647ee70
pszServer = 0x14a3ad10 "Domain Controller Name"
scode = -2147221231
status = 9
stat = struct __MIDL_nspi_0002
fRetry = 1

I run -2147221231 through error for an extra verification:

C:\WINXP\system32>err -2147221231
# for decimal -2147221231 / hex 0x80040111
ecLoginFailure ec.h
MAPI_E_LOGON_FAILED mapicode.h
CLASS_E_CLASSNOTAVAILABLE winerror.h
# ClassFactory cannot supply requested class
# 3 matches found for "-2147221231"

Now I am seeing the same the errors all I need to do is a code review to get a better understanding at what we are doing. After I finished my code review I debugged a bit further. I could see that we first try to connect anonymously, then connect with credentials. In this particular case we are failing to connect to the domain controller when using credentials. Once we returned back from this function call I got the true error code which status 1726. Now I am getting somewhere!!!

0:029> dv
hRpc = 0x173c7b78
pConnect = 0x149b49e4
pServerGuid = 0x1647ee70
pszServer = 0x14a3ad10 "Domain Controller Name" <- The domain controller we are trying to bind too!!
scode = -2147221231
status = 1726
stat = struct __MIDL_nspi_0002
fRetry = 0

I ran this error through err.exe as well.

C:\WINXP\system32>err 1726
# for decimal 1726 / hex 0x6be
RPC_S_CALL_FAILED winerror.h

# The remote procedure call failed.
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x6be
# for decimal 1726 / hex 0x6be
RPC_S_CALL_FAILED winerror.h
# The remote procedure call failed.
# 2 matches found for "1726"

With this information I now have the true error for why we failed to bind to the domain controller, the RPC call failed. All I have to do now is figure out why the RPC call is failing?? The customer and I checked the health of the domain controller and it was just fine.

This is where it gets very interesting. I can't mention certain aspects of the case, but I can say that I noticed that the customer had two applications installed on both cluster nodes that used low level kernel drivers. One of these applications was also reporting errors in the application log. Once we removed the application we suspected to be the problem and restarted the cluster we saw that the problem went away. The customer and I monitored the server closely for about 1 week and the problem never came back. To prove this was the problem we reinstalled the application and within 30 or so minutes of the application performing an update we hit this problem again. What we found out later was that the way the application was applying it updates, it was causing some sort of corruption to the TCP/IP stack (as the application was also hooked in to it) and this was derailing us.

The customer stated that there was no need for this particular application to be installed on the server, and the customer uninstalled it. At this point we were able to generate the OAB every day and the problem was solved!!!

I hope you guys found this as interesting as I did. This is pretty much what I do day in and out as an Escalation Engieer.

Until next time.

Dave

Is your Outlook client downloading an ANSI or Unicode OAB?

dgoldman — Sat, 16 Sep 2006 21:40:00 GMT

A Microsoft Exchange server can contain 3 different version of the OAB (OAB Version 2, OAB Version 3 and OAB Version 4). As newer versions of the Offline Address Book are created and rolled out through an organization, it can become difficult for administrators to keep track of which version of the OAB users are downloading. This blog will hopefully help to clear this up a bit.

A Microsoft Outlook client can download the OAB Version 4 only if the following criteria have been met:

The Microsoft Exchange 2003 Server has had Service Pack 2 applied
The OAB Version 4 has been enabled
The Microsoft Outlook client has had Office Service Pack 2 installed

For Exchange 2007

OAB Version 4 is being generated.
The OAB Version 4 needs to be replicated to a Web Distribution Point
If public folders are enabled, the OAB Version 4 needs to be posted
The user is running Microsoft Outlook 2007
The user is running Microsoft Outlook 2003 with Office Service Pack 2

This download will occur when an OAB update is required, for example this can occur due to the 24 hr update frequency or if the user manually attempts an incremental or full OAB download.

The location in the registry that contains the Microsoft Outlook provider settings is: HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider.

The following registry values will reside under this registry key (These can change based on Outlook client version):

"MOABU"                                 = eb6bc4ba-2128-4380-96d1-4fd912a0ba43
"OAB DNU"                               = The Unicode DN
"OAB Last Modified Time"           = Last Modified Time
"OAB Last Template LocaleU"     = Last OAB Template Locale Version
"OAB Last Template UI LocaleU" = Last OAB Template Locale UI
"OAB NameU"                            = Global Address List
"OAB SeqU"                              = OAB Sequence Number
"OAB Sort LocaleU"                   = OAB Sort Locale
"OAB Use UTF8"                       = 0 or 1

Note on MOABU - If the registry key is MOAB, the Exchange server is using an ANSI Offline Address Book. If the registry key is MOABU the Exchange server is using a Unicode Offline Address Book.

Note on "OAB Sort LocaleU" - The OAB can only use the sort locale and character set of the server where it is generated. For example, if you have an Outlook client that is Spanish connecting to an Exchange server, you can add the Spanish locale to the Exchange server. However, the primary locale is going to be whatever was installed; let's say English. When the OAB is generated, it will use the English locale.

Note on “OAB Use UTF8” - This value will be set to 1 if the client was able to download the Unicode Offline Address Book and to 0 for ANSI Offline Address Book.

Note on "OAB SeqU" - This is the sequence number for the OAB files that have been downloaded. For example, (Example - 0a 00 00 00 = 10)

Registry Locations
The following location in the registry will contain the settings for the Microsoft Outlook profiles:
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles.

The following location will contain the profile specific settings: HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook\13dbb0c8aa05101a9bb000aa002fc45a

How to configure your Outlook clients for only OAB Version 4 downloads
Now if an administrator wants to ensure that the users in an organization are *only* able to download the OAB Version 4, they can implement this two ways:

1. Manually set the registry key on each client (not a very efficient way).
2. Push out a policy through a GPO (Group Policy Object) in the Active Directory.

You will want to navigate to the following location in the registry: HKEY_CURRENT_USER\Software\Policies\Microsoft\Exchange\Exchange Provider and add the following value:

Registry Name = "OAB v4 Only"
Registry Type = DWORD
Registry Data = 0 or 1

0 - Is the default setting if the registry key is not set, not present or set to something other than this on the client. If this is the case the Microsoft Outlook will perform the same fall back logic as it does today.

Example - If OAB Version 4 is unavailable for download the Outlook client will try to download OAB Version 3 and so on.

1 - Outlook will only use OAB v4, regardless of what other OAB versions are present on the server. If no OAB Version 4's are available for download or this registry value is set, the Microsoft Outlook client will not use an OAB and will use the Online Global Address List instead.

Note:

This registry key is not applicable to ANSI mode profiles, which are limited and can only use OAB Version 2.
If you have an ANSI profile you will need to create a new profile as ANSI profiles can not be automatically upgraded to Unicode profiles.

For more information on how to tell which version of the OAB a Microsoft Outlook client is downloading, please see: Troubleshooting Offline Address Book Downloads with Outlook Diagnostic Logging (Event ID 27) - http://blogs.msdn.com/dgoldman/archive/2006/06/19/637139.aspx

So just as a recap. If you are are interested in finding out what version of the OAB your client it downloading, you need to check for the following:

"OAB Use UTF8" registry key and make sure it is set to 1. This will indicate that you are downloading a Unicode profile. Now even know this can be OAB Version 3 or OAB Version 4 you can still verify this.
Turn up advanced logging from with-in Outlook and look for the Event ID 27 in the Windows Application Log.
Follow blog http://blogs.msdn.com/dgoldman/archive/2006/06/19/637139.aspx.
Add the "OAB v4 Only" registry key via Group Policy Object.

Dave

Troubleshooting the Outlook Send/Receive with error 0x80040119

dgoldman — Mon, 03 Jul 2006 22:48:00 GMT

Another problem I have seen in the newsgroups and floating around on the Internet is that people are getting 0x80040119 when they try to download their Offline Address List.

Ok, so lets start by troubleshooting this guy.

One good troubleshooting step is to examine the HRESULT code that was returned and see what it is. I used err.exe on the error code 0x80040119, and what is returned is the following:

C:\WINXP\system32>err 0x80040119
# for hex 0x80040119 / decimal -2147221223
MAPI_E_EXTENDED_ERROR mapicode.h
# 1 matches found for "0x80040119"

Well, that was not too helpful this time, all we got back was an extended error. Since this is a rare instance where err.exe was not helpful with the error code, I need to reply on my knowledge of the Outlook client.

Since I know the inner workings of the Outlook client when it comes to OAB, I know that the client needs to obtain the correct Entry ID to the OAB system folders before it can download the OAB files. This Entry ID is called the PR_ADDRBOOK_FOR_LOCAL_SITE_ENTRYID. Once the Outlook client has this Entry ID it can then call OpenEntry and try to access the OAB message and attachments. Now for what ever reason if the Outlook client can not access the OAB Folder it was trying to download from (let's say OAB Version 4), it will fall back will try the OAB Version 3 and then OAB Version 2.

For more information on what your Outlook client did the last time it downloaded the OAB, see this blog: Is your Outlook client downloading an ANSI or Unicode OAB?

I know that there are two ways for an Outlook client to find out what Offline Address List it needs to go after.

1. Use the msExchUseOAB attribute that is stamped on the user account object in the Active Directory. This tells the client which OAB to download.
2. Read this attribute off of the mailbox store that my mailbox resides on. You can very this from the properties of the Mailbox Store using the Exchange System Manager.

NOTE: In the hosting world we use the msExchUseOAB attribute to override the default store Offline Address List. The reasoning for this is that in hosting there are multiple companies and each user needs to download their own specific OAB. Now If you populate this attribute on a user account, this will override the mailbox store default settings. This can also cause problems if the attribute is populated but not pointing to a valid Offline Address Book.

Well, since we are not in a hosting environment, I do not have that attribute populated on my user account in the Active Directory. This data points me to one location, my mailbox store.

So the next step in troubleshooting is to look at the mailbox store to see if there is a Default Offline Address List associated with it. This can also be done two ways:

1. Open the ESM (Exchange System Manager) and navigate down to the server and mailbox store. From here you can retrieve the properties for the mailbox store. If there is no Default Offline Address List populated, then I can then fix it there by selecting one and populating this. This will stamp the msExchUseOAB attribute on the mailbox store.

2. From your Exchange server you can run OABInteg /s:GCName /t:storealtest /v:2 /l and look at the c:\OABInteg.txt file and check all the mailbox stores for this association.

Now I can see that my mailbox store has no Default Offline Address List associated with it I have found the problem. Now I just need to populate one to fix this. Before I can download the Offline Address list I need to log out of Outlook, and then log back in. This is a necessary step because the Outlook client reads this information during the login process. Once I am logged back in, I can then Send/Receive and download my OAB.

What if your information store has a Default Offline Address List associate with it?
If you happen to have a Default Offline Address List associated with your Mailbox Store, it is very possible that your Outlook client is reading some stale or incorrect information from its previous settings. Creating a new profile usually fixes this problem, however there are additional steps we can take to verify this before we do so.

1. From either your Exchange server or Outlook client run OABInteg /s:exchsrvrname /t:hierarchylist/v:2 /l (creates a c:\oabinteg.txt file). \

NOTE: This will get all of the Entry ID's from the system folders on your Exchange server.

2. FROM YOUR CLIENT run OABInteg /s:exchsrvrname /t:oabfldcheck /v:2 /l (creates a c:\oabinteg.txt file).

NOTE: This will simulate an Outlook client logon. You can get the search the OABInteg.txt output for the following MAPI property: PR_ADDRBOOK_FOR_LOCAL_SITE_ENTRYID. Once you have found it, copy it and then search for in the hierarchy list output. This will show you what server and folder in the replica list your Outlook client is trying to open. If the Entry ID returned to the client is not the correct one, your Outlook client will be going to the Ex: folder which is the default because we had no Entry ID to give back to the client.

So in short:

Make sure that the OAB did in fact generate, you can see this with diagnostic logging turned to max. See this blog: How to troubleshoot the OAB Generation process
Make sure your information store is stamped with a Default Offline Address List.

Last but not least you can *try* to create a new profile. ANSI profiles will not be upgraded to Unicode profiles, which are needed for downloading the OAB Version 3 and OAB Version 4. Exchange 5.5 uses OAB Version 2.

Dave

OABInteg - The dumpPDNtable test is now a MAPI test

dgoldman — Mon, 03 Jul 2006 21:46:00 GMT

I wanted to post an update for OABInteg. I have been really busy as of late writing the Beta 2 Offline Address Book internal training, so I have not had a chance to work on OABInteg as much as I would have liked too. One change that I needed to mention was the dumpPDNtable test.

To avoid confusion, I have changed this test from an LDAP test to a MAPI test. If you have the latest version of OABInteg you will notice that when you run this test you will be prompted for a profile. You still need to provide a server name; however this parameter will now be ignored for the test.

There are two main reasons behind this:

1. I wanted to be able to produce the most accurate results when querying the Global Address List. Now this test will query the Global Address List the same way that the OAB Generation process does.

2. When I finish this test OABInteg will download the attachments from the public folder information store, and compare the PDN table to the data that was obtained from the Global Address List objects. This will show the PDNs that have been added or removed, thus showing the problematic objects. This is dependent on using a MAPI profile for logging on to the information store, and cannot be done via LDAP.

I will post progress updates when I can.

Dave

New Active Directory objects are not showing after Cached Mode clients download the OAB

dgoldman — Mon, 03 Jul 2006 19:52:00 GMT

I have been spending a fair amount of time in the newsgroups lately, and I have seen a lot of posts around new objects not showing in the Offline Address List when clients are running in Outlook cached mode. This is a fairly easy process to troubleshoot if you know where to look and what to look for. In order for new objects to show up in an Offline Address List, a few things must happen:

1. The RUS (Recipient Update Service) needs to stamp the appropriate policies on the object. Once this happens the object will be added to the Global Address list.

2. Active Directory replication has to take place. This is very important because in medium to large sized environments there is a good chance that when you update the Offline Address List, the OAB Generation server will be reading the Global Address List objects from a domain controller that has not received the new objects that were created.

3. The Offline Address List needs to be rebuilt. By default the scheduled rebuild time for an Offline Address Book is 5:00am every morning, unless changed by an administrator.

Now, here are some things that you need to remember about Microsoft Outlook clients:

1. Microsoft Outlook 2003 and 2007 clients are configured for cached mode by default.
2. An Outlook client will download the Offline Address Book assigned to the user when it synchronizes with the user’s mailbox for the first time. Once Outlook completes a full OAB download, it checks for updates every 24 hrs from the last successful update.

With that being said, there are several configurations that can be made to the cached mode OAB download.

How to Control Offline Address Book Download Behavior in Outlook in Cached Exchange Mode
You can use this procedure to create a new registry key to control offline address book download behavior in Outlook in Cached Exchange Mode.

Warning - Incorrectly editing the registry can cause serious problems that may require you to reinstall your operating system. Problems resulting from editing the registry incorrectly may not be able to be resolved. Before editing the registry, back up any valuable data.

1. On the Outlook client computer, open a registry editor, such as Regedit.exe or Regedt32.exe.
2. Navigate to: HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\11.0\Outlook\Cached Mode
3. In the left pane, right-click the Cached Mode subkey, point to New, and then click DWORD Value.
4. Type DownloadOAB, and then press ENTER.
5. Double-click DownloadOAB.
6. In the Edit DWORD Value dialog box, enter one of the following in the Value box:

0 – The offline address book does not download automatically.
1 – The offline address book uses the Download Full Items download mode. This is the default setting.
2 – Download the offline address book in any download mode, but download a differential update in the Header only download mode.
3 – Always download the offline address book and a differential update in one of the following download modes: Download Headers and then Full Items, Download Full Items, and Download Headers.

7. Click OK.
8. Close the registry editor, and restart the Outlook client for the change to take effect.

How to Download the Full Details Option for the Offline Address Book
You can use this procedure to download the Full Details option for the offline address book. Providing that replication has taken affect successfully and the Offline Address List in question has been rebuilt, you should see your new changes in your cache mode copy of the Offline Address List

To download the Full Details option for the offline address book

1. Start Outlook 2003 or 2007.
2. On the Tools menu, click ‘Send/Receive’.
3. Click ‘Download Address Book’.
4. Click ‘OK’.

If you want to do a full OAB Download, you can do the following:

5. Make sure you un-checked ‘Download changes since last Send/Receive.
6. Click ‘OK’.

How to check if the new changes have been replicated to all domain controllers
Once you have verified what domain controller you are connected to with your Active Directory Users and Computers MMC, you want to load up LDP. LDP is used to view objects in the Active Directory. You will need to expand the domain naming context, and navigate down to the container that your object resides.

You can look at properties like showInAddressBook and proxyAddresses to verify that the RUS has stamped the object successfully. If everything is ok here, you want to proceed to connect to the rest of your domain controllers and verify the same information. If you see inconsistencies between the objects, then you know that all of your changes have not replicated around yet. If this is the case, then you will need to use the Active Directory Sites and Services MMC to force replication.

The last key thing to check is to make sure that the domain controller that the OAB Generation process is going to be using has this updated information. To verify this you will need to turn up the diagnostic logging on the OAB Generation server and look for the 9117 event id. This event id will indicate which domain controller you are using.

Event ID : 9117
Category : OAL Generator
Source : MSExchangeSA
Type : Informational
Generated : 1/1/2007
Machine : OAB_Server1
Message : OALGen successfully opened a connection to Active Directory 'OAB_Server1' which will supply the current Address Lists.
- Default Offline Address List
For more information, click http://www.microsoft.com/contentredirect.asp.

NOTE: Before Exchange 2003 Service Pack 2 you will not see which domain controller name listed in the event id. If you do not have the domain controller name in the event id, you will need to take a Netmon trace. Taking a Netmon trace will show you where the NSPI calls are going, then you know that is the domain controller you are using.

How to verify that the changes have been generated by the OAB Generation Server
Before you see the changes you will need to rebuild your Offline Address List. Once you have done this there are several ways you can check for the new change files.

For OAB Version 2 and OAB Version 3a there will be a changes.oab file.
For OAB Version 4 there will be a binpatch.oab file.

1. You can use OWA to connect to the OAB folders and look for the latest changes.oab and or binpatch.oab file that was created.

2. You can run OABInteg /s:exchsrvrname /t:oabfldcheck /v:2 /l and then look at the c:\OABInteg.txt file.

By running this OABInteg command, you are having the tool connect to the public folder information store. Once it connects to the store it will dump out all of the information about the OAB messages and attachments that are posted for Outlook clients to download. You can look at the time stamps in the output file to verify that the changes.oab file has been created.

Once you have done all of the steps listed above, you can download the OAB files and you should see the new changes in your local Offline Address List.

For more information on troubleshooting the OAB Generation process see this blog:
http://blogs.msdn.com/dgoldman/archive/2005/07/16/439665.aspx

For more information on where the Outlook client files reside, see this blog:
http://blogs.msdn.com/dgoldman/archive/2005/04/28/413043.aspx

OAB Generation and the 9339 event id

dgoldman — Wed, 14 Jun 2006 17:55:00 GMT

The 9338 and 9339 event ID’s that are logged in the application log indicate that there was a problem with the data that is returned from the Active Directory during an Offline Address List generation. The reason why there are two possible events that can be logged, are due to the fact that we might / might not find a Display Name for the object that we failed on.

Knowing that OABGen.dll is a MAPI application, we know that OABGen.dll will need to make calls to the Active Directory via the NSPI interface for any data needed. Once the call has made its way to the Active Directory, it is the responsibility of the Active Directory to serve this data request back to OABGen.dll.

For those of you that are not very familiar with MAPI, here is a quick overview on how MAPI applications store/manipulate data. Before OABGen.dll can query the Active Directory for its data, OABGen.dll will have to create a MAPI table that is comprised of columns. Each column will be represented by a MAPI property (Example – PR_DISPLAY_NAME). Each row consists of a subset of properties. Now one thing to keep in mind is that a MAPI table is an abstract data type that represents a set of objects that all have common characteristics. Each object returned from the Active Directory will occupy a row in the table, and each column will contain the property data found. I do not want to go very deep in to MAPI because this is outside the scope of this blog, but I want you to have an understanding on how this works.

Below is a visual representation of a 1 row and 3 columns that would represent this object in the Active Directory:

Dave Goldman

PR_DISPLAY_NAME

PR_EMAIL_ADDRESS

PR_EMS_AB_MANAGER_A

Now that we have covered the basics of how MAPI sees data, we can move on to how the OAB Generation process obtains this data.

During the generation an Offline Address List, OABGen.dll will make MAPI function calls to QueryRows() to retrieve all the objects from the /Global Address List. For more information on the MAPI QueryRows() function call, please refer to this MSDN article. Each time OABGen.dll calls QueryRows(), it will pass in a set number of rows to be retrieved per call. OABGen.dll receives these query results in the form of pages.

Below is example of a QueryRows() call to Active Directory. Here we are requesting 5 rows per function call.

User 0	PR_DISPLAY_NAME	PR_EMAIL_ADDRESS	PR_EMS_AB_MANAGER_A
User 1	PR_DISPLAY_NAME	PR_EMAIL_ADDRESS	PR_EMS_AB_MANAGER_A
User 2	PR_DISPLAY_NAME	PR_EMAIL_ADDRESS	PR_EMS_AB_MANAGER_A
User 3	PR_DISPLAY_NAME	PR_EMAIL_ADDRESS	PR_EMS_AB_MANAGER_A
User 4	PR_DISPLAY_NAME	PR_EMAIL_ADDRESS	BAD PROPERTY!!

Now here is where the 9339 comes in to play. If one of the entries in the page has a bad property, the entire page retrieval will fail. In the event of a failure OABGen.dll will log the last entry it finds in the *last* page of results that were successfully retrieved. From the example above you can see that User 4 has a bad property in the Active Directory, so this page retrieval will fail.

Below is a list of events that you will see in the Windows Event Viewer Application Log.

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9339
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
Active Directory (domain controller) returned error 8004010e while generating the offline address list for '\Global Address List'. The last recipient returned by the Active Directory was 'Dave Goldman'. This offline address list will not be generated.
- OAB Address Book

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9330
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
OALGen encountered error 8004010e(internal ID 5000f5f) accessing Active Directory (domain controller) for '\Global Address List'.
- Address Book

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9126
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
OALGen encountered error 8004010e while calculating the offline address list for address list '\Global Address List'. This offline address list will not be available for client download.

Now that I have the events from the application log, I want to look closer to see what the error code being returned means. If you look at the three above listed event id’s you will see that there is a common error return ‘8004010e. By using The Microsoft Exchange Server Error Code Look-up I can see what the error code means.

C:\WINXP\system32>err 8004010e
# for hex 8004010e/ decimal -2147221234
ecInsufficientResrc ec.h
MAPI_E_NOT_ENOUGH_RESOURCES mapicode.h
# 2 matches found for "8004010e"

The error code MAPI_E_NOT_ENOUGH_RESOURCES is a default error code the NSPI interface returns whenever it catches an error while retrieving data from the Active Directory.

Invalid Active Directory Attributes
Mangled, null and incorrect attribute values can cause problems for the OAB Generation process, and stop an Offline Address List from being generated. By default directory attributes that are not populated will have a <Not Set> for its value. If you look at a property and you see that it is blank or null, this indicates that something is wrong with the property. In addition to the null or blank, values can also contain incorrect values.

As of right now there are only 4 known attributes that will cause a problem for the OAB Generation process if they have bad, null or incorrect values. This doesn’t mean that there couldn’t be more, but through most of the cases that I have reviewed and triaged, these are the 4 that always come up:

1 & 2. The HomeMDB and HomeMTA attributes - Which signify which Exchange server holds the mailbox. The homeMDB contains the complete DN of the mailbox store for the mailbox and the homeMTA contains the complete DN of the Message Transfer Agent (MTA) on that server.

3. The Manager attribute - Contains the distinguished name of the user who is the user's manager. The manager's user object contains a directReports property that contains references to all user objects that have their manager properties set to this distinguished name.

4. The Secretary attribute - Contains the distinguished name of the secretary for an account.

These values must point to a valid and correct object. For example, if a mailbox was moved from server one to server two and there was an active directory replication latency or problem, we might end up with an incorrect value for the homeMDB and homeMTA.

Another example is where the manager of a group of users leaves the company and their account is removed but the changes never replicate around.

How to fix this

This is where things start to get tricky. Right now without calling in a support call to PSS, you will have to try to do this manually. I am currently working on a solution for OABInteg that will allow you to scan for invalid entries and I hope to have this completed soon.

Looking above you can see that there are 4 attributes that have been the top offenders. You can run LDIFDE export commands to scan through your active directory and look for objects that have invalid entries. Below I am listing a KB Article that talks about using the LDIFDE utility. KB Article ID: 237677 Using LDIFDE to Import and Export Directory Objects to Active Directory - http://support.microsoft.com/?id=237677

Keep in mind that when you are looking at these attributes mentioned above, you need to make sure that they do not contain and null, invalid or mangled values. If they do contain values and are not supposed too, make sure they are set to <Not Set>. If they are supposed to contain values make sure that they point to valid objects in the active directory and are not set to null, or have a blank space.

One thing that will help you with your search is to look at the 9339 event and look at the last recipient returned. From here you can use the MAPI Editor (known as MFCMapi) to expose the Global Address List. You can download MFCMapi here: http://www.microsoft.com/downloads/details.aspx?familyid=55FDFFD7-1878-4637-9808-1E21ABB3AE37&displaylang=en

Steps for exposing the Global Address List

1. Run MFCMapi.

2. Click 'OK' on the main screen.

3. Select from the menu 'Session'.

4. Select 'Logon and Display Store Table'.

5. Select the profile that you want to use.

6. Select from the menu 'Address Book'.

7. Select 'View AB Hierarchy'.

8. When the dialog box appears, expand the Root Container.

9. Double click the 'Global Address List'

Now that you have the Global Address List exposed, you can search for the last recipient displayed in the 9339 event and start your search down from there. The next 50 results are the ones that you will want to look at, as this will be the next batch of results returned.

I will post more once I have added this functionality to OABInteg.

Dave

The DS Server registry key and rebuilding an Offline Address List

dgoldman — Sat, 06 May 2006 22:06:00 GMT

I wanted to talk briefly about using the DS Server registry value for hard coding Exchange servers to particular domain controllers, and the problems that it can cause.

The DS Server registry key was used back in the day as a means to hard code an Outlook client so it could use a particular domain controller. This was used to help redirect Outlook clients in the event of problem with certain domain controllers in an environment. Outlook clients and Microsoft Exchange servers specifically the System Attendant service (mad.exe) both use MAPI profiles for logging on to an Microsoft Exchange information store.

Below I will outline a high level overview on how MAPI clients use the DS Server registry value.

When a profile is created by an Exchange Server or Outlook client for the first time, we will write the current setting values (the global catalog server being use is one of them) to the following registry key:

Registry Location: HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\<profile_name>\dca740c8c042101ab4b908002b2fe182
Registry Value name: 001e6602
Registry Value type: String

We will also write certain OAB data values to the following registry key: HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider after the client has logged in for the first time and preformed a full OAB download.

Now anytime we logon with this profile we will follow this logic for logging on:

1. Check for the preceding registry key HKEY_CURRENT_USER\Software\Microsoft\Exchange\Exchange Provider\ and look for the "DS Server" value.

2. If this value is populated with a domain controllers name, the MAPI client will attempt to bind to the directory service interface for that specified domain controller.

If the global catalog server is available, Outlook will continue to use that global catalog server and only this domain controller.

If the global catalog server is not available the Outlook client will fail to bind to the directory service on the domain controller.

NOTE: If the DS Server registry key is not being used and the prior used domain controller is not available, the Outlook Client will ask the Microsoft Exchange Server for a new global catalog server referral. Once the client has been given back this referral the Outlook client will then attempt to communicate with that new domain controller.

Now using the DS Server registry key on a Microsoft Exchange server is bad!! If the DS Server registry key value is present but the global catalog server is not available, the System Attendant will throw a provider error and fail to log on. This will halt all communications from the System Attendant to the active directory, and can cause your Exchange server to stay is a decommissioned state until this value has been removed.

This Event ID can be seen in the application log as follows:

Event ID           : 9175
Event Type       : Error
Event Source    : MSExchangeSA
Event Category : MAPI Session
Description       : The MAPI call 'OpenMsgStore' failed with the following error: The information store could not be opened. The logon to the Microsoft Exchange Server computer failed. MAPI 1.0 ID no: 80040111-0286-00000000

If you were trying to generate an Offline Address List from the ESM (Exchange System Manager) there will be a dialog pop-up box with an error:

MAPI or an unspecified service provider.
ID no: 00000000-0000-00000000
ID no: c1050000 Exchange System Manager

There will also be additional Event ID's that will be present in the application log if you are attempting to generate an Offline Address List:

Event ID             : 9108
Event Type         : Information
Event Source      : MSExchangeSA
Event Category   : (13)
Description         : Offline Address List is being generated for \Global Address List.

Event ID             : 9126
Event Type         : Error
Event Source      : MSExchangeSA
Event Category   : (13)
Description         : OALGen encountered error [0x80004005] while calculating this OAL. This OAL will not be available for client download. (\Global Address List)

Event ID             : 9175
Event Type         : Error
Event Source      : MSExchangeSA
Event Category   : MAPI Session
Description        : The MAPI call 'OpenMsgStore' failed with the following error: The information store could not be opened. The logon to the Microsoft Exchange Server computer failed. MAPI 1.0 ID no: 80040111-0286-00000000

Using the err.exe (Microsoft Exchange Server Error Code Look-up Tool) you can look up both error codes:

C:\WINXP\system32>err 0x80004005
# for hex 0x80004005 / decimal -2147467259
ecError ec.h
MAPI_E_CALL_FAILED mapicode.h

C:\WINXP\system32>err 80040111
# for hex 0x80040111 / decimal -2147221231
ecLoginFailure ec.h
MAPI_E_LOGON_FAILED mapicode.h

The fix for this is to remove the DS Server registry key.
Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use the Registry Editor at your own risk.

Steps for removing the DS Server Registry key:
1. Start Registry Editor (Regedt32.exe).
2. Locate the DS Server value under the following key in the registry: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Exchange Provider
3. Click the ‘DS Server’ key
4. Click ‘Edit’
5. Click ‘Delete’
6. Click ‘Yes’ to confirm the deletion.
7. Quit Registry Editor.

Once you have confirmed that the DS Server registry value has been remove you can restart the System Attendant service and it should start up.

For more information on how the System Attendant uses a MAPI profile to build an Offline Address List see this blog: How does OABGen know what domain controller to connect to?

Dave

OAB full downloads and server side paged pool depletion

dgoldman — Sat, 11 Feb 2006 01:58:00 GMT

Recently I participated in another fun critsit where a customer had massive issues caused by full OAB downloads. The customer had several hundred thousand cached mode Outlook clients trying to download the full OAB all at the same time. This caused a very high network bandwidth consumption issue, and also contributed to server side paged pool depeletion problems.

After troublshooting this further with the customer we saw that each cached mode Outlook client had several open connections to multiple public folder stores. This was a huge impact for the exchange servers. The reason is that each connection from a cached mode Outlook client was increasing the user’s token size. Each client connection to a Windows server requires some paged pool kernel memory. Eventually this lead to paged pool depletion problems on some of the servers. Once the servers end up in this state, it is all down hill from there.

In this blog I list out some of the top offenders that can cause full OAB downloads. I have also linked a MUST read blog in regard to Tokens and Memory Usage.

PLEASE READ THIS BLOG -> Large Security Tokens and Kernel Memory Exhaustion

Here are some of the top offenders that I was talking about:

1. PDN Changes
2. The 1/8 Rule
3. Invalid proxy address’s that are over 64 characters – (fixed with a hot fix) (Signature Mismatch)

PDN Changes
1. If the PDN table changes in the OAB by either a new PDN or removal of a PDN then all Outlook cached mode clients using V2 of V3 OAB will also attempt a full download. The PDN table is the set of all PDNs (Parent Distinguished Names) found in the directory.

2. Manually modifying a legacyExchangeDN in the AD to create a PDN that did not exist before. This most often is done by accident if someone is editing this value and mis-types the value, thus creating a new PDN.

3. With Exchange 5.5 and ADC in place, creating a new container in 5.5 and inserting an object into it, or deleting the last object in a 5.5 container.

4. With Exchange 5.5 and ADC in place, ADC set to replicate the container hierarchy to 5.5, and the administrator creates new mail-enabled objects in a new AD container. The ADC will create the new container in 5.5 and back-replicate the new 5.5 distinguished name as the legacyExchangeDN of the AD object creating a new PDN.

5. Add an administrative group. The first mailbox created on a server in this AG will cause a new PDN to show up in the directory.

6. Deleting the last object with a particular PDN in its legacyExchangeDN or proxy Addresses. Example: A few years after consolidating and deleting a site, the last mailbox formerly in that site is finally deleted. The x500 placeholder is gone and reduces the size of the PDN table.

7. Adding/removing/modifying an X500 proxy address with new PDN. This can be done using ADU&C.

If the X500 address is in the local org, but the organizational units and containers are new or mistyped, a PDN will be added or deleted from the table.

The reason why this is such a problem is because the pdndex.oab file can not be re-indexed after it has been created on the client side. This can only happen on a full download. To stop Outlook clients from doing full downloads is to make sure that both the client and server must be running SP2. If the client is on SP2 it should automatically update to the version 4 providing the client is already using a Unicode profile and that they actually connected to the server where the v4 is. In SP2 we apply the diff files differently and do not rely on the PDN table.

8. Any applications that have the ability to write data to the active directory. Example - Importing proxy addresses or legacyExchangeDN's to the active directory and the import filters are set in correctly.

For more information on PDN changes please refer to this blog: OAB PDN Changes and Exchange Site Consolidations

The 1/8th Rule
When Outlook 2003 has not downloaded a .diff file for several days, and the sum of the .diff files is greater than 1/8th the size of the whole offline Address Book, Outlook 2003 downloads the whole offline Address Book. If there are many Active Directory objects that have a change to an attribute that is in the offline Address Book, that change may cause the .diff file to be greater than 1/8th the size of the whole offline Address Book. Therefore, all Outlook 2003 clients try to download the offline Address Book.

For example, the following changes might cause the .diff file to be greater than 1/8th the size of the whole offline Address Book:

1. You update the phone numbers of a large set of users with a new area code.
2. You add each user's department information to the directory.
3. You add a new address type or a new address book.

Outlook clients fail to apply the OAB diff files due to signature mismatches.
This problem occurs during the OAB Generation process when OABGen.dll reads in objects from Active Directory that contains a proxy address that exceed 64 characters in size. Objects that have proxy addresses that are longer than 64 characters are legitimate objects as far as the OAB Generation process is concerned, however the problem is when the Outlook client downloads the diff file it will encounter a signature mismatch. Once the Outlook client fails to apply the OAB difference file, the Outlook client will be forced to download the full OAB message to correct the issue.

Note: The 64-character limit includes the at sign (@).
Note: By default, the Offline Address Book logging feature is enabled in Outlook 2003.

These events are similar to the following one you will see with Outlook logged enabled:

Event Source : Outlook
Event ID : 27
Event Type : Error
Description : OAB ModDif failed. (Details record in event data).

Event Source : Outlook
Event ID : 27
Event Type : Information
Description : Starting OAB Download.

Event Source : Outlook
Event ID : 27
Event Type : Information
Description : OAB Download Succeeded.

Microsoft has confirmed that this is a problem and there is a hot fix available for this:
Microsoft KB Article for this: 909565 - When you run Outlook 2003 in Cached Exchange Mode, all the Outlook clients in the Exchange Server 2003 organization try to download the offline address book at the same time - http://support.microsoft.com/default.aspx?scid=kb;EN-US;909565

Large Security Tokens and Kernel Memory Exhaustion caused by Outlook cached mode client connections.
There are certain situations where and Exchange 2003 Server can experience a Windows kernel paged pool depletion problem caused by token sizes and the amount of Outlook Cached mode client connections.

Symptoms Experienced:
1. Users experience slow or unresponsive performance.
2. Server Bluescreening
3. You may have to restart the Exchange server to resume typical performance levels.
4. Paged pool allocation failure events that are similar to the following are logged in the System log:

Event ID: 2020
Source: SRV
Type: Error
Description: The server was unable to allocate from the system paged pool because the pool was empty.

CAUSE: This problem occurs when paged pool memory is exhausted because of too much memory being allocated to client access tokens.

Microsoft has confirmed that this is a problem and there is a hot fix available for this:
Microsoft KB Article - 912480 - An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory - http://support.microsoft.com/default.aspx?scid=kb;EN-US;912480

PLEASE READ THIS BLOG --> Large Security Tokens and Kernel Memory Exhaustion

How can we try to control the network consumption by using OAB Throttling?
The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.

Example 1: If a public store that supports 10,000 users receives 3,000 requests in one hour and the offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data. This traffic could potentially overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This traffic could potentially overload a gigabit-per-second LAN for longer than one hour.

How do we calculate the largest OAB message size?
1.To find out the biggest message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your oab files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state.

Here is an example of the Oabfldcheck summary:

Scan Completed

+------------------+

Message Class Normal found: 2

Message Class Differential found: 175

Message Class Unknown found: 0

Message Attachments found: 503

Messages found but unable to read the properties: 0

System folders found: 3

Highest sequence number found: 231

Lowest sequence number found: 16

Biggest attachment found: 16700 Bytes

Smallest attachment found: 102 Bytes

Biggest message found: 60234440 Bytes

Smallest message found: 106 Bytes

You can then take the size of the largest message (here it is 60MB) x total users downloading the OAB x request per hour = number of data to be delivered by the server.

To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool

How to enable OAB Throttling
Use one of the following methods to determine the OAB Bandwidth Threshold data value to use:

Method 1 – Less Accurate
1. Set the value preemptively. *Based on your knowledge of the network topology* and how full offline Address Book downloads from multiple servers can add up to overload the LAN, set the thresholds on the individual servers to the largest values that are consistent with not overloading the LAN.

Bandwitdh Example: Type 5000 to configure the server to use 5000 kilobytes per second (KBps) as the bandwidth threshold for offline Address Book download throttling. 5000 KBps is approximately 40,960 kilobits per second (Kbps), or 40.96 megabits per second (Mbps).

Method 2 – Most Accurate
1. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that overloads the LAN. This can be a short period of time.
2. Make note of the bandwidth used by offline Address Book downloads on each server.
3. Set the threshold values to a percentage of those values. For example, set the threshold values to 60% of those values.

To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.

Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.

***Use Registry Editor at your own risk***
1. Click Start, click Run, type regedit in the Open box, and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\
3. On the Edit menu, point to New, and then click DWORD Value.
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify.
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.

Things you need to Understand about OAB Throttling

1. To return to the default behavior, delete the registry value.

2. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.

3. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.

4. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:

If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.
If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.

5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.

Dave

OAB messages and message size restrictions

dgoldman — Tue, 07 Feb 2006 02:52:00 GMT

I had a chance to work a pretty interesting OAB issue with a co-worker the other day. We had a situation where Outlook clients from Site B were downloading their OAB just fine before midnight, and as soon as 7 am rolled around everything for the users in Site B came to a screeching halt. One thing that we did know was that this OAB was a replica from Site A.

Now this is common for large organizations to generate an Offline Address Lists, and then replicate them to other Exchange sites. But if things are not configured properly this can lead to problems.

Being that this was a replica, that kind of pointed us in the direction that we needed to go in to solve this issue.

Throughout the troubleshooting process we ran various tests and checked everything from the OAB folders; the posts in the store, etc and everything looked just fine. The OAB even generated fine in Site A.

So, we were able to fix this by doing the following:

1. Ran OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store so we could get the OAB message and attachment sizes. Here is the summary at the end of the oabfldcheck test.

Scan Completed

+------------+

Message Class Normal found: 2

Message Class Differential found: 18

Message Class Unknown found: 0

Message Attachments found: 174

Messages found but unable to read the properties: 0

System folders found: 3

Highest sequence number found: 23

Lowest sequence number found: 15

Biggest attachment found: 6700 Bytes

Smallest attachment found: 106 Bytes

Biggest message found: 11358214 Bytes <--- This is what we want!!

Smallest message found: 305 Bytes

2. Looked for Biggest message size and we found this in the Scan Completed section of the test .(end of the test).

3. Searched up for 11358214 in the c:\OABInteg.log file. 11358214 is the Biggest Message size. Below is what we find.

Message Conversation Topic: \Global Address List - details_u.oab - 23

Message OAB Name: \Global Address List

Message OAB DN: /

OAB Normal Message class

Message Code Page: 0x4e4

Message Size: 11358214 Bytes

Message Local Id: 0x409

Message OAB Sequence: 23

OAB Container GUID: ef06e64f-8ced-490e-aad0-f8030fa109b4

4. Being that this was a routing issue the next thing we checked was the message size restrictions on the SMTP connectors, and we saw that they were set to restrict messages that were over 10 MB in size. So this message was not going to replicate once it hit 10 MB, and you can see that it is a little over 11 MB.

Note on Message Restrictions - The routing engine determines the message size, priority, and message type (that is, system or non-system message). The routing engine compares these properties with available connector restriction information. It then excludes those connectors that cannot transfer the message due to effective connector restrictions from the list of potential connectors.

5. The Fix was to reconfigure the message restriction size limits to restrict message of 15 MB or greater. Once this was done we forced replication and the messages were able to replicate from Site A to Site B. We had the Outlook clients log out and then back in to Outlook and they were again able to download their OAB.

With this being said, when your organization is growing make sure that you configure your message restriction sizes limits so that they do not block your OAB messages from replicating around.

Dave

How does OABGen know what domain controller to connect to?

dgoldman — Fri, 27 Jan 2006 00:38:00 GMT

In this article I will explain how the OAB Generation process discovers the domain controller that it will use for rebuilding an Offline Address List. Understanding this process can help you in troubleshooting.

As I have mentioned in other blog posts of mine, the System Attendant (MAD.exe) is the process that is responsible for invoking OABGen.dll (the OAB generation process). This happens one of two ways:

1. Via Schedule

To view the schedule for an Offline Address list you can do the following:

1. Open the ESM (Exchange System Manager)

2. Expand the Recipients Container.

3. Select the Offline Address List Container.

4. In the right hand pane highlight an Offline Address List.

5. Select Properties.

At the bottom of the Properties dialog box you will see an area that shows the Update Interval. This drop-down list box will control when the Offline Address List is generated. The list box will also show several predefined times:

· Run daily as 2:00 am

· Run daily as 3:00 am

· Run daily as 4:00 am

· Run daily as 5:00 am

· Never Run

· Use custom schedule

2. Via the ESM (Exchange System Manager)

1. Open the ESM.

2. Expand the Recipients Container.

3. Select the Offline Address List Container.

4. In the right hand pane right client an Offline Address List.

5. Select "Rebuild".

By selecting "Rebuild" you will force replication to occur. In doing so the System Attendant then becomes responsible for making sure that the selected Offline Address List is rebuilt.

But, the real question here is "how does the System Attendant find the domain controller??"

Well, in order for the OAB Generation process to connect to the Active Directory to make its NSPI queries, it must have a mechanism for doing so. This connection to the Active Directory will be facilitated by a MAPI profile.

What is a MAPI and what is a MAPI Profile?

MAPI stands for Messaging Application Programming Interface.

A MAPI Profile is a named list of message services and configuration data created by a user or service. A profile can contain any number of message services. Profiles will contain a session object and use service providers to do the work on the behalf of the user or system.

A MAPI Session is a period of fixed duration (logon time ---> logoff time) which the work is done. In the context of MAPI, work refers to a series of client requests to MAPI that are services by one or more service providers.

Once a scheduled rebuild or manual rebuild of an offline address list is performed, there are a few things that need to happen before OABGen.dll can even connect to the Active Directory:

1. The system attendant needs to force the Directory Service call to synchronize the Offline Address List.

2. After the Directory Service calls are made necessary security checks will be done to ensure that we have the proper credentials to make this call.

3. After the necessary security checks have been preformed we will make a call to the Active Directory to get the Configuration Domain Controllers DNS name. This is needed so we can obtain information like (Forest Name, Organization, Administrative Groups, Server Name, etc).

4. Once the Configuration Domain Controller's name is obtained a Session Object can be set up.

5. Now try to create one of the Directory Service objects, and read all of the information we need from the active directory (this is the data mentioned above).

6. Make a query to get the name of the offlineABServer.

7. Once the offlineABServer is found, get and use legacyExchangeDN to query that object in the active directory. (Example - "/dc=local/dc=OABGen/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=OABGen/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=W2K3-E2K3".

8. Next setup our tasks that are necessary for the OAB Generation (profile creation, obtain the public folder information store, etc).

9. Get the server name and add it to the MAPI Session that was previously created.

10. Logon to the Admin portion of the profile (System Attendant profile).

11. Obtain the system directory so we can find out where the MAPISVC.inf file resides.

12. Read in the service specific settings from the MAPISVC.inf file so the profile can be configured.

13. Check to see if we are a domain controller or not.

14. Get the mailbox name for this profile - "/o=OABGen/ou=First Administrative Group/cn=Configuration/cn=Servers/cn=W2K3-E2K3/cn=Microsoft System Attendant"

15. Now log profile on to our session object using the MAPILogonEx API. For more information on the MAPILogonEx API please refer to: http://msdn.microsoft.com/library/en-us/mapi/html/9764438a-0e36-4124-af1e-8296759ed216.asp

16. Open the Address Book provider by creating a new IAB interface object.

17. Associate this Address Book with this MAPI session.

18. Load all of our necessary providers here for the profile.

19. Open the profile sections and also initialize emsabp.dll.

20. Load the Address Book data.

21. Look up the AB Server that will be responsible for holding the Offline Address List.

Here is where the magic is:

22. Check to see if the Address Book server can be contacted and is online. If we find that we are online then we need to Logon to the Directory Service.

23. Check to see if this server is an Exchange 2000 server.

24. Check the registry to see if we are hard coding the profile by using the following registry key [HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\Exchange Provider]. Please see this Knowledge Base Article for more information:

ID Q314737 - XADM: Offline Address List Generation Does Not Work on Exchange 2000 Server SP1

http://support.microsoft.com/default.aspx?scid=kb;EN-US;314737

NOTE: Using this registry key is not a recommended or supported method, and should not be used unless directed by a Microsoft PSS Support Professional for troubleshooting purposes only!!

25. Open Address Book profile section in the MAPISVC.inf file.

26. Read from the MAPISVC.inf file and see if we can find the following MAPI property tag 0x6602001e = PR_PROFILE_HOME_SERVER. If this value is present in the DS Section of the MAPISVC.inf file then this is the server that we are going to use for our Active Directory connections via our MAPI profile.

27. If this value is not present we are an Exchange 2000 server, we will then use the RFR Interface (Referral Service). Using the Referral Service will allow us ask the Exchange Server to connect to the Active Directory and find us another suitable domain controller that we can connect to.

28. If the server is not an Exchange 2000 server or our Referral failed to get the profile home server address we need to rebind back to the Active Directory again.

29. Once we find our data save it to the profile.

30. Now that we have the necessary profile information to connect to the Active Directory we pass this data along to OABGen.dll.

31. When OABGen.dll is invoked to rebuild an address list, it will have use the domain controller that is will use for its Active Directory queries.

For more information on the OAB Generation Process please see my post "Overview of the OAB Generation Process" here:

http://blogs.msdn.com/dgoldman/archive/2005/03/31/404329.aspx

Now under certain circumstances there are situations where these MAPI profiles do not get cleaned up by the sytem. This can lead to other problems which I do not go over in this blog.

For more information on how profiles are deleted, please see Stephen Griffin's blog on: When is a Deleted Profile Not Deleted?

MSFT Support Article - Description of the Offline Address Book Integrity (OABInteg) tool

dgoldman — Sat, 31 Dec 2005 01:13:00 GMT

Adding the link for the Microsoft Knowledge Support Article for easy finding :)

http://support.microsoft.com/?kbid=907792

Exchange Insider Articles on TechCenter

dgoldman — Wed, 07 Dec 2005 01:19:00 GMT

There are a number of the "Exchange Insider" articles starting to trickle out onto the TechCenter web site: http://www.microsoft.com/technet/prodtechnol/exchange/2003/insider/default.mspx. In case you are not familiar with these articles, they come from a weekly "tech bulletin" that is sent out. Over the years these weekly emails have hosted various guest writers on many interesting Exchange (and Exchange Related) topics.

I have also written a few in regard to OAB (Offline Address List Generation).

Offline Address List Generation Overview - http://www.microsoft.com/technet/prodtechnol/exchange/2003/insider/offlineadd.mspx

OAB Version 4 in Exchange Server 2003 Service Pack 2 - http://www.microsoft.com/technet/prodtechnol/exchange/2003/insider/oab4.mspx

I hope that you'll find them as useful as we do internally!!