Stemy's Blog

Signs, Sonograms, and Breakfast

On a more personal note, Alissa and I went to get a sonogram last week.  Camcorder in hand, I was very disappointed to find that they would no longer allow you to video tape or take pictures.  They would provide us a handful of photos throughout but no video at all.  Apparently there were some "legal issues" which prevented them from allowing any recording devices.  Translated -- Someone sued them because they were inaccurate about the due date, sex, or something else.  Another example of a one or two people messing things up for the rest of us.  Here's your sign... 

Not to let such a small thing ruin a great experience, we did get several good photos of our baby girl...

Measurements showed right on 20 weeks, so we'll keep the 13 Jan due date.

We celebrated with an early morning trip to IHOP for a chili-covered omelette with hash browns and some harvest grain and nut pancakes (with strawberry syrup).  We shared.  I don't see how any one person could eat that much food.

How to avoid the 75GB path to the unemployment line

Exchange Server 2003 Service Pack 2 is Coming!
All hail the latest service pack!

And there are some great changes being incorporated into E2k3 SP2...

• Improved compression in Mobile email
• Remote wipes of mobile devices (I love this one!)
• Cert-based authentication for mobile devices
• S/MIME for mobile devices
• Improved spam protection (updated IMF)
• Support for Sender ID
• New OAB format for increased performance
• Outlook cached mode enforcement
• Exchange 2003 STANDARD edition increases limit from 16gb -> 75 gb

And at that point, my jaw dropped....

Increasing the ceiling past 16 gb can be great for Small/medium businesses
I can't even begin to tell you how many calls I have taken that went something like this:
   Me:   "Thank you for calling Microsoft Exchange support.  My name is Stephen.  How can I help you?"
   Joe Admin:   "My information store won't start."
   Me:   "OK.  Is this the first time this has happened?"
   Joe Admin:   "No.  It's been crashing several times a day for the last week or so.  People would start calling me when they couldn't get their email so I would go and start it back up.  It never had any problems restarting until today, and now it won't start.  It keeps saying something about 16 gigabytes."

We would then proceed to temporarily raise the limit to 17gb (if we could) to quickly pull data out before the floodgates of inbound email are opened thereby dooming the poor guy to Exchange 2003 standard server with a 17gb priv1.edb.  Or we start the explanation of different options to try to remove data from a store already packed so full of data that it's like an overstuffed sock drawer that won't open.  Oh, you know what I'm talking about...

It's going to be great to not have to worry about the 16 gb limit!

Increasing the ceiling past 16 gb can be dangerous for Small/medium businesses
Not every business has the resources to hire a properly trained administrator who is dedicated solely to running the company's messaging infrastructure.  Believe me, I understand that fact, and I've got a lot of respect for the one-man shops out there.  That's a lot of hats to wear, and your email systems should just keep running without the need for your daily attention.  You've got too many other things to worry about every day.  But Exchange is "mission critical"... just ask anyone who's had to waste precious time enduring a lecture from the boss, the CIO, and the CEO (or dozens of other people who don't understand your systems) while you are in the middle of a disaster recovery scenario.  But what does that have to do with the 16gb limit?

1. When I hit the 75 gb limit...
   When.  Not if.  You remember back around 1992 when drives first started hitting 1gb?  "I'll never fill a 1gb hard drive!"  Yeah, I said that about 10gb.  I said that about 100gb.  And now I look at my PC at home with about 650 gb and worry about filling it up.  When Joe Admin called me, and we were in a position where we had to defrag his information store, we normally estimated about 2 gb an hour.  It often took 8 hours (or more) to complete the defrag on the store.  And then we had to try to get data out and defrag it again.  Now imagine this same scenario, but Joe is calling Microsoft Support with a 75 gb store.  (Because Joe's boss said "Why do we need mailbox limits if we can grow to 75 now?  We'll never fill 75gb!")  It's true that servers are faster now.  And drives are faster now.  But a defrag on a 75 gb store will still take many hours.  Many, many hours.  Not a good thing to have to tell Joe Admin's boss.  Or CIO.  Or CEO.
2. What do you mean "Outlook is retrieving data from the Microsoft Exchange Server"...
   In general on any given Exchange server, the bigger a store gets, the slower it gets.  It's a fact of life, and we can certainly do things to mitigate situation.  But in many situations, the solution to "making Exchange faster" is cost prohibitive to many small businesses.  Very often, this degradation of performance is due to a suboptimally configured (or architected) disk subsystem.  Or to put it in other terms, with Exchange we need to architect our disks for performance and not for storage capacity.
3. Murphy is watching...
   And he doesn't like you.  Sorry.  I hate to be the one to break it to you.  And sooner or later, he is going to do something to take down you server.  Now your database is inconsistent (aka dirty shutdown).  And you need to ESEUTIL /P it... then defrag it... then ISINTEG it.  Maybe you had backups... maybe you didn't.  But chances are that your boss wants that data back NOW.  You can always repair the database on a recovery server.  You do have a spare recovery server just sitting around waiting, don't you?  Well, that's ok, you can just find some spare hardware and follow your detailed and tested disaster recovery action plan that you've been practicing in all your free time.  You've got all of that written out and tested right?  Well, let's just get the backup from last night restored so we can return the server to production.  What do you mean that the last good backup was 4 months ago? 
4. We're okay as long as we have good backups...
   <Obvious statement> Backups are very important. </obvious statement>  But let's consider for a minute what happens when a backup starts on an Exchange server.  Actually, let's consider what doesn't happen -- a little subset of online maintenance called online defragmentation.  When we begin a backup on any database in a storage group, online defragmentation on all databases in that storage group stops until the backup completes.  With a 15 or 20 gb store, this probably isn't a problem.  We can even use NTBackup to back up a store of that size in a fairly short period of time.  And many small businesses don't have the resources to invest is blazing fast backup solutions.  So when the store grows to 60-70 gb, the backups take much longer.  Backups are now taking 10-12 hours instead of 2 hours.  Not a problem since we're an 8-5 shop.  Except that default, online maintenance is set to run from 1 am to 5 am daily.  And if your backup kicks off every evening... not much (if any) time left for online maintenance.  Well, what happens when my online maintenance doesn't run?  Lots of bad things.  Nothing dramatic and nothing immediate.  It's more of a slow rot.  No online defrag means new data cannot optimally use the white space within the database.  This can cause the file to continue to grow in size despite having free space in the database.  Other tasks that occur during the online maintenance -- Purge indices, tombstone maintenance, dumpster cleanup, public folder expiry, age folder tombstone, folder conflict cleanup, update server versions, secure folder cleanup, site folder check, and deleted mailbox cleanup.  These are all low priority tasks.  Bottom line is that if your backup times are consistently trampling all over your maintenance times, you won't have happy healthy databases.

So what do we do about it?
Well, with the new 75 gb limit (as with many things), just because you can, doesn't mean that you should.  The following best practices are nothing new, but should be revisited now to help you fully enjoy the 75 gb limit:

• Configure message size limits
• Configure mailbox size limits
• Do these tasks easier with System Policies
• Use Mailbox Manager to help manage old content that the users won't
• Run MOM 2005 Workgroup Edition (<10 servers) or MOM2005
• Use PerfMon to periodically get a pulse on the health of the server.  Historical data helps to isolate whether the CEO's new 10gb mailbox is causing issues.
• Plan storage to take advantage of the 75gb limit so it doesn't take advantage of you.

Object limits in Exchange 2000/2003

Will this affect you? 

Probably not unless you're in a huge environment, but conversations with my customer on this matter has led me to the conclusion that there are multiple points of  confusion on the topic.  An attempt to find detailed documentation frustrated me.  It was there.  I had seen it.  But I couldn't find it now.

What are the limits in an Exchange organization?

• 1000 Exchange servers
• 1000 Administrative Groups
• 100 domains
• 1000 connectors in a Routing Group

Why is there a limit?

Active Directory, by default, is confugured with a maximum page size of 1000 for any LDAP request.  So, in a default configuration, if we have 1001 objects that would be returned by the LDAP query, the query will return no results.  We can potentially encounter performance issues prior to hitting the 1000 object limit (depending on your environment's design), but the hard limit is 1000.

But wait, I can modify the MaxPageSize value for LDAP using NTDSUtil!

Yes.  Yes, you can.  But there are still problems.  There are possible performance issues with other LDAP applications that might be running against your domain controllers.  But the bottom line is that there are Exchange-specific issues with making the modification.  Microsoft doesn't recommend or support making the change for good reasons.  It will break some Exchange functions.

Well, if there's nothing that I can do about it, then at least help me understand the limit...

Now, that's what I'm talking about.  This is where some of the confusion tends to set in.  We'll work backwards and hit the easy ones first...

1. 1000 connectors in a Routing Group
   Two things to remember here. 
   First, if I have a Routing Group Connector (or any kind of connector) between RoutingGroup1 and RoutingGroup2, then it actually gets counted twice.  Once for each direction.
   Second,  third party connectors count as well.  So we're not just talking about your basic mail flow connectors.  We're talking about fax or any other kind of connector too.
   Fact is that since this is 1000 per Routing Group, I don't see many people hitting this problem.  Moving on....
2. 100 domains
   I really wanted to just say this is because "I said so" or some other flippant answer.  Unfortunately, I'm not the one who said so... the developers did.  When I first thought about it, it did seem kinda arbitrary since we're (obviously) not capping it at 100 because of the MaxPageSize on the LDAP query.  Actually, the recommended limit of 100 was imposed in E2k for performance reasons.  The DSAccess component (prior to E2k3 SP1) cannot cache objects that are more than 32kb.  When an Active Directory exceeds 100 domains, the associated ACLs can cause a directory object to easily swell over the hard limit of 32kb.  E2k3 SP1 modifies DSAccess so the objects over 32kb are chained together in multiple chunks.  Although this functionally removes the hard limit of 32kb, we can still run into performance issues if the size of directory objects get too large, so 100 domains is still a recommended limit for an Active Directory containing Exchange.
3. 1000 Administrative Groups
   Same reasoning as the 1000 connector limit here -- MaxPageSize in LDAP queries.
4. 1000 Exchange servers
   This is the good one.  What, precisely, counts as a server?  What if I cluster my backends?  What if I use NLB on the frontends?  How do the roles of the server impact this?  It's pretty easy when you think about why the limitation is in place and then work backwards.  The 'why?' at this point should be obvious -- MaxPageSize in LDAP queries.  But what are we querying?  Well, we know that the Exchange Organization is bound by the forest and not the domain.  This effectively prohibits the querying of the Domain partition of Active Directory for the information.  In fact, we are querying the Configuration partition (since it is shared by all domains in a forest) for objects with an objectCategory of msExchangeServer.  Now with a standalone Exchange server, it's pretty easy to see that the server name will match the cn of the object returned.  So if I have a (non-clustered) server named SERVER1, after I install Exchange, the server object in the ESM will show SERVER1.  But what about those nasty clusters?  I have 2 nodes of my cluster and the nodes are named CLUSTER-NODE1 and CLUSTER-NODE2.  I install Exchange on both nodes (following the first part of Q328875 for Exchange 2000 or Q895981 for Exchange 2003), and then I look in the ESM.  What do I see for this cluster?  NOTHING!  Why?  Because the Network Name resource for the Exchange Virtual Server has not yet been created.  So I create the Network Name resource EXCH-CLUSTER (along with the needed prereqs) and continue to follow the procedure referencing in the articles.  After I bring the Exchange Virtual Server (EVS) resource group completely online, I check in the ESM and see that it now references the Exchange server named EXCH-CLUSTER.  This means that with an A/P cluster, there will be one EVS.  But on an A/A cluster (which is wrong for so many reasons), you will have two EVSs.  And it is truly the number of EVSs that matters and not the number of 'active nodes' in the cluster -- although they are usually equal.  If I configured an A/A cluster (trust me, I never would), I could fail one EVS over to the other node.  At that point, one node would be passive (not owning any Exchange resources at the moment), however, we still have two EVSs.  So technically, the number of active nodes does not necessarily reflect the number of EVSs.  NLB?  Doesn't matter.  Shared IP address doesn't impact server count.  Role of the server?  Doesn't matter.  An HTTP front-end counts towards the 1000 server limit as much as a back-end.

Thx to Zach McNelis and Jeff Beckham for review. 

Thx to Ross Smith IV for helping me replace the "I said so" with useful information about the 100 domain limit.