Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

It's Bedlam all over again...

It's Bedlam all over again...

  • Comments 13

A really long time ago, I wrote a post about the "Bedlam DL3" event at Microsoft.

Well, a couple of days ago, we had another Bedlam DL3 event.  For some reason, the permissions on one of our internal DLs were messed up, and someone had granted "send-as" permission to all 2500 members of the DL.

Someone then sent a message to the DL with the From: field set to the DL (I have no idea why, or who did it, but they did it).

That person then realized that they had made a mistake and they tried to recall the message.

The problem is that message recalls are handled on the outlook client (all 2500 of them).  So every recipient of the message sent a "Recall Success" or "Recall Failure" message to the sender of the email message.

 

And the Exchange servers proceeded to.....

 

Slow WAY down.  Not surprisingly, given that they were handling what was estimated at 36G of email (2500 emails sent to 2500 recipients is 6.25 million emails, each email was about 6K bytes long).

But they handled it with aplomb.  Even a 36G email bomb was handled by the servers.  Email backed up for several hours, but the servers didn't crash.  Man, things have improved since way back when.

 

The best part of this was that the email alias in question was a security-related email alias.  So everyone on the DL was sending emails to the DL speculating about who was pen-testing the live Exchange servers :)  All the while the queues were being drained and clients were actively using the system.

 

I was pretty impressed, to be honest.

  • I think in this day and age of fast computers, the programmers should go out of their way to write software to prevent users from making mistakes like this one.

    In this case, it'd be great if the software warns the user when the from field is a big DL, and let the user confirm automated replies to big DL - OK it's not automated then, so the rule is: no automated replies to big DL.
  • Sure, but where do you draw the line?  And what happens when the user making the mistake uses an SMTP client which has no clue about the size of a mailing list.

    It's a potentially intractable problem.
  • Meh. You're thinking like devs. Look at it through a tester's eyes. To me, this is a *wonderful* bug. Give the person who found it a t-shirt or something. Something more substantial than a latte or a Dove bar. That's just great!

    And I love that although this behavior could be gated by Exchange it's a fundamental problem in Outlook instead. Neat.
  • I think what puzzles me here is why automated responses (recall success/failure) would be allowed to be sent through a mailing list. This seems like a recipe for a mail loop. If I understand the story correctly, the original message only hit O(N) and it was the automated replies that nailed the server with O(N^2) numbers.
  • 36G?  You call that an e-mail bomb?  I do that every millisecond, day in and day out.
  • I was a part of that DL, so I share the pain. I think that what would have fixed this problem is by simply changing the default options for message recalling. Most people that actually have used that feature already know that you won't ACTUALLY recall the message. Users are curious and will diff the messages to see where you screwed up. :)

    Anyhow, this made me think of the following, and I would like someone to answer this for me. How come even free e-mail services have a higher mail quota than the one provided to us on corpnet? WTH?!?
  • Is this Exchange 2003, or have you started dogfooding Exchange 2007 yet?

    Exchange 2007 requires x64 hardware, so may be much more capable of handling very large amounts of data!
  • We had something like this at the start of the year. At the time I was a contractor, and it turned out there was fairly large DL that had most contractors in the company on it. We found out about this when an HR person sent a small request that was meant to go to just one person to the whole DL.

    You can guess what happened next. It started with some people responding back to the original e-mail, say they didn't know what it was. Others pointing out that it was a DL, more people saying not to reply to the e-mails, even more people replying to the e-mails, and then the jokes started...

    All in all it ended up producing at least 300 pieces of e-mail (multiplied by the number of contractors) before the DL was pulled and all posts purged. A lot of people got a good talking to about the proper use of e-mail after that. It didn't kill the servers, but it slowed down e-mail for a good six hours.
  • Drew, actually KC's making a T-Shirt :)

    Mike, this was a mix of E2K3 and E2K7.
  • > How come even free e-mail services have a higher mail > quota than the one provided to us on corpnet? Oooh. Yet another feature that the maker of WNT inherited from the maker of VMS.
  • So what's the history behind it being called Bedlam?  After all, Bedlam is a lunatic asylum.  Any relevance?

  • PingBack from http://paidsurveyshub.info/story.php?title=larry-osterman-s-weblog-it-s-bedlam-all-over-again

  • PingBack from http://barstoolsite.info/story.php?id=3491

Page 1 of 1 (13 items)