this post was submitted on 28 Jun 2023
38 points (95.2% liked)

Sysadmin

7717 readers
95 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
[email protected]
[email protected]
[email protected]
[email protected]

founded 1 year ago
MODERATORS
 

It started off with an employee sending an email to a distribution list called "Bedlam DL3" asking to be taken off the list. With 13,000 recipients and everyone replying all with, "Me too!" and other messages, it was estimated that over 15 million messages were sent through the system in an hour. This crashed the MTA service due to a recipient limit. Each time the MTA service recovered, it would attempt to resend the message again which lead to a crash loop.

As a result of the incident, the Exchange team introduced message recipient limits and distribution list restrictions to Exchange, which is something we all use today!

More on the story here: https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643

cross-posted from: https://techy.news/post/2224

top 2 comments
sorted by: hot top controversial new old
[โ€“] Bransonb3 5 points 1 year ago

There is a great video from Dave's Garage about this: https://youtu.be/pBmuY6qFMPQ

[โ€“] philipstorry 2 points 1 year ago

I think the thing to take away from this is the poor state of management/maintenance tools that Exchange had. Thankfully over the years this has improved, but in those early years it was pretty bad.

I was using both Exchange and Lotus Notes/Domino in that period. If the same thing had happened in Notes, we'd just shut down the mail router task, open the mail.box database and remove the offending message. Easy. But that can only be done because Notes re-used its database system for everything, including the mail queue.

Exchange has a history of reinventing the wheel even within its own architecture. Let's just say I'mm very grateful that we're now on Microsoft 365 and these things are Microsoft's problem, not mine... ๐Ÿ˜‰