From time to time, we have major spam emergencies. Running a service, stuff invariably breaks. We try our best to monitor stuff, but something always comes up that we weren't aware of. Queues build up, perf monitors don't always get closely monitored, IPs get blocklisted, and then relisted and then delisted again, and so forth. When you're involved in a service business, things are moving very fast and there are lots of moving parts.
We hadn't had a major spam problem in a long time. Sure, we get reports of spam leaking through or messages being flagged as false positives, but I characterize those as routine. They happen all the time and we have a team dedicated to handling that stuff.
It gets tense when stuff breaks that hasn't broken in a long time because nobody can remember how to fix it. When you do it once and then leave it, it doesn't always stick in our memories. Over the past two weeks, two separate things have required immediate attention in the spam area. So, when that happens, they call in the expert who knows everything about how to deal with spam: me.
Last week, we had some issues with generating blocklists and keeping data up-to-date. We are in the process of migrating the technology to better hardware but that requires the maintenance of existing, legacy systems. The old hardware was coughing up mucous so I had to sit down, move files around, delete stuff (had some help with that part), and kick off the process manually. I ended up staying late that night but I got it working and managed to relieve the load off our content filters by about 30%. That's a lot.
Then, just recently, we had another spam problem. Without going into too many details, let's just say we were having outbound delivery issues. Seriously, this was just like an episode of 24. We had a bunch of people around a conference room table, and a couple of more on a conference room bridge. People were digging through logs, digging through delivery pools, analyzing messages, and throwing in spam rules. I was the one that ultimately diagnosed the problem, verified that the existing solution wasn't broken and how we could use it solve the problem. In other words, nothing was broken, we just needed to react to a new spam outbreak.
It was actually a lot of fun. Diagnosing and then solving a problem using a feature we created a few months ago gives me a great deal of satisfaction. It was humorous because just like Jack Bauer, all the people listened to my recommendation and ultimately the stuff I did is the stuff that will be used to solve the problem (minus any harsh interrogation techniques... spammers should be so lucky).
Now all I need is the cool phone with the "Doo-doo... dee-doo" ring.
PingBack from http://blog.postmaster.gr/2008/06/11/on-documentation/