Terry Zink's Anti-malware Blog

Countries with the most infected computers

All Spammed Up has a new post up referencing an article that security researchers have issued a report indicating that Spain is the country with the most infected computers, at 44.5%.  The United States is second at 14.4%.  The countries with the least infections are Sweden, The Netherlands and Peru.

The Microsoft Security and Intelligence Report, v7, doesn’t measure infection rates quite the same way.  Instead, it has a metric called Computers Cleaned per Thousand machines scanned, or CCM (where M is the Latin word for thousand – mille).  This is a measure of the number of computers cleaned per thousand executions of the Malicious Software Removal Tool (MSRT).  Below is a heat map of the countries with the most infections, for a better image either click the image (as it will be cut off in this blog) below or download the full report and zoom in your Adobe pdf reader (it is on page 41):

Click here for ginormous image.

Going from the above, we can see that Spain is definitely one of the hotter countries.  But, it is not the hottest.  Below is a table of the countries with the worst rates of infection:

Spain is clearly one of the worst but it is actually only number 4 behind Serbia and Montenegro, Turkey and Brazil.  There is no set pattern but in general, countries in the developed world (at the very least, the G7) are not found among the worst countries for malware infection.  Of course, the very interesting thing is that even within different countries, the types of infections are different.  Microsoft classifies the types of malware it removes and below is a table of what it looks like among various countries.  Click on the picture to see the full image as it will be cut off partially in this blog:

Click here for full sized image.

From our table above, Brazil and Spain are the worst offenders for malware infected computers, coming in at 3 and 4 respectively.  Yet the type of malware hitting them is different.  Brazil is plagued by Password Stealers that target Brazilian banks (led Win32/Bancos), followed by Worms and Viruses.  by contrast, Spain is hit hardest by Worms, then miscellaneous trojans and password stealers, which are substantially less than Brazil.

The United States was number 2 in the report that All Spammed Up referenced, but the most common malware affecting systems in the US are miscellaneous trojans, followed by trojan downloaders and droppers and then Adware (the pattern is similar in the United Kingdom).  So, different regions of the world are more prone to certain types of attacks than others. 

If we can make a generalization, then the countries with the highest malware infections rates as measured by the MSRT CCM metric are more prone to Worms.  The United States is actually about average with regards to infection (8.6 CCM vs 8.7 global average).  With regards to the lower countries, I am currently not seeing any discernable pattern and I would have to do a deeper statistical investigation.

Changing the title of this blog

For the very first time since I created this blog back in July of 2006, I am changing it’s title.  It is no longer “Terry Zink’s Anti-spam Blog”, it is now “Terry Zink’s Anti-malware Blog”.

I have not moved out of spam.  Instead, I have decided to broaden the focus of this blog to include malware as well as spam.  The relationship between the two is tightly integrated and I believe that I need to touch a wider array of the security space to remain relevant.  The only real change you will see will be that I will be writing about malware  more than I have in the past, and other security topics in general.  My sphere of interest has expanded from focusing on spam to focusing on the general security space.

Happy reading.

The Story of Conficker, part 3

Setbacks and Triumphs

The domain registration task became exponentially more challenging on March 4, 2009, with the discovery of Worm:Win32/Conficker.D. Investigators reverse-engineered the new variant and determined that it was programmed to generate 50,000 new domain names a day across 110 TLDs, beginning on April 1, 2009. Though this seemed at first like an impossible hurdle to overcome, CWG members immediately began working to counter the effects of the upcoming change. As security researchers continued to analyze the Conficker.D malware, ICANN staffers began contacting the registries responsible for each of the affected TLDs seeking cooperation in registering or blocking the domains, and the CWG compiled “go packs” of information for Internet service providers and enterprises about the steps they should take to help keep their customers and employees safe.

April 1, 2009, came and went, with the world outside the security community noticing little or no change. By that time, however, ICANN had secured the cooperation of all 110 TLDs used by Conficker, and the global DNS community was active and prepared to deal with the Conficker threat. Rapid, effective collaboration across borders and organizational lines had proven instrumental in containing what has been, and remains, a significant threat to the world’s computers and information.

The CWG Today

The CWG remains in place today, with more than 300 member organizations representing law enforcement, academia, and industry, and remains vigilant against new developments. In cooperation with ICANN and the DNS community, the CWG continues to block or register the 50,000 domain names generated each day by the Conficker algorithms. Each month the group supplies the 110 affected TLD operators with an updated list of generated domain names covering the next several months, so they can begin implementing countermeasures well in advance. Automated mechanisms verify that each domain name has been blocked before it is scheduled to be used and alert the CWG for any that have not, so activity for those domains can be closely monitored. Once in a while, a domain name generated by the algorithm happens to correspond to an existing domain owned by a legitimate party; in such cases, the CWG contacts the legitimate domain owner in advance and offers assistance managing the expected spike in traffic coming from infected computers.

In March, the group underwent a reorganization process to add structure and to segment its work by subject area to work more effectively. The group maintains a Web site at http://www.confickerworkinggroup.org with links to information in multiple languages about Conficker and resources that service providers and end users can use to determine if they are infected, and if so, what to do about it. The fight against Conficker is not over. The five identified variants continue to spread to new computers due to a lack of information or action on the part of some system administrators and end users. Even after Conficker recedes into insignificance, there will likely be other threats of similar magnitude to deal with in the future. As such threats appear, though, collaborative efforts, such as the CWG, can provide the global security community with unequaled tools for mitigation and resolution.

Conficker, Part 1
Conficker, Part 2
Conficker, Part 3

The Story of Conficker, part 2

 The Conficker Working Group Is Born

In January 2009, representatives from a number of security research companies and domain registrars, along with the anti-botnet Shadowserver Foundation, began discussing how best to implement a defensive Domain Name Service (DNS) strategy to handle domain registrations. To coordinate the significant amount of e-mail being generated by these discussions, the group established the CONFICKER e-mailing list on January 28, which drew a growing number of security researchers and members from law enforcement, academia, and industry, in addition to members representing each of the eight TLDs used by Conficker. Enlisting the support of the TLD operators would prove to be a vital step in containing the Conficker threat, enabling the group to block domain names more efficiently and at far less expense than would be possible through the commercial registration process.

By early February 2009, working group members had instituted a process for registering as many domain names as possible, before the Conficker operators could register them, and assigning them to IP addresses belonging to six sinkholes (server complexes designed to absorb and analyze malware traffic) operated by organizations belonging to the working group. Infected computers looking for command-and-control servers would contact the sinkholes instead, providing researchers with valuable telemetry for analyzing the spread of the worm. A number of Internet service providers (ISPs) were also able to use this telemetry data to identify infected computers.

Around the same time, the Internet Corporation for Assigned Names and Numbers (ICANN), which is responsible for allocating IP addresses and managing the Internet domain name system, invited the group to deliver a presentation on its domain registration efforts to a meeting of the ICANN board of directors. The board expressed its support for the program and assigned two staffers to help coordinate it. Despite these efforts, the Conficker operators were still able to register some domains before the working group could get to them. To mitigate this, researchers at Kaspersky Lab, an anti-malware vendor headquartered in Russia, worked with OpenDNS, a free network resolution service used by many organizations and individuals, to compute a year’s worth of Conficker domain names and proactively point them at the group’s sinkholes. Any infected computer belonging to an OpenDNS user would not be able to contact any of the Conficker command-and-control servers, even on domains the Conficker operators had been able to secure.

The formation of the Conficker Working Group (CWG) was officially announced to the public on February 12, 2009, as what a number of news stories characterized as an unprecedented example of global cooperation in the computer security industry, and a potential blueprint for dealing with threats in the future. The CWG had grown from an e-mail list for nine individuals to a group of more than 30 member organizations from around the world, coordinating complex activities through a robust communications infrastructure. On the day the CWG was announced, the group had successfully registered every Conficker domain name for the next 10 days, a genuine—if temporary—victory over the Conficker operators.

Conficker, Part 1
Conficker, Part 2
Conficker, Part 3

The story of Conficker

One of my favorite stories in the recent edition of the Microsoft Security and Intelligence Report v7, pp 29-32, is that of the story of Conficker.  I thought I would repost it here because it illustrates the problem of Conficker and the way the industry worked together to respond to the problem.

Case Study: The Conficker Working Group

The appearance in late 2008 of Win32/Conficker, an aggressive and technically complex new family of worms, posed a serious challenge to security responders and others charged with ensuring the safety of the world’s computer systems and data. (“Win32/Conficker Update,” beginning on page 95, explains the technical details of the Conficker worm and the methods it uses to propagate.) Working together, however, the security community was able to react quickly to the threat and contain much of the damage, in the process establishing a potentially groundbreaking template for future cooperative response efforts. On October 23, 2008, Microsoft released critical security update MS08-067, addressing CVE-2008-4250, a vulnerability in the Windows Server service that could allow malicious code to spread silently between vulnerable computers across the Internet.

The vulnerability affected most currently supported versions of Windows, although architectural improvements in Windows Vista and Windows Server 2008 made them more difficult to exploit than earlier versions. Like the worms that plagued the Internet earlier this decade, malware that exploited the vulnerability would be able to spread without user interaction by taking advantage of the protocols computers use to communicate with each other across networks. For this reason, and because actual attack code that exploited the vulnerability was known to exist in the wild at the time, the MSRC took the unusual step of releasing MS08-067 “out of band” rather than wait for the next scheduled release of Microsoft security updates, which takes place on the second Tuesday of every month. Security Bulletin MS08-067 happened to be released on the last day of the eighth annual meeting of the International Botnet Task Force in Arlington, Virginia, a suburb of Washington, D.C., where attendees agreed to closely monitor developments around what appeared to be the first legitimately “wormable” vulnerability to be discovered in Windows in several years.

The November appearance of Win32/Conficker, the first significant worm that exploited the MS08-067 vulnerability, marked a major challenge for security researchers, due to the aggressive tactics several of its variants used to propagate. Despite this, researchers soon discovered a way to limit or eliminate the Conficker bot-herders’ ability to issue instructions to infected computers. As described on page 96, the authors of the Conficker malware used an algorithm to generate 500 new domain names every day (250 for each of the first two Conficker variants discovered) to use for command-and-control servers. Computers infected with Conficker would attempt to contact each of these generated domain names every day. If the authors had a task they wanted the computers in the botnet to perform, they would simply use the same algorithm to generate domain names in advance and register a few of them, which they could then use to host command-and-control servers.

Fortunately, researchers from Microsoft and other organizations were able to reverse engineer the domain-name-generation algorithms used by the first two variants, designated Worm:Win32/Conficker.A and Worm:Win32/Conficker.B, soon after each variant was discovered. This enabled them to begin registering the domain names before the botnet operators could, thereby impeding the Conficker malware from obtaining new instructions. Initially, the researchers resorted to registering the domains commercially through the domain name registrars for the eight top-level domains (TLDs) (.com, .net, .org, .info, .biz, .ws, .cn, and .cc) used by Conficker, an approach that quickly became unworkable. Registering 500 domain names per day would cost thousands of (U.S.) dollars per day for the foreseeable future—and the cost would only increase if new variants appeared using different name-generation algorithms. It was clear that more help would be needed.


Conficker, Part 1
Conficker, Part 2
Conficker, Part 3

Microsoft’s Security and Intelligence Report, v7, now available

Every 6 months or so, Microsoft releases its Security and Intelligence Report for the previous 6 months of the year.  SIRv7 is now available here.  This is a very comprehensive document covering topics from the entire threat landscape that Microsoft is involved with combating.  This year’s report contains three key messages:

1. The redistribution of knowledge – Microsoft’s level of security intelligence will be unmatched and provided to individuals and organizations to help them make better security decisions.
   
   
2. OK, so what else is new? – The SIR contains the information that is relevant to people right now.
   
   
3. What do I do now?  - The SIR allows people to assess where they are and what action they need to take.

I thought I would post an excerpt from the Executive Foreword.  I think that this highlights the theme of this current SIR.

---

Welcome to the seventh installment of Microsoft’s Security Intelligence Report, which I hope you will find is the most extensive and comprehensive edition to date. The cover story in this report looks back at the major threats that have attacked customers over the last 10 years, and then the report drills deeply into the current threats that you need to understand and includes what you can do to best manage your risks.

At Microsoft, we remember the pain past incidents caused our customers and we reflect on them frequently. In particular, the Slammer and Blaster attacks that disrupted the Internet in 2003 are vivid reminders of the responsibility we have at Microsoft to ensure our products are as secure and privacy enhanced as possible.

As you can see from the timeline above, 2003 and 2004 were difficult times. [tzink note: see the report for a better image]  But, you can also see that since then, major security incidents have become less and less frequent. From the data in this report, you’ll also note that the scope and impact of major events have changed, as well. For example, from the press surrounding the Conficker worm that has been attacking customers over the past year, it’s easy to conclude that Conficker is just as widespread and impactful as Slammer or Blaster—but in most respects, it hasn’t been. In 2003, Blaster became one of the most prevalent threats impacting home PC users. Six years later, Conficker didn’t even make the Top 10 list among this audience. I don’t want to minimize the pain that many of our customers experienced fighting Conficker, because, as you’ll read in the report, it was the top threat detected and cleaned in enterprises in the first half of 2009, but Conficker emerged in a much different software industry than Slammer and Blaster.

Indeed, the software industry has matured a great deal since the days of Slammer and Blaster. Since 2003, the software industry has improved its ability to mobilize and coordinate resources to fight threats… The Conficker Working Group (CWG) was founded earlier this year, establishing a new model for how the collective industry can work together to mitigate global threats.

The industry was able to proactively get ahead of Conficker by discovering the vulnerability before attackers could use it in widespread attacks. The Security Science team at Microsoft was able to find the MS08-067 vulnerability, which Conficker uses to propagate, and work with the Microsoft Security Response Center (MSRC) to release its update before attackers could use it for a Blaster-type attack. Our industry partners helped protect many customers from attack via the Microsoft Active Protections Program (MAPP). MAPP supplies Microsoft vulnerability information to security software partners prior to security update releases from Microsoft… This program enabled the majority of MAPP partners to provide protections to their customers for Conficker 24 hours after the MS08-067 security update was released. This meant that many customers were protected up to a week earlier than traditionally possible, and certainly much earlier than customers could obtain such defense-in-depth protections and threat mitigations in 2003.

With the vulnerability that Slammer exploited, many administrators didn’t know whether they needed to apply a security update or that it had to be applied manually. Today, customers are notified and protected much faster; multiple communications channels exist to help customers find and understand information on security vulnerabilities. Security advisories help draw attention to security issues as they unfold, and provide customers with critical information before security bulletins become available. Microsoft’s advanced notification service provides customers with an insight into the number and nature of security updates that Microsoft will be releasing each month so they can plan more effectively for the deployment of the updates. Security bulletins provide information on vulnerabilities, along with workarounds and mitigations.

…

The progress that the software industry has made to better protect systems and customers might be small consolation to the users of those 5 million systems that were infected with Conficker in the first half of 2009. Still, it is a significant step forward, given that more than 100 times as many systems were protected from Conficker. This is in stark contrast to the Slammer and Blaster attacks of 2003 where many, many more systems were infected. The industry will continue to work together to make the frequency, scale and scope of emerging threats as minimal as possible.

We thank you for your help and efforts to protect the ecosystem, and look forward to continuing to work with you to create a safer, more trusted Internet.

George Stathakopoulos
General Manager, Trustworthy Computing Security
Trustworthy Computing Group

---

More excerpts to come over the next few days highlighting global trends in the threat landscape.

Live Free or Die Hard

Spoiler alert.

This past weekend, I got a chance to watch the 4th installment in the Die Hard series, Live Free or Die Hard.  I hadn’t seen the whole thing end-to-end before, only parts of it.  It was nice to finally get a chance to see the whole thing.

Overall, I like it.  It’s so far over the top that it’s completely unbelievable… but that’s the point.  It’s supposed to be unbelievable.  A jet plane flying around the city at low speeds and hovering like a helicopter in between parts of a freeway?  John McClane getting hit by a car and walking away?  Bad guys falling 20 feet onto concrete below and not even suffering a limp?  Whatever.

But what about the basic premise of the story?  In case you haven’t seen it, at the beginning of the film, various government agencies experience a major shutdown.  Hackers infiltrate the computer systems of the FBI, departments of transportation, nuclear facilities… well, nearly every agency in the United States and they proceed to shut it down.  The villain behind it is a disgruntled employee of the Department of Homeland Security who is a brilliant programmer and security expert.  After the events of September 11, he warned his superiors that the nation’s cyber infrastructure was vulnerable to attack.  Rather than listen to him, he was ignored and/of vilified, and fired from his job.  To get revenge, the villain plots a major hacking operation to demonstrate to his superiors that they should have listened to him; this proves that the nation’s infrastructure is vulnerable.  In reality, this is all a smokescreen as it is a diversionary attempt to steal billions, possibly trillions, of dollars of wealth.  In the hacking world, the villain would be classified as a cyber warrior.

Of course there are some things in the movie that are completely unrealistic like the physical stunts above.  Furthermore, why would the bad guys hack into a hacker’s computer and wait for them to hit the Delete key that detonates some C4, rather than them executing the explosion remotely?  That seems a little inefficient to me.

But that’s not the question I want to address.  What I want to ask is whether or not the nation’s cyber infrastructure is really as vulnerable to attack as the movie makes it out to be.

My answer?  Unlikely.  There are a couple of problems with this scenario:

1. The bad guy’s team was too small. 
   
   I counted a team of maybe 3 hackers on the bad guy’s team, not including himself.  That is way too small a team to control multiple that much computer systems.  Over here, we have a lot of people running a network that is not nearly as complicated as multiple government departments.  It takes constant monitoring and tons of documentation to keep things running smoothly.  And many times, things don’t run smoothly.  It would take a very long time to code something up, test it, deploy it, and control it while evading detection during the entire time the operation was running.
   
   Of course, something like that might be possible but three people is not enough.  It takes forever to get done all of the stuff I mentioned.  And it is very resource intensive.  Nobody writes code that executes as perfectly as the villain’s does the first time they try it out.  Of course, maybe they tested things but the government has a lot of independent systems.  The left hand doesn’t know what the right hand is doing.  So, you need guys who are familiar with each of the government’s various departments’ computer systems.  And know how to control them.  That just isn’t possible with 3 people.
   
   The computer hackers running the operation would be busy all day trying to evade detection and the amount of psychological pressure on them would be intense (especially when your boss is holding a gun and waving it around, and his girlfriend could knock your teeth into next week).  Nobody under that type of pressure avoids making mistakes, so you have to build automated mechanisms to control stuff for you.  And if you do that, it takes time to code it.  And if you take time to code it, even if you’re a great programmer, it’ll still have bugs.  The flawless execution of their stuff was completely unrealistic without having back up teams responding to issues that would inevitably come up.
   
   
2. The nation is vulnerable to attack, but not in the way they made it out.
   
   The uber-point of the nation’s security being vulnerable is correct, but not in the way they were making it out to be.  In my first point, I say that the team is too small.  I go on to say that government departments have all their stuff implemented differently.  I don’t know this to be true, of course, but I surmise that each department built their stuff independently of each other.  Some may have built their stuff on Linux and MySQL.  Others may have used Ruby.  Others, Perl.  Maybe there is some Java, Exchange, PHP (ugh) and Oracle.
   
   And when stuff is built independently, they don’t talk to each other.  And when they don’t talk to each other, it is very difficult to take them all over simultaneously. 
   
   Furthermore, when computer systems get big, particularly when they were implemented in the 1980’s or 1990’s, they aren’t documented very well.  If you work at a company whose infrastructure was written long ago, you’ll know how disorganized it is.  The code is poorly written, you will probably have GOTO’s going to GOTO’s, and there is no written support.  If you want to figure out what is happening, you have to “decompile” the code in your head or on paper.  It’s a mess.
   
   Thus, if an organization as large as the government is going to be attacked, what is more likely to happen is that rather than being controlled, it is more likely to be shut down than having control of it given to an external attacker.  A hacker can break in and deploy a worm, but this is much more likely to cause systems to crash and not boot than it is give control to a remote user.  Remember, it is not a single organization with a unified communication system, it is multiple computer networks that must be compromised and controlled. 
   
   Poorly written code doesn’t act like a cohesive unit.  Instead, it deadlocks and becomes unresponsive.  Memory leaks, and resources do not get released.  It’s the equivalent of having large paperweights on your desks (like my IP phone at work) and servers that sit there, spinning their wheels and doing nothing. 
   
   When the governments of Estonia in 2007 and Georgia in 2008 were attacked, and when Twitter suffered a DDOS attack in 2009), they shut down the nation’s, or web site’s, computer systems but they didn’t control them from the inside to make them do nefarious things.  They “just” rendered them inoperable.  So, we can all take solace in the probability that if a hacker ever takes over, traffic lights will only go out.  We don’t have to worry about them all turning green.
   
   
3. An emergency data dump wouldn’t go to only one server in one location.
   
   Or, I certainly hope not.
   
   As I said in my introduction, the taking over of a nation’s computer systems was only a diversion.  When this happened, all of the nations banks, financial institutions, trading accounts, etc, started downloading of all of its data into a data center located in Maryland (I think).  This computer data center was supposedly the Social Security Administration, but in reality it was designed to be a redundant backup in the case that a real emergency happened.  Of course, this emergency did happen, and the bad guy is the one who designed it that way.  Thus, his goal was to create an emergency, trigger this data download into the servers, and then walk away with all of the money (or delete it, sending America back to the Stone Age).
   
   Okay, I won’t get into all of the problems, but let me say this – if this guy was so brilliant, then his design has a flaw.  If you really were going to do this, you wouldn’t download all of the data into one location.  You would download it into two locations.  Remember, this is absolutely critical information and losing it would be disastrous.  Therefore, you’d have a backup.  That’s so obvious that a designer has to know that.  What you would probably do is download it to two separate (redundant) servers in the same data center, and then do the same thing in another geographically separate data center.  That way you have double-redundancy for a set of data that is so important.  Clearly, this bad guy can’t be that smart if he designed it to have one backup.  What a doofus.  No wonder he got fired.

I could probably name more problems, but this will do. But like I said, this movie isn’t real life, it’s entertainment.  It’s not supposed to be realistic.  And for what it was worth, it was a good ride.  I liked it.

Yippie-ay-yo-kay-yay!

The evolving MAAWG

MAAWG is an organization that started up in response to the spam problem.  Its official name is the Messaging Anti-Abuse Working Group, and they are meeting this week in Philadelphia to discuss all things abusive.  I didn’t go this time around, but maybe in the future I will secure my attendance.  DarkReading has an interesting article on the proceedings that you may wish to check out.  An excerpt:

---

"Email [abuse] will remain substantial," says Michael O'Reirdan, chairman of MAAWG and distinguished engineer in national engineering and technical operations at a major U.S. ISP. Even so, O'Reirdan says he'd like for MAAWG to change its name to more than a messaging title to better reflect the evolving threats to ISPs and their users. [tzink: emphasis mine]

Other MAAWG members, such as Cisco, note that malware distribution via email has become less of a threat in developed countries. "Email as a malware distribution [vector] is somewhat dead except in emerging economies," says Henry Stern, senior security researcher for Cisco's IronPort team. G-20 countries are now sending anywhere from 20 to 40 percent less spam this year than last, he says.

That's, in turn, pushing spamming botnets out of the U.S. to lesser-developed countries with emerging broadband infrastructures. "It's more lucrative for them to go outside the U.S. There's a migration away from old email spam here" and to other methods, such as attacks on social networks, for instance, says Patrick Peterson, a Cisco fellow.

---

Indeed, over the past year, the threat landscape has changed and shifted in various fashions.  The spam problem is not going away anytime soon.  People will continue to spam, ad nauseum, forever.  However, it is not the growth industry it once was.  I liken spam to the railroad industry.  Back in the 1800’s and 1900’s, railroads were the new and emerging transportation mechanism.  They were growing by leaps and bounds and revolutionized domestic trade (in the United States) and international trade (in Europe).  Trains could travel to places that boats could not.  Nowadays, we don’t really see a lot of railway expansion.  It’s an established industry.  There is certainly plenty of maintenance but there are other ways to get goods around – by automobile or by plane.  That being said, rails are not going away.  They are a very efficient distribution mechanism of transporting lots of goods, such as grain, steel, automobiles or passengers.  It is an entrenched part of our economy.  But it is not the growth industry of today.

In a similar way, spam is not a major growth industry.  It is harder for spam to get by filters and the spamming is done by more elite spammers.  That does not mean that cyber-abuse has gone away, however.  There are other attack vectors that have crept up over the past couple of years:

• Rogue antivirus
• Black search engine optimization (getting spammy webpages to the top of web queries)
• Hijacking of free web creation tools (like Blogspot or Live Spaces)
• Fast flux
• Social networking abuse
• Cyber riots in the form of DOS attacks against countries or services (like Twitter)

So you see, there’s a big chunk other than just spam.  Botnets are behind most of it, but they are a distribution vector for accomplish all of the above in addition to spamming.  To say it is only Messaging Anti-Abuse is too narrow in scope.  It is a natural progression to widen one’s view when the nature of the threat changes.

A couple of years ago, I attended the CEAS – the Conference on Email and Antispam.  They have since changed their name to the Collaboration, Electronic messaging, Anti-Abuse and Spam Conference (CEMAAS?).  It’s catching in other places, so why not MAAWG?  For every new communication medium, there will be someone who will attempt to take advantage of it and abuse it, and eventually organizations like MAAWG will have to figure out how to fix that one, too.  That’s simply the way it is.

What’s waledac up to these days?

Just for the fun of it, I decided to check some statistics on the waledac botnet.  I got the total number of distinct IPs sending us spam and broke them out by how much spam they were sending us, by country, for Oct 22, 2009.  Below are the results.

What’s interesting about this list of countries is the following:

1. China is nowhere on the top 10.  It’s not even on the top 20.  In fact, it comes in at #35.  For a country that has the second most amount of spams sent, by total volume and by total number of distinct IPs, it has proven itself to be fairly resilient against the waledac botnet.  To clarify that, I don’t mean that China is resistant so much as I mean that spammers/malware authors do not have as large a footprint in China as they do in the western world.
   
   
2. The one surprise entry on this list is #4, Saudi Arabia.  What?  Saudi Arabia is on this list?  What’s that all about?  Saudi Arabia is not one of the bigger spam problem countries that we see, but nearly 1/10 spams sent to us by waledac yesterday came from there.  I’m not sure what the deal is here but it certainly is unusual.

Things we can learn from Animaniacs

Does anyone remember that cartoon from the 1990’s, Animaniacs?

It was a pretty good cartoon for its short run.  One of the segments that they aired was called “Good Idea, Bad Idea”.  It was a short clip segment.  It would go something like this:

It’s time for another good idea, bad idea.  Good idea: giving a small child a balloon.  Bad idea: giving a small child a bunch of balloons (and the child then floats away).

It was a humorous segment.  And that brings me to advice that computer security experts give.  Good idea: using good password policies for all of the sites you visit on the web.  Bad idea: using different passwords for every site.

Why do I say this?  While we should always use good passwords (like letter/number combinations, nothing obvious like “123456” and “password”), it’s completely unrealistic to have different passwords for every site if you have a very wide reach on the web.  Consider myself:

• I have an online bank account from back in Canada
• I have another online bank account (which I opened when I moved to the United States)
• I have a third online bank account
• And I opened up a fourth online bank account!  In truth, I did this to get the free $100 for opening an account, but now that it’s open I think it’s kind of convenient to have since the bank is not local
• I have an online trading account
• I have an online retirement account from back in Canada
• I have an online retirement account when I moved to the United States
• I have a Facebook account
• I have a Twitter account
• I have Yahoo, Gmail and Hotmail accounts
• I have a login to my work computer
• I have a login to my Mac computer at home
• I have logins to two or three discussion boards which I participate in every once in a blue moon
• I have logins to a couple of websites (including this one) on which I write articles
• I have logins to a bunch of bill payment sites like electricity, rent and car insurance
• I have logins to online websites which I use to buy things

In total, I must have close to thirty different sites at which I login to.  How in the heck am I supposed to remember 30 different usernames and passwords?  On at least 1/3 of these sites, I have forgotten the password and I have to reset it nearly every single time I return to the site because I login maybe once a month.  It’s so frustrating! I know that using different passwords is good advice, but how realistic is it?  Humans cannot remember that many different combinations of things without resorting to some memory tricks.  Even then, it is still difficult.

There must be a better way.

Keeping track of botnets

A couple of months ago, I posted a one-day snapshot of how much spam we see from individual botnets.  I’ve been keeping track since July 29 on the biggest ones that have names, and only for IPs that get past our RBLs.  At the time of my first post, I thought that the stats wouldn’t really change much over time and that at any given time, they would be more or less the same.

That’s only partially true.

It turns out that rustock is the highest spamming botnet and is followed by bagle-cb, then cutwail (that surprises me about cutwail).  However, there is variation.  Rustock is the biggest sending botnet but only about half the time.  There is great variation amongst the others.  Below is a chart for the first month after I started tracking it, with the biggest spamming bot for that day highlighted in green.  It is tracking the total amount of mail marked as spam and divided up amongst all of them, expressed as a percentage of the total.

The number of “victories” for each botnet:

Rustock – 13
Bagle-cb – 10
Cutwail – 11

But below is a chart of the next month.  Rustock gets first place again, but there is more variation amongst the smaller botnets. Bagle and cutwail run out of gas while darkmailer and grum pick up the slack.  In reality, darkmailer is a bit of a darkhorse in all this because it tends to get overshadowed by the big three botnets and often comes in second (a lot like every sports team I ever cheer for – this should not be interpreted to mean that I am cheering for these botnets, I merely use the term as an analogy).

Number of “victories” for the winners:

Rustock – 15
Bagle-cb – 5
Cutwail – 4
Darkmailer – 3
Grum – 3
Mega-d - 1

The weird thing is that at first, whenever I would check the stats, I would dump everything into a text file, then use cat file.txt | grep rustock.  This was a quick way to see who’s spamming.  I would see that at the start of my tracking, rustock had large stats but after a while, it shrank down.  I thought that there was something wrong with my script to track this stuff even though nothing had changed.  I couldn’t figure it out.  But as it turns out, it appears that the worst of the botnets are cyclical in nature (based upon my two month data set).  I will continue to track this going forward to see who is the worst, and when rotations shift.

Interestingly, as an investor (who hasn’t put much money to work in months), I couldn’t help but think about how this relates to investing.  In the markets, sectors rotate.  Semi-conductors lead at some parts of the year, and six months later it is the pharmaceuticals.  These cycles are difficult to predict and the advice some give is to buy the index funds that tracks everything; since market timing is nearly impossible, you should purchase the index to ensure you catch the upswings on everything.

I was thinking, after viewing these stats, that botnet timing is nearly impossible to predict… other than rustock will be the worst about half the time.  Hmm, I guess that’s where the analogy ends.  Well, so much for that.

I don’t know what it is…

I don’t know what it is, but whenever I hear the name of the waledac botnet, I always think of Wario from the Super Mario Bros. series.  Something about both starting with the letters Wa, both being three syllables, both being bad guys, both using nefarious tactics to accomplish their goals…

I came across the following image today.  I think it pretty aptly describes the waledac botnets, malware, and the people behind them.

Dr. Wario/Waledac

Fooled today… almost

Today, I got a spam in my junk mail folder that nearly fooled me.  Below are the headers with some information removed to protect trade secrets:

Received: from VA3EHSMHS008.bigfish.com (unknown [10.7.14.235]) by
mail29-va3.bigfish.com (Postfix) with ESMTP id 0C2D9368054 for
<munged@microsoft.com>; Fri, 16 Oct 2009 23:46:34 +0000 (UTC)
Received: from waledac (110.46.151.204) by VA3EHSMHS008.bigfish.com
(10.7.99.18) with Microsoft SMTP Server id 14.0.482.32; Fri, 16 Oct 2009
23:46:33 +0000
Received: (qmail 28488 invoked from network); Sat, 17 Oct 2009 08:39:21 +0900
Received: from unknown (HELO tlvftz) (231.179.161.44) by waledac with SMTP;
Sat, 17 Oct 2009 08:39:21 +0900
Message-ID: <002701ca4eb9$e2b83600$e7b3a12c@LocalHosttlvftz>
From: Gertie Lockhart <munged@hwns.com.au>
To: <munged@microsoft.com>
Subject: What's she doing now
Date: Sat, 17 Oct 2009 08:39:21 +0900
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset="iso-8859-1";
reply-type=original
Content-Transfer-Encoding: quoted-printable

This is a spam from the waledac botnet.  From time to time, I like to inspect my spam to see who is spamming me and to verify whether or not I can tell, simply by looking at it, what spam is spamming me.  I was quite proud of our spam filtering engine because this IP isn’t even on any DNSBLs anywhere at the time of this writing.

Going from the first IP, 110.46.151.204, this is a mail host located in South Korea infected with waledac.  But nearly fooled me is the parts after it:

Received: (qmail 28488 invoked from network); Sat, 17 Oct 2009 08:39:21 +0900
Received: from unknown (HELO tlvftz) (231.179.161.44) by waledac with SMTP;
Sat, 17 Oct 2009 08:39:21 +0900

This makes it look like a mail server running qmail is also infected with waledac.  The first Received header is an actual (possible) qmail header.  The second is the exact type of format that qmail has – Received: from <ptr record> (HELO <helo>) (<IP>) by <string> with <SMTP or ESMTP>.  This type of header is in older versions of qmail, the newer ones have a different header.  What would have made this so unusual is that this would have been an example of an infected system running Linux (or Unix), not Windows.  It would have been clear evidence of a Linux botnet.

However, there are two problems with this:

1. You cannot trust headers after the Received header.
   
   
2. What makes this one fake is the IP 231.179.161.44.  This is a reserved multicast IP address (located in Algeria).  From what I understand, IPs in this IP address range cannot be used to send mail, or traffic, in this manner (I don’t understand multicast very well despite looking it up online and reading several documents).  According to RFC 3171, it is reserved but it doesn’t say for what.  This IP cannot be correct because it is reserved for multicast purposes.
   
   This IP is allocated to the African Internet Numbers Register, they are assigned 213.179.160.0/19, ASN 16214.

If it weren’t for that, I would have trusted that this is an actual infected Linux machine because it is unlikely to me that a spammer would spend that much time forging a pair of Received headers to look exactly like a popular MTA that runs only on Linux, and the Received headers have time stamps on them that are chronologically correct (ie, 08:39:21 +0900 is 7 minutes before 23:46:33 +0000).  Why would you spoof qmail?  I don’t understand the motivation, but it certainly looks like that is what happened.

If it were true, it would imply that waledac is not only used to spam, but it also installs its own MTA.  That would be quite an advancement.  At the very least, it is impersonating an actual one.

Best practices for sending outbound mail

One of the questions that I am frequently asked is if we get a sudden burst of outbound mail from a customer using us to send outbound, will we throttle their mail? 

Throttling is the process of slowing down outbound mail such that a sending organization can only send a certain amount of messages in a certain timeframe.  For example, if the rate limit is 2500 messages per hour and the organization wants to send 25,000, throttling would slow down their connection speed and it would take them 10 hours to send out their entire email campaign.  In a similar manner, I am frequently asked if we will block their mail if we detect a sudden burst in traffic due to an email campaign, the idea being that spammers are known for bursty mail behavior and this technique could potentially make them look like spammers.

In my department, throttling is not a mechanism that we have found to have extensive value.  However, elsewhere within Microsoft, particularly Hotmail and Exchange, throttling is used quite a bit.  By contrast, we are more content driven – as long as your email isn’t spam, then we will relay it.  The idea is that we try very hard not to interfere with legitimate mail flow.  If it is illegitimate mail, then we will interfere with it.  We will take action very quickly, as illustrated by my recent blog series.

Customers than ask me for a set of best practices when sending mail.  How can they ensure that mail they send, in bulk, arrives at the destination?  They are looking for advice for when they send from their own servers, or through our own servers.  Here is what I tell them.

1. Ensure that you have proper SPF records set up in DNS.  SPF records are a mechanism for ensuring that mail coming from a domain really is coming from that domain.  In other words, it is an anti-spoofing mechanism, and helps in the delivery of mail in some cases because it allows the receiver to verify the sender and build up a reputation on it.

   - Open SPF wizard
   - Microsoft’s SenderID wizard

   I’ll have a bit more to say on delivery to Hotmail below.
   
   
2. If they are sending mail in bulk (ie, emailing lots of receivers), then it is probably in newsletter format.  They should have a way of unsubscribing at the bottom of the email.  It would look something like the following:

   This email was sent to some.guy@microsoft.com by sender@example.com.
   Update Profile/Email Address | Instant removal with SafeUnsubscribe™ | Privacy Policy

   Some mailers require you to send an email to a certain email alias with “Unsubscribe” in the subject.  I don’t like those very much, I much prefer to click on the link.  But if you do it this way, at least have the courtesy to have it as a link and when it is clicked on, all the required fields are pre-populated.
   

3. Double opt-in is a great idea.  Here’s how it works: you know how sometimes you download software and there is a little checkbox saying “Yes, please sign me up for your annoying offers” and it’s checked by default, and often in small text?  That’s built on the assumption that people simply don’t notice that it is checked and it’s a free avenue for mailers to harvest your email address.  In the world of email security, it is considered a grey marketing technique.
   
   Double opt-in is different.  The check box for things like that are unchecked by default, so the user has to select it.  Secondly, once they submit the form, an email is sent to the end user with a link saying “Please click on the link below to verify that you really want to hear from us.”  That forces the user to opt into announcements but it gives the mailer a good reputation.
   

4. The sender of mail should have forward-confirmed DNS.  That means that if the sender is asdf@aselasdf.com, and aselasdf.com does not actually exist or resolve in DNS, they will have problems delivering to a lot of places since many spam filters consider that in their weighting.  For example, if the sending domain has no A-record in DNS, nor an MX record, a certain large ISP will reject the mail or at least throttle mail from that IP.  It is a common spamming tactic to fill in the sending email address with non-existent domains.
   

5. Similarly, the sending IP should have a reverse DNS entry.  For example, 12.129.199.61 is mail-haw.global.frontbridge.com.  A lot of senders do not have a reverse DNS entry for the IP.  Adding this in DNS makes it easier for spam filters to know who the mail is coming from.  It is common for spammers to send from IPs with no reverse DNS entries (ie, PTR records).
   

6. This is pretty obvious, but the subject line of the message should be what the message is about.  The sender of the message should reflect who is sending the message.  If you are sending out mail for the Shopper’s Handbag Company, here is what you should do:

   From: marketing@shoppershandbag.com
   Subject: New updated catalog for the Christmas season!

   Here is what you should not do:

   From: frank@gmail.com
   Subject: Catalogs

   The easier you make it for people to know who you are and what you are doing, the less you will annoy spam filters around the Internet.
   
   
7. Process bounces and remove non-existent email aliases from your lists.  Email addresses change over time, and people sometimes discard them.  Companies delete non-existent accounts and when you send mail to them, they will bounce it back to you.
   
   If you get an email bounce from somebody you sent mail to, remove it from your email list.  Not doing so annoys spam filters around the Internet.  Yes, I understand the bouncing email server may be sending you a form of backscatter mail, but you’re the one who originated the mail; deal with it.  Send to people’s email aliases that exist.
   

8. Windows Live Mail (Hotmail) has a program set up called Smart Network Data Services, you can check out the link here.  Basically, they allow senders to sign up and if they agree to be good senders, it improves their deliverability to Hotmail.  It also involves checking complaints sent by end users.  I don’t know that much about this program, but I hear that it helps some senders deliver mail to Hotmail.
   
   Hotmail can sometimes be one of the trickier ones to deliver to.  Steps 1 (especially), 4, and 5 really aid in delivery to Hotmail.  It also makes it much easier to escalate to them if you have everything set up properly, and it also improves delivery to the rest of the Internet.
   

9. Alternatively, if they were to do steps 1-5, they could consider talking to the folks at ReturnPath.  ReturnPath is a company that checks the reputation of bulk senders to ensure that they are good senders of email; they enforce compliance.  I know some guys there and I know that they check for items 1-5.  They do charge to get on that list, however, once a sender does get on and takes their sending reputation seriously, they will have a mcuh easier time delivery to Gmail, Yahoo and Hotmail.
   
   Disclaimer: Microsoft does not a formal relationship with ReturnPath, and neither do I.  I point this out based solely upon my own personal observations of how the email industry works, and I am familiar with the work that ReturnPath does.
   

10. There are companies out there that provide bulk mailing services and are familiar with sender’s best practices.  Two that I know of are ExactTarget and ConstantContact.  Both of these two are pretty sharp and on the ball, and can be of assistance when you need to keep in contact with lots of people.
    
    Once again, neither myself nor Microsoft has a formal relationship with these two companies, I point this out based solely upon my own personal experience and my familiarity with their work.

Those are the big ones I can think of.  I’m sure email marketing folks everywhere would more than likely agree with what I have said above, and may even have a thing or two to add.

I think that the good thing about this list is that these are best practices for delivering mail through all spam filters, not just ours.  They are paraphrased recommendations of some of the larger organizations that deal with email.  They are not special tricks that spammers could use to evade filters because it would de-anonymize them.  They are techniques that say “Here’s the proper way to send mail – make sure you can be authenticated, and don’t be deceptive.”  Good guys do that, bad guys do not.

How to reclaim your sender reputation, part 10 - Results

Results

Forefront Online (ie, us) has come a long way in reclaiming its outbound reputation. The question now is this – has it worked? I will report on some anecdotal evidence.

The Good

To determine whether or not we have gotten better, I prefer to check 3^rd party sources. While we may think that we are doing a better job, ultimately, the recipients of our mail will be validating this. In September 2008, I attended a conference where one of the other participants reported on reputation hijacking, and how it affected the big 4 web mail providers – Microsoft, AOL, Google and Yahoo, or MAGY for short. Microsoft was not the worst offender, but FOSE was the worst offender of outbound spam within Microsoft, and was the single largest source of reputation-hijacked mail of the various subgroups within MAGY. We got to work implementing a series of changes. Five and a half months later, in February 2009, we won the “award” for the most improved outbound IPs; in fact, we were one of the cleanest of the entire MAGY source IPs.

Another monitor of outbound sender reputation is ReturnPath. In September 2008, two of our data center IPs were rated 55 on a scale of 1-100. In other words, we were bad. In February 2009, two of our IPs were rated 90 and the other two were rated 95. We had improved dramatically. Since then, we have bounced around a little bit as we deal with the occasional spam outbreak, but we never drop back down to the 50s or 60s.

The Not-So-Good

Unfortunately, fighting outbound spam is a non-stop battle. We have new outbreaks of outbound spam at least once per week. These outbreaks have gotten larger and larger over time, not smaller. We have had several large web mail providers (not MAGY) block our outbound IPs even though we have outbound spam filtering in place. This suggests that new outbreaks do get past our spam filters.

Furthermore, our reputation according to ReturnPath has dropped back to down to the 70’s. While it is not as bad as it was late last year, it is not as good as it was earlier this year. Compromised customers are still costing us our reputation.

Summary

To summarize, cleaning up our reputation has worked. It is not yet perfect, but it substantially better than it was 18 months ago. We can react quicker to spam outbreaks and we have technology to avoid impacting our service when we do have problems. Our spam detection algorithms have improved over time and we are confident that as time passes, we will get even better at keeping our outbound pool clean.

The key pillars of outbound spam are three-fold:

1. Make sure that spammers are isolated from people with good behavior
2. Cut off the offending customers
3. Frequent monitoring requires a lot of granularity

It is not easy keep your reputation clean, but it is worth the time and effort put into it.

The end… for now.