MT History

From The Network People, Inc. - Wiki
Jump to: navigation, search

The hosting company I was working for (Lightrealm (which became HostPro (which became Interland (which became Web.com)))) had hundreds of boxes running sendmail. It has become a maintenance nightmare. Clients were hosted on a shared virtual server. The biggest problems were mail storms (joe job attack, forged return address, etc) which would DoS the entire machine, taking down the clients web site, email, database, along with all the other customers on that box. There would be so much load that as soon as sendmail was enabled, the load would spike and the system would become unresponsive. The only solution was to disable sendmail for a few days, until the storm subsided.


In late 2000 I was tasked to fix it. I determined the right solution was to get the mail off those boxes. To do that, I needed a mail solution capable of hosting a 100,000 email accounts. There wasn't much in the way of virtual hosted mail solutions available. I had experience converting mail systems to qmail to resolve security and scalability issues with sendmail so I was looking for a virtualized qmail solution.


I did a bunch of testing and benchmarking and then built a clustered qmail system using a heavily modified version of vpopmail. Soon after a client server got DoS'ed off the network so we switched its MX to my cluster which very gracefully handled processing a mail storm of several hundred messages per second for the next couple days. That built enough confidence that our systems admin team consented to migrating all the many (and widely varied) mail products onto my cluster. It rapidly scaled from one customer to tens of thousands of email accounts.


Mail-Toaster-v1.jpg


I published much of the instructions for building a Mail Toaster on my personal web site. Scores of people found it quite helpful as there was very little published documentation on getting all the mail system components to play nicely together as one cohesive system. At the behest of many very satisfied mail admins, I created the toaster mailing list and greatly enhanced the documentation for the growing user community.


When the hosting company was acquired and subsequently re-located to Atlanta, GA, Mail::Toaster 2.0 was born as I rebuilt the entire cluster on a new system in Atlanta. In Atlanta, we consolidated all the email accounts from three 1.0 version Mail::Toaster clusters (two in Seattle, one in Los Angeles) to one much larger cluster in Atlanta. We used some very clever techniques to migrate over 40,000 email accounts across the country to the new cluster in a one hour maintenance window. We also migrated many email accounts from other email systems to the Mail::Toaster.


As the toaster community grew, many features were added so that the toaster would work in more diverse environments. Rather than packing the toaster pages with volumes of "if you want this, do this" style instructions, nearly all the decisions about how your mail system is built were moved into two centralized configuration files (toaster-watcher.conf and toaster.conf). In this way, the toaster owner specifies how he wants it set up, and the build and maintenance scripts "make it so."