Carl Hutzler’s Blog

Hello everyone! I decided to start a blog because some other very nice people at AOL threatened encouraged me to do so. :-)

I have been with AOL for nearly 9 years now. I started out as a project manager in 1997 and worked on the AIM instant messaging system in our Operations team. I had the privledge of working along side some of the real pioneers of that technology (David Lippke, Barry Appelman, Colin Steele and others).

My stint with AIM was fairly short as one of my peer PjM’s who worked on email projects quit! All of sudden my manager, Scott Gries, was asking who wanted to work on Mail. Well it didn’t take me long to figure out my answer and its been AOL Mail ever since!

My first project was managing the implementation of a new architecture for AOL Mail. A lot of people don’t realize this, but the entire AOL system ran on Stratus Computer hardware in the beginning. As AOL’s mail system grew, we added more Stratus modules to the “ring network”. But there came a point when the machines were no longer getting fast enough and we had maxed out the #modules per ring. Thankfully that moment came within 3-4 months of the new architecture for mail being ready for launch.

Enter Tandem Mailboxes.

So we move a lot of mail from Stratus to another fault tolerant platform, Tandem. The application was totally redesigned and we built a very fault tolerant, very scalable solution. To this day AOL runs on this platform which has taken us from 10 million emails a day to over 700 million (at peak). We no longer have the horizontal scaling issues with Stratus and we support well over 100M mailboxes (and growing).

I managed the Operations team responsible for the new Tandem Mailbox system as well as the Stratus Operations team for several years. Both were a huge learning experience for me as I was somewhat new to operations especially on a scale of AOL and something as critical as email. I learned to get up at 2am and help the team service equipment. I slept with my pager. I found out what it was to be on a conference call at 2am when Matt Korn (VP of Operations) could not read his mail (why was it always Matt??).

The mail system kept growing and scaling. The team did well. The application did well, but there was a problem. The growth in mail did not exactly grow linearly with the growth in membership. In 2001 the budget for Email alone drawfed the budget for most other teams in Operations. Our VP at the time, Terry Laber, took the opportunity at one of his all hands to refer to the Mail Team as “budget pigs” in a cute way. Well this certainly got our attention and the attention of our director, Brian Sullivan.

Later that week, donning pigs noses we found at a local costune store, we sat down with Terry and mapped out where the budget was coming from. We decided on some changes to the architecture which needed help from our development team, but the single most important thing we decided was that we had a spam problem.

Shortly after that meeting, the world decided that the world had a spam problem and that everyone using email was fed-up. Our CEO, Jon Miller, put fighting spam on the company’s 2002 goals. Joe Barrett (our new VP of Operations) was enlisted as commander of the AntiSpam Hot Team. For some reason they picked me to form a new AntiSpam team in Operations.

More soon on what came next (if anyone cares :-) )
Signing off for now.

-Carl