The Journey Begins Part 3 - Abuse, Automation, Management, and Aquisition
… Continued from part two (part one):
As we last left my chronicle of my experiences in Web Hosting, I was the sole systems administrator at Virtual Servers. The systems were running high on load average, which isn't surprising given what we were running on the hardware that was available back then. I quickly realized that the only way to begin to address the problems in any meaningful way was to automate fixes and avoid short-term "band-aid" fixes. This is easier said than done however when you're the only sysadmin; To be blunt, time is of the essence and you don't have much time to be proactive when your customers are constantly contacting the support team to complain of slow service due to high server loads. I was able to convince my manager that I needed help, and he allowed me to hire someone to share my workload.
The first person I hired was a guy named Steve, who has since become a great friend. His primary job was to handle all of the abuse complaints. He knew web design very well but was just beginning as a systems administrator. I trained him on how to react to high load issues; The reason this was tied to the Abuse Desk position was that in many cases, high loads on the servers were caused by people hosting things that were against our AUP, such as porn or warez. The other cases were caused by customers whose web sites had outgrown the capacity of their shared hosting plan. Steve would contact these customers and tell them that there site was causing the server to fail and that he'd have to shut down their web site to keep the rest of our customers happy; He also gave them the alternative to upgrade to a plan that had more resources allocated to accommodate the needs of their growing Web site. It was mainly through efforts like this that we built our dedicated hosting business.
Now that I had a little more time, I began to automate everything I could. First, in order to help Steve, I built a number of scripts that he could use to track down users that were causing server performance issues. I also helped to beef up our monitoring and paging system (we called the system Zookeeper) so that it was better capable of paging a team of people rather than just me and my boss. I also integrated some of my data gathering scripts into the web interface of our monitoring solution so that when you saw a server alert come up, you could click a link to get more detailed information to track down the issues.
Through all of this, the Perl scripting language was my greatest friend. When it comes to coding a complex application such as a server, end-user app, or what have you you'd probably want to look at a language like C/C++ (or similar), but there is nothing better than Perl when it comes to system administration, especially on Unix/Linux/GNU systems. It's funny now that I look back at some of my earlier perl scripts - they were hastily written and didn't use the best programming practices. However, over the years I would grow to know Perl like the back of my hand.
By the Fall of 1999, I had hired another system administrator to my team (Phil), and was in the process of looking to hire a full-time Perl scripter to help with tool creation for systems administration automation. In December, It was announced that Virtual Servers (AKA Lightrealm Communications) would be acquired by Micron PC. Micron PC at the time was struggling to compete with the likes of HP, Compaq, and Dell and thought to start an Internet Hosting business in order to take its company in a new direction.
They acquired two other web-focused companies along with Lightrealm with the intent of forming them into a larger player in the web hosting space. Micron Internet Services, acquired from their parent company Micron Electronics, was mainly an Internet Service Provider selling dialup and broadband Internet services in the Boise, ID area. Hostpro (AKA NetLimited) was a hosting company based out of Los Angeles, CA that had an array of hosting products and services ranging from high-end customized dedicated hosting solutions to shared hosting. Our strength at Lightrealm was mainly our shared hosting business. While we had some dedicated hosting and colo customers, the primary focus had always been on the shared hosting business.
The merger brought on a new era. I went from working for a small private startup company with a very personal feel to working for a small part of a much larger publicly traded company. At the time, MicronPC was the large revenue player and we felt completely segmented from them. It was decided that the new brand for the combined companies would be HostPro. It would take a few years for us to begin to truly integrate our cultures and think of ourselves as one company. It is strange to look back now and remember how weird it was going from a company of 100 employees to a company that had well over 1000. In retrospect, this was nothing compared to working for a company like Microsoft, which has almost 60,000 worldwide.
One of the benefits of the acquisition was that we had a much larger budget with which to grow our company. In fact, one of the main reasons for the acquisition was that Lightrealm had grown so much that we'd hit a ceiling of what we could accomplish without some major capital. This capital was what Micron PC brought. Now instead of building our own custom hardware built in custom cases and using PC hardware, we began to deploy our servers onto enterprise class servers (still Intel based, but much higher quality). We were able to build out a new data center and office space, both of which were badly needed. Some of us still laugh with fondness when we remember the old days of working closets for a lack of available floor space in our overcrowded old office space. Having a new employee report to work was always exciting, as it meant trying to figure out if there was a spare cabinet or closet that might possibly be converted into a work area. Now that HostPro was formed with the backing from Micron PC, those problems were behind us.
To Be Continued...
