Deven Kampenhout's Tech Blog : About the Author

Microsoft PDC 2008

devenkamp — Mon, 27 Oct 2008 20:47:43 GMT

I'm attending the Microsoft Professional Developer Conference this week. If you're at the PDC or in the LA area, give me a shout out!

The Journey Begins Part 3 - Abuse, Automation, Management, and Aquisition

devenkamp — Sat, 09 Apr 2005 18:55:00 GMT

… Continued from part two (part one):

As we last left my chronicle of my experiences in Web Hosting, I was the sole systems administrator at Virtual Servers. The systems were running high on load average, which isn't surprising given what we were running on the hardware that was available back then. I quickly realized that the only way to begin to address the problems in any meaningful way was to automate fixes and avoid short-term "band-aid" fixes. This is easier said than done however when you're the only sysadmin; To be blunt, time is of the essence and you don't have much time to be proactive when your customers are constantly contacting the support team to complain of slow service due to high server loads. I was able to convince my manager that I needed help, and he allowed me to hire someone to share my workload.

The first person I hired was a guy named Steve, who has since become a great friend. His primary job was to handle all of the abuse complaints. He knew web design very well but was just beginning as a systems administrator. I trained him on how to react to high load issues; The reason this was tied to the Abuse Desk position was that in many cases, high loads on the servers were caused by people hosting things that were against our AUP, such as porn or warez. The other cases were caused by customers whose web sites had outgrown the capacity of their shared hosting plan. Steve would contact these customers and tell them that there site was causing the server to fail and that he'd have to shut down their web site to keep the rest of our customers happy; He also gave them the alternative to upgrade to a plan that had more resources allocated to accommodate the needs of their growing Web site. It was mainly through efforts like this that we built our dedicated hosting business.

Now that I had a little more time, I began to automate everything I could. First, in order to help Steve, I built a number of scripts that he could use to track down users that were causing server performance issues. I also helped to beef up our monitoring and paging system (we called the system Zookeeper) so that it was better capable of paging a team of people rather than just me and my boss. I also integrated some of my data gathering scripts into the web interface of our monitoring solution so that when you saw a server alert come up, you could click a link to get more detailed information to track down the issues.

Through all of this, the Perl scripting language was my greatest friend. When it comes to coding a complex application such as a server, end-user app, or what have you you'd probably want to look at a language like C/C++ (or similar), but there is nothing better than Perl when it comes to system administration, especially on Unix/Linux/GNU systems. It's funny now that I look back at some of my earlier perl scripts - they were hastily written and didn't use the best programming practices. However, over the years I would grow to know Perl like the back of my hand.

By the Fall of 1999, I had hired another system administrator to my team (Phil), and was in the process of looking to hire a full-time Perl scripter to help with tool creation for systems administration automation. In December, It was announced that Virtual Servers (AKA Lightrealm Communications) would be acquired by Micron PC. Micron PC at the time was struggling to compete with the likes of HP, Compaq, and Dell and thought to start an Internet Hosting business in order to take its company in a new direction.

They acquired two other web-focused companies along with Lightrealm with the intent of forming them into a larger player in the web hosting space. Micron Internet Services, acquired from their parent company Micron Electronics, was mainly an Internet Service Provider selling dialup and broadband Internet services in the Boise, ID area. Hostpro (AKA NetLimited) was a hosting company based out of Los Angeles, CA that had an array of hosting products and services ranging from high-end customized dedicated hosting solutions to shared hosting. Our strength at Lightrealm was mainly our shared hosting business. While we had some dedicated hosting and colo customers, the primary focus had always been on the shared hosting business.

The merger brought on a new era. I went from working for a small private startup company with a very personal feel to working for a small part of a much larger publicly traded company. At the time, MicronPC was the large revenue player and we felt completely segmented from them. It was decided that the new brand for the combined companies would be HostPro. It would take a few years for us to begin to truly integrate our cultures and think of ourselves as one company. It is strange to look back now and remember how weird it was going from a company of 100 employees to a company that had well over 1000. In retrospect, this was nothing compared to working for a company like Microsoft, which has almost 60,000 worldwide.

One of the benefits of the acquisition was that we had a much larger budget with which to grow our company. In fact, one of the main reasons for the acquisition was that Lightrealm had grown so much that we'd hit a ceiling of what we could accomplish without some major capital. This capital was what Micron PC brought. Now instead of building our own custom hardware built in custom cases and using PC hardware, we began to deploy our servers onto enterprise class servers (still Intel based, but much higher quality). We were able to build out a new data center and office space, both of which were badly needed. Some of us still laugh with fondness when we remember the old days of working closets for a lack of available floor space in our overcrowded old office space. Having a new employee report to work was always exciting, as it meant trying to figure out if there was a spare cabinet or closet that might possibly be converted into a work area. Now that HostPro was formed with the backing from Micron PC, those problems were behind us.

To Be Continued...

Almost There

devenkamp — Mon, 14 Mar 2005 22:26:00 GMT

Last week I was in Atlanta to pack up my families stuff and close on my house. Everything went relatively smooth, but I'll hold my breath until everything is delivered out of storage. The good news is that my house closed, so I'm officially homeless now. I've got a new home lined up that the family and I will be moving into on April 16th if all goes well. It sure will be nice to finally have everything settled. My wife Alisa really likes the new house, which is located in northern Ft. Worth. I like it because it is completely wired, and you can get up to 5 Mbps internet access for like $35/month there :)

The funny thing is that since we moved into temporary housing 3 weeks ago, I've spent more time on the road than I have living in Texas. So, my wife and kids have lived in Texas three times as long as I have. I'm excited to get moved into my new house so I can have my home office again. Granted, it's not so bad where I live now, right across the street from the Microsoft campus in Las Colenas, but I like it even more when all I have to do is walk down the hall to get to my office :)

Back from Vegas

devenkamp — Sat, 19 Feb 2005 05:00:00 GMT

I just got back from being in Las Vegas for a few days for the CSNA offsite meeting. Overall, I enjoyed the meetings as it helped me get a better understanding of the organization I'm working in, and to meet all of the members of my team in person for the first time. I have decided that I don't really like Las Vegas that much. It probably would be a fun place to visit for a single day, but after being there for 4 days, it gets old. The Venitian resort was fun, but smelled completely like cigarettes, which began to really irritate my lungs, eyes, and skin after a while.

On a more technical note, RedHat released RedHat Enterprise Linux 4 (Link is PDF) this past monday. I haven't had much time to read up on it, but one thing I know is that it utilizes the 2.6 linux kernel. I'm looking forward to playing around with it to see how it is different than version EL 3.

On another personal note, I'll be driving to Dallas with my family to relocate there, so my postings may be more limited this week.I'm really looking forward to getting this whole move behind me.

The Journey Begins Part 2 - The Early Days of Hosting

devenkamp — Wed, 12 Jan 2005 06:21:00 GMT

… Continued from part one:

In the beginning of 1999, the small hosting company I worked for (Virtual Servers, LLC.) realized that they were loosing business because they didn’t have a Windows based hosting product offering. By this time we were using NT 4.0 as workstations in the office, but we didn’t have any real expertise on IIS or running NT as a web server. They hired a New Yorker named “Vinnie” to put together our Windows based hosting solution. He did as good a job as anyone could do. The goal was to make it a similar style “virtual” product offering as our UNIX product. Unfortunately, true virtualization on Windows was impossible with the architecture at the time. In order to allow customers the “feel” of having more control of their Web server, we allowed customers to register their own dll’s. As you might imagine, this cause all kinds of troubles for our Windows Sysadmins. I also remember the company having all kinds of hardware troubles (only on the Windows side), as we had been used to building up our own servers to save on hardware costs. Eventually, we had to invest in some Dell servers and the major hardware problems went away.

During this time of our Windows hosting product beginning, I was busy creating the Unix systems administration team. At first, I was the only sysadmin. I had to wear all kinds of hats in that role: security, abuse, system administration, backup/recovery, and top tier support. All of our servers were hosted in a small NOC located in one of the Westin office towers in downtown Seattle. The NOC was owned and operated by our sister company, Lightrealm Communications. I wouldn’t quite call it a data center, as it couldn’t have been more than 1500 square feet in total size. Our office, where all personnel worked, was located on the East Side of Seattle in the city of Kirkland. As such, we had a “dark” data center.

We had built a custom monitoring system that would automatically page me if one of the servers went down. Each server’s power was hooked through a device called an “e-commander”, which allowed one to telnet in and power-cycle a box remotely if the box stopped responding and needed a hard reboot. This worked decently *most* of the time. Unfortunately, the UFS filesystem which was utilized by BSD/OS wasn’t the most resilient to hard booting. After coming up after a hard reboot, it would attempt to automatically repair the corrupted inodes, but every once in a while it wouldn’t be able to repair it automatically. If this happened, it meant that I had to make a trip downtown to give it personal attention. I don’t think I can count on one hand the number of times I had to make a trip downtown during business hours, but the systems had an uncanny knack of going down between 1-5 AM. The first few times I had to take a trip downtown in the middle of the night to fix a server, I thought it was pretty exciting. It didn’t take long to become quite a chore, however. I had been married for less than a year, and I can still remember my wife getting very angry at my pager.

Besides early morning file system recoveries, the biggest problem with the systems was performance. Since we allowed our customers a great deal of freedom on how they configured their servers, they were a very popular hosting solution for people who wanted to push the limits. Due to the backend system which created the “virtual” server environment, the systems tended to use more resources than a standalone hosting solution would have. The problems ranged from customers running bad “runaway” scripts, to sites which just got more traffic than the system could support. During the night, the systems tended to run pretty smoothly (when they weren’t crashing with filesystem problems), but during the day the loads could get pretty high. Any time the load averages rose above 7, the system performance would degrade to the point where customers would start to complain. I had to come up with a number of creative ways to automate managing the growing number of systems…

[To be continued] part 3

A little housekeeping

devenkamp — Mon, 10 Jan 2005 20:10:00 GMT

I guess I got a little more attention than I expected when I posted my "Linux is no longer free" post. In expressing my opinion about how the major linux distributions which target enterprises are now charging premiums for usage/support of their distributions, I drew out the trolls, along with some good discussion. Thanks to all who had something constructive to say (even if you disagreed with me). As for the trolls, I've posted my rules about how to keep your posts from being moderated.

The Journey Begins

devenkamp — Fri, 07 Jan 2005 00:34:00 GMT

With this post, a new blog is born. Keep reading, as this isn't your typical blog by your typical Microsoft employee. My name is Deven Kampenhout. I'm not new to blogging, but you probably won't find my name on other blogs, as I've posted as an anonymous alias in the past. My blog wasn't only technically oriented, but was more of a creative expression of whatever was on my mind. This blog will be focused primarily on technology, especially about the battle between Microsoft and Open Source in the Hosting industry. This blog may be unique in that it will log my journey as I transition as a UNIX/Linux/Open Source technologist into a proponent of Microsoft technologies, software, and solutions. The rest of this post will be a history of how my life got to this point, so if you’re interested in my background, read on.

This week, I started working for Microsoft in the Communication Sector as a Hosting Evangelist. Professionally, I've been in the web hosting business for almost 7 years. My first internet experiences had me working on HPUX UNIX systems at the College of Engineering labs atBrighamYoungUniversity in 1996. I was captivated by the Internet and found myself spending all-nighters in the 24 hour lab figuring out how the Internet worked. Before too long, I had jobs working for the University building Web sites, creating content, and eventually building a rudimentary E-Commerce solution for the University book store.

In the summer of 1997, I went up to the Seattle area to begin working in an Internship. While I was there, I helped the small electrical engineering firm (ATD Northwest) establish its Web presence. I set up their web site with a hosting company named Virtual Servers, LLC. That same summer, I met my future wife. I returned to Utah to finish another Semester of school, returning to Seattle in the spring of 1998 to get married. While I was up there for the summer, I was fortunate to land a "summer" job with Virtual Servers, the same company that provided the web hosting services for ATD Northwest. I kept working through the summer and finally came to the point where I had a difficult decision to make: continue my exciting and promising career with “VServers”, or go back to school to complete my college degree. After much struggle over the decision, I decided that I was learning a great deal more at my new job than I was in college.

Virtual Servers focused on creating hosted web solutions for small to medium businesses. What differentiated this company from the rest of the fledgling hosting industry was our shared hosting product offering. Traditionally, you could either spend large monthly sums for your own “dedicated” server, or spend a fraction of the cost and host your web content on a server that shared other web sites. The disadvantage of “shared” hosting was a loss of flexibility and manageability. With a dedicated server, you could have full administrative access and customize your configuration to support your content. However, on the shared servers, most solutions gave very limited ability to add custom configurations, customize email and ftp account settings, etc. “VServers” came up with a hybrid solution, which created virtual server solutions which allowed each different account hosted on a shared system to be managed and configured as if it were a dedicated server. Initially, we only sold shared “VServer” solutions, but eventually some of our customers grew out of the resources that a shared system could offer, so we developed the “VServer Dedicated”. It was one of the first of a new type of hosting platform that would eventually be known as “managed dedicated solutions”.

At first, all of the VServer products were built on the “UNIX-like” BSD/OS operating system, distributed by BSDI (Berkley Software Design Inc.) At the time, Linux (RedHat) was still in its infancy (i.e. not very reliable yet), and Windows NT (IIS) was just beginning to pick up popularity in the infant web hosting industry. Before too long, it was apparent that without a Windows NT-based product, we were loosing business…

[ To be continued… ] (Part 2) (Part 3)