PatHelland's WebLog

Leaving Microsoft and Moving to San Francisco

Pat Helland — Fri, 30 Sep 2011 15:03:41 GMT

Hello, Friends!

Lots has happened since I last blogged. For almost two years, I've worked on Cosmos, some of the plumbing for Bing. It stores hundreds of petabytes of data on tens of thousands of computers. Large scale batch processing using Dryad with a high-level language called SCOPE on top of it. Working on this team (with some amazing colleagues and friends) has been one of the highlights of my career.

Last June, I was honored to be able to write a paper for the Communications of the ACM. It is called "If You Have Too Much Data, then 'Good Enough' Is Good Enough" and can be found here: http://cacm.acm.org/magazines/2011/6/108666-if-you-have-too-much-data-then-good-enough-is-good-enough/abstract. If was quite an honor and a lot of fun.

Now for some big news, today is my last day at Microsoft after over 15 years (that's 17-1/2 minus the 2 years at Amazon). My wife, Lisa, and I have decided to move to downtown San Franciso. We will be only about 30 minutes drive from the grandkids, my daughter, and son-in-law. We will be about an hour from two of my brothers. Since my father passed away in April, 2010, there has not really been a close family tie to keep us in Seattle. We are really longing to be down where we can see the grandkids.

While I will miss my wonderful colleagues on the Cosmos team, it is time to search for something new. I don't have a new job lined up and plan to move to SF, help unpack, and attend the bi-annual High Performance Transaction Systems workshop (www.hpts.ws) in late October. HPTS has been a big part of my life since 1985. I will see my friends who work in scalable computing and look around to see what opportunities are available in the Bay Area. I know there's lots going on and I am looking forward to returning to California after my 17+ years in the Seattle area.

It is my hope to get a new website and blog going at www.pathelland.com but there's nothing there yet. It's amazing how busy you can be working at Microsoft and preparing to move to California!

If someone wants to contact me, my personal email is phelland0215@hotmail.com.

Thank you all for your interest and kindness through the years... I would really like to offer more thoughts and ideas moving forward but today I am off to my last day of work and my exit interview. You'll hear more from me soon!

All my best,

Pat

The Importance of Teams in Software Engineering

Pat Helland — Thu, 13 May 2010 19:09:14 GMT

Sorry I haven’t been blogging for a while. Last Fall, I switch to work on Bing Infrastructure and have been very, very busy (and having a wonderful time). The projects I’m working on include COSMOS and Autopilot. COSMOS is a petabyte store (working towards being an exabyte store) which runs over tens of thousands of inexpensive computers. In addition to reliable storage, COSMOS supports Dryad based computation with application development in SCOPE which is a SQL-like language. Some public papers include: SCOPE and COSMOS, and Partitioning and Parallel Plans in SCOPE and COSMOS. The Autopilot team in OSD (Online Services Division which includes Bing) makes hardware selections for our ever-increasing bunch of servers, networking, systems support, automatic deployment and load balancing. See Autopilot. I have been having a blast working with the team in Bellevue and a team in Beijing with lots of talented people.

Recently, I’ve been trying to pull the various sub-teams together and open up communications and camaraderie within them. This is an approach I’ve taken numerous times since the 1980s and I believe it is super powerful and important. The idea is SO simple that I am always amazed it is not more common. We pick a regular day of the week and meet for lunch on that day. Simply reserve a conference room and have everyone run to the cafeteria (or wherever) and grab their lunch. As people trickle into the room at a few minutes after noon, the discussion begins. Here are some basic rules:

1) You want to pick a team size of 10-25ish which has a set of common technology interests. It should include development, test, and program management.

2) No topic is off limits but it needs to be broadly interesting to the folks on the team. Realistically, this quickly settles into some challenge the team is facing, some architectural issue, or some customer issue. What I like to do is let it go wherever it goes but watch the folks in the room. Once in a GREAT while, some small group gets into something that is kinda boring the others and you need to steer the discussion back to one of general interest. After a team is used to the meetings, it happens less often.

3) One person at a time talks as the discussion moves back and forth. It is important to listen to everyone and let the discussion evolve. No one should dominate the time. The more senior people should contribute (they usually have lots of valuable stuff to add) but they shouldn’t talk for too long.

4) It is important to try and get as many people (senior and junior) to participate as possible. When someone fresh out of college and new on the team feels they can contribute to the discussion, they feel more like they belong. To ensure this is safe and OK, it needs to be understood that critical responses are not OK. Gently discussing where a proposal may not be the best for some concrete reason is great as long as everyone contributing can feel safe. Of course, healthy and safe teasing and joking around are fun. One thing I’ve learned (by screwing it up) is that you shouldn’t call on the shy people to participate. Sometimes, a really shy person would rather hide under the table than talk. If you call on them, they’ll stop showing up for lunch. Now, when they WANT to speak, it is OK to stop the table to listen (unless the folks at the table have already done the polite thing as they should). It is normal for me to look around the room through the hour and make a mental note of who has not yet contributed. If I have sufficient context, I may try to steer the topic to something near to their interests to make it easy and natural for them to chime in… that’s not threatening to them. When this goes well, you will find that in a team of 15 to 20 everyone (or almost everyone) will have participated during the hour.

5) As much as possible, it is good to not cancel the weekly lunch discussions. We all find crunches and deadlines. The team gets busy and has to work its butt off for a few weeks. Still, everyone needs to eat lunch and the emotional decompression with your friends (and talking about the fun challenges in your product) is a good diversion from tracking down those bugs.

6) I think it’s important to not prepare for the lunch nor to announce a topic for the lunch. If the topic is known in advance, sometimes people say: “That’s not my favorite topic” and blow off the lunch. If, on the other hand, they get used to not expecting to know the topic and that the sessions are fun, they will show up and it can evolve into something fun. Also, if you have announced: “Sam will talk about XYZ”, it gets really tough to handle when Sam is a bore. Not everyone is capable of keeping a lecture interesting. It is better to say that the lectures are not during the “Friday Lunch Discussions” but are at some other time. You will learn that having a pre-planned topic takes away from the team’s sense of ownership of THEIR time to chat together. If there is a speaker, their won’t be as heavy a participation and it is a different event.

I like the following definition of the word “Culture”. Culture is “the way we do things around here”. It’s my experience that once the expectations are set, the team just naturally behaves in accordance with how they are used to behaving. That means you can set this discussion group up and it just happens moving forward.

There are many positive results from having these weekly meetings. The members of the team get to know the some of the larger team better than they would have… healthy fun discussions make you LIKE your colleagues. With a team size of 10-25ish, there will always be folks you don’t know as well as you’d like to know. The discussion of the main challenges for the team can forge and evolve the shared vision for the team. It is essential that the broader team understand the big picture and then work down into some of the details of the other pieces.

For me, this is both a major investment and a really nice treat. I’m working to provide architectural guidance to all of COSMOS and Autopilot. It’s pretty hard to keep up with all of these efforts and the people involved in them. I’ve wiped out my lunches for the week. Monday is Autopilot’s discussion meeting. Tuesday’s is the COSMOS Store team. Wednesday this week was the first for the SCOPE Compiler and Optimizer team. [Thursday’s are my day to write quietly… like this blog entry.] Friday’s are the Execution Engine’s meeting. I am still trying to figure out which of these days will have TWO meetings so the Manageability team can have a discussion (and I will have to switch off between them).

Each time I’ve started a team going with this, there’s quite a bit of confusion before the meeting but afterwards we have a good consensus that it was fun and useful. For me, it allows a precious opportunity to learn about the technology, the people, and how the people interact with their technology. Sometimes, I can contribute some comment here and there which moves the way people think about the goals and direction of the product. Always, it gives me an opportunity to work to help all the members of the team to learn and grow. A healthy team is built out of people who are growing and stretching in their abilities. This requires all the members of the team helping their friends in that process.

While typing this blog entry, an email came through from a teammate wanting to start a small effort on a utility. The discussion at lunch yesterday encouraged him to push forward with an idea he and another teammate were thinking about. I’m going to meet with them next week and get to know them better. It is tough to quantify all the ways these discussion help the team!

Anyway, I just wanted to share this experience with you. Software is much more than just bits and bytes. Many times, it is building the caring and comfort that allows you to be friends with shared responsibilities and shared goals. Sharing meals and camaraderie make a big difference. An occasional beer is good, too.

Great to be blogging!

- Pat

Trip Report -- HPTS (High Performance Transaction Systems) Workshop -- Part 1 of the Trip Report

Pat Helland — Mon, 02 Nov 2009 17:35:00 GMT

Trip Report – HPTS (High Performance Transaction Systems) Workshop – Oct 25^th-28^th, Asilomar, Pacific Grove, CA. (www.hpts.ws). This is Part 1 of the trip report… I am chiseling my way through my notes and expect this to take a couple more blog entries before I can describe everything I plan to cover. Believe it or not, this 11 page “Part 1” is only taking us through Monday morning – there’s Monday afternoon, all Tuesday, and Wednesday morning yet to cover!

Overview:

HPTS was founded by Jim Gray, Dieter Gawlick, Andreas Reuter, and some other industry luminaries in 1985. It is a workshop designed to bring together practitioners and academics working in super large, scalable, high throughput systems. There is a lot of overlap between this community and the folks doing DBMS systems. In the early days, the biggest challenges included getting 1000 transactions per second through a mainframe. At the time, that felt like getting a man to land on the moon and come back safely. Now, this includes huge web based sites like Amazon, eBay, Google, and Bing. Our area of interest also includes integrating massively complex heterogeneous Enterprise and Internet (B2B) systems. Of course, we are fascinated with hardware trends, networking, and all sorts of geeky stuff.

I first attended HPTS in 1985. It was a magic opportunity to be with legends in the industry. Gene Amdahl drove up in his Rolls Royce and parked next to my beat up old van. Mike Blasgen(System R), Mike Stonebraker, Bruce Lindsay, Mohan, Andy Heller, Dieter Gawlick, Hector Garcia-Molina, Irv Traiger, Phil Bernstein, Jim Gray, and many more folks who had become well known to me through their papers were there in the flesh and treating me (a 29 year old punk) with kindness and respect. It was a magical 3 day gathering.

Let me explain the Asilomar Conference Center (www.visitasilomar.com). Asilomar sits right by the Pacific Ocean between Pebble Beach and Monterey in the Pine Trees and includes dunes leading right to the beach. It was built as a YWCA around 1910 or so and contains a number of beautiful historic buildings in the arts and crafts style built by architect Julia Morgan. In the 1950s, it was sold to the state of California which has operated it as a conference center (and built more buildings in the 1960s). The rooms are rustic (no TVs or phones – crappy mattresses), the food is served in a large dining hall when the bell rings on the roof and it has a magic feel. It is normal to walk out of your room, hear the surf, and see deer walking around. I absolutely love the evenings by the real wood fireplaces.

In 1985, Gray decided we could all go cheap and lined us up to sleep four in a room. I shared a room with Dieter Gawlick (inventor of IMS/FastPath among other things), Hector Garcia-Molina, and Theo Haerderer (a wonderful professor from Germany). After the first night, Hector informed us we ALL snored too much and moved to stay in a hotel down the street. For the next 8 or 10 gatherings, we were two to a room but now most of us pay a bit more to have a private room.

I’ve attended HPTS every two years (for three days) since 1985. I was program and general chair in 1989 and general chair in 2007. Since Jim Gray’s disappearance, I’ve assumed the role of local arrangements chair and “godfather” – kinda’ watching out for the whole workshop. This year, James Hamilton (now of Amazon) was general chair and Margo Seltzer (of Harvard) was the program chair. We had a wonderful (but exhausting) three days and I will tell you about them soon.

One of the traditions is that the folks leading HPTS wander around during the first two days trying to figure out who gets suckered into doing the work as the next workshop happens two years later. I am thrilled to report that Mohan (from IBM) will be the general chair in 2011 and Mike Carey (now of UC Irvine) will be the program chair. I will remain as local arrangements chair and “godfather”.

So, every HPTS starts with Sunday dinner and a casual beer/wine afterwards. Monday is all day presentations and after dinner we have poster sessions (volunteer 10 minute talks – we had 15 this year). Tuesday is all day presentations and this time we had a 90+ minute panel after dinner. Finally, we have sessions Wednesday morning and adjourn after lunch. It is a long, exhausting, and exhilarating 3 to 4 days.

Of course, the most important part of the workshop is the breaks and the opportunity to talk with people working in the same area as you are working. In addition to the grizzled old luminaries, this year we went out of our way to get more graduate students and invited 16 of the brightest and best from Berkeley, Stanford, MIT, Harvard, Brown, and more. I spent each of the three evenings staying up late talking by the fire with these wonderful young folks (and some other diehard old farts). As I type this the following weekend, I am still tired from the sleep shortage!

So, let’s get to the content:

Session 1) The New Memory Hierarchy –
Session Chair: David Cheriton (Stanford)

Steve Kleiman – Chief Scientist at NetApp: “Flash on Compute Servers”

So, Steve Kleiman is the Chief Scientist at NetApp so you would expect him to be focused on network attached storage and we were not disappointed.

Steve started with a discussion of hardware trends (not all of which I was able to capture in my notes). He observed that with multi-core and VMs, more bandwidth (BW) and IOs per Second (IOPS) are needed for each node. Also, as a trend, Steve pointed out that flash memory has more than 5X the IOPS and less than 100 microseconds access time.

The original case for network storage lies around the benefits of aggregating disk which include more random IOPS and bandwidth as well as the management simplicity of concentrating data into a centralized place. It has been the case that a single application per host has somewhat limited ability to consume bandwidth.

Considering flash in a networked storage environment, it does not have the same benefits of sharing. Since there are enough IOPS in a flash drive to placate most apps, the equation changes. Is network storage fast when the network latency is noticeable compared to flash? We can do 5 SSDs to get 40,000 IOPS but will the latency be a problem?

We will see flash on compute servers. Servers can already accommodate flash and it is cheap enough with 250GB < 10% of the server cost.

A new tier in the storage hierarchy: Steve argues we will see storage split into two tiers: the IOPS tier and the capacity tier. The IOPS-Tier will offer the best $/IOPS with low latency and will have sufficient capacity for the active data in most typical enterprise systems. It will be the new “primary storage”. The Capacity-Tier will offer the best $/GB – Deduplication, compression, and good serial performance.

Steve then outlined three broad categories of Host-Based Flash usage. DAS (which I think means “Direct Access Storage but my notes failed me), Primary Storage Service, and Cache.

· Host-Base Flash: DAS: Single point of failure and so you would use normal backup or application based replication like Exchange or Oracle). You can have boot files, VM, temp files.

· Host-Based Flash: Primary Storage Service: File or Block. Requires mirroring or RAID to peers. Allows capacity sharing. Steve believes it is likely that Microsoft and VMWare are likely to compete here – the integration of this is complex. Making reliable primary storage is hard and this may impact the roles of folks in IT.

· Host-Based Flash: Cache –File or block. This will be a writethrough cache. If you do a write-back you will occasionally lose stuff. Using a cache can allow for a centralized data management (in a remote network storage server). This can help with keeping the current IT storage management roles by providing an automatic “tiering”.

So, data management spans the IOPS tier and the Capacity tier. You need to have automated data movement and global access. It is impractical to have manual placement at huge scale. It is going to be important for us to have a technology independent language to specify the desired properties for the data. Say what properties you want and not how to get them. New SLO (Service Level Objectives) for Max Latency, Bandwidth, Availability, etc.

In summary, the new host-based flash is the new IOPS-Tier. It will replace high-performance primary storage in the cloud. This IOPS-Tier s is not going to be economical as a secondary or as archival since the $/GB is higher than spinning media. The Capacity Tier will be implemented as network storage. It is optimized for the best $/GB. Data management between these two tiers will be important.

There was a lot of discussion around the SLO (Service Level Objectives). Steve indicated that their vision was not super-flexible set of knobs but more a small set of options for defining the data characteristics. Margo offered that one aspect of what was important was the “value” of different parts of data which can, in turn, tell the system how hard to work to make the data remain available.

Someone asked if there was any performance analysis for 3-tier storage (memory, IOPS-Tier, Capacity-Tier) over more traditional 2-tier storage (Memory and Disk). Steve answered that there had not been an explicit performance analysis.

Stonebraker observed that enterprises seem to like NAS/SAN but that the Cloud guys are using direct attached storage. Are the forward looking guys walking from NAS/SAN? Steve replied there are people very interested in NAS/SAN because it gives them a way to control the data and make servers stateless. Adding in the IOPS-Tier is a simple extension.

This was a fascinating and engaging presentation!

Memory Technologies for Data Intensive Computing –Andy Bechtolshein (Sun Computers)

What an honor to get Andy to come and present to us. Unfortunately, I didn’t get a chance to chat with him personally as he was only at the workshop for a short while.

Andy is a well respected and talented hardware engineer who was a founder of both Sun Microsystems and of Arista Network. His slides are not going to be posted and I typed as aggressively as I could during his lightning fast presentation. I would recommend you look both at my friend James Hamilton’s blog post of Andy’s talk at: http://perspectives.mvdirona.com/2009/10/26/AndyBechtolsheimAtHPTS2009.aspx and then read my notes attached below. I have not attempted to coalesce the information between James’ notes and mine and there may be overlap. James has done a nice job cleaning his up and I will focus on some of the other sessions in my editing.

Here’s my rough notes:

è Andy speaks VERY fast and I lost a lot of what was covered – The slides were very dense – Also, I apologize for some inaccuracies… These are RAW notes that I made during the fast paced presentation and some of them don’t completely make sense to me.

Semiconductor Technology Roadmap

Still increasing

Number of devices on the chip still doubling

Flash is increasing faster than logic and DRAM – Flash faster than Moore’s law

Between 2010 and 2022, Flash should increase by 100X – After that (10nm) optical problems may become severe – don’t know after 2022

Emerging devices

· Carbon-nano tubes

· Graphene Nanoribbons

· … <more that I lost…>

Silicon Roadmap Predictions

· 128X increase in transistors by 2022

· 1000 cores per chip by 2022

CPU Module Roadmap

· IO bandwidth will not allow the same bandwidth per core. IO going a little faster but number of pins hasn’t increased in 10 years.

· 1000 cores would need several terabytes of bandwidth to memory. Of course, this would need power and cooling

· <more lost…>

CPU Challenge

· Power per core --

· Clock rates not increasing – need lower power level per core

· <more lost…>

Power efficiency – Increasing frequency – a Square effect on the power

· CPU power strategy –

· Clock rates look flat

· High clock rates are less power efficient

· To reduce power, simplify architecture

· <more lost…>

Multi-Chip 3D packaging

· Thru-Si via Stacking

· Need to combine CPU and Memory on one module

High Density 3D Multi-Chip Module

Benefits of MCM Packaging

· Enables much higher memory bandwidth

· More channels, wider interfaces, faster I/o

· Greatly reduces power

Challenges with MCM packaging

· Total Memory – 64 device <…??...>

· With 64 GNB/device – 4TB

· Assuming 1K cores – 4BG/core

MCM enables Fabric I/O

· 2010 –1 32 Gb/direction

· 2020 1.73Tbps/direction

· Mesh or higher radix fabric topologies

Expected Link Data Rates

· 10GBps shipping today

· 2012 – 25 GBps

· 2016 – 50 GBPS

· 2020 – 100 GBps

ExaScale Storage

· Forget Hard Disks – Disk are not getting faster

Flash Experience so far

· Not easy to get to 1000000 IOPS

· Writes are a problem

· SAS/SATA interface not optimal

· Direct PCI-Express interface looks more promising. – Move FLASH closer to the CPU -- Lower latency and more throughput

Flash in the Memory Hierarchy

· Flash is not random access more like a block device -- random access about 1000 slower than DRAM

· Flash can be used as a stable storage device -- writes are committed – Supercap magic behind the scenes.

· Tremendous throughput and size – terabytes of capacity

Flash Latency

· Latency expected to drop by a factor of 10

· Flash transfer rates will double each year for the next several years. – 250MBS/sec 2012

Flash prices

· 2004 -- $100 GB

· 2009 $1 – GB

· 2012 -- $.25 / GB

· Prices are dropping dramatically –

Flash Summary

· FLASH will be very expensive, very dense,

· Costs falling by 50% per year

· Access times falling by 50% per year

Tech Summary

· Moore’s Law for at least 10 more years

· Frequency Gains not happening

· Flash will be interesting

Software Challenges

· Limits of App Parallelism

· Need to exploit Parallelism

Session 2) Using the New Memory Hierarch
Session Chair: David Cheriton (Stanford)

Scaling-Out without Partitioning – Phil Bernstein and Colin Reid (Microsoft)

This session was, in many ways, one of the most interesting at HPTS. It was not new to me since I had been privileged to spend a number of hours talking to Colin (and a bit to Phil) about Hyder back at Microsoft. Phil did a great job explaining a radical new approach to scale-out and serializable database!

Hyder is a software stack for transactional record management. It can offer full database functionality and is designed to take advantage of flash in a novel way.

Most approaches to scale-out use partitioning and spread the data across multiple machines leaving the application responsible for consistency.

In Hyder, the database is the log, no partitioning is required, and the database is multi-versioned. Hyder runs in the App process with a simple high-performance programming model and no need for client server. This avoids the expense of RPC.

Hyder leverages some new hardware assumptions. I/Os are now cheap and abundant. Raw flash (not SSDs – raw flash) offers at least 10^4 more IOPS/GB than HDD. This allows for dramatic changes in usage patterns. We have cheap and high performance data center networks. Large and cheap 64-bit addressable memories are available. Also, with many-core servers, computation can be squandered and Hyder leverages that abundant computation to keep a consistent view of the data as it changes.

The Hyder system has individual nodes and a shared flash storage which holds a log. Appending a record to the log involves a send to the log controller and a response with the location in the log into which the record was appended. In this fashion, many servers can be pushing records into the log and they are allocated a location by the log controller. It turns out that this simple centralized function of assigning a log location on append will adjudicate any conflicts (as we shall see later).

The Hyder stack comprises a persistent programming language like LING or SQL, an optimistic transaction protocol, and a multi-versioned binary search tree to represent the database state. The Hyder database is stored in a log but it IS a binary tree. So you can think of the database as a binary tree that is kept in the log and you find data by climbing the tree through the log.

The Binary Tree is multi-versioned. You do a copy-on-write creating new nodes and replace nodes up to the root. The transaction commits when the copy-on-write makes it up to the root of the tree.

For transaction execution, each server has a cache of the last committed state. That cache is going to be close to the latest and greatest state since each server is constantly replaying the log to keep the local state accurate [recall the assumption that there are lots of cores per server and it’s OK to spend cycles from the extra cores]. So, each transaction running in a single server reads a snapshot and generates an intention log record. The transaction gets a pointer to the snapshot and generates an intention log record. The server generates updates locally appending them to the log (recall that an append is sent to the log controller which returns the log-id with its placement in the log). Updates are copy-on-write climbing up the binary tree to the root.

Log updates get broadcast to all the servers – everyone sees the log. Changes to the log are only done by appending to the log. New records and their addresses are broadcast to all servers. In this fashion, each server can reconstruct the tail of the log.

Now we get to transaction commit. All machines are marching ahead rolling the log forward. When an intention record is generated (indicating a proposed new root for the log), each machine in the cluster checks for a conflict. If any of the records updated conflicts with previously committed work, the transaction is aborted. Conflict in this case is traditional read/write conflict using all the classic set of rules for various degrees of consistency. If there are no conflicts, the transaction is committed. If there are conflicts, the transaction is aborted.

Because there is an append in the log controller which assigns order, each intention record will race to the log and the controller will pick a winner. Because log ordering is deterministic, each of the server nodes can reliably come to the same conclusion about which transaction won the race and, hence commits rather than aborts in the face of conflict. Conflict detection and transaction commit is done redundantly and predictably at each node.

So, each node generates an intention log for a transaction it wants to do. Every server checks for conflicts and then either aborts or commits. The log order provides determinism.

Performance of Hyder: The system scales without partitioning. The system-wide throughput of update transactions is bounded by the update pipeline. It is estimated this can perform 15K update transactions per second over a 1GB Ethernet and 150K update transactions per second over a 10GB Ethernet. Conflict detection and merge can do about 200K txs per second.

The abort rate on write-hot data is bounded by the transaction’s conflict zone. This is determined by the transactions end-to-end latency. This conflict zone is about 200 microseconds in the prototype. It is estimated that this would handle about 1500 TPS (assuming all the transaction have conflicts).

Major Technologies: This is using flash as append only (and it is not using SSD flash but raw flash with a special controller). The custom controller over raw flash has mechanisms for sync and fault tolerance. Storage is striped with a self-adaptive algorithm for storage allocation. It includes a fault tolerant algorithm for a totally ordered log. Also, it has a fast algorithm for conflict detection and merging of records.

Status: Most of Hyder has been prototyped but there is a long way to go. Phil Bernstein and Colin Reid are working on a paper and HPTS is the first time Hyder has been discussed publicly.

Again, this was a wonderful talk with some fascinating and different ideas. Great work!

A Performance Puzzle: B-Tree Insertions are Slow on SSDs (Bradley Kszmaul – Tokutek – Research Scientist at MIT)

So, this talk was about some surprises in SSD performance in supporting MySQL. Much of the discussion was about the datasheets for some of the drives not matching the measured reality.

Bradley had measured the value of running MySQL against SSDs and expected the SSDs (with higher IOPS) to get better Btree performance. Here’s my rough and raw notes:

Intel X25E Specs

· Read BW up to 250 MB/s

· Write BW up to 170 MB/s

· Random 4KB read rate – 35 KIO/s

· Random 4KB

· <lost some…>

Measure Berkeley DB

· Trending to 4500 write per second

Performance Model --

· Startup cost plus bandwidth cost

· Looking at read performance as a function of block size

· As block size gets bigger, bandwidth goes up

Discussion à Margo points out that many Filesystems see large blocks and do read ahead. That can perturb the results…

When you read and write large blocks, Bradley sees Filesystems doing the right thing and keeping contiguous blocks adjacent.

The datasheet for the drive is not matching the reality for larger block sizes

Mixing reads and writes seems to impact the performance quite a bit…

What Block Size to Use?

· For point-queries, BTrees are insensitive to block size.

· For range queries, block size if important

· Tension: Large block sizes make range queries faster; Small block sizes make exact match faster

· Set block size so half of the time is accounted by seek, half by bandwidth

· Reads: 50KB for SSD but 500KB-1000KB for rotating disk

· Writes 10KB SSD , 500KB-1000KB for HDD

· Mixed read/write 21KB for SSD

Cache-Oblivious Approach

· Use data structures that are fast for any block size

· Also can speed insertions without slowing searches

· Tokutek’s MySQL storage engine uses these ideas.

Discussion: James Hamilton layering of software and release by release changes makes it hard to get predictable behavior. When all these releases are separately evolved, the performance behavior is very unpredictable. Bradley: fragmentation starts messing with this, too. James: all the layers interact and we need to measure by the workload and adapt. The complexity makes performance very hard to predict.

Implications of Storage Class Memories (SCMs) on Software Architectures (C. Mohan – IBM)

Storage Class Memory (SCM)

Mohan gave a very thought provoking talk on the implications of Storage Class Memories. Where do we need them? What good will they do?

If we have non-volatile memory that is as easy to access as DRAM, will we have corruption problems? Will we see a running system trash the non-volatile memory like it trashes DRAM? One of the advantages of our current system design is that we have to pass a “sanity test” of an IO driver or messaging stack to get nastiness outside of a running system. That dramatically reduces the chance of this happening.

Anyway, Mohan’s talk was really thought provoking and enjoyable! It was good to see my old friend back at HPTS after taking a stint in India for a few years.

Raw notes:

· Blurs the distinction between DRAM and Disk

Industry SCM Activities

· IBM

· Intel/ST spun out of Numonyx (FLASH and PCM)

· Samsung, Numonyx sample PCM chips –

· <lost som>

Speed/Volitility/persistenct

· DRAM plus SWCM – Fast and persistent

· Persistent storage will not lose data

Competing technologies for SCM

· Phase Change RAM (most promising)

SCM as a part of the storage stack

· The distinction between RAM and Disk – the line in between is blurring

· Memory-like and also Storage-Like

SCM Design Triangle

· Speed – endurance – Cost/bit

· Speed with Write endurance is more memory-like

Phase-Change RAM

· When density increases, the failure rate goes up as bits flip more often

· SCM is on Moore’s law

PCM is fast

· 100-1000 ns

Architecture

· SCM may be used as Memory

· SCM may, alternatively be used as a storage (through IO controllers)

· Discussion à Pat: Bugs in non-volatile memory can sometimes be preserved

PCM Use Cases

· PCM as Disk

· PCM as Staging device

· PCM as memory

· PCM as extended memory

Exploring DBMS as an exploiter of PCM

· Should log records be written directly to PCM?

· Should log be more like Disk (first to buffer and then forced to disk)?

· PCM is not as reliable as disk à do you need to offload to a disk

· If you use Group Commit, it is unclear that PCM helps a DBMS

PCM replaces DRAM – Buffer Pool in PC

· The PCM Buffer Pool access will be slower than DRAM

· Writes will suffer more than reads!

· Should we instead have a DRAM BP backed by a PCM BP?

· Similar to DB2 z in parallel sysplex environment with BPs in coupling facility (CF)

· But DB2 situation has well defined rules on when pages a move from DRAM CP to CF BP

· Variation was used in SafeRAM work at MCC in 1989

Assume the whole DB fits in PCM

· Apply old main memory DB concepts design directly

· Shouldn’t we leverage persistence specially?

· Every bit change persisting may not be good!

· Today’s failure semantics has flexibility on tracking changes to DB pages.

· Memory overwrites will cause more damage

· If every write assumed to be persistent as soon as write completes, then L1 and L2 caching can’t be leveraged.

Assume whole DB fits in memory

· Need to externalize periodically since PCM won’t have good endurance

· If DB spans both DRAM and PCM, hard to figure out what goes where!

What about Logging?

· If PCM is persistent and whole DB is in PCM, do we need logging?

· Of course it is needed to provide at least partial rollback even if data is being versioned (at least to track versions to invalidate)

The end of Monday morning! More soon…

- Pat

Building Blocks and the Three Bears

Pat Helland — Tue, 14 Apr 2009 02:41:54 GMT

So, about a year and a half ago, I was working in Developer Division and there were lots of discussions about different kinds of components… it seemed that everyone was talking past each other and meant really different things about they meant by a component. I decided to try to identify three different meanings of the word “component”. To avoid debates over the naming, I chose the really ridiculous names: “Baby-Bear”, “Momma-Bear”, and “Papa-Bear” to describe the different things meant by different folks when they talked about a component… of course, these names were not meant as serious but it did lead to some illustrative discussions about the “Three Bears”. Last week, Krzysztof Cwalina asked me to post a description of the “Three Bears” because it would be helpful for him in describing their project, MEF. Here goes…

The idea is that there are three different types of components.

Baby-Bears are much like the classic object with inheritance, reference-passing, and synchronous interactions with their callers.
Momma-Bears (closely related to what is provided by MEF), are comprised of multiple Baby-Bears. While exhibiting synchronous call patterns, they do not support pass-by-reference nor inheritance. They will, however, allow for support of dependency injection.
Papa-Bears are the biggest of the three and are most similar to a SOA Service. A Papa-Bear will typically be composed of multiple Momma-Bears. Papa-Bears interact with their partners asynchronously, are frequently remote, do not support pass-by-reference or inheritance, and may be offline.

My interest has always been in the realm of the Papa-Bear and other projects have focused more in the realm of the Baby-Bear and the Momma-Bear. I will leave it to Krzysztof to explain MEF (which I believe is more akin to the Momma-Bear) type of component.

Anyway, this is what I like to call “Building Blocks and the Three Bears”.

- Pat

“Book Report” on the UC Berkeley Paper “Above the Clouds: a Berkeley View of Cloud Computing”

Pat Helland — Sat, 11 Apr 2009 03:57:29 GMT

As I mentioned in a previous blog post, I am very impressed with a recent paper from my friends at the Berkeley RAD Lab called: Above the Clouds: a Berkeley View of Cloud Computing. it seemed to me that it would be important for these concepts to be better understood by a broad audience within Microsoft so I took the time to create a presentation covering the contents of the paper. You can see the presentation I created covering the paper’s contents here.

If you are interested in cloud computing, you want to understand these ideas.

Also, I highly recommend the Berkeley blog on cloud computing available at: http://abovetheclouds.cs.berkeley.edu.

Thanks to the folks at Berkeley for their hard work and permission to show this to you.

- Pat

Great Paper from UC Berkeley on Cloud Computing

Pat Helland — Sat, 21 Feb 2009 03:05:00 GMT

My friends at the UC Berkeley RAD Lab have written a wonderful paper on Cloud Computing. It can be found (with a video interview and some other stuff) at http://abovetheclouds.cs.berkeley.edu. I strongly recommend looking at their work!

- Pat

Slides from SQL Service Broker Design Preview from August 2001

Pat Helland — Wed, 21 Jan 2009 05:41:35 GMT

I know this is a LONG time ago but these slides were never made public before and there's some interesting (and still relevant) stuff in this collection. At the time, it was confidential but that is no longer the case.

In August of 2001, the SQL Service Broker team did a design preview and I wrote five presentations totaling 280 slides. I presented these over a 12 hour period (actually, that time also includes a sixth talk with another 60 slides on "Fiefdoms and Emissaries" but I'll push that out later).

CAUTION: These slides are from August of 2001 and SQL Service Broker shipped in late 2005. Many things are changed! Do NOT count on these slides for accuracy of the product but they can be a big help with inspiration.

The following slide decks are now out in my new public skydrive folder.

SQL Service Broker Overview

An overview and introduction to SQL Service Broker. This slide deck has the following major sections:

Introduction: Motivates the challenges in loosely-coupled messaging, the advantages of deeply integrating messaging with the database, support for scaling and fault tolerance, and some scenarios for SQL Service Broker.

High-Level Architecture: This covers web services, brokers, message fragmentation, service environments (which provide a namespace for services), message-types, service-contracts, services, and service programs.

Programming Model: This section covers an overview of the programming model, service instances and state management, dialogs, and again ties these programmatic concepts to the service instance. Caution: the syntax has changed between this and the shipped product!

Relationship to Other Stuff: Here we visit management issues, interoperation, and look at products that create services.

Conclusion: We revisit the scenarios and summarize the points made.

The SSB overview talk has 87 slides and can be found here.

SQL Service Broker Messaging

This talk examines the message protocols used in SQL Service Broker. Reminder: This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

Introduction: This covers the concepts of reliable messaging, brief discussion of web services, Internet and Intranet issues, transport support, binary and lossless transformations, and a sketch of the protocol stack.

Adjacent Broker Protocol: The adjacent broker protocol talks to a nearby SQL Server to propagate messages and get them closer to their destination. This section outlines the challenges at this level with delivery (sometimes tunneled through firewalls), lifetime management, and format negotiation.

Endpoint Broker Protocol: The endpoint broker protocol functions across the SQL Server engines that actually constitute the two endpoints for the communication (and it layers atop the adjacent broker protocol). This section covers dialog endpoint state, fragmentation and sequence numbers, piggy-backed acknowledgements, and unsequenced ack messages.

Message Formats: This covers pieces of the messages, encryption challenges, and ASCII versus binary sections of the SOAP message.

Monolog Contracts: Monologs were not implemented but they are VERY interesting if you care about Pub-Sub. This short section on Monologs covers some of the issues related to the messaging formats. It discusses dialog contracts to implement monologs.

Web Services and SOAP Bubbles: Some brief framing of the relationship of the SSB protocols to Web Services and SOAP.

Conclusion: a summary of the content covered.

The SSB messaging talk has 31 slides and can be found here.

SQL Service Broker Performance

This talk examines performance, scaling, and high-availability issues with SQL Service Broker. Reminder: This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

Introduction

Work Within a Service: This covers messages in and out of a service, service instance state management, data access from services, and scalable and highly-available ASP.NET apps.

Fault Tolerant Services: In this section, we cover designing services for fault tolerance, deploying services for fault tolerance, and more issues with fault tolerance.

Scaling and High-Performance: Here, we examine application issues for scale-up, execution of scale-up services, application issues for scale-out, and execution of scale-out services.

Conclusion

The SSB performance talk has 46 slides and can be found here.

SQL Service Broker Monologs

One of my favorite talks, this describes the unshipped feature of SSB Monologs. Monologs implement a very flexible form of pub-sub. While they are an unimplemented feature in SSB, the serious implementer could follow the guidance in this talk and avoid a lot of difficulties. Reminder: This is not an accurate depiction of what is in the product (it has not shipped)!

This slide deck has the following major sections:

Monolog Overview: This covers "What's a Monolog?", programmatic interfaces, and broker-to-broker contracts. Included are the notions of "History versus Currency", Monolog-Groups, and Subscriptions.

Programming with Monologs: This section includes programming a publisher, requesting a subscription, and monolog delivery in a subscriber.

Some Special Monolog Features: A really cool concept called a Monolog-Checkpoint is covered as well as the lifetime of monologs.

Plumbing Monologs Over Dialogs: We examine fan-out trees for scale, plumbing the fanout using dialogs, publisher agents, and subscriber agents.

Conclusion: Here we examine content syndication and content aggregation and then summarize the talk.

The SSB monologs talk has 53 slides and can be found here.

SQL Service Broker Security

Again, one of my favorite talks, this describes some of the special challenges with security that arise in a messaging and services system like SQL Service Broker. Classic session-oriented security only works when messages are received in order. That is just one of the challenges. Reminder: This is not an accurate depiction of what is in the product!

This slide deck has the following major sections:

Security Behavior: This section covers identity, dialog authorization, monolog authorization, and auditing.

Encryption Concepts: Here we examine symmetric key encryption, public key encryption, and some practical rules for using encryption.

Encrypting Dialogs: In this section, we introduce a concept called "handshakes" which helps optimize the performance of certificate management in a dialog-based system. We look at handshake certificates, messages and dialogs, and cryptographic guarantees.

Encrypting Monologs: Monologs are N-party communications. We examine in this section why 2-party security works best and then how we can use a fan-out tree to implement a secure scalable publisher.

Conclusion

The SSB security talk has 61 slides and can be found here.

I hope these talks are fun as there's still lots of interesting concepts in them 8 years later.

- Pat

Transaction Processing and the Barbarian Hordes

Pat Helland — Tue, 20 Jan 2009 04:53:00 GMT

So, I'm starting to figure out how to put stuff up on Windows Live Skydrive and I have beginning out here. I've only spent a few hours on this and there's lots more old stuff to add soon. I will send another blog post when this is in better shape.

I just couldn't resist sharing some fun, though...

As I was sorting through old files, I came across a video of Transaction Processing and the Barbarian Hordes and placed it up for all to see. This was my favorite moment on stage (at least so far) and was the source of a rare bout of stage fright before starting the presentation...

This was at the Long Beach PDC in November of 1996. The PDC was titled "The Renaissance of Distributed Computing" and I had a one hour General Session to introduce Microsoft Transaction Server. We wanted to show that MTS was easier to install than a classic Transaction Processing Monitor. So, my friend Greg Hope came out to do a 2-1/2 minute setup on MTS. While this was happening, I told a five minute joke...

There were 7000 people in the room and (I think) 10000+ on video feed. They had NO idea what I was doing for at least three minutes.

- Pat

A Wonderful Time at CIDR (Conference on Innovative Database Research)

Pat Helland — Sat, 10 Jan 2009 04:35:39 GMT

Sunday, January 4th through Wednesday, January 7th was the Conference on Innovative Database Research (CIDR) at Asilomar.

First, let me talk for a minute about Asilomar. It is located in the forest right off the beach in Pacific Grove, CA (right next to Monterey). From 1913 to 1956, it was a YWCA and has a number of magnificent and beautiful historic buildings. In 1956, it was sold to the State of California which has run it as a conference center since then. Asilomar has a deep and special place in my heart. I've spent 3 days every two years there since 1985 at the HPTS (High Performance Transaction Systems) workshop. This gathering has been attended by many wonderful friends in the transaction processing and database community. Every job I've gotten since the 1980s has been through contacts forged at HPTS. I chaired the HPTS workshop in 1989 and (believe it or not), my first grandson was born DURING the workshop (after the keynote, before the first session, and immediately after I ordered more coffee in a crisis for the first break). I still smile at the location of the (now removed) pay phone where I continually called to find out how the birth was progressing. Nick (my grandson) is now at Cal State San Jose. It was at Asilomar that I last saw my dear friend, Jim Gray. Now, I go two times each two years for CIDR (January of the odd years) and HPTS (October of the odd years). The smells, the feel, the fireplaces, the wonderful Chapel Building (where we meet to present to each other), and the amazing participants offer me a rich combination of memories and current stimulus.

So, this year's CIDR was a good one. The agenda was packed. There was a healthy combination of seasoned database folks and young grad student. We listened to a lot of presentations and had a lot of stimulating conversations. I saw a bunch of old friends and made a bunch of new ones. Someone (who... me??) managed to bring some nice sipping whiskey each night and get a gang off to chat quietly by the fire and ensure we ended up sleep deprived but with a better understanding of our new and old friends. It was a very, very nice event.

On Tuesday evening, I was on a panel with my old friend from Tandem days, Shel Finkelstein. We were joined by a friend of mine I met last year, Michael Armbrust, who is a graduate student at UC Berkeley studying in the Berkeley RAD Lab. The panel was on the ways in which cloud-based systems are, in many cases, evolving towards availability over consistency. I presented a summary of "Building on Quicksand". Shel did a marvelous job presenting "Principles for Inconsistency". Finally, Michael presented about his project at Berkeley called "SCADS: Scale-Independent Storage for Social Computing Applications" and then we had a joint Q&A. You can find the papers for these talks at http://www-db.cs.wisc.edu/cidr/cidr2009/program.html and then navigating the program to Tuesday evening.

The panel was a blast... I've known Shel for almost 20 years and knew he would do a great job. He did. This was Michael's first conference but I was THRILLED with his talk. He was clear, organized, and thought provoking. An absolute home run for someone's first conference!

FInally, I want to thank Mike Franklin, Joe Hellerstein, and Gerhard Weikum (and many others) for all their hard work to make CIDR 2009 possible.

- Pat

"Building on Quicksand" Paper for CIDR (Conference on Innovative Database Research)

Pat Helland — Sat, 13 Dec 2008 05:34:00 GMT

I just submitted the final (photo-ready) version of a new paper called "Building on Quicksand" to the Conference on Innovative Database Research. This is a paper I coauthored with my friend, Dave Campbell. We submitted an earlier version (to try to get into the conference) in October, I wrote a presentation on this for TechEd EMEA, and now we have a new and longer full paper complete. I like this version best of all.

- Pat

Building on Quicksand

Reliable systems have always been built out of unreliable components. Early on, the reliable components were small such as mirrored disks or ECC (Error Correcting Codes) in core memory. These systems were designed such that failures of these small components were transparent to the application. Later, the size of the unreliable components grew larger and semantic challenges crept into the application when failures occurred.

Fault tolerant algorithms comprise a set of idempotent sub-algorithms. Between these idempotent sub-algorithms, state is sent across the failure boundaries of the unreliable components. The failure of an unreliable component can then be tolerated as a takeover by a backup, which uses the last known state and drives forward with a retry of the idempotent sub-algorithm. Classically, this has been done in a linear fashion (i.e. one step at a time).

As the granularity of the unreliable component grows (from a mirrored disk to a system to a data center), the latency to communicate with a backup becomes unpalatable. This leads to a more relaxed model for fault tolerance. The primary system will acknowledge the work request and its actions without waiting to ensure that the backup is notified of the work. This improves the responsiveness of the system because the user is not delayed behind a slow interaction with the backup.

There are two implications of asynchronous state capture:

1) Everything promised by the primary is probabilistic. There is always a chance that an untimely failure shortly after the promise results in a backup proceeding without knowledge of the commitment. Hence, nothing is guaranteed!

2) Applications must ensure eventual consistency. Since work may be stuck in the primary after a failure and reappear later, the processing order for work cannot be guaranteed.

Platform designers are struggling to make this easier for their applications. Emerging patterns of eventual consistency and probabilistic execution may soon yield a way for applications to express requirements for a “looser” form of consistency while providing availability in the face of ever larger failures. As we will also point out in this paper, the patterns of probabilistic execution and eventual consistency are applicable to intermittently connected application patterns.

This paper recounts portions of the evolution of these trends, attempts to show the patterns that span these changes, and talks about future directions as we continue to “build on quicksand”.

An Interview with .NET Rocks while I Was at Barcelona Last Week

Pat Helland — Thu, 20 Nov 2008 22:55:41 GMT

Mark Dunn of .NET Rocks asked me to do an interview at TechEd EMEA last week. We chatted about one of my talks called "If You Have Too Much Data, then Good Enough Is Good Enough". The link to the interview is here.

Thanks, Mark! It was fun!

- Pat

Channel 9 Just Posted an Interview with Me on the Drive to Many-Core Processors

Pat Helland — Tue, 11 Nov 2008 13:27:31 GMT

This was filmed at TechEd NA in Orlando last June. I received the link to the video today. It was a nice chat with Patrick Weikle about the inevitability of Many-Core Processors and how this will inevitably drive us to parallel computation.

The link is here.

I'm Doing Five Talks at TechEd EMEA Developer (Barcelona -- November 10-14)

Pat Helland — Fri, 17 Oct 2008 01:05:02 GMT

Well, those folks at TechEd EMEA continue to be nice to me. Looks like I get to do four talks for the architecture track AND the closing general session. It is quite an honor to be told I can make up whatever I feel like and they will let me present it.

Here are the titles and abstracts for the four talks for the architecture track:

Building on Quicksand
Quicksand is a mixture of sand and water that looks like it is solid but will slowly consume anyone that tries to walk over it. This talk will argue that for many years our systems have built their reliability by layering on top of unreliable components and, effectively, built on top of quicksand. Furthermore, the granularity of the unreliable components has been steadily increasing through the years and is gradually showing new and challenging semantics to the application developer.

Fault tolerant algorithms are built by creating pieces that are idempotent or repeatable without harm. Ensuring these idempotent operations can be performed by components that are unreliable, a reliable design must capture the state ACROSS the failure units and keep going if something fails. This has been done with circuitry, mirrored disks, multiprocessors, and more. In the past, failures were usually masked from the running application. Now, the size of the unreliable component is inexorably getting larger as the economics of our systems changes. This is making it impossible for the underlying hardware and software platforms to hide the failures. Increasingly, applications must design to cope with the failures of these unreliable components.

This talk will sketch the theory of fault tolerance and show a number of examples of how systems have been designed through the years and how their evolution has gradually impacted application design. Hopefully, this discussion will help application designers understand the new responsibilities they face in building reliable distributed systems. It is possible to stay afloat on top of quicksand with sufficient care!

When You Have Too Much Data, “Good Enough” is Good Enough

Classic database management systems have very tight control over their data and over the correctness of the answers they provide. A number of pressures are being placed on classic database systems:

· We are awash in vast quantities of data
· There are many sources of the data, it frequently has inconsistent schema, and its interpretation is up to… well… interpretation
· Data are streaming into the system(s) at the same time as we are attempting to analyze it
· The machines holding the data may not all be concurrently available

This talk will examine some of these forces on classic database management systems and propose that there are a number of new ways to think about data. We will explore a number of different dimensions in which the “classic” database world is being relaxed to meet the needs for our customers in a changing world.

RIAs and Emissaries

The emerging field of RIA (Rich Internet Applications) is a hybrid between the classic rich-client application and a browser. Designed to provide the richness of the client environment without installation, RIA applications must be small and lightweight. To provide safety, they must have strictly limited access to resources on the client machine. Sandboxing of the client access limits the RIA app to per-app/per-user/per-machine state. The state of the art is evolving in how sharing can be done within RIA environments across applications and across users while providing safe computing.

The emissary design pattern is one where there is non-trusted code running outside of a service which is specifically designed to support interactions with the service. Leveraging reference data published by the target service, the emissary prepares requests with a sophisticated enough understanding of the service’s expectations that the request is likely to be accepted and processed. All of this is done while respecting the trust boundaries imposed by the target service. This pattern is clarified in the talk by showing parallels to the usage of paper forms in the business processes that proceeded the advent of computers.

We will explore the use and evolution of the Rich Internet Application and discuss some of the fun challenges the industry faces as RIAs evolve. Then, we will describe the emissary design pattern and show how RIAs are very well suited to working with their parent web environment using this pattern.

This talk is simply the wild ideas of the presenter and does not reflect any product commitments. It might, however, help you understand and build your applications differently.

Metropolis: Buildings and Applications

The Metropolis series of talks refers to a collection of presentations which examine cities and their evolution since being connected by the railroads and shows how Information Technology is undergoing the same transformations as IT shops are connected by the Internet. This second talk in the series examines the parallels between buildings in the city and applications in the IT shop.

Buildings come in many shapes, forms, and uses. Some are built with major investments and the builders anticipate a long and vibrant lifetime for the building. Others are built for quick turnover and short lives. As buildings age, different aspects of the building have different lifetimes and amortizations. It is common to see the roof, siding, plumbing, or wiring redone a number of times before the building is discarded. We are also seeing an increasing trend towards buildings being designed for a class or category of occupant without knowing the specific occupant. You see this in retail malls, office buildings, and more. As time moves on, a larger percentage of buildings are designed for a generalized use and a smaller percentage customized for the buyer.

Applications are following the same trend (although lagging the construction industry by a number of decades). We see major investments in some pieces of software and then surprisingly quick-and-dirty implementations of others. We see renovation of applications across releases with some aspects of the application remaining intact while other aspects may change many times over the lifetime of the code. Finally, we are gradually seeing the creation of patterns or classes of applications and supporting infrastructure to ease their development.

This talk will cover a number of parallels between buildings and applications. We intend to show that the same forces that have shaped the building industry are shaping the development of applications. It is possible to look to the building industry to tell us where applications will be going.
Some (but not many) of you may have seen this talk a few years ago but it is still guaranteed to make you think differently about the composition and evolution of applications.

And, finally, the closing general session at which the poor attendees have no alternative entertainment but listening to me! The organizing folks offered me this spot and asked me what I wanted to present. Since I was curious about data centers and felt REALLY uneducated, I had to select that as a topic. This led me to the following title and abstract for the big presentation.

Green Computing through Sharing
Reducing both Cost AND Carbon

Data centers consumed 1.5% of the total electricity in the US in 2006 and are on track to double as a percentage every five years. It is about 2% of the US total in 2008. Western Europe’s use is increasing at a slightly faster rate (from a slightly lower base percentage). The consumption of electicity within data centers is of significant financial and environmental importance.

Where the heck is all this power going? Why is the electrical load increasing so much? What can be done about it?

This talk will examine both traditional and emerging data center designs. We will start by examining how a data center is laid out, constructed, and managed. We will show two emerging trends: the change to designing data centers for the optimization of power and the emergence of new economies of scale in data centers which is contributing to the drive towards cloud computing. Microsoft is actively moving to compete in the space of cloud computing as we are seeing at the PDC (Professional Developers Conference) a few weeks before TechEd EMEA Developer.

Next, we will examine the sources of waste in the system today and examine why so many of our resources are underutilized. Because we are reluctant to share computing resources, they are left idle much of the time. Why is this currently the dominant choice? What can be done in the design of applications, systems, and data centers to make them more green (both carbon and cash)? What can developers do to make a difference?

Anyway, I am super excited to be going to Barcelona and humbled by the trust and opportunity I've been given! Hope to see you there! If so, please come up and say hi! In the meantime, I'm going to work on my slides...

- Pat

First You Live, Then You Die, Live Now!

Pat Helland — Fri, 10 Oct 2008 16:18:17 GMT

I didn't think I wanted to blog about this but it's been rattling around inside of me so I thought I would share it!

Life is short and life is precious... none of us know when our time is up and we all need to hold close what is dear to us.

Monday evening, Sept 29th, I was away from home for the evening at a Microsoft internal conference. At around 10:30PM, I checked the local newspaper and found out that a friend of mine, Keith Grinstein, had passed away the day before. Coverage was in the Seattle Post-Intelligencer, Puget Sound Business Journal, and later in the Seattle Times. A few days later, a nice article was in Crosscut.

Keith was my neighbor and I really, really liked him and really, really like his wife, Claire. We'd only been out to dinner once as the two couples and kept talking about wanting to set up another dinner. I would see Keith at least once a week hanging around in the lobby of the building we share. We all live on top of a hotel and there are folks working for the hotel down in the lobby all the time. It is a busy and vibrant place. Keith had been a successful entrepreneur and businessman. He was a lawyer, did a stint as an aid to various politicians, worked in telecommunications (he was formerly CEO of NexTel amongst other things), was an early investor in Amazon, and much more. He had set his life up the way he wanted it and was now doing venture capitol work out of a small company he set up with some friends. His office was one block away and he spent most mornings down in the "Studio" area off the hotel lobby where you can get breakfast. He was there for a couple of hours each day, pacing around, doing business over his cell phone, and teasing, laughing, and smiling with whoever he would encounter. Keith's wife, Claire, is a beautiful, vibrant, fun, and hard-working ophthalmologist. Keith had the energy to be on countless boards of charitable organizations. He was on the board for our home owner's association. One week before he passed away, Keith and Claire opened their home to 300-400 people on a condo tour to benefit the Pike Place Market.

While Keith was certainly not my closest friend nor I his, it is so very hard to see such a vibrant, good, kind, productive, and funny man go so suddenly. In our building, we have a community and it is hurting from the loss. The wonderful people working in the lobby are so sad... they interacted with Keith almost every day. Miles, one of the concierges, said "I miss Keith pacing back and forth across the lobby in his bare feet doing business over the cell phone in his loud and laughing voice". Denny, the general manager of the hotel, said that two days before he passed away, Keith had his head stuck through the door of Denny's office making him laugh hysterically. None of this surprises me.

We couldn't go to Keith's funeral as my wife, Lisa, had sinus surgery last Monday (she is doing fine). About two hours after the surgery, she was getting ready to go home and Lisa said "I couldn't be more thrilled with the surgery... I just wish we had been able to go to the funeral!". The surgeon said, "What, Keith's funeral?". We had not had any discussion of Keith with him but the doctor had served on a charity board with Keith and had known him for years. I've heard reports that the gathering was standing room only with over 1000 people in attendance. It would be amazing to say how many lives Keith had touched in his 48 years but I saw his spirit and that makes it all seem so plausible.

What can you say when a wonderful person's life is cut short? My dear friend, Jim Gray, frequently said, "First you live, then you die, live now!" It is SO true! I miss Keith...

- Pat

A Wonderful Few Days at a Wedding in Chicago

Pat Helland — Tue, 02 Sep 2008 04:53:31 GMT

Blogging on the bus again!! This was written on August 18th and 19th but it's taken a few weeks to get around to editing the photos... No computer stuff in this blog, just personal fun!

So, on Wednesday, August 13th, Lisa and I flew to Chicago to attend a wedding of a friend from our building in Seattle. We met John at the building in which we live. He recently fell in love with a wonderful lady named Helen and when we were invited, we had to say yes! Neither of us had been to Chicago (I am NOT counting two work trips where I didn't get out of the conference rooms). We were very excited to go and simply had the BEST time.

The wedding was at the downtown Ritz-Carlton right next door to the John Hancock Tower. As we were leaving home and walking out the front of the building to get to the Seattle airport, we saw John and Helen loading up to go! After some hugs, we determined that they we on an Alaska airlines flight landing 30 minutes before our United flight and we would see them at the hotel later in the evening since they were having dinner with Helen's parents who live in Chicago. We landed, got to the hotel, unpacked, and had dinner in the hotel. We stayed up late Wednesday night drinking at chatting with the almost newlyweds since we were the first to arrive into town.

Helen is a beautiful Chinese lady originally from Beijing and an only child. Her father came over to become a university professor (I think in material sciences) and started a successful company... the real American dream! John is from Philadelphia and is the 11th of 12 children from an Irish family. He has worked his butt off for years in the financial industry and has held amazing positions at very influential companies having climbed up solely with his wits, hard work, and honor. It is a joy to have them both as friends. They waited to later in life to marry since neither of them found their true love until recently. Just watching them together (and knowing they are our neighbors) is so cool!

Thursday, August 14th -- Walking, checking out the Chicago Spire, an amazing dinner, and partying with new friends!

We were going to be attending the wedding on Saturday but had Thursday and Friday as a mini-vacation to acquaint ourselves with downtown Chicago. The weather was perfect! The town was alive and pumping. We were in the middle of the Magnificent Mile of shopping on Michigan Avenue and we just had a blast. After sleeping in on Thursday, we had to try Chicago Pizza! We headed to Gino's East and were one of the first there for lunch (since this was our breakfast after sleeping in). I wanted to try deep dish pizza. Lisa wanted a thin crust... So, not wanting either of us to be disappointed, we ordered both! It takes a LONG time for deep dish pizza... When they both came, we were astonished that it is DEEP dish pizza! It seemed like a lasagna more than like a pizza. I actually liked Lisa's pizza better! The locals told us later that it is always better the next day after sitting in the refrigerator all night. I'm not convinced!

Next, since Lisa and I are fascinated by cool buildings, we went to the presentation center for the Chicago Spire (also see). This condo building will be 2000 feet tall hand hold about 1100 homes in it! It is a really novel architecture right where the Chicago River meets Lake Michigan. It will punch up on the Chicago Skyline about 500 feet taller than the Sears' Tower and be the tallest building in North America when it opens in 2012. The construction is just about at ground level now. Each of the 1100 (or so) homes has a different floor plan since the tower gets gradually smaller as it rises and the window (and structural) placement gradually rotates (about 2.5 degrees per floor). It was a BLAST to see the project even though we are currently quite content with our condo home in Seattle that we are planning to move into next year (and the commute to Redmond from downtown Chicago would be harder than from downtown Redmond). Still, it was fascinating checking it out and we will certainly remember on subsequent visits when it is a world famous landmark!

After leaving the Chicago Spire offices, we walked to Navy Pier which is wonderful public pier extending into Lake Michigan with amusement rides and boat tours and much more. There were lots of people just having fun in the sun on a beautiful day. Soon, the roar of the jet engines told us that the Blue Angels were in town! Now, we had just spent an afternoon watching the Blue Angels less than two weeks ago! Also, Lisa had bumped into the pilots at the Washington Athletic Club where we exercise. The WAC is where they stay when in Seattle and Lisa had a wonderful conversation with these fine servicemen. So, when we saw them flying over us again, it was just a LOT of fun to watch them practice on Thursday.

By 3:30PM, we needed to start our walk back to the hotel as we had 6:00PM dinner reservations at Charlie Trotter's Restaurant and needed to get all dressed up!

So, Lisa had heard about Charlie Trotter's and we were told it was one of the finest restaurants in the world. Now, you never know how that should be interpreted but we have to say, it was phenomenal! There was a seven course menu with wine pairing. A coat and tie is required and it was my first opportunity to try out my new suit (since the old one from 18 months ago was too loose).

We had the most amazing food and had a wonderful time! They have two seatings (6PM and 9PM) and by 8:30, we were heading back to the hotel. Unfortunately, Lisa was getting a headache so she crashed for the night. I shot a text message to the groom but the recently arrived gang was still at dinner so I walked and explored the beautiful city in the Magnificent Mile area on an absolutely lovely evening. The place was abuzz with action and it felt SO good to walk and burn some calories.

Soon, I was meeting some of John and Helen's friends and family and we were out partying! Helen's friend Eva is currently living in Shanghai. Most of John's friends are from Washington DC but his family (six brothers and five sisters) are from Philadelphia. I met Mike and Karen (from Austin), Reza (from DC), TJ and Emily (from DC), and John's brother, Tony! John, Tony, and I closed the place down and had a blast!

Friday, August 15th -- Sushi Lunch, The Bus Tour, the Boat Tour, and Meeting more friends!

So, I slept in until 10AM (Lisa had popped to life at around 8AM but let me sleep). We started by tracking down sushi for my lovely (sushi-loving) wife. We had a lovely lunch and the set out to figure out what to do. We decided to hop onto one of those double-decker busses where you can sit on top, see a LOT, and get a sunburn. We did all these!

We rode around on the tour bus for two hours in the sun and saw SO many fascinating buildings, parks, the Chicago River, and more.

We decided to bail out near the Chicago River and walk down to Grant Park. This would only lose us the last mile of the tour loop and that was the Magnificent Mile of shopping and we had already walked that. As we crossed the river, we saw a tour boat on the river and decided that would be more fun and hiked over to Navy Pier (where we already knew they originated). So, off we were on an "Architecture Tour" from a boat up the Chicago River. This time, it was an hour in the sun and again, snapping photos like crazy! Here's some of the best!

We had promised to get back to the hotel by 5:30 to rendezvous with some of the folks flying in from other places. That evening was the rehearsal dinner and, while we weren't in the wedding party, we were hanging out with some of them, their spouses, etc, while the rehearsal itself happened. It was a BLAST. Soon, we were running off with our new friend, Reza, to grab a steak a couple of blocks away while the wedding party did the dinner together. We came back to the hotel later and had drinks with everyone until we bagged it at around 1AM. Again, an absolutely LOVELY day of adventures, partying, and nice people.

Saturday, August 16th -- A lovely lunch and an AMAZING Wedding!

So, again we slept in and rested. We got ourselves going at around 10:30 and knew we couldn't be too adventurous because we needed to be in formal clothing and ready for the wedding by 3:30. We decided that the Chicago River was just TOO cool and so we hiked the mile down Michigan to the River and started exploring for a restaurant along the river to sit for lunch. It took a little while but we found a wonderful place called Flatwater and had a GREAT lunch. Walking back, we explored the hotel lobby for the new Trump Tower along the river. Having an OPEN hotel while major construction is going on 90 stories above is really cool! But there they are open while the building is being built! We walked back and checked out the Fourth Presbyterian Church of Chicago which was only a block away from our hotel. It is a beautiful 140 year old church!

Here's the building with our hotel, the Ritz-Carlton.

And, right next door is the John Hancock Center.

Soon, it was back to a shower and getting the fancy clothes on.

We showed up at around 3:30 for the 4:00PM wedding and it was gorgeous! Our new friends were arriving and soon we saw the groom (our neighbor, John) and his six brothers. There was a string quintet playing while everyone was seated and a trumpet player just waiting for the "Here Comes the Bride" procession... that's ALL the trumpet player did!

The wedding itself was short and beautiful! Soon, we were outside the room for appetizers and cocktails while the wedding party had jillions of photos taken and the room was reset for dinner. It was wonderful!

We sat at dinner with some of John's family (a sister, nephew, niece, and nephew's girlfriend). Let me clarify... the nephew was 30, built like a mountain, and back from his second tour in Iraq. They were fun! There were also some of John's good friends from some of his jobs in the financial industry. During the (lovely) meal, there were four musicians playing traditional Chinese music while we ate. It was amazingly cool.

Then, out came the THIRD group of musicians, a 9 piece orchestra to do the dance music. Helen has spent years doing ballet and John and Helen did the most impressive "first dance" I have ever seen. It was elegant, graceful, and charming. The singer from the musicians said she had seen a LOT of first dances and that was the classiest. Then dancing began, the wedding cake was served, and more and more dancing! Lisa had to (temporarily) abandon her beautiful (but painful) high-heels in order to dance!

Our new friend, Reza, caught the garter belt!

We managed to dance on and off until about 1AM. My new suit DEFINITELY needed dry cleaning and we had a wonderful time. It was just one of the classiest and most fun weddings I've ever been to.

So, back to the room by 1AM... Packing... Too little sleep... Catching a ride to the airport by 9AM... Change planes in Denver and glad to be home! Happy (and tired) to be at the gym by 6AM Monday morning... Between the cruise and Chicago, I need more time at the gym!

A trip we won't forget! Chicago is a great city!

-- Pat

Confidence in the Cloud

Pat Helland — Tue, 02 Sep 2008 02:13:37 GMT

So, one morning a few weeks or so ago, my colleague, Ruwen Hess, stuck his head in my door and commented on an Article on Gmail and User Confidence he had just read. He also pointed out a very interesting Article on the S3 failure from July 20th (which I had seen before). Now, before I start commenting on the point of this blog post, I want to say that this is NOT meant as a slam on our competitors. This is a complex and nascent space in which we are all learning. I want to complement my former company, Amazon, on the openness exhibited in their discussion of the event of July 20th. It is a new thing for Amazon to explain when stuff happens and I think it is great for the industry and great for Amazon. Google is just fighting the good fight to provide highly available services. They are both fascinating companies and worthy competitors. Still, there are some interesting aspects to this publicly available news.

Of course, this leads my mind down too many different paths. This was going to be a short post but I always get carried away! Let me cover:

Some Observations about Reliable Process Pairs,
Less Is More,
N-Version Programming,
Availability Over Consistency,
Eventual Consistency,
Front-Ending the Cloud, and
It's Going To Be a Fun Ride!

Some Observations about Reliable Process Pairs

Let me start by discussing independence of failures (and the need to focus on it to achieve high availability). During the 1980s, when I was developing system software at Tandem Computers, some of our important programs were process pairs. In this scheme, the same software ran in two different processes on different computers within the distributed system connected via messaging. The goal was that if only one of the computers (and, hence, processes) failed, the other would "take-over" and continue offering service. Of course, it was essential to be able to restart a backup and fill it up with sufficient state to continue the computation. Specifically, every failure and take-over created the need to restart the other process (which used to be the primary). Restarting implies the filling of the new backup with enough state to be ready to take-over.

During this time, I was privileged to be able to see a number of different implementations of process pairs within the Tandem System and, indeed, to do major development and implementation on a large process pair (in addition to a number of other pieces of software with different approaches to fault tolerance). As I lived and breathed the triage of product support, crashes, dump analyses, patches, and major product upgrades, it became clear to me that there was a pattern. Some process pairs were implemented to ship entire data structures. Some were implemented to send very narrow and minimalist descriptions of an operational change in state (e.g. in the transaction manager: "We just advanced transaction-X to the stage of flushing the log records to disk"). In a different example, one process pair would send a message containing its entire address space of data to its partner whenever a change occurred! Lots of lively debates raged about the best way to implement reliable process pairs.

I observed that the implementations which sent the minimum amount of data seemed to be the most resilient. There were many crash dump analyses that showed a process running along and it would trash some data structure. Then, a checkpoint would copy the data structure to the backup process, including its corruption. (A checkpoint is the message used to communicate state from the primary process to the backup process in a process pair.) Soon, the corruption caused the primary process (and its computer) to crash. Now, the backup (with the corruption received from the checkpoint) takes over to do the work of the process pair. Before long, the backup gets tangled up because of the corruption in the data structure and IT falls over, too! The entire fault tolerant system has succumbed to a single failure because there is insufficient isolation between the parts.

Less Is More

Communicating less information within a message is usually best. If you send extra stuff, it can cause corruption!

The failure of Amazon's S3 on July 20th is fascinating and, yet, very typical. S3 uses a Gossip Protocol to spread state around the network without any master. The knowledge is simply propagated to anyone that will listen and, very quickly, spreads around the network. Unfortunately, there was a bit of poison introduced into the system. Now, it just so happens that this was introduced because Amazon's MD5 hash protection of their messages did not cover some of the system status bits, but that is not my main point. All (non-protected) data communications will occasionally have transmission errors and, unfortunately, the stuff being gossiped by S3 was not protected. The article simply describes it as system status state.

I know that in these complex distributed systems,it is essential to keep somewhat accurate track of the state of the system. What is not clear from the Amazon S3 article from July 20th was the shape and form of the system status information and why it did not settle out automatically after the corruption. The article mentions that the system status state is given a higher priority and was starving out the real work as it was gossiping about the system state.

The main point is that when you are communicating information, it is essential to keep the information flow as sparse as possible.

While S3 is a brilliant system which is designed to continue functioning during a data center outage, it had a flaw (which I'm sure is now fixed) which allowed for some bad state to propagate to ALL the data centers. To eliminate the problem, all the data centers were taken offline and then the entire system was restarted.

Let me reiterate my deep respect for the engineering at both Amazon and Google in addition to what I see at Microsoft. Still, as the industry matures, we will all learn by the school of hard knocks to keep the information we spread across failure units as crisp and concise as semantically possible.

N-Version Programming

I've never personally experienced it but have read about uses of N-Version Programming. In this scheme, specifications of a program are drawn up and presented to N different teams, each of which will produce a version of the system. Now, if you develop 3 versions of the system, you may then implement a voting scheme in which when there are two disparate answers, you pick the one with two votes. The concept is to reduce the probability of a software bug causing the entire system to fail. Of course, one of the challenges in this approach is the development of comprehensive and clear specifications.

The reason I think of N-Version Programming is that it involves the "Less Is More" principle. To do N-Version Programming correctly, you need to isolate your development teams and prohibit any out-of-band communication. If the teams communicate (other than through the specifications), it increases the chances of propagating some corruption.

Availability Over Consistency

I am not aware of any information from Google about the cause of their outage but, again, we as an industry are still learning how to keep ever more complex applications and services available. The article cited above, expresses the dismay of many users as a service they've grown to depend upon is simply not there for a while.

In many cases, the user would gladly take a "good enough" answer NOW rather than wait for the "correct" answer LATER. In fact, it is MORE common to see users just want to keep going. Right now, I am typing most of this on the bus with Windows Live Writer inserting notes to myself about the hyperlinks to fix when I am back online... It's great to just "keep going" even with a reduced experience!

Amazon published a wonderful paper on Dynamo at SOSP 2007. This paper provides an excellent overview of the Dynamo storage system (which I had a role in encouraging from the sidelines -- I can't take credit for it). Dynamo is in production with at least two running services and numerous fascinating techniques employed in its implementation. I would encourage you to read the paper.

The reason that I raise this here is that Dynamo provides availability over consistency. In a distributed system, it has been proven that Brewer's CAP Conjecture is true... Hence, it is now called the CAP Theory. The idea is that you can have only two of Consistency, Availability, or Partition Tolerance. You have have a consistent (and by this, the idea is a classic transactional ACID consistency) and partition tolerant system but it may not be available under some partitions. Alternatively, you can have an available system which tolerates partitions but it won't have the classic notion of consistency. Increasingly, I see applications designed with looser notions of consistency. Most of the time, customers really want availability at the expense of classic consistency! New means of expressing looser consistencies are emerging to provide availability even when failures occur!

I learned over 20 years ago working in transactional systems to ASK a customer what their priority was when dealing with an outage. Indeed, some customers wanted you to ensure that every transaction was correct before bringing the system online. At first, it shocked me to learn that many, many customers wanted the system up even if the results of the work might be somewhat "less than perfect". Indeed, I saw this in some banking systems with humongous amounts of money being pushed around. It wasn't that they were doing anything wrong... if the system came up, most of the transfers could be accomplished and overnight interest gathered for them. The funds involved were large enough that hundreds of bank workers would simply stay up all night and verify the accuracy of the work, cleaning up as necessary. The timeliness mattered more than the accuracy!

Again, more often than not, availability matters more than strict (classic) consistency!

Eventual Consistency

Also fascinating (at least to me) is the area of eventual consistency (and, again, this term implies a LOOSER form of consistency than classic ACID transactions).

The basic idea is that it is OK to diverge opinions across replicas. What is needed is a system that coalesces when communication is reestablished between the replicas. I talked about this in my talk The Irresistible Forces Meet the Movable Objects. I am bringing this up because (I am arguing) that people really want availability over consistency. When divergent changes are made while disconnected, they also want the messes to clean themselves up as much as possible.

It is my opinion that we will be designing systems to support eventual consistency. Part of this trend will be to back away from our traditional separation of storage (e.g. database) from applications. In my opinion, we will evolve towards integrated solutions that combine the storage with the execution of the application in a way that will make it easier to redo the operation by the application and create eventual consistency across different executions of the app. This is much easier to do than to cope with reorderability of write operations against a data store.

Front-Ending the Cloud

So, I love and believe in cloud computing. I think the entire space will grow and evolve. The cost structures for many companies will make it very cost effective to use big data centers for hosting lots of their computation. As I look at data center cost structures, it is clear that it is going to be a competitive business with many advantages to large data center managers with large economies of scale. In a handful of years, most companies will look to offsite providers for their reliable servers.

Still, I wonder if there will be a market for local front-end systems running a replica of a portion of the applications needed to keep the business going. Not all the computation a company performs is "mission critical". In other words, the company can function with a subset of its computation (at least for a short while). Will there be an emerging space of loosely-coupled eventually consistent computation run either at the local company or at a different hosting company than the main cloud-purveyor? If there is an outage of the cloud services, this front-end would keep the business going. When reconnected, the results of the work would be shared and reconciled.

This is very much like the relationship between Outlook and Exchange. Outlook does a LOT of its functionality while disconnected and that is of huge value.

This may be an interesting economic point -- combining the cost savings of the cloud with independent failures of the front-end.

It's Going To Be a Fun Ride!

I am SO glad to be working in this space! It seems to me this is a great time for distributed systems geeks!

- Pat

The Emissary Design Pattern and RIAs (Rich Internet Applications)

Pat Helland — Sun, 10 Aug 2008 18:28:00 GMT

Here is a first draft of a new presentation. I gave it a couple of months ago just after TechEd and thought I would share it as I try to write up some of my thoughts on RIAs. I plan to rework this a bit more and present it again at TechEd Europe. The talk is titled: "The Emissary Design Pattern and RIAs (Rich Internet Applications)"

Abstract:

The Emissary design pattern was first described in 1999 in the old "Fiefdoms and Emissaries" talk. The concept of a "fiefdom" is very similar to what we today call a service in a Service Oriented Architecture. The fiefdom is a separate trust sphere and transactional boundary. An emissary is a prescriptive pattern for interacting with a service (or fiefdom) which leverages reference data and a deep understanding of the service to prepare requests for service and maximize the chance those requests will comply with the requirements of the service. An emissary may be richly interactive and anticipate the validation requirements of the service.

The emerging world of RIAs (Rich Internet Applications) is a fascinating blend of a classic smart client and a browser-based web application. In a RIA app, client code runs in the browser but still must comply with the browser enforced sand-boxing and not cause harm to the host client machine. Navigation, naming, linking, and much more are being defined in a fashion drawing from both the web style and the client style. Many of the design issues with RIAs are under discussion today as this support for these applications is emerging.

This talk examines both the emissary design pattern and the nascent space of Rich Internet Applications. It motivates how one can look to the workflow patterns contained in our parents' use of paper forms for workflow to understand the possibilities of implementing user-centric workflow as shared replicated data. The talk concludes with some preliminary concepts of a shared and declarative definition of the "paper form" model and its constraints and how these may someday be used in the automatic generation of emissary-based RIA clients.

More RIA stuff soon!

- Pat

SeaFair in Seattle!

Pat Helland — Fri, 08 Aug 2008 04:05:25 GMT

One last personal post and then I'll get to the work stuff. Last Friday we got off the ship... Saturday was moving the office with Bob... Sunday we went to a SeaFair party!!! I've lived in Seattle for 14 years and never really paid any attention to SeaFair but it was a blast!

So, SeaFair is a combination hydrofoil racing and airplane exhibition featuring the Blue Angels. It turns out that some friends John and Wendy have an amazing house overlooking Lake Washington right above the hydrofoil races. We were invited to a party, the weather was awesome, and we had a blast! I have to share some photos and our thanks to John and Wendy for including us!!!

All these photos were taken from their lovely deck!

- Pat

I Moved into My New Office in Building 35

Pat Helland — Fri, 08 Aug 2008 03:05:54 GMT

So, as I mentioned a few weeks ago, I am starting in the SQL team this week after the one week vacation. My friend, Bob, came over on Saturday with his pickup truck and we moved everything from building 42 to building 35. The Microsoft moving organization always gets confused when I move myself but there's no way a standard moving team can deal with my office. They did move my phone, though...

I am now moved into my new office which (this time) does not have a window... we're supposed to do an office move in a few months and I should get a real window. In the mean time, I have a substitute window so I can still have a window office. Also, as evidenced by the hallway outside, my office is definitely considered to be the place where a bunch of asses hang out.

It's great to be relocated into the SQL team! It was a long (seven hour) Saturday moving stuff! Thanks for the help, Bob!!

- Pat

A Fun Cruise to Alaska with Friends

Pat Helland — Thu, 07 Aug 2008 19:45:23 GMT

I'm starting this blog entry on the bus riding to work on Monday, Aug 4th... it's taken all the bus rides until today, Thursday, to finish. There's NO computer stuff in this post... This is a vacation travelogue!

Wow! We had a good time in spite of Lisa being seasick for 1-1/2 days and the weather in Alaska (mostly) sucking... Still, it was FUN!

First of all, it was lame of me to think that I was going to exercise every day on the ship... the exercise room was small and had no water or music. I worked out one day and went to the buffet lines the rest of the time. I am back in Seattle now and working hard to get rid of the two or three pounds I packed onto my butt... I weigh within two pounds of what I did three years ago (but more is muscle so I am in 36 inch pants for the first time since the 1960s). Giving away clothes when you get smaller is an important trick! When the clothes get tight, STOP EATING SO MUCH... My clothes are a little too tight after the cruise!

I did, however, succeed at my other goal of reading all 750ish pages of the last Harry Potter book. JK Rowling is VERY creative and the book held lots of surprises! I have a lot of respect for her work and enjoyed it immensely.

So, we started off Friday by meeting up with the other two couples, Bob and Rita and Kirk and Kathy, at our condo for brunch and then we caught cabs to the cruise ship dock which is TWO MILES from our home. It was literally a 10 minute cab ride. We can see all the cruise ships, including Rhapsody of the Seas coming and going from our living room. There are two or three cruise ships in Seattle every Friday, Saturday, and Sunday each week from May through September. Frequently, I am up at 5AM (at least on Fridays) to watch them arrive. In the evening, at around 5pm, it is normal for them to all pull out within minutes of each other. Most of them back out of the Port of Seattle and do a 180 degree spin right in front of our home. I call it the cruise ship ballet! Here are a couple of photos of the ship we took (Rhapsody of the Seas), taken from our home as it departed the evening after we arrived back in Seattle.

So, Friday (July 25th) was the day to arrive and unpack. We got on board at around 1PM and started exploring the boat with our friends. The luggage arrives over a few hours so you need to kill time a bit and then unpack. So, we discovered the Windjammer Cafe. This is the place where there are GOBS and GOBS of mediocre food, crowds of people jostling to pack their plates, and then (at some times of day), seriously insufficient tables to sit down and eat. Now, here and there, you can find something really good in those buffet lines but it is VERY hard to do appropriate portion control because you think you've found the right stuff and the right amount of food but then you walk over to the next stuff and there's something else you want to slap on your plate! Still, there is always something there and that's kinda' nice when you're locked up on the ship, you can always get something to eat.

Shortly before departure, we had the mandatory lifeboat drill... we all have to practice getting on our life preservers and going to our muster station where we can look at the 150 person lifeboats. It is a necessary hassle. Here's my wife, Lisa, looking beautiful in her life preserver!

Here is the view of Seattle as we were departing Seattle. This photo includes the building with our condo. It's the shorter one which is darker on the top half in the center of the photo.

Next year, we're moving to a different building only one block from Pike Place Market. Here's the view of it under construction with the yellow crane in front. Both of us LOVE downtown living and it was pretty darned cool to only take a 10 minute cab ride to catch an Alaska cruise!

I mentioned the "Cruise Ship Ballet" from Seattle. Three ships set off at basically the same time that Friday. Here is a Celebrity ship in from of us and a Holland America ship following behind us!

We went to the opening show and had a great time! Dinner was fun and we met one of the singers at the next table having dinner with her mother and aunt who were cruising that week. This set a pattern where we made a big point of sitting in the front row every evening and cheering on the performers... it was a blast!

As we cruised up the Pacific on the outside of the British Columbia and Southeast Alaska islands, the weather got a bit rougher... our stateroom was quite a bit forward and Lisa got seasick. Consequently, she stayed in the room on the second dinner (which was the first formal dinner). So, there were only five of us at dinner that night. Here's Bob and Rita (all dressed up)

and Kirk and Kathy (all dressed up)

and me (in a tux) posing with Kathy since Lisa was upstairs in the stateroom being miserable. She made it clear to me that I should go to dinner since if I stuck around while she was sick, I would annoy the heck out of her and make her MORE miserable... she was right!

On the third day, we pulled into Juneau and saw that it was in a beautiful location but it was POURING rain... Other than Lisa, none of us brought rain gear. It turns out to get to town you need to board a bus but the line was outside in the rain. We waited for a while and then set out to get onto the bus. Standing (and freezing), we decided to bag it and go back into the ship... That was NICE and WARM. I found time to read that last Harry Potter book!

Again, we very much enjoyed dinner and the show in the theater on the boat as we were getting underway from Juneau to Skagway.

The next day, we went to Skagway. It is a cute little tourist town which will get between 5000 and 10,000 tourists every day of the week during the 5 month season. We spent a couple of hours walking through the shops and exploring. It was a nice break getting off the ship.

Again, we really enjoyed dinner (casual attire) and the show... it became a lovely rhythm as we met each day to spend the evening in the dining room with friends and head off to the show!

The next day, we got up at 6AM... The original itinerary had us seeing the glacier in Tracy Arm Fjord. The Captain announced that we would not be able to get very close (like 2 miles away) from that glacier because there were so many icebergs floating down the fjord that he would need to stay way back for safety... Instead, he would take us down Endicott Arm Fjord and we would see a different glacier (Dawes Glacier) much closer up. To make that happen, the visit would need to be much earlier so... we all got up! Lisa and I met up with our friends in our stateroom to hang on our balcony... I didn't see it but I understand there were hoards of guests (many in their pajamas) on the 10th deck (with no wind protection). We had a great view from our balcony. Here are shots of the approach up the fjord and of the glacier as this huge ship did a 180 degree turn only a few thousand feet away. Our stateroom happened to be on the port side and so it had a magnificent view!

While in front of the glacier, the crew lowered a lifeboat so they could capture some pieces of floating ice. It just so happened that the boat they used was directly below our balcony! One of the pieces of ice was later carved to become a salmon and put on display.

We were right behind the bridge and saw LOTS of fun stuff with the glacier...

So, later that day was the second formal dress night. Lisa loves getting dressed up and (with her help), I think I am doing OK at it!

There was a very cool drum and dance act from Argentina and we had a blast with Bob and Rita at the show.

At the end of the evening, we found a cute folded towel puppy with Lisa's reading glasses on the bed in the stateroom.

The last stop on the cruise was Prince Rupert, British Columbia (necessary so that the ship makes an international stop). It was a pretty town but there's not much to do there... Lisa was thrilled to find a sushi restaurant, though!

So, this takes us to our last cruising day home through the inside passage. It was mostly a quiet and uneventful day (as captured in this photo):

We were taking the inside passage (going South or home to Seattle). This winds its way between Vancouver Island (which is about 400 miles long) and the mainland. There are beautiful views and narrow passages between the various islands. Coming home was the top line in this chart:

At lunchtime on that last day (Thursday), the ship was heading through a narrow passage while we were sitting down and eating. Now, Bob and I have done quite a bit of boating in our day (especially Bob who is a licensed captain). We were fascinated (but not worried) when the boat started listing to the starboard by what we thought was about 8 to 10 degrees. This went on for a minute or two and LOTS of desert plates fell, carts went rolling, and people were quite surprised! A couple of minutes later, the ship's captain announced that the ship was fine and not to be stressed.

Now, there is quite a difference between a roll (where the boat goes back and forth) and a list (where is stays for a while in one direction). When the seas get rough, everything gets stowed and you may get a roll (or a pitch) motion which can involve a lot of motion. When the seas are calm (like in the inside passage), stuff comes out from being stowed and it will fall if there's too much movement.

So, this was an UNUSUAL event! The cruise ship company doesn't like to have this happen. Both Bob and I could tell from the motion that we had been moving through a pass with a following current. The islands and other geography in the inside passage are a special area. As the tides change, you can have large bodies of water a few feet (or more) higher than what is lying on the other side of the narrow passage. Consequently, the water squirts through one way (landing onto some calm but lower water) for a few hours and then when the tides change, it squirts through another way. We could feel this in the motion of the ship.

Now, when a 79,000 ton ship does this, normally everything is lined up, it runs through the shoot and people barely feel a thing. Both Bob and I (sitting at lunch) notice the slight variation and were chatting about it. When we listed to the starboard (right), that got our attention as we could feel the rudder had gone over to make a turn to the port (left) and the following current was pushing the ship on the port side causing her to lean (or list) to the starboard side. This lasted only a minute or two and the ship recovered nicely.

Later that day, Lisa and Rita went to a Q&A with the captain and his senior crew. He described that he was definitely responsible (and on the bridge) while the pilot (a fellow NOT with the cruise line but who knows the local waters) was at the helm. As the Rhapsody of the Seas was shooting through the pass with a 5 knot following current (which is a very noticeable but reasonable current to sail through), a couple of fishing vessels cut in front of the Rhapsody of the Seas and the pilot needed to take evasive action to avoid hitting them. That caused us to make a slightly uncomfortable maneuver. He claimed we listed 5 degrees (although it seemed like more to Bob and me). Also, the captain said that was the worst unexpected listing in his 47 year career. Anyway, for a couple of boat-geeks, it was cool!

Finally, on the last night, we had a Las Vegas style juggling and balancing act (Justino and Daniela) who were great! It seems that the singers in the theater group set me up... I could tell that Justino normally doesn't call on someone from the audience but he was clearly looking for the bald headed guy in the front row. He came and shook hands. Then, a few minutes later he made a comment and asked me my name (and I told him "Pat"). Again, a few minutes later, he asks me to stand up and hands me three juggling balls with the comment "Would you please hold my balls?". Of course I said yes at which time he produced a long-haired wig for me to wear... I kept it on for the next 45 minutes since I certainly am not going to back away from a good joke (and I was VERY glad I didn't get asked to help juggle). When the singers came on about 15 minutes later, they were only a few feet above me and struggling VERY hard not to bust out laughing! It was fun!

Here's some pictures of me with Justino and Daniela after the show but before I returned their wig to them...

And then, the last evening's dinner and a goodbye to Rocky from Beijing who brought us cocktails with a warped sense of humor ("I'm the 'Boozer', not the 'Loser'!").

That night we packed and set the suitcases in the hallway. We awoke the next morning to see the Holland America's Amsterdam backing into her berth right in front of us and then a very familiar view of the Seattle skyline. It was GREAT to be back home!

I spent the rest of Friday, August 1st, reacquainting myself with my laptop and deleting most of the 538 messages in my inbox!

Saturday was the office move to building 35 and the SQL team (more on that soon).

It was a LOT of fun to just goof around for a week!

- Pat

Transferring to the SQL Team

Pat Helland — Wed, 23 Jul 2008 21:17:40 GMT

Since I returned to Microsoft in March of 2007, I've worked in the Developer Division on a project in the area of RIAs( Rich Internet Applications). This is a fascinating space in which lots of fun stuff is happening. I've been privileged to work on a wonderful team developing an upcoming project in this area. While I have had a blast and can see the impact of my work, my real passion is scalable distributed back-end applications and data management. Maybe I'm in a rut, but I've worked on variations of this space since 1978 and still find it fascinating.

For many years, I've worked with my friends in the SQL team on a number of different projects (both directly on the projects and by coming over and offering various opinions to rile them up). Now, it turns out there is a really special opportunity for me to join a team in SQL called SIA (Strategy, Infrastructure, and Architecture). This spot allows me to work on the business, technology, architecture, infrastructure, and more for one of the industry's leading products. It also allows me to participate in driving some of the new changes in scalable cloud computing (which is, of course, where I am extremely passionate). The leaders of the team and most of the folks working on the team are old friends that I thoroughly enjoy! Also, it is a natural part of the job for me to be communicating externally and, hence, you are still going to see me causing trouble at a number of Microsoft and industry conferences... Expect to see me at TechEd EMEA for sure!

I am heading out on an Alaskan cruise on Friday with my wife, Lisa, and two other couples who are dear friends. We will return on Friday, August 1st and then over the weekend my friend, Bob, and I will move my office to building 35. Needless to say, moving the office is a project!

While I am not going to disclose anything about the project I've been working on for the last 15 months or so, I do have a whole bunch of thoughts on RIAs. I plan to drive to get this thoughts out as blog posts over the first few weeks of August but I've decide to actually leave my laptop at home and not take it on the cruise! It's time for me to read the last Harry Potter book...

I want to thank the Developer Division, my bosses, Shanku Niyogi, ScottGu, and Soma for the wonderful opportunity and blast it has been working in DevDiv. Also, I can't forget to thank all the wonderful members of my team who will remain good friends!

Talk to you soon!

- Pat

A Weekend at Foo Camp

Pat Helland — Thu, 17 Jul 2008 01:29:03 GMT

Well... I was lucky enough to get invited to Foo Camp (which was last weekend) and I figure "What the heck! Let's do it!".

Foo Camp is held at the O'Reilly headquarters in Sebastopol, CA which is 1 to 2 hours (or more depending on traffic) north of San Francisco. It is an invitation only event whose name stands for "Friends Of O'Reilly" and involves about 300 diverse and interesting individuals from different walks of the computer industry (and related industries). Tim O'Reilly host it at their headquarters and supplies very nice buffet food, showers, rest rooms, and meeting rooms. I was informed that the best way to enjoy the event is to camp there which means either pitching a tent on their lawn or finding an available space in a meeting room or hallway to throw a sleeping bag. It was an option to get a hotel room in town. Now... I haven't camped in about 20 years so this required some thinking...

I concluded I had two options:

Stay in a hotel room and ensure I remained sober enough to drive at the end of the evening, OR
Buy a tent at REI in Santa Rosa and then hit the high-end liquor store in Santa Rosa to buy enough whiskey to lubricate a serious subset of the 300 attendees.

Naturally, I chose the second option and bought six bottles (some of my favorites and others I hadn't tried).

So, I flew from Seattle to SFO late Friday morning (July 11th). After picking up a car at Avis, I hit the road north (and the traffic) at around 1:15PM. It took until about 3:30 to get to Santa Rosa and the REI camping store. Buying a tent was an experience, too. REI doesn't have any cheap tents but, boy, do they have some nice ones! So, I ended up with a two-man tent which, in reality, was exactly the right size for me, my duffle bag, and the box of whiskey. By the time I'd hit the REI, the liquor store, and driven all the way to Sebastopol, it was about 5:30 and things at the O'Reilly campus were really hopping.

The campus has three buildings and is the regular workplace for quite few people. Naturally, some of the areas were labeled as not for the Foo Camp attendees since not everyone wants someone sleeping in their office filled with all their things. Lots of the space was graciously made available. A large circus-style tent was set up for dining and around the back of the campus were a number of tents with chairs set up for meetings. It was really very well done.

So, the next job was pitching the tent (which is NOT my forte). The lawn had been pretty well staked out by the time I got there so I went around to the side of the dining room tent and found a spot between the big tent and the sidewalk. This made me the closest tent to the beer and wine which really wasn't a problem for me (I brought earplugs) but the WELL lit sidewalk meant my tent was, well, WELL LIT. This, too, didn't phase me but obviated the need for the flashlight I'd scored at REI. I was surprisingly successful at tent construction and soon moved into my new 54" by 80" abode.

At dinner on Friday, I saw my friend Jesse Robbins who (I believed scored me the invitation to Foo Camp. I met Jesse during my tenure at Amazon and he is one of the most knowledgeable people I know about Disaster Management. Jesse has been trained as a firefighter and is ALWAYS thinking about how can a data center and its applications survive failures. It was good to see him! At dinner, I met Adam Jacob who is working with Jesse at a new gig called HJK Solutions which is working to help companies develop their operations. Cool stuff! I also met Brady Forrest of O'Reilly. I had known of Brady because of the work he has done in running various conferences.

So, the format for Foo Camp is pretty informal. There is a large board set up that has columns for rooms (or tents) and rows for time slots for meetings. People rush over and slap up a topic that they want to present. Partnering with people of like interest is encouraged. It is also encouraged to reduce your one hour time slot to 30 minutes if you think your topic is best served by a shorter period. The entire conference is self-organized and pleasantly quirky.

The first evening was random discussions as people drifted in before, during, and after dinner. There was beer and wine and lots of fun people to talk to. After a while, I realized I was tired and set off to find a place to brush my teeth. Having succeeded with that, I crawled into the sleeping bag at 11:30.

So, camping is (sort of) OK... One thing I remembered from my youth but, now, vividly remember is my LEAST favorite part of camping. That's when you wake up in the middle of the night and REALLY have to go pee. Now, to politely accomplish this, you have to successfully unzip the sleeping bag but the zipper always gets stuck. Then, you have to put on enough clothes to avoid getting arrested or, worse, hit on by a Foo Camp attendee. Don't forget your shoes as you are about to search for a Men's Room... yuck.... Finally, you have to unzip the tent flap AND the rain cover for the tent and crawl out on the lawn. Hopefully, you remember the location of the facilities in your exhausted stupor. After finding the Men's Room and returning, you have to undo the entire process in order to get back to sleep. I hate this part!

On Saturday morning, I was pretty close to the first one up since I was seriously wanting to take a shower. Remember, there are a couple of hundred people competing for one men's and one women's shower. Unlike at most baseball games, the ladies have the advantage in contention for scarce resources at Foo Camp. So, I scooted into the shower at around 6:15AM (not too bad since I crashed at 11:30).

The sessions started at 10AM and I went to a number of excellent ones. Sometimes, it was hard to select between the various cool sessions. I remember the following ones:

Data center power -- James Hamilton (my friend from Microsoft) and Jeff Hammerbacher who leads the Facebook data team (but will be leaving soon). Both James and Jeff were filled with information about running large and dynamic data centers. The power issues for data centers have been on my mind the last few years and I find that James is a wonderful font of knowledge. I most definitely love that he is at Microsoft and my friend... I plan to come pepper him with additional questions in the months to com. Jeff, also, has tons of knowledge from supporting the data needs of Facebook as it has undergone its explosive growth. This was a fun and invigorating discussion in which I met an attendee, Roger Magoulas who is a research director at O'Reilly. I have a feeling that there will be opportunities for me to work with Roger, too.
Parallel Programming -- Kerry Hammil of Microsoft Research. We had a fun discussion of the difficulties of getting applications (and, indeed, their libraries and OSes) to be parallel. There were about 25 great and interesting people participating in this group and, not surprisingly, I participated, too. This was such a lively discussion for me that it ended up in the hallway and we skipped the next session.
Beyond Relational DB -- Emil Eifrem (of NeoTechnology) and Terry Jones (of Fluid Technology) -- These were really two different folks with two different reasons for wanting to build a datastore that is different than relational. For NeoTechnology, it was about making an optimized graph representation of data. By custom designing for a graphs, they can do specialized queries quite fast. For Fluid Technology, they are allowing data to be represented in a fashion where annotations (or additional attributes if you will) can be stored by any observer of the data. Basically, if you can look at an entity, you can add an attribute which is then stored in YOUR store with your privileges. Filtering of the entities can be done based on whatever attributes you can see (ones you've stored or others you can see). Both systems are interesting. I had a number of discussions during this session (and afterwards). During this, I met Esther Dyson (who is an investor in Fluid Technologies and quite a fascinating person). I also met Jeff Jonas (who I very much enjoyed, too).

Next, came dinner... Actually, what happened was Esther, Jeff, and I slipped away to the big dinner tent and Esther spent some time online showing us a demo of a startup she is on the board for called 23andMe. This is a fascinating company which makes its money by allowing you to pay them for a genetic analysis. This analysis can yield information about how closely your genes are like those of your family. Esther had a large number of family members whose genes were processed so they could be compared to each other. We spent some time looking at the variance and similarity of siblings, aunts, parents, half-siblings, and more. It was very interesting. This then led to a discussion of the implications of certain markers that are related to different disorders. What do the genes mean? When should lay people gain access to information about their genetics? How can it be presented to people so they understand (and do not react improperly) what a certain pattern of genes will mean for them? Apparently, some states (I think California but may be wrong) are demanding that genetic information only be made available to someone via their doctor. What are the tradeoffs associated with that?

Continuing over dinner, I got a chance to hear more of Jeff Jonas's work at Systems Research and Development (SRD) which was acquired by IBM in January 2005. Jeff's interest is in processing VAST quantities of information to see surprising correlations. He applied this technology to help flesh out folks that were trying to cheat the casinos in Las Vegas (which is where he lives). Later, it was tapped by the folks at Homeland Security to find patterns associated with folks trying to do acts of terrorism. Jeff is very interesting and I hope to get the chance to work with him more.

So, after dinner at about 8:30, I figured it was time to break out the six bottles of whiskey (two scotches and four bourbons). There was a fire pit and that seemed like a great place to set up. The beer that the nice O'Reilly folks had supplied was brought over and soon we had a party going. One at a time, three other whiskeys were produced by other volunteers. A few hours later, I noticed some Gin and Tonic had been added. Since there were about 300 people, this was not an overwhelming amount of trouble. Also, a walk around the campus revealed many different people involved in different passions. There was one group in a meeting room with about 8 different musical instruments just making wonderful sounds. Others had people doing puzzles (I think my friend Pavel Curtis was one of the ring-leaders in that effort). Others were playing various games and I did notice some hula hoops in play. Someone had set up a croquet mallet, a ramp, and a rack to break old china... this makes a MARVELOUS sound whose effect is enhanced by whiskey. The following photo was taken on my friend Jesse Robbins camera by another attendee. You can see James Hamilton, me, and Jesse by the fire.

Check out James' blog for his perspective of the event. (Picture stolen from James' blog and, in turn, from Jesse's Flickr site).

So, I partied until about 2AM. I remember long and interesting discussions with Mark Fitzsimmons and then later on getting to know some of the Microsoft Research folks better including Lili Cheng, Jennifer Chayes, Christian Borgs, and Linda Stone (now ex-MSFT). Just an amazing group of people that I would love to get to know even better!

Then, it was back to the JOYS of the tent and the middle of the night zipper-fight. I managed to sleep until 8:30.

The next morning, it was waiting in line to shower and then I had to pack up all the stuff! I got all done except the sleeping bag and tent when I went off to see a talk by Terry Jones on the calculations to track the changes in Flu virus genetics and the challenges of creating an effective vaccine. Esther Dyson (who is both funding Terry's company and working on the board of the genetic company 23andMe) was there for the discussion. It was (and is) fascinating the extent to which data analysis of LOTS of information is having a profound impact on something as surprising as the design (and prediction) of the contents of the annual flu vaccine so as to create immunity to infection for what the scientists THINK (and hope) is going to be the dominant mutation of the flu virus. It was VERY interesting and Terry's knowledge was very impressive.

So, that morning, Esther realized I was heading to SFO asked me to give her a ride. What a treat to chat with here for a few hours! I scrambled to pack up the tent and give it away to a nice couple that are Foo Camp regulars... they keep a stash of supplies for later Foo Camps. I asked them NOT to keep it for me. Having discerned that there is at least one taxi in Sebastopol, if I get invited back, I will exercise a THIRD option:

3. Hit the high-end liquor store for whiskey, cause trouble on the second evening, call a taxi, and stay in a local... I think I'm spoiled!

Esther and I had a lovely chat and covered our respective backgrounds and passions (she is into space flight amongst other things). Quite a lady and quite an enjoyable drive to SFO. I dropped Esther off, returned the rental car, and managed to get an earlier flight home that I had reserved (which was GREAT since I needed the sleep and was happy to see my wife, Lisa!).

I hope I get the privilege to attend this eclectic and fun event again (but without the tent).

- Pat

Foo Camp was fun! Thanks to Tim O'Reilly for having me!

- Pat

Controllers and Doers

Pat Helland — Sun, 06 Jul 2008 22:05:00 GMT

First of all, I've put off writing this for a while since I wanted a term for "controllers" that does not feel pejorative. I really mean this characteristic in a healthy and natural fashion!

So, it's my belief that the people working in IT fall into two broad categories (with occasional cross-over and hybrids). Folks generally are either:

Controllers -- These people struggle every day with the overwhelming number of applications, data sets, schemas, projects, and the chaotic interaction between them. They strive to bring order to the chaos and rationalize how all this crap can work together. They are fervently backing the enterprise-wide schema rationalization and struggling to get the architecture team to be successful in its attempt to unify many different aspects of the application install base. Part of this effort is to unify the logging, monitoring, and management of all the applications within the enterprise. As a first step in that, they support the ongoing project to create an accurate inventory of the various applications within the enterprise. It is always a work in progress!

Doers -- These folks see an important business imperative that requires application support. They WANT to work within the framework advocated by the Controllers but it seems very challenging to meet their business constraints while dealing with all the overhead (and apparent confusion) of doing things compliant with the Controllers' constraints. Sometimes, they comply. Other times, they say "Screw It" and just do their application changes the way they want to do them.

So, this dichotomy seems stable just like the American Two-Party System of Politics seems to be stable. The Democrats and Republicans always seem to squabble and that clarifies the argument. Over time, the center viewpoint migrates (and American political views are surprisingly homogeneous -- just ask any non-American). The existence of the debate (and the tension it causes) drives the system towards a new balance over time. Similarly, the Controllers and the Doers move the center of gravity for the IT organization -- in both its centralized capacity and its per-department capacity. The proclivity of the Controllers adds some small sense of consistency (which is of tremendous value) while the business demands of the Doers keeps a spark under the IT organization's butt.

Some observations:

When I was a kid, the Controllers had great big mainframes and multi-year waiting lists for application development. The Doers bought mini-computers using their departmental budget and really annoyed the Controllers.
Later, the Controllers were using minicomputers (usually running Unix) and the Doers were building PC applications! Some of these were even built in Excel and met the business needs just fine (even if the data wasn't properly "Controlled" which did pose challenges).
Now, the PC is "The Man" and has moved into the mainstream of the world of the Controller. The Doers are using lots of different tools including mash-ups!

So, when I look around Microsoft and other vendors, I always ask myself "Is this product for the Controllers or for the Doers?" Different divisions and groups within the various vendors build stuff targeting the two constituencies. Neither is wrong and neither is right. The tension and the balance seems to be stable while the manifestations of each groups efforts evolves forward relentlessly. What a hoot!

- Pat

Link to the Video of the "Irresistible Forces Meet the Movable Objects"

Pat Helland — Sat, 03 May 2008 03:07:56 GMT

A while back, the fine folks at TechEd EMEA sent me a link to the video of me presenting last November... I screwed up and didn't blog this.

The powerpoints are at Blog of the Powerpoint of Irresistible Forces and the PDF version of the powerpoints are at Blog of the PDFized version of Irresistible Forces.

To run the video, click on the talk title below.

- Pat

The Irresistible Forces Meet the Moveable Objects

Knowledge and Wisdom -- A Tribute to Jim Gray

Pat Helland — Sat, 26 Apr 2008 03:48:00 GMT

I've been busy with a number of things; one of them is helping to prepare for the official Tribute to Jim Gray. There will be an all day gathering at UC Berkeley on May 31st (see here). This will include a general session (in which I am honored to have a small presentation) and a technical session covering many of the aspects of Jim's technical contributions to the industry. There's still time for you to attend if you can make it... please sign up!

As a part of the tribute, it was decided that a special edition of the SIGMOD Record would be published with papers matching the presentations done on May 31st. I helped wrangle all the papers together working with the presenters so this could come together.

So, I asked to present about Jim as a Mentor in Industry and that led me to write down my thoughts in a two page paper to be included in the special edition of the Record. I wanted to share that with you. It it titled: "Knowledge and Wisdom" and the abstract (and pdf) are below. I miss my friend...

-- Pat

Knowledge and Wisdom - by Pat Helland

Jim Gray was my mentor and friend for almost 25 years. Even before we met, he was changing my life through his writings. Jim continually and patiently nudged me to strive for more, accomplish more, and to pass these gifts on to others. He embodied those attributes that I most want to emulate.

Jim influenced those around him both by sharing in his immense knowledge and by listening and guiding. Even more, he created a community that fostered growth and sharing across competitive boundaries with a clear ethical compass. This paper summarizes my experiences with Jim's approach to nuturing and mentoring.