Over the last several months in my role here in China, I have given talks at several leading universities and met with many of the leading faculty and students working on technologies related to the Data Platform. I’ve also spoken at several industry conferences, meeting with customers, partners, analysts and other industry folks. There are many topics that come up at these meetings – changing technology trends, distributed development, the tremendous growth of Asia etc. But one topic that seems to come up more than almost any other is the question of how we organize and conduct our product development in Microsoft. I suppose this is only natural – Microsoft is one of the most successful software companies in the world, and the software industry here in this region is poised for tremendous growth, so it makes sense that people in the industry are eager to learn from the our experience over the last quarter century.

This is actually a very big topic and within Microsoft we have an Engineering Excellence group that actually runs courses that can span several days and provide an overview of Microsoft’s software development methodology, our engineering system, organizational structures, best practices, tools and technologies we use internally ensure quality, reliability, security etc and a variety of related topic. By no means would we claim that we have all this figured out perfectly and have a perfect system, but there is indeed a lot of accumulated knowledge and experience that we can share. And we do actually share this information, in appropriate form, with others in our industry, worldwide and also in this region.

As this is indeed a large topic, I don’t want to get too deep into this here, but I do want to address one aspect of our engineering system – the core disciplines that we organize our R&D teams around and the particular roles that each of these disciplines plays. I want to discuss this because I believe Microsoft does this a little bit differently from the rest of the industry even in the US, and especially here in China there is not a good understanding of these core disciplines and what role each of them plays.

Traditionally, the Microsoft engineering system has consisted of 3 “core” disciplines: “Development”, “Test”, and “Program Management”, also known as Dev/Test/PM for short. I’m going to touch on each of these briefly here, but I like to introduce them in a different order:

PM: When we think of engineering disciplines, most people start with “Dev”.  For me however, things really start with the Program Management discipline. At Microsoft, “PM” means many different things, but for me the core essence of the PM role is two things:

1.       The first part of the PM’s job is to understand the customer’s requirements and translate that into a functional specification of what we should build. This is where it all begins. If we don’t understand the customer, it is not very likely that we’ll end up building the right thing.

2.       The second part of the PM’s job is to work with Dev and Test to translate the initial specification into a living, breathing product.

I find that many people, especially here in China, think “Project Management” when they hear PM. Indeed, Project Management is part of a PM’s job (under #2 above), but it is only a part of the PM’s job. The real skill that a PM brings is the expertise to listen to customers, understand the world from their point of view, and then to design a solution for their problem. This does not just mean giving customers what they ask for literally, but to truly understand them and design a solution that solves their problems even if the customers could never imagine the solution – as the famous saying goes, if we had only listened to customers, we would have looked for a faster horse, not come up with the automobile.

Dev:  Of all the engineering disciplines, this one is probably the one people think about the most commonly. Dev is short-hand for “Development”, the folks who responsibility it is to actually design and build the software that we ship. The essential job of Dev is to take the functional specification produced by PM and translate that into an actual implementation. In the world of mission-critical system-level software, this implementation better be extremely reliable, secure, manageable, scalable and high-performance.  And the designs and implementations Dev produces better stand the test of time and last for several versions and years to come.

Test: The test discipline in Microsoft is much misunderstood, certainly externally, but sometimes internally as well. When I first came to Microsoft many years ago, I was (pleasantly) surprised to find that Microsoft had almost as many, if not more, testers as developers. Coming from a company that had a much less developed testing discipline (and where as a result, quality assurance was considerably weak), it took a little while to get used to what the essence of the Test discipline really is. The reality is that, in Microsoft, how fast we can ship software depends on not how quickly we can design and implement it but rather on how quickly we can test it. This is because every piece of software we ship, especially on the systems-software side, has to pass an extremely high quality bar. The Test discipline is really an complex area, and one where have learnt a lot over the years in terms of different types of testing that we employ – unit tests, functional test, integration tests, stress and long-haul tests, performance tests, security tests, localization tests, etc. etc. The set of tools and techniques we employ in test is truly some of the most impressive and complex – automated test harnesses, automated test generators, automated test failure analyzers, automated security “fuzzers”,  fail-point and state-machine based testing.

The three “core” engineering disciplines described above are like the 3 legs of a chair – you need all three of them, and in a balance, to have a proper engineering organization. No one leg can dominate the other – otherwise, you get an organization that may not be in touch with customers needs or one that does not pay enough attention to quality. Indeed, the three disciplines are a little bit like the branches of government – they form a system of checks and balances that ensures we understand what customers want, we design and build that with high quality, and we ensure that we deliver a product that meets customer expectations in every regard.

It is also important to emphasize that we aim to attract the best talent to all three core disciplines – the bar is equally high for all the disciplines, it just happens to be that the passion and skill-set for each is a little different. PMs usually have a passion for working with customers, conceptualizing what the product should do, and then working with their Dev and Test peers to coordinate all the work to make sure we deliver exactly that. Developers have a passion for building top-quality software – software that is innovative, simple, reliable, secure, scalable, high-performance and stands the test of time. And Testers are passionate about finding all kinds of ways to break software and making sure making sure we find all the issues and bugs before we ship it to customers.

When we interview candidates, a very important part of what we do is find out which discipline the person’s talent and passion really lie in and directs them accordingly. Of course, over the course of one’s career, one’s passion and talent may change, and the person may change disciplines as a result – I myself started in the Dev discipline before switching to PM. This is only natural and we actually encourage that as a way to build better teams.

Other disciplines

It is also important to point out that although the three disciplines mentioned above are what have traditionally been considered the “core” disciplines at Microsoft, there are several other disciplines that are also becoming increasingly important. For example, User Experience (UX) professionals are essential to ensuring that products are intuitive and natural for users to use. A great user experience can make the difference a product that customers love versus one they merely tolerate. UX is certainly very important for products aimed at end consumers, but it is also important for all our audiences – Developers, IT Professionals, Information Workers.

As we move into the Software+Services era, a variety of disciplines related to architecting, building and running extremely large-scale infrastructure becomes increasingly important. Again, while this has been true for some time for our consumer facing web properties such as MSN and Live, it is now becoming increasingly important for all our product groups as more and more of them take steps to evolve their products along the Software+Services model.

Many candidates I talk to often want to discuss what role at Microsoft would be the best fit for them and how they can grow their careers. The best advice I can think of is to work on a technology and a role that they are really passionate about. As I mentioned above, we value all the disciplines equally and a well-balanced organization needs great people in all the different roles. While different disciplines appeal to people with different passions and skill-sets, all the disciplines offer opportunities for innovation and great work. And all of them offer opportunities for advancement and leadership. Indeed if you look across the senior levels of Microsoft, there are leaders who emerged from various disciplines – what they shared was a passion for what the work they were doing.

I hope this discussion of the different engineering disciplines at Microsoft and the approach we take to them shall be useful for the many people who seem to be interested in this topic. If you have any questions or comments, feel free to post a reply to his entry.

Until next time – 再见!

In my last post on the SQL Server China R&D team, I brought up culture as one of the most important leadership challenges in building up a team. How do we ensure that a team grows up with the “right” culture? And what exactly is this “right” culture anyway? In this post, I want to delve into these questions in a bit more detail.

Let’s start with the definition of the word first – Merriam-Webster defines culture as

Main Entry: ¹cul·ture Pronunciation: 'k&l-ch&r
Function: noun
Etymology: Middle English, cultivated land, cultivation, from Anglo-French, from Latin cultura, from cultus, past participle

the integrated pattern of human knowledge, belief, and behavior that depends upon the capacity for learning and transmitting knowledge to succeeding generations b : the customary beliefs, social forms, and material traits of a racial, religious, or social group;  c : the set of shared attitudes, values, goals, and practices that characterizes an institution or organization d : the set of values, conventions, or social practices associated with a particular field, activity, or societal characteristic

So, culture is about shared attitudes, values, goals and practices. Every organization has a culture, whether one that has been fostered intentionally, has evolved in a grass-roots manner or some combination of the two. Sometimes, organizations are very explicit about verbalizing the kinds of values and cultural attributes they would like to foster. But it should be no surprise that most people cannot understand, or even remember, what the “mandated” culture of an organization is supposed to be. Each of us grew up with some cultural background, but probably not many of us would say that we acquired our cultural leanings by reading a set of values in a document. Most of us imbibe culture through our daily lives, through everyday experiences, actions, reactions, positive and negative rewards and reinforcements, from a variety of quarters – our elders, superiors and peers. Over time, these values and patterns of behaviors become ingrained in our very being and we then become part of the “established culture”, doing our part in passing that culture on to others in the organization.

Research shows that culture, once ingrained, can be very difficult to change or “unlearn”, for individuals and even more so for groups or organizations. At the same time, over the long term, culture has been shown to be a much more reliable and important factor that differentiates successful organizations from average or below-average ones, much more so than business strategy, a particular technological or process advantage, or even particular leadership howsoever charismatic or visionary it might be. That is why it is so important, IMHO, to start off a team with the “right” culture, howsoever each organization might define right.

For a team such as the SQL Server R&D team in China, clearly a lot of the “right” culture for us follows from the culture of our “parent” team in Redmond. Of course, we will interpret and adapt and bring our own local flavor to this culture here in China, but many of the essential characteristics that define who we are and how we behave we will inherit from the larger SQL Server R&D team. So what are some of the elements of this “SQL Server culture”? It is hard to give a definition or boil down to a few bullet points to things that have been accumulated over a period of many years, but let me try to list here a few things that I believe are core to the SQL team:

Systems Culture: We are a systems team. We build mission-critical platform software which is used by tens of thousands of organizations to build and run business-critical applications that are in turn used by hundreds of millions of people world-wide every day, 24 hours a day, 365 days a year. We build software that has a lifetime of years, indeed decades. Any mistake or weaknesses, whether in design or implementation or in our processes or methodology, can and will show through. The bar is high! To be successful in this kind of an environment, we need a mind-set of being professional grade engineers (see below). Amateurs need not apply!

IC Culture: Building mission-critical systems software is a craft that takes years to learn. We often tell engineers when they’re hired, especially those coming straight out of universities, that they won’t even be fully productive until 3 to 4 years into their job! While this might sound hyperbolic, it is actually true. By the time an engineer learns what customers want, how to translate that into a product or feature, what is good design versus bad design (versus great design), what tradeoffs to make, how to write robust, secure, reliable, scalable high-performance code that is maintainable and supportable in the field, and to do all this in a productive way working in a team environment, it is going to be several years. This is the process of transforming a smart fresh graduate into a professional grade engineer and there are no short-cuts.

So what is IC culture? IC is Microsoft-speak for Individual Contributor – as opposed to lead or manager. When a business depends on deep technical skill and knowledge like ours does, we have to value the individual contributor. We cannot build a product like SQL Server if every smart engineer wants to stop writing code 4 years into their career and start only managing people. I often cite this statistic – Microsoft has dozens, perhaps hundreds, of VPs world-wide, but only 14 Technical Fellows (and we are fortunate to have 2 of those in SQL Server). The take-way is not that a VP is not an important post, but that at Microsoft, and definitely in SQL Server, we very highly value people who want to remain technical through their career. In a way, Bill Gates himself is the ultimate IC – sure he has a staff, but his main role as Chief Software Architect (CSA) of the company is to help chart the technical direction for the company. There can be no greater testament to the value placed on ICs and deep technical knowledge in our environment.

In fact, I believe this point is so fundamental, I’d like to step back here for a moment from SQL Server or Microsoft or any particular company. In my travels around the region, many people express a belief/hope that the Asia-Pacific region as a whole and in particular countries such as China and India shall increasingly take on the mantle of technological leadership in the world economy.  Obviously there is a wealth of talent in this region that provides the necessary conditions for this to happen. However, there is also no doubt in my mind that if this region is really to get to that next step where long term product and industry innovations are being driven from labs and companies here, we have to create a culture that values deep technical achievement. In today’s environment, when I talk to college students about their career aspirations, 8 out of 10 want to be managers within a few years of graduating. We simply cannot expect to build the next generation of technological leadership on such a foundation. I see valuing deep technical knowledge and the IC culture as an imperative for the entire eco-system in this Asia-Pacific region, not just for a particular company or group.

Innovation culture: The database industry is a mature one: 30+ years, 20+ billion USD in annual revenues world-wide. It’d be easy to think that innovation is not a driving force in this industry at this stage. You’d be dead wrong! In my earlier post on the transformation from Database to Complete Data Platform, I described the unprecedented breadth and depth of challenges facing our field. To tackle these challenges successfully, innovation needs to be a core value in our genes, or else we’ll be extinct soon enough. Innovation can be big or small, technological or process oriented, but innovation needs to be a daily value that every employee lives and breathes. SQL Server has had a history of innovating in ways that fundamentally changed even this mature industry – pioneering ease of use and auto-management, integrated BI capabilities in the core platform, dramatically simplified developer experience. Today, we continue that tradition with many break-through technologies in the next release of SQL Server such as the Entity Data Model. We will absolutely need to continue to this focus on innovation going forward.

Customer-focus: In today’s super-charged environment, it is easy to focus on the competition and forget about the customer. But it is vital to understand that the way to beat the competition is by serving your customer, not the other way around. Customer-focus is not the job of one particular department or role. Sure there will be customer-facing departments like our Customer Support Services (CSS) and various other field organizations. But customer-focus is vital to job of every single individual throughout the organization. Without knowing what the customer ultimately wants, you cannot do your job properly – regardless of whether you are a developer, tester, program manager, architect or even an admin! There is no easier way to become obsolete than to lose touch with what the customer wants. On our team, it is mandatory for every employee to spend some portion of their time with customers, whether it be through participation in newsgroups and forums, attending customer conferences, or a variety of other ways.

“Do the right thing”: I could go on about other values we cherish within SQL Server and Microsoft, such as respect for diversity, being open and honest yet respectful, taking on big challenges etc. While these are all great values and ones we do indeed cherish, if you talk about everything you talk about nothing. So I would like to conclude by focusing on a summary value that I call “do the right thing”. I admit it may sound a bit sappy, but it is true that this is something we strive hard to live by in SQL Server. Whether it is with regard to a customer issue, a product issue or an internal issue, this is a motto that guides us in our day-day work. I’ll give you a couple of examples. I’m sure every SQL Server customer remembers the Slammer worm to this day. This one event completely changed our approach to software development within SQL Server. When the worm hit, we had actually just gone through a security push and released SQL Server SP3 (which actually included a fix for the exploit used by this worm). However, once this problem hit, it was no longer a question of whether we had issued a fix or not. We had to help our customers get their systems back up and running as quickly as possible in a safe way. The SQL Server team mounted a Herculean effort in the short term to produce the tools and guidance needed for customers to recover their systems. But it is the long term changes we made – in our development processes, in our disaster-preparedness, in our entire approach to security – that stand out in my mind. The results of those efforts have proven themselves over the last several years as SQL Server has proved itself to be one of the most secure products on the market. Does this mean we will never have a vulnerability or an exploit again? Clearly, the answer is no. But what we can say is that we’re working hard every day to make that as unlikely an event as possible, and we’re also constantly prepared to be ready to respond if such an event does occur again.

Another example I’ll cite is the case of the Database Mirroring (DBM) feature in SQL Server 2005. As one of the most important and popular features in the release, clearly there was a pressure on us to ship this feature. However, as we got close to the end-game of shipping SQL Server 2005, it became clear to us that we would not have the time to run this feature in production (both internally within Microsoft and at selected customer sites) for the duration we would need to have a crucial high-availability feature like DBM be certified for production use. So we had a difficult set of choices – delay the release, ship DBM anyway, or pull the feature from the release. None of them were easy choices – if they were, we would not be talking about them. But in the end, we decided the right thing to do was to ship SQL Server 2005 but not certify DBM for production use, complete the production-test runs in due course, and then certify DBM for production use with SP1. Sure, there was pain associated with this decision in the short term, but looking back customers now tell us they appreciated the extra care and effort that we put in to make sure the feature was completely ready before we let them use it in production.

Hopefully these couple of examples give you an idea of what we mean by “do the right thing.” There are so many more examples that come to mind but this post is long enough already. I hope you also got a sense for what we mean when we say “SQL Server culture.”  It is going to be an interesting challenge to make sure our SQL Server China R&D team grows up with the important elements of this culture ingrained in us as we grow the team over the next few years. But that is a core part of the commitment we made when we decided to create this team.

Until next time – cheers and zai-jian!

Prakash – 孙博凯

This past week, I was in Hong Kong in connection with Microsoft TechEd Hong Kong. Ron Jacobs and I gave the closing keynote where we highlighted the new wave of innovation in Microsoft’s Application Platform. I covered the Data Platform part of course, and I had the pleasure of giving the audience a quick glimpse into the many innovations coming to market in SQL Server 2008 (which is scheduled to be released next calendar year). We’ll talk more about SQL Server 2008 in a future posting, but if you are interested in learning more and playing with the bits right now, here’s the place to start.

In addition to speaking at TechEd, we also met with members of the local HK media and briefed them on SQL Server’s future direction as a product. And we met with some of our valued customers in the region – learning about the latest projects customers are undertaking on the Microsoft platform and the types of issues they need help with is always an educational experience.  

But perhaps the most interesting aspect of the trip for me this time were the talks I gave at the University of Hong Kong (HKU)  and also at the Hong Kong (HKUST) of Science and Technology. The sessions were entitled From Database to Complete Data Platform and the goal was to connect with students and faculty doing research in areas that we now broadly refer to as the “Data Platform” and give them an industry perspective of what we believe is a historic transformation of the field from its traditional roots in “Databases”.

The field of modern databases has been around for over 40 years. From the early work on Hierarchical and Network models, through Codd’s ground-breaking work on the Relational model, through the many innovations in the area of transactions, isolation levels, access methods, declarative query languages and query processing, cursors, APIs, and so on, the field that has provided the basis for building the robust enterprise-scale mission-critical applications that drive much of today’s “Information Economy”. As such, when an average student in a university thinks about the field of “Databases” they may picture a mature, somewhat musty, field in which all the exciting problems were solved many years ago and all that remains is incremental advancements that wring the last bit of life out of a dying field.

How wrong they would be.

Over the last few years, a powerful confluence of trends – technology trends, consumer and business trends, application trends – has resulted in the broadening and redefinition of Database field like never before, with more exciting problems to solve than at any previous time in the long history of the field. Let’s examine these trends briefly:

Technology Trends: Everyone is familiar with Moore’s law – processing capacity doubling every 18 months, first manifested as increasing MHz and now as multiple cores. The trends in hard disk storage capacity (and price) have actually been even more amazing. Just to give an example, hard disk prices per GB have dropped from about USD 40,000/GB in 1980 to about USD 0.5/GB today!!! Memory and flash memory prices are on an even steeper trajectory down, with capacity going up exponentially to match. At the same time, there is a tremendous proliferation of devices – cell-phones, PDAs, gaming devices, GPS devices and so on – all of which generate, store, process and send/receive/synchronize data at tremendous rates. And of course, the ubiquitous Internet has not only made possible new types of applications but also changed expectations around the characteristics of existing applications – more on this in a minute.

Consumer and Business Trends: Enabled by the technology trends mentioned above, there have been huge changes in how consumers and businesses relate to data and information. First, there is the sheer explosion of data – the amount of new data being generated, much of it born electronically, has been growing exponentially (ever notice how your hard-disk, no matter how large, is always significantly full?). Email, documents, digital photos, music, video, streaming data from sensor, satellite imagery, all are part of this great data explosion. But it is not just about storing all this data – consumers and businesses want to derive value from all this data – being able to search, share, synchronize, analyze, visualize and manipulate the data so it becomes useful information – the notion of “Your Data, Any Time, Any Place”. And all this while ensuring the data is secure, privacy is protected, and all regulatory requirements –both external and internal – are met.

Application Trends: First, there was “batch processing” – basically an automation of human processes that had previously existed for centuries. Then came “OLTP” – Online Transaction Processing that in many cases changed the way business was done – what used to take many days could now be done instantaneously. The OLTP architecture and underlying platform technology went through several iterations, but the core concept remained the same. As companies started to accumulate more and more data from these OLTP systems, they discovered an opportunity to gain a vital competitive advantage by analyzing this data and understanding their customers better – thus was born the Business Intelligence, including Data Warehousing, Online Analytical Processing (OLAP), Reporting, Data Mining and so on. But today we live in a Web 2.0 world – applications surface through a variety of end-points (rich client, browser, device, laptop/desktop, …), they seamlessly bring together data from a variety of data sources, and they provide a variety of rich services including query, search, analysis (increasingly real-time analysis), reporting, visualization, and so on. And they do all this while operating at unprecedented levels of scale, reliability and security.

A Complete Data Platform

The changing trends described above are driving a fundamental transformation of our field – from just Databases to what we now call a Complete Data Platform. This Data Platform builds upon and extends the concept of a Database in three different dimensions:

All Data: It is no longer sufficient for the Data Platform to be able to store and manipulate words and numbers as databases have done for a long time now. A Complete Data Platform must be able to work with all kinds of data – including text, XML, Objects, documents, files, streaming data from sensor networks, and any kind of user-defined data. And it must be able to provide the appropriate services for each type of data – storage, indexing, query etc.

All Tiers: Gone are the days when a Database ran only on a “server”. Today, a Complete Data Platform must provide data services across the entire spectrum of hardware tiers – from phones and mobile devices, to laptops, desktops, servers, server farms and finally cloud-scale mega-service infrastructure. And it must do so with seamless interoperability for data and applications across these tiers.

All Services: The range of services on data is no longer restricted to store, query, backup, restore and a few other verbs. A Complete Data Platform must provide a broad range of services including those and others – search, cache, synchronize, analyze, mine, integrate, report, visualize, secure, audit, archive, … In short, it must service the entire data life-cycle, from birth to archival.

We can drill into a lot more details (and we likely will in future posts) but that in a nutshell is the scope of what we call a Complete Data Platform – one that handles all data, on all tiers, and provides all relevant services on that data. And does so while maintaining consistency in some vital dimensions – for example in the data model, security model, management model, data access APIs, development tools, etc. And of course it goes without saying it does all this with high performance, rapid time to solution, and low TCO. Quite simple – isn’t it? J

Our Opportunity

I hope the discussion above gives you a sense about the unprecedented opportunity for innovation in the Data Platform space. At no other time in the history of the field of modern Databases has there been such a wide array of technical challenges, such a broad canvas on which to paint. If you are a student in university – as were some of the bright minds I met in Hong Kong this last week – this is a time of unprecedented opportunity for you. The spectrum of problems to pick from is so vast and varied. And not just for “Database” majors - there are interesting problems in this space for those with an interest in virtually any aspect of computer science – computer architecture, networking, programming languages, data mining, XML, search, visualization, web-scale computing, semantic web – the list is quite long. The database field has always been one where people easily spend a long time – it is not unusual to find people who have built multi-decade careers, indeed spent their entire professional life, in this space. The opportunity to do so, if you choose to, is even more compelling today. After all, we live in the information age – this is our time.

Until next time - cheers,