Engineering SharePoint
This is the second post in the run up to the SharePoint Conference on October 19th. Last time, I covered the history of SharePoint. This time, I will talk about how the broader Microsoft Office team designs and builds SharePoint highlighting improvements for the 2010 cycle. This is our fourth release of SharePoint plus several releases of Office before that. There has been continuous learning and improvement in our product design and development approach, particularly as we have expanded to servers and services during the last decade. For additional perspective, check out the Office 2010 Engineering Blog.
Process as a Reflection of the Product
You told us you wanted easy, comprehensive, flexible and reliable productivity tools and infrastructure and as a result we have continually evolved our team and process to reflect that mission. There are three key areas we focus where we differentiate our development approach - shared vision, innovative ecosystem and holistic quality.
First, a shared vision is the foundation of everything we do. SharePoint is a large and integrated product serving a broad set of customers. We have learned from discussions, research and instrumented builds, that organizations and users do a broad range of tasks to get work done. There’s ad-hoc brainstorming, statistical research, scheduling and project management, metrics calculation and analysis, internal and external communications and more. However, these activities should not be silos. Just as you told us you want your word processor and spreadsheet to be consistent and work together, you asked for simplicity spanning team collaboration sites to reporting portals. Hence, we always start with a shared vision, design, development and release process bringing together the teams building Office and SharePoint.
Second, we know you need SharePoint to be adaptable to specific industries, organizations and individuals. Building a healthy and innovative ecosystem on our platforms has always been a priority at Microsoft and we increased our investment in the SharePoint team for the 2010 release. We made big bets on customization, programmability, interoperability and upgradability that I will cover in the next post. While we have been working with a range of customers and partners for quite a while on 2010, we will ramp the broader ecosystem education and feedback cycle at the SharePoint Conference next week.
Finally, the most important driver of our engineering process is to meet your mission critical needs for the holistic quality of Office and SharePoint. We know this extends beyond the software to documentation, support, training and skills across the industry. We have the largest quality assurance investment (people, schedule, industry testing) in the industry and made a number of key improvements for 2010 to deliver SharePoint as a server and service. I will highlight some of these below. As I covered in the previous post, we do a few things out-of-band including value-added innovation spanning Microsoft, 3rd-parties partners and the open source community but we make tradeoffs in our core engineering approach towards a well-designed, integrated and reliable wave of productivity solutions.
The Team
SharePoint is built by the Office group which includes approximately 40 teams and thousands of people. The core teams are in Redmond, Washington, but we have teams around the world with some of the larger sites in Silicon Valley, Boise, Boston, Ireland, Norway, India, China and Japan. These teams provide diverse experiences and perspectives and are a great test bed for our global collaboration tools. We often get asked about our development culture. I think the best description is a balance of top-down and bottoms-up vs. one extreme or the other. We try to balance consistency with innovation and empowerment with governance. If that sounds a lot like our vision for the SharePoint product, that is the idea. The team should reflect the product. While we consider a shared vision, schedule and process core to our approach, we have lot of brainstorming and collaboration as we iterate on this shared plan. After the vision is set, we push down almost all the decisions to the teams about what to build and how features should work as long as they are consistent with fundamental tenets like security and performance. That bottoms-up innovation is how features like My Sites and the Business Data Catalog got created and we have many examples like that coming in 2010. Of the 40 teams, almost everyone is contributing to SharePoint or building light-up scenarios that work with SharePoint. There is a core SharePoint team (actually two – one for Windows SharePoint Services and one for SharePoint Server building on top). However, the code comes from across the organization. Some teams (Word, Excel, PowerPoint, Visio, Access, InfoPath, OneNote, etc.) have been client-centric but increasingly are building client-server scenarios including the Office Web Apps (e.g. the 2010 PowerPoint Web App) or SharePoint features (e.g. InfoPath Forms Services). Other teams are more server-centric such as our Content Management, Search and Line-of-Business Integration groups but even they are focused on end-to-end scenarios. There is a mantra inside our group to optimize customer scenarios top of mind and not “ship the org chart”. Interoperability is a key part of this process from day one including publishing all the interfaces between our clients and servers. We have published SharePoint APIs and protocols for a long-time. We formalized this approach as part of Microsoft’s Interoperability Principles and significantly expanded the depth of our documentation in the last couple years including sharing preliminary information earlier than ever before for the 2010 wave.
Team investments and innovations come from a lot of different places and we used all of these approaches as part of the 2010 development:
· Existing Teams – Most of the 40 teams (e.g. Excel, SharePoint) evolve from previous releases coming up with ideas based on customer research, technology investigation, brainstorming, etc. Some of the work the Excel team has done for business intelligence in 2010 is a good example of a team evolving its mission in a big way.
· New Teams Created for the Release – OneNote is a great example of an effort where we created a new team within the Office organization. The core Office Web Applications team is the best example of this in 2010. Prior web work had been done in Outlook + Exchange, Excel and InfoPath, but we wanted to create a shared team to provide shared capabilities for the 2010 wave just as we have done in the rich Office client for several releases.
· Incubations – As I mentioned last post, SharePoint itself evolved from a couple different incubations. The Line-of-Business Integration team is the best example of this in 2010. We had a sub-team inside SharePoint Server building the Business Data Catalog (BDC) in SharePoint 2007, but we beefed that up considerably with the LOBi team who had been working outside SharePoint on some early stage work that we decided to bet on for this wave. We have a number of partnerships with Microsoft Research and other teams looking at things that will come in future releases.
· Acquisitions – While we don’t do a lot of acquisitions, we have been fortunate to make some great companies join us dating back to PowerPoint, SharePoint Designer (previously FrontPage), Web Content Management (from NCompass Labs) and SharePoint Workspace (previously Groove). The FAST Search team is a great example of an acquisition enhancing our team in 2010.
Hopefully, this gives you the sense that to build a broad product we have invested in a large team with a variety of sources of experience and innovation, but all striving towards a shared vision for building easy, integrated and interoperable SharePoint and Office clients, servers and services.
The People
The roles within Microsoft R&D have been covered a few books and blogs before so I will not go into great detail. At the highest level:
· Software Development Engineers (SDE) own the architecture and write the code
· Software Development Engineers in Test (SDE\T) own the test strategy, plans and automation investment and test the code
· Program Managers own the vision, specifications, project management and coordination outside the team with marketing and others
· Product Planners own the customer, market research and feedback processes
· User Assistance owns the documentation from in-product help to IT guidance to SDKs
· Designers owns the look and feel of the product from the overall approach to refining specific features
· Usability owns the prototype and product usability testing , analysis and recommendations
In addition, we have lots of key roles outside R&D helping us design, build and support SharePoint. Product Managers own marketing programs, customer and partner engagements, and readiness activities (like the SharePoint Conference) and provide some of the most important feedback to Program Managers.
While this sounds like a lot of roles, it is not as compartmentalized as you might think. Most of the work in Office and SharePoint is done in what is called “Feature Crews” consisting of a PM, SDE and SDE\T. They work extremely closely together on the whole lifecycle of an area of the product from brainstorming to scheduling to prioritization to development to testing to dogfooding to early adopter customer projects. Some Feature Crews use approaches like Scrum, some don’t. At this level we really leave it up to the team how they want to operate day-to-day and we try to keep the organization chart very flat above them. Most new hires have just 2-3 layers between them and an area VP/GM like me. We have a common engineering system (specs, build, test framework, etc.) but we encourage a lot of creativity and best practice sharing across Feature Crews. One we used for 2010 is promoting great early work the SDE\T team had done using real world data sets to validate upgrade much earlier in this cycle than 2007. Each release, there are dozens of new things we try in our approach to building better software. Some work. Some don’t payoff. For SharePoint 2010, our big new investment in people and roles was our Customer Assistance Team (CAT) that I mentioned in the last post. We were initially worried about creating another role. We already had the SharePoint Center of Excellence within our Microsoft Consulting Services (MCS) organization who looks at key customer and partner projects and brings the feedback into the team which was a key part of us designing 2007. However, we decided with the growth of the business and range of deployments, we really wanted dedicated people sitting within the engineering group living and sharing complex manageability, governance and application development customer scenarios. Our CAT team shares customer overviews, architectures and feedback on the http://office SharePoint site and in atrium sessions and works with developers and testers 1x1. In fact, I expect some of our top ranked sessions at the upcoming SharePoint Conference to be delivered or co-authored by members of our CAT team. In sum, while a broad product needs some expertise and specialization, the culture of the group is very much “make the right decision for the customer, make others great and get stuff done” so it really is a team effort which we think is an ideal culture for building a collaboration product.
The Feedback Loop
Customer and partner feedback has long been the key to the Office and SharePoint engineering process. We are fortunate to serve a very diverse set of organizations and users around the world. As you would imagine, one of our most valuable sources of input is discussions with IT leaders in large organizations. However, we are careful not to get skewed too much away from direct end user feedback. We talk to lots of end users and partners. We synthesize what is similar and unique when we to school teachers about collaborating with their students vs. VPs of manufacturing trying to share best practices in plant operations across their factories. A few of the mechanisms we emphasized while building SharePoint 2010:
· Statistical Research – We have a few different instrumentation and telemetry mechanisms we use in Office and SharePoint including SQM (actions and profile data), Watson (crashes) and Feedback from the App (e.g. Send a Smile/Frown). At scale, this is the best data we can get to help us go from gut decisions to data-driven decisions so we appreciate you using them. We have rigorous privacy guidelines around this data. This helps us understand everything from typical network latencies to query term patterns to document library sizes. Over the last few years, we have also gotten lots of operational data from SharePoint Online. While we always looked at activity profiles from Microsoft’s own 100,000+ person intranet / internet running on SharePoint and a set of early adopter customers, it is valuable to have detailed data from our much larger online farms supporting a broader range of customers. This also helps tuning the user experience, performance, deployment documentation, etc.
· Product Planning – Planning does a broad range of primary and secondary research we share across the team. One of our favorite things they do is “Day in the Life” research where we follow real people around doing their jobs irrespective of whether they use software to get tasks done to keep us grounded in addressing unmet needs.
· Internet Blogs and Forums – We read these intensely looking for the key trends and feedback so please keep it coming.
· Conference Chalk Talks –Some of our highest bandwidth discussions come after sessions at the SharePoint Conference, TechEd, the PDC and regional events where we have someone from the development team brainstorming with people on a particular topic – be it designs for high availability or how to evangelize SharePoint to new users. There is something about getting an unplanned set of people together without too much structure that sparks a lot of creativity about where we should take things next.
· MVPs, Customer and Partner Advisory Councils – These are more formal activities where we try to balance being inclusive but small enough to be manageable early in the cycle when our plans are still forming. Our MVPs have always given us great feedback during every stage of the development process. Our Partner Advisory Council meeting is small enough that everyone fits around U shaped table so people feel comfortable chiming in if they like or hate an idea. We have extranet SharePoint sites and Live Meetings for these relationships so we can collaborate virtually as well. We work hard to make sure these cover a cross section of organizations and geographies as we are not able to include as many people as we would like in these forums.
· Field Advisory Councils – We have a group of Microsoft employees in technical sales, consulting and support we call “SharePoint Insiders”. They are mix of new and old employees - some have been part of our feedback loop since pre-release SharePoint V1 @ 2000. They are a very direct group and do not mince words when reflecting the feedback they hear from you or opportunities they see.
· Usability Labs – During the development cycle we do 1000s of tests (paper prototypes, terminology, screenshots, working code, etc.) with a sample of people following a pretty rigorous methodology. The labs are here on campus but anyone can watch a test live from their office. This is one of my favorite parts of the development cycle. If we show a design for a new feature to 8 people and only 2 of them can complete the task, it is back to the drawing board.
· Early Adopter and Beta Program – We have a staged disclosure where we work with a cross section of early adopters to get early product feedback followed by a broad beta to help us understand what we need to fix across a broad cross section of scenarios. The upcoming broad beta is a very critical part of the process, so again, thanks for your participation when we roll that out.
I could keep going but this is covers the biggest investments.
The Vision
My second favorite day in a SharePoint release (after the day we ship) is the day the team rolls out the vision. This includes a vision statement, pillars (e.g. embracing cloud delivery, supporting business processes), tenets (e.g. upgradability, security) and, of course, the schedule which grounds us in the realities of tradeoffs needed in any software business and project. The schedule is based on all the data we analyzed from the last release about how long it took us to complete various parts of the engineering cycle vs. hopes. It gets rolled out to the team in three forms – a one page Word document we can all keep on our desktops or walls to guide us on tradeoffs through the release, a longer Word document that goes into detail and a big PowerPoint demo to the team. The latter is my favorite part. We take everyone in team into a big conference center for the vision roll-out meeting. There are a couple text slides to describe the pillars and tenets but 90% of the meeting is the experience we want users, developers and IT to see. One big giant three hour demo. Everyone leaves very fired up. With the vision in place, we unleash specing, development and testing. Here is where lots of details get written down, debated, reviewed and finalized. Along the way, the teams come up with new ideas, get feedback from customers, partners and the market and make tradeoffs to adapt our plans but that is much easier to do from a grounded vision. All of the collaboration and tracking is via our intranet http://office SharePoint site I will describe below. However, we have a few key face-to-face checkpoints along the way. The first is “demo day” after the first coding milestone. We get everyone back together to demo the real code to see how it holds together and where we can share ideas or need to adjust the plans. The second checkpoint comes later when the code is close to polished. The design team takes the original vision PowerPoint and replaces the mockup bitmaps with real screenshots of the working builds. Again, it is a great chance to holistically see where the designs have evolved from the vision day (which they always do a lot!) and where we still need to tweak. There is always a mix of some cool new things we added and some things that we had to defer but each release we have been about 80-90% consistent with the original vision. I think a clear collaborative vision is a key to building a successful team and broad product like Office or SharePoint. We obviously hope you use SharePoint to foster the same kind of planning and collaboration in your organization.
The Process
I will not going into the detail of how we do software engineering. As with our roles, there’s been a number of good books and blog posts over the years that have covered at least parts of what we do. One of the best has been the Engineering 7 blog done for Windows 7. I did want to highlight a few things we evolved for SharePoint 2010.
· Dogfood – A lot has been written about the importance of dogfood (running your own software internally for real work) at Microsoft to build usability and quality in and give us a data on the project status. When I first joined Microsoft in 1992, I was put on a very early dogfood version of what became Exchange for all my e-mail and used a pre-alpha version of what became Access for a customer tracking app. It was rocky at first but a great chance to get data, give feedback and help ship these products. The Windows NT team compiled Windows NT using early builds of Windows NT which was called self-hosting and developers got a visit from Dave Cutler, the NT development manager, if you they broke this. Dogfooding is the most critical to driving a sufficient level of real world stabilization to facilitate external feedback. This release, we embraced dogfooding earlier and more frequently than ever in SharePoint. We have been using Office and SharePoint 2010 builds to do our work in the team for over 18 months. For me it has been no net ever since. Mails via Outlook 2010. Documents to SharePoint 2010. For the client software, we have a http://voteoff site where people in the development team vote and comment if a particular build is good or not. While we have lots of manual and automated testing where we find the vast majority of our issues, but the data we get from dogfooding is an indispensable part of the process. If we find an issue during dogfooding, it not only improves our product but also flags areas for increasing our test automation. For SharePoint, over the course of the release we have dozens of upgrades to our dogfood and puppyfood (the corny name for our pre-dogfood servers for the really brave– another 2010 cycle invention). We just upgraded http://office again this week and the build is looking very good. We will start the process to upgrade the SharePoint servers for all of Microsoft which we see as a critical part of validating the beta release before making it available to all of you.
· Early Adopter Interaction – In the past, we gave the software to early adopters who described their scenarios and feedback. This release we had Feature Crews actually build sites with these early adopters. This turned out to be extremely educational. Another new thing we did year was hosting trade fairs in our Building 16 atrium where each team had a booth to show off their customers’ projects and developers would walk around and learn about how features would be used and feedback. We gave fun awards for the best sites including most creative scenarios and most interesting bug found in the process. It was a great chance for developers in the team to see customer scenarios face-to-face, ask questions, etc.
· Performance Analysis – SharePoint performance is very complex and multi-dimensional depending on the hardware, software and network configuration, data set, usage profile, etc. Our CAT team has done a lot of great work to publish much better performance guidance for SharePoint 2007 than we had at 2007 RTM. They also helped the entire team dramatically increase its performance analysis and investments for the SharePoint 2010 cycle. We probably collective 2 orders of magnitude more performance data vs. during the 2007 cycle. For example, we don’t just want to understand the median latency of a page but the 95th percentile latency across a browser and bandwidth matrix as users don’t want the product to be slow even 5% of the time when an asynchronous job might impact performance. From looking at hundreds of our largest customers in detail, the top two opportunities for performance improvement are the hardware configuration and custom code written to SharePoint. So a lot of what we did to help our development team we put in the product for 2010. Some of this is tuning, isolation and throttling but one of our favorites is a report that shows you your slowest pages and lets you turn on the “Developer Dashboard” on a page so you can see exactly where the time is spent. A couple releases ago when we upgraded Microsoft overall corporate portal to SharePoint 2003, it took us several hours to track down a slow page to a custom web part that did too many round trips to a non-SharePoint database. In SharePoint 2010, you will know this proactively and can diagnose in minutes. We also have a new way to prevent this Web Part from slowing down the site which I will describe in the next post.
· Grid Team – In the last couple of posts I mentioned that SharePoint Online is big part of our development cycle. It is a way to reach many new customers. It also gives us a much larger scale environment than our own SharePoint intranet, extranet and internet apps to analyze and tune. This release, we created a dedicated team called “The Grid” team to analyze and optimize in great detail the total costs, time, reliability and other factors of a SharePoint deployment we hope to reach 10s and eventually 100s of millions of users. Almost all of what we learn there helps make the product more reliable for customers and partners hosting their own servers from a small company to a 250,000 employee enterprise. There is a bunch of great learning from things like virtualization with Hyper-V to best practice analysis to operations monitoring we will share in the coming months. The Grid team will be at the SharePoint Conference.
SharePoint on SharePoint
The key to successful dogfooding is picking a real scenario and not just a playground. So we run the Office and SharePoint projects on early builds of SharePoint 2010. Except for the first couple builds, there is no fall-back site. We have to make it work. I often get asked about how to balance empowerment with oversight by SharePoint customers, so I will describe in a little detail what we do on http://office as we have a mix of ad-hoc stuff with some pretty formal engineering processes. Part of http://office is collaboration sub-sites for each team, where they can do whatever they find useful except, and this is a big one, open up security to anyone they want in Microsoft. We use SharePoint’s security permission and inheritance features to restrict who in Microsoft has access to information about future versions of Office. The core of http://office is more structured sites including Vision, Schedule, Specs, Test Plans, Status, Metrics, etc. A few people running the overall project customize these but almost everyone contributes and consumes. Right now, I spend a lot of time looking at the Metrics site. I have two monitors on my desktop and one usually has an IE window with this maximized. It uses our BI features including Excel Services and SQL Server integration to bring together data from across the teams on a variety of metrics. It uses Excel features like heat maps (e.g. teams with more than 5 bugs per developer are red) and sparklines (new mini charts that fit in a cell in Excel 2010 showing) to show areas for focus and key trends. Finally, we have a bunch of social networking tools on SharePoint. I keep my internal team blog there. We have Engineering Wikis for developers, testers and program managers that are living training and best practice sites that new hires in particular find very useful. Everything on http://office is indexed so rather than everyone sending e-mail to everyone about everything, we use alerts, RSS feeds and some new 2010 social networking features I will describe in the next post to keep track of things that are of interest to us. I mentioned our design and usability specialists earlier. I subscribe to a weekly alert for all the test and research posts they make to the SharePoint site. Others may want that more or less frequently. http://office is used by thousands of people around the world daily so we also get great operational data that helps us tune the work-in-progress code as well. So dogfooding helps throughout the cycle - from validating new collaboration features to ensuring good code quality to final stabilization and documentation work. Of course this is not sufficient; it complements manual and automated testing, early adopter testing and pilots, broad betas and other parts of our process.
Three Common Questions
I thought I would wrap-up this post with the top three questions I hear from customers and partners about how we think about building SharePoint.
· Do you guys think of SharePoint as an application or a platform? SharePoint is both but we start by aspiring to be an application. We want to solve as many problems as we can out-of-box without requiring a developer. Of course, many sites and solutions need developers and in the core SharePoint teams we have lots of people working on developer features and 2010 will be a major leap in programmability and developer productivity as I will cover in the next post. But I think we should always be looking at custom solutions and asking how we could have enabled them using less code. While this is this the right philosophy, there will always a huge variety of needs for building solutions on top and we’re very focused on supporting that as well. As I covered in the last post, while we are focused on solutions, we made some very clear architecture bets in SharePoint on integrated storage, rendering, security, etc. so the platform is a big internal focus as well.
· How do you guys make tradeoffs vs. product X, Y or Z? SharePoint is a broad product and we position it as such so we understand we invite this comparison. We have been asked to contrast SharePoint to everything from SAP to Facebook! We make tradeoffs focused on broad deployment of a general purpose server and service spanning collaboration, portals, content management, search, business intelligence and business process integration. All else being equal, we will prioritize capabilities used by the broadest number of organizations and users. Over the last couple of releases we have substantially increased SharePoint R&D to invest in depth as well. To use the Office analogy, not every Excel user creates PivotTables but most organizations have lots of people creating them and many more people interacting with them. You saw a leap in depth investments in content management in 2007 and will see a similar leap in Search and BI in 2010.
· What do you guys really think about the cloud? Is it an opportunity or a threat? What if I want to or need to keep running my own servers? We love the cloud! We have been running SharePoint Online for customers based on SharePoint 2007 for a few years. Last year, we announced some of the largest enterprises adopting SharePoint Online as well as a multi-tenant service for small-medium sized organizations. You can choose to use SharePoint Server, Online, partner hosting or a combination. We see all models going forward. While SharePoint Online helps organizations who may not have access to IT resource, we believe it helps in larger organizations with deep IT groups well. We think we’ll do the very best job running it which translates to happier users and IT and lets IT professionals spend more time higher business value activities for technology adoption than just low level operations tasks. Even for organizations who want to run SharePoint, you will benefit because we build our experience running high availability, low cost services into the product and guidance. Again, your choice: server, service or mix. We apply and share what we learn from our SharePoint Online experience.
Thanks again for reading and your feedback. Next post from me will be an overview of the 2010 capabilities. I promise!
Jeff
Jeff Teper – Corporate Vice President, SharePoint Server, Microsoft
