As we begin to close down on Ask Mode for the Platform and Design Time teams we enter the final stage of the release, Whidbey RTM Escrow.   

DevDiv Whidbey RTM Escrow – the phase before the completion of the RTM milestone where the product goes through a period of bake time.  No additional work is planned for the Whidbey RTM ENU product during this period of time, “we are done”.  All QA teams have completed their final test passes and have signed off on the product, including all Whidbey release criteria.  The only work left to complete will be media verification. 

Once the Platform teams meet the set of entry criteria required for entering escrow we will complete our final long-haul stress runs and turn our focus to locking down the Design Time teams.  When the Design Time teams meet the set of entry criteria for escrow we will turn our focus towards meeting our escrow exit criteria.  We will build what we expect to be our final build and QA will begin media verification. 

Escrow Entry Criteria

• 0 recall class bugs left to be fixed for Whidbey RTM
• QA teams have completed their Final Test Pass and signed off on the release
• Final release criteria met/signed off
  
  

Escrow Exit Criteria

• Media verification completed
• No Bugs marked as Escrow Reset under investigation

What will people be doing during escrow? 

• Long-Haul Stress – we will continue to run Long Haul Stress to confirm that we have not encountered any new recall class bugs
• App Building – we will continue to do App Building  to confirm that we have not encountered any new recall class bugs
• Media Verification – during the first week of Design Time escrow we will complete media sign-off
• DevDiv shiproom will continue to meet daily while we are in real-time bug mode (24-hour triage), where all new escrow bugs filed must be triaged within 24 hours.
• Investigating issues and regression testing any changes taken during escrow

Where is the triage bar?  What would reset escrow?  During escrow only issues deemed “recall class” by shiproom meet the bar and would reset escrow.  This would have to be a serious enough issue requiring us to recall the media, such as an MSRC or GDR class security issue.

Would we restart the escrow clock?   Yes, depending on where the fix is we will reset escrow based on QA’s expected Whidbey Response Time for verifying and regression testing a given fix.  The expectation is that if we take a serious enough escrow resetting fix that we would need a maximum of 2 weeks from the completion of the fixed build to ship RTM.

We are almost there!!!

-Mike

Most of you are already aware of the Visual Studio 2005 Community Technology Previews (CTP) and are familiar with the philosophy and benefits of the CTPs – you get working bits more frequently, great opportunity for feedback from you to the product teams (using Ladybug), potential of finding issues earlier than RTM, ability for you to verify requested fixes and lots of other good stuff. We are working on striking a balance between releasing CTPs frequently, the amount of time we spend verifying their quality and their effect on the main product schedule.

We have now released the July CTP, which is available to our MSDN subscribers. The SKUs already released in the July CTP includes the VS Pro, VS Std, VSS, VSTO, VS SDK and VS PPE. VSTS and VSTF will be released next week. 

So, what really happens behind the scenes before you get the bits? Basically, a group of people work together and start cranking the CTP machine J Just to give you an idea, let me walk through my life as a PM on the release team before the release of Phase 1 of the Visual Studio 2005 July CTP, which was scheduled to release in 3 phases to account for the large number of SKUs:

            Phase 1: VS SDK, VS PPE, VS STD due on July 8th

            Phase 2: VS Pro, VSS, VSTO due on July 15th

            Phase 3: VSTS, VSTF and VS SDK (compliant with the phase 3 bits) due in late July

June 28^th- 29^th:

Phase 1 work starts a week in advance. I started to scout the Main branch to select a build that would be released as our July CTP. Some people ask “A week early? Can we not just wait until July and then pick a build a day before it is released?” While that is the ideal case, we want to make sure that our customers get a minimum quality of bits. To do this, we need to allow sufficient time (as you will see in the rest of this blog) for install and uninstall testing as well as media verification (For example, virus scans need to be run so that you don’t get viruses as part of the download). To gauge quality, I look at each build to see if it had passed the Integration tests that we run daily on all builds. In this case, Murphy’s law prevailed - the Main lab, which was doing just fine the week before, suddenly had some build issues so I had to wait for those to be resolved and wait for a new build to be generated.

June 29^th- July 1^st:

- Selected a build as the CTP build since it had passed all the Integration tests.

- Created a save request for the build as new daily builds routinely replace older builds.

- Discussed SQL Server Express dependency with the Yukon team to determine the build of SSE to ship with the July CTP.

July 2^nd:

Came into work (thanks to an issue with my smart card not allowing me to VPN into the corporate network) to request actual media to be burned for the CTP. This will ensure that the requests are seen by whoever first walks into the burn lab on Tuesday. Media needs to be burned for purposes of QA who test whether the images on the bits (actually being posted) match the bits of the CTP build on the build drop server.

July 5^th:

10:00 AM – So, our detection logic is causing SQL Server Express installation to look like it failed (as the post-install check looks for the version the build was generated with). Decide to push this into Known Issues that accompany the download as the install actually succeeds but only looks like it has failed. Also, its too late now as the media is being burned as we speak.

July 6^th:

-Quickly hand over all the media to the test team and create media verification requests for all the SKUs.

-4:00 PM - Test verification finds that the VS Pro un-install fails! After remembering to breathe, I find the Setup devs - Is there a fix? Can we investigate and fix it fast enough to turn around and ship this thing? So, I move the Pro SKU to ship with Drop 2 so that we can get the fix into the build.

- Request MSDN to stage the bits so that the bits will be available when we want to go live as this step usually takes 2-3 days for all the bits to propagate.

- Created a request to post the public symbols on the Internet server so that customers can debug their applications

July 7^th:

Create internal CTP site so that folks have one place to view everything about CTPs and their schedule. Provide MSDN with all the documentation they need and the marketing blurb and give them the go-ahead to post the bits.

July 8^th:

The bits are live on MSDN! Drop 1 goes out to our customers right on schedule and I periodically check MSDN to ensure that the bits have been replicated. But, what is this – the content looks a little different than what I sent out. Where are the Known issues? I picked up the phone and called the MSDN contact and turns out the content was indeed different. A quick change and a couple of emails later, the content looks good. I then send out a status email to the whole Division informing them of the same. Start to work on Phase 2………

 Known issues (July CTP - Drop 1 and 2):

While we haven’t released an official “Known Issues” document and these are also published as Notes in the download location, I wanted to post these here:

1) At the end of the Visual Studio 2005 installation, it might indicate that SQL Server Express is not installed, even though SQL Server Express was installed successfully. Look at Add/Remove Programs to confirm that the SQL Server Express has been installed.

2) Installing on Win2k3 gold will fail until you install MSI 3.1

As you can see, there is still a long way to the realization of our vision where we are releasing builds on a daily basis. We are constantly thinking about ways in which we can streamline these processes and automate as much as of them as we can. But, we are constantly improving on releasing bits “over the wall” to you in a predictable, consistent manner.

Thanks,

Kavitha

I mentioned in my last blog that we’re heading towards a first step in the "Final Security Review" for Whidbey. A couple of weeks ago, our central security team finished a 9 week long Penetration Testing (pen-testing) effort of Whidbey. They had 8 people working just on our product. We began by doing "education" sessions for the security testing team, spending 1-3 hours on each area of the product. Based on these sessions and their expertise and experience, the testing team chose a number of areas to focus on. With a product as large as ours, it was impossible to cover everything. Remember that this is somewhat of a safety-net review. We have done extensive training and invested a lot of time in testing and reviewing our product ourselves.

Our next step is to apply for the SWI Final Security Review signoff. This is being done through a sort of documentation tool. I will use the tool to document that he have met the requirements of SDL (and a few others that we imposed on ourselves). The documentation involves supplying the team with "proof" like threat models, results of tools run on our product etc. One thing to remember is that throughout the engagement with SWI, we had a member of their team be a virtual member of our security group. This helped us be always in sync with the requirements and make sure we don’t miss any requirements. Right now, I fully expect us to pass with flying colors.

In parallel to this sign off, I’m working on publishing our release criteria for security. For the most part, it is the same as our Beta2 criteria. Since Beta2 allowed customers to "go-live" with it, we could not afford to hold off any security work for post that release. We do, however, are constantly improving our tools, and from time to time we identify new potential attacks that we want to ensure we are not vulnerable to. So I have a couple of tools (FxCop and PreFast) that we’ve improved that I need to set the bar for. There were also another three requirements that our SWI buddy sent us that we have to factor in.

Last I blog’d I talked to the Product Metrics/Release Criteria… and now we’ve released Whidbey Beta 2!  It has been a long but satisfying road…   As you can see from Chris blog below “In the home stretch…” he mentioned stress was going to be a wild card.   He was right, which is why Beta 2 took just a little longer than expected.

Over the final few months of Beta 2 we met on a daily basis in shiproom to discuss release criteria, bugs, fixes, stress, builds…  With Beta 2 having a “go live” license there was no way we wanted to release Beta 2 before it’s time, before meeting our quality goals.  I think that was the right choice and I hope are customers agree!

As we roll from Beta 2 I am in the middle of reviewing our Whidbey RTM release criteria.  I will again be holding reviews with all the release criteria owners.  The main topics this time around will be about how we did in Beta 2, defining any necessary changes, setting checkpoints, and building a great quality RTM product!

Now go download and start using Beta 2!

Check off another mini-milestone for Whidbey! Beta 2 Tell Mode is over, and we started "Ask Mode" today.  Over the course of the last week, we finished up a round of what we called “aggressive” integrations of the source code branches and are currently in the process of moving to a single Beta 2 branch (we had a tremendous group of dedicated developers in the build lab all weekend).  Once all the labs are up into the Beta 2 branch, we begin the stabilization process there and begin fixing bugs in earnest on the live branches.

To recap our Beta 2 schedule (based on Fridays):

• ZBB: 11/19/2004
• Security Push: 11/15/2004 through 01/14/2005
• Tell Mode: 01/17/2005 through 02/04/2005
• Ask Mode: 02/07/2005 … until we meet our release criteria, which defines all of the various quality measures for Beta 2.

To be exact, we finished Tell Mode on Friday, but have two teams that need through Wednesday of this week to reduce their bug backlog a little more. So, everyone went into Ask Mode on today except these two teams. We’ll take one last integration of their sources for Beta 2 on Wednesday evening, and then they will be in Ask Mode too.

So starting today, every bug that a team wants to fix for Beta 2 needs to be brought to ship room for approval.  The first question asked is “what is the customer scenario?” and if we decide that the benefits sufficiently outweigh the risks, we’ll ask the team to go and check in the fix.  Otherwise, they’ll move the bug to the next milestone and fix it in the live trees.  We’ll probably batch up a few fixes early this week as we get the Beta 2 branch building.  We typically lock the trees from any check-ins so we can stabilize off a non-moving base.  Once we open the trees, we’ll take all the approved fixes into Beta 2 and begin building every night.  My expectation is that we’ll have solid Beta 2 builds that we can hand off to QA by Thursday or Friday of this week.

Two weeks ago, we entered a phase of the Beta 2 product cycle we call “Tell Mode”.  This is the point which marks the transition to the end game for Beta 2.  Here’s what has happened so far, and an idea of what’s to come for Beta 2.  

• ZBB: 11/19/2004
• Security Push: 11/15/2004 through 01/14/2005 (rest assured, these are not the only weeks we thought about Security in Whidbey!  We spent this time doing concentrated efforts and implementing new processes which are now a part of our daily activities.  The push let us take Security to the next level.)
• Tell Mode: 01/17/2004 through 02/04/2005
• Ask Mode: 02/07/2005 … until we meet our release criteria which defines all of the various quality measures for Beta 2. 

Tell Mode is when the Division begins to meet daily.  Every team reports on the number of bugs fixed the previous day, with those fixes to be available in the current day’s build.  The idea behind Tell Mode is to start the shut down process.  Teams move into a centralized triage process, where every bug that comes into the product team is scrutinized as to whether or not is should be fixed for Beta, moved to the next milestone, or punted from the project completely.  Over time, we start to look at samplings of bug fixes and begin to set the bar, ensuring that teams are fixing the right set of bugs and that we're positively affecting the quality of Beta 2.  This additional process slows the churn on the product because the more fixes we take, the more chances of regressions or build breaks.  Our goal is to slow the number of check-ins, stabilize, take the necessary fixes to meet our release quality criteria, give the product a little time to “bake”, and then ship it.

Tell Mode is scheduled to last three weeks, through 02/04.  We expect to then move into Ask Mode, where every bug that we want to be fixed has to be approved by our centralized ship room.  The first question we ask on every one of these bugs is “what is the customer scenario?”  Every fix needs to have a positive, tangible, and critical customer impact or else we don’t take it in Beta 2 at that point.  Once we are in Ask Mode, we’ll do another test pass and we’ll finish driving up our quality criteria such as our Stress pass rates.  

So far, we’re doing pretty well.  We’re running a little hotter than I would like, but all the right things are happening.  The next week will really give us a good indication on how well we’re tracking towards our goals.

Chris

SQL Server 2005 (code named “Yukon”) is built on the Whidbey platform. The integration of the Whidbey .Net Framework with the SQL server engine is well known; this provides the “SQLCLR” functionality that enables managed code stored procedures, user defined types, aggregates and functions. What’s less well known is that the Yukon administrative and business intelligence (BI) tools are largely written in managed languages and heavily leverage the Whidbey .Net Framework. In addition, the Yukon tools incorporate VS Whidbey components and several of them are implemented as VSIP applications that plug into VS. Yukon developers are daily users of VS Whidbey and the SQL buildlab utilizes Whidbey compilers and tools to build Yukon on a daily basis. It would be fair to say that the SQL division really cares about Whidbey progress and quality.

With so much exposure to Whidbey, it should come as no surprise that the SQL division reports Whidbey bugs. For much of the product cycle we’ve seen a slow but steady decline in the total active Yukon originated Whidbey bug count. The incoming rate varies with SQL's transitions to newer Whidbey builds and QA activities (bug bashes, stress runs, etc.) and we’ve been consistently resolving bugs at a slightly higher rate. Whilst the total active number was never very large, as we close in on Whidbey Beta2, our fix rate has increased and we’re down to a small number of remaining Yukon originated bugs. Most of these are SQLCLR stress related bugs, which is not unexpected at this point, with teams focusing on meeting stress and other release criteria goals. We know we can count on a bug flow from stress runs to Whidbey teams until release criteria are met and Yukon is ready to ship

Yukon's Whidbey exposure is clearly good news for customers. Having a Microsoft division the size of SQL pounding on Whidbey and probing its limits helps smooth the way for customers to extract great results from Whidbey when it ships next year.

-James