As we are getting close to releasing VS whidbey, I was thinking about my contributions to this release and one of the major item that appeared in the list was testing C# edit and contine (here after EnC). This is one of the primary features I was responsible for testing in whidbey. Then I thought of writing a short summary about how we tested this feature. Below is the result of that exercise.
Sometime in Jan 2004 when we were half way into the whidbey product cycle, decision was made to provide Edit and Continue feature for C# in whidbey. This was based on the overwhelming feedback from customers asking for EnC for C# and that it was the #1 feature request listed in MSDN feedback.
EnC is one of those features thats very simple to use but complicated when you get into the details of its design and implementation.It touched the core areas of VS - CLR, Compiler, Editor and debugger, making testing very challenging. However our initial understanding was that since VB already had support for this (in whidbey), it was just a matter of making changes to C# compiler and C# editor to make this happen and the support in CLR and debugger already existed (and tested). Its another matter that this assumption turned out to be incorrect (atleast partially) in the end.
Intially the test team comprised of me and my collegue Daigo (he blogs in Japanese and can be found here). Santosh joined the team some time later.
We started of by coming up with the quality metric that would help us drive our testing efforct. These are the two primary attributues we wanted to target:
EnC should reliably succeed or fail. If EnC does not succeed, the user should always be able to resume the original debugging session. However, when the original debugging session is resumed, code changes will remain.
EnC should accurately reflect supported code changes in the new executable image, while maintaining the purity of the rest of the image. The new image should be accurately reflected in the resumed debugging session.
Pieces making up EnC
After talking to developers about the different components making up this feature and the details about their inner workings we came up with the following breakup list that made up EnC:
Once we had the list of sub features making up the overall EnC feature, our next task was to come up with the approach to actually test these pieces individually and also test the feature as a whole. Below list kind of summarized the approach: