One of the favorite part of my job as a Program Manager in product engineering groups at Microsoft is seeking customer feedback for products we build.  It is really energizing for me to let people take our under-development products for a test run and observe them to see what was hard/easy for them to use.  In past I have observed people participate in usability studies and folks from product groups observe them behind a glass door.  I recently had my first experience with an “agile usability study” we bring in 5-8 participants from within Microsoft to a conference room for couple of hours and give them a set of tasks.   The study was organized by our user experience group.  In addition to the satisfaction of making a difference in a product, each participant also gets a $25 gift card!

The session format was:

·         10 minute introduction (5 minute study intro and 5 minute product context)

·         1 hour and 30 minute of actual study

·         20 minute wrap up


Folks from product groups (PM, SDE, SDET) in the room observe the participants during the study, write the feedback on huge stickies and put them on the whiteboard.  We categorize the feedback during the session into high level categories and then we discuss the most relevant category in the post-study 20-minute wrap up.   I found this study format to be great as it was low overhead and makes it quite interactive.    We provided virtual machines with pre-loaded bits to the participants – all they had to do was bring their own laptops.  At recruitment, we had specifically asked for .NET developers with SQL experience for this study. 


In my 5 minute context (why) about SQL Server Modeling Services, I didn’t mention “M” or “Quadrant” at all.   Our usability study for System.Runtime domain had the following three primary evaluation goals:


1)      User comprehension of the System.Runtime database model (tables, views, table value functions)

2)      Usability of LoadAssembly.exe tool

3)      Usability of writing T-SQL queries over the model


The scenario, sample application and task list is available at: System.Runtime Model/Loader Usability Task List.


The sessions like these make us (folks from product groups) get out of our cocoon and see the ‘truth’. J 


I will do another post with analysis of the feedback once we have compiled it all, but here were my high level takeaways:

1)      Don’t make me write T-SQL queries, provide me a tool ( seems i am the only one who likes writing T-SQL queries ;-) )

2)      It is very difficult to navigate the structure of the schema in SQL Server Management Studio (SSMS)

3)      Provide an EDM / object model head over the schema so I can then write queries in code


It was apparent in this session, that unless our target customer is an ISV developer who is building metadata-aware tools on top of the SQL Server Modeling Services database, an enterprise developer or architect will not write T-SQL queries for analysis.   If our value proposition is ‘impact analysis over codebase’ then we need to provide a targeted tool for those users (and they need not worry about the database structure).  

What are your thoughts? Can you benefit by having your .NET applications metadata in a database?  How much metadata do you really need in there?  Are you interested in participating in our agile usability studies (i.e. if you are in the Seattle area)?