AdamU's WebLog

Will it Blend? That is the question!

adamu — Thu, 01 Feb 2007 22:47:00 GMT

We're having fun watching the BlendTec commercials since we've announced that the product code named Sparkle is now Blend.

Since my last post 6 months ago in July where I visited a company in Raleigh NC, I've been amazingly busy.

I've visited Roxio in Santa Clara, CA.
I've done a recruiting trip to U Penn in Philly
I've done a recruiting trip to Shanghai

Most importantly, I've shipped the Interactive Designer Sep CTP, the Blend Beta 1 in Dec, the Blend Beta 1 Japanese in Dec, and the Blend Beta 2 in Jan.

I hope that everyone that's seen Beta1 and Beta2 come away with how deeply the product team cares about quality, and how hard we're working on delivering a great user experience. If you've visited the sparkle/blend newgroup, I hope that you understand that the majority of people answering your questions are people who've been working for years on building this product, and that they love hearing how you are trying to use the product, and are anxious to take your issues and try to resolve them.

We've got two more big milestones coming soon to hit for Blend: RC1, and English RTM. And then the RTM for an additional 8 languages (German, Japanese, French, Italian, Spanish, Korean, Simplified Chinese, Traditional Chinese). For me personally, I've also got two more recruiting trips planned in the next 3 weeks: MIT and CalTech.

So I'll likely be scarce for a while, again, but only because I'm working hard on hiring great people, and driving us to completion on a great product for Pro Designers to build rich compelling UI experiences.We fixed almost 700 bugs between Beta 1 and Beta 2. We added a couple key features that we were missing.

Adam

on the Frontline

adamu — Wed, 05 Jul 2006 00:26:00 GMT

Two weeks ago, I spent an entire week at a customer site. This customer is in Raleigh, NC, and I got to see their early progress on a WPF prototype. They see the User Experience opportunities with WPF as a key differentiator in their market.

This opportunity came about because of a program that the Server and Tools Business within Microsoft runs, called Frontline. This program is why I spent the week with our Customer Support organization in April in Texas.

We always try to look for win win situations. In the Frontline case, I think that this program is a true win-win. I get to see first hand how customers see their business, and what excites them and frustrates them about Microsoft technologies. They get to have someone from the product development group in their office to answer questions (or at least put them in contact with those who can answer their questions), vet their plans, and provide a deeper glimpse within Microsoft.

I had a great experience in Raleigh and I am excited about future customer visits.

June CTPs now available

adamu — Wed, 05 Jul 2006 00:20:00 GMT

We've made available new CTPs of Expression Interactive Designer and Graphic Designer that work with the June CTP of the .Net Fx 3.0. Links are below.

Interactive Designer

Graphic Designer

That's right, you're not from Texas, but Texas wants you anyway

adamu — Tue, 25 Apr 2006 06:15:00 GMT

With apologies to Lyle Lovett....

I'm spending this week in Texas; specifically, at Microsoft's Las Colinas Customer Support Center in Irving, Tx. Now why in the world would I do that, voluntarily? Well, I'll tell you...

Each year as I review my performance, I look at ways I can improve. Everyone at Microsoft should do this, and most people I know here do a healthy amount of introspection during review time. Normally, you take a look around at training and work experience opportunities, and choose 1 or 2 that you'd like to do this year. The last couple of years I'd been choosing people manager training opportunities. This year I wanted to do something different. In my first 6 years at Microsoft, I'd worked on components that shipped in VB5 and VB6, Visual Interdev 1.0 and 6.0, SQL 7.0, as well as Office Developer 9.5 and 10. But I have been working on Expression since July 2002, and it's been a long time since I've truly 'walked in the customers shoes'.

So here I am in Texas, and I'm going to spend a week here, building a relationship with my product's Customer Support Engineers, listening to calls, understanding their workflow, with the goal of bringing back to my group more customer focus.

March CTP of Interactive Designer now available

adamu — Fri, 10 Mar 2006 07:34:00 GMT

enjoy!

http://blogs.msdn.com/expression/archive/2006/03/09/543851.aspx

recruiting at Syracuse U

adamu — Tue, 07 Mar 2006 04:37:00 GMT

I went to Syracuse a little over a week ago to recruit for Microsoft. Since I started on the Sparkle project nearly 4 years ago, I've had the opportunity to go to Shanghai, Romania, Bulgaria, U-Penn twice, Duke, Carnegie Mellon, Va Tech, UCLA, USC, Arizona, Texas A&M, Purdue, and Florida, and I've done phone interviews with candidates from Waterloo and Ohio State. I enjoy meeting bright students with passion for technology, I do enjoy traveling, and college recruiting is one part of my strategy on building a great team.

Syracuse had surprisingly little snow for mid Feb. Having lived in the area 15 years ago (I've lived in 16 states, so no surprise that I lived near there, huh?), I was prepared for feet of snow. Instead, you could actually see grass, and the roads were all clear.

We invited the students to a pizza dinner on Sunday evening, and I was a little tired after flying from Seattle to Dulles, and then to Syracuse, but 30 minutes after we got to the hotel, we were in a room with about 40 students answering their questions about Microsoft, the recruiting process, etc. Unfortunately, we screwed up, and we only order pepperoni pizza, and I think half the people there were vegetarians. Anyone who was there, I apologize for the mix-up, and we'll do better next time.

Monday and Tuesday we conducted interviews for about 40 students. I went on this trip with 2 leads who had never been on one of these types of recruiting trips before, so I spent time observing them (and them observing me) and providing feedback. Wednesday was the long flight via Dulles back home to Seattle.

One thing I've never been able to coordinate on one of these trips was attending a sports event. I was at Purdue when they were playing Notre Dame during football season, but my flight left in the middle of the game. I was at Arizona, but they were on the road. Texas A&M, but they were over in Waco. At Duke, but they were on the road that week. But this time, Syracuse was hosting West Virginia in basketball, and WV was ranked in the top 10. So Monday evening after a day of interviews, I walked up to the Carrier Dome, paid $10 over face value outside the stadium, and got a seat 6 rows behind the Syracuse bench. The place was rocking, and it was alot of fun. I have to say, thought, the Carrier Dome is really not a great basketball facility. Syracuse upset WV 60-58, and the home fans went home very happy.

What's the difference between SDE and SDET at microsoft?

adamu — Fri, 27 Jan 2006 23:39:00 GMT

I was exchanging email with a candidate last night who was confused about the differences between what dev and test do at microsoft, and as I tried to compose my thoughts about what it is that SDETs do, I thought that it would be good to share this on my blog.

You can envision the SDET job in two vectors, one in feature work, another in tools work. We have a product to test, and we need to eliminate defects, so we spend time in both proactive and reactive activities to help PMs specify the right thing, as well as validating what it is that our SDE counterparts built. Here’s the typical workflow for the feature work we do.

Feature Work

PM writes a spec for a feature. PM, SDET, and SDE sit down and discuss the feature, make changes and get general agreement on the spec.
SDET takes the spec, writes a test spec based on it, describing the testing in general terms. The test spec also defines an object model that you can think of as an API to the feature.
PM, SDE, and SDET sit down and discuss the feature spec and test spec, and agree that the OM defined by test is correct.
SDE begins to implement the feature
SDET implements the OM, and then starts writing tests against the OM.
As features come online
1. SDE and SDET together run tests agasinst the feature (manual, automated, and writing unit tests).
2. SDET does adhoc buddy testing against the feature to get more testing done before check-in.
Post code complete, SDE fixes bugs.
Post code complete, SDET analyzes automated test failures.

While there’s a lot to get excited about in both jobs (I can’t believe they are actually paying me to do what I do in my spare time for fun!) there’s drudgery in both jobs. Test case writing, automation analysis, and adhoc testing for the 1000^th time can be mind numbing, but fixing bugs for a months at a time is equally as painful.

Tools Work

To extend our existing automation infrastructure, or to do testing in additional vectors, we build test tools. Lots of them. Test Labs can have 100s of machines, and if we’ve written the tests correctly, we can determine the current quality of the build far more efficiently than with manual testing. So while we do some manual testing, the focus is on building test automation that can be used milestone to milestone, release to release. Obviously, the quality of the tests and test infrastructure is super important. As the saying goes, who tests the tests, and who tests the tests that test the tests? Tests that don’t test properly can give false positives, and can actually be worse than no test at all because of the false sense of security they encourage. And infrastructure that is poorly designed will give you big problems as you approach v2, v3, etc., as well as sustained engineering efforts on your currently released products. So we have code reviews for every checkin, and we have design reviews for all tool work.

Some of these tools we created, some we took a snapshot of the code from other teams and make our own modifications, others are shared amongst many teams. Here is a list of some of the tools we build to test:

lab, machine and run management – infrastructure that drives our test automation in our test labs. Machine re-imaging and configuration, automated logins and installs of pre-reqs, fail/pass reporting, run management, machine allocation, etc.
test harness that allows for functional tests or unit tests to be executed. Builds test binaries and reports pass fail results to the run management system.
Team libraries – we build tons of our own libraries: for my team, we divide them like this
- Physical OM
- Logical OM
- Internals OM – we build add-ins into the product to get at internal data structures to determine current state \
- Extensions to those OMs to help us write fewer numbers of tests that test more of the product, and lower maintenance through ideas like
  - systematic variation of common data and behaviors
  - loosely coupling test execution and test verification to make verification more comprehensive and reusable
  - pairwise testing tools for combinatorial testing
Buddy build tools that Allows SDEs to run specific sets of tests on their desktops (or offload it to the lab and send them an email with the results) before checkin.
Basic UI automation libraries - we take these building blocks and build our Physical OM.
Static Analysis for
- security
- style guidelines
- common bugs on looping/branching
Code Coverage/Branch Analysis
quality dashboard - Web portals that track key metrics
- Code coverage
- Test automation pass/fail rates
- Bug Stats: Current total defects, defects per developer, incoming defect rate, fix rate, regression rate.
- Code base size
- performance
Auto-analysis tools
Performance Testing tools
Stress / mean time to failure testing
watson crash bucket analysis

When I think about SDE vs SDET, what I tell folks is that:

You are better suited to be an SDE if
- you like to get really deep in one technology space for long stretches
- you don’t stop until you write the perfect algorithm, or the most elegant code
You are better suited to be an SDET if
- You like system integration type work:
  - you take the technology that this team did, make use of it and tie it to this other piece of technology, etc.
- You may have been a TA and graded other peoples code, or were the guy in group projects who enjoyed and was good at poking holes in other peoples designs
- You have strong big picture thinking, and are focusing your energy on solving the whole problem.
  - You find yourself coming up with “good enough” solutions to problems. Your solutions may or may not be elegant or perfect, but you are happy with it and you’ve moved on to the next part of the problem.

So I describe the SDE problem space as more narrow and deep, and the SDET problem space as broader, not as deep. Some SDETs have more penchant for tool building, less for feature work, or the other way around.. In my group, I try to balance between both of them so that we get smart people thinking about the hard problems within feature work, giving them good cross functional exposure and scope of influense, and still mixing in a healthy amount of tools work. Other teams (many I've been on in the past) structure it differently, but my personal feeling is that this balance gives people the right amount of both for healthy career development.

Expression Interactive Designer - Now available!

adamu — Tue, 24 Jan 2006 12:13:00 GMT

It's been a work in progress for what seems like a lifetime... But as of right now we FINALLY have a build of Expression Interactive Designer (EID) available for public download. We've also updated Expression Graphic Designer with a new XAML exporter. You can find both here:

http://www.microsoft.com/expression

Windows Live Local - Virtual Earth

adamu — Sat, 14 Jan 2006 06:12:00 GMT

I was playing around with Virtual Earth on Windows Live Local, and they've recently added a bird's eye view to the mapping software. Here's a bird's eye view of my building on the Microsoft campus.

http://local.live.com/default.aspx?v=2&cp=47.639567~-122.133399&style=o&lvl=2&scene=3729997

Christmas Presents and teams long since past

adamu — Thu, 29 Dec 2005 11:20:00 GMT

I went to a White Elephant gift exchange with about 20 other people about 2 weeks ago. What was unusual was that this was with people who were on my very first team here at Microsoft, over 9 years ago. We hadn't had a daVinci team function in a long time, and it was great seeing people that I rarely get to interact with anymore. In fact, we had people who had retired from Microsoft or moved on to other companies come back for the function. I hope that it's not another 7+ years before I can get together with one of the funnest group of people I've ever had the pleasure of working with. The hallway nerf wars, the office decorations while people were out of the office, along with building a really great piece of software that still ships in various forms in SQL Server, VS, and Office nearly a decade since it's inception was an awesome first experience at Microsoft. I think the biggest things I took away from that experience were:

you can work hard and have fun at the same time.
building today's highly complex software, by its very scope, must be a collaborative effort, and team building is an important part of facilitating that collaboration.

And that leads me on to my second point. There were some neat presents at the gift exchange, and I ended up taking home some european chocolate; which was a top 10 gift, definitely. But what I wanted most was the new Mr. Potato Head Darth Tater, but alas, it was not in the luck of the draw for me to end up with it.

I mentioned this Darth Tater to my kids, and of course, I ended up with Darth Tater under the tree, which was pretty cool. See, my kids figure that if they get me a toy, they will get to play with it at my office when they spend the day with me, which, since we homeschool our kids, is a few times a year (way more than I spent time with my dad at his job). Thus I've gotten Star Wars lego sets, hot wheels cars, etc. over the years from the kids.

So when I mentioned it to the kids, I knew that one of them was bound to get it for me, and they didn't disappoint. I really thought it was pretty cool. But what was so suprising (and thus blog-able) was that I not only got Darth Tater, but I also got a SpudTrooper potato head! What's next? Princess Lay's? Luke Frywalker?

Greetings from Shanghai

adamu — Wed, 09 Nov 2005 22:12:00 GMT

I was in Shanghai last week to interview candidates for Microsoft, and got a day to do a little sightseeing. Shanghai is a pretty neat city!

What is Acrylic For?

adamu — Thu, 27 Oct 2005 08:36:00 GMT

Andrew Watt, an infopath MVP asked, "What exactly is Acrylic for?"

Acrylic is intended for pro designers (people who use today's pro design tools) to create rich visual content, primarily for use in user interfaces, either desktop apps or web apps, although for the pure artist it's also an excellent tool. It's target audience is not developers or hobbyists or information workers, so these folks may be initially frustrated attempting to use the tool because of their lack of familiarity with this class of tool.

Whether the pro designer primarily uses vector tools or bitmap tools, Acrylic will enable them to create designs combining the best of both.

Acrylic does overlap somewhat with some existing pro designer tools, but I think it is more complementary than anything. If you are an artist, you can use Acrylic, and you'll have a blast just drawing stuff. It's variation palette, among other things, makes this an amazing experience. But the core focus for Acrylic is in areas that are fairly unique, because Acrylic's uber-goal is to help designers become more deeply integrated into the application development process. Acrylic has the ability to output XAML, which can be used as the UI for an application. In todays typical workflow, a designer uses a design tool to prototype the UI, and throws it over a wall to a developer, who attempts to implement the UI in code, but the developer may miss nuances in the design, or it may take too much time to get it just the way the designer wants, and much of the aesthetic appeal of the design can be lost. With Acrylic's XAML export, developers can use the designer's full content in the application interface; no need to attempt to recreate it.

Adam

Sparkle Video Link

adamu — Fri, 23 Sep 2005 09:34:00 GMT

In case you missed the Sparkle Video on Channel 9, here's the link: http://channel9.msdn.com/showpost.aspx?postid=115387

And the NorthFace Concept Video: http://channel9.msdn.com/Showpost.aspx?postid=116327

You've Got Questions, I've got Answers... Part 2

adamu — Thu, 22 Sep 2005 11:47:00 GMT

OK, here's your chance. Blast away. I did this last year, and got lots of questions from people about how hiring and recruiting works at Microsoft, as well as some test questions.

So, what would you like to know? Something about Sparkle? Or Acrylic? Something about testing at microsoft? Or more on hiring and recruiting?

Here's an Acrylic stitched picture I took during an August camping trip to the Mt St Helens area. The mountain to the left missing it's right (northerly) side, is Mt. St. Helens. The lake that is now 200 feet higher than it was 25 years ago is Spirit Lake. Yep, the bottom of the lake is where the side of the mountain is now.

Sparkle and AdamU

adamu — Thu, 15 Sep 2005 07:12:00 GMT

Well, given that we (finally!) announced today (at PDC) the project I've been working on for the last 3 years, I can now put in my blog that I've been working on "Sparkle Interactive Designer". You can check it out here. And yes, if you get a chance to use it in the coming months, or you want to, I want to hear all about how much you love it, hate it, how it could be better, and where we are missing the boat (especially on quality).

Haven't posted much during the last 60 days, as during August and September, I've travelled a lot. August, we went camping at Mt Hood, and at Mt St Helens, and I did a 2nd trip to Beijing. I put 700 miles on my motorcycle (and 2000 miles in an airplane) during one very busy weekend. I'm currently in LA, doing our annual visit of my mom and step-dad. We have a lot of fun; visiting with my mom and sister is great, and my 5 boys turn into fish and spend alot of time in her pool, and we go to disneyland. I think we are still going to try and get one final camping trip in later this month too.

Speaking of Disneyland, the reopened space mountain rocks! A lot faster, and a lot darker. We did it twice! And the 50th anniversary special fireworks at the end of the day were great, highlighting the coolest rides in the park. The fireworks for 'pirates' was awesome. But the best part is being there with no really long lines. We didn't spend more than 20 minutes in any line, and most were walk right on to 5 minutes.

I think that I'll include a picture from each of my travels in my blog entries over the coming few months. Here's a picture of Mt Hood from Lost Lake Butte, a 2.5 mile hike from Lost Lake gaining about 1500 feet in elevation. Pictured are my father in law, my youngest son Jackson (4), my oldest, Aaron (16), my 2nd son, Garrison (12), and me.

Static Code Analysis

adamu — Wed, 27 Jul 2005 08:30:00 GMT

I started using a tool this week that measures the lines of code (LOC) that are going into our source tree. I discovered several things that make me believe that measuring how much code we've written (along the lines of TSP or PSP) and some of the other properties of the code base can help me assess the state of the project. I learned:

It takes at least as much code to test something as it does to build something. Our test depot has 245,000 LOC, and our dev depot has 215,000. Automated testing simply has to embrace best development practices; the scope of the investment demands it.
Making incremental measurements so that I see the change over our milestone on a per week basis shows me when the dev team has really stopped adding code, and is stabilizing. Our dev team was averaging 1500 LOC per week for 9 weeks, and this last week, the number of files and code actualy decreased slightly. This is good, because we are entering a bug fixing/stabilization phase.
I need to learn how to apply cyclomatic complexity to our code base. According to the numbers I saw, our maximum complexity was 97 in the dev code base, and 52 on test. According to the references I reviewed, anything over 40, and you should think hard about rewriting this code.

Myth Busting Testing #3: Test Automation is like car insurance.

adamu — Fri, 22 Jul 2005 10:15:00 GMT

Ok, Myth #3: An automated test must find bugs to be useful.

Here's a scenario for you: you have a car, and you have an car insurance, right? Do you really need car insurance? if you're a safe driver, maybe you have a claim every 5 years, and maybe you pay out more in premiums than you get back (which must be the case, generally speaking, otherwise insurance companies would be out of business, right?). You haven't had a ticket in 5 years, you haven't had an accident in 10. But aren't you glad you have the insurance? You know if something happens, you'll be covered.

So think of test automation like that insurance policy.

Reality: Test's don't have to "find bugs" to be useful. Test automation's real value lies in validating that defects have not been introduced in previously working code. Tests that pass provide meaningful data about the state of the codebase under development and the development process.

Just as insurance gives you peace of mind, so to does test automation.

Let me give you another example.

My team has been in a coding milestone for 10 weeks now. Monday, we start a stabilization period. When we started the coding milestone, all our automated testing was passing 100%. Our smoke tests, for the most part, have been at 100% throughout the 10 weeks. Up until about 4 weeks ago, our set of daily automated tests were passing at 100% . But we took a partner drop, and sure enough, we had 2 failures in that larger set right after that. Next week, another pickup, and now 4 failures, the next week 7, and the next week 9. And we had fixed many of the previous weeks failures and bugs filed against our partner, but we couldn't get ahead of the incoming breaking changes.

So here I am, using my test automation finding bugs as an example of why a test's entire purpose in life is not necessarily finding bugs. Huh? Adam, didn't you say that the real value lies in the test validating that defects haven't been introduce? Well, what useful data can you derive from the tests passing consistently?

Confidence: higher confidence in giving my partners drops of our code.
Where to spend time: less time testing in those more stable areas, and focus more on the areas that we've seen defects in.
Our Churn: If things aren't failing at the rate we expected, we probably aren't seeing as many dev check-ins as we expected.
Partner Churn: If things aren't failing when we pick up partner drops, then we need to find out why the breaking changes we expected aren't causing failures. Did they actually make it in? Or are we just doing an amazing job of fixing all the breaks before we pick their code up?
Assessment of process: I can look at the test check-ins, and see if we aren't seeing failures because our dev and test teams are communicating efficiently about the breaking changes and addressing them before they fail in our test lab.

There's really alot of valuable data from a well designed test automation system that has well designed tests that "don't find bugs" because they are passing regularly.

One more time:

Test's don't have to "find bugs" to be useful. Test automation's real value lies in validating that defects have not been introduced in previously working code. Tests that pass provide meaningful data about the state of the codebase under development and the development process.

Myth Busting Testing #2

adamu — Sun, 26 Jun 2005 00:07:00 GMT

Myth #2: Testing does not, by itself, improve software quality.

I love this one! I must have heard it like a gazillion times. And taken at its most literal, it is entirely correct. Huh? Yep, it's entirely literally correct. So why am I calling it a myth? Because it implies that testing and quality are not related, and that is completely false.

Reality: Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Here’s a great quote from one of the best books on developing software, Code Complete by Steve McConnell:

“Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, they don't improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't test more; develop better.” - Steve C McConnell, Code Complete: A Practical Handbook of Software Construction

McConnell’s quote above is a fantastic one; it makes great copy. However, I completely disagree with the conclusion that implies that testing and developing are at odds (at the best the tone probably implies that they are orthogonal), I believe that the quote actually shows just how important it is to test more, not less. Because in order to achieve improvements in your software by 'developing better', you simply must test more. Test has to become inextricably entwined in the development process. Here’s what I mean…

You are sitting in a restaurant somewhere, perhaps it’s a Saturday, and you’ve decided to take a ride to explore highway 2, and you found a little greasy spoon diner, and since it’s lunch time, you stop in to grab a bite. As you’ve mostly polished off the blue plate special, the spatial relationship between what’s left of the open faced sandwich and the side of fries gives you a moment of elucidation: you’ve hit upon an idea to solve a problem you were stuck on for most of the week. As you design, before it’s even going down on the napkin with the crayons that are supposed to be for the kids, you are testing your ideas. You form them up, write them down, and then somewhere in version 27, after you’ve done this iteration of design and test, design and test, you eventually translate those ideas to the napkin. It’s even mostly readable, although the crayon tore the napkin a couple times. You pocket the napkin, pay the bill, and you are happily on your way again, seeing the sights on highway 2, and they seem rosier than before lunch, and they probably are, because now your mind is free from the problem it had been background processing on.

When you return home that evening, you bust out the laptop, and carefully unfold the napkin, and you start to translate from napkin to code. Hey, you are a geek and love writing code, what else are you going to do on a Saturday night? And somewhere in the iterative process of translating from crayon wax to electrons, you’ve designed something that you believe will solve the problem. At which point, what do you do? Why, you hook this thing up to the rest of your system, you create some test data, and then you test it. You build even more test data, and try it another way. You find a corner case you missed, you fix it, and you test it again. You throw garbage data at it, and you fix a couple bugs that were there that weren’t validating some inputs. And because you are such a great developer that your ego never gets in the way of creating great code, you run through all those same tests again to make sure that the changes you made didn’t break anything.

When you get back to the office, you schedule a design review. Yep, you test it again. Once you’ve made changes to the design from that review, you implement the code, and then you have a peer do a code review. You see, you’ve measured (tested) your coding proficiency, and you know that for every 100 lines of code you write, there are probably 3 bugs. And sure enough, you find 4 during the code review (yep, you tested it again). You are ready to check it in to the source tree, so of course you run the standard set of check in tests on it. And then, because you are a great developer, you write unit tests for it so that when someone else checks some code in, or makes a change somewhere else in the code base, there's an early warning system that can signal that there's a problem.

As part of the check-in, you write a test release document on the code you’ve checked in, and you fire it off to your test partner. You know that the software design engineer in test (SDET) who will read it, will probably think of how this integrates with the larger system under test, and write scenario based tests for it. If he’s really good (and he probably is, because great developers always want to work with great testers, they have no ego problems) he probably has already had the scenarios coded up, and now he can go back in and fill in the gaps that your document has pointed out as potential areas of concern. Wow, you tested it yet again. You review the unit tests with your SDET partner, and he finds a corner case you missed, and he says he'll go ahead and add it to the test suite, and then he adds them to the check in tests and daily automation run. He can do this because the development and test teams use the same test harness for unit tests, checkin tests, functional tests, so any test that dev writes can be leveraged by test in the test lab, and any test that an SDET writes can become part of the dev unit or check in tests.

So if you want to lose weight, you do weigh yourself all the time. Why? Because you want to know where you stand. You want to make immediate course corrections, not wait a month and find out that you’ve put back on 3 pounds. You want to do it far more frequently. And you don't use just one scale. You measure yourself with a tape measure, and when you put on your jeans. You measure yourself when you look in the mirror. And while some measurements are more qualitative and/or subjective than others, all of them are useful and help you towards your goal of losing weight.

Think and Test, Design and Test, Develop and Test. Fix and Test. Fix more and Test. Then you hand it to someone else and help them test it. It’s what great developers, and great development teams, do.

Better development is better testing. If you never measure your code’s quality by testing, you'll never know if the quality is sufficient.

I will say it again (because I can, it’s my blog, and because it’s important)

Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Do you have that kind of relationship (communication and tools) with your test partners? Shouldn't you? At Microsoft's engineering excellence days during the last 3 years, we've talked alot about 'pushing quality upstream.' This is half of what I think it means. There's more on what test and pm can do as well. What do you think 'pushing quality upstream' means?

Myth Busting Testing and Test Automation

adamu — Thu, 23 Jun 2005 13:27:00 GMT

I've been reading a lot lately, and wow, there are so many opinions on the advantages and disadvantages of writing test automation. I thought I’d share my observations as to the common misperceptions that anyone in the software development business (engineer or manager, development, test or pm) needs to understand about testing in general and test automation in particular.

Myth #1: There are a set of tests that I would execute manually, but would never write test automation for because I'd only ever run them once.

If you would only run a test once, then you've written the wrong test, or you haven't started testing soon enough.

I am going to say it again (because I can, it’s my blog, and because it’s important)

If you would only run a test once, then you've written the wrong test, or you haven't started testing soon enough.

Quite simply, if there is a test that you’d only run one time, your test design is wrong. The test should be more generalized, perhaps use data sampling to provide variability to the test. Well designed tests are the key to testing, whether that testing is automated or manual.

OK, Adam, assume that I believe you about writing more general tests, but what do you mean about testing soon enough? I thought that writing test automation too soon is a bad thing. It breaks all the time, and my maintenance cost is too high.

My point here is that if you’ve waited to test a component such that if you run all your tests and, assuming that there were no failures, it’s ready to ship, then you haven’t been involved in the development process early enough. I think that this is a great topic for a future more detailed post, so for now, I'll focus on the test design part of the problem.

A quote from James Bach from Test Automation Snake Oil:

Once a specific test case is executed a single time, and no bug is found, there is little chance that the test case will ever find a bug, unless a new bug is introduced into the system. If there is variation in the test cases, though, as there usually is when tests are executed by hand, there is a greater likelihood of revealing problems both new and old. Variability is one of the great advantages of hand testing over script and playback testing.

And another from Brian Marick on When Should a Test be Automated

The fact that humans can’t be precise about inputs means that repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug. For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.

While I completely agree with what James and Brian are saying regarding variability, I completely disagree that you can’t have the same variability in your automated testing as your manual testing. It’s harder to design and implement test systems that solve the variability problem and yet can be reproducible, but hey, that’s one of the reasons we build software: to solve complex problems. It does mean that typical techniques that James mentions (scripted and playback testing) are not useful mechanisms for more effective test automation.

Now there may be a set of tests that may be too expensive to automate, but automating because of expense is an entirely different reason, and one that I will discuss another time.

Here’s another great quote from James Bach’s snake oil:

One day, a few years ago, there was a blackout during a fierce evening storm, right in the middle of the unattended execution of the wonderful test suite that my team had created. When we arrived at work the next morning, we found that our suite had automatically rebooted itself, reset the network, picked up where it left off, and finished the testing. It took a lot of work to make our suite that bulletproof, and we were delighted. The thing is, we later found, during a review of test scripts in the suite, that out of about 450 tests, only about 18 of them were truly useful. It's a long story how that came to pass (basically the wise oak tree scenario) but the upshot of it was that we had a test suite that could, with high reliability, discover nothing important about the software we were testing. I've told this story to other test managers who shrug it off. They don't think this could happen to them. Well, it will happen if the machinery of testing distracts you from the craft of testing.

Again, I agree entirely with the sentiment, but I completely disagree when he indicates that automated testing was the culprit. This is not a test automation issue, it’s a test design issue. You could just as easily have run 450 manual tests that were poorly designed that would also tell you nothing useful about the code under test.

That it’s harder to write well designed automated tests is a given, but heck, as I stated above, why do we write any software? To solve some highly complex problems that would take us longer to do manually. Test automation, while having its own set of unique problems, is still, at the end of the day, just software that is attempting to verify the state of another piece of software at any given point of time.

When tests are correctly designed, and you are involved in the process early enough, there should never be a test that you’d only run once.

Engineering Excellence and TrustWorthy Computing

adamu — Wed, 22 Jun 2005 08:28:00 GMT

For the last several years, Microsoft has had days devoted to Engineering Excellence (EE), and to TrustWorthy Computing (TWC). Much of EE days in the last couple years have been about pushing quality upstream, which I interpret as getting more testing done sooner in the process, whether it is design reviews, code reviews, or unit testing, but ultimately before code is checked in. TWC has been about the pillars of privacy, security, reliability, business process integrity and transparency.

This year, they combined the two events into a single 3 day event, with dozens of in depth talks on a broad range of subjects related to EE and TWC. You can learn more about TWC here.

The keynotes were good, and the final keynote was by SteveB, who was as fun and spontaneous as ever. It is amazing to see how far we've come as a company with regards to these two key initiatives in just 3 years.

The most interesting thing I saw today was on Team Software Process (TSP) and Personal Software Process (PSP) that the BUIT has been using with good results. I am going to read a couple of Watts S. Humphrey's books on this and see if we can apply it to what my team is doing. More importantly, it got me to think about what we are measuring with regards to productivity (lines of code written) and defects injected. There was a quote used, something along the lines of "If you aren't measuring, you aren't engineering. I don't know what it is, but it is not engineering." If anyone knows the source of the quote, and the exact quote, please let me know.

Father's Day Camping - Acrylic Style

adamu — Tue, 21 Jun 2005 07:24:00 GMT

I was camping this weekend in Central Oregon. Each year my wife's father's siblings (all 8 of them) and many of their children and grandchildren get together (50 to 60 people) and go camping just outside Sweethome, Oregon, which is about 30 miles east of Salem. I say camping, but with the facilities at Camp Koinonia, it's hardly camping; more like a retreat. It's always fun; swimming, the annual horseshoe tournament, wonderful food, and catching up on what everyone's been up to. I did a geocache (www.geocaching.com) while I was there, hiking to the top of the north butte (just north of foster dam), and brought my little minolta digital camera along, and got some great panoramic pictures by stitching them with Acrylic. The first picture is a complete 360, the second is the about the 3rd 8th of the first picture, more detail on the lake and dam area. Click on them to see them much larger. Acrylic did a great job, especially with the light to dark transitions. I was showing the photo stitching to several of the moderately tech savvy folks at the campout (one was an eastern oregon ranch realtor, another was a IT student at BYU-Idaho, another was a landscape architect), and they were ecstatic that they could get the beta to try it out. (www.microsoft.com/expression).

Acrylic Greetings from Beijing

adamu — Fri, 17 Jun 2005 15:26:00 GMT

From the 8th to the 14th, I travelled to Beijing to meet with 5 vendors who've been working on one of my projects for the last year. Yanchun (Tracy) Guo, who is a test lead for Microsoft in Beijing, helped me out by taking me out to lunch, and in general being my babysitter for a couple days at work. It was great to get face to face time with them, see their offices in Beijing, and meet with the vendor companies at their offices. And experiencing Chinese culture first hand was a great experience. I am looking forward to a visit again in a few months.

One of the projects I've been working on, code named Acrylic, went to Beta on the very first day that I met with them in Beijing. There's been quite a buzz about Acrylic since the release, which to me is quite surprising. The Expression site with the version 3.3 bits has been up for over a year now. Was seeing a new beta really surprising? The beta has a new coat of paint for the interface, has the ability to have pixel layers in addition to the vector layers, has live effects, photo stitching, and I'm sure there's a couple other features that I'm forgetting.

Anyway, I got to see the sites while I was in Beijing over my weekend there, and I used Acrylic to stitch my photos together for some great panoramic photos. The top photo is 180+ degree view of one of the many courtyards between gates at the forbidden city. The bottom 3 are from Tiananmen Square, the mausoleam where Chairman Mao in entombed, the Chinese National History Museum, the Monument to the People's Heroes, and the other building is the Great Hall of the People, where the congress and standing committee meet. Beijing is busy getting ready to host the 2008 summer olympics.

Test Automation, plus what is it like to be in Test at Microsoft?

adamu — Thu, 16 Jun 2005 06:39:00 GMT

It's not sexy, but it sure is fun.

Santosh recently had a great post on what it's like being in test at microsoft. For those who read my posts from last year on campus and industry recruiting at microsoft, I would encourage you to also read this to get the sense of why being an SDET at Microsoft can be so rewarding.

Test Automation

Michael Hunter, aka the Braidy Tester, has had an excellent set of posts on my team's approach to test automation. It is essential reading, as will be his upcoming example series.

not as simple as Red vs Blue

adamu — Tue, 26 Apr 2005 07:47:00 GMT

I ran into Adam Barr (www.proudlyserving.com) last week while volunteering to assist with a local Vocational High School's computer curriculum. Adam had left a few responses on my Blog on exchanges on testing at microsoft, and it was a pleasure to meet him in person.

I was reading Adam's blog post from the 24th, and it's clear that he and I see eye to eye on many things such as diversity and it's effect on business, but one large disagreement that I had with his post was how he viewed being in a "blue" state as an advantage. Here's a great take on the red vs blue states; I think it's a more accurate breakdown of the US political landscape. Finally, here's what I see as the most accurate analysis of pork spending. Based on this report, it's clear that the reason it's so good to be in a "blue" state is because of all the pork that comes back when you have democratic representation.

great updates for winXP

adamu — Fri, 03 Sep 2004 00:21:00 GMT

I've been on vacation with the family and busy with tons of other stuff, and I realized that I hadn't posted in over a month! Wow, time flies. While my next test topic post is a discussion of why you should do test automation (it's about half complete), I thought I should post that windows xp sp2 just rocks. The additional security features are going to make my life alot easier. Why? Well, I'll tell you.

First, SP2 isn't for me; I mean, it's not for the type of user I am. It's for my mom, my aunt, my sister, brother, my kids and my wife. They are not professionals with computers, although they use them alot. They probably don't understand all the implications of clicking on a link in a browser or in email. I do. I know what havoc can be wrought upon your machine. I've seen Slammer, MyDoom, and a dozen other things in action in the last several years in my corporate environment. I've seen the evilness that Kazaa is as it brings your machine to a grinding halt on friend's and familes' computers.

So SP2 makes your windows xp computer 'secure by default'. instead of pulling the trigger on something and getting a hole in your foot (it's painful, takes a while to recover, but you won't die), you'll probably have to pick up the gun, put a bullet in it, point it at your foot, undo the safety on the gun, and then pull the trigger. If my analogy failed for you, the point is that you have to jump through alot more hoops to shoot yourself in the foot now.

It also means that some programs that weren't written in a way to comply with windows standards won't work anymore. If they use part of the registry they shouldn't, they will now fail. So it will be painful for some companies to fix their programs.

So why does it make my life easier? If you are like me, you spend alot of time in family and friend customer support. Uninstalling Kazaa for a friend, fixing something because someone got a worm on their box, is a bunch of my free time.

So thanks XP SP2 team for giving me back some of that free time! Now I better get back to finishing my Why Test Automation post.