notes and rants

I finally received my feedback from the futuretest conference. When I get feedback from a talk or class, I usually take about 3 seconds to look at the overall numbers, then start reading the comments. The numbers are nice, but I find the comments to be a much better gauge on how well my talk was received.

Overall, the comments were good – people (in general) felt like they learned something and liked the insights. One comment, however, had me a bit confused. It said:

This is a redo of what you said at STPCON. How do you differentiate the two (presentations) when you rerun?

Of course, I’ve never presented at STPCON, and I’m pretty sure the other MS folks who have presented there never gave a presentation similar to this one. Perhaps it was an extreme case of déjà vu (or a big glitch in the matrix)?

A channel 9 interview with James Whittaker is now available on channel9.

You may know James from his “How to break software…” series of books. He’s been at MS for a year or two (after 10 years at FIT) and has recently moved to the team that makes Visual Studio Team System Test Edition.

My boss is taking a little bit of heat for his latest blog post. The entire point of his blog is to be provocative, but in my opinion, he didn’t get his intended point across.

So, how do you tell your boss that you think he’s stupid? The answer is easy – you don’t. But - you can tell him that you think his idea is stupid…as long as you’re careful.

I love conflict (although I’m told that “open, inclusive discussion” is a better phrase). I believe that if people hash out their issues with no holds barred, that they will reach a solution that they can support – even if the solution is contrary to their original beliefs. In other words – I am more apt to support someone’s decision if I felt that my side was properly heard.

Of course, in order to have a conversation like this, you need one important thing – trust. If I think by boss will fire me or take away opportunities if I disagree with him, it’s in my best interest to keep my mouth shut. This also means that I won’t be very motivated to support my bosses decisions. He also needs to trust me, and know that I’m not disagreeing with just to be petty or to undermine him. You need trust to have productive conflict. Conflict without trust is just a fight, and a fight is just a waste of time.

Even if you’re comfortable disagreeing with your boss, you can’t just say “you’re wrong” and wait for the response. Your side will be better supported if you show a little empathy for the other side. Rather than say “Eric – this is stupid”, it’s better to say something like “Eric – I think I realize what you’re trying to say here, and I do appreciate the provocative style, but the main points that are coming across in this post are way outside of my comfort range. For example … can you help me understand your thinking?”

If you never disagree with your boss, do you truly trust them 100% in their decision making? A survey in Britain last summer says that nearly half of all employees claim those on the next rung up of the ladder are poor decision-makers. I wonder if those who don’t think their boss is a good decision-maker ever bother to disagree with their boss…or if there is enough trust in the relationship to enable an environment where disagreement is a good thing.

BTW – my boss isn’t a moron, but I do love to disagree with him whenever I find it appropriate, and appreciate that we have enough trust between us to make the disagreement productive.

Matthew Heusser recently asked What is a software architect anyway? Matt when on to ask what a test architect does. This question was so good (and the answer so long), that I thought I'd respond here.

So, what is a test architect. One answer is that it's a fancy title for an experienced tester. At Microsoft, we try to avoid this definition, and instead describe test architect as a senior test role with wide strategic scope. That's the short answer.

Here's the long answer.

There is no “typical” test architect role. Test architects focus on a diverse set of goals and perform a wide variety of tasks. Some spend time developing testing infrastructure, test authoring frameworks, or evaluating features in order to create complex tests. Some are in charge of a particular technology for their group. Others spend time consulting on how to improve test effectiveness. The common thread across all test architect roles and the primary responsibility of a test architect is to provide technical leadership and strategic direction for their testing organization. It is also expected that in addition to accountability to the current product, that senior test architects will consistently look beyond the current release and may have several deliverables not tied to a particular product release.

A test architect has in-depth knowledge of a variety of testing techniques and methodologies used both inside and outside of Microsoft. They often provide technical assistance and/or advice to the test Manager. The question often arises - “Shouldn’t the test manager be doing all of this anyway?” Typically, the test Manager is the individual providing leadership and formulating team strategy. However, as organization sizes continue to grow and/or the test manager takes on additional responsibilities, it often makes sense for a test architect to step in and assist with or deliver on these responsibilities. The test architect often takes on some leadership roles typically associated with the test manager. For example, while test managers certainly will implement change to grow their teams, the test architect is frequently the individual who provides technical leadership and strategic direction for their organization.

A test architect is expected to be able to affect change not only across the testing community, but between other engineering disciplines as well. Test architects must drive quality across all disciplines, providing guidance, feedback, and suggestions to improve quality practices across an entire engineering team.

While a few test architects may be focused on a specific problem or improvement, the goal for the test architect investment should be long-term improvement of the organization testing process and growth. Senior test leaders, when faced with an urgent problem or situation in need of quick improvement can typically find a solution. Broad or recurring issues, however, may require a test architect. The test architect should be thinking long-term and laying out a path for solving big issues over a long period and not focused on fighting daily fires. The test architect must be focused on defining the testing process and setting the test team up for continued success.

In my previous post, I mentioned that Goals in GQM should be S.M.A.R.T. (Specific, Measurable, Attainable, Relevant, and Time Bound). A reader questioned how applicable SMART is to every scenario, and I thought it would be worth it to reply here. I use SMART when I am working with measurements - in other words, I believe that when dealing with measurements, SMART goals should be applied.

Can I apply SMART, for example, to solving world hunger. I don't know - let's see...

Let's start with a goal "Solve world hunger". That goal fits none of the SMART criteria, so I better try again.

Specific - for this goal, what I want to focus on is number of deaths and disease attributed to malnutrition. This doesn't cover everything to do with world hunger, but it's important that goals are specific.

Measurable - How do I tell if my efforts have worked. I do not know what the current data is on malnutrition and hunger issues, nor do I know what has been done already, but for the sake of exercise, I'll say that my goal is to reduce deaths and disease each by 10%.

Attainable - again, I'm not well versed enough in the area to know if my goal is attainable, but let's assume it is

Relevant - I'll assume that less sick and dying people is a good thing and is relevant to the above points

Time bound - Oh yeah - didn't mention this in the measurement above, but let's do this by the end of the 2009 calendar year. I can also attach time bound to the measurement by saying that the 10% improvement will be based against 2008 data.

Put it all together, and my new goal is:

"Reduce the number of deaths and sickness caused by malnutrition and hunger related issues by 10% in 2009 (as compared with 2008 data)."

Now - is that enough to solve world hunger - of course not. There will be sub goals that deal with the root causes of world hunger. For example, I will likely have goals around the number of bags of strong stock seeds and quality fertilizer distributed to appropriate locations, how training on growing food is distributed, or how wealthier countries are sharing their crops.

Will you now apply SMART everywhere? Only if you think you can build a house with only a screwdriver. I do believe that it is highly applicable to areas where you want measurement, but of course it's possible to do meaningful work where you don't want measurements.

I will be happy to fake my way through any examples you want to throw at me in the comments.

I was typing a response to Matt on his blog to this post, when I realized two things.

1 - the comment forms on blogspot blow

2 - due to the length of my reply and #1 above, I thought it would be ok to respond here.

NOTE: If you don't know what GQM is and need context, read this (wikipedia), and look at Matt's previous post.

In short, the purpose of GQM is to measure something meaningful rather than the random assortment of metrics that most teams look at. Like most things in life, if you do it wrong, it doesn't work.

The OIM concept that JB responded with is hard to disagree with. It basically says "look and think about what is happening, adjust appropriately, and repeat". I don't like that the example is "Is X doing testing well" - people metrics are almost  always going to be wrong. Perhaps that is why I don't get the model...?

I'll be the first to admit that GQM is easy to get wrong, but bear with an example (and one not based on individual performance).

For GQM to work, the goal needs to be SMART (Specific, Measurable, Attainable, Relevant, and Time Bound). Let's say for example, that you're team is struggling with estimates and are trying to improve. In fact, management has asked you to improve estimates and provide data! A (smart) goal could be "Improve the accuracy of estimates for all projects by 20% in the next year compared to last year"

Questions may be results based: "What percentage of tasks planned for last year or this year were completed?", or progress based: "How accurate were my initial estimates for the last milestone?", "Did our team accomplish all of its objectives for this month?". From that, you may come up with metrics like:

• % of tasks planned versus completed for last year versus next 12 months
• Error of estimate for last milestone abs(actual-estimate)
• % of tasks accomplished versus planned for last month

But wait - metrics can be gamed. In fact, it's easy to game metrics. This may be where JB's OIM model comes in. If you think your metrics will always tell you what you think they will, you are in for a big, big surprise. The answer, of course, is that you test your metrics before using them, then watch and make sure they are measuring what they think they are. Metrics aren't special - anything you follow blindly will come back to bite you.

I’ve been thinking about testing and quality again lately. Across the industry, testers call themselves “quality assurance”, or “quality engineers”, and testers say they “measure quality”. I’ve had serious thoughts recently whether the two terms  (“quality” and “testing”) are as related as most people seem to think.

Testers don’t measure quality (and they certainly don’t assure it). The role of test is to provide information – but that information is just data – data about how the product was engineered. Things like code coverage, test pass rates, and bug rates are just status about the engineering effort.   Product quality – or perhaps better put – customer quality is different. Customers don’t care if you had 90% code coverage. Customers don’t care what your test pass rate is, and don’t care how many bugs they found. To them, quality is more emotional – quality products make them feel good, and help them accomplish a task in a new or exciting way. Somewhere along the line, someone decided that engineering quality was the same as product quality. Granted, the choices testers make to measure engineering quality are based on experiences in customer reactions, but it’s a crapshoot how much the the efforts align.

Graphically, you have two different measurements. In the worst case, they miss completely. For example, think of a level 5 cmm company that delivers bad software – they did it because they missed the boat on matching their process to measuring customer quality.

In a perfect world, the two measurements match exactly. In reality, I bet that engineering quality and customer quality match somewhere in the 25-50% – maybe 60% at the most.

To get customer and engineer quality to match, you need measurements that can predict how the customers will perceive quality. This dictates that the best measurements would be from customer feedback during product development – things like beta feedback, newsgroups, and customer surveys may be more valuable than you think.

So what do you do with the engineering metrics? You keep them. If the role of test is to provide information, the right set of measurements can be the perfect tool for assessing risk. All that stuff like code coverage and pass rates that the customers don’t care about – they don’t care, but you still need to do it. If I buy a new car, I don’t care how many parts were replaced on the assembly line as long as it looks and works perfectly when I drive it away. Even though I don’t care about how the car was engineered, my perception of quality is affected by the quality of the engineering process – whether I know it or not.

How can such a simple question have so many answers. ISO has an answer, Crosby has an answer, Juran has an answer, Weinberg has an answer, and thousands of others have their own answers or variations on the answer. The problem with most answers is that they are as vague as the question. The problem with the question is that quality means so many different things to so many different people.

Taguchi gets it close by defining quality as “The loss a product imposes on society after it is shipped". The key thing I draw from this statement is that many aspects of quality seem to be unknown until after a customer uses the product. The challenge in defining quality in any field is coming up with a definition that can help create quality in the first place.

I sit here looking around my desk – I see a phone, some headphones, pens and paperclips. What is it that defines quality for these items. What makes one paperclip better than another, what defines quality in headphones or a telephone? Would your answer be the same as the person across the hall? What is quality? I think I hate that question.

It's not hard to find articles or blog postings on the web talking about when a test should or shouldn't be automated. I also know that there are plenty of "test automators" in the industry who spend too much time trying to automate things that shouldn't be automated. As extreme as the automation camp can be at times, I am even more surprised to see testers at the other end of the spectrum - those who say that only humans (or thinking humans) can test effectively.

Test automation is, perhaps surprisingly, context driven. If I'm testing an internal application, I doubt I would ever write automation for it. I may not even write automation for a shrink wrap application if the context was right.

Of course, the context at Microsoft dictates a higher level of automation. I would estimate that 70% of the automation written at MS is non-UI testing. Given that we create operating systems and platforms, we have a lot of APIs and object models to test. Add to that the number of configurations these platforms and systems run on, testing without automation would be idiotic. I would also estimate that 99% of the UI testing (within the 30% of UI testing for those of you doing math at home) is done via an object model or via an MVC interface such as IAccessible. This sort of UI automation tends to be a bit more resilient to changes and breaks (a little) less often.

The "other" UI automation - that tiny bit done by attempting to send keystrokes and mouse clicks is always problematic, and I don't think it ever has a reasonable ROI.

The problems with test automation, and probably the root cause of everything the anti-automators have with automation, is automating the wrong tests. If you've ever talked to me about this subject, you know that my opinion is that you should automate 100% of the tests that should be automated. Where people make mistakes on both sides is determining which tests should be automated. If you tell me that everything should be automated, or that nothing should be automated, I think you are telling me that you don't know how to figure out which tests to automate - which means you're not thinking about the problem.

Michael Feiner wrote a book called The Feiner Points of Leadership (get it?). The book has 50 laws, but they mostly say the same thing - Leadership is about managing relationships. A better subtitle for the book may be "50 examples of leadership", but it doesn't matter - it's a good book, and if you are a leader (or want to be one) I think reading it is worth your time.

As a manager, I have obvious responsibilities to manage my relationships with my employees - as well as try to manage their relationships with each other. I also need to manage the relationship with my own boss. Every manager has these same relationships, but not every manager is necessarily a leader. I don't think you can be a leader by only managing direct relationships (i.e. your direct reports and manager) - that's just management. The challenge in leadership is managing all of the other relationships you are responsible for. Leaders must manage relationships with peers, peer teams, customers and many others to be successful. Good leaders build trust and credibility in all of their relationships, and use those attributes to make everyone perform better. Influence with out Authority is something that leaders need to be able to do. Making people to do stuff just because they work for you is never going to be a key to leadership success.

Of course, I'm not saying I do all of this stuff, but I do think about it. I know that I don't work enough with my peers, and I know that I need to do a better job managing relationships with our customers, but it's something I think about often, and something I hope I can  continue to get better at.

I'm somewhere over the Rocky Mountains on my way back home from New York (via Atlanta...long story). While I was in Long Island, I borrowed a charger for my Zune (I forgot mine), so I was happy to have a full charge for my return trip. Unfortunately, one of the ear buds on my headphones (shure E3) went awol (I think it's a design defect that they fall of so easily), so I'm unable to tune out the idiocy of my fellow passengers.

I'm a firm believer of the golden rule (probably a defect as well). That's why I'm amazed when someone walks up the aisle grabbing and pulling on the back of every seat...then does it again...and again.

The guy across the aisle from me (and about 1/3 in the aisle as well) is rocking out. To be fair, I have no idea what he's listening to, but I can tell you the tempo of every song as he slams his hand into the armrest to the beat (for the record, about 160 beats per minute).

The good (no great) news is that I don't think I have any travel on my agenda until September. I don't mind traveling, but there's been a lot for me lately (defective planning on my part), and it will be nice to be on the ground for an extended time.

Almost forgot to mention - I had a great time in Long Island. The forefront folks are a great bunch of folks to work with and did a great job putting up with me.

Several years ago, I was on a product team in the midst of a ship cycle. I was part of the daily bug triage, where we reviewed, assigned, and sometimes - postponed bugs to the next release. Postponements happen for a variety of reasons, and are a necessary part of shipping software. A few months before shipping, we had some time left at the end of the meeting, and I asked if we could look at the bugs assigned to the next version of our product. The number was astounding. It was so big, we started calling it the "wave". The "wave" meant that after we shipped, we would be starting work on the next release with a huge backlog of bugs. Yuk.

Our bug backlog is a form of Technical Debt. We constantly have to make tradeoffs when developing software, and many of those tradeoffs result in technical debt. It's a tough thing to deal with, and we (software industry) aren't always very good at handling it. 

Matt Heusser has blogged about this before, and is holding a peer workshop on technical debt in Michigan in August. The call for proposals is here.

Check it out.

Futuretest wrapped up today. I thought my talk went well - it was a bit of a challenge for me to take a huge topic and whittle it down to a 45 minute presentation, but I did it and managed to make all of the points I had planned.

Joel Spolsky spoke immediately before me and did a fantastic job. His topic was software quality, but it didn't matter. He could have talked about the price of Tonka toys. He had the audience laughing constantly and linked his points together flawlessly. There were, however, two minor drawbacks to his presentation (none having anything to do with his content). The first is that he was engaging and entertaining enough that I knew following him would be hard. The second is that his sense of humor is close to mine, so I had to consciously change my approach a bit.

Despite Joel, I received a lot of complements on the talk and was happy to see it made a mark on on the participants.

Perhaps a more important statement to make is that this was one of the best conferences I have attended. I enjoyed every single talk, and it was refreshing to see a test audience who saw the value in a holistic approach to testing and quality assurance. I think I may spend too much time trying to play in a part of the testing community that I ultimately don't end up agreeing with very often.

I had a conference call to attend to and missed the final session, but apparently the dates for the 2009 conference were announced. I'm going to see if I can weasel my way back in next year.

Btw - if you found my blog from the futuretest conf (i.e. you were there and saw the link on my final slide), feel free to ask any follow up questions here, or drop me an email.

When I started at Microsoft, I thought it was a pretty big company (near 15,000 employees worldwide). These days, it's huge (pushing 90k employees or so).

There have been some growing pains over the last decade. Communication across teams is harder, and haring tools and processes is harder. Decisions sometimes take longer, and there are more layers of management between the top and the bottom of the organization.

There are some advantages as well. If anyone ever wants to work on a different product - or even change fields completely, it can be done without leaving the company. I know HR folks who have become testers, and developers who have become marketers. I don't think I'll ever leave test, but it's nice to know I could.

Another advantage of a big company is that it's everywhere. I'm in New York this week for futuretest, and the wireless internet at my hotel is horked. I have a ton of work to get done today, and about 75% of it required access to files and data not on my laptop. I knew MS had a Manhattan sales office, as well as a small R&D shop in the village. So, this morning after I gave up on the wireless, I went to my hotel lobby to talk to the concierge and asked for a copy of the white pages. I found the Microsoft entries (there were several), and wrote down the address of the biggest one (the one with several phone numbers listed). If you haven't been to Manhattan before, one of the most important things to know is that the street numbers, while ordered, have nothing to do with the numbers of the streets. For example, 340 West 8th is between 8th and 9th, and 660 Madison Ave. is near 60th street. There's a method to the madness - or at least a convoluted algorithm here. I don't mind the confusion, and can usually get to where I'm going without any problems (or ask a taxi driver to take me there).

Anyway, as I was looking up the MS addresses, I saw that there was an office about two blocks from my hotel. I figured it was something small, but I was hoping there would be a table somewhere where I could set up shop. I walked in the building, flashed my ID card and followed someone else with a similar badge to the 6th floor. Within minutes, I was set up in a small conference room and drinking a familiar cup of coffee. The office itself is nice sized. There are a few training rooms, and at least fifteen conference rooms (sales people and their meetings ...).

Now, I just need to find a Microsoft office in Paris or Italy.

I'm in New York next week - first for Future Test, then off to Long Island to teach a quick class before heading home. New York is one of my favorite cities, and it's been a few years since I've been there - so I'm taking an extra day while I'm there to wander around and have some fun.

Apparently, I've been running a bit beyond full capacity for the past few weeks. A few days ago, I noticed that I mis-booked by hotel stay by one night, but I quickly cleaned that up. Then today, I was looking up conference details, and realized that while I thought the conference was on Monday and Tuesday, it is actually on Tuesday and Wednesday. It really doesn't have any impact on my trip since I planned to head out to LI on Wednesday anyway (I'll just go a few hours later than I originally planned), but I can't help but be a bit troubled when I make mistakes like this.

Let's hope the week goes well...