As a continuation from my last blog entry, here are some more thoughts I have about code coverage measurements.
· Never roll up code coverage data without the interpretation of that data. Without some text about what your numbers mean, people will draw different conclusions which sometimes will require you to then provide more detail than is truly necessary. So always include key takeaways or explanations with your data. And remember that in most cases the numbers are relative, not absolute, so normalize.
· For example: developers start writing unit tests and find their code coverage is up in the 90% range. Wow! How is that possible? My testers have been writing tests for months and are only at 45% CC. After some investigation, we found that we were reporting coverage numbers on a different amount of files. The devs had 90% CC for one file and the testers were at 45% for 10 files. Once we normalized this, the developers’ unit tests only had 2% coverage.
· Hey, my developer just checked in a ton of code and now my code coverage number is down, what do I do? Well, you can't stop your developers from creating features for the product. And you can't stop them from creating new files in the process of doing this. So instead you need to make sure to explain the reason why the numbers are down, and what your plan is to fix it. Obviously, you'll need to run more tests on their new features. A decrease in code coverage numbers always deserves an explanation.
· Code coverage does not prove quality; it only proves how effective the test team is at hitting all the code. Used correctly, it can help drive quality and efficiency to some extent.
· Some lines of code will get hit many times during a test pass. That won't be represented in code coverage. But that doesn't mean that your testers should only hit a line of code once. Your team needs to run through all the customer scenarios during a full test pass and if in doing that, the same code gets hit many times, then so be it.
· I've heard some great ideas around using code coverage to drive focused testing, especially for hot fixes. If you can map specific test cases to specific lines of code that they hit when they are run, then if your product needs to release a hot fix (a bug fix after product release), targeted efficient testing is possible. You'll know the specific lines of code the developer changed for the fix. All you need to do is run the test cases that map to those specific lines of code that were changed. In general, it sounds like a great idea. But I've never seen it fully implemented and proven.
· If you have some old code in your product and those files generate low coverage numbers, should your team spend time writing more test cases to increase coverage on that old code? There are some significant tradeoffs with doing this. If your product has been used by customers for some time and therefore no critical issues are being reported in that area, what exactly will increasing the code coverage in that area solve? What's the benefit? Your testers will spend a lot of time increase a number, yet potentially not increase product quality at all.
Code coverage is a powerful measurement that can drive great improvements in your testing when used correctly. Make sure you understand what the numbers mean and check them twice. Accurate code coverage numbers are sometimes more elusive than you'd expect.
Instrumenting your product's source code to get an indication of how much of the code was covered during testing is a really, really smart measurement to get. If you aren't at least measuring this number, you should be. Visual Studio provides features that continue to make this easier. What does this have to do with fire? Well, code coverage measurements are to managers like fire was to cavemen. Discovering how powerful these measurements can be is enlightening! It's totally awesome! You can do so much with it! Grill up some mammoth, roast marshmallows, oh wait, are we talking about fire or code coverage? But just like fire, with code coverage measurements you need to understand what you have, or you will get burned. Ok, so play with fire, just don't start a forest fire - Don't draw conclusions from data that isn't fully understood. Here are some tips so that you can keep it all real when dealing with code coverage numbers.
-
Know what you are measuring. You instrument a file, you run your tests, your code coverage is high, you think you are great. Really? Do you know what you are measuring?
-
How large is your file? Small files can potentially have high coverage numbers.
-
Are you measuring the right file? I've seen some mistaken data that was being pulled from the wrong file and therefore more inflated coverage numbers were shown than were actually occurring.
-
What is the code and feature you are measuring? C++/C# code - easy to instrument. Jscript - a bit harder to measure but possible. XSLT - is that even possible? SQL - possible with the right tools. ETL - still figuring that one out ourselves. What your features do and therefore what language they are written in (and what technologies they use) have a huge impact on what your coverage numbers will be.
-
Maybe your numbers were unexpectedly low. Can all the code in the file be instrumented? Do you share the file with other teams and are you really expected to cover their code when running your tests? This leads me to my next point.
-
Be very careful about setting one code coverage target number.
-
Your code coverage numbers on each file could vary a lot for many good reasons. Trying to hit one universal code coverage number will drive you crazy. Predict code coverage numbers per file and base it on how many lines (blocks, arc, whatever is your base unit) of code is actually be covered.
-
Years ago in some of the large teams I've been on, there was a code coverage target for everyone. Yep, all those thousands of people testing thousands of files, and we all had to have them roll up into one number, 70%. The logic seemed reasonable. With that many files, they should average out easily. The problem was that in feature teams where they weren't hitting 70%, the testers were doing crazy things with their test cases to drive the numbers up. Instead of inventing test cases and scenarios based on the customers, testers were creating unrealistic scenarios in order to hit some obscure line of code that may have been in the product for years and never really used. Good intentions sometimes drive the wrong behavior.
-
Take action based on the extremes
-
Honestly, as a manager, 70% CC doesn't tell me much more than my test team is doing well at covering the dev code. And the conclusion is to keep doing what they are doing. I don't take action on CC numbers that aren't extremely low or high.
-
In some places, like testing APIs, I expect a higher number like into the 90%.
-
When CC numbers seem too high, investigation and root cause analysis is the action plan to make sure the data is accurate.
-
Only when the code coverage numbers are low do I really believe my team needs to take action and focus differently on the work than what they have been doing.
Those are a few of my thoughts. Code coverage is an interesting topic. I'll let everyone absorb my info above for a while before I continue with more of my thoughts and tips on this topic. It's cold outside so I think I'll go light a fire now.
Filed under: test manager, test, quality, planning, management, project, results, quality assurance, shipping products, software development, engineering, source code, developers, code coverage, effective testing
Since the nights are getting longer and it's Halloween, let's talk about going dark. This isn't about wearing a black costume for trick-or-treating! Going dark is a term used to describe the situation where someone hasn't communicated in a while. This could be on any topic but mostly for individuals it is about the lack of giving status on how their work is progressing and what blocking issues they have. And for leads it can be many different items around project status, team dynamics, etc. Regular communications is needed to shed light on certain topics. The situation of going dark happens very often because it's a natural tendency to be focused on doing the work that generates a lot of results. I usually find that leads who are very knowledgeable and have a lot of experience in running their projects are the ones that go dark most often. When things are going well and there are no issues to report, why report anything? The individual's justification and thinking is that they are too busy to communicate status and it wouldn't really help anyway.
That's where the misconception lies. How does one know what their manager or skip-level manager is thinking, discussing, and deciding? They usually don't. Potentially they are having conversation about the project you are working on, the progress that has been made on certain work, or other topics in order to determine items such as workload balancing, schedule slippages, etc. And they are making these key, impacting decisions based on the knowledge that they have. So although you may think your status isn't important, it may be very crucial in managers making better decisions.
The worst place for going dark is in a servicing organization. I ran a tools team a few years ago that delivered tools inside of Microsoft and then needed to support those tools. When internal customer requests come in, there needs to be regular communications back to the customer on the progress on their issue. Many times this didn't happen because the team was too busy, personalities were too introverted, and the overall thinking was that the customers should know that they will be working on the issue. But the customer was left hanging, not knowing that their issue was getting any attention. In this case, as in most cases, over-communicating was necessary to fix the mindset of going dark.
Whether the communication happens through emails, meetings, or other forms, it needs to occur. If your team is going dark too often, consider being more prescriptive on what the cadence of communications needs to be. Regular communications and not going dark is key to running a successful team and delivering results.
Filed under: planning, management, results, deliverables, software development, project management, cadence, status, engineering, going dark, communication
Since Windows7 released today, I think it is a good time to reflect back on my experience in the Windows Client team and working on the different versions of Windows. Vista and Windows7 were done much differently. When Vista shipped and we were making big changes to how we were going to manage the next project, there was an analogy floating around that I really liked and it helped the team move forward. Here is basically how it goes.
Vista was like American cooking. When you go to an American restaurant, you almost immediately order a drink, then get an appetizer, next your main dish, and finally dessert. If not planned out well, you may get nachos for an appetizer (Mexican), spaghetti for your meal (Italian), and then run out of room to even have a dessert. And how many times does the waiter/waitress come to the table to give you that one condiment you need or other items? It's not a very well planned out meal. But it is fun. Luckily, you aren't required to have dessert or have foods that go well together. And most of the time, you aren't in a hurry so it doesn't matter how many times you send the waitress away for more items. Creating Vista was fun too. But the project also had these characteristics similar to a typical dinner out in the U.S.
Windows7 is like Asian cooking. When you go out to some Asian restaurants, once you order, they bring all the ingredients that are required to make your meal to the table. Then they make it right there in front of you. It's all planned out, is done quickly, and ends up being exactly what you ordered. I was in the Windows7 team for the first 1.5 years of that project and a significant amount of that time was spent planning.
Finally today, the world gets to see the results of all that planning.
Ok, this analogy has made me hungry. Thanks for listening and go get yourself a copy of Windows7. I'm going to go get myself dinner.
15 years ago today was my first day at Microsoft. In the last 15 years, a lot has changed. Here is the way things were 15 years ago:
There was no IE. There was no Outlook. There was no concept of mobile or digital media.
There was no Office, just separate apps like Word, Excel, Powerpoint. There was no Visual Studio, just separate language apps.
There was no C#, .Net framework, or 64-bit anything – 32-bit was new and cool.
As for me, in the last 15 years:
I was an STE, SDET, and SDE while in the exact same role.
I’m the only test manager that ran the Visual Studio debugger team and the WinDBG team - the two most popular debuggers Microsoft has shipped.
I was on the Visual C++ team that created C# and managed the testing of the first Visual C# IDE (UI).
I was the person responsible for getting the first working build of the 64-bit compiler and handing it over to Windows to see how well it compiled Windows source.
I was the test manager responsible for the first N SKU Windows releases (the special ones for the EU).
The best ship-it party I attended was the free weekend trip to Coeur d’alene resort and Silverwood.
I had a test lead years ago that did a great job at setting expectations for his people. But for himself, his main goal was always stated as "do whatever it takes to ship the product". This was really difficult to measure him against. But the concept is an interesting one. Many teams fall somewhere in the spectrum between doing-whatever-it-takes and doing only what each individual wants to do (or product-focused vs. task-focused). I could call the latter selfishness, but is there really anything wrong with a person doing what they want to do? If it helps their career or keeps them motivated, then what's wrong with that attitude? Actually nothing is wrong with it as long as it is in moderation (and not always at the far end of that spectrum). There will always be a portion of the job that isn't a person's favorite, but should be accepted and completed so as to get to the more exciting interesting parts. But this isn't really just about being selfish; it's about trust within the team. Doing-whatever-it-takes-to-ship infers trust that the rest of the team will do their part. Nobody can do everything, but if a person needs to help out in a non-traditional area or task, why not? If everyone is doing their part, and the team has matured enough that there is trust, everyone should be thinking about how else to help out. Where on the spectrum does your team fall?
There are advantages in making sure people are identified by discipline (such as dev, test, PM, support, etc). Roles and responsibilities are clearly defined which include priorities, ownership and accountability, and clear growth paths. But the disadvantage is that it leaves the team inflexible. The discipline with the least resources or slowest ability to produce results now becomes the bottle neck for shipping a product. And instead of helping out, the rest of the team may blame or sit idle waiting for them to catch up because they are so task-focused.
Doing-whatever-it-takes allows the team to have agility and efficiency. It also builds pride in what the team is shipping. If the team is more focused on staying in their defined disciplines and focusing on their specific enjoyable tasks, then they must not have pride in their product, otherwise they would be doing whatever it takes to ship it. The downsides to this are risks around confusion of ownership, work duplication not managed closely, and some people who are overly enthusiastic about doing-whatever-it-takes may burnout.
My current team is still trying to figure this out, find the right balance, and decide where on the spectrum they fall. How about yours?
Filed under: test, management, commitments, project, results, deliverables, goals, quality assurance, selfish, task, roles and responsibilities, shipping products, software development
At Microsoft, we set commitments regularly for all employees. These are basically goals and focus areas that are written down to help employees remember what to work on and to clarify how their work is being measured. Here are some guidelines on verbosity that help when I review others' commitments:
- Some people get carried away with this and write these commitments as all-encompassing text, almost like an super job-description with absolutely everything that they do written down and measurable. It's a maintenance nightmare and very difficult to make sure the correct work is being focused on.
- Are they memorable? In 2 months from now assuming that they aren't looked at regularly, would the individual remember what they are supposed to be focusing on? Too many commitments and they won't remember any.
- Are they truly goals? Commitments about individuals attending meetings or giving status to their managers in many cases are fairly regular expectations (like brushing your teeth in the morning), so they don't need to be written down as commitments (unless the individual is one that forgets to brush their teeth in the morning). Too many items cloud the real work that needs to be focused on.
As a manager, I find it valuable to not do each employee's commitments in isolation. For the employee, it will be an individual experience, but as the manager, to make sure everyone in the same type of role and experience level has the same level of goals to aim for, some calibration needs to occur. I have both leads and individuals on my team. For my leads, I typically calibrate them myself, compare them to their counterparts in dev, or calibrate across other test orgs in the division. I find the best approach for the rest of my team members is to combine commitment text from all individuals in each job level into one document (one doc for level1s, a separate doc for level2s, etc) and review with my leads. I print-out these docs for my leads and we meet and read through these commitments. Sometimes this ends up being a lot of text so the calibration is more of a scan of the text to see what jumps out as different or unusual instead of reading through every detail. What comes from doing this is that:
- We can make sure that we aren't setting expectations too high or too low for people at the same level
- We usually find some commitments or measurements for success worded really well and pull those phrases into other commitments
- We find places where we should set stretch goals or that some individuals may be close to promotion
- Grows leads ability to see the different levels of expectations between experience levels of employees
- We create some standard commitments as a template that each person can then customize
For Test, the general high-level template would be: 1). Testing and shipping the product, 2). Writing automation and tools, 3). This third one usually is interchangeable depending on the team and individual focus but could be about Being the Customer or Driving Cross-group Collaboration. The 4th commitment is always a private one for the individual around areas of improvement. This may be a theme through all the other commitments, but it's spelled out clearly in this commitment specifically made to address areas that need improving. These are just my high-level descriptions, there would definitely be more text describing the plan on executing on these commitments and the ways to measure success of these commitments to make sure lead and individual are on the same page.
In many project teams, I see people mistaking actions for results. One of the best inputs a manager can give to a team is to evaluate when the team is too action-oriented and guide them to be more results-focused. If a project team is always really busy and working long hours but then they keep missing deadlines, the problem may be that they are too action-oriented. Being busy doesn't necessarily mean that they are busy doing the right work or doing it efficiently enough to get the right results on time. It's up to the manager to realign the team towards showing results.
When I came into a new team as a Test Manager, the team was very proud of a cross-team effort they had in place to do automation work. For the testers on this cross-team effort, the first priority was their testing roles within their own project teams. This general, broader automation work was secondary and in situations like this when it's not the first priority, focusing on results is even more critical. They were using ideas from the agile development process such as having monthly sprints and more frequent stand-up meetings. So I attended a monthly sprint meeting and I was shocked. Nobody was making any progress on these automation tasks. I saw goodness in their list of tasks that were all assigned to owners. But when the owners gave status, almost all of them hadn't made any progress on this work over the last month. When I asked what the months before looked like, they were the same. They were meeting and keeping status on these work items. Their tasks involved investigating and discussing issues, but not completing actual deliverables. They also weren't given the time to ever get to this secondary automation work. The other part of this cross-team work that they were proud of was that they rotated through sprint owners every month. This allowed everyone to have an opportunity to lead the cross-team effort and this didn't over-burden one person in managing this whole effort. But I believe this actually helped contribute to the action-oriented thinking. Nobody was accountable for results from this whole cross-team effort because the owner changed every month.
So I had to make changes to move this effort to results-driven. First, I assigned one owner for the whole cross-team automation effort, every month, always the same owner. That set up some accountability for showing results. Then, I went through the list of tasks, verified they were based on deliverables, and helped the new owner to prioritize them. I made sure that each person in the team could contribute time to do the work and if they couldn't, they weren't assigned a task, or that task was made inactive or lowered in priority. Finally, instead of tasks being tracked as either complete or not complete, tasks had completion percentages so that it was clear that some progress was being made. These are all simple changes, but together they can make a huge difference.
Another aspect to think through as a team changes to results-focused is on how the goals or tasks are being described. For example, some action-oriented goals would be:
-
Attend two training classes
-
Research the best model-based testing tool
-
Write two tools each month
Now here are the same goals written to be results-driven, paying attention to the deliverables:
-
Give a presentation or demo on two training classes you attended (turn attending a training class into a deliverable back to the team so they can learn as well)
-
Research the best model-based testing tool, give a demo on it, and incorporate it into the team's testing processes (researching can't be measured but a demo is a deliverable and incorporating it into a standard process will show continuous results during a project)
-
Deliver proposals to team or managers on gaps in testing or technology including solutions in the form of tools (writing two tools a month will leave a team with many unused, and potentially unneeded tools and a bunch of time wasted for the tester that wrote it - having the testers give proposals that can be properly evaluated will validate that the right tools are being created - and that may not be 2 a month!)
Overall, there are a few things to consider to move a team from action-oriented to results-driven. First, there needs to be accountability for the work. Secondly, goals need to be deliverable-based. Thirdly, progress needs to be monitored by the percentage of work that is being done and not just that a task is done or not. Finally, and this is an indirect results from being a manager, having the manager focused on the work being produced just naturally makes everyone on the team show more progress.
I heard the statement "Test is not a landfill!" today from one of my leads and thought it was a great way to describe what Test is striving for. Test should not be like a landfill. When I picture a landfill, I see a fenced in area that garbage gets dumped into. The garbage is the low-quality builds or untested code coming from the Development team. And the fence is the lack of communication on what's coming. Devs need to understand that unit tests are very important to run before checking in their code. And if they are going to check in something that will break BVTs, they need to communicate that beforehand.
As for unit tests, I find the best ways to get devs to increase their priority in writing and running these tests is to 1). Report on them and 2). Don't accept a build for testing without them being run. In the first case, widely communicating reports showing a lot of zeros for number of unit tests or code coverage of unit tests is a great motivator. Rejecting a build after devs potentially spent a lot of time coding but no time testing their code is also a good motivator. It's not meant as punishment, just as a firm statement that testers aren't going to waste their time testing a low quality build until devs run some tests to prove it at least has an adequate level of quality. If they run tests before every check-in, it is easier for them to debug a problem because that individual dev will know specifically what code was changed. Waiting until a build is produced and then having testers stumble into problems, costs more time because now testers have to debug through all the dev code (not just what was changed by one dev) and sometimes multiple testers have installed/deployed the build and are blocked. This is costly.
So a great statement for all test teams is that "Test is not a landfill"!
This is the gray part of testing. Should the goal be that BVTs pass 100%? On every build, all the time? The right answer is YES. But is that really the right answer for all cases. Well, of course not. I do believe it is the right answer for most cases. BVTs (build verification tests) are tests that are run right after a build compiles successfully (and links, and produces binaries). They should be test cases that cover the most common functionality or scenario that proves that the build can be tested further. BVTs should help prevent a team of testers from spending time installing a build only to find out that it does work well enough to do further testing.
For small projects, BVTs should pass 100%. There's really no reason why this goal can't be hit. Testers and developers on small projects should be communicating enough that feature changes should be reflected in the BVTs. The complexity of these projects, though potentially high, shouldn't be high enough to prevent the goal of a 100% pass rate. As the automation maturity of the team grows (and therefore the automation suite grows), the expectation of 100% BVT pass rate should even be more strictly enforced. If there are always certain tests that fail and cause the BVTs to not run at 100%, they should be disabled until they can be corrected. Otherwise, these continuously failing tests cause testers to re-investigate them more often than is necessary. A BVT suite is no place for unstable test automation. Also, BVTs should be automated. A manual BVT is almost a contradiction in terms since the goal of BVTs are to streamline the process and adding any manual intervention slows the process down.
So at what time is it ok to not pass at 100% (and still allow the test team to do further testing on a build)? I know of 3 situations: 1). the feature set is huge, 2). and related to that, the dependency chain is long and 3). The amount of test cases in the BVT is huge. For these situations, let's consider testing Windows. What would BVTs for testing Windows builds look like? Even running through a set of common scenarios will still make the BVT suite very large. When I owned running BVTs for the Client Windows Division, almost every day, there would be failures. But should we stop the release of a build to the Printer team just because the Media Player BVT tests failed? No. Of course, the real question is why did the Media Player tests fail? If the goal is 100% passing, then shouldn't it be as simple as fixing or disabling the tests for the next build to get to that goal? Well, yes, until the next build uncovers another set of failing tests. And the trend continues. This is because Media Player is at the top of the technology stack. The UI features are dependent on the underlying SDK, and the SDK features are dependent on the underlying codec, and that tech is dependent on the deep underpinning of the OS, and so on. So a bug (or even a deliberate change) could occur lower in the stack and then the change results in functionality being different in Media Player and the BVT tests fail. And is there really a solution to fix this? The first solution that comes to mind is that dependent teams need to communicate about their breaking changes. And that's a great solution when they know a change will break others dependent on them. But many times, these changes aren't expected to break. And communications on every change can be overwhelming to the other teams. So instead, you accept BVTs not passing at 100% and work to make the investigation of those tests as efficient as possible.
Finally let's consider a team like the Graphics team in Windows. They run tests on every configuration of graphics cards and drivers. This becomes a huge matrix. Then add the amount of BVTs that need to run on each of these configurations. We were producing over a million BVT results each day. Even when the BVTs passed 99%, there would be 10,000 test cases to investigate. That was impossible to accomplish, so we would watch the trends and only do investigations of failures when the pass rate dropped significantly. Otherwise, we would be stuck in BVT "analysis paralysis". We wouldn't be able to do anything else because we would spend all the time investigating BVT failures.
So you see, the simple question on whether BVTs should pass 100% has a lot of complex answers. That's the gray world of testing!
As I mentioned in a previous entry, it's review time at Microsoft. I already gave my guidelines on how employees should write their comments. Now it's time for my guidelines on how managers should write their evaluations of their employees. Many guidelines from my previous post apply here. Being a manager of managers, I find it very helpful to both me and my managers to review their comments for each of their employees. It helps them grow as managers as they continue to improve their ability to give clear feedback, and it helps me understand what the individuals in my team are doing. This works well with a team up to about 25 people. Larger than that and I find it too much to read.
Here are some specific guidelines for managers to consider:
1. Respond to employee comments. Depending on the review form employees fill out, sometimes there are employee comments in various sections. Managers should add comments in all those sections to address what the employee has written. If instead all the manager comments are gathered at the end, the employee may question if all their comments were read.
2. Although the goal is to be concise and not verbose, a manager needs to compare the amount of comments they are writing to what their employee wrote. For example, if the employee was very excited about their work in a certain area, they may write a few paragraphs on it. The manager shouldn't just write a statement or two saying that they agree...ok, that is part of my guidelines for employees writing their comments, but in reality I wouldn't expect managers to only write that! If the employee is excited about their work, then the manager should document their excitement as well (assuming this is a place for positive feedback). The employee will want to read about how well the manager thinks they did. Give examples from the manager viewpoint of some of the items that the employees accomplished. Sometimes this is a repeat of their evaluation, but instead it adds a different perspective, try to find examples that are different, or combine all their items into a more broad example of a skill they are exhibiting.
3. A manager's evaluation needs to set the correct tone for feedback to the employee and to reflect the overall compensation. This is a balance between positive and negative feedback. If overall the employee's performance was poor, more constructive feedback on areas of improvement needs to be in the evaluation. There should still be recognition of good work, but within the degree that they exhibited during the review period. If the employee's performance was stellar, then more positive feedback should be given. Some constructive feedback should also be given since most employees that are doing well still want to know where they can improve and grow.
4. Remove yourself from the evaluation. Employees don't want to read about the manager in their evaluations. Avoid starting sentences with the phrase "I know", such as "I know you delivered that automation on time". Of course managers should know that! Managers shouldn't need to prove what they know about an employee’s performance within the evaluation. Only use the phrase if the employee is afraid everyone overlooked some of the work they did.
5. State the positives, state the negatives, conclude with a summary of positive (toned appropriately for the employee). For examples, I always like seeing managers use motivational phrases in their comments such as 1). "Great job", 2). "Keep up the good work", and 3). "You are a valuable contributor to the team". For any of these statements, there are degrees that when set appropriately will set the tone. Don't say "great job" if really it was a solid performance, and not great. Instead say "good job" or "thank you for your solid contributions". Similarly, the third statement can be made overly positive with "the success of the team was solely based on your wonderful contributions".
6. Get peer feedback and use it. Many managers forget to get peer feedback, yet their employees spend most of their days doing work without the manager present. Getting a different perspective on how an employee is doing is a great resource for giving feedback or support the feedback that was already planned. In some cases, I will pull phrases from peer feedback and add them as comments with a section that just says, "Here’s what your peers are saying" and then put quotes around each phrase. I don't put enough text in there for the employee to determine who gave the feedback because it needs to stay anonymous.
7. Managers typically write strong direct feedback when it's positive. But when they have to say something negative, they add so many phrases and words that the exact message isn't clear. I ask the manager to tell me verbally what they want the message to be. Most people talk more concisely than they write. Once they state it verbally, they can then go back and rewrite the negative feedback more concisely. Along with this, after giving negative feedback, explain what success looks like so that they know what to aim for next time.
8. Don't use third person. Managers should be writing their comments directly to the employee who will be reading it. So use the term "you". Don't use the employees name so that it reads like the feedback is written about that employee to someone else. It's more personal to say "you". It's also harder to give direct negative feedback, but when done correct, the message is received a lot clearly when addressing the employee directly. Also consider adding their name at the beginning of the evaluation similar to addressing a letter to help direct the comments to them individually.
9. Use extremes. This may seem like an odd guideline, but reading comments that are neither strongly positive nor strong negative make it difficult to get the message across to the employee and have it stick. It's ok to use both positive and negative extremes in a review. It drives home what the manager thinks really went well and what the manager thinks really went poorly. This will impact the employee and they will remember this better through the next year as they work towards their next review. Be careful and don't exaggerate details or feedback. The use of extremes is only for the tone.
10. Use company terms. Many times, employees want to follow up on feedback from their managers. Many companies have some standard terms or documentation; point the employee to those as appropriate. Microsoft has competencies that employees are expected to exhibit. Instead of explaining the details around how to get better at focusing on the customer (for example), I can point employees to the competency on this and they can see what it looks like to exhibit this skill at different levels of experience.
Many engineers, especially testers, ask me how they can grow their careers and how I know when they should be promoted. I've learned over the years that there are really three key focus areas that show that an engineer is growing. These are in addition to the standard skill set that all engineers need around technical excellence, critical thinking, communications, attention to details, and drive for results.
The first is Anticipating Problems. A junior engineer/college hire will run into obstacles while accomplishing tasks. When they are blocked, they will usually go to their lead or more experienced coworkers and ask for help to get unblocked. As the engineer gains more experience, when they get blocked, they can come up with a few different solutions to get unblocked but are unsure which option to choose. When a tester goes to their lead with a multiple choice answer, that's a good sign that the engineer is growing in this area. Eventually, the engineer will approach the lead in a different way, not with multiple choices, but with one choice - the one they believe is the best option to getting unblocked. At this point, all the engineer needs is confirmation from the manager that this is a good solution. But growth in this area isn't complete yet. The next step is that the engineer will come to the lead stating what the blocking issue was, what solution they came up with, and that they have already implemented the solution and are no longer blocked. The engineer is confident enough to not need confirmation from the lead. Finally, the engineer has experienced getting blocked in a certain way so many times that they don't need to come up with solutions to be unblocked. Instead, before the problem even presents itself, the engineer is working through ways to prevent it and the blocking issue never occurs.
The second focus area is Long-term planning. A junior engineer or new hire will typically be focused on the tasks for the current day. Their preparation and planning is to make sure that work gets done within that day. With more experience, they can plan tasks for a longer horizon such as a week or month, and eventually multiple months. This focus area is very much related to the next one about scope because as the engineer works with a longer horizon in mind, they can impact whole project teams. Since projects get done within months or years, as the engineer's long-term planning is within that same period, they can plan out their work and potentially other engineers' work for that whole time. Managers need to be planning out not just for the next product cycle but the one after that. That is why many higher-level managers do 3-year plans. They have mastered this skill more than most.
The third focus area is Scope. A junior engineer typically works within the scope of their day-to-day tasks which for testing may involve finding bugs, writing test cases, and automating. Their scope is focused on one person, themselves, and the testing tasks they have to complete for their features.. As they grow, the size of the group that they impact grows as they do their work. Usually this grows to the feature team which may also include their devs. They review dev specs and do more code reviews, and potentially as they write automation, they think about how others could use it and make the code more modular and ready for reuse. As the engineer matures and the scope grows, they impact a whole project or product team. Their opinions are sought out. Finally, as they do their work or develop new opinions, they communicate these broadly beyond their project or product team to multiple product teams or a whole division. The true final step to have the widest possible scope in a role is to impact the company you work for. Each step along the way to broadening scope involves growing confidence, knowledge/experience, and communications skills.
It's that time of year again at Microsoft where we get to write our reviews. I find it helpful as a manager to go over some guidelines with my team as a refresher on how to write good reviews. These are my opinions from what I've learned after doing reviews for the last 15 years. Some of these guidelines may not work for everyone. Also, there are different guidelines for managers writing evaluation comments and feedback for their reports. I'll get to that in a future post.
- The first thing people need to understand is that review writing is an important task and skill. Don't short-change it. It's the main method used to record all the work that was done in the last year. If not enough time and thought goes into it and it is written poorly, it becomes a bad representation of the work that was done, even if the work was spectacular.
- Who is the audience? Let's face it. The audience of a completed review document is not the individual writing it or the manager who adds their evaluation comments. It's that future hiring manager or future boss (within the same company) that wants to know more about an individual as that individual moves through product groups on the journey of their career. Given this, there are some items to consider which will make the comments more clear to this audience:
- Don't overuse acronyms - the individual and their manager may understand it, but the manager a few years from now won't know what it means and the text will lose it's usefulness.
- Don't use project names without first describing them.
- Don't infer or indirectly state something. Describe it so new readers will understand.
Here's an example: Before a reorg was announced, I needed to decide what team members I should move into new roles testing databases. I hadn't worked on the team long enough to know the backgrounds of everyone so I decided to look at their past reviews. I found that this didn't help me at all. All I stumbled into was acronyms, project names, and cursory inferences to situations I wasn't a part of.
- Be honest and objective. The goal should be to write comments that are an accurate evaluation of work that was done, both the good and the bad. Along with this, don't overstate your accomplishments or take credit for something you didn't do. When the manager writes comments, they should theoretically only need to write "I agree" because the individual's evaluation comments are objective, accurate, and complete.
- Related to the last point, comments should be balanced. This is a time to brag about your accomplishments and a time to critique yourself. Many people only state all the good things that were done. What happens in this case is that the manager then needs to also mention all the areas that need improvement. More time will be spent on these improvements in the manager comments which then makes the review become unbalanced. The individual may have met expectations, but since the manager needs to spend enough time in the text stating the areas of improvement, the tone then becomes negative. The same is true in reverse. I've read many reviews where individuals get down on themselves for not being perfect or not accomplishing everything. Then the manager's comments need to state all the great things that were done and again, the review's tone becomes unbalanced. This also comes across as the manager and individual not being on the same page. So aim for writing comments so that all your boss would need to write is "I agree".
- Don't just state "what" was done, but "how" it was done with examples and details. This is important. Getting a task completed but upsetting people along the way is not good. Managers should not have to do damage control for mismanaged tasks. When a task is completed, how it was completed is sometimes as important as just completing it.
Here's an example: I've managed tools team and once had someone that wrote a tool and got broad acceptance for it - many people in other teams started using it. I found out they were using it because they were bullied into it, and weren't given a choice. In another case, the tools developer did surveys to find out what his potential customers (people in other groups) wanted before writing the tool. In the end, both people had the same result, a widely used tool. But how the second person got to that end result was much more appropriate and productive.
- Polish the written text by confirming that strong action words are used. And get sentences in the right tense, past or present. Avoid using "have", "has", and "been". And finally, don't use bullet items, but write full paragraphs while trying to be concise.
If writing review comments is difficult, consider getting into the habit of weekly status reports or saving important emails into a separate folder throughout the year. Then, at review time, go back through those status reports or emails to remember the details of what was done over the past year.
The last few months have been a staffing challenge for me and my team. I lost a quarter of my team in a 3 month time period due to reorgs….and I couldn't replace any of them. Due to the economic climate, things are tight and that includes headcount. Then, my team was affected by lay-offs and I lost some more people. So at this point, as a test manager, I think the expectation is that I would be frustrated and screaming for more people. But what good would that do? It is what it is. It's a serious situation and it needs my focus, but being the squeaky wheel right now given the economy will not be beneficial. What I need to be concerned with is the amount of work and project scope within my team. With the help of my very-capable counterparts, we continue to scope project down, lengthen deadlines, communicate expectations to our business stakeholders, and review our budget numbers. The most important management goal I have right now is to make sure that my team members aren't killing themselves to get all the work done that was left from the multiple vacancies on the team, and to make sure necessary work isn't falling off and not getting done (it's instead getting prioritized correctly).
So here are some ideas to help in situations like this:
-
Creative staffing:
- Contractors/vendors - beyond just hiring them, have them do more overtime especially on weekends to give full-time employees a break
-
Offshore employees can allow the business to run day and night and shouldn't cost too much
-
Internships is a great way to bring people in with a fresh perspective and a lot of energy and cost less than fulltime employees
-
Job sharing allows a manager to have two team members for the price of one and even though they both will do half the work, the manager gets two different perspectives and there's less likelihood for burnout
-
Clearly and visibly communicate changing expectations and deliverables
-
On reports (status reports or quality metrics), show some red - raise some flags around risks and tradeoffs being made. This will allow upper management to agree with the decisions that were made to deal with the staffing issues.
-
Review processes on the team and dig into the details. This will help uncover inefficiencies. Gaining efficiencies will allow the team to do more with less people.
-
Specifically for test teams: automate, automate, automate!
-
And finally, be patient and think strategically. At some point, the economy will get better and managers need to have a plan on where to add resources and what work to put back on the schedule when that happens.
Every manager would love to have more people and get more work done. And some managers use that desire and their team situation to become the victim and create excuses for not getting work done. Or they continue to complain about being understaffed. What I've learned is that being understaffed is a perpetual state that every team is in and it is the responsibility of the manager to work through it and accepted it. That is why I question if the state of being understaffed really exists? Or isn't that just the norm? As I continue to change the amount and the way we work to accommodate team size, then we are never truly understaffed. If managers let this block them, they do a disservice to the team members that are there ready to do great work!
As a manager in an organization, there are different directions to apply focus and communications - downward, upward, and sideways. Test Managers who focus downward spend most of the time dealing with their team, with testing issues within the team, and generally push results through their team. If this is the only direction that they focus on, then they will do well when it comes to manager feedback and organizational health ratings. But what is missed is overall efficiencies, personal visibility, and opportunities with large scope for their teams.
The sideways direction for a Test Manager is defined as peers such as other Test Managers, Dev Managers, etc. It depends on the organizational structure, although managers shouldn't limit themselves to peers that are specifically at the same level in an organization since some orgs have multiple middle levels. TMs need to build solid relationships with these peers in order to gain efficiencies. For example, after I moved over to be a new Test Manager in a team, I sought out the other two Test Managers in the division and met with them to start building a relationship. They both basically told me to buzz-off because our teams didn't have any commonality. They were missing the point that this was the exact reason why I wanted to meet with them - there's always commonality and once found, efficiencies can be gained. In this case, my team was temporarily isolated. Other test teams were too far removed to make any significant gains by partnering with them. I found that my team had a lot to work on! Test teams don't just test their features, they write automation which involves significant deliverables and due dates. Test teams also need to collect measurements on the quality, create standards and best practices around engineering processes on the teams, set up lab environments to run tests, and interpret and incorporate customer feedback. This is a lot of work to do on one team. But if test managers partner with other test teams (and other Test Managers) they can end up sharing automation tools, finding better engineering processes, and plugging into measurement systems that already exist, all of which can gain a test team a ton of efficiencies. After a re-org, my team moved away from isolation and into a team with welcoming Test Managers and I was finally able to start gaining these efficiencies. The other benefit from focusing sideways is personal career growth by the growing scope of impact and influence. As I built solid relationships with those welcoming Test Managers, the tools and processes that my team developed started getting used by those teams and as a result, my team and I had a larger impact on the division as a whole.
Focusing upward is the final direction and that is defined as the bosses of a Test Manager (their management chain). I've seen Test Managers whose teams are miserable, they don't have clear direction and don't meet deliverable deadlines. But the TM just continued to get pet projects and recognized at meetings, and his career grew quickly. This was because he was focused upward. His top priority was to impress his boss and to do items that were the most visible and the highest priority for his boss and management chain. In this case, he focused too much in one direction and not in the others. For many people, it's more difficult to focus on this direction because the challenges and problems that need solved are potentially beyond a test manager's comfort zone. To some degree, managers focusing upward will have to turn on a bit of the sales pitch, self-promotions, and boldness. Managers that don't naturally focus upwards should start with small projects and tasks.
So which direction should a Test Manager focus on? All of them, of course. Typically, one direction gets the most focus and that is unintentional. But if the manager can see what direction gets the most focus and increase focus in the other directions, they will be a more well-round test manager. I find it natural to focus downward - and easier for most. So here are some ideas on how to focus upward:
Skip-level 1:1s up management chain with prepared agendas
Emails to VPs and higher ups when opportunities arise
Get onto newsletters with items about test team initiatives
Focus on items or issues that gain you some added visibility
Now for those managers who naturally focus upward, but maybe not downward, here are some ideas:
Walk the hallways and start getting to know individual testers
Find reasons to stop by testers' offices to talk (preferably about work)
Hold regular meetings with their team
Ask the leads for invitations to meetings where individual tester attend
Do regular skip-level 1:1s with individual testers
Ask them questions about what they are working on and allow them to give feedback and input without any ramifications - potentially in the form of a post-mortem exercise
I have found that keeping a balanced focus in these 3 directions works well!
