Summary
Motley: A tester's job is to find bugs, so measure them on the amount of bugs they find. More test automation is always better.
Maven: Do not measure testers by the amount of bugs they report. Think of the test team as more a quality assurance team than a quality control team. Too much test automation can be a problem due to brittle test infrastructures, too much time troubleshooting tests, suboptimal investment in tests, and lack of thinking like a user.
______________________________
[Context: Motley has been having issues with another counterpart team in the company]
Motley: I am a little disappointed in our test team. They really haven't generated many bugs in the product as of late. I feel like I have to kick them in the pants to get anything done.
Maven: I always thought we had a great test team. We have a bunch of really smart people, and they have made previous releases great. What is the problem?
Motley: Lately all the bugs are generated either by the development team or by others using the product. We pay the test team to test the product and find issues. If I was the test manager who is measuring them by the number of bugs they find, they would all fail this year's annual review.
Maven: Whoa. Before we explore the real issues of why the test team is not reporting bugs, let's talk about your statement around measuring testers.
Motley: Why? The best way to measure a tester is by the number of bugs they find. You can mine the bug database easily to get the numbers and it makes evaluation of the test team really easy.
Maven: Sorry, bud. Measuring testers by the number of bugs they find is a really dumb thing to do.
Motley: Don't EVER call me "dumb" - <pow>.
Maven: Ouch! Fine. I deserved that, and I apologize. Anyway, just because a tester is not reporting bugs does not mean they are not being productive. Think of the test role less as "quality control" and more as "quality assurance". The best tester can prevent a bug before it happens.
Motley: But they do control quality, and how would they prevent a bug before it happens? Developers hand over the code when they are ready expecting the tester to find all the issues.
Maven: Maybe in the past. Now, particularly with agile teams, testers are involved in the process right from the beginning. They participate in functional specification reviews, contribute to user scenario development, design reviews, code reviews, do private buddy testing, help compute metrics like code coverage, and many other quality-related activities. They do not have a sole test responsibility - they are another member of the team. A good tester helps assure quality by finding issues in all aspects of the development process and helps to improve the team's processes to prevent bugs. The best testers find no bugs in the code because they worked to prevent them right from the beginning. As a result, it makes no sense whatsoever to measure a tester by the amount of bugs they report in the bug database.
Motley: I guess that has some merit. The best tester may actually have a lower bug count than the poor testers. That is kind of an odd way of looking at the performance of a tester, though.
Maven: Odd, but reasonable. The test team should still be testing end-to-end scenarios in the product and reporting test results, however. You are saying that isn't happening? Why?
Motley: I talked to Morgan, the test lead of the multimedia feature. They are focusing all of their time on improving the test automation to ensure we have full coverage. However, the test infrastructure they are using is very unreliable and often reports incorrect results. Another team in the company is responsible for that infrastructure, but because our tests are a little more complicated, Morgan's team ends up troubleshooting the issues. It sucks up so much time that they don't have time to do the tasks that matter, like attend reviews and actually test the product.
Maven: A common problem when teams focus too much on automation.
Motley: What do you mean? Test automation is a good thing. It allows us to re-run tests without human intervention on a regular basis. Without test automation we risk breaking many areas of the product every time we make a change.
Maven: Do not get me wrong - automation can be a great thing. Unit testing is a form of automated testing and something that we have been discussing that absolutely every software developer must be doing. You make a change, you run your tests, and perhaps tests from other teams, to ensure you have not broken any functionality. The more unit tests, the better. The higher the code coverage on those unit tests, the better.
Motley: You are contradicting yourself all over the place Mr. Maven. First you said you can focus too much on automation, and now you are saying more is better. Inquiring minds want to know - which is better?
Maven: More unit tests - better. More scenario-based automated tests - not necessarily better. There is a great value in having a basic test suite that tests the core user scenarios of the product. However, relying too much on automation at the scenario level has disadvantages:
- If the architecture of the test infrastructure is very brittle (i.e. not robust), every time the product changes, major changes in the automated tests may be required. Testers spend more time trying to get the automation to work than adding real value to the product.
- A large focus on automation perpetually puts the test team behind the development team, if that is how your team is organized. If the developers design for testability, as they should, this situation is less of a problem as the testers can start developing automation fairly quickly. If automation is an afterthought, the developers may be on to the next feature before the test team is done, which means the feature isn't "done" according to our definition. It is also a context switch to go back and diagnose and debug issues later as the code is not as fresh.
- If the infrastructure on which the current features are built is expected to change in the next release, often a rewrite of the test automation is required. Should the team spend a large amount of time building automation infrastructure if it will change in the near future? Tough call.
- An automated test cannot think like a user, even if you inject some randomness into the tests. Over-reliance on automation creates some nasty test holes in the product that may only get discovered after shipping.
Motley: And here I thought more automation all the way around is always better. You actually make some good points. I am going to have a chat with the test people and ensure we are focusing on the right things. We should give up some work on automation if it means that we ship the product sooner and we do not cause ourselves any long term harm.
Maven: Sounds great. Let me know how you make out. I'm now going to stick some tissue up my nose to control the bleeding thanks to you.
Motley: Ah, you know you deserved it.
______________________________
Maven's Pointer: At Microsoft, I (James) have worked with a test team that seems to spend more time diagnosing and troubleshooting organization-wide automation infrastructure than it does improving processes, working closely with developers on test plans, and manually testing the product as a user would. Automation is a tool to detect regressions earlier in development, but it should not be used as a crutch. Given that philosophy, limit the amount of time you spend on automation to a reasonable number and pay attention to diminishing returns.
Maven's Double Pointer Indirection: We talked about code coverage in the past. One mistake test teams make is that they demand that they own the code coverage numbers, and that coverage should be measured against the test team's automated tests. Scenario-based test automation is not about testing individual components, modules, methods, and lines of code - it is more about ensuring that a scenario works as it should. Leave code coverage metrics to the developers and measure it with unit tests.
Maven's Resources:
- I.M. Testy's blog, from a guy in the Microsoft Engineering Excellence who has been teaching about testing for a long time.
- Alan Page's Notes and Rants, from another guy in Engineering Excellence who really knows his test material
- How We Test Software at Microsoft, by Page, Johnston and Rollison, Microsoft Press, ISBN: 9780735624252, August 2008.
Motley: James, that slacker, is not in the office again. That guy spends so much time outside of work that it's a wonder we pay him.
Maven: Well, everyone is entitled to a bit of vacation now and then, don't you think?
Motley: If that's what you want to call it.
Maven: Actually, this time around he's off doing a backpacking trip for charity. James is doing a 4 day hike to support Washington's National Park Fund (WNPF). It's a really good cause. We all do lots of hiking and camping in the backcountry in our national parks and generally take trail maintenance, repair, rangers, etc. for granted. Money to support that stuff needs to come from somewhere. James is raising money for the WNPF to help out. Care to give?
Motley: Ummm, I gave at the office.
Maven: Well, if everyone said that then nothing would get done. James will even take care of having Microsoft match whatever funds you donate. In case you change your mind, you can donate at http://www.firstgiving.com/waletzky.
Motley: Ugh. Fine. I can probably part with a few morning lattes. I'll have a donation up there within the hour.
___________________________________
James: No blog entry this week due to a charity event. Back next time. In the meantime, if you feel like donating to the WNPF, please click the link above. Thanks!
Summary
Motley: Use Scrum in my personal relationships? Don't be such a geek.
Maven: You can apply the same lessons as Scrum teaches to your personal relationships - lots of communication, planning, iteration, and retrospectives focusing on continuous improvement. Just translate the really hard stuff (relationships) to the what we geeks understand.
______________________________
[Context: It's Monday. Motley is not a very happy developer after spending the weekend with his girlfriend]
Maven: Hey, Mot. You look a little depressed today. Did you not have a good weekend?
Motley: Personal stuff. Leave me alone.
Maven: Ah, c'mon. Maybe I can help.
Motley: Dude - you write software for a living. You have no personal life. You don't know the first thing about relationships. Ah, crap. I already gave too much away.
Maven: Girl troubles. I see. What's the problem?
Motley: Ok, I'll humor you. I may as well tell someone my problems. It helps to get them off your chest. It's pathetic that I have to resort to you, however. Anyway, I have a great relationship with my girlfriend - usually. We are going through a rough patch. We haven't been communicating as much as we should, expectations are not being set appropriately, and I feel like we keep digressing to old behavior.
Maven: Typical problems in a more mature relationship. I remember you saying you live with her, so I presume it's mature. How about starting with some examples? What's the communication issue?
Motley: Wait a second. You see James listening again hoping for some blog material? Hah. Not getting anything this time. For once we are having a conversation about a topic that has nothing to do with software engineering! Sucker.
Maven: Knowing him he'll turn this into a software engineering topic somehow.
Motley: Anyway, one of the main issues in my relationship centers around chores. She has some responsibilities and so do I. The problem is that she does a poor job at the chores and I have to clean up and finish. It is frustrating.
Maven: It's your fault.
Motley: What do you mean it's my fault?!? She does the chores!
Maven: Have you sat down and communicated expectations you have for the chores? The two of you have not specified the completion criteria for the chores, so there is a misunderstanding of what needs to be done.
Motley: That sounds so robotic, dude. But, fine. I guess if we don't talk about expectations, that leads to issues. I'll have a chat with her about that.
Maven: What are the other problems?
Motley: Still related to chores, when she starts on something, it takes forever to finish it. She just bites off way too much at one time. And by the time it's done, it's too late to be useful. Yeah, yeah - you want an example. Well, she started working on the yard in mid-summer - bought all the plants, and by the time the plants were all in the ground it was fall and they died off shortly thereafter. Definitely not the way I would have done it.
Maven: Have you encouraged her to split the larger tasks in smaller pieces? Instead of doing one big task she could do a bunch of smaller tasks and adjust as needed. If the yard is taking much longer than expected, she could adjust what plants to put in the ground to be more seasonal.
Motley: Probably not a bad suggestion. I'll bring it up. While I am in a whine mode, another thing I am not happy about is that many chores get overlooked. Taking out the garbage is a regular thing - if you don't do it, you miss garbage day and have to put up with a nasty odor for a week. Don't you just love the smell of two week old rotten vegetables?
Maven: Perhaps what you two should do is have a brief discussion at the start of the week to come up with a shared to-do list of tasks you need to accomplish. Think of it as a family meeting or something.
Motley: Have you ever had a relationship? Never mind - I withdraw the question. Your suggestion is way too formal! She will likely laugh in my face.
Maven: It doesn't need to be formal, and it goes back to setting expectations. If you clarify the plan for the week, you will save yourself some stress and headaches later. Write it down so there is no confusion, and then you get the pleasure of checking off the task when it is complete, as we talked about previously in our checklists conversation.
Motley: I am not going to bring it up as a "family meeting", but the concept is reasonable.
Maven: If you do a daily check-in as well, you ensure that things are kept on track for the week. Just bring it up in everyday conversation.
Motley: I am afraid that if we take these suggestions that we will be back to our same old selves at the end of the first week.
Maven: Change is hard. We have talked about that too. When you make up your plan for chores for the next week, think back about the previous week and talk about how you can make the next week better. Scrap the to-do list if it isn't working. Make more time to talk if that's what it takes. Additionally, have a chat about what went well, and continue to do those things.
Motley: I don't know if I can sell all of this to her. I am obviously going to have to rephrase everything that you have just said in a way that she can understand, and not sound so ridiculously formal. All of this thinking sounds familiar though, but I am having a hard time placing it.
Maven: At this point you likely think I am completely off my rocker (crazy), but in a previous relationship my girlfriend and I were seeing a counselor and he recommended everything above. I thought it sounded very familiar to the work environment, and then it hit me like a tire on the side of the head - it's Scrum! Who knew that you could apply Scrum concepts to your personal life. I didn't.
Motley: Scrum. Yeah, I thought that sounded familiar. You just ran over planning, the meaning of "done", doing work iteratively, the daily check-in meeting, updating your backlog, and retrospectives. You are such a geek. You did say "previous relationship" with your girlfriend, so I am assuming this didn't work. Why bother?
Maven: Don't be a wise guy. Apply all the stuff we just talked about to both your relationships and to software engineering and you will be fine. Just remember to apply some right brain thinking when you use it with your girlfriend.
______________________________
Maven's Pointer: Building and maintaining relationships is often hard for techno-geek logic-based beings like software developers. It is amazing how the metaphors that we apply in everyday software development also apply to life in general. The counselor in the story above had no idea whatsoever what "Scrum" was, but he presented all the concepts in his language. Many concepts that we leverage as developers are applicable to other domains and disciplines. You just have to look for it.
Maven's Resources:
- Yeah, right. This topic is pretty far out there. If you find any related resources, be sure and let me know!
Summary
Motley: Features sell a product. When in doubt, add more features!
Maven: These days, software is less about features and more about reliability, fit 'n finish, performance, and usability. Use the Kano model to help you focus on the right scenarios for the user.
______________________________
[Context: Motley is struggling with a common project management problem]
Motley: I am wondering how we are ever going to make any money on this product. We had to make so many cuts to make it to market in a reasonable time that the product is not all that useful, in my opinion. We need more features, and we need them quickly.
Maven: Do you think it's just features that make a product successful and sell lots of copies?
Motley: Well, pretty much. If the software doesn't do what you need it to do, why would you bother buying it? Features trump everything else. For a v1, we should even consider shipping sooner with a few more bugs to get a few more features in.
Maven: Hmmm… you were telling me the other day that you have an iPod Touch, right?
Motley: Why are you wasting my time here? Yes, I have an iPod Touch. Get to the point.
Maven: One more question: why do you have an iPod Touch vs. a Creative Labs Player vs. a Zune? The iPod costs significantly more doesn't it? And don't the other players typically have more memory for the same money, and perhaps include FM tuners?
Motley: That was three questions, wise guy. I chose the iPod Touch after playing with it for a while. It does what I need it to do, it's reliable, and it's extremely well refined and polished. I could flick the list views and pan around for hours. I love watching the acceleration of the list view as you start to flick and slow down as "friction" takes over. They really did a nice job with the interface. I had to have it. It's a really fun device to use.
Maven: So it's more than just features that attracted you to that particular model?
Motley: Well, in this case, yes. I looked at the other MP3 players out there, but I wasn't impressed. They had the same features, but the iPod Touch was just, well, fun. Additionally, one of the other players that shall go unnamed crashed within just a few minutes of use. I was not impressed.
Maven: Does the iPod Touch do everything you want it to do?
Motley: Not necessarily. It would be great if it had a built in GPS, for example, so I could hop on a wi-fi network and get directions from where I am. An FM tuner would be nice for those nights I go to the races (they broadcast the commentary over FM). But seriously, I have to get some work done. Why are you bothering me about all this?
Maven: Let's go back to the software you are working on - you were saying we need more features and should even ship with slightly lower quality to get features in. Does that equate back to your iPod Touch experience?
Motley: Well, errrrr, ummmm…
Maven: That's what I thought. A decade ago, when there was little competition for many software products on the market, features won the war. End users actually put up with an operating system like Windows 95 that blue screened every other day when it was released. It was the best thing out there and the reliability issues were tolerated. Times have changed. Users expect software to "just work". They want it to solve their problems their way with a minimum of fuss. If it meets the majority of their scenarios and does it well, users are generally happy. These days there are always other options if a product does not meet their needs.
Motley: You did mention "their way" above, though. In order to do things their way, you need to make it as configurable as possible and including knobs and buttons allowing them to customize their experience. That takes time to build.
Maven: I would actually argue that too many knobs to twist, turn and pull just ends up confusing the user creating a frustrating experience. Look at the iPod Touch - two buttons. Compare the browser on the iPod Touch to something like Internet Explorer on the desktop - very few configurability options. But, users are happy. In fact, they are ecstatic about the device. It satisfies 80% of the core user scenarios. It may not keep the power users happy, but there are other options out there, including third party software updates that could potentially add some of the knobs and buttons they need.
Motley: There is still a minimum feature set we would need to be successful.
Maven: No argument there. However, that feature set is usually not as large you think it is. Nail the top priority user scenarios extremely well and you will likely be successful.
Motley: Easier said than done.
Maven: Sometimes. You need to really understand the user in order to make good decisions. Here are a few other keys to success:
- Reliability: The user's first experience with the product should be positive. It should "just work". It should be working out-of-the-box within a few minutes. Spend more time during development ensuring it installs easily and doesn't crash vs. adding features. What features are there must be solid.
- Fit 'n Finish: Were the gravity and friction capabilities on the iPod Touch really necessary for a decent user experience? No. Do they set the device apart from competitors? Yes. Do they make the device fun to use? Absolutely. Do they set the bar for interfaces on mobile devices? You bet. Nice icons, nice graphics, and little effects add up to a really positive experience with the device when combined with the other criteria listed here.
- Performance: Waiting sucks. Apple put the right hardware in the iPod Touch to ensure performance would not suffer. I am sure they spent a lot of time tuning the experience to make it snappy. Users will no longer accept sluggish software. Performance actually is a feature.
- Usability: The user interface must be incredibly intuitive such that you do not even need a user's manual. The user should be able to pick up the device and be productive within minutes. An understanding of human psychology is a requirement when developing a good interface.
As another example, how many features of Microsoft Word do you use?
Motley: Out of the 10000? Probably 20.
Maven: Exactly. A few extremely polished features will likely satisfy 80% of your users (see the Pareto Principle, or 80-20 rule). Spend time making an incredible experience for those core features, and add features once the polish is in place. Aim to delight the user.
Motley: Delight the user? That sounds a little "foo-foo" to me. What is "delight the user" supposed to mean?
Maven: Think about the Kano model, which comes from Six Sigma. A diagram illustrating the concept is shown below:
Pasted from <http://www.isixsigma.com/library/content/c030630a.asp>
The lower curve represents the basic needs of the user that must be present in the product for it to stand a chance to be successful. With the iPod Touch example, it better play MP3 files. That is a basic need that the device must fulfill, and if it didn't, it wouldn't sell, at least not as an MP3 player. These features likely won't set you apart from the competition either.
The middle curve represents those features or criteria that you can never have enough of. With the iPod Touch example, more memory, faster performance and better sound quality are always welcomed and can set the product apart from the competition.
The upper curve provides the greatest opportunity to beat competitors, and are those things that truly "delight" the user. With the iPod Touch example, the polished user interface, the flick and pan operations, the automatic playlists, etc. help set it apart from other players and make customers say: "Wow, this is a cool device!" Often these types of "features" are unexpected, but very welcome.
Motley: Interesting model. You are saying that we must met the basic needs, consider performance characteristics, but to really set ourselves apart, we need to consider those things that delight the user. This typically revolves around ideas that really catch the user's attention, and are often the result of polishing existing user experience paradigms and features.
Maven: Exactly. Although polish does not always equate to delighters, it does contribute to a solid user experience - moreso than the addition of several more half-baked features.
______________________________
Maven's Pointer: Although I (James) work for Microsoft, I will gladly give another company kudos for a job well done if it is warranted. I do own an iPod Touch and think it's a fabulous device that other MP3 players cannot keep up with. I have been extremely happy with my purchase, even though I could have gotten a significant discount (relative to other players) on one unnamed player. "It just works" really matters to me - the device and/or software needs to be polished, usable, reliable, performant (is that really a word?), and just do what I need it to do. That's not to say Microsoft doesn't make some fabulous products as well, such as OneNote, Money and Vista (yes, I am happy with Vista). Apple does a set a great example, however, when it comes to building software and devices that consumers love.
Maven's Double Pointer Indirection: There is more to Kano analysis and mentioned here. There is guidance available in some of the resources below on how to ask user positive and negative questions and plotting them on a 2-D table, to determine whether ideas are delighters, performers, or basics.
Maven's Resources:
Summary
Motley: Milestones are useless for agile development. Our feature team can ship at the end of every iteration, so milestones have no value for us.
Maven: Milestones provide a synchronization point across a set of features, helping to ensure the overall product is of high quality. Every milestone has a set of exit criteria that the integrated product must satisfy before moving on.
______________________________
[Context: Motley is analyzing his team's upcoming deliverables and is questioning some of the Cynthesis Software development process]
Motley: Overhead, overhead, overhead. Why is the project manager making our team go through all the hassle of hitting a milestone when we are doing iterative development anyway? We have mini-milestones every time we finish an iteration. We produce a shippable piece of functionality that is ready to go. Milestones are useless for us.
Maven: Actually, milestones have use even in an agile world. You are not the only team contributing functionality to the release, right? There are other teams building features?
Motley: Yeah, so? Some of the other teams are doing agile as well so they likely have the same questions as we do. For the others not doing agile, well, they need to wise up and get with the program.
Maven: It's not so easy to change a company, as we discussed in previous conversations on change management. At some point you need to bring all the feature teams together and make sure everyone is synchronized. That's one of the primary purposes of a milestone. It is a checkpoint that all teams have in common with a set of exit criteria. A milestone brings everyone together to ensure all teams can integrate their features and come together to create a product. Milestones may not be technically necessary if there is one small team developing a release. You can ship at the end of every iteration, at least in theory. However, when many teams are involved in creating a large product, you need to periodically bring everyone together.
Motley: Milestones just seem too process-heavy. Why not just sync-up on an as-needed basis?
Maven: Don't forget - milestones also have use to people outside the company.
Motley: Why should anyone outside our development group care about our internal milestones? Perhaps your girlfriend needs to know so she can expect you to work some overtime around the end of a milestone, but other than that, I can't think of any reason.
Maven: With any significantly-sized product, Cynthesis typically ships technical previews, does integrated product demos for customers, and ships beta releases. Typically these interim releases prior to ship align with milestones to ensure product quality is high. Because each small feature team may have their own iteration schedule with their sprints, the milestones provide that date where sprints can align.
Motley: Fine. I can accept all that. It still seems process heavy though. You mentioned the phrase "exit criteria". That phrase has "formal process" stamped all over it. Pepto Bismol is my exit criteria after eating bad seafood.
Maven: Ah, dude, that really wasn't necessary.
Motley: I know, but potty humor is always fun anyway <laugh>.
Maven: Back to business. Yes, "exit criteria" is one of the more formal-sounding software engineering phrases, but it does have use. At the end of any given milestone there is typically a checklist of quality criteria that each team and the overall product has to satisfy before the milestone is considered done. Think of it as a team-wide definition of that ever-so-important definition of "done".
Motley: Ah, so it's a fancy phrase meaning that once we put all the pieces together that they all work well together. That would include overall performance, end-to-end integrated scenario testing, stress testing over time, lack of cross-product memory leaks, general usability and consistency across features, consistency of documentation, how it works with the rest of the system, and other general product health issues.
Maven: Exactly! We put all the pieces of the puzzle together and hopefully it produces a beautiful picture. Without that checkpoint you could ship a product with puzzle pieces that don't match creating a jumbled mess.
Motley: I see. Makes sense. One thing that I would encourage the project managers to do, however, is not make those exit criteria too strict and that we focus on the right things.
Maven: That's a very astute observation, Mot. I worked in one company where the exit criteria list was pages and pages long. After about the first third in priority order, the rest just created needless overhead and diminishing returns. It felt like the project managers were micromanaging the feature teams. Instead of speeding up the development process, it slowed everybody down, in my opinion.
Motley: Ouch. I guess we should consider ourselves lucky here at Cynthesis. I can put up with a reasonable list of exit criteria, but you don't want to know what I do when someone tries to micromanage me.
Maven: Imagine nag nails for every little thing like high priority bugs you haven't updated in the bug database for a few hours, or nag mails because the project manager thinks you haven't triaged your bugs but your process is different than other teams, or people screaming at you because bugs are not assigned to a specific individual person and are instead on a bug backlog. I'll stop ranting now, but just know that these things can be a reality even in great software companies.
Motley: Enough! Enough already! You are going to make me start pulling my hair out just thinking about it, and I really don't have that much hair to begin with.
I understand the need for milestones. From my perspective of being on an agile team, we keep going as usual, make sure we integrate on a regular basis, be strict about our meaning of "done" for tasks, fold the exit criteria for milestones into our everyday development and guess what? Our team won't struggle through trying to meet exit criteria in the high pressure closing of a milestone. Smooth sailing.
Maven: That's the spirit. Your team is going to rock.
______________________________
Maven's Pointer: Learn the intricate details of the milestones in your company. At Microsoft, practically every team has a "code complete" milestone. Most developers take that milestone for granted. However, the definition of "code complete" is different on every team. Do all features need to be done? Are to-do's allowed in the code? Do you need to run static analysis prior to code complete? Is some measure of API documentation required by the milestone exit? Learn what "code complete" and the other milestones mean on your team and address the exit criteria early.
Maven's Double Pointer Indirection: Avoid cheating on the exit criteria when the end of a milestone draws near and the project manager does not want to change the date. You are only cheating yourself. Do your best to meet the exit criteria and maintain high product quality even in early milestones. Revisit the exit criteria for future milestones if the exit definition is too strict.
Maven's Resources:
Summary
Motley: Don't be anal about Application Programming Interface (API) design.
Maven: Good APIs are discoverable, consistent, simple, usable, hard to misuse, cohesive, lack side-effects, strongly typed, documented, has tests and samples, extensible when necessary, and are developed with anal minds.
______________________________
[Context: Motley has just been involved a design review for a companion team and has some concerns he is sharing with Maven]
Motley: I am not sure about this. I just attended a design review with Murdoch's team. The issue at hand was a new API they were designing for clients to talk to their logic layer. Their piece is in native code, which I am not that fond of, but design is design no matter what language. This is the fourth API in a row that I have been reviewing that simply extends the IOleCommandTarget interface in COM.
Maven: What's the issue with that API? Forgive me, I do most of my development in managed code.
Motley: Here is the API for IOleCommandTarget::Exec() directly from MSDN:
HRESULT Exec(
const GUID *pguidCmdGroup, // Pointer to command group
DWORD nCmdID, // Identifier of command to execute
DWORD nCmdExecOpt, // Options for executing the command
VARIANTARG *pvaIn, // Pointer to input arguments
VARIANTARG *pvaOut // Pointer to command output
);
You typically use this API to enable some kind of object and the container that it sits in to dispatch commands to each other. It's mostly used with toolbars and menu items.
Maven: Ah, yes, the old "command" API. I worked with a developer in the past who threw one of those into every API set so that he could extend it without publishing a new interface. It became a big grab bag for every little extension and got out of control real fast. At least the Exec() method above is somewhat typed - his command API took in a string that indicated what the method should do. You may as well have that one API as your object model and that's it!
Motley: I don't want to be anal about the design review. This one isn't so bad, but it can be overused, in my opinion. The APIs Murdoch's team are adding to the product are not meant to be used by third parties - they are internal APIs. Why rely on a command-based API set if the APIs are internal?!?
Maven: That's a good question. What was his answer?
Motley: Consistency. They already have a bunch of extensions using Exec() so they are just adding more to be consistent.
Maven: Sounds like some refactoring is in order for the internal APIs. Even for public APIs, unless there is some post-ship customer extensibility, I would not likely follow this model.
Motley: I am having a tough time convincing him that this API approach is suboptimal. I am about to give up. I hate asking this, but any advice? Should I continue to pursue this issue or concede defeat and fight another battle?
Maven: Well, we have talked a lot about design already, and most of those principles hold here. API design, however, is an art that has principles in itself.
Motley: I know of some good examples. There are many examples of good APIs inside the .NET framework. You can get by with just Intellisense for the most part when developing against many of the classes. Well, that is if you forget about ADO.NET for a little bit. I have to relearn the ADO.NET APIs and reread the documentation every time I develop something in that framework, but I digress. And don't get me started on the SharePoint APIs - "site" vs. "web" anyone?
Maven: You are right - .NET holds many good examples. Now based on those examples, what makes the APIs you mention "good"?
Motley: Let's see. Discoverability is high. I can pretty much guess the namespace, object name, and method name much of the time. Microsoft has named the APIs very intuitively so I don't need to rely on documentation other than the Intellisense comments.
Maven: That's a good one! APIs that are easy to find, even without docs, make development a pleasure. What else?
Motley: Along the same lines, APIs should be consistent with the rest of the system. I should just "know" the namespace it's in and the names should make it easy to understand the responsibility of the object or method.
Maven: Good stuff! Keep going.
Motley: Simplicity is also key. I want to avoid jumping through hoops just to use an API. Back to our design principles - coupling can get in the way of simplicity here if I have to create one object before another before I can use the API I really want. I should be able to easily explain what one API does, it should meet the needs of the audience, and there should be few parameters.
Maven: You can tie many of these things up into ensuring APIs have a high degree of usability. Usability does not just apply to user interfaces, but also developer APIs. To complement that, the API should be hard to misuse. Make it difficult for a developer to make a mistake. The IOleCommandTarget interface above is easy to misuse based on the generic in and out pointers. It also contains many parameters, which is contrary to your last statement. Anything else to add to designing great APIs? What else don't you like about IOleCommandTarget?
Motley: It should do one and only one thing (i.e. be cohesive) and have no side-effects. You could argue that the IOleCommandTarget really does only one thing - dispatch commands - but it folds up much functionality within it. I also want my interfaces to be strongly typed so that any mistakes I make are caught at compile-time vs. run-time.
Maven: Wow, Mot, those are some fantastic tips! Any more?
Motley: That about covers it for now. I guess I didn't need your advice after all. BUT, as I have come to know Mr. Always-Has-The-Last-Word Maven, I am willing to bet my paycheck that you have a few more things to add.
Maven: That's a bet I am NOT willing to take! You know me too well, Mot. Here are a few more. Good APIs:
- Are documented. In the world of managed code, always supply XML comments with your APIs. As you noted earlier, Visual Studio's Intellisense parses the comments and shows them to developers while they are editing. This is a great help.
- Has tests and samples. Why not provide your unit tests with your APIs, at least to internal teams? They provide an indication of how the API is used and what kind of boundary conditions there are. Sample code is also extremely important, both internally and externally. Most developers start with a sample and build from it. As a result, samples need to be of extremely high quality as they form the basis for much (eventual) product code.
- Are extensible, but only if you need it. There should be a good reason for extensibility, otherwise you expand your test matrix and open yourself up to all sorts of creative development from the outside world.
- Are developed with anal minds. Yes, that sounds a little weird. Yes, you should be anal when you are reviewing Murdoch's APIs. Once an API is published, you must support it for many years to come. Try your best to get it right the first time. Point out every little issue in an API, no matter how small it seems. Aim for perfection. Be anal when asked to create or review an API!
Motley: This was actually a good discussion. I didn't really consciously realize there was that much to creating a solid API.
Maven: The job of a software engineer is forever complicated.
______________________________
Maven's Pointer: Usability testing applies just as well to APIs as it does to user interfaces. Why not ask for developer volunteers, give them coding scenarios, and ask them to program a small solution to a problem using your new API set? Ask them to think out loud, and ensure you document their every move. It may provide some valuable information on whether your APIs are usable.
Maven's Resources:
Hello faithful readers!
A small announcement: I have decided that with the summer months approaching here in the Pacific Northwest I am going to decrease the frequency of blog posts for a while to once every two weeks (biweekly). The reasons are several:
- Work is taking up quite a bit of time lately, and the blog is a non-work activity.
- I am working on an exciting project that is related to the blog, and that will require some free time.
- Since the weather is, well, suboptimal during much of the year in the Seattle area, the summer time typically brings bluer skies and I want to get outside more :-).
Rest assured I am still very much committed to the blog and will continue posting. I just need to slow it down a bit as it's a lot of effort.
Thanks for understanding. As always, keep the comments and suggestions coming. I love hearing from you.
James.
Next blog posting: Tuesday, June 10, 2008.
Summary
Motley: Bug fix sprints are a Scrum anti-pattern. Quality should be kept high so as not to have to focus on bug fixing.
Maven: A clear meaning of done for sprint tasks is important, but even if you follow this best practice, there will always be integration issues, bugs, and changes required late in a cycle. A sprint focused on these tasks is reasonable.
______________________________
[Context: Motley and his team of merry developers have been fixing bugs for the past week, and Maven has noticed something odd]
Maven: Morning, Mot. How are things going?
Motley: Not too bad, although I'm a bit late getting in this morning. It's tough to get out of bed when all you have to look forward to is fixing bugs for the next couple of weeks. The bug database is the team's to-do list for the next little while.
Maven: Yeah, this is a part of the development cycle that is rarely a developer's favorite. Is morale still high in the daily Scrum meeting?
Motley: Daily Scrum? Why bother? We are done with Scrum now that we are integrating and bug fixing. Besides, bug fix sprints are a Scrum anti-pattern. Didn't you tell me the other day that Scrum only works for new feature development?
Maven: Um, you must have heard that statement from someone else, or misunderstood something I said. I would partially agree with your statement that bug fix sprints are a Scrum anti-pattern. In fact, most of the Scrum/agile purists would likely say that you are absolutely right. However, past experience and the reality of software development have made me realize that bug fix sprints can be a necessity in large organizations practicing agile development.
Motley: Why?
Maven: One of the most important concepts of Scrum, and where many teams fail, is having an explicit meaning of done for each and every task on the sprint backlog. For example, a typical coding task may involve the following in its "done" definition:
- Design document reviewed (if necessary)
- Feature fully implemented (including error handling, performance, reliability, security, maintainability and other quality concerns)
- Buddy build complete
- Buddy test complete
- Unit tests written
- Code coverage from unit tests > 80%
- Static analysis run
- Code reviewed by at least one peer
- Code checked-in to the source tree
Motley: That's getting pretty detailed isn't it? What's the point?
Maven: The point is that we all have a common understanding for what it means to complete a specific coding task, and understand what "shippable quality" means. If you do not explicitly define the task, some tasks may get missed costing you valuable time and effort later in the cycle when change is more difficult. With agile, we want to produce high quality deliverables very early and not rely on a long "stabilization" phase at the end. If we do everything we can to prevent bugs up front, we'll actually save time in the long run and theoretically ship sooner. Having defined meanings of "done" for all our tasks is a big key.
Motley: Which reinforces my point that a bug fix sprint is an anti-pattern. Jeez, Mave - did you not have your morning jolt of caffeine?
Maven: If everything goes according to plan and you integrate components as you go, then yes, longer bug fix periods are somewhat of an anti-pattern. But let's be realistic. There is some measure of integration and late discovery of unknowns in any large software project. For a small project of a few thousand lines of code, you may be able to avoid a bug fix stage like this. However, take James' experience in working for a previous company. One thing his team was responsible for was porting a large code base from one platform to another. The team added unit tests where possible but much of the code was inherited legacy code that did not come with tests. The test team worked closely with the development team to find as many problems as possible up front, but when you have tens of thousands of lines of code running on a platform that is changing underneath you, you must expect to find bugs later.
Motley: Not to mention when people in the organization start hammering on your code as real users would. I think James called that "dogfooding" when I talked to him, as in eating your own dogfood. That does not sound very appetizing.
Maven: Correct. You cannot possibly predict all the issues that will arise and fix them proactively. As a result, his team undertook what they called a "quality sprint", which focused on fixing bugs. They took an agile approach and refused to develop new features until the product was at shippable quality, and they hit ZBB.
Motley: What the heck is a "ZBB"?
Maven: "Zero Bug Bounce". It's a moment in time where you have no bugs that are older than, say, three days and there are no ship stoppers. At this point, the team can make a call that quality is high and new features can be developed.
Motley: You called it a "quality sprint". You could also call it an "integration sprint" depending on what your goals are. In fact, that might be a better name for what we are doing now.
Maven: Agreed. The term I don't like ,however, is "stabilization sprint". To me, this implies that the code you wrote was crap to begin with. In reality, there may be things beyond your control that crop up upon integration even though you focused on quality, so I prefer to give it a more positive spin. "Stabilization" also typically implies a huge focus on testing. Testing is part of what happens here, but testing is also happens in every sprint.
Motley: Game it if you want. If it looks like a duck, and quacks like a duck, well, you know. You still have not answered the question as to why I would want to practice Scrum when I am fixing bugs!
Maven: You don't necessarily have to practice Scrum while bug fixing, but you may find it valuable. When your team needs to fit within a larger ship cycle on a big team, you may not have full control over how to structure your bug database. In addition, your team may live within the confines of a waterfall-oriented organization, but you can be agile as a feature team by practicing Scrum and transferring prioritization of fixes to the sprint backlog. The team does the usual sprint planning figuring out which bugs are most important to fix. The sprint backlog is then a prioritized queue of bugs, and devs grab bugs from the queue as soon as they are done fixing their current bug and validating with test.
Motley: Is it really worth the overhead though? I already have my list of things to do in the bug database. Why duplicate it?
Maven: I generally don't consider Scrum as "overhead" - it's a very lightweight process, although I would agree that with any process you arguably get some level of overhead. You could fairly easily write a tool that automatically populated the sprint backlog with the top bugs from the bug database. Note, however, that you also gain the following benefits by practicing Scrum while fixing bugs:
- The daily meetings ensure improved communication and collaboration
- Data gathered is useful for future planning sessions
- Frequent retrospectives allow the team to constantly improve
- The team can adjust priorities during the sprint
- The team need not rely on a rigid bug database schema, and can use the sprint backlog to customize information about the bugs
- The team remains focused, and has a Scrum Master to help
Motley: Fine, fine, fine. We'll try it your way for a sprint and see if it pays off. I did kind of lose track of what Marvin was doing with his bugs and the daily meeting should help.
Note: I'd love to hear some better suggestions on how to deal with the problems discussed in this story. Arguments based on reality (vs. theory) are much appreciated in comments!
______________________________
Maven's Pointer: Here's another argument for tracking time spent on tasks in a sprint: over several short quality-oriented sprints you can gather data on the amount of time it takes to fix each bug. By mining that data, you can compute an average length of time per bug, which helps you with effort estimates for capacity planning on future bug fixes. Our team, for example, knows that it takes just under 5.4 hours of effort to fix an average bug. We can then estimate how many we can fix during any given sprint, and set expectations accordingly.
Maven's Resources:
Summary
Motley: OneNote is just like Microsoft Word, and is not a place where a developer should be spending time.
Maven: OneNote is perfectly suited to feature teams and developers. It provides simple, organized, efficient, and easy to manage method of organizing ideas and general content. It is a developer's best friend!
______________________________
[Context: Motley wanders by Maven's desk while taking a break and notices that Maven is spending a lot of time in one particular application]
Motley: Hey, Mave. I heard you typing like mad on the keyboard as it was time to take a break. I figured an interruption would do you good.
Maven: Yeah, thanks. I think. What can I do for you?
Motley: I was just noticing that you have a ton of content in that application on your screen. Why is there so much junk in there? What is the application you are using?
Maven: Ah, yes, the greatest application that Microsoft has ever shipped!
Motley: You mean Microsoft has shipped an application that you call "great"? Ha, ha. Just kidding. I couldn't live without Visual Studio, for example.
Maven: Always the character. The application I spend the good majority of my day in is Microsoft OneNote. It comes with the Office suite of products, and it is a software developer's best friend.
Motley: Shouldn't you be spending more time in Visual Studio as a developer? Are you slacking off again, Mave?
Maven: As a developer, how much time do you really spend coding?
Motley: Not as much as I would like.
Maven: Exactly. There are lots of other things to do, particularly as a development lead. For example, you have to build people's careers, write design documentation, investigate issues and plan for the future. I use OneNote to help me gather my thoughts.
Motley: Why not just use a Word document? OneNote just looks like it reinvents the wheel! Why do I need to learn another application when I am proficient with the 10 useful features of Microsoft Word?
Maven: OneNote is completely geared towards quick note taking. OneNote's user experience stresses organization and simplicity. For example, embedded within it is the hierarchical idea of notebooks, sections, pages, and subpages. Simple drag/drop operations let me organize my notes in this hierarchy. It hides the ideas of files - sure, there are files under the hood, but I never really ever have to see them.
Motley: Wait a second, Mr. Wise Guy. What about saving your notes? I have you there!
Maven: You're pretty smart, but you're question is misinformed in this case. OneNote completely abstracts away files and saves content changes under the hood. You never press a save button.
Motley: That's pretty cool. I wish more applications did that. As long as it saves frequently and I don't lose data when the inevitable crash happens.
Maven: I don't think I've ever had OneNote 12 crash.
Motley: I suspect you are lying, but enough about files. What else makes this thing a developer's best friend? I have never ever viewed Microsoft Word that way!
Maven: OneNote is always open on every PC I use - home, work or laptop. I set up a shared notebook on my home PC for personal stuff like to-do lists and blogging, and I set up multiple shared notebooks at work for work stuff. The first time you access the shared notebook from a P