Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

Sisyphus 'R' US

Sisyphus 'R' US

  • Comments 18

Lately I've felt like Sisyphus.  It's a natural part of the process of software engineering, but it doesn't change the feeling.

In every software project, after all the new code's been written, focus shifts to resolving and removing the bug backlog.  Of course fixing bugs is an ongoing part of the process, but you've always got the temptation of new work to keep you going.  But on a given project, when you're done with the new work, you've got to fix the bugs before you can move on to the next project.  During this period, you live and breathe by your outstanding bug count, looking for the time when you hit the magical ZBB (Zero Bug Bounce).  This is, without a doubt my absolute favorite part of a project - I absolutely LOVE this phase (I think it's a hunter-gatherer thing).  It is unbelievably satisfying to find the 18 month old bug that's been wreaking havoc throughout the system (that happened last week).

Having said that, while you're trying to get to ZBB, the testers in your organization are constantly finding new bugs.  The good news is that you're not writing new code, so you're hopefully not introducing new bugs (it happens, but bug fixes tend to have fewer bugs than the original code).  But there's a period of time when the incoming bug rate is higher than your bug fix rate.  You work your bottom off trying to reduce your bug backlog.  There are really three ways you can reduce your backlog.  First off, you look at bugs and eliminate the duplicates (I found 4 dups in my bug backlog on Friday, for example).  Second, you can investigate the bug and determine that the bug is actually in a diferent component and assign it to the new component.  And thirdly you can fix the code. 

All the while, while you're trying to reduce your backlog, the testers are still finding new bugs.  In addition, other developers in the organization are busy investigating their bugs, and as a result, they're reassigning their bugs to you.

After a while, it all settles down - bugs get triaged appropriately, duplicates get weeded out of the database, the testers start running out of bugs to find (because you've fixed them), and eventually you start heading on the path towards shipping your product.

But right now, it's Sisyphus time - my incoming rate is as high as or higher than the fix rate.  It doesn't seem to matter how many bugs I resolve per day, my outstanding bug count seems to go up.  The other day, I resolved ten bugs during the course of the day, and at the end of the day I had the same outstanding bug count as I did when I came in.

I'm working as hard as I can, but against the outstanding bugs metric,  I'm not quite keeping up.  This can be an incredibly poisonous situation to be in, because you can't ever relent on the pace.  If you slack off at all, then your all important bug total will explode and you've got still more work to do.

The good news is that it's temporary.  As I said, it settles down in a while, and the incoming rate will start drop below the fix rate. 

Right now, on the other hand, I feel like that guy sitting there rolling the ball up the hill, never to see it hit the top.  I know I'll get it up there, and I'll be able to stand at the top knowing that I've really achieved something.

 

So I go into the office every single weekday to push that dad-blamed rock up the hill again.  Who knows, maybe I'll make it over the top this time!

Edit: Changed some text to remove some negativity.

  • I must admit that I quite like this bug-fixing phase of software development! Why? There are some really great challenges involved in tracking down some of the more obscure bugs - I find it really satisfying. Good luck Larry - maybe it will inspire some more "What's wrong with this code?" articles.
  • Matt, thanks for adding a smidge of perspective. I've actually changed the text to point out that this is absolutely my favorite time in the development cycle.

    I LOVE fixing bugs, it's one of the things that I'm really good at (I've still got the flyswatter that the NT networking group had made as a prize for "most bugs fixed").
  • Larry, I always tend to take it personally when bugs are filed against my code. Almost like it's a reflection on my coding skills. I know that feeling of being Sisyphus all too well, but what do you do to overcome that?
  • Todd, you don't. What you learn is that the bugs aren't personal. They're opportunities to learn from your mistakes.

    And you also need to realize that this is just a small stage in the life of a project, and that "this too shall pass". Every project has this time, it's actually a good time, because it means you're about to hit the downward slide to finishing up.

    The other cool thing about this part of the project is that the easy bugs are gone, and you start seeing the really hard ones. Those are ALWAYS fun :)
  • Larry, did you ever catch the Dilbert animated TV series that lasted for a single season?

    PHB: You've gotta look at it from the rock's point of view!

    I've got my own personal hell courtesy of COM aggregation to deal with at the moment, it should make a good story once it's actually fixed ;p
  • Larry said: "bug fixes tend to have fewer bugs than the original code"

    But in _Code Complete_, Steve McConnell (citing Yourdon) says, "defect corrections have more than a 50 percent chance of being wrong the first time." [p. 637]

    So I guess it depends how you count bug fixes.

    I worked at one company that truly grokked how often bug fixes end up introducing new bugs. A known bug can often be better than fixing it and introducing an unknown one in it's place. Nearing release, you practically had to plead with Change Control to be allowed to fix a particular bug. Fixes in the late game had to be important from a customer perspective, the cause had to be well-understood, and the fix had to be reviewed by multiple developers.

    Despite shipping with what some might consider to be a high known-bug count, the software's reputation was excellent, as most of those known bugs were low-impact. And we always shipped on time. ALWAYS.

    That said, I can totally relate to the feeling Larry is describing. I try not to look at anything other than the bug I'm working on right now. The bug count is going to do what it's going to do. Deal with them one at a time, according to priority. Put in a good day's work and then forget about it.

    I enjoy bug-fixing phase, because the work is more discrete. Sure the count may have gone up, but at least you can pat yourself on the back and say that you've fixed 10 bugs today, and that means customers are going to get a better product. That's concrete progress.
  • Even if every fix has a 50% chance of introducing a new bug, if you fix 100 bugs, you've still got 50 fewer bugs in the product than you did when you started.

    When you get close to shipping, the bar for bug fixes goes WAY up, and there is a huge amount of effort that goes into ensuring that only high quality fixes go in. But when you're months away, it doesn't matter.

    Also, in my experience, the reality is that fixes are usually good, the problem is that the fixes then expose other bugs that previously existed but was hidden by the previous bug (I had one of these happen just Tuesday).
  • My 2 cents: I view the phases of a software project like the change of seasons. Summer (coding) is over and now it's fall, time to buckle down, expand test coverage, etc. Winter is coming soon (the desolate but exciting times when you have to evaluate the risk of every bug fix because while you might fix X, you could change how Y and Z work...) but then spring and the next product.

    Re: bugs and ego:

    At some point you realize that nobody's perfect and that the "engineering discipline" required to produce a "perfect" product is beyond the GNP of the entire planet earth. Bugs are inevitable - the key is to make them discoverable and diagnosible without spending 6 months trying to figure out who's corrupting stack randomly. This is why I focus on coding and error handling techniques so much.

    Frankly it's one of the "growing up" things that I learned when I stopped working alone on projects and instead started working on a team, especially alongside teams with highly esteemed engineers. The existence of any one bug is not a personal affront; it's just economics. It's trends that matter.

  • > bugs get triaged

    Of course.

    > appropriately

    Only on lucky occasions.

    ===

    > What you learn is that the bugs aren't
    > personal. They're opportunities to learn
    > from your mistakes.

    Yes. But repeated bugs, and refusals to correct triaging errors, and refusals to release bug fixes (the 50 out of 100 that are really fixes) aren't covered by this, and they still happen too often.

    > When you get close to shipping, the bar for
    > bug fixes goes WAY up

    That doesn't seem like the right way to do it. If the ratio of needed bug fixes to allowed bug fixes explodes when the bar is raised, you need to convert the shipping target to a beta. Meet the shipping date but start confessing to the number of problems, and next year deliver a service pack to convert the beta to the intended product.
  • I am really encouraged that even a guru has bugs in his orginal code. Where I work it is looked upon as a small disaster if your code has any bugs in it and introducing another bug after fixing a bug is unforgivable. I feel much better after reading this - thanks Larry.
  • Johnny, if anyone tells you their code is bug free, they're lying.

    Everyone has bugs in their code.
  • > Who knows, maybe I'll make it over the top this time!

    :) Thanks for sharing, Larry. My little app got bug-bashed Friday so me and my little boulder are currently at the bottom of the mountain. :) It's little consolation that I did't write the code myself or that I'm contingent staff--I *own* these bugs, at a very real, visceral level. I agree that the bug-fix phase has a positive side. I find the pinpoint problem-solving focus to be challenging and, obviously, rewarding when I get to resolve the bug. Gotta go...
  • The differences between a guru and a non-guru is the rate at which we fix our bugs and the rate at which we learn to make new bugs next time instead of repeating old ones.
  • s/is/are/

    s/current preview functionality/preview functionality/
  • Spare a thought for those of us who have to pick up someone else's awful but production-critical products and spend our time trying to fix deep-seated bugs without breaking anything else, and never having to enjoyment of going through that mythical thing called "development".

    Or even better, take over someone else's awful but production-critical products, get tasked with "rewriting" them but being given no opportunity to give our input on what direction the design should take. In fact, what this "rewrite" turns out to be is just a huge scale bugfix.
Page 1 of 2 (18 items) 12