Progressive Development : refactoring

Motley says: "Code comments are for sissies"

James Waletzky — Tue, 28 Aug 2007 19:03:00 GMT

Summary

Motley: My code is so readable and self-documenting that comments are unnecessary.

Maven: Comments should address the intent (the why) behind the code, not the how. Extraneous comments around how may indicate refactoring opportunities.

______________________________

[Context: Maven is doing a code review of Motley's code and notices there are no comments in it]

Motley: How's that code review going?

Maven: Not bad. Is there a reason, though, that you don't have any comments whatsoever?

Motley: Not needed - code comments are for sissies! My code is so readable and self-documenting that they are unnecessary. Just like a natural beauty doesn't need make-up, my code doesn't need comments!

Maven: Your code is good, I'll give you that. However, I can see a few places where comments are necessary.

Motley: I have to assume that the people reading my code are smarter than slugs - and hopefully less annoying. Beings above slugs in the evolutionary chain don't need comments.

Maven: Score one dig for you! Ouch. There are good practices and bad practices with comments. Let's start with the bad. If you are commenting code to meet some kind of comments-to-lines-of-code metric, that's the wrong reason.

Motley: Hahahaha. That's as ridiculous as your last haircut!

Maven: Don't laugh - I've seen it done. Comments obviously have to add value to the person reading the code. A comment like this one is totally useless:

// increment i

i++;

Motley: Oh, come on. What mor-

Maven: Ah! No insults please. I've seen that done too. It's a waste of bits.

Motley: Ok, Mr. Einstein. When is it good to have comments? I'm paid to write code, not stories.

Maven: Comments are extremely valuable in code when they convey useful information that the code itself cannot. For example, the code itself cannot express the intent that the developer had when she first wrote it. The code itself cannot tell you why the code is the way it is. Sure, a good developer can figure out how it works, but the why may not be obvious. Good comments express intent.

Motley: Isn't that what a design doc is for?

Maven: Good question! Design docs can express overall intent, but for details, why not keep that really close to the code? That way it may even get updated, whereas we often forget to update design docs when we make an intent change.

Motley: Fine. Keep the low-level design documentation with the code, as we talked about.

Maven: You bet. And you can use the XML comments built into C#. You can use that mechanism to document public APIs, which is another good reason for comments. Let's save the details on how to do that for another conversation.

Motley: I'm not familiar with XML comments in C#, but it sounds geeky enough to peak my interest! Back to what to comment - I was reviewing Mort's code the other day and he had a ton of comments in there describing how he implemented things. Since I'm not a slug, I can delve into the code and easily figure out what it's doing without looking at the comments. In addition, there is a high probability of the comments getting out of date. I just went and removed them all. Perhaps you should have this talk with Mort?

Maven: The best way to solidify learning is by teaching, as Dr. Covey says, so I'll let you have that conversation with him. He's going to be a little pissed that you removed all his comments though. Just blindly removing comments is not usually the right thing to do. If someone comments a block of code describing the code itself (instead of intention), there is probably a refactoring opportunity. Usually blocks of code like that have some complexity (thus, the comment). If you leverage the refactoring tools we talked about and Extract Method, you can isolate the complexity behind a nice readable method or class. Comments about how often indicate refactoring opportunities.

Motley: Ok, you're right - Mort's probably going to be a little upset. I'll roll back my change and have a talk with him about refactoring opportunities. I'll also set him straight on commenting why. I don't often fully agree with you, but this time I'm easier to please since I was expecting you to tell me to comment the crap out of my code. It's nice to see your advice is not slug-like.

Maven: You think you know me so well, don't you Mot? Ha! If we go back to agile principles, we want to do the simplest thing possible, always add value, and do just enough. Everything we talked about fits within that mould.

______________________________

Maven's Pointer: Steve McConnell talks a lot about comments in his books and web sites. He has a relatively good checklist that lists good commenting techniques that is worthy of reviewing. I don't agree with all of the checklist (e.g. author's name and phone number in the listing), but there are indeed some gems.

Maven's Resources: Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, 2004.

Motley says: "Refactoring means no more up-front design"

James Waletzky — Tue, 10 Jul 2007 18:08:00 GMT

Summary

Motley: Refactoring keeps my design clean from the start, so no more up-front design!

Maven: Up-front design is still necessary to achieve clarity on the overall approach (preventing rework later) and needs to be documented to allow others to review your thinking.

______________________________

[Context: Motley is trying to educating himself on agile development and is reflecting on some reading with Maven]

Motley: This is really cool - I've been doing some reading on agile development and because I am doing refactoring continuously, that means I can do away with all that up-front design stuff we used to do.

Maven: Do away with it? Hardly! Maybe for a small couple-hundred line program, but any bigger than that, there is still value in thinking through the problem before you code anything - at least, if you want to prevent major rework later in the cycle when mistakes are more costly to fix.

Motley: But refactoring is going to keep my design clean. What's the point of all the up-front thinking?

Maven: That's true - refactoring helps you continuously keep quality high. However, if you make an investment in a bad high-level design and architecture, it becomes extremely hard to refactor yourself out of it. Then you are talking about reengineering, which can be a huge task. Refactoring is for small design changes without changing external functionality. Large changes are not as well suited to refactoring as it may lead to drastically changing the way requirements are implemented.

Motley: But I want my design to emerge based on the learning I get while coding!

Maven: You want the details to emerge, but you need clarity on the overall approach before you flesh out the detail. A design phase is needed to understand the problem, investigate various high-level solutions and their trade-offs, and document your overall approach.

Motley: I'm not sure I buy that - the agile manifesto states that we should favor "working software over comprehensive documentation." No design document needed!

Maven: Not exactly. You stated the keyword: favor. It does not say there is no need for documentation whatsoever. We need that documentation to keep the stakeholders in our design happy and to provide something concrete for people to review.

Motley: And I thought I was done with 100 page design documents…

Maven: You are! You want to document "just enough" in your design. Documents of 100 pages in length are generally overkill and violate the lean principles we talked about previously.

Motley: But how do I know what is "just enough"? If I have 3 days on my schedule for design, "just enough" usually just means I hit the end of day 3.

Maven: Let's save that for another conversation. I have a great way to measure "just enough", but I am going to leave you in suspense.

Motley: Fine. I don't want to be here all day anyway.

Maven: Of course, the other topic of interest is once you have "just enough", you want your design to be "good".

Motley: Great. I want "just enough" - I have no idea what that really means - and I want it to be "good" - I have no idea what that really means either. Judging "good" on a design is all in the eye of the beholder.

Maven: You're comparing judging a design for "goodness" to judging looks?? Hah. Fortunately, we can get a little more scientific than that and focus on the principles that generally make one design better than another. Let's do that...

______________________________

Maven's Pointer: Be careful with the term "emergent design". Many developers equate the phrase "emergent" design with jumping into Visual Studio and coding. What "emerges" at the end is some kind of design (usually junk). Truly "emergent design" uses a feedback technique to provide you continuous confirmation that you have a clean and working design. An example of an emergent design technique is test-driven development (TDD). Tests drive your design, refactoring keeps it clean, and test automation validates it all works.

Maven's Resources: Agile Development Manifesto, Wikipedia entry on Continuous Design

Motley says: "Test both private and public methods"

James Waletzky — Tue, 12 Jun 2007 10:41:00 GMT

Summary

Motley: To get high code coverage, we should be testing both public and private methods

Maven: Only test public methods. Testing private methods gets in the way of refactoring

______________________________

[Context: Maven checks in on Motley's unit testing practices and notices something odd]

Maven: Hey Mot - how is the unit testing thing going?

Motley: Pretty good, I must say. I'm leveraging Test-Driven Development (TDD) as much as possible and it has really improved my code coverage. Check out all these tests!

Maven: Wow - that is pretty good! In fact, that's more tests than I was expecting. Are you literally testing everything?

Motley: You betcha. I have test cases around every method I write.

Maven: Even the private methods?

Motley: Duh. Of course! I gotta get that 100% code coverage, man! That was your idea don't forget.

Maven: Okay, I am willing to bet you are doing TDD without the refactoring part of the process. Am I right?

Motley: Well, um, I guess. I'm trying to do a bit but it's such a pain in the butt. But wait, how can you tell just by looking at my tests???

Maven: Well, if you are testing private methods that typically leads to a slightly larger number of tests, because a good practice is one test (or more) per method that you are testing. If you just test public methods, you can typically get the same code coverage with a slightly lower number of tests.

Motley: So what? So I have a few more tests. It doesn't harm anything.

Maven: Ah, but it does. Testing private methods makes it really hard to refactor. You see, refactoring does not usually affect a public interface but instead the implementation behind that interface, i.e. the private methods. With TDD, you need to be free to refactor, or you defeat one of its primary purposes - to drive a better design. If you need to update tests every time you make a refactoring change to your implementation because you are testing private methods directly, it really gets in the way and you are not going to do it. I am willing to bet that's why you are skipping an extremely important part of the TDD process.

Motley: The private methods do get in the way of refactoring, but code coverage is everything!

Maven: With TDD you don't write a line of code without having a test in place first. When you do refactoring, you rerun your tests to ensure you haven't broken anything. You should theoretically end up with the same coverage even just testing via public methods. If you can't hit your private methods through your public methods, why are they even there? Besides, the code coverage number in itself is not super interesting. What is more interesting is the analysis of why coverage numbers are low. This can lead you to more tests that you may otherwise miss. Don't go overboard trying to get to 100% - it's not worth it. Some C# language constructs, like foreach, have extra compiler-generated code in the intermediate language, which makes it real tough to get 100%. It's not worth the effort. Oh, and by the way, when I say "public", I typically also mean protected methods (public for subclasses) and even internal methods (public within the assembly).

Motley: So you're saying I should just test public methods and code coverage will just come if I use TDD. You're also saying that getting to that magical 100% code coverage number is often not worth the extra effort, and although code coverage is a good measure of your tests, the analysis is more important.

Maven: You should write a book, Mot. I could not have put it better myself.

Motley: I'm a quick learner, as you can see from my stellar deliverables. I can even teach you a few things, like how to be likeable in a social atmosphere.

Maven: What are you saying?!?

______________________________

Maven's Pointer: You may ask, "How do I test private methods anyway? They are private!". In .NET, there are several ways to test private methods should you have a rare case where you definitely need to test a private:

Leverage a test framework that uses reflection to call methods. The System.Reflection APIs can access private members of a class
Use Visual Studio 2005 (and later) for unit testing. It will generate a test context object when you use the wizard to create a test that lets you access privates.
Create a friend assembly using the InternalsVisibleTo attribute that lets you access internal methods at least.
Create/Run a tool that flips some bits in the .NET assembly metadata to change the scope of classes and methods all to public (you are not testing the original compiled assembly in this case though).
Create/Run a tool that generates a proxy for your class and accesses members via reflection. Your tests then use the proxy instead of the original class.
Use preprocessor definitions to change the scope of the private methods to public in a special build.

Maven's Resources: How to Test Private and Protected Methods in .NET provides similar arguments for/against testing private methods, and talks about a subset of the above list for testing private methods.

Motley says: "Refactoring is too hard - it's not worth the effort"

James Waletzky — Tue, 24 Apr 2007 21:42:00 GMT

Summary

Motley: Every time I make a change in refactoring I either cause compile problems or break functionality. Refactoring is slow and not worth the effort.

Maven: Use tools to refactor and leverage unit tests to improve your confidence that your refactoring does not break functionality.

______________________________

[Context: Motley is refactoring some code but having a tough time with it.]

Motley: #$%#%! This refactoring thing is a load of crap. It's too hard and not worth the effort. You were definitely high on something when you gave me this little tidbit.

Maven: Maybe I'll take a teeny step back for a second and let you cool off.

Motley: Not if I chase you into the kitchen with my iron fists of fury!

Maven: Well, errr, what's the problem?

Motley: Every time I make a change to "improve" the design I end up breaking something. Sometimes functionality breaks with my change and I discover that when I run the application, and sometimes I just have compiler error after compiler error. It's REALLY frustrating!

Maven: You know what I'm going to say next?

Motley: Grrrr. Yeah. "I can help with that". Well give it to me before I wring your neck!

Maven: I can definitely help with that. Let's start with the compiler errors. Even doing simple things like renaming variables - which is a simple refactoring - can cause compiler problems. Using Find/Replace across all files in a project can help, but you can still miss a file - particularly if you change an interface. But that was the old days. Let's try a basic refactoring example in Visual Studio 2005 that will make your life easier. Double-click the name of the variable you want to rename.

Motley: Grrrr… Fine. Now what?

Maven: Right-click on the variable that is highlighted. See that little "Refactor" menu that appears? Mouse over it, look at the pop-up menu, and click "Rename". Type in the new name of the variable and click "Ok". VS gives you a preview of what it is going to do. When you accept the changes it goes through and takes care of all the renames in a safe way leveraging all of the symbol information it creates when it compiles your project. I've come to trust it quite a lot.

Motley: Now that is pretty cool, I must say. But I don't use VS all the time.

Maven: Most other modern IDEs these days have refactoring support built in. In fact, the idea has been around for quite some time. Smalltalk developers are quite familiar with the technique. It is a huge time saver.

Motley: Ok, and I see there are quite a few different refactorings available. I'll have to play with that. But what about breaking functionality? That's still getting in the way of refactoring!

Maven: You're doing Test-Driven Development, right? Which means you have unit tests?

Motley: Yeah, so?

Maven: Are you running your unit tests after each and every refactoring?

Motley: No. That seems like overkill.

Maven: Well, that's part of the problem. You've spent a bunch of effort creating unit tests. They definitely help drive the design, but they also give you confidence to refactor the code. Every time you make even a small change, passing unit tests give you confidence that you have not broken any functionality.

Motley: Fine. I can see how that would help, but unit tests only cover small increments of code. I could still be breaking other dependencies/clients.

Maven: I can't argue with you there. That's reality and a most excellent point.

Motley: I guess validating the unit has not broken still goes a long way in preventing breaking functionality. I'll stick with this a little while longer and try and prove you aren't as dumb as you look.

Maven: I appreciate that. I think.

______________________________

Maven's Pointer: Visual Studio 2005 offers several different refactorings - mostly from Martin Fowler's refactoring catalog. Examples include rename, extract method, encapsulate field, extract interface, promote local variable to parameter, remove parameters, and reorder parameters. Experiment on some sample code by highlighting chunks of code and applying the refactorings. Once you get used to it, you'll save huge amounts of time. Other IDEs like Eclipse for Java also offer refactoring support.

Maven's Resources: Refactoring C# Code Using Visual Studio 2005

Motley says: "I don't have time to Refactor - I need to get it right the first time"

James Waletzky — Thu, 19 Apr 2007 16:34:00 GMT

Summary

Motley: Refactoring takes too much time - I have to ship you know!

Maven: Refactoring leads to much more maintainable code, and if you have tests in place, the return on investment far outweighs the cost.

______________________________

[Context: Motley has just embarked on his first adventures with TDD, but Maven wants to check-in to ensure Motley really understands the process]

Maven: So how are things going with TDD, Motley?

Motley: I've only been at this for a couple of hours - why are you bugging me already?

Maven: It's good to start off on the right path. Just want to make sure you understand the TDD process.

Motley: Yeah, yeah, yeah. Write a failing test. Write code to pass it. Move on to the next requirement.

Maven: Whoa there, Nelly! You forgot one of the most important steps - refactoring.

Motley: Not really sure what you mean by that. If I have syntax errors I fix them. Duh.

Maven: Refactoring is all about improving the design of your code without changing any existing functionality. You mentioned previously that one of your values is writing great software. Well, refactoring is not just about writing it great the first time, it's about being realistic with yourself that you won't get it right the first time. You make the necessary changes to prevent it from starting just OK and turning into a big turd later as you maintain it. Obviously it takes more time than just writing the code once, but the pay-off down the road is huge.

Motley: You've been sniffing too much glue, Maven. I barely have enough time to write my code once let alone keep redoing it over and over. I have to ship. If I don't ship, the business suffers, and most important, I get fired, and I suffer.

Maven: What about when someone has to maintain that code moving forward? Do you want to make it difficult for them? How much do you think that costs the business?

Motley: Like everything in life, there is a trade-off. I have to do my best given that one shot is all I get. I vote ship and deal with maintenance issues later.

Maven: That's what has gotten us into all this trouble with our legacy code! How do you like fixing bugs in all that old stuff?

Motley: Well… it sucks. Our legacy code is a "turd" - as you so eloquently put it.

Maven: Exactly. We need to make it easier for the next person to maintain. Refactoring is a way to get us there. Write your code, improve it, and then move forward with the next requirement. It's actually quite natural once you get in the rhythm.

Motley: But I'm afraid that once I have it working that if I make changes I'll break something.

Maven: Ah, but you're doing TDD, right? That means you have tests in place so that if you do happen to break something you know about it right away. It's quite beautiful, really. Give it a shot and I'll check in with you later.

Motley: Why do I keep letting you do this to me? Fine, I'll try it. But if I'm late on my deliverables, it's your paycheck. Deal?

Maven: I'm so confident that you're going to rock that yeah, I'll take that bet.

______________________________

Maven's Pointer: Many times the primary reason that developers are afraid to refactor isn't because they don't have the time, but because they don't have the confidence to make a change without breaking existing behavior. Having unit tests in place increases that confidence and makes refactoring easier and fun. Avoid refactoring without first having tests in place!

Maven's Resources: Refactoring: Improving the Design of Existing Code, by Martin Fowler et al, Addison-Wesley Professional, ISBN: 0201485672, 1999.

Motley says: "I can't write tests before application code - there's nothing to test."

James Waletzky — Mon, 09 Apr 2007 18:32:00 GMT

Summary

Motley: You can't possibly write tests before code - there's nothing to test.

Maven: Writing tests for a method before coding it has all kinds of design advantages.

______________________________

[Context: Motley has been writing unit tests for a week. Maven wants to take the next step]

Maven: Hey. Looks like you're getting pretty good at this unit testing thing.

Motley: Yeah, I can definitely see the benefits of having a thorough suite of tests. I can make sure that with every code change I don't break anything.

Maven: Want to take it even farther? Do you want to make an even bigger difference in quality?

Motley: You really are a poop disturber, Maven. Fine. Let me hear about your hair-brained scheme.

Maven: How about this? Try writing your tests first before you write your code. It's called test-driven development, or TDD for short.

Motley: I can't write tests before I write any code! There would be nothing to test! You've been sniffing to much glue for your own good.

Maven: No, I am serious. Thinking about how you are going to test your code before you write it has all kinds of advantages, including driving your design. In fact, TDD is more of a design technique than a test technique when you really think about it.

Motley: So how could this possibly drive my design? What difference would it make if I write my tests before my code - which still seems crazy to me - or after? It's like saying that I should edit my book before I write it.

Maven: Maybe if you thought more carefully about how your book was to be edited, you'd do a better job writing the book the first time. With TDD, since you write your tests first, your design automatically is testable, which the test team will love. You can also view your APIs from the perspective of a client even before the API is implemented, so you know right up front whether the API is usable. In addition, coupling and cohesion problems in your code come to light immediately. I am sure you hate programming to difficult APIs. Well, this way you make sure that your APIs are clean before you start because you are using them. TDD essentially forces a good design to emerge based on your tests.

Motley: I'm still not sure about this. So all I have to do is write my tests before the code?

Maven: Well, to truly do TDD, here are the steps to follow:

Start with a requirement. A requirement could be one method you want to implement.
Write a test for that requirement.
Write just enough stub code to make the test and code compile.
Write just enough code to fail the test.
Execute your tests and watch them fail.
Write just enough code to pass the test.
Execute your tests and watch them pass.
Refactor.
Execute your tests to ensure you have not broken anything.
Go to step 1.

Motley: That's a lot of steps. Seems too complicated to really be useful. And what's the point of writing a failing test first?

Maven: Actually, it's not complicated. Once you get in the groove, it actually feels quite natural. Oh, and the point of writing a failing test is essentially to test your test. If you write a test and expect it to fail but it passes, you know you have a problem with either your test or your code. By starting with a failing case and then watching it pass, you are more confident that your test is good.

Motley: Let me finish what I am doing with my current set of modules and then maybe I'll give this a shot. No promises. You'll also have to enlighten me a bit on the refactoring step, as I'm not sure what to do there.

Maven: No problem, Motley. I applaud you for having an open mind to try this stuff out.

Motley: Whatever. The boss says you were hired for your sound engineering ideas, so I guess I should humor him and listen. Hehehe. Kidding, of course. Maybe.

Maven: Stay tuned. I have a lot more to say about TDD. It's a great developer technique.

Motley: Can't wait. Ugh.

______________________________

Maven's Pointer: A precursor to effectively doing test-driven development is understanding how to write good unit tests. Use a proven test framework to make it easy to write tests, and ensure your tests execute quickly. After all, with TDD, you are perpetually running your tests. It's such a great way to develop code that I don't think I'll ever go back to writing tests after code.

Maven's Resources: Test Driven Development: By Example, by Kent Beck, Addison-Wesley, ISBN: 0321146530, 2002.