Ron Jacobs

Windows Workflow Foundation

Judging the quality of open source projects

Judging the quality of open source projects

  • Comments 12

It seems to me that people who want to use open source projects have a big problem.  Several problems actually, but perhaps the most insidious of these is quality.  How do you know if that really cool project you heard about is good quality design and code or is perhaps something that is going to be more trouble than it is worth in the long run?

 

For most open source projects, the quality is simply a guessing game.  There is word of mouth, but you never know just how deeply someone has looked into a project.  And what is more, perhaps they never found the bug that you will run into.

 

Perhaps the most important measure of quality for a project is the response of the community. 

  • When a bug is posted, are people assigned and are they fixing it quickly?
  • Are questions responded to in a timely fashion?

 

If the community is active and supportive, you do get a sense that even if you do find a problem, someone is going to fix it and in a timely fashion.

 

What I want to see is the principles of test driven development being fully exploited in the community.  This would involve a full set of unit tests that achieve substantial code coverage.  Of course, code coverage doesn’t tell you everything you need to know, but it does tell you something.  

 

My bottom line is that you need to know the quality of the project you want to use.  An ideal community infrastructure would measure both the quality of the testing but also the quality of the team in terms of responsiveness.

 

I don’t know of anyone who has built a community like that yet, but I would sure love to see it.

 

 

  • CPAN (http://www.cpan.org/) has done this quite well - a comprehensive library of open source perl modules that are all covered by automated unit tests. They also have integrated bug tracking and software rating.

    I'd really like to see a CPAN-quality community for a lot of other languages.
  • the two questions you list are easily answered by checking the forums for the project. It isn't at all confined to open-source projects and I would say that those questions are much more important for software that is closed and that you have to pay for.
    GotDotNet, SourceForge, etc, have forums where you can easily look and see exactly how fast questions are answered (or not). Many proprietary software makers have newsgroups and/or forums where you can test the same. I've seen many "hello? are these forums still supported" posts (often never answered and months old) in both opensource and proprietary support forums (including some Microsoft newsgroups, though they aren't exactly official support routes so they don't really count).
  • "Several problems actually, but perhaps the most insidious of these is quality. How do you know if that really cool project you heard about is good quality design and code"

    Playing devil's advocate, how do I know that closed source software is "good quality design and code"? Can I audit it myself? Can I fix it myself? Can I hire people to do either of these things?

    "or is perhaps something that is going to be more trouble than it is worth in the long run?"

    Continuing, how do I know that my commercial closed software will continue to be supported for 10, 20, 30 years?

    "For most open source projects, the quality is simply a guessing game. There is word of mouth, but you never know just how deeply someone has looked into a project. "

    Which is also true about closed source commercial software.

    "And what is more, perhaps they never found the bug that you will run into."

    True enough, but the same thing is true about closed source commercial software. Even worse, if a closed source software project decides that a bug isn't economical to fix, or a high enough priority, there's nothing consumers can do.

    Just talk to a VB6 developer who sank all that time developing Vb6 code, only to be given an entirely new platform, a mediocre conversion tool and sent on their way.

    Closed source software relies on the assumption that the vendor can be trusted. Open source software makes no such assumption.
  • Keep in mind that I am not trying to bash the quality. You bring up several important points that I agree with. What I am trying to come up with is a way in which we can honestly expose the quality of open source projects as a means to make them better.

    Keep those comments coming!
  • Are you saying that since a project exists it has quality? I hope not. We have all seen projects that are half baked ideas that never got quite finished.
  • Since you work at Microsoft, why don't you come up with a way to honestly expose the quality of MS products. Why look at open source projects when you can't even do this with your own?
  • "What I am trying to come up with is a way in which we can honestly expose the quality of open source projects as a means to make them better. "

    This gets kind of hand-wavy (and far away from the mechanics of writing code), but bear with me. I think the key insight is that open source software brings the power of the market economy to bear within a given software product, rather than bringing it to bear between software products.

    Backing up a minute, let's say that I'm trying to decide between Microsoft Windows and Apple MacOS as a platform for my corporate desktop computers. As a consumer, I have two choices: Windows or MacOS. That's binary yes/no choice is the extent of my ability to create incentives for the two vendors. Even worse, if I have an investement in Windows and the surrounding infrastructure, that makes it more difficult for me to switch to MacOS. It makes the cost of saying "to heck with windows" that much higher, and makes it more costly for me to use my dollars to communicate my dissatisfaction with Windows back to Microsoft (by switching).

    Now, let's say I'm using Linux. I have a choice between any number of vendors selling and supporting what are effectively the same product. If I'm not happy with the support I'm getting from Red Hat, I can switch to somebody else that'll be more responsive to my quality issues. If I decide that I need a particular bug fix, I can take responsibility for explicitly driving the fix rather than relying on a closed-source company's defect resolution process and my compromised ability to move my dollars elsewhere.

    My contention is that by lowering the barriers to competition as well as increasing the visibility into the process, open source creates more liquidity in the software market and makes the incentives for creating high quality software that much more effective.

    "The driving factor is the feeling of doing good and more realisitically seeing your software used by other people."

    I'm sure that's what IBM's management thinks when they look at the money they're sinking into Linux/Eclipse/etc.

    Of course, IBM is using OSS to drive other businesses. This might be the fundamental problem with the corporate-sponsored OSS model: everybody sinking money into large scale OSS development is doing it for reasons other than just to sell the software.
  • Lately, I've read several interesting blog posts and news articles on the differences between open source and traditional closed-source commercial software. To mention three firestarters, here are some articles that led to me posting this: The Luxury of Ignorance by...
Page 1 of 1 (12 items)