CUSEC 2010 Keynote: Douglas Crockford – The Software Crisis

At long last, the fifth in my series of notes taken from keynotes at CUSEC 2010, the 2010 edition of the Canadian University Software Engineering Conference. These are from Douglas “JavaScript: The Good Parts” Crockford’s keynote in which he talks about how hard it is to make software. Doug is a senior JavaScript architect at Yahoo!, the guy behind JSON, wrote one of the first important pieces on JavaScript’s unrealized power and utility and in a former life wrote videogames (of which he has some interesting stories).

My notes from his keynote appear below.

The Software Crisis

"I made the first important discovery of the 21st century: that JavaScript has good parts!"

• The Software Crisis dominated the computer industry for a decade: 1960s
• The crisis was that software projects were becoming too complex to manage
• Projects were:
  • Over budget
  • Late
  • Unreliable
  • Not fully meeting their requirements
  • Unmaintainable
  • Failing
  • Insecure (this is a relatively new thing)
• The software crisis isn't over -- we're now in year 45
• It never went away, we just stopped talking about it
• "In other news, the sun will eventually exhaust its supply of hydrogen"

Craft vs. Engineering

• Some people said that we weren't bringing enough engineering discipline to the practice of programming
• Some said programming isn't the same thing as engineering and therefore engineering discipline doesn't apply
• Both miss the point
• The real problem with software is that it's the most complex thing we make
• It requires perfection, which humans aren't good at
• It's amazing we can make software at all

Software construction

• Computer science has taught us lots of things, but not how to manage software projects
• In some ways, building software is like any other sort of construction
• But in some other ways, it's quite different
• You have to keep in mind that "construction" when applied to software is just a poetic metaphor
• Take the case of building a wall:
  • It's easy to estimate, based on the size of the wall how much time it will take, and also how far progress will be at any given time during its construction.
  • Try doing that with software!
• Part of the problem is the nature of software
  • It's both powerful and malleable
  • The qualities are both a blessing and a curse
• One problem: The lack of metrics
  • There's no objective measure of quality or completeness
  • We can test software, but the tests are on the same level of quality as the software being tested
  • Consider one of the old stand-by metrics: LOC (lines of code)
    • It's not an indicator of quality
    • Nor is it an indicator of completeness
• Another problem: Estimation is difficult
  • Programmers, by nature, are optimists
  • You have to be and optimist to do this kind of work! You wouldn’t be able if you weren't
  • "I think rational people could not do programming"
  • As a result, we're probably the least qualified people to do estimating
  • What doesn't help is encouragement from management like "You're a smart guy, you should be able to do it in 3 months!"
• Another problem: Programmers don't understand how they spend their time
  • They think they spend most of their time typing
  • Measurements show that most of their time is actually spent in meetings or tech conversations
  • (And there's also the time spent staring at the screen going "My God, what have I done?")
  • As a result of this disconnect between perception and reality, programmers tend to be skeptical of process improvements that might require more keystrokes
    • [Story about developer who refuse to switch from a CamelCase naming scheme to a words_delimited_with_underscores naming scheme]
    • The programmer had an emotional attachment to the CamelCase naming scheme and argued that the underscore-based scheme slowed things down
  • Consider the first rule of optimization: optimize the process that is taking the most time
  • When it comes to programming, that process isn't the typing!
  • It's the thinking, worrying and puzzling
• Programming is a social activity
  • Solo cases are the exception
• Cost of innovation
  • Doing what's been done before vs. doing what hasn't
  • We're constantly doing things we haven't done before
  • That's most likely to lead us to mistakes
• Legacy
  • In other industries, legacy is seen as the wealth of practice and tradition, the wisdom of experience
  • Not ours! In software, past accomplishments are considered to be a liability
  • The age at which programs become "legacy" is getting younger and younger
  • Some programs become legacy before they're finished\
  • No industry disrespects legacy the way the software industry has

Leaps

• Software is not governed by Moore's Law; it's governed by Murphy's Law
• Leaps tend to come at 20 year intervals, not two year intervals
• Leaps make it possible to implement more complex projects
• Adoption of "leaping" software technologies tends to be very slow
• Common responses to leaps: controversy and reluctance
• Leap 1: Plugboards

• Single machine programmable with plugboard
• The original spaghetti code
• Leap 2: Symbolic assembly language
  • The first software tool -- the first program to make it easier to write programs
• Leap 3: High level languages
  • In the 1960s, there was a lot of research on automatic programming
  • It was believed, even back then, that programming too complex and error-prone for humans to do
  • We wanted to get computers to write programs for us: we’d feed them specifications; they’d output programs
  • The result? FORTRAN
  • Still a leap forward
  • Even today, with DSLs, it's still progamming
• Leap 4: Structured Programming
  • Getting rid of GOTO
  • Let us write more complex programs
  • Argued for 20 years as to whether it was good idea to get rid of GOTOs
  • People who benefited most and opposed were programmers
  • Arguments and counter-arguments
  • "Had to wait for those guys to get old and die"
  • So today, virtually all our languages are GOTO-less
  • Good thing -- more sophisticated programs possible
  • What was the fuss all about?
• Leap 5: Object-Oriented programming
  • 1967: Simula released
  • 1972: Alan Kay begins Smalltalk, which was originally intended for children
  • Most of good today in tech owes a debt to Smalltalk
  • 1980: Smalltalk released. Smalltalk took 8 years -- Netscape spent 8 weeks getting JavaScript ready!
  • 1985: C++ released. Essentially some of the good ideas in Smalltalk added to C
  • 1995: Java released
  • 2004: PHP with version 5 says "Okay, we'll do it that way too"
  • A lot of time has passed between 1967 and now. Simply because a new good idea appears means it won't be adopted quickly
  • OOP is not the last leap. It's just that we're so fixed on objects that we can't see past them
• So we're late for the next leap
  • I think it'll be distributed programming, the idea of taking n components and put them together to make a functioning application
  • Will live in the cloud
  • Nobody thought that we'd see signs from the next step forward in JavaScript and browsers, but that's where it'll likely come from
• Some Failed Leaps
  • Artificial intelligence
    • Intelligence is much harder to understand
  • 5th generation languages
    • Japanese invested heavily it, but without much result
  • Computer-aided software engineering
    • Reborn in IDEs
  • Subjective-oriented programming, etc.

Bugs

• Software does not have enough self-awareness to be afraid of bugs. That's why it works as well as it does.
• The term “bug”  is attributed to Edison, who used them to refer to flaws
  • From the Pall Mall Gazette, March 11, 1889: "Mr. Edison, I was informed, had been up the two previous nights discovering a bug in his phonograph."
• Ever since then, defects in inventions have been called bugs and it's become part of popular culture
• There's also Commodore Grace Hopper's bug, a literal bug: a moth caught in the relay of a Navy electromechanical computer.
  • She pasted moth into her logbook
  • It's the "first actual case of a bug being found"

Snake oil

• Even though we know better, we love "silver bullets"
• We tend to like people who make fantastic promises to fix our problems, if we just follow some crazy methodology of theirs:
  • Diagramming?
  • Documentation style?
  • Have all your meetings standing up?
• Saying "nothing works" is discouraging; I'd like to give you a better story
• The Mythical Man-Month
  • Written by Fredrick Brooks; I highly recommend this book
  • He said a lot of things in the book that seemed unreal at the time, but they were proven to be true over time
  • The source of what we know as Brooks' Law: "Adding manpower to a late software project makes it later."
  • One of the first books to show an understanding that programming is social
  • Another idea in the book: the Second-System Effect
    • Take a team of programmers who've done something brilliant
    • Give them a blank slate for the next project
    • The result? They fail...hard
    • Why? They become overconfident. Their earlier success causes them to overreach
  • Another idea in the book: Prototyping
    • Software is like waffles: Make the system, throw it away, make it again
    • Counter to the notion of getting it right the first time)
    • "Make the first one to throw away" turns out to be a good idea
  • One last concept from the book: "A project become a year late one day at a time"
• Literate Programming
  • Created by Donald Knuth
  • The idea: Programs designed to be read
  • I've seen lots of good programs done in this style
  • It doesn't appear to work well in groups yet -- but that's mostly what we're doing now
• Incrementalism
  • The basic idea: "We'll see you in two weeks" vs. "See you in two years"
  • Disappointments are smaller and more frequent, which are easier to manage; it lets us manage our expectations
  • It also makes us less ambitious, but that lack of ambition is helpful
  • It avoids second system problem -- this doesn't let you dream big
  • The project is always a beta, and perpetually unfinished
  • The downside of incrementalism:
  • There's the potential that your project will become something like the Winchester House in San Jose
    • Built by Mrs. Winchester, wife of the Winchester behind Winchester guns
    • Her husband and child died tragically, and a medium told her that they were killed by the spirits of people killed by the Winchester repeating rifle
    • She was warned that they'd come after her too unless she found some way to confuse them
    • She came up with the idea that as long as she kept working on her house, the spirits would not find her
    • So her house was constantly under construction, with hired people adding new things to it all the time
    • It was built up incrementally. "This is an agile house!"
    • End result: things like stairways leading up to a ceiling:

• "You're programmers, you know what happened here!”
• Chimneys that didn't go up all the way to the ceiling

• No rhyme nor reason
• Go watch Mr. Blandings Builds His Dream House (1948)
  • It's the best movie ever made about project management
  • Blandings' problems:
    • Lack of knowledge of technology
    • Poor control over requirements
  • [Joey’s note: This movie’s been remade in modernized form with Tom Hanks in The Money Pit(1986) and Ice Cube in Are We Done Yet? (2007).]
• Feature Cost
  • Development time
  • Deployment cost
  • Maintenance cost (bloat, cruft)
  • Download time
  • User confusion/training
  • Bug delivery
• Code Value
  • Microview: good coding conventions
  • Macroview: good architecture
• We need better program architecture
• Architecture is hard, but conventions is easy
• Think of how hospitals were improved just through simple conventions of hygiene, such as washing your hands
  • The simplest thing we can do to increase the value of our code is to make it readable
    • For example, when returning a value from a method, we should use return value, not return(value) -- the latter implies that return is a function
  • Whatever improves human parsing of programs is helpful
• Communicating
  • You should think of programs as a medium of intentional communication:
    • Communicating detailed instructions with the machine
    • Communicating with your development community
    • Communicating with yourself (and especially your future self)
• Good architecture is about not collapsing into a puddle of confusion, and should enable you to change a correct program into another correct program
• Cruft
  • It's software scar tissue
  • Comes from lots of sources:
    • Premature optimization
    • Inexperience
    • Misread source
    • Feature enhancement
    • Death marches (which are the most avoidable source of cruft)
    • As cruft accumulates, complexity grows and progress slows
• Eventually, the code itself becomes a source of friction

Bloat

• It's "software obesity"
• In the old days, memory was expensive and you didn't get much of it, so your programs were lean by necessity
• Having only 8K - 64K of RAM in a machine put incredible constraints on what you could do
• I used to dream of the day when memory would cheap and abundant enough so that we wouldn't have to do evil things to get programs to fit
• Now I wonder if maybe those constraints weren't so bad
• "Bugs hide in those rolls of fat in our programs"
• Insecurity
  • Most programs these days have marginal security
  • Programs that once didn't have security implications now do
  • Any bug is potentially a security bug
  • Good secure programming is good programming
  • Thinking about security might make programming easier
• Refactoring
  • The process of code refinement, involving things like:
    • Correction of formatting
    • Insertion of documentation
    • Removal of cruft and bloat
    • Restructuring
• Perhaps we should take advice from Exodus: "Plant and harvest crops for six years, but let the land rest and lie fallow during the seventh year."
• Maybe do six sprints where you add new features, but on the seventh sprint, don't add new features at all
• Sometimes the best course is to start over
• This is quite difficult to accept
• The pain of the crash
• Loss of a week's work, a day's work, an hour's work, is unbearable
• The illusion of completion
  • "It's in the can!"
  • I once worked as an expert witness on conflict between 2 game companies
    • The original game company had split up; the new company was being sued for plagiarizing the original company's game engine
    • I reviewed both game engines and determined that the new game company had written a whole new engine in 2 month -- they'd learned for the experiences writing an engine for the old company
    • The owner of the old company couldn't accept that; he thought the value of the codebase was 100 man-years, not 1
    • An experienced team can redo what they just created quickly. The team's focus must be on simplicity to avoid second system effect
• Have regular code readings
  • Don't wait until release to do code reviews
  • Do team code reading regularly during development
  • Problems can be discovered early
  • Good techniques can be shared early
  • Experienced developers can lead by example; novice developers learn from the group
  • They should be: frequent, helpful and respectful
    • You need a healthy team - a dysfunctional one will tear itself apart

Conclusion

• Quality first. Quality last.
• Readable code. Code reading.
• Invest in the quality of your code base.
• Invest in your programmers and managers.
• Quality is the most important feature.
• If you look at all the polished, completed code you write at the end of a year, you could type it all in a day
• Typing is not what we do; it's thinking and communicating

[This article also appears in Global Nerdy.]