There are a lot of software wheels that get re-invented all the time. Re-inventing is a time consuming waste that many projects can cut. In almost all cases it’s much better to do some due diligence and use an off the shelf component where you can. Invest your time, passion and expertise in your actual product.

Save your energy for your real innovations

Engineers are smart. Sometimes we get cocky and think because we know how to build the best X on the web, we also know better about everything else too. Don’t fall into this trap. It’s very unlikely a genius web developer can really create a better database than off the shelf. There are too many good ideas begging to be programmed. Don’t waste time on something that’s outside your core innovation. The world needs customer focused applications that work well with people. Don’t waste your time creating the ten thousandth backend widget you could have gotten for free on the internet.

Existing protocols are good

Nothing makes me crazier than software that implements a broken version of a standard protocol. You can spend time designing a brand new protocol, but if you are doing something that is well known on the ‘net you should try hard to work with existing protocols. Existing protocols have some miles on them and are usually good compromises for everything they do. Designing and testing new protocols is just quicksand for most projects. If you can’t say that the new protocol is the heart of what’s cool about your product, then don’t make a new protocol.

Get your design patterns off the shelf

Most software project fall into some pretty well known patterns. Console applications, three tier web applications, client-server applications and peer to peer applications are very common patterns. If your product falls into any well known pattern, follow the best practices for your pattern. Innovate where you are really on the cutting edge and try to use cookies cutters wherever else you can.

Good planning is critical to delivering software. Anyone who has shipped a product knows that the plans we start with rarely survive the entire release process intact. Too many of the plans we start with are useless by the time we ship our software. Ideally our plans would serve us better through the entire process. Here are some lessons about planning I have learned that should help you spend you planning time better.

Good business starts with a vision

A clear vision will do more to help you ship software that delights your customers than any other planning you do. This is very important so I will repeat it. You must have a clear vision and communicate it clearly to your team. Teams who can’t do this have no way to live up to their potential. Your vision must unify your marketing goals with your engineering goals and it must be realistic and achievable.

Good products flow from a vision. You can have the most technically sophisticated components known to man, but without a unifying vision they are doomed. This part of your planning is really, really important. If you neglect it, I promise you will cause suffering and randomization in your team about midway through the cycle!

Define one or two primary pillars and two or three secondary pillars.

The pillars of your product should be a rough guide to where you will spend your time downstream. One approach to software pillars is to imagine each attribute your software can have is a dial from one to ten. You can set each dial at a number and then arrive at the probably amount of time it will take to achieve your vision. The trick is that you have a limited budget. Most projects can only survive having one or two pillars be set to “10” and two or three more set above “5”.

How can you decide what pillars are important? Talk to your customers. Find out what they really want. They will want everything, of course. Find a way to make them stack rank what they want, and proceed from there.

Some sample pillars you might work with:

·         Feature rich

·         Usable

·         Robust

·         Scalable

·         Maintainable

·         Testable

·         Technically advanced

·         Performing

·         First to market

A lot of software projects pay lip service to pillars, but fail to really nail down the vision for each one. If you want to create a product that scores a 10 on each one of those scales, you will have to pay the price in time, talent and your resale cost is going to be high to compensate.

The best software projects start from a vision that is specific about what pillars are important, which ones are secondary and which ones are going to be “left for the next version.”

Some pillars are diametrically opposed to others. If you want to score a ten out of ten on “first to market” you can pretty much forget about scoring very high in the other pillars.

Make sure before you start to design features that you know what you want the software to achieve in the market. Make this vision clear and public for the team. Don’t use weasel words in your vision statement. When conflicting priorities present themselves, a clear vision will make for smooth sailing. For example, if a feature is causing the ship date to slip, a clear vision will allow you to quickly decide if you should slip or cut the feature.

Finally it’s important to say what pillars aren’t part of this release. If performance isn’t a primary consideration, say so in the vision. Otherwise you will find your team spending a lot of time trying to performance tune a product that shouldn’t be tuned in this release. You can’t have everything in one release. So prioritize and learn to let go of everything else.

When you vision is clear and clearly communicated, your team is set up to succeed.

If your feature crew are struggling ask if the support systems are in place. Can they get builds and run automated tests at will? If not, no amount of planning is going to help them when they discover something nobody thought of. Is it clear how important the feature is to the project, how well it must work and how fast? If not, you will have resource contention issues that will drain valuable resources from your team.

No plan survives contact with the enemy

I worked on a team that had a pretty good process for creating software. They sketched out some pillars, whipped up some prototypes, tried out the combination and then did some mid course corrections. However, parts of the process could have been optimized. The problem was that the plan on paper didn’t match the plan in reality. So anytime you wanted to change the process you got stuck with the fact that no one was following the “real” plan in the first place.

If your actual process doesn’t follow the script it’s likely the script is flawed. This is probably a very healthy occurrence. Try to capitalize on the evolutionary nature of your organization. It will adapt to overcome challenges. Don’t make a rigid insistence on outdated plans one of those challenges.

Today I wanted to export a hyper-v machine to a share on a computer with more disk space. However I got this error: 'General access denied error' (0x80070005). I didn't have enough disk space to just make it locally and transfer it over. I really should get another disk for that system, but not today.

I quickly realized the problem. Hyper-v doesn't try to export with my permissions. It uses the permissions the service started under. I figured out two work arounds. Both are probably security issues, so tread with caution and common sense. Either method could allow unauthorized users and programs to write to the host you do the steps on. The second method at least contains this to one directory. Don't trust any executables that show up in that share. You should probably also remove the permissions when your task is done.

First, and untested, change the service accounts that hyper-v runs under to a domain account that has permissions to the share in question. The restart hyper-v. That will probably work. However the amount of damage that can happen if something goes wrong is unpredictable.

Second, add the MACHINE itself to the share permissions on the desination. You can do this the exact same way you would add a user (Click the advanced sharing button in Vista). However, in the permissions dialog box you have to click "Object Types" and check "Computers" in order to be able to see the computers.

From the Object Types screen click "Ok" then enter the name of the computer you want to give the permissions to. Then check "full control".

Once you hit OK on all the layers the machine itself will have permissions to write to the share.

A situation came up this week with two different testers in my team. They both asked about the approach they were taking to a problem. Both of them were doing a lot of testing and they were trying to decide on a depth-first or breadth first approach.

In the end they ended up taking different approaches to similar problems. However, both approaches minimized randomization.

Minimize context switches

A typical test cycle on a feature involves a lot of thing. At some point you generate test cases, run them and report results. In most cases you need to figure out the approach that will make you most productive. Designing one test case, running it, dissecting the results and then reporting that one result before moving on to the next one isn’t an ideal way to tackle testing. You will be switching applications and thought processes very frequently. These context switches are very expensive in terms of easy things like mouse miles and clicks. They are also very disruptive to your brains natural process. You are preventing yourself from developing a rhythm or identifying the big picture.

Group similar tasks

Henry Ford figured out that an assembly line boosts productivity for cars. A lot of the tasks we do in test can be assembly lined. For example, you need to create test cases in the test case repository.

You could take a depth first approach, where you complete an entire test case before you move on to the next. In most cases you can do better. For example, you might use a tool to input all the test cases titles in one shot. I have even used some hacked together scripts to generate test case titles for predicable parts of a project. Being able to upload hundreds of test case titles in a few seconds is really rewarding.

The important thing is to identify task groupings and try to get all the tasks at a certain level done as a group.

Do the easy stuff first

Once you start running the tests you might find out that some of the test cases aren’t easy to run or verify. You can easily get hung up trying to debug, diagnose or improve these trouble children. Don’t do it. You will be paying a time tax to context switch. If something breaks the pattern, kick it on to another pile and keep moving.

It’s much better to finish 80% of your test cases on Monday and then spend the rest of the week diving deep into the 20% that gave you trouble. Your boss has a much better idea of the health of the software. Your developers have bugs you already found to work on. Best of all, if you run out of time, you did a significant fraction of the work.

Bug investigations fall in the same bucket. It might be best to log a lot of bugs that don’t have all the information you ultimately want in the bug. It’s much better to log 20 bugs a day that are missing details than log one bug a day with all the possible details. Be sure to note in the bug that you plan to come back later and provide details, if that’s your plan. During a test pass you want to put emphasis on finding bugs. When the test pass is over you can shift to fixing bugs.

Whenever you run into problems, ask if you can safely postpone a deep dive in order to knock out the easy stuff. Many times you can do just that and it will save you a ton of time.

Another example of this is when you have people helping you out. Say you just hired a tester to help out in the area you are an expert. You could make them do everything you do and let them call or talk to you anytime they have issues. It’s a lot more productive to have them do everything they can do by themselves and make a list of all the questions or problems they have and deal with them all at once. This same trick works with your boss when he or she “pitches in” in your area.

Circle back for the hard stuff at the end.

Of course you don’t want to neglect the tricky problems. You do need to address them. Just don’t let them distract you and cause a lot of context switching through the entire endeavor. In fact, you will probably benefit from having a rhythm with the deep dives at the end as well.

Do it better

Everyone in test gets distracted by trivia at one time or another. If you pay attention to what you are doing, you can catch yourself “rat-holing” and get back on track. Things get hard. We get behind schedule. Sometimes “pulling it in” is just a matter of re-arranging the order of our work.

Untested software has bugs. As we test software we find a lot of different kind of bugs. We want to identify the root causes for as many bugs as we can. Finding root causes can be fiendishly difficult and time consuming. Using a systematic, scientific approach can help you with the hard problems.

Say you have a program that used to work, but now it crashes. You look into it and realize it’s been broken for quite some time and fell through the holes in your testing. Now you have several related check ins and dozens of others that have a remote possibility to be the culprit.

Stop poking at problems with that stick

When we find problems in software we often have a good idea what the problem might be. We do some informal checking and poking around. Most of the time, this behavior pays off. Our experience and intuition are valuable and we get to the heart of the problem without a lot of thrashing. The problem comes when this approach fails us. All too often we keep blindly poking with a stick and end up thrashing around and not finding the answer.

A good rule of thumb is to switch to a more rigorous approach when your guesswork has failed three or more times. In our example you might guess it was the latest check-in and back that out. You find out that is no good and the program still crashes. You check the dependencies and they don’t reveal anything. Next you run the test on some other machines and find it’s a universal problem. At this point you should make a solid plan for when to stop guessing and when to switch to a more controlled approach.

Use the scientific approach

At its heart science is testing your beliefs. When you were guessing about the source of the problem you were doing this in a relaxed way. Now you realize that nothing obvious is wrong and you will need to do some deeper technical work. You need to use the scientific method. For software testing there are really just three key things to keep in mind.

Control the variables

You need to make sure you are changing things one at a time while you run your tests. You might get the program to work after you re-installed the operating system on the host, changed some code and rebuilt the database. What you won’t know is what fixed the problem and if it’s gone for good. When you switch to a rigorous method, you have to be careful and control the variables one at a time. It’s easy to fall into the trap of changing lots of stuff and trying again. You might get the program working again quicker, but you lose too much valuable diagnostic data. In lab environments this is especially tempting. You just need the environment “up” to complete your work. The problem is you don’t know if it was a trivial error you will never see again, or if you just put a tarp over a deep pit.

You should be taking notes on the variables and how you change them at this point. You don’t have to exhaustively note every little thing, but you should be able to back out any changes you make. In code this means having a checkpoint. In an installation this may mean a snapshot image. Maybe it’s just a list of values you change in the database. Just be sure you can unwind the stack if you have too.

Create a hypothesis

Once you have control of the variables you need to guess again about what is wrong. But you need to do it in a controlled way. State your idea in a way that you can prove wrong. It’s a good idea to write your hypothesis down.

In the example of the crashing program, we might hypothesize anything from a corrupt pointer assignment to something outrageous like “it won’t work on Tuesdays when the moon is full.” The most important thing is that your hypothesis is narrow enough to be tested. Often it’s a really good idea to make a list of things you don’t think are wrong and test them first. A good example might be “The file was corrupted at install time.” We don’t really think this is causing the program to crash, since we have several versions of the installer and it’s unlikely they would all be corrupt. However its possible and until we clear it as a culprit it could be masking other problems.

Try to prove it wrong

Once you have a hypothesis do an experiment. You need to design your experiment so that it will prove your hypothesis wrong. You also need to design it so that you change the fewest variables possible to prove your hypothesis wrong.

In our example we could diff the file with the one on the build server. If they are identical, we know that the corruption hypothesis was wrong. Scratch it off the list and move on to another one.

Rinse and repeat.

Jump start the car before you take the engine apart

The problem with this approach is the sheer number of hypothesis there are. Defining them all narrowly and then designing an experiment to test them will be very time consuming. You need to use an approach that weights getting results quickly. An analogy is a car that won’t start. Nearly any part in the car could be bad. Based on your history with the car you might have some guesses. While you are going through all the possibilities you might as well try to jump start the car. It’s easy to do, and only take a few minutes. If the car starts, wonderful, you are done. If not, you quickly ruled out a very common reason for cars not starting (dead battery).

Rank your hypotheses to get most coverage

When you start to think about your hypotheses rank them by how hard they are to do and how much information they give you. Start with the easiest experiments that give you the most information.  Every test you do gives you data. All of that data can help you decide what tests you should focus on.

In the example of the crashing program, you might have one hypothesis that a particular line of code is at fault and another that the network connection on the host machine is at fault. An experienced network engineer can rule out 99% of network problems with a few simple tests. If that’s quicker and easier than tweaking the line of code, start there. On the other hand, you may have the code open in an editor and building and installing may be trivial. In that case you can test your “line of bad code” hypothesis much easier than a network problem.

Stick with the science

It’s tempting to go back to poking your problems with a stick. You rule out a few major problems and you start exploring at random again. This is rarely productive. If you find yourself changing lots of variables at once, you are probably wasting time. If you are stuck for more experiments, then you can explore some to get more data. But if you have plenty of possible culprits, keep working through the list. It’s just like running any other set of test cases.

Trouble, right here in software city! 

That’s trouble with a Capital T. That rhymes with P and that stand for pool! Wait, pool is the “trouble in river city”. In software city the real problem can be what I call the “Think System.” In case you never caught a rendition of “The Music Man” (by Meredith Wilson, opened in 1957 and ran for 1375 shows) the “Think System” is what the con man uses to "teach" music and disguise his ignorance. In software we run into the think system all the time. When we tell our teams to “Think Agile!”, “Think Quality!”, “Think Security!” or “Think Idea of The Moment!” we can cause Trouble (with a capital T).

In my last post I described some of the reasons that make changing your process hard in organizations. Now that you know how to avoid the pitfals, how do you get past use a "Think System"? Luckily there are good ways to build lasting change in your organization. Here are some experiences I have had that should help guide you.

Don’t fall prey to a Think System

Management and even the worker bees (IC’s in Microsoft Speak) sometimes get really excited about a concept. Sometimes these concepts are just the current fad and don’t amount to much, sometimes they are vital to the next level of software evolution. You usually have to try them on to see. Change isn’t easy, and sometimes group handicap themselves more than they really need to. Usually this is when we tell everyone to “Think Idea X” but don’t do a good job communicating what the idea really means.

Intuition can be flawed

In software construction the obvious path can sometimes lead over the cliff. I see teams get fired up with ideas. This is great, but when there is no substance behind them we get into trouble. We can all take off in ten directions thinking we are pursuing the grand vision. Getting everyone back together can be like herding cats. Worse, we can all nod and smile then run off the cliff like a heard of lemmings.

I have seen software projects go off the rails in both ways. I worked in a company where my boss believed wanted to have a superior lab. His think system was “Think Super Lab”. There was one camp that believed that a neat lab with completely documented processes was the key to “Super Lab.” There was another camp that believed that functionality was paramount and neatness was a nicety best left for later. This was a giant disaster. Everyone was pursuing a different version of the vision. There were pitched battles and end the end, the lab was neither functional nor neat. Each camp had an intuitive approach to having a “Super Lab”, but both approaches were flawed.

I have also worked on products where everyone was heading the same direction. Sadly, like the lemmings, we were headed over a cliff. It was a more pleasant experience in some ways. We all knew we were doomed, but nobody would rock the boat. Our instinct to go along with the herd meant the product launch was a lot like a train wreck. Don’t let this be you.

Ask what success looks like

The acid test for a “think” system is the lack of a visible or knowable finish line. Engineering is a place where we should be able to use facts to drive our actions. All too often we merely think and feel we are doing a good job and don’t have a way to prove it. Some bosses I really respect constantly ask the question “How do we know we are winning?” They draw the finish line on the track and point it out clearly to everyone. Metrics can let you down, but that doesn’t mean you should never use them. There is a place for instinct in business, but software construction that is based purely on instinct is doomed.

Two way communication is key

Knowing what success is will help you make course corrections along the way. Make sure your message is being heard. Make double sure that you are hearing the messages coming up from the guts of the project. Not only do you need status reports, but you need to know the people giving them understand what you are asking of them. It’s easy to drown a team in busywork and generate a boatload of metrics. You get better information from people who understand what the metrics mean and how they drive the business. When you are trying to change your culture with a new idea it’s really important to get timely feedback and make sure the team knows about it. I have seen Technical Support operations where the wrong metrics were being given as feedback to the team. Things like average call time; hold wait times and call abandonment were put up on a big board. The person with the shortest calls and greatest volume got cash rewards every week. The problem was customer satisfaction was in the toilet. Management was very upset and started firing “poor performers” with “bad numbers”. Customer satisfaction went even lower. The problem was with communication. Management was looking at one set of numbers (customer satisfaction) and showing the workers another set of numbers. The communication was very one sided. The more management pushed, the more people who were being nice to customers left or were fired. The only people left were the ones good at keeping calls short.

We make the same mistakes in software all the time. We want high quality software and then we publish a leader board with the highest number of bugs logged. This leads to cherry picking and avoiding diagnosing complex problems. The testers get in a race, and the product quality suffers.

Be clear in both directions with what you are trying to accomplish and how you plan to get there. Don’t assume people will automatically understand. Double check your assumptions. Listen closely to critical feedback. When I worked I help desk I learned that every call had a lesson for the company. A way to do things better was there even if the person on the other end just calls and curses and yells.

Build frameworks to guide you

You can break all the rules you want. You just have to know the rules first, and then have a damn good reason to break them. Make sure your team has a framework to build from. A little guidance goes a long way to keeping everyone on the same page, productive and happy.

Build from a solid foundation

It’s important your framework be on a solid foundation. Time after time I have seen frameworks built on the quicksand of “best case scenario” or be paralyzed by “worst case scenario.” You also need to make sure your house is basically in order before you charge off in a bold new direction. If your team doesn’t have the basics down, make sure you get up to speed before you try to swallow a new idea, practice or work plan.

Be honest about what isn’t nailed down

Sometimes you need to put some stakes in the ground for planning purposes. Make sure your team understands what the hard assumptions are and where you are just sketching. This comes back to communication of course. You need a framework, but you don’t need to have the whole thing nailed down before you start to work.

Use built in flexibility

Make sure your framework isn’t too rigid. There is a military saying “No plan survives contact with the enemy.” It’s the same in software. The very best plans are ones that have built in flexibility. They often take very small bites and then have decision points where you can act on that flexibility. This brings me to my next section.

Iterate on your ideas

People will tend to fall back into old habits. In order to institute something new you have to keep at it. Human beings need fairly constant feedback in order to change a habit.

Do pilot programs

A good way to start a new program is to try it on a small scale. Set aside a small portion of the team and give them the new rules. Give them permission to fail. Let them complete a cycle and report back on what went well and what went poorly. You can learn a lot of valuable lessons on a very small scale. A pilot program gives you a chance to test and fine tune your cool new ideas. Don’t pass up using such a powerful tool if you can help it.

Take small bites

Once you get the feedback from your pilot project find a way to implement it in short phases in your team. If you aren’t getting feedback about how you are doing at least once a month you are in trouble. A week long feedback loop is probably close to ideal. Don’t bite off more than you can chew. Once your team gets used to the new system (it takes six weeks or more to form a new habit) you can ease off on the feedback loop frequency. Make sure everyone still knows what success is and can self measure if you do cut back on the feedback. Taking small bites also means not trying to change too much as once. In sports like golf of tennis players often work on their swing. They will take it apart and fix one aspect at a time. A side effect of perfecting your backswing is usually that your whole game goes downhill while you are learning. This is normal and will happen whenever you change something at work too! Factor in time and resources to “take the hit” for making improvements. You will see a short term decline for long term results.

Be retrospective often

Plan in time to ask and effective answer how you are doing. This should be in your framework. Be critical and ask if the changes you are making are having the desired effect. Remember that things may go downhill for a while until people can “put their game back together.” Make sure you are being focused and you take advantage of the flexibility in your framework. Ask what’s going well, and what’s not. Make small course corrections. If you make a small correction once a week, you will get your team heading the right direction without overshooting. If you overcompensate every week, you will never get on course and you will add a lot of chaos to your work.

Don’t just Think. Do.

The Think System is a sham. Make sure you don’t fall for it. Software teams do it all the time. They say they are going to be “agile” or “quality focused” or “security champions” but they don’t have a solid plan to backup their slogan. When you want to change the way you work, make sure you have a good transition plan.

Don’t fall for the flim-flam man!

Software is a completive business. It’s not unlike a race. No runner who seriously wanted to win would enter a race with a 30 pound pack. In order to be a competitive athletes (or businesses) need to make sure they have as much gear as they need and no more. The ideal amount of equipment like running shoes and water bottles is something that comes with experience. Don’t make the mistake of tossing out things you really need, like your blood, in order to lighten the load.

Race to Win

In any competitive landscape it’s important to keep your eye on the prize. In software, that means having a strong customer focus. The problem comes when you get myopic about this focus. Just because the customer won’t directly touch, see or hear it doesn’t mean it’s not important or even vital. There are two things in software that are vital but still tempting to toss out. Just like the runner, they are really the life’s blood of the engineer effort and removing them will kill out chances of winning. The first is testing and the second is your automated infrastructure.

I have talked before about building your test automation on bedrock. You simply must have the things on that list up and running or you will be spending more time compensating than it would take to get them running in the first place.

Training and eating right

Athletes spend a lot of time training and focus on their diets. There is a cycle to the training as they approach a competition. Good software development teams also follow a cycle. They start with getting the engineering of their work process working well. All too often software teams think they can skip this stage. They think that the “getting in shape” phase of training isn’t needed. They should be able to jump in with both feet. Just like a marathon runner who decides the best way to train is to just start racing, they run out of steam early in the process. It’s not really possible to catch up. This is the point when teams start to try crazy things. Thinking they need to be lighter to catch up, they will do desperate things. Just like our runner friend who thinks letting out a couple quarts of blood with lighten them up, their efforts usually do more harm than good. Sometimes you have to start slow now to go fast later. Don’t enter a race with no training the night before.

Don’t be that team

You really want to be customer focused, so don’t stop with asking what the customer wants to touch, see and hear. Ask how well they want your product to run. Ask how deeply they expect it tested. Ask how smoothly they expect your process to run. Winning a race starts long before you wake up on race day. You don’t want to saddle yourself with needless bureaucracy. You don’t want to try and run a race without preparing or bringing vital equipment like your shoes. The worst teams end up entering the race with a 30 pound pack and no shoes. You don’t have to work hard to beat those guys. But if you want to run your best race and delight your customers, you have to find the balance and enter the race strong. Your strongest competitors already know this, you have to know it too.

The business bottom line

In order to run a strong race, you need to budget some effort to the “meta-work.” You have to accept the fact that 10% of your effort will be tithed to making your process powerful, lean and agile. Keeping all your best players and resources 100% focused on customer facing features is a recipe for disaster. Spending the time and money to make sure your tools are world class will go a long way towards making your product world class.