Wednesday, July 16, 2008

Process Herd

Process is hard. One of the coping techniques we use is to work together. Obviously, many human accomplishments could only have happened with help. Skyscrapers, the moon landing, improved health care, etc. Still, many human accomplishments could only have happened because there was only one person involved or at least one person in charge. The ceiling of the Sisteen Chapel, The theory of Relativity, etc. So what does any of this have to do with the fact that process is hard?
I believe that the only way to come up with a working process is to have one person work on it. I don't mean that the person should go off into a cave and work on it. They should seek input from others. They should seek correction from others. Still one individual should make all the decisions. I am suggesting a sort of benign dictatorship. Why is this so essential? To understand let us think about why Process is hard.
Process is hard because software development is hard. Therefore deciding how one should go about it is hard. Software development is hard like driving is hard. My daughters are just hitting that magic 16th year. As a result, I have taken the time to think about driving. One thing that I have told my daughters is that no single part of driving is really hard. OK, maybe parallel parking :-). Seriously though, driving is many many simple things. What's hard about driving is that you have to do all these things at the same time and you have to keep doing them. So while you are maintaining your lane position, you also have to be aware of the cars around you, and were you are in your route to your destination. Additionally, you need to be looking for road hazards, you also need to be looking for other drivers around you that maybe about to do something you will have to respond too. Then there is cross traffic to keep an eye on, you must be paying attention to traffic signs and signals. Finally, if you are 16 it is also important to be on the right radio station :-). All of that and you haven't made any maneuvers yet. Software development is like that. As a developer you are typing in code, that must be syntactically correct. Furthermore, it should be efficient and maintainable. It must fit within the plan for the piece of functionality you are working on. It may be useful to others so you should keep reuse in mind. You should be adding comments such that a person who has never looked at this code can figure out your intent the first time through. I could go on, but I won't. Just so I am not accused of saying that developers have it harder than anyone else in software development, lets consider QA.
Our QA Manager and I were talking about tracking the testing of our products yesterday. We came up with the following list of inputs that effect the testing effort on a project:
  • Functional Testing (Status, Planning, etc)
  • Status of Translation
  • Rate of Code Change
  • Rate Defects are being found
  • Rate Defects are being resolved
  • Rate of Functionality Change

That's not in any particular order by the way. There are six inputs (probably more) that he needs to be aware of to management the testing effort on a project. We could make similar lists for each person involved in the process (testers, documentation, development, management, and so on). The reason Process is hard is because it needs to take all of that into consideration. By the way, this is also why Process is essential. If each person stopped to think about how each task should be done while they were doing it, things would take substantially longer. If each person decided for themselves how they were going to communicate with others on the team, there would be no communication.

So Process is complicated and hard to get right. Therefore we should have many people involved in creating our Process right? wrong. Why isn't it be better to work on process as a group? Simplicity and consistency are the answers. A process should be simple and must be consistent.
Process discussions tend to attract people who really like process. You know the type, if a little process is a good thing then a lot of process must be a great thing! Of course this is not true. Process should be as simple as possible, but no simpler. One way to help keep process simple, is to have process be decided on by one person. People involved in the conversation will bring things to the process. They will likely be good ideas, but they will be more than is strictly necessary to keep the team working efficiently. Process is a minimal pursuit. It is not that anyone should be excluded, but like good writing one needs to keep removing things that are not moving you toward the goal.
Many people means compromise. Compromise as an issue that it can lead to inconsistent results. Here is an example from a real world Process dialogue I was involved in. The team had decided that they wanted to use Agile methods (Scrum in particular). They liked the idea of minimal design as they had struggled with analysis paralysis in the past. Unit testing sounded like a good idea, and refactoring was definitely something they wanted to try. As they meetings went on it was decided that the goal of 100% Unit Test coverage was way too aggressive. There was a lot of existing code after all. One group thought that minimal unit testing of "hot spot" code was the place to start. Another group felt that 60% or 70% was a good goal to begin with. A compromise was reached that a goal of 40% to 50% coverage be unit tests would be used. A cycle started about 6 weeks into the project. In this cycle, a change (read refactoring) that passed all the unit tests would leave the application unusable. So while one or two members of the team tried to isolate the issue, the rest of the team was forced to work on either an older version or rely on the unit tests only to verify the changes they were making. This cycle was happening two time a week. What went wrong?
The process the team had adopted was inconsistent. If you accept minimal up front design, then there will certainly be sweeping changes later on in the project as design flaws are discovered. In order to be able to make those sweeping changes, you must be able to test if the functionality you did not intend to change is consistent. One way to do that is to have near 100% coverage with automated tests (i.e. Unit Tests). In other words a Process that involved minimal up front design, must include refactoring and unit testing to be consistent. The process the team had settled on did not contain all of the elements so it was in consistent.
So Process is hard, but it is still best designed by one person. In order for it to be effective it must be adopted by all. It should be simple, or as simple as it can be. It should be minimal. It must be consistent. The herd needs a shepherd. We need a guide so we can focus on what we are trying to accomplish. We need process so we can be efficient and stay sane.

No comments: