Process Herd

Process is hard. One of the coping techniques we use is to work together. Obviously, many human accomplishments could only have happened with help. Skyscrapers, the moon landing, improved health care, etc. Still, many human accomplishments could only have happened because there was only one person involved or at least one person in charge. The ceiling of the Sisteen Chapel, The theory of Relativity, etc. So what does any of this have to do with the fact that process is hard?
I believe that the only way to come up with a working process is to have one person work on it. I don't mean that the person should go off into a cave and work on it. They should seek input from others. They should seek correction from others. Still one individual should make all the decisions. I am suggesting a sort of benign dictatorship. Why is this so essential? To understand let us think about why Process is hard.
Process is hard because software development is hard. Therefore deciding how one should go about it is hard. Software development is hard like driving is hard. My daughters are just hitting that magic 16th year. As a result, I have taken the time to think about driving. One thing that I have told my daughters is that no single part of driving is really hard. OK, maybe parallel parking :-). Seriously though, driving is many many simple things. What's hard about driving is that you have to do all these things at the same time and you have to keep doing them. So while you are maintaining your lane position, you also have to be aware of the cars around you, and were you are in your route to your destination. Additionally, you need to be looking for road hazards, you also need to be looking for other drivers around you that maybe about to do something you will have to respond too. Then there is cross traffic to keep an eye on, you must be paying attention to traffic signs and signals. Finally, if you are 16 it is also important to be on the right radio station :-). All of that and you haven't made any maneuvers yet. Software development is like that. As a developer you are typing in code, that must be syntactically correct. Furthermore, it should be efficient and maintainable. It must fit within the plan for the piece of functionality you are working on. It may be useful to others so you should keep reuse in mind. You should be adding comments such that a person who has never looked at this code can figure out your intent the first time through. I could go on, but I won't. Just so I am not accused of saying that developers have it harder than anyone else in software development, lets consider QA.
Our QA Manager and I were talking about tracking the testing of our products yesterday. We came up with the following list of inputs that effect the testing effort on a project:

• Functional Testing (Status, Planning, etc)
• Status of Translation
• Rate of Code Change
• Rate Defects are being found
• Rate Defects are being resolved
• Rate of Functionality Change

That's not in any particular order by the way. There are six inputs (probably more) that he needs to be aware of to management the testing effort on a project. We could make similar lists for each person involved in the process (testers, documentation, development, management, and so on). The reason Process is hard is because it needs to take all of that into consideration. By the way, this is also why Process is essential. If each person stopped to think about how each task should be done while they were doing it, things would take substantially longer. If each person decided for themselves how they were going to communicate with others on the team, there would be no communication.

So Process is complicated and hard to get right. Therefore we should have many people involved in creating our Process right? wrong. Why isn't it be better to work on process as a group? Simplicity and consistency are the answers. A process should be simple and must be consistent.
Simplicity
Process discussions tend to attract people who really like process. You know the type, if a little process is a good thing then a lot of process must be a great thing! Of course this is not true. Process should be as simple as possible, but no simpler. One way to help keep process simple, is to have process be decided on by one person. People involved in the conversation will bring things to the process. They will likely be good ideas, but they will be more than is strictly necessary to keep the team working efficiently. Process is a minimal pursuit. It is not that anyone should be excluded, but like good writing one needs to keep removing things that are not moving you toward the goal.
Consistency
Many people means compromise. Compromise as an issue that it can lead to inconsistent results. Here is an example from a real world Process dialogue I was involved in. The team had decided that they wanted to use Agile methods (Scrum in particular). They liked the idea of minimal design as they had struggled with analysis paralysis in the past. Unit testing sounded like a good idea, and refactoring was definitely something they wanted to try. As they meetings went on it was decided that the goal of 100% Unit Test coverage was way too aggressive. There was a lot of existing code after all. One group thought that minimal unit testing of "hot spot" code was the place to start. Another group felt that 60% or 70% was a good goal to begin with. A compromise was reached that a goal of 40% to 50% coverage be unit tests would be used. A cycle started about 6 weeks into the project. In this cycle, a change (read refactoring) that passed all the unit tests would leave the application unusable. So while one or two members of the team tried to isolate the issue, the rest of the team was forced to work on either an older version or rely on the unit tests only to verify the changes they were making. This cycle was happening two time a week. What went wrong?
The process the team had adopted was inconsistent. If you accept minimal up front design, then there will certainly be sweeping changes later on in the project as design flaws are discovered. In order to be able to make those sweeping changes, you must be able to test if the functionality you did not intend to change is consistent. One way to do that is to have near 100% coverage with automated tests (i.e. Unit Tests). In other words a Process that involved minimal up front design, must include refactoring and unit testing to be consistent. The process the team had settled on did not contain all of the elements so it was in consistent.
So Process is hard, but it is still best designed by one person. In order for it to be effective it must be adopted by all. It should be simple, or as simple as it can be. It should be minimal. It must be consistent. The herd needs a shepherd. We need a guide so we can focus on what we are trying to accomplish. We need process so we can be efficient and stay sane.

The Road Ahead?

Project Management have an impossible job. Part of their job is predicting the future. The goal is to identify when a project will complete so that resources can be managed. Project Managers also have an interesting way of dealing with the difficulty of this situation. They pass it on. That is they ask the developers when they will be done. At least that is the way it has worked on the software development projects I have been involved with. As a developer I don't like the question "When will I be done"? To begin with it involves the question how done I am. As I wrote last week, I don't think that is a very useful question. Of course, starting with the shaky footing of how done am I and adding the uncertainty of the future into the equation leads to unreliable answers. Of course, it also allows the project manager plausible deniability. After all, it wasn't the project manager who incorrectly estimated.

Why shouldn't the developer estimate when the work will be done? Isn't the developer in the best position to estimate the work? Actually, the developer is not. Now if you are a developer you likely take offense to that, but let's consider the facts. A developer is, or should be, focused on the thousands of detail decisions that go into, well development. What data structure should be used? What UI paradigm should be followed? Is this a part of the the system a candidate for some type of an extensible framework, or should it just be coded and forgotten? To estimate when the work is going to be done, the developer needs to focus on the work that remains to be done. She needs to consider the work schedule not only for herself but for others who's work supports, or interacts with the work that needs to be completed. As developers we like to think that we can do anything, but we have to admit that we cannot do that all at the same time.

So now that I am off my soapbox, how do Agile Methodologies deal with the need for precognition in Project Management? First they avoid the infatuation with dates. A completion date is a pretty poor piece of information. Without know the expectations that went into arriving at the end date, one cannot use it to evaluate how changes in staffing might effect the date. Is the end date 2 weeks out because there is 2 weeks of work to do, or because the developer is waiting on something that will arrive in 10 days and then there is 3 days of work. If all you have is an end date, how do you know the difference? In Agile Methods we are interested in velocity, and issues. From velocity we get an estimate of how long the remaining work (to do estimate from the developer) will take to complete. Obviously the project manager will take any time off or other distractions into account. The issues tells the project manager what is slowing development so they can do something about it, hopefully. What I especially like about Agile Methods is that they look for ways to ensure people are working together.

So that is about it for what I like and don't like about Agile Methods. Not, sure what I will write about next, but having a series like this seemed to work pretty well.