Tuesday, December 09, 2008

Natural Selection does not explain Evolution

I have been reading Dr Dawkins' book "The God Delusion". In chapter 4 he reiterates a statement I have heard time and again that is simply not true. The statement is that natural selection removes the improbability of apparent design from biological structures (that means us :-). This is not true. Before you warm up your flamethrowers, please hear me out.

First let me say that I am not saying that the theory of evolution is wrong. Nor am I likely to go on about how evolution is "only" a theory. Most of science is a theory because it is very difficult to prove something true.

Still, I feel that there is a fundamental misunderstanding of the process of natural selection. You see natural selection does not result in speciation. What's speciation? Speciation is the process or processes that result in a new species. Though it is widely accepted that natural selection play a role in speciation, its role is to remove the cruft from the work table. I know of noone who suggests that natural selection causes new trait to come into being. Rather natural selection is the tendency over time to select traights that are beneficial (assuming they are genetic in origin). So, given a black moth and a white moth of the same species, natural selection causes the black moths to become a large percentage of the overall population because toxins introduced into the moth's environment kill lichens that had previously hidden the white moths. Still natural selection did not cause the black mutation. You see natural selection can only work on traits that exist.

Another interesting thing about natural selection is that it works very quickly. I am speaking in terms of evolutionary time scales here. Say four or five generations is enough for a species to become predominantly of one trait. This time will vary depending on the animal, but even if it where 500 years, that is a minor amount of time on an evolutionary scale (millions and billions of years). If fact the combination of natural selection and genetic drift, obviously play a predominant role in the shaping of species on the planet. I make this statement because they act so quickly and are always exerting their influence. So what does that say for evolution?

Lets say that evolution introduces change at a rate of 1 change per year (I suspect this is high, but lets go with it). Lets say that on average 50-100 changes are necessary to create a new species. A new species is defined as enough genetic difference to make sharing genetic material impractical. Lets assume a one year gestation period on average (I suspect this is a little high, but we are talking in generalities here). What does that mean? It means that we are creating new species at a rate of approximately one every 50 years. The number of species in the world is declining at the same time at a rate of about 1 every 5 years. This means that we should expect to see the number of species in the world in decline. Most biologist accept that the number of species in the world is currently declining. But evolution predicts that the number of species in the world should be increasing.

Wait, if you think I just proved that evolution does not work, you need to wait. There is another shoe. The numbers I was throwing around there where general averages. They were probably not right even at that. Most biologist today believe that something happens every so often that results in many new species. Then the number of species declines over time, until again something happens. There is little consensus about that something. My point in spelling all this out was to show that though natural selection explains a portion of evolutionary theory, it is also an issue for other portions of the theory. Rarely in science is anything a slam dunk. One must be careful of ones statements. I agree with Dr Dawkins regarding the beauty of natural selection, but it is selection, not creation. Until we have a better idea of how speciation works, there is little reason to get excited. On the other hand, this stumbling point is no reason to toss evolution out with its own bath water.

It needs work, but that is what science is all about.

Wednesday, July 16, 2008

Process Herd

Process is hard. One of the coping techniques we use is to work together. Obviously, many human accomplishments could only have happened with help. Skyscrapers, the moon landing, improved health care, etc. Still, many human accomplishments could only have happened because there was only one person involved or at least one person in charge. The ceiling of the Sisteen Chapel, The theory of Relativity, etc. So what does any of this have to do with the fact that process is hard?
I believe that the only way to come up with a working process is to have one person work on it. I don't mean that the person should go off into a cave and work on it. They should seek input from others. They should seek correction from others. Still one individual should make all the decisions. I am suggesting a sort of benign dictatorship. Why is this so essential? To understand let us think about why Process is hard.
Process is hard because software development is hard. Therefore deciding how one should go about it is hard. Software development is hard like driving is hard. My daughters are just hitting that magic 16th year. As a result, I have taken the time to think about driving. One thing that I have told my daughters is that no single part of driving is really hard. OK, maybe parallel parking :-). Seriously though, driving is many many simple things. What's hard about driving is that you have to do all these things at the same time and you have to keep doing them. So while you are maintaining your lane position, you also have to be aware of the cars around you, and were you are in your route to your destination. Additionally, you need to be looking for road hazards, you also need to be looking for other drivers around you that maybe about to do something you will have to respond too. Then there is cross traffic to keep an eye on, you must be paying attention to traffic signs and signals. Finally, if you are 16 it is also important to be on the right radio station :-). All of that and you haven't made any maneuvers yet. Software development is like that. As a developer you are typing in code, that must be syntactically correct. Furthermore, it should be efficient and maintainable. It must fit within the plan for the piece of functionality you are working on. It may be useful to others so you should keep reuse in mind. You should be adding comments such that a person who has never looked at this code can figure out your intent the first time through. I could go on, but I won't. Just so I am not accused of saying that developers have it harder than anyone else in software development, lets consider QA.
Our QA Manager and I were talking about tracking the testing of our products yesterday. We came up with the following list of inputs that effect the testing effort on a project:
  • Functional Testing (Status, Planning, etc)
  • Status of Translation
  • Rate of Code Change
  • Rate Defects are being found
  • Rate Defects are being resolved
  • Rate of Functionality Change

That's not in any particular order by the way. There are six inputs (probably more) that he needs to be aware of to management the testing effort on a project. We could make similar lists for each person involved in the process (testers, documentation, development, management, and so on). The reason Process is hard is because it needs to take all of that into consideration. By the way, this is also why Process is essential. If each person stopped to think about how each task should be done while they were doing it, things would take substantially longer. If each person decided for themselves how they were going to communicate with others on the team, there would be no communication.

So Process is complicated and hard to get right. Therefore we should have many people involved in creating our Process right? wrong. Why isn't it be better to work on process as a group? Simplicity and consistency are the answers. A process should be simple and must be consistent.
Process discussions tend to attract people who really like process. You know the type, if a little process is a good thing then a lot of process must be a great thing! Of course this is not true. Process should be as simple as possible, but no simpler. One way to help keep process simple, is to have process be decided on by one person. People involved in the conversation will bring things to the process. They will likely be good ideas, but they will be more than is strictly necessary to keep the team working efficiently. Process is a minimal pursuit. It is not that anyone should be excluded, but like good writing one needs to keep removing things that are not moving you toward the goal.
Many people means compromise. Compromise as an issue that it can lead to inconsistent results. Here is an example from a real world Process dialogue I was involved in. The team had decided that they wanted to use Agile methods (Scrum in particular). They liked the idea of minimal design as they had struggled with analysis paralysis in the past. Unit testing sounded like a good idea, and refactoring was definitely something they wanted to try. As they meetings went on it was decided that the goal of 100% Unit Test coverage was way too aggressive. There was a lot of existing code after all. One group thought that minimal unit testing of "hot spot" code was the place to start. Another group felt that 60% or 70% was a good goal to begin with. A compromise was reached that a goal of 40% to 50% coverage be unit tests would be used. A cycle started about 6 weeks into the project. In this cycle, a change (read refactoring) that passed all the unit tests would leave the application unusable. So while one or two members of the team tried to isolate the issue, the rest of the team was forced to work on either an older version or rely on the unit tests only to verify the changes they were making. This cycle was happening two time a week. What went wrong?
The process the team had adopted was inconsistent. If you accept minimal up front design, then there will certainly be sweeping changes later on in the project as design flaws are discovered. In order to be able to make those sweeping changes, you must be able to test if the functionality you did not intend to change is consistent. One way to do that is to have near 100% coverage with automated tests (i.e. Unit Tests). In other words a Process that involved minimal up front design, must include refactoring and unit testing to be consistent. The process the team had settled on did not contain all of the elements so it was in consistent.
So Process is hard, but it is still best designed by one person. In order for it to be effective it must be adopted by all. It should be simple, or as simple as it can be. It should be minimal. It must be consistent. The herd needs a shepherd. We need a guide so we can focus on what we are trying to accomplish. We need process so we can be efficient and stay sane.

Sunday, July 06, 2008

The Road Ahead?

Project Management have an impossible job. Part of their job is predicting the future. The goal is to identify when a project will complete so that resources can be managed. Project Managers also have an interesting way of dealing with the difficulty of this situation. They pass it on. That is they ask the developers when they will be done. At least that is the way it has worked on the software development projects I have been involved with. As a developer I don't like the question "When will I be done"? To begin with it involves the question how done I am. As I wrote last week, I don't think that is a very useful question. Of course, starting with the shaky footing of how done am I and adding the uncertainty of the future into the equation leads to unreliable answers. Of course, it also allows the project manager plausible deniability. After all, it wasn't the project manager who incorrectly estimated.

Why shouldn't the developer estimate when the work will be done? Isn't the developer in the best position to estimate the work? Actually, the developer is not. Now if you are a developer you likely take offense to that, but let's consider the facts. A developer is, or should be, focused on the thousands of detail decisions that go into, well development. What data structure should be used? What UI paradigm should be followed? Is this a part of the the system a candidate for some type of an extensible framework, or should it just be coded and forgotten? To estimate when the work is going to be done, the developer needs to focus on the work that remains to be done. She needs to consider the work schedule not only for herself but for others who's work supports, or interacts with the work that needs to be completed. As developers we like to think that we can do anything, but we have to admit that we cannot do that all at the same time.

So now that I am off my soapbox, how do Agile Methodologies deal with the need for precognition in Project Management? First they avoid the infatuation with dates. A completion date is a pretty poor piece of information. Without know the expectations that went into arriving at the end date, one cannot use it to evaluate how changes in staffing might effect the date. Is the end date 2 weeks out because there is 2 weeks of work to do, or because the developer is waiting on something that will arrive in 10 days and then there is 3 days of work. If all you have is an end date, how do you know the difference? In Agile Methods we are interested in velocity, and issues. From velocity we get an estimate of how long the remaining work (to do estimate from the developer) will take to complete. Obviously the project manager will take any time off or other distractions into account. The issues tells the project manager what is slowing development so they can do something about it, hopefully. What I especially like about Agile Methods is that they look for ways to ensure people are working together.

So that is about it for what I like and don't like about Agile Methods. Not, sure what I will write about next, but having a series like this seemed to work pretty well.

Saturday, June 28, 2008

Are We There Yet?

So far I have been talking about aspects of agile methodologies that I am not a fan of. So lets look at what I am a fan of. One aspect of Agile Methods that I am a big fan of is the definition of done. Done is a sore spot in project management. People tend to define down based on what they have to do personally. For a developer, done typically means that the code is complete. The trick is that the no one cares if the code is done, except the developer. The customer is not buying code. The customer wants to know if they can use function XYZ. Project Management for development projects looks for the completion of functionality XYZ, but typical development projects are not organized or focused on completing XYZ. The more common organization is that the Developers work on their own, the QA people work on their own, the Doc Writers work on their own, etc. Of course they interact, but they do not work together. This leads to a "my part is done" mindset. If I am a Developer in a group of Developers, what can I say about a feature except that the code is complete?
The Agile approach is not to group Developers, but to put a couple of Developers together with a few QA people and a Doc Writer. Now I have a team (hopefully), that is focused on accomplishing some chunk of functionality. Obviously putting people together in a room does not make them a team. Still you have a group that has the possibility and perspective to be focused on completing some functionality for the Customer. What is even better is if the Customer is part of that team as well. Then you can be very confident that the functionality is not only complete, but actually what the customer wants. It wont be all of what the customer wants, but it will be something.
This is a huge change from the way people in this industry typically work. Its a change no one will really want. Still, it can have a dramatic effect on what you get down, and how happy your customer is.

Monday, June 23, 2008

Through a Mirror Dimly

I don't know what I am doing part of the time. That is a difficult thing for any engineer to admit, but it is true. The early days of any software project are filled with uncertainty. This is when you find out what is real and what is hype. This is when the development team really pushes back on the customer/marketing. The customer doesn't have to justify what they want, but they do have to understand it. I can't create an application that is "easy to use", "responsive", or "pleasant to look at". These are all requirements I have received in the past, but I cannot fulfill them. I can build an application that has buttons on the left that define the application steps, properties for the current tool/action in the lower center and help in the lower right. I can ensure that the user will get some type of feed back for every action in less than a quarter second. I can ensure that the widgets drawn on the screen have rounded corners and a 3D glass effect like those in Vista. The difference may see like trivia to some, but as an engineer the difference is huge. These differences are also important for QA and acceptance. The first set of requirements cannot be tested. The second set can.
So I find that most of the effort in the first few iterations of a project is actually spent prototyping and refining requirements. This is the time where Stories or Backlog Items are broken down because they were too general to begin with. This is the time when tasks are defined. This is the time when the original high level estimates are tested for sanity. This is also the time when the calendar is consulted to see if Joe UI Guru is going to be on vacation during the bulk of the UI prototype work. This is what used to be called design.
Now I know that we don't have a design phase anymore, but you gotta have a plan. I do agree that it does not need to be a thorough and detailed plan. Still one should identify the areas that represent the most risk. We are not going to spend time trying to figure out how we are going to put text on the screen for most apps. We know how to do that. In fact we know several ways to do that and that choice is an implementation detail. We are going to spend time on how the UI flow works to avoid the steps in the wrong order bug. Or we should. I believe the TDD, code and refactor approach assumes design is inherent in the act of coding and therefore assumes a pretty good, fairly experienced developer. If your team is made up of such people, then it will likely work because of there experience level. If you have say interns working on your team, forcing them to stop an think about the code they are about to create is probably worth while.
So what do we call this part of the development cycle? I like detailed planning.

Check back next week to hear how the story ends.

Monday, June 16, 2008

I am not a brick in the wall

I enjoy what most people would call "Hard" music. I grew up with the Sex Pistols, Black Sabath, and of course Pink Floyd. I enjoyed the Wall when it came out (both the album and movie), and still listen to it from time to time. The title song contains the lyric, "All in all you're just another brick in the wall". Its not what most would call a pretty picture. Yet its the way most Agile Methodoologies view development staff.

I am told that I should not consider individuals when I am planning my projects. It is better to plan for the team. I can just guess how much the team will get done. Then as I track how much the team actually gets done I will start to understand how bad my guesses are and I can plan accordingly. Setting aside for the moment the idea that I should not try to become better at estimating how much work a task is, lets focus instead on the idea of scheduling for the team.

I have spent most of my professional life working on integration code. I have a background in hetworking (I used to be a network administrator and troubleshooter). This gives me a special insight into how the network is going to respond to the various schemes for getting two (or more) applications to talk to each other. I tend not to make certain mistakes because it is obvious to me that the underlying network will have issues with it. As a result of this experience I tend to get integration tasks done more quickly than other programmers. More then just getting them done faster, my solutions typically have fewer problems in the wild.

This is not just me patting myself on the back, however. I am especially bad at creating GUI code. I tend to take a long time. I tend to focus on things that turn out to be of little importance and not spend enough time on things that are important. Not to say that my code is trash, but it is not nearly as good as some other people on my team who have sort of specialized in GUI development. They just natuarally avoid certain mistakes that their experience has told them to avoid.

Now let's consider that I am planning a vacation (which I am). I will be gone for a week and a half. If it turns out that there is little integration work during that time, this will have a minimal impact on the schedule. After all the GUI work that I would have gotten done in that time is limited by my ability to do GUI work (or lack there of I guess). If there where a great deal of integration work then the story would be reversed. However I am told not to worry about such things. This seems odd to me. Am I really supposed to believe that my manager and I can switch places (see last weeks post) and that will not have an effect? That I can trade work with the Intern who is not yet finished with his degree and the outcome will be the same?

No I think not. Maybe I just don't understand, but it has not been due to lack of trying. I think Agile methodologies have it dead wrong when I am told not to do detailed planning and to ignore people's strengths. Of course I think Agile methodologies get other things right. But more on that next time.

Monday, June 09, 2008

I'm a Nerd God

NerdTests.com says I'm a Nerd God.  What are you?  Click here!
I am a Nerd God.

Well that's what the test says anyway. My boss is not a Nerd God. He was a nerd and maybe still is, but not a Nerd God. I have dreams entirely made up of code. Note, I was not coding in the dream. Rather all of existence was code. I suspect that I have a little more than healthy obsession with code. I learn new programming languages for fun. I practiced coding before Dave Thomas started Code-Kata. When I am stressed out from a day of coding at work I come home and the first thing I do after saying hello to my lovely wife and daughters is to start thinking about the projects I am working on at home, it helps me relax.

My boss has not done any type of coding for years on the other hand. My boss is well spoken and articulate. He has a good mind for dates (I do not!). He is good at reading people, knowing if they are upset. He is excellent at explaining the problem in a way that all can understand and that does not upset anymore unduly. I am good at inciting a riot with a single statement. OK, not quite that bad, it usually take two or three.

So no, my boss could not do my job, but then I could not do his either. Don't misunderstand me, he could figure it out given time, but the results would not be up to snuff. He has written code in the past. Good code as far as I know. With some practice I am sure he could write good code again, but probably never great code. Not to say that all the code I write is great, but I do write great code from time to time.

I have been a project manager. I received an award for my skills running a project. Kept people on track, got things done and generally hated every minute of it. I am pretty sure the people who worked for me did not find it a picnic either. I would never be a great project manager. My boss is a great project manager. I'm not just saying that in case he reads this blog. I doubt anyone reads this blog :-). He does some pretty amazing things with moving people on our team around. He know just when to pull someone off team A and put them on team B to get both projects done on time. That's what I mean when I say he has a head for dates. We have done some amazing work these last two years that I have worked for him. You would never hear him say it, but a good chunk of what we have accomplished is a direct result of his management. Still he could not do my job.

There is an implied elitism in the idea that my manager should be able to do my job. Its like saying "If you are smart enough to be a programmer then obviously you can manage a project". Does anyone expect the general contractor to be able to do plumbing, electrical and carpentry work? What about sheet rock, masonry, and landscaping? Obviously no one can be good at all of that (and there is more to do). That's why we have different people on the team. Different skills make the team stronger.

Does my boss understand my job, yes certainly. Is he capable of it, no decidedly not. That is part of why he is now a manager. If he was a skilled and adept programmer he would never have been allowed to go into full time management. At least he would not have been allowed to do that at the company where I currently work. My present company values technical people enough that it is never necessary to "make the switch" to management just to get paid more.

What does this have to do with Agile Methodologies though? You'll have to keep reading to find out.

Thursday, June 05, 2008

Being Agile

I have recently taken on a new role at work. I am the tool wright for our Business Unit. This mean that it is my responsibility to ensure that the tools we use are "sharp" and ready to be used. One of the tools we use is our process. I mean the low level development process as apposed to the higher level software life cycle process (though I am responsible to maintain that as well). When I say I am responsible for our development process, that means it's my job to document it. It is also my job to create Templates/Tools/whatever else to make the process work as smoothly as it can. No process has zero impact on those who follow it, but the impact should be minimal and relatively painless.

All this means that I attend many meetings about our process. As we are making the transition from an ad-hock process (read that as essentially no process) to an Agile process (more or less XP), there are many things we have to figure out. One thing I keep hearing from the developers is that we are not Agile, to which management responds we are agile and have been for a while. Furthermore, management insists that we have to be agile to be effective in our market place. You gotta love it when some one takes a word that everyone knows the meaning of and gives it a new meaning.

So what does it mean to be agile and what does it mean to be Agile? Well, Agile used to mean something. In fact it meant something pretty useful after the Agile Manifesto was originally drafted. So you don't have to follow the link I will copy the Agile Manifesto here:

We are uncovering better ways of developing software by doing it
and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools

  • Working software over comprehensive documentation

  • Customer collaboration over contract negotiation

  • Responding to change over following a plan

That is, while there is value in the items on the right, we
value the items on the left more.

That's it. Its fairly loose, but it defines an idea, an approach to the problem. What it doesn't do is cause someone to purchase a book, or high a consulting company. Don't misunderstand me, I have no issues with XP, or Scrum, or DSDM or whatever your favorite Agile Methodology is, but the strict adherence to any methodology is contrary to the fourth point above. The goal of Agile methods (all agile methods) is to respond to the real world rather than focus on the model. A Methodology is a group of methods that have been identified as typically having a favorable result. Sometimes people call them best practices, but the goal is to continue to do what has worked in the past. This is normal and useful. You don't figure out how to get to work each morning, you go the same way you went yesterday and know that you will likely make it.

The issue come when you turn a corner and find that a construction crew has dug up the road to work on the sewer. Obviously you cannot continue to go that way. It might be possible to get through, but it would be risky at best. Likely it would end badly, so instead you think to yourself what other path can I take to get to work. Project management should work the same way and it often does. The issue with Waterfall Methodologies it turns out wasn't that they didn't allow for change, but that people did not recognize and respond to the need for change. Almost all Waterfall methodologies have an idea of a feedback loop built into them somewhere. It is usually very long and so relies on people to remember problems and be interested in fixing them weeks or months after they have been a problem. Agile methods stress shortening this time frame and that is a good thing, but you still have to recognize when the train is off the tracks.

I like the train off the tracks image because I locomotive that has left its rails does not stop right away. It is going to stop soon however and it will make a mess as it does so. I hear people say things like "We are still getting things done, so we must be alright." I don't know why people think that way. It would be like skidding sideways down a freeway and saying we are still going toward our destination so we are alright. No you are not. Yes your moving in the right direction, but you have no control, etc. This will end badly.

So where am I going with this? Response to change is not what makes Agile Methods different. What allows for agility (the ability to change direction) in Agile Methods is knowing where we are. What is important and different about Agile Methods is defining work that can be done in a time frame that allows you to change course. Having things that you can get done in days or weeks means that you can change direction in days or weeks. If everything you are working on is going to take months, then it is very difficult to change priority and work on something different. Yes, but you say it really does take months to do ... I know it does, but to accomplish that you are really doing 100 smaller things that take days or weeks to do. What Agile Methods are about is identifying what those smaller things are and assigning them. So that you can change who is helping with the work, or even what work is getting done this week as the project goes on.