Software Hygiene: Your Editor is Important

This entry is part of a series called Software Hygiene. The idea comes from a whitepaper I read back in the 70s or 80s called something like "On Brushing Ones Teeth, Hygiene in Software Development". The overall point was that though methodologies and languages can be important, the habits you have are very important also.

Why does the editor matter? Why are there endless debates about Emacs vs VI to this day? The reason is simple, if you develop software you will spend a huge amount of time working in your editor. As editors have become Integrated Development Environments (IDEs) this has become even more true. Of course many reading this will not know that editors did not always allow you to launch a debugger. So given that you are going to spend some serious time with your editor, you should at least get to know it.

One of the first things I do with my editor is change the color scheme. The default color scheme is designed to look good, especially from a distance. It is designed to attract your attention to an advertisement for the IDE. Sorry if this seems negative, but this is how business works. So you should try some things. You will notice that I am not telling you what color to make your editors background. I'm not because I don't know. The goal is to make the screen you are going to be stairing at as pleasent as you can. You should try some different color schemes and see what sticks. You may find that your eyes were suffering and you did not even know it.

Obviously, colors are not the only aspect of the editor that matters. Modern editors have all kinds of interesting and clever ways they can help you. One aspect you may not have thought of, and that is the key bindings. An extream approach is to try a completely different key set. I like to use the VI key bindings myself. Still, before you change the world, consider the things that you do often. There are likely things that you do often that do not have an easy keyboard shortcuts. Every time you take your hands off the keyboard to use the mouse, your flow is interupted. Perhaps it is not a big deal, but why take the hit? Why not spend a few minutes to setup a keyboard shortcut so you can use it and keep moving. Another idea along the same lines is to find out how your editor uses templates. They may not be called templates, but most editirs have a method for expanding a few characters into a block of code. Again, most of us find that we write similar peices of code over and over. Of course the ultimate version of a template is a project template. They will typically write multiple files worth of code at once.

At this point most people talk about how they don't know how to do any of this. I appreaciate that these ideas can be time consuming to learn about. Still consider the return on your investment. Spend an hour or two working through how to build a template that will save you 10 minutes. If you use that template three times a week, inside of a month or two you have made up the time, then you are getting ahead.

It is interesting to me how little most developers know about thier tools. A carpenter has likely tried a dozen or so different hammers. They know the wieght, length, etc of the hammer that is right for them. They do this because they have a hammer in their hand all day long. In the case of the capenter they can actually become injured if they use the wrong hammer. Fortunately, as a software developer it is unlikely that we will be injured. However, we can all stand to have a little more time in our schedule. So spend the time to get to know your editor, or take the plunge to find an editor that you really love.

Natural Selection does not explain Evolution

I have been reading Dr Dawkins' book "The God Delusion". In chapter 4 he reiterates a statement I have heard time and again that is simply not true. The statement is that natural selection removes the improbability of apparent design from biological structures (that means us :-). This is not true. Before you warm up your flamethrowers, please hear me out.


First let me say that I am not saying that the theory of evolution is wrong. Nor am I likely to go on about how evolution is "only" a theory. Most of science is a theory because it is very difficult to prove something true.


Still, I feel that there is a fundamental misunderstanding of the process of natural selection. You see natural selection does not result in speciation. What's speciation? Speciation is the process or processes that result in a new species. Though it is widely accepted that natural selection play a role in speciation, its role is to remove the cruft from the work table. I know of noone who suggests that natural selection causes new trait to come into being. Rather natural selection is the tendency over time to select traights that are beneficial (assuming they are genetic in origin). So, given a black moth and a white moth of the same species, natural selection causes the black moths to become a large percentage of the overall population because toxins introduced into the moth's environment kill lichens that had previously hidden the white moths. Still natural selection did not cause the black mutation. You see natural selection can only work on traits that exist.


Another interesting thing about natural selection is that it works very quickly. I am speaking in terms of evolutionary time scales here. Say four or five generations is enough for a species to become predominantly of one trait. This time will vary depending on the animal, but even if it where 500 years, that is a minor amount of time on an evolutionary scale (millions and billions of years). If fact the combination of natural selection and genetic drift, obviously play a predominant role in the shaping of species on the planet. I make this statement because they act so quickly and are always exerting their influence. So what does that say for evolution?


Lets say that evolution introduces change at a rate of 1 change per year (I suspect this is high, but lets go with it). Lets say that on average 50-100 changes are necessary to create a new species. A new species is defined as enough genetic difference to make sharing genetic material impractical. Lets assume a one year gestation period on average (I suspect this is a little high, but we are talking in generalities here). What does that mean? It means that we are creating new species at a rate of approximately one every 50 years. The number of species in the world is declining at the same time at a rate of about 1 every 5 years. This means that we should expect to see the number of species in the world in decline. Most biologist accept that the number of species in the world is currently declining. But evolution predicts that the number of species in the world should be increasing.


Wait, if you think I just proved that evolution does not work, you need to wait. There is another shoe. The numbers I was throwing around there where general averages. They were probably not right even at that. Most biologist today believe that something happens every so often that results in many new species. Then the number of species declines over time, until again something happens. There is little consensus about that something. My point in spelling all this out was to show that though natural selection explains a portion of evolutionary theory, it is also an issue for other portions of the theory. Rarely in science is anything a slam dunk. One must be careful of ones statements. I agree with Dr Dawkins regarding the beauty of natural selection, but it is selection, not creation. Until we have a better idea of how speciation works, there is little reason to get excited. On the other hand, this stumbling point is no reason to toss evolution out with its own bath water.





It needs work, but that is what science is all about.

Process Herd

Process is hard. One of the coping techniques we use is to work together. Obviously, many human accomplishments could only have happened with help. Skyscrapers, the moon landing, improved health care, etc. Still, many human accomplishments could only have happened because there was only one person involved or at least one person in charge. The ceiling of the Sisteen Chapel, The theory of Relativity, etc. So what does any of this have to do with the fact that process is hard?
I believe that the only way to come up with a working process is to have one person work on it. I don't mean that the person should go off into a cave and work on it. They should seek input from others. They should seek correction from others. Still one individual should make all the decisions. I am suggesting a sort of benign dictatorship. Why is this so essential? To understand let us think about why Process is hard.
Process is hard because software development is hard. Therefore deciding how one should go about it is hard. Software development is hard like driving is hard. My daughters are just hitting that magic 16th year. As a result, I have taken the time to think about driving. One thing that I have told my daughters is that no single part of driving is really hard. OK, maybe parallel parking :-). Seriously though, driving is many many simple things. What's hard about driving is that you have to do all these things at the same time and you have to keep doing them. So while you are maintaining your lane position, you also have to be aware of the cars around you, and were you are in your route to your destination. Additionally, you need to be looking for road hazards, you also need to be looking for other drivers around you that maybe about to do something you will have to respond too. Then there is cross traffic to keep an eye on, you must be paying attention to traffic signs and signals. Finally, if you are 16 it is also important to be on the right radio station :-). All of that and you haven't made any maneuvers yet. Software development is like that. As a developer you are typing in code, that must be syntactically correct. Furthermore, it should be efficient and maintainable. It must fit within the plan for the piece of functionality you are working on. It may be useful to others so you should keep reuse in mind. You should be adding comments such that a person who has never looked at this code can figure out your intent the first time through. I could go on, but I won't. Just so I am not accused of saying that developers have it harder than anyone else in software development, lets consider QA.
Our QA Manager and I were talking about tracking the testing of our products yesterday. We came up with the following list of inputs that effect the testing effort on a project:

• Functional Testing (Status, Planning, etc)
• Status of Translation
• Rate of Code Change
• Rate Defects are being found
• Rate Defects are being resolved
• Rate of Functionality Change

That's not in any particular order by the way. There are six inputs (probably more) that he needs to be aware of to management the testing effort on a project. We could make similar lists for each person involved in the process (testers, documentation, development, management, and so on). The reason Process is hard is because it needs to take all of that into consideration. By the way, this is also why Process is essential. If each person stopped to think about how each task should be done while they were doing it, things would take substantially longer. If each person decided for themselves how they were going to communicate with others on the team, there would be no communication.

So Process is complicated and hard to get right. Therefore we should have many people involved in creating our Process right? wrong. Why isn't it be better to work on process as a group? Simplicity and consistency are the answers. A process should be simple and must be consistent.
Simplicity
Process discussions tend to attract people who really like process. You know the type, if a little process is a good thing then a lot of process must be a great thing! Of course this is not true. Process should be as simple as possible, but no simpler. One way to help keep process simple, is to have process be decided on by one person. People involved in the conversation will bring things to the process. They will likely be good ideas, but they will be more than is strictly necessary to keep the team working efficiently. Process is a minimal pursuit. It is not that anyone should be excluded, but like good writing one needs to keep removing things that are not moving you toward the goal.
Consistency
Many people means compromise. Compromise as an issue that it can lead to inconsistent results. Here is an example from a real world Process dialogue I was involved in. The team had decided that they wanted to use Agile methods (Scrum in particular). They liked the idea of minimal design as they had struggled with analysis paralysis in the past. Unit testing sounded like a good idea, and refactoring was definitely something they wanted to try. As they meetings went on it was decided that the goal of 100% Unit Test coverage was way too aggressive. There was a lot of existing code after all. One group thought that minimal unit testing of "hot spot" code was the place to start. Another group felt that 60% or 70% was a good goal to begin with. A compromise was reached that a goal of 40% to 50% coverage be unit tests would be used. A cycle started about 6 weeks into the project. In this cycle, a change (read refactoring) that passed all the unit tests would leave the application unusable. So while one or two members of the team tried to isolate the issue, the rest of the team was forced to work on either an older version or rely on the unit tests only to verify the changes they were making. This cycle was happening two time a week. What went wrong?
The process the team had adopted was inconsistent. If you accept minimal up front design, then there will certainly be sweeping changes later on in the project as design flaws are discovered. In order to be able to make those sweeping changes, you must be able to test if the functionality you did not intend to change is consistent. One way to do that is to have near 100% coverage with automated tests (i.e. Unit Tests). In other words a Process that involved minimal up front design, must include refactoring and unit testing to be consistent. The process the team had settled on did not contain all of the elements so it was in consistent.
So Process is hard, but it is still best designed by one person. In order for it to be effective it must be adopted by all. It should be simple, or as simple as it can be. It should be minimal. It must be consistent. The herd needs a shepherd. We need a guide so we can focus on what we are trying to accomplish. We need process so we can be efficient and stay sane.

The Road Ahead?

Project Management have an impossible job. Part of their job is predicting the future. The goal is to identify when a project will complete so that resources can be managed. Project Managers also have an interesting way of dealing with the difficulty of this situation. They pass it on. That is they ask the developers when they will be done. At least that is the way it has worked on the software development projects I have been involved with. As a developer I don't like the question "When will I be done"? To begin with it involves the question how done I am. As I wrote last week, I don't think that is a very useful question. Of course, starting with the shaky footing of how done am I and adding the uncertainty of the future into the equation leads to unreliable answers. Of course, it also allows the project manager plausible deniability. After all, it wasn't the project manager who incorrectly estimated.

Why shouldn't the developer estimate when the work will be done? Isn't the developer in the best position to estimate the work? Actually, the developer is not. Now if you are a developer you likely take offense to that, but let's consider the facts. A developer is, or should be, focused on the thousands of detail decisions that go into, well development. What data structure should be used? What UI paradigm should be followed? Is this a part of the the system a candidate for some type of an extensible framework, or should it just be coded and forgotten? To estimate when the work is going to be done, the developer needs to focus on the work that remains to be done. She needs to consider the work schedule not only for herself but for others who's work supports, or interacts with the work that needs to be completed. As developers we like to think that we can do anything, but we have to admit that we cannot do that all at the same time.

So now that I am off my soapbox, how do Agile Methodologies deal with the need for precognition in Project Management? First they avoid the infatuation with dates. A completion date is a pretty poor piece of information. Without know the expectations that went into arriving at the end date, one cannot use it to evaluate how changes in staffing might effect the date. Is the end date 2 weeks out because there is 2 weeks of work to do, or because the developer is waiting on something that will arrive in 10 days and then there is 3 days of work. If all you have is an end date, how do you know the difference? In Agile Methods we are interested in velocity, and issues. From velocity we get an estimate of how long the remaining work (to do estimate from the developer) will take to complete. Obviously the project manager will take any time off or other distractions into account. The issues tells the project manager what is slowing development so they can do something about it, hopefully. What I especially like about Agile Methods is that they look for ways to ensure people are working together.

So that is about it for what I like and don't like about Agile Methods. Not, sure what I will write about next, but having a series like this seemed to work pretty well.

Are We There Yet?

So far I have been talking about aspects of agile methodologies that I am not a fan of. So lets look at what I am a fan of. One aspect of Agile Methods that I am a big fan of is the definition of done. Done is a sore spot in project management. People tend to define down based on what they have to do personally. For a developer, done typically means that the code is complete. The trick is that the no one cares if the code is done, except the developer. The customer is not buying code. The customer wants to know if they can use function XYZ. Project Management for development projects looks for the completion of functionality XYZ, but typical development projects are not organized or focused on completing XYZ. The more common organization is that the Developers work on their own, the QA people work on their own, the Doc Writers work on their own, etc. Of course they interact, but they do not work together. This leads to a "my part is done" mindset. If I am a Developer in a group of Developers, what can I say about a feature except that the code is complete?
The Agile approach is not to group Developers, but to put a couple of Developers together with a few QA people and a Doc Writer. Now I have a team (hopefully), that is focused on accomplishing some chunk of functionality. Obviously putting people together in a room does not make them a team. Still you have a group that has the possibility and perspective to be focused on completing some functionality for the Customer. What is even better is if the Customer is part of that team as well. Then you can be very confident that the functionality is not only complete, but actually what the customer wants. It wont be all of what the customer wants, but it will be something.
This is a huge change from the way people in this industry typically work. Its a change no one will really want. Still, it can have a dramatic effect on what you get down, and how happy your customer is.

Through a Mirror Dimly

I don't know what I am doing part of the time. That is a difficult thing for any engineer to admit, but it is true. The early days of any software project are filled with uncertainty. This is when you find out what is real and what is hype. This is when the development team really pushes back on the customer/marketing. The customer doesn't have to justify what they want, but they do have to understand it. I can't create an application that is "easy to use", "responsive", or "pleasant to look at". These are all requirements I have received in the past, but I cannot fulfill them. I can build an application that has buttons on the left that define the application steps, properties for the current tool/action in the lower center and help in the lower right. I can ensure that the user will get some type of feed back for every action in less than a quarter second. I can ensure that the widgets drawn on the screen have rounded corners and a 3D glass effect like those in Vista. The difference may see like trivia to some, but as an engineer the difference is huge. These differences are also important for QA and acceptance. The first set of requirements cannot be tested. The second set can.
So I find that most of the effort in the first few iterations of a project is actually spent prototyping and refining requirements. This is the time where Stories or Backlog Items are broken down because they were too general to begin with. This is the time when tasks are defined. This is the time when the original high level estimates are tested for sanity. This is also the time when the calendar is consulted to see if Joe UI Guru is going to be on vacation during the bulk of the UI prototype work. This is what used to be called design.
Now I know that we don't have a design phase anymore, but you gotta have a plan. I do agree that it does not need to be a thorough and detailed plan. Still one should identify the areas that represent the most risk. We are not going to spend time trying to figure out how we are going to put text on the screen for most apps. We know how to do that. In fact we know several ways to do that and that choice is an implementation detail. We are going to spend time on how the UI flow works to avoid the steps in the wrong order bug. Or we should. I believe the TDD, code and refactor approach assumes design is inherent in the act of coding and therefore assumes a pretty good, fairly experienced developer. If your team is made up of such people, then it will likely work because of there experience level. If you have say interns working on your team, forcing them to stop an think about the code they are about to create is probably worth while.
So what do we call this part of the development cycle? I like detailed planning.

Check back next week to hear how the story ends.

I am not a brick in the wall

I enjoy what most people would call "Hard" music. I grew up with the Sex Pistols, Black Sabath, and of course Pink Floyd. I enjoyed the Wall when it came out (both the album and movie), and still listen to it from time to time. The title song contains the lyric, "All in all you're just another brick in the wall". Its not what most would call a pretty picture. Yet its the way most Agile Methodoologies view development staff.

I am told that I should not consider individuals when I am planning my projects. It is better to plan for the team. I can just guess how much the team will get done. Then as I track how much the team actually gets done I will start to understand how bad my guesses are and I can plan accordingly. Setting aside for the moment the idea that I should not try to become better at estimating how much work a task is, lets focus instead on the idea of scheduling for the team.

I have spent most of my professional life working on integration code. I have a background in hetworking (I used to be a network administrator and troubleshooter). This gives me a special insight into how the network is going to respond to the various schemes for getting two (or more) applications to talk to each other. I tend not to make certain mistakes because it is obvious to me that the underlying network will have issues with it. As a result of this experience I tend to get integration tasks done more quickly than other programmers. More then just getting them done faster, my solutions typically have fewer problems in the wild.

This is not just me patting myself on the back, however. I am especially bad at creating GUI code. I tend to take a long time. I tend to focus on things that turn out to be of little importance and not spend enough time on things that are important. Not to say that my code is trash, but it is not nearly as good as some other people on my team who have sort of specialized in GUI development. They just natuarally avoid certain mistakes that their experience has told them to avoid.

Now let's consider that I am planning a vacation (which I am). I will be gone for a week and a half. If it turns out that there is little integration work during that time, this will have a minimal impact on the schedule. After all the GUI work that I would have gotten done in that time is limited by my ability to do GUI work (or lack there of I guess). If there where a great deal of integration work then the story would be reversed. However I am told not to worry about such things. This seems odd to me. Am I really supposed to believe that my manager and I can switch places (see last weeks post) and that will not have an effect? That I can trade work with the Intern who is not yet finished with his degree and the outcome will be the same?

No I think not. Maybe I just don't understand, but it has not been due to lack of trying. I think Agile methodologies have it dead wrong when I am told not to do detailed planning and to ignore people's strengths. Of course I think Agile methodologies get other things right. But more on that next time.

I'm a Nerd God

I am a Nerd God.

Well that's what the test says anyway. My boss is not a Nerd God. He was a nerd and maybe still is, but not a Nerd God. I have dreams entirely made up of code. Note, I was not coding in the dream. Rather all of existence was code. I suspect that I have a little more than healthy obsession with code. I learn new programming languages for fun. I practiced coding before Dave Thomas started Code-Kata. When I am stressed out from a day of coding at work I come home and the first thing I do after saying hello to my lovely wife and daughters is to start thinking about the projects I am working on at home, it helps me relax.

My boss has not done any type of coding for years on the other hand. My boss is well spoken and articulate. He has a good mind for dates (I do not!). He is good at reading people, knowing if they are upset. He is excellent at explaining the problem in a way that all can understand and that does not upset anymore unduly. I am good at inciting a riot with a single statement. OK, not quite that bad, it usually take two or three.

So no, my boss could not do my job, but then I could not do his either. Don't misunderstand me, he could figure it out given time, but the results would not be up to snuff. He has written code in the past. Good code as far as I know. With some practice I am sure he could write good code again, but probably never great code. Not to say that all the code I write is great, but I do write great code from time to time.

I have been a project manager. I received an award for my skills running a project. Kept people on track, got things done and generally hated every minute of it. I am pretty sure the people who worked for me did not find it a picnic either. I would never be a great project manager. My boss is a great project manager. I'm not just saying that in case he reads this blog. I doubt anyone reads this blog :-). He does some pretty amazing things with moving people on our team around. He know just when to pull someone off team A and put them on team B to get both projects done on time. That's what I mean when I say he has a head for dates. We have done some amazing work these last two years that I have worked for him. You would never hear him say it, but a good chunk of what we have accomplished is a direct result of his management. Still he could not do my job.

There is an implied elitism in the idea that my manager should be able to do my job. Its like saying "If you are smart enough to be a programmer then obviously you can manage a project". Does anyone expect the general contractor to be able to do plumbing, electrical and carpentry work? What about sheet rock, masonry, and landscaping? Obviously no one can be good at all of that (and there is more to do). That's why we have different people on the team. Different skills make the team stronger.

Does my boss understand my job, yes certainly. Is he capable of it, no decidedly not. That is part of why he is now a manager. If he was a skilled and adept programmer he would never have been allowed to go into full time management. At least he would not have been allowed to do that at the company where I currently work. My present company values technical people enough that it is never necessary to "make the switch" to management just to get paid more.

What does this have to do with Agile Methodologies though? You'll have to keep reading to find out.

Being Agile

I have recently taken on a new role at work. I am the tool wright for our Business Unit. This mean that it is my responsibility to ensure that the tools we use are "sharp" and ready to be used. One of the tools we use is our process. I mean the low level development process as apposed to the higher level software life cycle process (though I am responsible to maintain that as well). When I say I am responsible for our development process, that means it's my job to document it. It is also my job to create Templates/Tools/whatever else to make the process work as smoothly as it can. No process has zero impact on those who follow it, but the impact should be minimal and relatively painless.

All this means that I attend many meetings about our process. As we are making the transition from an ad-hock process (read that as essentially no process) to an Agile process (more or less XP), there are many things we have to figure out. One thing I keep hearing from the developers is that we are not Agile, to which management responds we are agile and have been for a while. Furthermore, management insists that we have to be agile to be effective in our market place. You gotta love it when some one takes a word that everyone knows the meaning of and gives it a new meaning.

So what does it mean to be agile and what does it mean to be Agile? Well, Agile used to mean something. In fact it meant something pretty useful after the Agile Manifesto was originally drafted. So you don't have to follow the link I will copy the Agile Manifesto here:

We are uncovering better ways of developing software by doing it
and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools


• 
  
  Working software over comprehensive documentation


• 
  
  Customer collaboration over contract negotiation


• Responding to change over following a plan

That is, while there is value in the items on the right, we
value the items on the left more.

That's it. Its fairly loose, but it defines an idea, an approach to the problem. What it doesn't do is cause someone to purchase a book, or high a consulting company. Don't misunderstand me, I have no issues with XP, or Scrum, or DSDM or whatever your favorite Agile Methodology is, but the strict adherence to any methodology is contrary to the fourth point above. The goal of Agile methods (all agile methods) is to respond to the real world rather than focus on the model. A Methodology is a group of methods that have been identified as typically having a favorable result. Sometimes people call them best practices, but the goal is to continue to do what has worked in the past. This is normal and useful. You don't figure out how to get to work each morning, you go the same way you went yesterday and know that you will likely make it.

The issue come when you turn a corner and find that a construction crew has dug up the road to work on the sewer. Obviously you cannot continue to go that way. It might be possible to get through, but it would be risky at best. Likely it would end badly, so instead you think to yourself what other path can I take to get to work. Project management should work the same way and it often does. The issue with Waterfall Methodologies it turns out wasn't that they didn't allow for change, but that people did not recognize and respond to the need for change. Almost all Waterfall methodologies have an idea of a feedback loop built into them somewhere. It is usually very long and so relies on people to remember problems and be interested in fixing them weeks or months after they have been a problem. Agile methods stress shortening this time frame and that is a good thing, but you still have to recognize when the train is off the tracks.

I like the train off the tracks image because I locomotive that has left its rails does not stop right away. It is going to stop soon however and it will make a mess as it does so. I hear people say things like "We are still getting things done, so we must be alright." I don't know why people think that way. It would be like skidding sideways down a freeway and saying we are still going toward our destination so we are alright. No you are not. Yes your moving in the right direction, but you have no control, etc. This will end badly.

So where am I going with this? Response to change is not what makes Agile Methods different. What allows for agility (the ability to change direction) in Agile Methods is knowing where we are. What is important and different about Agile Methods is defining work that can be done in a time frame that allows you to change course. Having things that you can get done in days or weeks means that you can change direction in days or weeks. If everything you are working on is going to take months, then it is very difficult to change priority and work on something different. Yes, but you say it really does take months to do ... I know it does, but to accomplish that you are really doing 100 smaller things that take days or weeks to do. What Agile Methods are about is identifying what those smaller things are and assigning them. So that you can change who is helping with the work, or even what work is getting done this week as the project goes on.

Global Warming Heats Up

I hate it when the press starts talking about science. The press doesn't get science. Most science does not create sound bites that are useful. Of course the sound bites that are created are usually only useful to convince people of something that there is no scientific basis for. I thought I would throw some more information into the fray. I am not a climatologist, nor a scientist of any kind. Science is essentially a hobby for me. I am fascinated by how things work, and understanding natural systems. Add to this a general conservationist mindset and I am interested in Global warming, of course. But let's look at what we know about Global Warming.

Global Warming Facts
Is There Global Warming?
This is not really a question. I live in Wisconsin. One of the things everyone in Wisconsin has heard of is the Ice Age trail. The Wisconsin landscape was carved by Glaciers. There are currently no Glaciers in Wisconsin. Obviously the planet is warmer than it used to be, therefore global warming. This is the problem with answering this question. It's really the wrong question.

98% of Scientist Support Global Warming
Yea ,team, but who cares? I don't care what sociologists, political scientists, or archaeologists think about global warming. Don't get me wrong these are likely to be really smart people. It's just that they are not likely to know any more about the Earth's climate than I am. What I what to know is how many climatologists and meteorologist think that the Earth is warming in a way not explained by climate cycles. I also want to know where they think the temperature will end up?

Human activity has an effect on Global Warming
OK, this one is a great example of a bad scientific statement. Human activity produces CO2 (among other things). CO2 is a greenhouse gas and therefore hold heat within the atmosphere. Obviously, human activity has a contributing effect on global warming. The issue is how much? It is true that when you find your self in a hole you should stop digging, but are we digging with a teaspoon, a shovel, or a backhoe?

I'd love to give you answers to these and I will do some research, but I am here to tell you that the answers are not easy to find. In part because the press is talking so much about the wrong questions.

I'll post more as I can figure some of it out.

The Computer is Dead, Long Live the Computer

Finally the follow up to my February 2007 Post.

If you look up the definition for Computer you will find that it has a second definition that may surprise you. This is actually not the second definition, but the original definition of the term computer. You see a computer is "a person who computes; computist." So where did that come from? Well, back in the stone age before the personal computer or fire, accounting departments used to be full of people who would keep the books. These people were not accountants, an accountant is someone you hire to help you make sense of all the numbers. An accountant can tell you if you are making money and how much. This is often much more difficult than you might imagine. It is so difficult that we still don't have computer programs that will do it. Instead we have Excel.


Obviously Excel means that we no longer need computers (the human version) in an accounting department, right. True, but only sort of. It seems that though we no longer have rows of people with their heads bent tallying columns of numbers, we do need to have book keepers. When personal computers began to automate what people computers were doing we quickly learned that they had task that were easy and tasks that were exceptionally hard. By easy and hard here I am talking from the computers perspective, not the people's. Accurately tallying a column of numbers is difficult for many people. When you add in the need to do it quickly, and without error it is nearly impossible for most people. Computers find this very easy however. Other tasks that the computers (people not machine now) were doing are very difficult or impossible for a PC. Tasks such as deciding which category an entry belongs in. Maybe you can create a rule like all invoices from Krispy Kreme Doughnuts goes into business expenses, but these type of rules typically break down quickly. So even though the people in the accounting department are not doing sums anymore there are not that many fewer people in the accounting department than there were prior to the PC revolution.

Why do I bring this up? Especially in connection to software development. The issue is similar. Back in February of 2007 I wrote an article about a company called Intentional Software. I talked about the issues facing software development and why I didn't think Intentional Software was going to change the software industry. The trick is that they are trying to create a piece of software that makes software development easy, and software development is hard. Software development is not hard because we have to learn cryptic languages to be able to talk to the computer. Software Development is hard because it is process automation and processes are inherently hard. We are back to making all those fuzzy decisions like what category does this invoice go into?

I am currently working on a project to create a panel that is ridiculously easy to use. There are some particular challenges to this project as it is a computer that will be used on a shop floor. Keyboard and mouse are not completely out of the question, but it will have to work without them most of the time. This has lead me to think about what easy to use means. Here are my ideas.

The Interface Must be Apparent
This is why GUI programs are often seen as easier to use. They are more apparent. When people are working they do not want to go looking for things, they want to just do what ever they are doing. As shocking as this may be to computer programmers, most people do not enjoy spending time on a computer. Furthermore, most people find figuring out arcane steps to complete a process a bore and more than a little annoying. At any given moment the interface needs to make the option as obvious to the user as it can.

Actions Should Work with as Little Interaction as Possible
Every question that the user is asked is a distraction. When I ask my word processor to put today's date into a document, it should use the format that makes sense given my locale. It would be nice if I can then change the format. However asking me to choose the format every time I ask for the date is bad. The vast majority of the time I want the format that is the default for my locale (locale is language and country). Similarly when I save a file I should be asked where and what to name it once, not every time. After I have establish the files name and folder, I will rarely wish to change it. These decisions depend on the use case that is expected, but I think you see where I am going. Another aspect of this is that I should not have to go hunting through the menus to find something I want to do. If it makes sense in the current context then it should be available with one or two mouse clicks. Everything should be available through the keyboard. In the case of the project I am working on there needs to be a button on the touch screen.

Side Effects Should be Minimal
When I select an action I should not have to spend any time wondering what else this is going to do. Each command should be self contained. In this way the user can start to think in terms of steps to accomplish some goal. If too much is wrapped up in a single action then the user can become torn between what they want to do and the side effect they wish to avoid. Furthermore, having minimal side effects fosters the user to use the software in way that you did not expect.

Action Should be Complete
This is the balance to the minimal side effect rule. The user should not have to build what they consider to be simple actions. To save a file it would hardly make sense to ask the user to select the default directory. Then go through the menus and set the current file name. Finally to go through the menus again and choose save. In addition to being cumbersome, this makes it more likely that the user will miss steps or in other ways mess up the process. There is a balancing act between minimal side effects and complete actions that is often difficult to manage. However when it is managed properly there is a real sense of power in using that software.

Software must be Reliable
You may not think of this as an ease of use requirement. You may say all software should be reliable and I would agree. In this case, however, reliability does enhance the usability of software. The goal of easy to use software is to create an environment where the user can focus on their goals rather than on using the computer. Any time the software fails it interferes with this goal. Furthermore, it can cause the user to second guess themselves when they are attempting to accomplish something. They will wonder if this mouse click is going to cause the program to exit.

So what do we need to make it easier to program? We need an environment that does these things, but we also need an environment that can choose data structures and algorithms well. Yes I know you are an Object Oriented Programmer (OOP), so you don't use data structures or algorithms. Actually when you create an instance of that Set class you are choosing a data structure, and when you choose to use the Singleton design pattern you have chosen an algorithm. These decisions are difficult. They tend not to have pure right/wrong answers. They tend to be highly dependent on how other questions where answered. Furthermore they are dependent on how the process "should work". This is almost never know in a definitive way.

The short answer is that though software automation will continue to improve. Libraries will become larger and more powerful. Software Developers will not be wholesale replaced by machines any time soon. It is difficult to think about the things that need to be decided when developing software. Currently people still think much better than machines.

Unintentional software

In my last post I came down fairly hard on a company called Intentional Software as I said in the last entry I do not mean to single them out. I do think they are jumping a bit far. I think the next step up the abstraction ladder is Component Base Programming. This is neither my idea nor a new idea. Building software from reusable and cheap components has been the dream of software developers essentially from the beginning. So far the components are not cheap (most projects still build their own) or particularly reusable, but they still have many strengths. In my last job I had the experience of working on a project that was using Component Oriented practices. After a couple of years of work on the system a team of about 10 developers had created 78 discrete components. What was more amazing is that we had created 4 distinct (though related) applications. I had the experience of prototyping one of those applications. I was able to pull together several of the existing components, write one that combined and extended the functionality of three other components and wrap it all in enough code to create a functioning prototype of the new application in 2 weeks. It really blew me away how powerful this paradigm is.
This is what software developers have been looking for, except that we were creating our own components. We had also created our own framework for the components to live within. The framework incorporated both the services necessary for components (creation, version control, etc) and an inversion of control container. This allowed us to inject customer specific functionality into the architecture at the component edges. The project went on to be a success with about 3 months work to complete the first customer implementation. The next implementation took 7 weeks.
The "failing" of component oriented development is that this is still work being done by programmers. So it does not reach as far as "non-programmer" writing a system, but it is definitely a major step toward that goal. I am hesitant to call it a failing because it is a major step forward. So what is the next step toward non-programmer programming? Well as the question implies the problem is in understanding the term non-programmer.
If someone takes a set of actions that result in a program that does something, isn't that programming? I think, as I said previously, that the core of this goal is flawed. Someone who has learned enough to cause the computer to accomplish some goal is a programmer. The issue is not to remove the programmer, but to allow a business expert to become a programmer. The goal is to create an environment that is easy enough to use that the people closest to the problem can work on automating it directly. I did not say that the environment had to be easy enough to learn, but easy enough to use.

In my next post I will talk about the difference between easy to learn and easy to use. I will also talk about why Excel hasn't put more people out of work.

Everything Old is New Again

One of the aspects of having been involved with computers for 20+ years that I enjoy is being able to see how things keep coming back around. One story that keeps coming back is how "non-programmers" are going to be able to program one day. I just read a story in the New York Times about Intentional Software. For those too who choose not to read the article, Intentional Software is a company that is in the process of making a set of tool that allow "non-programmers" to define the intention they have for how the software should work. Then those intentions are rendered as code. Finally, the code is compiled into an executable.
Let me start by saying I think this is a great idea. Despite the fact that it may mean fewer software development jobs in the long run, I still think it is a great idea. Unfortunately it will not work. You see, it attacks the wrong problem. The article describes three advantages of Intentional Software's solution:

1. The people who design a program are the ones who understand the task that needs to be automated.
2. The design can be manipulated simply and directly, rather than by rewriting arcane computer code.
3. Human programmers do not generate the final software code, thus reducing bugs and other errors.

The first advantage is simply not true. The idea that there is a domain expert that completely and thoroughly understands any business process is a myth. There are certainly people who understand some aspects of a process, but there is no one who understands the process completely. Furthermore, even if you could find one person who understands the process well, that person has an ideal implementation in mind. This is almost always not what the business is doing now. What the business is doing now is a result of the interplay of many people who have different ideas of what the business should be doing now. Some of these ideas are well thought out and the result of debate and/or experience. Others are a result of people wanting to get to lunch, or leave at a certain time. This mess is what defines the non-automated process. This is why collecting requirements for a software project is so difficult. Since there is no one who understand the entire process, many people are interviewed to collect the requirements. The requirements are collected based on how clearly each person can explain their views on how things should be done. They are also effected by how well each person can sell their ideas. Even if the system is implemented close to this description of the intent of the software, if the right people were not involved, the final system will be less than ideal. Of course there is also the problem that while the software is produced and tested the needs of the company are in flux as a result of many internal and external factors.

The next advantage is closer to the mark. "Arcane computer code" is what I do 5 days (sometimes more) a week. I find it neither arcane nor do I think of it as "computer code". It is difficult for some people to understand this, but in many ways I am more comfortable with say C# than I am with English. The advantage of an arcane computer language is that they tend to be very precise and limited. I know that the computer is going to do in most cases. It is true that sometimes I overlook or confuse things, but I do this much less often in C# than I do in English. The higher the abstraction, the more effort goes into understanding what is actually going to happen. Very high level languages have the issue that one must be familiar with the subtly of the language. There is a greater risk that my interpretation will not match the interpretation of the computer. Furthermore, high level abstractions tend to be very domain specific. Take an inventory abstraction for instance. Is it tracking the inventory in a vending machine, a retail store, a warehouse, a factory? Even within these distinction there are distinctions. Is the vending machine selling discrete items (i.e. candy bars, sodas, etc) or is it vending coffee (by the cup). Is the warehouse holding drugs, steel, food? Each of these variations must be represented somehow. Either there needs to be a huge third party ecosystem that creates each of these specialized abstractions, or the abstraction must be generic enough that it is adequate to cover all the possibilities. Adequate solutions are not good enough to differentiate your business, by the way. This is why most companies use software, after all. They want a competitive advantage. So they also need to be able to create their own high level abstractions.

This brings us to the final advantage. Since we have removed the "human" element from the process there will be fewer errors right? Not really. Firstly we have not removed humans we are using a different set of humans. If we have a high level abstraction that is widely applicable we will reduce errors, because we have a large number of people that are sharing the effort to find errors in code. This is why GUI tool kits actually do make programmers more productive. As you move the abstraction layer up you naturally are working with smaller audiences. The inventory abstractions I spoke of in the last paragraph show how high level abstractions must target a specific domain to be of real value. Even then, the parts of the software that really make a difference tend to be unique to a single product. That is why they make a difference. So now you are back to using software produced in house for one application. All of the advantages of having many people reviewing and correcting the software go away.

So where does that leave us? Software is hard because:

1. No one really knows what it should do
2. Software that give me a business advantage is different than software that other people are using
3. There are technical challenges in creating software

Intentional Software's solution will help with item three, but does little or nothing to help with the first two. I don't mean to pick on them. There has been any number of technologies that where supposed to make software development easier that failed prior to them. If one looks at Delphi, Visual Basic, or an ancient Dos programming tool called Layout, they all had the same promise, and none of them changed the world. They all try to make the act of developing software easier, but if you look at item number one, it has nothing to do with the act of writing software, except that it is a prerequisite. Item two means that any sort of software factory won't really solve the problem either. Solving the third problem is helpful, but not enough to change the experience significantly. Software is hard for real world reasons.

To Paraphrase Albert Einstein, "Software Development should be as simple as possible, but not simpler".

Well, my experiences here have proven something I already knew to be true. Nobody cares about epistemology. OK, so nobody is overstating it a bit, but very few people do anyway. So I will turn my attention from all things epistemological, to a more general mélange of stuff about programming, politics, Christianity and epistemology. Hope you and yours are having a Joyous Christmas and hope to see you in the new year.

Is faith the same as Ignorance?
On a TV show that I like to watch, the main character made the comment “Faith, isn’t that the same as ignorance?”. That line has stuck with me. Obviously, I am interested in knowledge and to an extent against ignorance. So this idea that faith and ignorance are the same has really caused me to think. Is faith the same as ignorance? Surely not, but it is not enough for me to say that, I must be able to support that idea. So let us look at what ignorance and faith are, and how they relate.
Let’s start with ignorance. I suspect that you know, or think you know, what ignorance is. The simplest definition of ignorance is a lack of knowledge. Given that definition all of us are ignorant. I know nothing about nuclear medicine, or asian music, or any number of other subjects. There is a broader definition of ignorance, which is choosing not to know. This is the derogatory use of the word. Obviously, people who choose not too learn are making a mistake. Still, not learning about things outside of our profession and interest is perfectly reasonable, and inevitable. There is a good side to ignorance. Ignorance is where all new scientific and philosophic though comes from. Ignorance keeps us humble. The double doctorate professor who has too take his car to a mechanic without a high school diploma, is reminded that despite his knowledge there is still much he does not know. So like all things there is a good and bad to ignorance.
What then can we say about faith? What is faith? Faith is confidence or trust in a person or thing. Again given that definition we all have faith. When you stand up you have faith that the floor will hold you. You have faith that the Sun will continue to warm the Earth. You have faith that time will continue to run, etc. So faith, like ignorance is a matter of fact. Not something one can avoid. So how do we come by faith? We trust the floor because the floor has always held us up. We trust time because we have never experienced it to vary. Faith typically come from experience. Faith does not typically exist without support, just not with empirical support. Faith is confidence that is not supported by logic or reason. This does not mean that faith is not logical or reasonable, it means that faith is the set of beliefs that are not supported by logic or reason. Faith is all the things you believe without proving them.
Do scientists have faith? Yes, there are many things that scientists have faith in. The most obvious is the scientific method. Why is science done the way it is done? Because we believe it to be the most successful process for achieving knowledge and understanding. Scientists have faith that the universe is a very consistent place. That is they believe that the same rules of physics work here and in other galaxies. We have only limited evidence that this is true. We have never been to other galaxies after all. Scientists trust their observations. Observations are notoriously unreliable and scientists have come up with ways to help mitigate the errors, still what we see is often not what really happens. So we see that science relies a great deal on faith.
Still there is faith and there is Faith. What is Christian Faith? Let's start with the Bible definition of Faith. Hebrews 11:1 say "Now faith is assurance of things hoped for, a conviction of things not seen." I think this is the same thing as faith in the floor. I am confident that God has a plan and that it will be for my benefit. I am convicted by the power of God. Even when things do not appear to be going my way. Is it foolish, or ignorant of my to have faith in God? Let's assume for the moment that there is a God and that he is omnipotent (all powerful), omniscient (all knowing) and omnipresent (everywhere at once), what else would there be to have faith in? Given an all knowing, all powerful, God present everywhere at once, there is no other logical place to put ones faith. So it all comes down to the question is there such a God. I believe there is, but that is based on my experiences.

You will have to ask God for yourself.

Pat O

I have been reading Brian Glass' Blog about the descent of man and how that should effect your choices in the election tomorrow. In general I agree with his points (If you want to read them you should start at the beginning here). Man is very gifted at making a mess. I blame this on our sinful nature. It is not a question of knowing what is right. It is a question of doing what is right.
How often have we hear, or even said ourselves, I knew it was wrong/bad idea/etc, but I just couldn't help myself? Still, it helps to be informed, if one can be. I believe that I have the power to choose the right through the Holy Spirit. Perhaps you choose the right through self discipline, or maybe thought a Ouija board. Still most people base their decisions. on some type of information. So where do you get the information that you make election decisions with? From the candidates? Isn't that a little like having the fox guard the hen house? So from Political Action Groups that buy time on the television? This is not much of a step up. The local newspaper? Again not typically the bastion of rational fact that one might hope for. So if anyone is reading this I am interested. As you return from your civic duty of voting, please share with us how you go about deciding who to vote for? I am not asking who you voted for, but how you gather information to decide. What information sources do you trust?

May God have mercy on us tomorrow.
Pat O

Not All Bad

My last post might get one to thinking that I am down on Humanity. This was not my intent. It's simply an explaination why so many efforts that start out with the best of intentions go wrong. Human history is full of institutions that are created for good reasons, but still these institutions eventually becaome a problem in their own right. Why? Because they get big. As the size goes up, the effort required to fight the effects predicted by Chaos Theory increase. Eventually there is too much force, or too little resistence and they fall. Many remain fallen, some are reborn by a new vision and a new resistence. So what are we to do?
The secret is to keep things small. It is difficult to hide sloth or greed in a group of 5 or 10 people. As I said before these issues are hard to resist in an organization of hundreds or thousands. So we should look to create organizations that have a few members, say less than 100. We should focus on small business, local government, etc. It is in these organizations that innovation often occurs. These organizations are often closer to the problem and therefore can respond in a more surgical way.
Understand that this is not the most efficient approach. Small businesses tend not to make as much money. Passing laws at the local level is often just as much work as passing a law at the State level with substantially less pay off. Still, I believe this is where our thgoughts and time should be focused. Of course, to really make this work there needs to be an effort to return control the the local governemnts. More on that another time.

Tanks for your support
Pat O
  _      _      _
 /*\==  /*\==  /*\==
<ooo>  <ooo>  <ooo>

Chaos Theory and Organisations

As I call my blog Current Chaos I though it is appropriate to share my ideas about how chaos theaory and human organization come together. I should start by pointing out that I am niether an expert in chaos theory or organizational theory. I am a computer geek. I work in a cube. I prefer the company of my computer to most anyone outside of my immediate family (I love my wife and daughters very much, just in case they decide to read this :-). In my spare time I write software for free, or design, build and program robots. By now you should be getting the picture that I really am a geek full time. Still, I do look up from the monitor every now and again. One thing likely to get my attention is science. In particular any sciencentific news that effect how people look at problems. I see a lot of similarity between the scientific method and developing software. Therefor I am always interested in how scientist think about the world, and how they present the complex systems they are trying to understand. So when I heard about Chaos Theory I figured it was worth a look.
One aspect of chaos theory as I understand it is that complex systems are expected to show the traits shared by most of thier parts. That is not to say the average behaviour, but that the most common traits will show through. One way to think about this is to imagine a box full of Mexican Jumping Beans. For those that do not know, Mexican Jumping Beans are beans that spontaneously "jump" about a quarter inch into the air. The direction of the jump is essentially random (it's actually chaotic, but that is a discussion for another article). So you can imagine a box full of beans bouncing up and down in random directions, the result of that movement would be a pretty even spread of the beans throughout the box. Now if we introduce a trait to the jumping, say by lifting one side of the box an inch, obviously we would expect the beans to collect at the low side of the box. This is true even though some of the beans would make pretty good headway toward the highside of the box, eventually most of the beans would be at the low side.
Now let's consider something more interesting than a box of Mexican Jumping Beans, (not that I have any issues with Mexican Jumping beans mind you :-) let's consider an organization. The parts of an organization are people. No matter how else you think of an organization being put together, ultimately they are made up of people. So what traits, if any, do people share. This is where my Christian beliefs come into play. As a Christian I believe that all people have a fallen nature, that is that all people are sinful by nature. This is not to indicate that I have lost all faith in my fellow man. I know and work with a number of people that are honest, hardworking folks. Still the traits that are shared by most people are the seven deadly sins. The sin nature of people would naturally be the common trait. What would we expect given this hypothesis?
To answer that lets ask a different question. How would you describe most organizations? Large corporations, governments, any other large organizations, are typically described as selfish, greedy, mean and even cruel. This is unfortunately what this hypothesis expects. The sinful nature of human beings comes out as the number of humans involved increases. So small organizations good, big organizations bad.

Well that's sure to be controversial, but then I have never been afraid of controversy.

Hope you are yours are well,

In honor of talk like a Pirate day earlier this week my daughter suggested I get a Priate name. Here it is:

My pirate name is:

Iron Sam Rackham

A pirate's life isn't easy; it takes a tough person. That's okay with you, though, since you a tough person. You have the good fortune of having a good name, since Rackham (pronounced RACKem, not rack-ham) is one of the coolest sounding surnames for a pirate. Arr!

Get your own pirate name from piratequiz.com.
part of the fidius.org network

Here you will find the wondering and chaos that is my current thought process. I also hope to post ideas about a Tactical Turn Based Hex Map war game engine that I will likely call smallwarz. In between look for the occasional diatribe about the insanity that is the software industry. Hopefully it will prove useful, entertaining, or ideally both.