Every teardrop is a waterfall

I was listening to Coldplay the other day and got to thinking about waterfalls. The classic waterfall diagram is written something like this:

Analysis
    leading down to...
Design
    leading down to...
Implementation
    leading down to...
Testing.

The Testing phase at the end of the process is perhaps the biggest giveaway that something is very wrong. In waterfall, the testing phase at the end is what's known as a euphemism. Or, more technically, a lie. Testing at the end of waterfall is really Debugging. Debugging at the end of the process is one of the key dynamics that prevents waterfall from working. There are at least two reasons:

The first is that of all the activities performed in software development, debugging is the one that is the least estimable. And that's saying something! You don't know how long it's going to take to find the source of a bug let alone fix it. I recall listening to a speaker at a conference who polled the audience to see who'd spent the most time tracking down a bug (the word bug is another euphemism). It was just like an auction! Someone called out "3 days". Someone else shouted "2 weeks". Up and up it went. The poor "winner" had spent all day, every day, 9am-5pm for 3 months hunting one bug. And it wasn't even a very large audience. This 'debug it into existence' approach is one of the reasons waterfall projects take 90% of the time to get to 90% "done" (done is another euphemism) and then another 90% of the time to get to 100% done.

The second reason is Why do cars have brakes?. In waterfall, even if testing was testing rather than debugging, putting it at the end of the process means you'll have been driving around during analysis, design and implementation with no brakes! You won't be able to stop! And again, this tells you why waterfall projects take 90% of the time to get to 90% done and then another 90% of the time to get to 100% done. Assuming of course that they don't crash.

In Test First Development, the testing really is testing and it really is first. The tests become an executable specification. Specifying is the opposite of debugging. The first 8 letters of specification are S, P, E, C, I, F, I, C.
A test is specific in exactly the same way a debugging session is not.

Coupling, overcrowding, refactoring, and death

I read The Curious Incident of the Dog in the Night Time by Mark Haddon last week. I loved it. At one point the main character, Christopher, talks about this equation:

P_g+1 = α P_g (1 - P_g)

This equation was described in the 1970s by Robert May, George Oster, and Jim Yorke. You can read about it here. The gist is it models a population over time, a population at generation_g+1 being affected by the population at generation _g. If there is no overcrowding then each member of the population at generation g, denoted P_g, produces α offspring, all of whom survive. So the population at generation g+1, denoted P_g+1 equals α P_g. The additional term, (1 - P_g ) represents feedback from overcrowding. Some interesting things happen depending on the value of α

• α < 1: The population goes extinct.
• 1 < α < 3 : The population rises to a value and then stays there.
• 3 < α < 3.57 : The population alternates between boom and bust.
• 3.57 < α : The population appears to fluctuate randomly.

In biological systems each generation has a natural lifespan. The cycle of death naturally helps to reduce overcrowding. But even with the inevitable death of each generation, the system's behaviour is delicately poised. Once α gets into the 3 - 3.57 range the growth starts to outpace the death leading to a rising population, but this leads to overcrowding which reduces the growth leading to a falling population. That reduces overcrowding, and growth rises again. The population repeats in a stable up/down cycle. But only because of the inevitable death. And only while the rate of death is sufficiently high to maintain the cycle. Once α gets beyond 3.57 the rate of death can no longer regulate the increased rate of growth and the system destabilises.

You can think about the process of writing software with this equation.

You can think of over-crowding as being analogous to over-coupling. We feel that a codebase is hard to work with, difficult to live in, if it resists our attempts to work with it. When it resists it is the coupling that is resisting.

You can also think of death as being analogous to refactoring. Just as death reduces overcrowding, so refactoring reduces coupling.

Refactoring is a hugely important dynamic in software development. Without refactoring a codebase can grow without check. Growing without check is bad. It leads to overcrowding. Overcrowding hinders growth.

Incremental development

Out of the crisis

is an excellent book by W. Edwards Deming (isbn 0-911379-01-0). As usual I'm going to quote from a few pages

All industries, manufacturing and service, are subject to the same principles of management.

Quality comes not from inspection, but from improvement of the production process.

Today, 19 foreman out of 20 were never on the job that they supervise.

Fear amongst salaried workers may be attributed in large part to the annual rating of performance.

Absenteeism is largely a function of supervision. If people feel important to a job, they will come to work.

He that would run his company on visible figures alone will in time have neither company nor figures.

There has never been a definitive study of quality in the dental profession; nor is there likely to be one. Partly because they tend to work alone, dentists resist the idea of being evaluated, or even observed, by others.

Where there is fear, there will be wrong figures.

It is well known that rework piles up: no one wishes to tackle it.

This company had been sending a letter to every driver at every mistake. It made no difference whether this was the one mistake of the year for this driver, or the 15th: the letter was exactly the same. What does the driver who has received 15 warnings, all alike, think of the management?

Cause is effect and effect is cause and vice versa

Defects cause lateness.

The more defects code has the more time and effort it takes to get it to done. This seems a self-evident truth. But beware! The Causation Fallacy says it is not easy to know what is cause and what is effect. If a feature misses its deadline pressure often builds to ensure it doesn't miss the next deadline. And under pressure people don't think faster. Extra pressure usually increases the likelihood of defects. This suggests

Lateness causes defects.

So do defects cause lateness, or does lateness cause defects? Or do they rotate around each other like partners on a dance floor?

Sprints, time-boxing, and capacity

A team is doing Scrum with 3 week sprints. Suppose at the end of a sprint they've got nothing to done. What should they do? There's a strong temptation to ask for more time. To make this sprint a 4 week sprint. Most of the work in progress is 90% done they say. Another week and things will have got to done, they say. It seems reasonable.

Trying to run systems beyond their capacity is not a good idea. In this situation Scrum's fixed-duration time-box constraint has served it's purpose admirably. The problem is not the choice of 3 weeks. Changing 3 weeks into 4 weeks is not addressing the problem. The problem is the team planned to pull in an amount of work and get it to done in 3 weeks. But they're not yet in control of their process - they don't know what their capacity is. They pulled in more than 3 weeks worth of work. Probably a lot more. But we just don't know!

In The Toyota Way, Jeffrey Liker writes:

Taiichi Ohno considered the fundamental waste to be overproduction, since it causes most of the other wastes.

Advice from a genius with a lifetime's experience. Toyota manufactures cars. It makes cars. Its production line is an actual line. If manufacturers are prone to overproduction imagine how much more prone software developers are! The things we make are not even physical things. In software, things are mostly invisible. It's difficult to manage what you can't see. In Quality Software Management volume 2, First-Order Measurement, Jerry Weinberg writes:

Without visibility, control is not possible.
If you can't see, you can't steer.

Rather than asking for another week, the team should really be thinking about addressing their real problem. Their real problem is that they're pulling in too much work. They have to somehow learn to pull in less work. So they can start to be in control of their process rather than their process being in control of them.

Culture quotes

I've collected together some culture quotes from previous book snippets:

Culture makes its presence known through patterns that persist over time. (from here)
Jerry Weinberg

Building a culture takes years of applying a consistent approach with consistent principles. (from here)
Jeffrey Liker

Culture makes language, then language makes culture. (from here)
Jerry Weinberg

Culture is what we do when we do not consciously decide what to do. (from here)
Russell Ackoff

Consultants who see culture change as something distinct from the work and, as a corollary, something that can be the subject of an intervention, miss the point. When you change the way work is designed and managed, and make those who do the work the central part of the intervention, the culture changes dramatically as a consequence. (from here)
John Seddon

One aspect of almost every culture is the belief in the utter superiority of that culture. (from here)
Donella Meadows

Culture change is free [because] it's a product of the system. (from here)
John Seddon

Culture is changing faster than it has ever changed before...what once took many generations of gradual development is now attempted by a single individual. (from here)
Christopher Alexander

Hunger is the best source

I've previously blogged about being taught ITA spelling at primary school. About how it causes me spelling problems. I was reminded of this when speaking to Geir Amdal at the excellent Agile Coach Camp in Oslo. Geir showed me this wonderful blog post with lovely twist on the famous quote:

Knowledge is power.
Francis Bacon

It reminded me of something my Mum used to say to me when I was little:

Hunger is the best source.

For many many years I didn't understand what she was saying. I was seeing the word sauce as source. She was actually saying:

Hunger is the best sauce.

Food tastes better when you're hungry. Reflecting on my confusion I realize I'm actually quite proud of this mistake. This was a long time ago remember. I was a small boy at the time. Even then, it seems, software was calling me.