Being wrong

is an excellent book by Kathryn Schulz (isbn 978-0-06-117604-3). As usual I'm going to quote from a few pages:
One extremely good way to become wedded to a theory you just idly expressed is to have it contradicted... from noncommittal to evangelical in a matter of milliseconds.
We take our own certainty as an indicator of accuracy.
The instant an implicit assumption is violated, it turns into an explicit one.
When we ask people to look for something specific they develop a startling inability to see things in general.
The genius of statistics was that it did not ignore errors, it quantified them. [Laplace]
In ancient Indo-European, the ancestral language of nearly half of today's global population, the word 'er' meant "to move", "to set in motion", or simply "to go."
It is all too common for caterpillars to become butterflies and then maintain that in their youth they had been little butterflies.
the stakes of our mistakes.
Realizing that we are wrong about a belief almost always involves acquiring a replacement belief at the same time.
"fallor ergo sum" (I err, therefore I am) [St Augustine]
When other people reject our beliefs, we think they lack good information. When we reject their beliefs, we think we possess good judgement.
As with so many systems, the strengths of inductive reasoning are also its weaknesses. For every way that induction serves us admirably, it also creates a series of predictable biases in the way we think...
When a framework serves us well... we call it brilliant, and call it inductive reasoning. When it serves us poorly, we call it idiotic, and call it confirmation bias.
Being wrong can be funny; other people being wrong can be very very funny.
Without some kind of belief system in place, we wouldn't even know what kinds of questions to ask, let alone how to make sense of the answers.

Adapt - why success always starts with failure

is an excellent book by Tim Harford (isbn 978-0-349-12151-2). As usual I'm going to quote from a few pages:
Cross the river by feeling for stones [Deng Xiaoping]
Accepting trial and error means accepting error.
Darwin, a meticulous observer...
The art of success is to fail productively.
Complexity is a problem only in tightly coupled systems.
Make sure you know when you've failed, or you will never learn.
What Palchinsky realised was that most real-world problems are more complex than we think. They have a human dimension, a local dimension, and are likely to change as circumstances change. His method for dealing with this could be summarised as three 'Palchinsky principles'
  • seek out new ideas and try new things
  • when trying something new, do it on a scale where failure is survivable
  • seek out feedback and learn from your mistakes as you go along
If we are to take the 'variation' part of 'variation and selection' seriously, uniformly high standards are not only impossible but undesirable.
When John Nagl served in Baghdad in 2003, he found that while his young inexperienced soldiers had the authority to kill, he - a major with a doctorate and a decade of experience - didn't have the authority to print his own propaganda pamphlets to counteract the clever PR campaign that the local insurgents were running.
Speciation - the divergence of one species into two separate populations - rarely happens without some form of physical separation.
Tight coupling means the unintended consequences proliferate so quickly that it is impossible to adapt to the failure or to try something different.
The first thing Timpson does when it buys another business is to rip out the electronic point-of-sale machines (there are always EPOS machines) and replace them with old fashioned cash registers. 'EPOS lets people at head office run the business', explains John Timpson. 'I don't want them to run the business.'

John Timpson describes one instance where he couldn't buy half-price happy hour drinks at a hotel bar, because midway through giving his order, the hour ended and the bar's computerised sales system refused to allow the half-price offer to be applied.

Timpson's company training manual describes the twenty easiest ways to defraud the company, making it clear that the company understands the risks it is running and trusts its employees anyway - and many people respond to being trusted by becoming more trustworthy.
A central point of the corporation, as a legal structure, is that it is supposed to be a safe space in which to fail. Limited liability companies were developed to encourage people to experiment, to innovate, to adapt - safe in the knowledge that if their venture collapsed, it would merely be the abstract legal entity that was ruined, not them personally.
Fail better. [Samuel Beckett]

The conquest of happiness

is an excellent book by Bertrand Russell (isbn 0415378478). As usual I'm going to quote from a few pages:
It will be found that a quiet life is characteristic of great men, and that their pleasures have not been of the sort that would look exciting to the outward eye. No great achievement is possible without persistent work.
Worry is a form of fear, and all forms of fear produce fatigue.

Every kind of fear grows worse by not being looked at.
More and more it becomes possible to choose our companions on account of congeniality rather than on account of mere propinquity.
Like the heroes of Valhalla who spent every day hunting a certain wild boar, which they killed every evening but which miraculously came to life again in the morning…
Fundamental happiness depends more than anything else upon what may be called a friendly interest in persons and things.

To like many people spontaneously and without effort is perhaps the greatest of all sources of personal happiness.

The secret of happiness is this: let your interests be as wide as possible, and let your reactions to the things and persons that interest you be as far as possible friendly rather than hostile.
The more things a man is interested in, the more opportunities of happiness he has.
Events only become experiences through the interest that we take in them.
In the best kind of affection a man hopes for a new happiness rather than for escape from an old unhappiness.
Work, therefore, is desirable, first and foremost, as a preventive of boredom, for the boredom that a man feels when he is doing necessary though uninteresting work is as nothing in comparison with the boredom that he feels when he has nothing to do with his days.
Two chief elements make work interesting: first, the exercise of skill, and second, construction.

All skilled work can be pleasurable, provided the skill required is either variable or capable of indefinite improvement.
We may distinguish construction from destruction by the following criterion. In construction the initial stage of affairs is comparatively haphazard, while the final state of affairs embodies a purpose; in destruction the reverse is the case: the initial state of affairs embodies a purpose, while the final state of affairs is haphazard.
Destruction is of course necessary very often as a preliminary to subsequent construction; in that case it is part of a whole which is constructive.
Few things are so likely to cure the habit of hatred as the opportunity to do constructive work of an important kind.
I should seek to make young people vividly aware of the past, vividly realising that the future of man will in all likelihood be immeasurably longer than his past, profoundly conscious of the minuteness of the planet upon which we live and of the fact that life on this planet is only a temporary incident; and at the same time with these facts which tend to emphasise the insignificance of the individual I should present quite another set of facts designed to impress upon the mind of the young the greatness of which the individual is capable, and the knowledge that throughout all the depths of stellar space nothing of equal value is known to us.
Happiness must be, for most men and women, an achievement rather than a gift of the gods, and in this achievement effort, both inward and outward, must play a great part.
The only man totally indifferent to power is the man totally indifferent to his fellow-men.
It is better to do nothing than to do harm. Half the useful work in the world consists of combating the harmful work.

smell-driven development

After much research by our scientists I can finally report a major advance in software methodology - smell driven development.

Our scientists have perfected a set of software metrics which accurately monitor the health of your software. These metrics are gathered in real time by our patented custom SmellWare servers. Each SmellWare server (prices to be announced shortly) is connected via USB to three glass vials and a sophisticated spray-atomizer.
  • When the metrics read by the SmellWare server indicate your software is in a poor state, the atomizer will spray scent from the first glass vial.
  • When the metrics indicate a further deterioration, the atomizer will spray scent from the second glass vial.
  • When the metrics indicate a really disgusting codebase, the atomizer will spray scent from the third glass vial.
By choosing smells with an increasing tendency to get on the developers' olfactory nerves you can "visualize" the state of your codebase in a way that cannot be ignored!

The genius of this approach is that smell, being the oldest sense, has had the longest to evolve. The olfactory sense is, at the same time, both the most primitive sense and the most sophisticated. Even in humans about 1 in 50 genes is devoted to smell-related protein receptors!

Our odour scientists have studied the four aspects of threshold and tolerance; odour concentration, odour intensity, odour quality, and hedonic tone, and after extensive field testing the first (cheese based) vials are ready.
  • Vial 1 (Yellow) is based on Pong-Leveque (Normandy) and is ranked "distinct, strong, and unpleasant" on the Odour Awareness scale.
  • Vial 2 (Amber) is based on Epoisses de Pieds (Southern Italy) and is ranked "very strong, liable to cause rashes and sore eyes". (Epoisses de Pieds is banned from public transport in Italy).
  • Vial 3 (Red) is based on Peixe-Podre et Meia (Northern Portugal) and is ranked "intolerable, liable to cause vomiting, diahorrea, and heart attacks". (Peixe-Podre et Meia is based on Blue-vein Smegma which had a 97% mortality rate before the EU made its manufacture a federal offense punishable by 25 years imprisonment.)
Believe us when we tell you that once your developers have experienced Peixe-Podre et Meia (Red) they will never again sit idly by while your codebase is merely in Epoisses de Pieds (Amber) state!

Ivan Marks - the people's champion

is an excellent book from Media Press (isbn 978-0-9567015-9-6). As usual I'm going to quote from a few pages:
Young anglers were encouraged and taught the basic skills by more experienced anglers. The club would fish matches where novice anglers were paired off with the top men. For two hours the older angler sat patiently with his pupil explaining the correct tactics to use.
There are, of course, times of the year when the facility to make maggots sink at a reduced rate of fall is a great advantage, especially in still water.
A 4lb chub will eat a half-pint of casters.
His [Benny Ashurst] maxim was always to keep the fish feeding for as long as you can.
Tanked fish have been fed yellow maggots and given an electric shock every time the made a move to eat one. The same fish were fed white maggots and allowed to take what they liked without interference. Later, when those same fish were fed mixed yellow and whites they wouldn't look at the yellows - and can you blame them? That experiment proves that fish can learn.
The real secret with loose-feeding is regularity.
You must not panic and fool yourself into believing that the more stuff you throw in, the more fish will come out.
I told the Italian team manager that I might have taken the individual honours had I been able to stay catching bleak of around that weight. "No Ivan," said the Italian, "it isn't possible!" He explained that my 1oz bleak was a fat and none too healthy fish, and he was right. It looked spawn-bound. In fact he said it had a worm infection. And he explained that fish in that condition are not fit fish. They don't move to the bait fast enough - even if there are enough of them - to allow true speed fishing. Which helps explain just how deeply the Italians have delved into this type of fishing.
A barbless hook punctures the bait, but fills the hole it makes.
Roach and bream don't grab your bait with their teeth. They simply inhale water into their mouths and this sucking process puts the bait where you want it to go.
The biggest risk of breaking a line comes from shock impact. Sudden shock is to be avoided at all costs.

Fun and learning at Ericsson

I had the pleasure of teaching a TDD C++ course at Ericsson's Jorvas centre in Helsinki this week. As usual I made heavy use of cyber-dojo. Above is a screen shot of the dashboard of one of the practice katas. My evals were 5.8 for the course as a whole and 5.9 for teaching skills (out of 6). Some of the comments were:
  • Very clear.
  • You really got us understanding TDD.
  • Transformation of the mindset became clearly visible during the course.
  • Thanks!
  • Best course I've taken at Ericsson, thank you.
  • The cyberdojo is an excellent environment.
  • Very good hands on training.
  • cyber-dojo extremely nice.
  • It was fun!
  • You really need to try to do TDD in practice to see how beneficial and fun it is.
  • You made it visible in practice.
  • Fun fun fun.

Nicer HTML radio buttons

One of my pet peeves is plain html radio buttons. If I have a set of 5 radio buttons with 5 text labels of differing lengths then I want to be able to click anywhere on the shaded area and not just the text labels.

This is quite easy to achieve. Simply put each <input> into a <div>...
<div class="radio">
  <input type="radio" ... value="Red"/>
    <label>Red</label>
</div>
<div class="radio">
  <input type="radio" ... value="Amber"/>
    <label>Amber</label>
</div>
<div class="radio">
  <input type="radio" ... value="Green"/>
    <label>Green</label>
</div>
...and use a bit of jQuery
var $j = jQuery.noConflict();
$j(document).ready(function() {
  $j('div.Radio').each(function(n,node) {
    $j(node).click(function() {
      $j(this).children(':first').attr('checked', true);
    });
  });
});
Using a <div> changes the layout - so you might want to put each <div> into a table layout.

You can add some CSS to highlight the clickable whitespace...
  .Radio { background-color: Moccasin; }
  .Radio::after { content: "\00a0"; }
If you want to remove the bullet simply add the following
$j(document).ready(function() {
  $j('input[type=radio]').hide();
};
If you want to see an example of this, just try CyberDojo

Complexity and postmodernism - understanding complex systems

is an excellent book by Paul Cilliers (isbn 0-415-15287-9). As usual I'm going to quote from a few pages:
Complex systems operate under conditions far from equilibrium. There has to be a constant flow of energy to maintain the organisation of the system and to ensure its survival.
If resources were limitless, i.e., if growth could take place unrestricted, no meaningful structure would evolve. Boundaries, limits and constraints are preconditions for structure.
The theory of self-organised criticality tells us the following. A self-organising system will try to balance itself and a critical point between rigid order and chaos.
The classic definition of stability states that in a stable system small causes produce small effects.
The classical definition of instability, at least as used by Poincaré, is probabilistic. Unstable events are defined as events that have no observable cause.
The system of language transcends the choices of any individual user, and therefore has stability.
In the last analysis, the two facts are interdependent: the sign is exposed to alteration because it perpetuates itself.
When dealing with complex phenomena, no single method will yield the whole truth.

Bare bones ruby unit testing

This morning I spent a happy hour exploring a little of ruby's Test::Unit::TestCase. I started with this:
require 'test/unit'

class MyTest < Test::Unit::TestCase

  def test_one_plus_one_equals_two
    assert_equal 2, 1+1.1
  end

end
I wanted to see how little I needed to write my own, super-minimal implementation of Test::Unit::TestCase...
require 'test/unit'

class MyTest < MyTestCase

  def test_one_plus_one_equals_two
    assert_equal 2, 1+1.1
  end

end
After 204 traffic lights in cyber-dojo I ended up with this...
require 'assertion_failed_error'

class MyTestCase

  def self.test_names
    public_instance_methods.select{|name| name =~ /^test_/}
  end
  
  def assert_equal( expected, actual )
    message = 
      "#{expected.inspect} expected but was\n" +
      "#{actual.inspect}\n"
    assert_block(message) { expected == actual }
  end

  def assert_block( message )
    if (! yield)
      raise AssertionFailedError.new(message.to_s)
    end
  end

end

at_exit do
  ::ObjectSpace.each_object(Class) do |klass|
    if (klass < MyTestCase)
      klass.test_names.each do |method_name| 
        begin
          klass.new.send method_name
        rescue AssertionFailedError => error
          print "#{klass.name}:#{method_name}:\n" +
                "#{error.message}"
        end
      end
    end
  end
end
class AssertionFailedError < RuntimeError; end
which allowed me write...
require 'my_test_case'

class MyTest < MyTestCase

  def test_one_plus_one_equals_two
    assert_equal 2, 1+1.1
  end

end
and finally, I added this...
class MyTestCase
  ...
  def self.test( name, &block )
    define_method("test_#{name}".to_sym, &block)
  end
  ...
end
which allowed me to rewrite the test as...
require 'my_test_case'

class MyTest < MyTestCase

  test "1+1 == 2" do
    assert_equal 2, 1+1.1
  end

end
Fun :-)

The Vizzini school of bad management

  • Am I going mad or did the word "think" escape your lips?
  • Hurry up
  • Inconceivable
  • Faster!
  • You know what a hurry we're in
  • I'm waiting!
  • Catch up with us quickly
  • I do not accept excuses
  • Did I make it clear that your job is on the line?
  • There will be no one to hear you scream
  • Stop doing that. We can relax, it's almost over


renaming a sub-directory with a space in it in git

Suppose you have a git sub-directory with a space in it...
/outer/old name/
It appears you cannot rename the sub-directory with a space in it to a different name which also has a space in it. For example, the following does not work...
git mv -f "outer/old name/" "outer/new name/"
However, after much trial and error, I have found the following which does work...
git mv -f "outer/old name/" "outer/temp-name/"
git commit -m "rename subfolder with space part 1/2"
git mv -f "outer/temp-name/" "outer/new name/"
git commit -m "rename subfolder with space part 2/2"
Hope this proves useful to someone.

my kanban 1's board game

Here's a slide deck explaining the essence of my kanban 1s board game. Jon Jaggers Kanban 1s Board Game
  • You can play an early session with no clips so the players can see how inventory builds up (you can also push done story-cards to the next edge's corner, rather than waiting for them to be pulled).
  • The clips that hold the story-cards are a crucial part of the game. They make it a kanban game.
  • You can limit the number of clips per edge to create a natural work-in-progress (wip) limit.
  • You can add a new rule: players can also spend a 1 to split a story-card in two, eg a 4 into a 3 and a 1 (assuming they have a spare clip).
  • You can record the day a story-card comes off the backlog, and also the day it gets to done and thus measure the cycle time.
  • You can simulate scrum-style discrete sprints.
  • You can vary the number of dice at different edges.


Isolating legacy C code from external dependencies

Code naturally resists being isolated if it isn't designed to be isolatable. Isolating legacy code from external dependencies can be awkward. In C and C++ the transitive nature of #includes is the most obvious and direct reflection of the high-coupling such code exhibits. However, there is a technique you can use to isolate a source file by cutting all it's #includes. It relies on a little known third way of writing a #include. From the C standard:

6.10.2 Source file inclusion
...
A preprocessing directive of the form:
  #include pp-tokens 
(that does not match one of the two previous forms) is permitted. The preprocessing tokens after include in the directive are processed just as in normal text. ... The directive resulting after all replacements shall match one of the two previous forms.


An example. Suppose you have a legacy C source file that you want to write some unit tests for. For example:
/*  legacy.c  */
#include "wibble.h"
#include <stdio.h>
...
int legacy(int a, int b)
{
    FILE * stream = fopen("some_file.txt", "w");
    char buffer[256];
    int result = sprintf(buffer, 
                         "%d:%d:%d", a, b, a * b);
    fwrite(buffer, 1, sizeof buffer, stream);
    fclose(stream);
    return result;
}
Your first step is to create a file called nothing.h as follows:
/* nothing! */
nothing.h is a file containing nothing and is an example of the Null Object Pattern. Then you refactor legacy.c to this:
/* legacy.c */
#if defined(UNIT_TEST)
#  define LOCAL(header) "nothing.h"
#  define SYSTEM(header) "nothing.h"
#else
#  define LOCAL(header) #header
#  define SYSTEM(header) <header>
#endif

#include LOCAL(wibble.h)  /* <--- */
#include SYSTEM(stdio.h)  /* <--- */
...
int legacy(int a, int b)
{
    FILE * stream = fopen("some_file.txt", "w");
    char buffer[256];
    int result = sprintf(buffer, 
                         "%d:%d:%d", a, b, a*b);
    fwrite(buffer, 1, sizeof buffer, stream);
    fclose(stream);
    return result;
}
Now structure your unit-tests for legacy.c as follows:
First you write null implementations of the external dependencies you want to fake (more Null Object Pattern):
/* legacy.test.c: Part 1 */

static FILE * fopen(const char * restrict filename, 
                    const char * restrict mode)
{
    return 0;
}

static size_t fwrite(const void * restrict ptr,   
                     size_t size, 
                     size_t nelem, 
                     FILE * restrict stream)
{
    return 0;
}

static int fclose(FILE * stream)
{
    return 0;
}
Then #include the source file. Note carefully that you're #including legacy.c here and not legacy.h and you're #defining UNIT_TEST so that legacy.c will have no #includes of its own:
/* legacy.test.c: Part 2 */
#define UNIT_TEST
#include "legacy.c" 
Then write your tests:
/* legacy.test.c: Part 3 */
#include <assert.h>

void first_unit_test_for_legacy(void)
{
    /* writes "2:9:18" which is 6 chars */
    assert(6, legacy(2,9));
}

int main(void)
{
    first_unit_test_for_legacy();
    return 0;
}
When you compile legacy.test.c you will find your first problem - it does not compile! You have cut away all the #includes which cuts away not only the function declarations but also the type definitions, such as FILE which is a type used in the code under test, as well as in the real and the null fopen, fwrite, and fclose functions. What you need to do now is introduce a seam only for the functions:
/* stdio.seam.h */
#ifndef STDIO_SEAM_INCLUDED
#define STDIO_SEAM_INCLUDED

#include <stdio.h>

struct stdio_t
{
    FILE * (*fopen)(const char * restrict filename, 
                    const char * restrict mode);
    size_t (*fwrite)(const void * restrict ptr, 
                     size_t size,  
                     size_t nelem, 
                     FILE * restrict stream);
    int (*fclose)(FILE * stream);
};

extern const struct stdio_t stdio;

#endif    
Now you Lean On The Compiler and refactor legacy.c to use stdio.seam.h:
/* legacy.c */   
#if defined(UNIT_TEST)
#  define LOCAL(header) "nothing.h"
#  define SYSTEM(header) "nothing.h"
#else
#  define LOCAL(header) #header
#  define SYSTEM(header) <header>
#endif

#include LOCAL(wibble.h) 
#include LOCAL(stdio.seam.h)  /* <--- */
...
int legacy(int a, int b)
{
    FILE * stream = stdio.fopen("some_file.txt", "w");
    char buffer[256];
    int result = sprintf(buffer, 
                         "%d:%d:%d", a, b, a*b);
    stdio.fwrite(buffer, 1, sizeof buffer, stream);
    stdio.fclose(stream);
    return result;
}    
Now you can structure your null functions as follows:
/* legacy.test.c: Part 1 */
#include "stdio.seam.h"

static FILE * null_fopen(const char * restrict filename, 
                         const char * restrict mode)
{
    return 0;
}

static size_t null_fwrite(const void * restrict ptr, 
                          size_t size, 
                          size_t nelem, 
                          FILE * restrict stream)
{
    return 0;
}

static int null_fclose(FILE * stream)
{
    return 0;
}

const struct stdio_t stdio =
{
    .fopen  = null_fopen,
    .fwrite = null_fwrite,
    .fclose = null_fclose,
};    
And viola, you have a unit test. Now you have your knife in the seam you can push it in a bit further. For example, you can do a little spying:
/* legacy.test.c: Part 1 */
#include "stdio.seam.h"
#include <assert.h>
#include <string.h>

static FILE * null_fopen(const char * restrict filename, 
                         const char * restrict mode)
{
    return 0;
}
    
static size_t spy_fwrite(const void * restrict ptr, 
                         size_t size, 
                         size_t nelem, 
                         FILE * restrict stream)
{
    assert(strmp("2:9:18", ptr) == 0);
    return 0;
}

static int null_fclose(FILE * stream)
{
    return 0;
}

const struct stdio_t stdio =
{
    .fopen  = null_fopen,
    .fwrite =  spy_fwrite,
    .fclose = null_fclose,
};
This approach is pretty brutal, but it might just allow you to create an initial seam which you can then gradually prise open. If nothing else it allows you to create characterisation tests to familiarize yourself with legacy code.

You'll also need to create a trivial implementation of stdio.seam.h that the real code uses:
/* stdio.seam.c */
#include "stdio.seam.h"
#include <stdio.h>

const struct stdio_t stdio =
{
    .fopen  = fopen,
    .fwrite = fwrite,
    .fclose = fclose,
};
The -include compiler option might also prove useful.

-include file
    Process file as if #include "file" appeared as the first line of the primary source file.


Using this you can create the following file:
/* include.seam.h */
#ifndef INCLUDE_SEAM
#define INCLUDE_SEAM

#if defined(UNIT_TEST)
#  define LOCAL(header) "nothing.h"
#  define SYSTEM(header) "nothing.h"
#else
#  define LOCAL(header) #header
#  define SYSTEM(header) <header>
#endif

#endif
and then compile with the -include include.seam.h option.

This allows your legacy.c file to look like this:
#include LOCAL(wibble.h) 
#include LOCAL(stdio.seam.h)
...
int legacy(int a, int b)
{
    FILE * stream = stdio.fopen("some_file.txt", "w");
    char buffer[256];
    int result = sprintf(buffer, "%d:%d:%d", a, b, a*b);
    stdio.fwrite(buffer, 1, sizeof buffer, stream);
    stdio.fclose(stream);
    return result;
}    


every teardrop is a waterfall

I was listening to Coldplay the other day and got to thinking about waterfalls. The classic waterfall diagram is written something like this:

Analysis
    leading down to...
Design
    leading down to...
Implementation
    leading down to...
Testing.

The Testing phase at the end of the process is perhaps the biggest giveaway that something is very wrong. In waterfall, the testing phase at the end is what's known as a euphemism. Or, more technically, a lie. Testing at the end of waterfall is really Debugging. Debugging at the end of the process is one of the key dynamics that prevents waterfall from working. There are at least two reasons:

The first is that of all the activities performed in software development, debugging is the one that is the least estimable. And that's saying something! You don't know how long it's going to take to find the source of a bug let alone fix it. I recall listening to a speaker at a conference who polled the audience to see who'd spent the most time tracking down a bug (the word bug is another euphemism). It was just like an auction! Someone called out "3 days". Someone else shouted "2 weeks". Up and up it went. The poor "winner" had spent all day, every day, 9am-5pm for 3 months hunting one bug. And it wasn't even a very large audience. This 'debug it into existence' approach is one of the reasons waterfall projects take 90% of the time to get to 90% "done" (done is another euphemism) and then another 90% of the time to get to 100% done.

The second reason is Why do cars have brakes?. In waterfall, even if testing was testing rather than debugging, putting it at the end of the process means you'll have been driving around during analysis, design and implementation with no brakes! You won't be able to stop! And again, this tells you why waterfall projects take 90% of the time to get to 90% done and then another 90% of the time to get to 100% done. Assuming of course that they don't crash.

In Test First Development, the testing really is testing and it really is first. The tests become an executable specification. Specifying is the opposite of debugging. The first 8 letters of specification are S, P, E, C, I, F, I, C.
A test is specific in exactly the same way a debugging session is not.

Coupling, overcrowding, refactoring, and death

I read The Curious Incident of the Dog in the Night Time by Mark Haddon last week. I loved it. At one point the main character, Christopher, talks about this equation:

Pg+1 = α Pg (1 - Pg)

This equation was described in the 1970s by Robert May, George Oster, and Jim Yorke. You can read about it here. The gist is it models a population over time, a population at generationg+1 being affected by the population at generation g. If there is no overcrowding then each member of the population at generation g, denoted Pg, produces α offspring, all of whom survive. So the population at generation g+1, denoted Pg+1 equals α Pg. The additional term, (1 - Pg ) represents feedback from overcrowding. Some interesting things happen depending on the value of α
  • α < 1: The population goes extinct.
  • 1 < α < 3 : The population rises to a value and then stays there.
  • 3 < α < 3.57 : The population alternates between boom and bust.
  • 3.57 < α : The population appears to fluctuate randomly.
In biological systems each generation has a natural lifespan. The cycle of death naturally helps to reduce overcrowding. But even with the inevitable death of each generation, the system's behaviour is delicately poised. Once α gets into the 3 - 3.57 range the growth starts to outpace the death leading to a rising population, but this leads to overcrowding which reduces the growth leading to a falling population. That reduces overcrowding, and growth rises again. The population repeats in a stable up/down cycle. But only because of the inevitable death. And only while the rate of death is sufficiently high to maintain the cycle. Once α gets beyond 3.57 the rate of death can no longer regulate the increased rate of growth and the system destabilises.

You can think about the process of writing software with this equation.

You can think of over-crowding as being analogous to over-coupling. We feel that a codebase is hard to work with, difficult to live in, if it resists our attempts to work with it. When it resists it is the coupling that is resisting.

You can also think of death as being analogous to refactoring. Just as death reduces overcrowding, so refactoring reduces coupling.

Refactoring is a hugely important dynamic in software development. Without refactoring a codebase can grow without check. Growing without check is bad. It leads to overcrowding. Overcrowding hinders growth.

Incremental development



Out of the crisis

is an excellent book by W. Edwards Deming (isbn 0-911379-01-0). As usual I'm going to quote from a few pages:
All industries, manufacturing and service, are subject to the same principles of management.
Quality comes not from inspection, but from improvement of the production process.
Today, 19 foreman out of 20 were never on the job that they supervise.
Fear amongst salaried workers may be attributed in large part to the annual rating of performance.
Absenteeism is largely a function of supervision. If people feel important to a job, they will come to work.
He that would run his company on visible figures alone will in time have neither company nor figures.
There has never been a definitive study of quality in the dental profession; nor is there likely to be one. Partly because they tend to work alone, dentists resist the idea of being evaluated, or even observed, by others.
Where there is fear, there will be wrong figures.
It is well known that rework piles up: no one wishes to tackle it.
This company had been sending a letter to every driver at every mistake. It made no difference whether this was the one mistake of the year for this driver, or the 15th: the letter was exactly the same. What does the driver who has received 15 warnings, all alike, think of the management?

Cause is effect and effect is cause and vice versa


Defects cause lateness.


The more defects code has the more time and effort it takes to get it to done. This seems a self-evident truth. But beware! The Causation Fallacy says it is not easy to know what is cause and what is effect. If a feature misses its deadline pressure often builds to ensure it doesn't miss the next deadline. And under pressure people don't think faster. Extra pressure usually increases the likelihood of defects. This suggests


Lateness causes defects.


So do defects cause lateness, or does lateness cause defects? Or do they rotate around each other like partners on a dance floor?



sprints, time-boxing, and capacity

A team is doing Scrum with 3 week sprints. Suppose at the end of a sprint they've got nothing to done. What should they do? There's a strong temptation to ask for more time. To make this sprint a 4 week sprint. Most of the work in progress is 90% done, they say. Another week and things will have got to done, they say. It seems reasonable.


Trying to run systems beyond their capacity is not a good idea. In this situation Scrum's fixed-duration time-box constraint has served it's purpose admirably. The problem is not the choice of 3 weeks. Changing 3 weeks into 4 weeks is not addressing the problem. The problem is the team planned to pull in an amount of work and get it to done in 3 weeks. But they're not yet in control of their process - they don't know what their capacity is. They pulled in more than 3 weeks worth of work. Probably a lot more. But we just don't know!


In The Toyota Way, Jeffrey Liker writes:

Taiichi Ohno considered the fundamental waste to be overproduction, since it causes most of the other wastes.

Advice from a genius with a lifetime's experience. Toyota manufactures cars. It makes cars. Its production line is an actual line. If manufacturers are prone to overproduction imagine how much more prone software developers are! The things we make are not even physical things. In software, things are mostly invisible. It's difficult to manage what you can't see. In Quality Software Management volume 2, First-Order Measurement, Jerry Weinberg writes:

Without visibility, control is not possible.
If you can't see, you can't steer.

Rather than asking for another week, the team should really be thinking about addressing their real problem. Their real problem is that they're pulling in too much work. They have to somehow learn to pull in less work. So they can start to be in control of their process rather than their process being in control of them.



culture

Back to quotes table-of-contents

From Quality Software Management: Vol 2. First Order Measurement
Culture makes its presence known through patterns that persist over time.

One of the most sensitive measures of the cultural pattern of any organization is how quickly it finds and removes problems.

From The Toyota Way
Building a culture takes years of applying a consistent approach with consistent principles.

From XP and Culture Change
A process change will always involve a cultural change.

Because culture embodies perception and action together, changing culture is difficult and prone to backsliding.

From Quality Software Management: Vol 3. Congruent Action
Culture makes language, then language makes culture.

From Beating the System
Culture is what we do when we do not consciously decide what to do.

From Freedom from Command and Control
Consultants who see culture change as something distinct from the work and, as a corollary, something that can be the subject of an intervention, miss the point. When you change the way work is designed and managed, and make those who do the work the central part of the intervention, the culture changes dramatically as a consequence.

From Leverage points
One aspect of almost every culture is the belief in the utter superiority of that culture.

From John Seddon
Culture change is free [because] it's a product of the system.

From Notes on the Synthesis of Form
Culture is changing faster than it has ever changed before...what once took many generations of gradual development is now attempted by a single individual.

From Slack
Successful change can only come about in the context of a clear understanding of what may never change, what the organization stands for... the organization's culture... If nothing is declared unchangeable, then the organization will resist all change. When there is no defining vision, the only way the organization can define itself is its stasis.

From The Hidden Dimension
As Freud and his followers observed, our own culture tends to stress that which can be controlled and to deny that which cannot.

From The Silent Language
Culture hides much more than it reveals, and strangely enough what it hides, it hides most effectively from its own participants.

An often noted characteristic of culture change is that an idea or a practice will hold on very persistently, apparently resisting all efforts to move it, and then, suddenly, without notice, it will collapse.