assert_not_diff

The output of Ruby's Test::Unit::TestCase's
assert_equal(expected, actual) when expected and actual are not the same is often not as useful as it could be. For example:

expected = [1,99,2,3,{:a=>34,:b=>43}]
actual = [1,2,3,4,{:a=>324,:c=>555,:b=>43},5]
assert_equal expected, actual

produces:

<[1, 99, 2, 3, {:a=>34, :b=>43}]> expected but was
<[1, 2, 3, 4, {:a=>324, :c=>555, :b=>43}, 5]>.

For large objects which differ only slightly the output that tells you where they differ quickly gets lost in the mass of output telling you where they don't.

Perhaps what's happened here is a sort of Primitive Obsession.
It's as though assert_equal assumes it will only ever be called with primitives.

So I put together a little bit of code to help out. The simplest solution I can think of uses json and diff, so I thought I'd call it assert_not_diff

require 'json'
require 'Tempfile'

def assert_not_diff(lhs,rhs)
  if lhs != rhs
    puts `diff -y #{file_for lhs} #{file_for rhs}`
  end
end

def file_for(obj)
  exp = Tempfile.new("bk", "/tmp").open
  exp.write(JSON.pretty_generate(obj))
  exp.close
  exp.path
end

expected = [1,99,2,3,{:a=>34,:b=>43}]
actual = [1,2,3,4,{:a=>324,:c=>555,:b=>43},5]
assert_not_diff expected, actual

This produces:

[                  [
  1,                 1,
  99,            <
  2,                 2,
  3,                 3,
                 >   4,
  {                  {
    "a": 34,     |     "a": 324,
                 >     "c": 555,
    "b": 43            "b": 43
  }              |   },
                 >   5
]                  ]

Which I think is a lot more useful.
Hope this proves useful to someone!

smart swarm

is an excellent book by Peter Miller (isbn 978-0-00-738297-2). As usual I'm going to quote from a few pages:

As successful foragers return to the nest with seeds, they're met at the nest entrace by foragers waiting in reserve. This contact stimulates the inactive ants to go out. Foragers normally don't come back until they find something. So the faster the foragers return, the faster other ants go out, enabling the colony to tune its work force to the probability of finding food.

Instead of attempting to outsmart the desert environment, the ants, in a sense, were matching its complexity with their own.

Instead of trying to keep fine-tuning a system so it will work better and better, maybe what we really ought to be looking for is a rigourous way of saying, okay, that's good enough. [Deborah Gordon]

If a scout bee was impressed by another scout's dance, she might fly to the box being advertised and conduct her own inspection, which could last as long as an hour. But she would never blindly follow another scout's opinion by dancing for a site she hadn't visited.

J. Scott Turner considers the mound's function as a respiratory system so essential that the termites couldn't live without it. In a sense, he argues, the mound is almost a living part of the colony.

If individuals in a group are prompted to make small changes to a shared structure that inspires others to improve it even further, the structure becomes an active player in the creative process.

Unlike our systems, which are tuned for efficiency, the termites' systems have been tuned for robustness, which they demonstrate by building mounds that are constantly self-healing.

What really made the lights go on was the realization that termites don't pay attention to the environment itself but to changes in the environment.

Not only does this complicated structure represent an indirect collaboration among millions of individuals, it also embodies a kind of ongoing conversation between the colony and the world outside. The mound might look like a structure, but it's better thought of as a process.

We should think of it [the termite mound] as a dynamic system that balances forces both inside and outside its walls to create the right environment for the termites.

When you feel like you belong to something, it gives you so much more freedom and so much more energy that might otherwise be used up in anxiety, to do other things.

On January 12, 2006, several hundred thousand pilgrims had gathered in a dusty tent city at Mina, three miles east of Mecca...
By noon... about a half-million or more pilgrims filled the Jamarat plaza in front of the bridge... The pressure inside the crowd was crushing... More than an hour later, victims were piled up seven layers deep: 363 men and women were dead.
"Those in charge need to remember the root cause of the problem: too many people trying to get through too small a space. The ingress rate at the bridge was 135,000 per hour. The thoughput rate of the pillars was only 100,000 an hour. You can't put a pint into a half-pint jug." [Keith Still]

the life of brian

is an excellent screenplay by Monty Python (isbn 0-413-74130-3). As usual I'm going to quote from a few pages:

Why aren't women allowed to go to stonings, Mum?
Because it's written, that's why.

Jaguar's ear lobes. Wolf nipple chips. Gem 'em while they're hot. They're lovely. Dromedary pretzels, only half a dinar. Tuscany fried bat.

The People's Front of Judea fucking gets things done!

What's this then? 'Romanes Eunt Domus'?
It says 'Romans go home'.
No it doesn't. What's the Latin for Romans? Come on … come on

And what have they ever given us in return?
The aqueduct?

Oh … I'll give you nineteen then.
No, no. Do it properly.
What?
Haggle properly. This isn't worth nineteen.
But you just said it was worth twenty.
…
Eighteen?
No, no, no. You go to fourteen now.
Fourteen.
Fourteen, are you joking?

Siblings!! Let us not be disheartened. One total catastrophe like this is just the beginning!

What is the secret?
Leave me alone.
Yes! Tell us the secret!
What is the secret?
is is the secret of Eternal Life?

Look … you've got it all wrong. You don't need to follow me. You don't need to follow anybody. You've got to think for yourselves. You're all individuals.
Yes, we're all individuals.
You're all different.
Yes, we are all different.

Always look on the bright side of life.

Perfect Software and other illusions about testing

is the title of an excellent book by Jerry Weinberg. As usual I'm going to quote from a few pages:

Testing a system is a process of gathering information with the intent that the information could be used for some purpose.

Without a process that includes regular technical reviews, no project will rise above mediocrity, no matter how good its machine-testing process.

At least half your testing costs can be cut before anybody ever runs a test, if only your systems are designed with testability in mind.

"We absolutely need this software in place twenty-four weeks from tomorrow. We need two weeks to staff up and get approvals. Then we'll need four weeks for requirements, four weeks for architecture. four weeks for design, and eight weeks for coding. That adds up to twenty-two weeks, so we'll have two weeks left for testing."

If you have ten kilograms of pure uranium-235 and you add another ten kilograms, you'll have twenty kilograms. But if you do this a few more times, you won't have fifty kilograms, you'll have a nuclear explosion. One plus one doesn't always equal two.

The more bugs you find, the more you're going to find, not the other way around.

The human mind craves meaning. If you feed people a random bit of data, they'll struggle to divine meaning from it - and they'll move from the intake phase to the meaning phase so fast they won't be aware of doing so.

You have to know what you're expecting before you give meaning to a test report, otherwise everything looks or sounds right. That's why I'm a strong advocate of the test-first philosophy, whereby developers write their tests to include expected results before they write a line of code. It's what we did fifty years ago, but the practice was gradually lost when industry trends separated testing from development.
That separation occurred initially because it's psychologically difficult for people to test their own programs. There's still significant risk if you rely on test-first without pair-programming or some other process that casts more than one pair of eyes, and more than one brain, on a program.

If test-first is a good idea, then significance-first is even better. Why? … if you actually perform even an enormous number of tests, you would likely lose the valuable information among all the worthless crud. The number of tests performed should be as small as possible, but no smaller.

"We didn't have any problems until we started testing. We were right on schedule. Testing screwed up everything."

"What the American public wants in the theatre is a tragedy with a happy ending." [William Dean Howells]

Experiential Learning 3: Simulation

is an excellent book by Jerry Weinberg. There's no isbn - you can buy it from Leanpub. As usual I'm going to quote from a few pages:

Would be writers commonly trap themselves by speaking their stories, rather than writing them.

The essential principle of the fieldstone method is "energy".

In a few days we can raise the participants' level of both creating and reviewing, which are the yin and yang of such creative work.

Really, the possibilities are endless, so there's never an excuse for saying you can't think of an experiential exercise.

If they cannot overcome their feelings that the simulation is "silly", then that is an important perception which we'll want to examine during the invention.

A simulation doesn't have to be "real" to be successful as an experiential learning too. What has to be real the feelings it stimulates in the participants, for feelings are what drive learning.

Paradoxically, realism often interferes directly with learning from a simulation.

Frequently, a VW company will sell a poem without knowing whether they can build it.

"Rules" are frozen solutions. Rules are solutions to yesterday's problem, carried forward to the present, but usually without reference to the problem they were intended to solve. Each rule is really an "if-then" rule, but "if-then" part is seldom stated.

You are not a grader, but a teacher.

Every time you jiggle one of your control points, you're ruining some of the teaching power of the simulation. Why? Because you're simulating a universe in which there are powerful hidden gods who control the world.

surely you're joking Mr Feynman

subtitled 'Adventures of a Curious Character', is an excellent book by Richard Feynman (isbn 978-0-099-17331-1). As usual I'm going to quote from a few pages:

Radio sets were much easier to understand in those days because everything was out in the open. After you took the set apart (it was a big problem to find the right screws), you could see this was a resistor, that's a condenser, here's a this, there's a that; they were all labelled.

I finally fixed it beause I had, and still have, persistence.

They didn't even know what they "knew". I don't know what's the matter with people: they don't learn by understanding; they learn by some other way - by rote, or something. Their knowledge is so fragile.

In this room there were wires strung all over the place! Switches were hanging from the wires, cooling water was dripping from the valves, the room was full of stuff, all out in the open. Tables piled with tools were everywhere; it was the most godawful mess you ever saw. The whole cyclotron was there in one room, and it was complete, absolute chaos!
It reminded me of my lab at home. Nothing at MIT had ever reminded me of my lab at home. I suddenly realized why Princeton was getting results. They were working with the instrument. They built the instrument; they knew where everything was, they knew how everything worked. ... It was wonderful! Because they worked with it. They didn't have to sit in another room and push buttons.

I sat with the physicsts, but after a bit I thought: It would be nice to see what the rest of the world is doing, so I'll sit for a week or two in each of the other groups.

So you must be always jiggling a little bit, testing out which way seems to be the easiest.

One day I was watching a paramecium and I saw something that was not described in the books I got in school - in college, even. These books always simplify things so the world will be more like they want it to be.

I chose to play a thing called a "frigideira" which is a toy frying pan made of metal, about six inches in diameter, with a little metal stick to beat it with. ... I practiced all the time. I'd walk along the beach holding two sticks that I had picked up, getting the twisty motion of the wrists, practicing, practicing, practicing.

One day, shortly before Carnival time, the leader of the samba school said, "OK, we're going to practice marching in the street. ...
It was rush hour in Copacabana, and we were going to march down the middle of Avenida Atlantica.
It said to myself, "Jesus! The boss didn't get a license, he didn't OK it with the police, he didn't do anything. He's decided we're just going to go out."
So we started to go out into the street, and everybody, all around, was excited. Some volunteers from a group of bystanders took a rope and formed a big square around our band, so the pedestrians wouldn't walk through the lines. People started to lean out of the windows. Everybody wanted to hear the new samba music. It was very exciting!
As soon as we started to march, I saw a policeman, way down at the other end of the road. He looked, saw what was happening, and started diverting traffic! Everything was informal. Nobody made any arrangements, but it all worked fine.

One exercise they had invented for loosening us up was to draw without looking at the paper. Don't take your eyes off the model; just look at her and makes the lines on the paper without looking at what you're doing.
One of the guys says, "I can't help it. I have to cheat. I bet everybody's cheating!"
"I'm not cheating!" I say.
"Aw, baloney!" they say.
I finish the exercise and they come over to look at what I had drawn. They found that, indeed, I was NOT cheating; at the very beginning my pencil point had busted, and there was nothing but impressions on the paper.
When I finally got my pencil to work, I tried again. I found that my drawing had a kind of strength - a funny, semi-Picasso-like strength - which appealed to me. The reason I felt good about that drawing was, I knew it was impossible to draw well that way, and therefore it didn't have to be good - and that's really what all the loosening up was all about. I had thought that "loosen up" meant "make sloppy drawings," but it really meant to relax and not worry about how the drawing is going to come out.

When it came time to evaluate the conference at the end, the others told how much they got out of it, how successful it was, and so on. When they asked me, I said, "This conference was worse than a Rorschach test: There's a meaningless inkblot, and the others ask you what you think you see, but when you tell them, they start arguing with you!

nothing new ever works

I'm having a new bathroom put in at the moment. The plumber was here doing some plumbing. Then the tiler was doing some tiling. He proudly mentioned that he had a new van. When he said that I thought of the New Law from Jerry Weinberg's The Secrets of Consulting

Nothing new ever works

Then he says he has to go out to get something. He says he'll only be half an hour. Ninety minutes later he's still not back. For a moment I start to think negative thoughts about the stereotype of an unreliable workman. Then I remember his new van. Sure enough, he turns up a few minutes later and he's fallen foul of the New Law. His new van had got a puncture, and being the first one it had taken him ages to change the wheel.

Buckminster Fuller once said...

If you want to change how someone thinks, give up. You cannot change the way people think. Instead, give people tools that help them do something different.

wip limit

I bumped into a nice example of a wip limit. It's in a Columbo episode called The Conspirators. In it the murderer, a man called Joe Devlin, likes to drink whiskey. He's careful not to like drinking whiskey too much though, so before drinking he uses a diamond (set into a ring) to mark the side of the whiskey bottle. As Devlin makes the mark he says "This far and no farther". Once the level drops to the mark he stops drinking. Thus Devlin is using a WIP limit - a Whiskey In Progress limit.

Ackoff's Best

Ackoff's Best, by Russell Ackoff is an excellent book by (isbn 978-0-471-31634-3). As usual I'm going to quote from a few pages:

Analysis looks into things; synthesis looks out of things … Analysis yields knowledge; synthesis yields understanding … The former enables us to describe; the latter, to explain … In analytic thinking, the thing to be explained is treated as a whole to be taken apart. In synthetic thinking the thing to be explained is treated as part of a containing whole. The former reduces the focus of the investigator; the latter expands it … Synthesis, or putting things together, is the key to systems thinking just as analysis, or taking them apart, was the key to Machine-Age thinking.

The essential properties of a system taken as a whole derive from the interactions of its parts, not their actions taken separately. Therefore, when a system is taken apart it loses its essential properties. Because of this - and this is the critical point - a system is a whole that cannot be understood by analysis.

Organisms and organisations are systems that usually have purposes of their own. However, the parts of an organism (i.e., hearts, lungs, brain) do not have purposes of their own, but the parts of an organisation do … An organisation with purposeful parts almost inevitably generates internal conflict … An organisation is a system whose major deficiencies arise from the ways its parts interact, not from their actions taken separately.

Automation is fundamentally different from mechanisation. Mechanization has to do with the replacement of muscle; automation with the replacement of mind.

The publicly owned company became a corporation (derived from "corpus", meaning "body") and the chief executive became the "head" of the firm.

Development of individuals and corporations is more a matter of learning than earning.

The most variety-decreasing type of social system is one we call a bureaucracy. A bureaucracy is an organisation whose principal objective is to keep people busy doing nothing. … The problem created by people who are busy doing nothing is that they frequently obstruct others who have real work to do. … Bureaucracies obstruct development. … Because bureaucracies tend to be inefficient and obstruct development they invite and encourage corruption.

If you're going to sin, sin against God, not the bureaucracy. God will forgive you but the bureaucracy won't. [Admiral Hyman G. Rickover]

Wisdom is the ability to see the long-run consequences of current actions, the willingness to sacrifice short-run gains for larger long-run benefits, and the ability to control what is controllable and not to fret over what is not … Even the best planning of which we are capable requires at least as much art as it does science … Planning is not an act but a process …

Today's students are over instructed in what they can better do alone: take things and concepts apart; and they are under instructed in what is very difficult to do alone: put what they have learned together into an understanding of the world and their role in it.

Effectiveness is evaluated efficiency. It is efficiency multiplied by value, efficiency for a valued outcome. Intelligence is the ability to increase efficiency; wisdom is the ability to increase effectiveness … Growth does not require an increase in value; development does.

Separation of the relevant information from the irrelevant is a critical part of problem formation … the reasons for wanting to answer a question determines what is the right answer to it.

A wrong solution to the right problem is generally better than the right solution to the wrong problem, because one usually gets feedback that enables one to correct wrong solutions, but not wrong problems. Wrong problems are perpetuated by right solutions to them.

Even those aspects of an organisation's future that are not affected by what it and others do (e.g. the weather) cannot be forecasted well for more than a short period. As in the case of the weather, however, the need to forecast it can be eliminated by bringing it under control, as we do by building structures within which work can proceed whatever the external weather may be.

stuka pilot

is an excellent book by Hans Ulrich Rudel (isbn 978-1908476876). As usual I'm going to quote from a few pages:

He tells me that a large scale offensive is in preparation in my sector... approximately three hundred tanks are to be employed in this operation... The number three hundred flabbergasts me... I reply that I find some difficulty in believing it... He says to me, half in earnest, half in jest: "If I didn't know you, for two pins I would have you put under arrest for saying such a thing. But we will soon find out." He goes to the telephone and is connected with the Chief of the General Staff. "You have just given the Fuhrer the figure of three hundred tanks for operation X." "Yes I did." "I want to know the names of the divisions concerned with their present strength in tanks. I have somebody with me who is well acquainted with the position." ... the Chief of the General Staff has the bad luck to begin with the 14th armoured division. He says it has sixty tanks. Goering can hardly contain himself. "My man reports that the 14th has one!" A lengthy silence at the other end of the line. "When did he leave the front?" "Four days ago." Again silence. Then "Forty tanks are still on their way to the front. The rest are in repair shops on the line of communications, but will certainly reach their units by zero day, so that the figures are correct." He has the same answer for the other divisions. The Reichsmarschall slams down the receiver in a rage. "That is how it is!" The Fuhrer is given a totally false picture based on incorrect data and is surprised when operations do not have the success expected... The South Eastern zone with its network of communications is being incessantly blanketed by the enemy's bomber formations. Who knows how many of those forty tanks, for example, will ever reach the front or when? Who can say if the repair shops will get their spare parts in time and if they will be able to complete their repairs within the specified time?

In ministries and departments, however, mistakes are denied on principle.

Another confirmation of the truth of our old Stuka maxim: "Nothing comes off - except what you have practised."

We have long since ceased to develop practice from theory; we do just the opposite.

The fitters have their hands full, for the aircraft have been heavily damaged by flak. The life of such an aeroplane will always be limited.

Little by little I discover all the tricks. Skill is often the result of getting hurt.

I now see that perfectly plainly. We are alone to possess this knowledge; the responsibility is ours.

the power of pairing

In a previous post I talked about an example where a pair performed better than either individually. Here's another small example that happened to me today.

My very good friend Syver was running a coding dojo in Erlang. A test for the filter exercise looked like this:

odd(Integer) when Integer rem 2 == 1 ->
    true;
odd(_) ->
    false.

filter_test() ->
    ?assertEqual([3, 1], 
      filter(fun(Elem) -> odd(Elem) end, [1, 2, 3, 4])).    

I don't know Erlang at all, so Syver helped me out with this first-cut version:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter_helper(Fun(Head), Fun, Head, Tail, []).

filter_helper(true, Fun, TrueElement, [Head|Tail], Result) ->
    filter_helper(Fun(Head), Fun, Head, Tail, [TrueElement|Result]);
filter_helper(true, _, TrueElement, [], Result) ->
    [TrueElement|Result];
filter_helper(false, Fun, _, [Head|Tail], Result) ->
    filter_helper(Fun(Head), Fun, Head, Tail, Result);
filter_helper(false, _, _, [], Result) ->
    Result

I realized Erlang is a lot like Prolog which I knew in the dim and distant past. I stared at the code and after several minutes something about it started nagging me. It was the splitting of the list into it's Head and Tail elements in both filter and filter_helper. I wondered if this duplication could be avoided by making filter_helper call back into filter. After several false attempts I came up with:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter_helper(Fun(Head), Fun, Head, Tail).

filter_helper(true, Fun, TrueElement, List) ->
    X = filter(Fun,List),
    [TrueElement|X];
filter_helper(false, Fun, _FalseElement, List) ->
    filter(Fun, List).

Then Syver showed me how the use of X could be collapsed:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter_helper(Fun(Head), Fun, Head, Tail).

filter_helper(true, Fun, TrueElement, List) ->
    [TrueElement|filter(Fun, List)];
filter_helper(false, Fun, _FalseElement, List) ->
    filter(Fun, List).

We did some argument renaming:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter_helper(Fun(Head), Fun, Head, Tail).

filter_helper(true, Fun, Head, List) ->
    [Head|filter(Fun, List)];
filter_helper(false, Fun, _, List) ->
    filter(Fun, List).

Then Syver noticed that we could pass Head,Tail as a single [Head|Tail] argument:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter_helper(Fun(Head), Fun, [Head|Tail]).

filter_helper(true, Fun, [Head|Tail]) ->
    [Head|filter(Fun, [Head|Tail])];
filter_helper(false, Fun, [_|Tail]) ->
    filter(Fun, Tail).

We agreed that filter_helper was better as filter:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail]) ->
    filter(Fun(Head), Fun, [Head|Tail]).

filter(true, Fun, [Head|Tail]) ->
    [Head|filter(Fun, [Head|Tail])];
filter(false, Fun, [_|Tail]) ->
    filter(Fun, Tail).

As a final polish Syver refactored to this:

filter(_, []) ->
    [];
filter(Fun, [Head|Tail] = List) ->
    filter(Fun(Head), Fun, List).

filter(true, Fun, [Head|Tail]) ->
    [Head|filter(Fun, [Head|Tail])];
filter(false, Fun, [_|Tail]) ->
    filter(Fun, Tail).

and then, again, after the dojo, thanks to Syver's comment below, to this:

filter(_, []) ->
    [];
filter(Fun, [Head|_] = List) ->
    filter(Fun(Head), Fun, List).

filter(true, Fun, [Head|Tail]) ->
    [Head|filter(Fun, [Head|Tail])];
filter(false, Fun, [_|Tail]) ->
    filter(Fun, Tail).

We agreed that the final version was definitely better than either of us could have come up with individually.

salmon fishing

is an excellent book by Hugh Falkus (isbn 0-85493-114-9 ). Unusually I'm going to quote from one page - the preface:

No creature on earth treats the dogmatist more sternly than Salmo salar.

Contradictory in everything he does, Salmo cannot be tied down by dogma.

In addition to providing days of rare enchantment, it taught me that much of what of what I had read about Salmo's behaviour was pretty dubious. Often, the fish I was watching seemed a different creature from the one encountered in print. And over the years it began to dawn on me that far from being fact, a lot of what I had hitherto taken for granted was indeed fallacy; that most authors had written about what they thought salmon did, not what they had seen them do.

Since then I have tested certain of their statements, some of which will be found amongst these pages: assertions by acknowledged experts and regarded as Gospel - but unsound. Most of the accepted tenets of salmon fishing started as conjecture; but gradually, parroted by writer after writer, part of this conjecture became "fact".

In salmon fishing there is no magic substitute for water-sense, skilful presentation, and persistence.

We are wise to move with the deliberation of the heron, that most stealthy of waders.

I think small hooks are preferable to big hooks - they usually get a better hold.

I remain convinced that black is the most attractive colour for a salmon fly.

I suggest that the nearest we can come to a definitive statement is to say that salmon tend to react to large flies sunk deep at temperatures below 45°, and to small flies near the surface at temperatures above 50°.

fishing on the river tay

At Dunkeld, during a snow storm!

kanban

is an excellent book by David Anderson (isbn 978-0-9845214-0-1). As usual I'm going to quote from a few pages:

The essence of starting with Kanban is to change as little as possible.

Failure to make a delivery on a promised date gets noticed much more than the specific content of a given delivery does.

Counter-intuitively, most bottleneck management happens away from the bottleneck.

Speed is most useful if it is in the right direction… If your priority is to find and reduce the constraint you are often solving the wrong problem... The dramatic success of the Toyota Production System (TPS) had nothing to do with finding and eliminating bottlenecks. Toyota's performance gains came from using batch-size reduction and variability reduction to reduce work-in-progress inventory [Don Reinersten]

Kan-ban is a Japanese word that literally means "signal card" in English. In a manufacturing environment, this card is used as a signal to tell an upstream step in a process to produce more work. The workers at each step in the process are not allowed to do work unless they are signalled with a kanban from a downstream step.

Trust is a hard thing to define. Sociologists call it social capital. What they've learned is that trust is event driven and that small, frequent gestures or events enhance trust more than large gestures made only occasionally.

High-trust cultures tend to have flatter structures than lower-trust cultures.

The traditional approach to forming a commitment around scope, schedule, and budget is indicative of a one-off transaction. It implies that there is no ongoing relationship; it implies a low level of trust.

The more groups involved, the longer the meeting is likely to take. The longer the meeting, the less frequently you are likely to hold it…

Buffers and queues add WIP to your system and their effect is to lengthen lead time. However, buffers and queues smooth flow and improve predictability of that lead time. By smoothing flow, they increase throughput, so more work is delivered through the kanban system. Buffers also ensure that people are kept working and provide for greater utilisation. There needs to be a balance, and buffers help maintain it. In many instances you are seeking business agility through shorter lead times, and higher quality partly through lower work-in-progress. However, do not sacrifice predictability in order to achieve agility or quality. If your queue and buffer sizes are too small and your system suffers from a lot of stop-go behaviour due to variability, your lead times will be unpredictable, with a wide spread of variability. The key to choosing a WIP limit for a buffer is that it must be large enough to ensure smooth flow in the system and avoid idle time in the bottleneck.

The first principles of Kanban are to limit work-in-progress and to pull work using a visual signalling system.

You need slack to enable continuous improvement… In order to have slack, you must have an unbalanced value stream with a bottleneck resource.

The width of a bottle's neck controls the flow of liquid into and out of the bottle. We can pour quickly from a wide neck, but often with a greater risk of spillage. With a narrow neck, the flow is slower but it can be more precise… In general, a bottleneck in a process flow is anywhere that a backlog of work builds up waiting to be processed.

As we all know, there really is no such thing are multi-tasking in the office; what we do is frequent task switching.

test code is code but it's different

Here's a bit of code...

def to_roman(n)
  result = ''
  ['V',5,'IV',4,'I',1].each_slice(2) do |ch,unit|
    while n >= unit do
      result += ch
      n -= unit
    end  
  end  
  result
end

And here's a bit of test code...

def test_arabic_integer_to_roman_string
  assert_equal    "I", to_roman(1)
  assert_equal   "II", to_roman(2)
  assert_equal  "III", to_roman(3)
  assert_equal   "IV", to_roman(4)
  assert_equal    "V", to_roman(5)
  assert_equal   "VI", to_roman(6)
  assert_equal  "VII", to_roman(7)
  assert_equal "VIII", to_roman(8)
end

I'm sometimes asked which is more important, the code or the tests? Does it matter if you refactor the code but let debt accumulate in the tests? A question like this tells me that the questioner probably doesn't really understand that test-code is the yin to the yang of the code it tests and vice versa. That they form a co-evolving system. When you feel the tests need refactoring that isn't a sign you did something wrong. That's just part of development! The tests mean that code under test is not as closed a system as it would be without them. And being less closed it can stave off entropy for longer.

That's not to say that code and tests are the same. They're both code, but they're not the same.

For example, suppose I have a metric that calculates a measure of the complexity of code. I use this metric to calculate the complexity of the code under test and also the complexity of its tests. Imagine if the complexity of my tests is greater than the complexity of the code it tests. How do I know the tests aren't wrong? Should I write some tests for the tests? And if the pattern repeats and the complexity of the tests for the tests is higher still should I write some tests for the tests for the tests? No. That way leads nowhere. I don't want complexity in my tests. It want them simple. I want them linear. The complexity of my tests might be greater than one, but it must be smaller than the complexity of what it's testing.
Test code is code but it's different.

Or consider the names of functions. In code under test I want Goldilocks names - names that are not too long, and not too short. On the one hand I want reasonably long names - long enough to express intention. On the other hand, names always occur as sub-expressions of larger expressions. If my names are too long, the full expressions they're part of quickly become unwieldy. It's in the full expressions I really want understandability, so I can turn the name-length-dial down a bit.
But the names of my test functions are different. They are never part of larger expressions. So for them I can turn the name-length-dial up up up - all the way to, deep breath, no_newline_at_end_of_file_msg_is_gobbled_when_at_end_of_common_section
Test code is code but it's different.

the lean startup

is an excellent book by Eric Ries (isbn 978-0-670-92160-7). As usual I'm going to quote from a few pages:

What if we found ourselves building something that nobody wanted? In that case what did it matter if we did it on time and on budget?

I've come to believe that learning is the essential unit of progress for startups.

If you cannot fail, you cannot learn.

Qualitative learning is a necessary companion to quantitative testing.

Pay no attention to the eight people behind the curtain.

When I could think of nothing else to do, I was finally ready to turn to the last resort: talking to customers.

That which optimizes one part of the system necessarily undermines the system as a whole.

Large batches tend to grow over time.

The paradoxical Toyota proverb "Stop production so production never had to stop."

We routinely asked new engineers to make a change to the production environment on their first day... They would ask, "What will happen to me if I accidentally disrupt or stop the production process?" ... We told new hires, "If our production process is so fragile that you can break it on your very first day of work, shame on us for making it so easy to do so." If they did manage to break it, we immediately would have them lead the effort to fix the problem as well as the effort to prevent the next person from repeating the mistake.

Switching to validated learning feels worse before it feels better... the problems caused by the old system tend to be intangible, whereas the problems of the new system are all too tangible.

certain to win

is an excellent book by Chet Richards (isbn 1-4134-5376-7). As usual I'm going to quote from a few pages:

Man is the child of custom, not the child of his ancestors. [Ibn Khaldun, 1377 A.D.]

Financially massive organizations warp the environment they inhabit much like the way gravitationally massive bodies warp space-time in physics.

Since what you're looking for is mistakes, a general rule is that bad news is the only kind that will do you any good.

A similar effect, a breakdown in the quality of energy, is well known to students of physics as "entropy". The energy is still there, but it isn't available for doing work. The insidious thing about entropy is that within a closed system, it always increases. In other words, closed systems run down.

The more you try to control people, the less control you get.

If a just-in-time production line had to wait for a formal decision process to work, it would hardly move at all, and it would never improve.

Complex methodologies can turn the attention of the company inwards.

We polish individual techniques and also train as a group.

It is not a matter of "getting it", it's a matter of doing it.

Mutual trust comes from mutual experience.

Most readers are familiar with the dualities that seem to inhabit everything oriental, the best known being the yin / yang. Yin signifies such things as endurance and the maintenance of the present states, while yang represents vitality and change. In the Taoist cosmology that underlies eastern strategy, neither can exist without the other, and the familiar symbol shows each depending on and containing the seed of the other.

security scan kanban

There's something about the typical kanban board I see that feels wrong. Let me try to explain. It's to do with what kanban means; what a kanban is. A true kanban is a physical thing used to regulate supply and demand. For example, consider putting your hand-luggage through an airport security scanner. You put your hand-luggage into empty trays. The trays go through the scanner. At the other side you take your hand-luggage out of the trays. The emptied trays go back to the start (often on a separate conveyor belt underneath) for someone else to put their hand-luggage into. Often you're ready to put your hand-luggage through the scanner but you have to wait because there are no empty trays. The number of trays represents the capacity of the system. Each tray is a kanban.

Now let's take a look at a typical kanban board such as the one on the left. It has two columns; one column for the activity of Wibbling which has a work-in-progress (WIP) limit of 5, and one column for the activity of Fubaring which has a work-in-progress limit of 4. There are 3 blue stories being Wibbled and 4 blue stories being Fubar'd.

How is this board supposed to be used? I count the number of blue stories currently being Wibbled - 1,2,3. I look up to the top of the Wibbling column and see its WIP limit is 5. I know 3 is less than 5 so I know there is some spare Wibbling capacity. If I want to know how much I can do the math, 5-3==2. And repeat... I count the number number of blue stories currently being Fubar'd - 1,2,3,4. I look up to the top of the Fubaring column and see its WIP limit is 4. I know 4 is the same as 4 so there is currently no spare Fubaring capacity.

It seems odd to have two columns that look exactly the same when one has spare capacity and the other does not!

My problem is that 5 and 4 are abstractions!
5 is not the same as 5 trays.
5 is just 5.
But 5 trays is [Tray Tray Tray Tray Tray].

What's important about the 5 isn't the 5.
What's important about the 5 is that it represents 5 trays.
What's important about the 5 is the relationship between the 5 and the number of blue stories it limits. The Wibbling blue story WIP limit is abstracted to a 5 but the blue stories are not. I have 1,2,3 blue stories in the Wibbling column but I'm not representing them with a 3, I'm representing them with 1,2,3 actual blue story cards.

Here's the alternative 'scan kanban' I'm thinking of.
I remove the 5 WIP limit from the top of the Wibbling column. Instead I put in 1,2,3,4,5 orange  Tray s.
I remove the 4 WIP limit from the top of the Fubaring column. Instead I put in 1,2,3,4 red  Tray s.

I add the rule that every  Story  must always be in a tray.

Now I can instantly see the Wibbling column has some spare capacity because there are some  Tray s with no  Story  in them. I can count the spare capacity if I want -  1 ,  2 . No math needed. Easy. I can instantly see the Fubaring column has no spare capacity because all the  Tray s contain a  Story .

Now I have a kanban board that actually has kanban on it!
Each tray is a kanban.
Now I can experience pull.

Suppose the workers to the right of the Fubaring column use green  Tray s. They send an empty  Tray  to the Fubaring column to signal they're ready to pull a  Story .

The workers in the Fubaring column move a done  Story  from their  Tray  into the empty  Tray .

The workers to the right of the Fubaring column pull the  Story  in its  Tray  back into their column.

The workers in the Fubaring column now have an empty  Tray .

The workers in the Fubaring column can send their empty  Tray  to the Wibbling column to signal they're ready to pull a  Story .

Thus each  Story  flows left to right while each empty  Tray   Tray   Tray  moves right to left.

Once we have a system under control we can try to improve. For example, we can see what happens when we reduce the WIP limit by 1 in the Wibbling column. We simply remove a  Tray .





Caveat Emptor: I've never seen this style of kanban in actual use. It's just an idea so far. As always, if anyone has any feedback it would be much appreciated.

Update! Here's a follow up blog entry.

Andy and the crusty cobs

My friend Andy works on the railways. He used to work as a master baker at one of the big supermarkets. One of the items he baked was Crusty Cobs.

Clearly customers can only buy Crusty Cobs if there are some Crusty Cobs actually on the shelves. So the supermarket also employed a bakery supervisor to check Crusty Cob availability. (Naturally the supervisor was paid more than Andy. After all he was the supervisor.) Every day, at 4pm, the bakery supervisor arrived, clutching his clipboard, and counted the Crusty Cobs on the shelves; one, two, three, four, five, six, seven... Andy got praised if there was good Crusty Cob availability and chastised if there wasn't.

So Andy adapted. He baked extra batches of Crusty Cobs but waited to put them on the shelves until shortly before 4pm. Crusty Cob availability, as recorded by the supervisor, was very good. Customers wanting Crusty Cobs before 4pm were often disappointed.

A short while later as well as counting the Crusty Cobs on the shelves (always at 4pm), the supermarket also started counting the number of Crusty Cobs actually sold at the tills during the day. Now Andy also got praised if lots of Crusty Cobs were sold.

So Andy adapted. As well as sticking the Crusty Cob labels on the Crusty Cobs, he also printed extra Crusty Cob labels and stuck them on some Soft Rolls (same price). After all, who looks at the labels? So Crusty Cob sales, as recorded at the tills, were always very good. The number of Crusty Cobs actually bought by customers was somewhat less.

Not long after, the supermarket noticed a drop in the sales of Soft Rolls. They asked Andy not to bake so many Soft Rolls.

So Andy adapted. He baked less Soft Rolls, labelled them correctly, and put the extra Crusty Cob labels on the French Sticks. Customers wanting Soft Rolls, at any time, were often disappointed.

Goodhart's Law states:

When a measure becomes a target, it ceases to be a good measure.

Or, more colloquially:

Targets are where measures go to die.

the mind of war

is an excellent book about John Boyd by Grant T. Hammond (isbn 978-1588341785). As usual I'm going to quote from a few pages:

…in order to determine the consistency of any new system we must construct or uncover another system beyond it… One cannot determine the character or nature of a system within itself. Moreover, attempts to do so lead to confusion and disorder.

he was always testing the limits - of airplanes, people, science, the military, and, most especially, bureaucracies.

In dozens of interviews, conducted for this book, the most consistent theme and nearly universal comment was that John Boyd was the essence of an honourable man and incorruptible.

Oral, not written, communication and conviction, not accuracy, still rule in military culture.

Boyd liked putting things together (synthesis) better than analysis (taking things apart)...

He also came to appreciate the routine practice and repetition that was required to become really good at something and to overcome the boredom by focusing on minute improvements.

He observed very carefully.

Boyd could go from 500 knots to stall speed, practically stopping the plane in midair, which would force any aircraft on his tail to overshoot him and thus gain the advantage for Boyd. In another trick, he would stand the F-100 on its tail and slide down the pillar of its own exhaust. Fire would come out of the intake in the nose of the aircraft and the tailpipe simultaneously. A seemingly impossible feat, it was challenged by others. Boyd when to Edwards Air Force Base in California, where NASA had two fully instrumented F-100 aircraft, and demonstrated it and other techniques to a series of nonbelievers. The test pilot at Edwards who challenged him at the time was a fellow by the name of Neil Armstrong.

Boyd had never designed an airplane before, but as he told Colonel Ricci and Gen. Casey Dempsey, "I could fuck up and do better than this."

The rule of thumb in the Air Force is that a plane will gain a pound of weight a day for the life of the aircraft.

High entropy implies a low potential for doing work, a low capacity for taking action or a high degree of confusion or disorder… The tendency is for entropy to increase in a system that is closed or cannot communicate with the external systems or environments.

A natural teacher, he understood that if he told you something, he robbed you of the opportunity to ever truly know it for yourself.

We are never deceived. We only deceive ourselves. [Goethe]

Boyd's dictum: "Ask for my loyalty and I'll give you my honesty. Ask for my honesty and you'll have my loyalty."

Quality Software ManagementVol 4. Anticipating Change

is an excellent book by Jerry Weinberg (isbn 0-932633-32-3). This is the second (or is it third?) snippet review for this book (here's the first). As usual I'm going to quote from a few pages:

Ultimately what helps you most in managing system size is courage and realism.

Without action things will only get less visible over time.

Human systems don't change unless the individuals change, one at a time.

Growth is always non linear.

... the incidence of test failures is directly proportional to the square of the size of the crowd multiplied by the rank of the senior observing official [Augustine's Laws]
The dynamic behind this law is simple: A large crowd of high dignitaries means that the event is planned according to external, not internal, events.

It is easy to look at this diagram and believe that you're seeing a defined process. You're not. What you're seeing is an optical illusion.

"model" is just another name for a guide to anticipating the future.

Cultural changes have a much greater potential impact than process changes because one cultural change - such as driving fear out of the workplace - can affect hundreds of process changes.

Feedback works on continuity. The pieces in the model cannot be too large (or response will be slow), and they must be stable (or response will not be predictable).

Error prone modules are born error prone and stay error prone.

go pair

At the recent Agile on the beach conference I was chatting to Paul Massey who runs BlueFruit Software in Redruth (which, by the way, is a great company to work if you're looking for a job as an embedded developer). I was reminded of something Paul mentioned to me a while back that relates to pair-programming. He and a friend played go against other people on a server. Their go rankings were 8k and 9k respectively. Interestingly, they then played against other people as a pair. And their joint-ranking went up to 6k. Paul writes:

When I play go, I have a problem that I struggle to find the patience for strategic thinking. I combat this problem by doing strategic thinking when my opponent is playing and tactical thinking when it's my turn. Yesterday I realised the parallel with pair programming, i.e. tactical thinking when at the keyboard, and strategic thinking when away from the keyboard. Both are important.

cyber-dojo dates

If you'd like me to run a private cyber-dojo for you please email me. I love travelling (especially to places where you can fly-fish for atlantic salmon) and meeting new people. There is often no charge but I do ask you to cover my travel expenses.

past cyber-dojos

I no longer keep this list updated...

• Agile on the Beach conference, 5th-6th September 2013, Falmouth, U.K.
• C course for Cisco, in Chennai, 29th-31st July 2013
• C course for Cisco, in Bangalore, 24th-26th July 2013
• Geoffrey Dunn (@gd_twit) installed and ran a cyber-dojo in Sydney, Australia, 26th July 2013
• Cambridge Software Craftsmanship meetup, organized by Ceri Shaw, 18th June 2014, in Cambridge, U.K.
• Malooba startup, with Errol Chavannes, 23rd May 2013, in Falmouth, U.K.
• IDBS with Brian Hobbs, 21st May 2013, in Guildford, U.K.
• Agile Adria conference, 22nd-23rd April 2013, in Zagreb
• ACCU 2013 conference 10th-13th April 2013, in Bristol, U.K.
• SkillsMatter's 3 Day Agile Development course, 25th-27th February 2013, in London
• Synch-Norwich meetup, 24th January 2013
• C++ patterns course for Cisco, in Oslo, 20th November 2012
• With Chris Vaux at GCHQ, 16th November 2012, in Cheltenham, England
• For Ericsson, 13th November 2012, in Croatia
• Ed Sykes ran a recruitment cyber-dojo 24th October 2012
• Vincent Baur (of Schlumberger) ran another C++ cyber-dojo with even more attendees in Clamart (suburb of Paris) on 23rd October 2012
• Olve Maudal (of Cisco) ran a cyber-dojo, 10th October 2012, in Oslo
• SkillsMatter's 3 Day Agile Development course, 26th-28th September 2012, in London
• At a BCS SPA meeting, 2nd October 2012, London
• Mark Chen (of Schlumberger) ran a C++ cyber-dojo in Calgary on 26th September 2012
• Vincent Baur (of Schlumberger) ran a C++ cyber-dojo in Clamart (suburb of Paris) on 24th September 2012
• Mike Long and James Grenning ran a full day coding dojo workshop in Houston, USA, week of 10th September 2012
• At Agile On The Beach conference, Cornwall, England, 7th September 2012
• For Aviva, in Perth, Scotland, 13th-15th August 2012
• For Cisco, in Bangalore, India, 31st Sept - 4th August 2012
• SkillsMatter's 3 Day Agile Development course, 18th-20th June 2012, in London
• at IKON 12th-14th June 2012, in Teddington, England
• Mike Long ran another big cyber-dojo for 32 people in Beijing, Thursday 17th May 2012
• After attending the Scandinavian Developer Conference Emil Jönsson ran his own cyber-dojo
• At Skillsmatter's Progressive Java tutorial London, 14:00pm, Friday 4th May 2012
• at the bar in the evenings at the ACCU conference, Oxford, 24th-28th April 2012
• Scandinavian Developer Conference, as part of a Mastering Agile Practice tutorial co-run with Kevlin Henney, 18th-19th April 2012
• At Skillsmatter's London office, 6:30pm, Thursday 5th April 2012(not the 3rd)
• Ericsson in Helsinki, Finland, 19th-20th March 2012 (read blog-entry)
• Agile Riga in Latvia, 17th March 2012
• at Manpower, in Uxbridge, England, with Steve Brownett, 22nd Feb 2012
• Ericsson in Budapest, Hungary, 8th Feb 2012
• Ericsson in Helsinki, Finland, 18th January 2012
• Olso Coach Camp in Norway, 7th January 2012
• Opera in Oslo, Norway, 15th December 2011
• Mike Long used cyber-dojo during the Global Day of Code Retreat in Beijing, 3rd December 2011
• XP Day 21st November 2011, London, England
• SkillsMatter's 3 Day Agile Development course, 9th-11th November 2011, in London
• Ericsson, 3rd November 2011, Helsinki, Finland
• Diamond Light Source with Mark Booth, 28th October, Didcot, England
• IKON 27th October, in Teddington, England
• Ericsson, 20th October, in Budapest, Hungary
• Lars Thorup ran a TDD coding dojo using cyber-dojo at the Silicon Valley Code Camp 8th-9th October
• Agile .NET Europe in Zebrastraat, Ghent, Belgium, 10th, 11th October
• Ericsson Agile Conference, 20th-21st September, Helsinki, Finland
• Tampere Goes Agile conference, 17th September, Finland
• Agile on the beach conference, 15th-16th September, Falmouth, England
• Agile Lean Europe conference, 7th-9th September, Berlin, Germany
• McAfee, 31st August, Aylesbury, England
• Jeremy Lightsmith and Gary Bernhardt tried to run a TDD coding dojo using cyber-dojo at the Agile 2011 conference, 7th August, Salt Lake City, Utah
• SkillsMatter's 3 Day Agile Development course, 3rd-5th August, in London
• Mott MacDonald in Croydon, 21st July. Read blog entry
• Olve Maudal (@olvemaudal) ran a cyber-dojo in San Jose, California, 23rd June
• with Kevlin Henney (@kevlinhenney) in a 2 day version of our Mastering Agile Practice course, 21st-22nd June, for CommaSoft, in Germany. Read blog entry
• Erik Bakstad (@ebaxt) and Johannes Brodwall (@jhannes) ran a cyber-dojo at JavaBin, Oslo, Friday, 17th June
• Mike Long (@meekrosoft) ran a cyber-dojo at Schlumberger, Oslo, Friday 17th June
• NDC conference, 8th-10th June, in Oslo, Norway. Read Gil Zilberfeld's blog posting.
• Software Craftsmanship conference, 26th May, at Bletchley Park, England.
  Read Jean-Laurent Morlhon's (@morlhon) blog writeup (original in French)
• Absolute software 24th May, in Redruth, UK
• SkillsMatter's 3 Day Agile Development course, 4th-6th May, in London
• Olve Maudal and myself ran a cyber-dojo (at the bar!) during the ACCU 2011 conference, April 13-16th, in Oxford
• Kevlin Henney and myself used cyber-dojo as part of our ACCU 2011 pre-conference tutorial , A Day of Deliberate Practice, April 12th, in Oxford
• Kevlin Henney and I ran a Day of Deliberate Practice partly using cyber-dojo in our DevWeek 2011 Post-Conference Workshop, 18th March, at The Barbican Conference Centre, London
• The return of cyber-dojo, Oslo Coding Dojo meetup, Wednesday 16th February, 6pm at The Scotsman, Karl Johan's Gate 17
  Read David Kowalski's blog writeup
• Opera in Linköping, Sweden, January 17th-20th 2011
• Alex Graff ran a cyber-dojo at Pinesoft where he works.
• Lean Kanban eXchange conference, December 3rd
• XP Day London conference, November 29th, 30th
• Pure in Kings Langley, Herts, 16th November
• GothPy meetup, Göteborg, Sweden, 3rd November
• JAOO 2010 conference Aarhus, Denmark, 4th-8th October
• Agile Cambridge 2010 conference, UK, 15th October.
  Read Simon Prior's blog writeup.
  Read Paul Grenyer's blog writeup.
• ACCU London meeting, hosted by SkillsMatter, 16th September.
  Read my blog entry
• JavaZone 2010 conference Oslo, Norway, 8th September
• Trondheim Coding Dojo Norway, June 30th, 2010.
  Read Anders Hammervold's blog writeup
• NDC 2010 conference Oslo, Norway, June 16th-18th.
  Watch Scott Bellware's video
• XP 2010 conference Trondheim, Norway, 1st-3rd June.
  I ran one session with Jeremy Lightsmith on Tuesday morning. Together with Olve Maudal I ran another session at the Open Space on Thursday evening
• Oslo Coding Dojo Norway, 18th May 2010
• ACCU 2010 conference Oxford, UK, 14th-17th April.
  I ran three cyber-dojos with Olve Maudal.
  Read my blog writeup
  Read Mike Long's blog writeup

the first cyber-dojo Raspberry Pi

Here's the first Raspberry Pi raised from cyber-dojo donations :-) Everything works so it's off to my old school in the morning.

In case you're thinking "where is it?" (it's only credit-card sized) it's to the right of Lightning McQueen and Tow Mater, between the keyboard and the monitor with an aptly raspberry coloured ethernet cable attached. Here's a close-up.

And here's cyber-dojo running in Midori. More Pies have been ordered and should be arriving in the next few days. Many thanks for your donations. Keep them coming!