Average Time Since Green

Last week I spent an afternoon discussing the Average Time To Green Game with my friend Olve Maudal from Tandberg. We discussed making the game browser-based where everything happens on a game server; the source files are held on the game server, they're compiled on the game server, the unit tests are run on the game server. Suddenly the game morphs from the average time to green into the average time since green (atsg). This is a profound change. It means the atsg can be continuously monitored during an iteration rather than merely being calculated once at the end. (It reminds me of the difference between Linear Thinking and Systems Thinking.) It also opens up masses of further interesting deliberate practice possibilities for the game. And perhaps also for actual software development. I'm working on a proof of concept using Ruby on Rails.

File extension repetition considered harmful?

Don't Repeat Yourself is a well known software development mantra. Suppose your working on a C project and all the #include lines name header files whose name ends in dot-h. Aren't all those dot-h's repetition? They are. But you probably don't consider that as repetition because in this context it doesn't matter. However, sometimes file extensions become a bigger part of the context and then the repetition can start to hurt. For example, I have some PHP files whose names end in dot-php. The content of these PHP files contain some hardwired redirections to sibling dot-php files. These files live on my ISP's server and they run under PHP4. My ISP has upgraded to PHP5 - but only for files ending in dot-php5. Is it worth upgrading I ask myself.

Java initializer block trick

Here's a neat trick Kevlin Henney found in some code Michael Feathers posted at http://pastie.org/534364 Instead of doing this:

ArrayList<Integer> values = new ArrayList<Integer>();
values.add(1);
values.add(2);
values.add(3);

You can do this:

ArrayList<Integer> values = new ArrayList<Integer>() {{
    add(1);
    add(2);
    add(3);
}};

Abstraction

I came across this astonishing illusion the other day (via a Jerry Weinberg twitter). The blue spirals and the green spirals are the same colour. Really. Jerry said he'd looked at the pixels in Photoshop to confirm it. It's not that I don't trust Jerry but I looked too (in Gimp) because it's just so counter-intuitive. It is true. It's amazing isn't it.

The way we perceive the world is based partly on what information it's sending us but mainly on how we interpret that information. If we consciously saw everything we just couldn't cope with the information overload so our minds filter it for us. That's abstraction. We don't see what's there and we see what's not there.

You can see the learning

This is the barchart from The Average Time To Green Game I blogged about earlier. It's interesting to study this barchart.

1. The first green bar on the left is the highest. At this point the students hadn't realized what the aim of the game was - to control the average time to green. Everyone was busy beavering away on what they incorrectly thought they were being judged on - the trivial exercise of stripping backslash newline characters from a buffer. A couple of laptops would probably have carried on into the night without some gentle prompting. The height of the bar represents the average time to green remember. There were four very high individual times and the standard deviation was a hefty 174 (seconds).
2. The second green bar is marginally lower, indicating the average time to green had reduced slightly. Remember that before each iteration everyone swapped pairs (and moved to a new laptop). The pair swapping was the primary mechanism by which strategies for controlling the average time to green were passed on. At this point only one swap had taken place so not much strategy passing on had occured. There were three very high individual times and an even heftier standard deviation of 196.
3. The third green bar shows a bigger drop. Since there had now been two pair swaps the effect of passing on strategies was now more marked. Only now was the group really starting to realize what the true aim of the game was. They were starting to experience first hand how effective simply strategies like frequent compilation can be. There were two high individual times and the standard deviation was now down to 142.
4. By the fourth iteration three pair swaps had taken place and all the groups now genuinely understood the aim of the game (which was not to produce the world's greatest unsplice function!) Every single group got to green within 8 seconds. It was noticeable that on this iteration, as soon as the bell was rung each group chose to get to and stay at green at their first possible opportunity. The standard deviation was only 22.
5. The fifth iteration bar is perhaps the most interesting of all. By this time everyone understood the aim of the game and everyone was able to get their laptop to green within a very short time of the bell ringing. However, with some gentle prodding we suggested that they didn't have to stay at green at their first opportunity. Consequently, they were starting to decide whether to stay at green or continue for a little longer. They were starting to think about all the pairs not just their pair. This was the first real point when they were in control of the development process and not vice-versa. The standard deviation was 71.
6. For the sixth iteration we suggested the group aim for an average time to green of 60 seconds (they were at 130 seconds for the fifth iteration). On the seventh iteration the average was 85 seconds with a standard deviation of 74. Pretty impressive.

At the end of the game people were really starting to act with more team awareness. On the first iteration none of the green pairs offered to help a red pair - they just sat watching. In contrast by the end (with a little prompting) some green pairs were offering help to a red pair.

Something else that was noticeable too: at the first iteration everyone looked quite tense but by the end they all realized it really was a game and they all looked at lot more relaxed. Their manager Lars commented on how marked this difference was.

In a review the next day one of the developers commented "suddenly baby steps were being encouraged and large steps were being frowned upon." One developer recounted instinctively firing up the debugger, only to be persuaded by their partner that there were other more effective strategies.

Campaign for small change

When I'm buying something at a shop and I pay with cash I rarely have the right money so I usually get some loose change back. When this happens I look for a charity tin to drop the small coins into. I'd like to claim this is an act of generosity and altrusim but I can't. For one it's not very generous since it's only small change, and for another it's not very altrusitic since I do it mostly to reduce the hassle in my life. I go to great lengths to try to keep my life hassle free and I don't like lots of loose change in my pocket.

Of course, giving away my small loose change means I'm unlikely to ever have the correct change which creates a pleasant reinforcing feedback loop - broken only when there is no charity tin :-(

Doing this on my own doesn't make a lot of difference, but if everyone did it... Are you willing to give away your small change each time you shop? Would you sign up to a "campaign for small change"?

10,000+ warnings is people problem

When you read "there is always a problem and it's always a people problem" it's easy to get the wrong idea. Technical problems and people problems are almost always deeply intertwined.

For example, suppose you're part of a team that has let warnings accumulate one on top the other over many months until they now number 10,000+. Aside from the obvious technical problem of having 10,000+ warnings the team has a much deeper people problem.

Ask yourself the question - why does the team continue to live with the pain of 10,000+ warnings? Their answer is "that's how its always been". Sure the number creeps ever upward, but what does that matter when they're up to 10,000+? The team have had so many warnings for so long that they no longer even think about them! That's abstraction!

And given that the team don't even see 10,000+ warnings as a problem will they be motivated to get rid of them? Unlikely. They've lived with 10,000+ warnings for so long they've become comfortable with the discomfort they cause! Of course they claim they're not in discomfort. Abstraction again!

They are caught in a vicious circle of blindness and numbness. To solve this people problem the first step is to somehow get the team to see and feel the pain 10,000+ warnings are causing. That will be difficult. Then you'll have to get individuals to change their behaviour. That will be difficult too.

Oh, and you also have a smaller problem - 10,000+ warnings.

Ha ha

An old couple go to the doctors. The doctor says they are fine for their age but he's noticed they are starting to become more forgetful and suggests they write things down so they don't forget them. Back at home the husband asks the wife if she'd like anything to eat. "A bit of ice cream would be nice" she replies. Anything with it he adds. "Some chocolate sauce. And you'd better write it down or you'll forget". The husband says he doesn't need to write it down. "And a spinkling of nuts too" adds the wife. "And you'd better write it down". Again the husband is sure he doesn't need to write it down. "Oh and a cup of tea" adds the wife again. "You'd better write it down". Still he doesn't write it down. Of goes the husband to prepare the food. Fifteen minutes later he walks into the lounge with a lovely english breakfast on a plate: fried eggs, bacon, sausage, mushrooms, tomato. The wife looks at the plate and says "where's the toast?"

Complexity, testing, coverage

Several years ago I was bitten by a bug where my C# unit tests weren't being run because I had accidentally left the public specifier off my test class definition (and therefore NUnit could not see the class). It was easy to fix - I simply added the public keyword. Wondering how many other times I had accidentally done the same thing I decided to write another NUnit test whose job was to use reflection on its own assembly looking for test classes that weren't public. I was pondering this episode again recently while doing some C training. In C (and C++) you don't have reflection. It occured to me that you might be able to use code coverage to avoid this potential problem. If the coverage could separate out the test code from the tested code you could use less than 100% coverage of the test code to help indicate one or more accidentally uncalled test functions. This highlights that test code and tested code are not the same. Test code typically has lots of detail complexity but very little dynamic complexity.

Agile Software DevelopERS

De Luca's Law says that Managing software developers is 20% technology and 80% psychology. Dale Carnegie believed it was 15/85. On my bookshelf are half a dozen Agile books. All are about Agile Development, Agile Management, etc. None are focused on Agile Developers - on how to improve your agility as an individual. Conway's Law says "An organisation is constrained to produce designs which are copies of the communication structures of the organisation." Can this law be extended to consider agility?

A design cannot be more agile than the developers who designed it.

Intuitively it seems a reasonable idea. If XP aims to improve technical practices, and Scrum to improve management practices then perhaps it's time for a movement aimed at improving personal practices?

Bill Bailey's Law

Here's a law I invented which I use sometimes when training or consulting.

Never believe any assertion containing the words never or always.

It can be applied to itself of course. And to many other laws. If you believe it then you don't and if you don't then you do. It essentially says you should think for yourself. Bill Bailey is a UK comedian. On UK tv one time he recounted being shown into a big cat enclosure in a zoo in Chile. The keeper, whose English was very broken, said "Never face the cats". Bill said ok and they went in. After they had been inside for a while the keeper turned to Bill and said, "No no, sorry, always face the cats". Bill joked that the keeper added "we lose a lot that way".

The Average Time To Green Game

Something special happened last week. I was in Bangalore doing some training at the request of my good friend Olve Maudal of Tandberg (now part of Cisco). A day on Test Driven Development was scheduled for Monday and I fell asleep Saturday night thinking about how to really get across the idea and nature of TDD to a group of developers. I woke up at 2am Sunday morning with The Average Time To Green Game pre formed and named, ready in my head!

Game setup

1. Each computer is given a label (eg Alligators, Bears, Cheetahs, etc).
2. Minimum of two people per computer.
3. Each computer must have a TDD framework installed (better still, use CyberDojo).

Game play

1. Every 10-15 minutes ring the Average Time To Green Bell (we found a small brass bell in a local shop).
2. When you ring the bell you also start a timer and project it so everyone can see it. This timer starts at zero and increments second by second.
3. The aim at each computer is then to get to green (all tests passing).
4. When a computer has got to green two things have to be recorded for that computer: the iteration number, the time it took to get to green since the bell (simply look at the projected timer).
5. When your computer is green you have to cover your laptop (we provided sheets of green paper) and wait till all the computers have got to green.
6. The data for all computers is recorded in a spreadsheet.
7. When all computers are at green everyone briefly looks at the graph made from the spreadsheet.
8. Then you have to swap partners and computers and a new iteration starts.

Game Goal

The goal of the game, which was clearly and explicitly printed on the instruction sheets was simply to control the average time to green across the whole group. The group naturally didn't understand that at first - they focused instead on the problem. The problem was completely trivial - Olve and I picked stripping backslash newline pairs off a character buffer - as in C/C++ preprocessor logical lines. It was utterly fascinating to watch how things progressed, and we feel it worked really well (and more to the point I think the participants did too), both in the TDD sense and in the team building sense.

Game photos

1. Graph of the average time to green over several iterations.
2. Helping to solve one computer that was holding the team up.
3. Total number of tests passing split by computer.
4. Relaxing at the end.

Game retrospective

1. The group were all well above average ability so we could have perhaps run fewer iterations, or used a less trivial problem.
2. We could have used staged goals. First measure the average time to green, then lower it, then control it.
3. Once the group felt they had control of the average time to green we could have let them choose their own goals.
4. We could have encouraged participants to write down their choice of strategies and their experience of pair programming.

Here's a follow up blog-entry/

C# 2.0 - Visitor return type

In this entry I'm revisting the Visitor Pattern, this time using C# 2.0 generics rather than Java generics. One addition to the previous design is to declare two interaces, one for a void return and one for a non-void return:

public interface IVisitable
{
    void Accept(IVisitor visitor);
    R    Accept(IVisitor1<R> visitor);
}
public interface IVisitor
{
    void Visit(A visited);
    void Visit(B visited);
    void Visit(C visited);
}
public interface IVisitor1<R>
{
    R Visit<R>(A visited);
    R Visit<R>(B visited);
    R Visit<R>(C visited);
}

Each concrete element then simply implements both Accept methods:

public class A : IVisitable
{
    ...
    public void Accept(Visitor visitor)
    {
       visitor.Visit(this);
    }
    public R Accept<R>(Visitor1<R> visitor)
    {
        return visitor.Visit(this);
    }
    ...
}

I could do this in Java too of course. However, the C# version edges out the Java version in two respects. First, C# allows two types to have the same name as long as they have a different number of generic arguments.

public interface IVisitor
{
    void Visit(A visited);
    void Visit(B visited);
    void Visit(C visited);
}
public interface IVisitor<R>
{
    R Visit<R>(A visited);
    R Visit<R>(B visited);
    R Visit<R>(C visited);
}

Secondly, generic IVisitor works with all types (including value types such as bool) without the need for wrapper classes:

public class ExampleVisitor : IVisitor<bool>
{
    public bool Visit(A visited) { ... }
    public bool Visit(B visited) { ... }
    public bool Visit(C visited) { ... }
}

Statement Refactoring

I've just noticed this somewhere:

public string MakeRequest(string hostname, string message)
{
    Socket socket = null;
    IPAddress serverAddress = null;
    IPEndPoint serverEndPoint = null;  
    byte[] sendBytes = null, bytesReceived = null;
    int bytesReceivedSize = -1, readSize = 4096;
    serverAddress = Dns.Resolve(hostname).AddressList[0];
    serverEndPoint = new IPEndPoint(serverAddress, 80);
    socket = new Socket(AddressFamily.InterNetwork,
    SocketType.Stream, ProtocolType.Tcp);
    bytesReceived = new byte[readSize];  
    sendBytes = Encoding.ASCII.GetBytes(message);
    socket.Connect(serverEndPoint); 
    socket.Send(sendBytes);
    bytesReceivedSize = socket.Receive(bytesReceived, readSize, 0);
    socket.Close();
    if(-1 != bytesReceivedSize)
    {
        return Encoding.ASCII.GetString(bytesReceived, 0, bytesReceivedSize);
    }
    return "";
}

Why do programmers still write code like this in C#? Do they like verbosity? Why not write this instead:

public string MakeRequest(string hostname, string message)
{
    IPAddress serverAddress = Dns.Resolve(hostname).AddressList[0];
    IPEndPoint serverEndPoint = new IPEndPoint(serverAddress, 80);
    Socket socket = new Socket(AddressFamily.InterNetwork,
                               SocketType.Stream, ProtocolType.Tcp);
    int readSize = 4096;
    byte[] bytesReceived = new byte[readSize];  
    byte[] sendBytes = Encoding.ASCII.GetBytes(message);
    ...
}

It seems to me there are loads of these statement level refactorings that would prove really useful. Do any tools support "small" refactoring like this? Do any books talk about them?

Here's another: Instead of writing:

if (expression)
    return true;
else
    return false;

why not just write:

return expression;

How Buildings Learn

I've been rereading How Buildings Learn by Stewart Brand. It's a really great read. It's author took the time and effort to do what few authors do. He didn't do the simple thing and just write about buildings at a certain point in time. He did the difficult thing and wrote about the underlying processes that govern the evolution of buildings over time.

At the heart of this book is the idea of change - of time. That buildings change. That people change them. That the elements change them. This, fundamentally, is the reason the parallels with software leap off every page. Understanding and managing change is arguably the fundamental aspect of understanding and managing the process of developing software. Software isn't written perfectly in a sudden flash. It takes time. It takes lots of small changes. The author writes "My approach is to examine buildings as a whole - not just whole in space, but whole in time". He laments the aphorism "Form ever follows function" written in 1896 by Louis Sullivan (A Chicago highrise designer) because "it misled a century of architects into believing that they could really anticipate function". The idea is to aim for software that gets better over time. Or, more accurately, that is capable of getting better over time.

Here are some quotes:

Our basic argument is that there isn't such a thing as a building. A building properly conceived is several layers of longevity of built components.

Many buildings are demolished early if their outdated systems are too deeply embedded to replace easily.

Hummingbirds and flowers are quick, redwood trees slow, and whole redwood forests even slower. Most interaction is within the same pace level.

The dynamics of the system will be dominated by the slow components. with the rapid components simply following along. Slow constrains quick; slow controls quick.

Trust, intimacy, intense use, and time are what made these buildings work so well.

Architects offered themselves as providers of instant solutions, and only the look of a building gives instant gratification.

Hackers and Painters

I've just finished reading Hackers and Painters by Paul Graham. It's a really excellent read which I highly recommend. Here are some quotes:

Hackers need to understand the theory of computation about as much as painters need to understand paint chemistry.

Hackers start original and get good, and scientists start good, and get original.

Everyone by now knows the danger of premature optimization. I think we should be just as worried about premature design.

When you switch to a new model, you realize how much software development is affected by the way it is released.

Paying attention is more important to reliability than moving slowly.

Steve Jobs once said that the success or failure of a startup depends on the first ten employees. I agree.

Object oriented programming offers a sustainable way to write spaghetti code.

C# Gotcha

I've just spent 10 minutes blankly staring at my screen trying to figure out why the heck an ultra simple piece of C# didn't work... Suppose you create the following and compile it into an assembly called JSL.HelloWorld.dll

namespace JSL.HelloWorld
{
    public class Example
    {
        //...
    }
}

And then you write a test for it...

namespace Tests.JSL.HelloWorld
{
    using JSL.HelloWorld;
    using NUnit.Framework;

    [TestFixture]
    public class ExampleTest
    {
        [Test]
        public void Eg()
        {
            Example puzzle = new Example();
            //...
        }
    }
}

Looks fine. Yet when you compile the test csc complains that "the type or namespace name 'Example' could not be found...". Ten minutes blank staring follows. The problem is there are two namespaces called JSL.HelloWorld. There's one that's global that contains the class called Example we're after. And there's another one inside the namespace called Tests (called Tests.JSL.HelloWorld). So the compiler tries to resolve Example in Tests.JSL.HelloWorld but not in JSL.HelloWorld. One solution is as follows:

namespace Tests.JSL.HelloWorld
{
    using global::JSL.HelloWorld;
    using NUnit.Framework;

    [TestFixture]
    public class ExampleTest
    {
        //...
    }
}

BCS SPA 5th Jan 2005

Immo Hüneke has mercilessly bullied me into speaking at a BCS SPA meeting in central London. It'll be essentially the same talk I gave to the North East branch - a hotch potch of interesting quotes and various related topics loosely knitted together into something vaguely resembling a presentation.

The End of Certainty

is the title of an excellent book by Ilya Prigogine. It's subtitled Time, Chaos, and the New Laws of Nature and I reckon at least two of those three apply to software. Prigogine is a Nobel Prize winner (in Chemistry) and has worked on problems of non-equilibrium, complexity, self-organization, entropy, etc, for over four decades. Rather than writing an extensive review I'm going to open the book at a few random pages...

A nonequilibrium system may evolve spontaneously to a state of increased complexity.

Bifurcations are a source of symmetry breaking...Bifurcations are the manifestation of an intrinsic differentiation between parts of the system and its environment.

Once we have dissipative structures we can speak on self-organization.

...order can only be maintained by self-organization.

...constraints do not eliminate creativity, they provoke it.

...our position is that classical mechanics is incomplete because it does not include irreversible processes associated with an increase in entropy.

There is a necessary trade off between certainty at a given time for continuity through time.

I've won something!

I'm chuffed to learn one of my articles has won best Overload article for the year (the year in question being 2003). Writing unpaid articles isn't as altruistic as it appears. You learn a great deal by writing. If you're a developer your job is writing descriptions. Writing matters. Richard Gabriel's advice to developers is to learn poetry. And Dijkstra said that the best indicator of a developers skill was mastery of their native tongue. One of Michael Jackson's lexicon entries is called simply 'Descriptions'.