Siddhartha

is an excellent book by Herman Hesse. As usual I'm going to quote from a few pages.

He learned more from the river than Vasudeva could teach him. He learned from it continually. Above all, he learned from it how to listen, to listen with a still heart, with a waiting, open soul, without passion, without desire, without judgement, without opinions.

It also happened that curious people came along, who had been told that two wise men, magicians or holy men lived at the ferry. The curious ones asked many questions but they received no replies, and they found neither magicians nor wise men. They only found two friendly old men, who appeared to be mute, rather odd and stupid. And the curious ones laughed and said how foolish and credible people were to spread such wild rumours.

Is it not perhaps a mistake on your part not to be strict with him, not to punish him? Do you not chain him with your love? Do you not shame him daily with your goodness and patience and make it still more difficult for him?

Within Siddhartha there slowly grew and ripened the knowledge of what wisdom really was and the goal of his long seeking. It was nothing but a preparation of the soul, a secret art of thinking, feeling and breathing thoughts of unity at every moment of life.

From that hour Siddhartha ceased to fight against his destiny. There shone in his face the serenity of knowledge, of one who is no longer confronted with conflict of desires, who has found salvation, who is in harmony with the streams of events, with the stream of life, full of sympathy and compassion, surrendering himself to the stream, belonging to the unity of all things.

In every truth the opposite is equally true. For example, a truth can only be expressed and enveloped in words if it is one-sided. Everything that is thought and expressed in words is one-sided, only half the truth; it lacks totality, completeness, unity.

The sinner is not on the way to a Buddha-like state; he is not evolving, although our thinking cannot conceive things otherwise. No, the potential Buddha already exists in the sinner; his future is already there. The potential Buddha must be recognized in him, in you, in everybody. The world, Govinda, is not imperfect or slowly evolving along a path to perfection. No, it is perfect at every moment; every sin already carries grace within it, all small children are potential old men, all sucklings have death within them, all dying people - eternal life.

In order to learn to love the world, and no longer compare it with some kind of desired imaginary world, some imaginary vision of perfection, but to leave it as it is, to love it and be glad to belong to it.

It may be a thought, but I confess, my friend, that I do not differentiate very much between thoughts and words. Quite frankly, I do no attach great importance to thoughts either. I attach more importance to things.

I think it is only important to love the world, not to despise it, not for us to hate others, but to be able to regard the world and ourselves and all beings with love, admiration and respect.

The thing to me is of greater importance than the words; his deeds and life and more important to me than his opinions. Not in speech or thought do I regard him as a great man, but in his deeds and life.

Uncontrollable tears trickled down his old face. He was overwhelmed by a feeling of great love, of the most humble veneration.

The Hidden Life of Trees

is an excellent book by Peter Wohlleben (isbn 1771642483). As usual I'm going to quote from a few pages.

In forked trees, at a certain point, two main shoots form, they continue to grow alongside each other. Each side of the fork creates its own crown, so in a heavy wind, both sides sway back and forth in different directions, putting a great strain on the trunk where the two parted company. ... The fork always breaks at its narrowest point, where the two sides diverge.

The process of learning stability is triggered by painful micro-tears that occur when the trees bend way over in the wind, first in one direction and then in the other. Wherever it hurts, that's where the tree must strengthen its support structure. ... The thickness and stability of the trunk, therefore, builds up as the tree responds to a series of aches and pains.

There is a honey fungus in Switzerland that covers almost 120 acres and is about a thousand years old. Another in Oregon is estimated to be 2,400 years old, extends for 2,000 acres, and weighs 660 tons. That makes fungi the largest known living organism in the world.

You find twice the amount of life-giving nitrogen and phosphorus in plants that cooperate with fungal partners than in plants that tap the soil with the roots alone.

Diversity provides security for ancient forests.

There are more life-forms in a handful of forest soil than there are people on the planet.

As foresters like to say, the forest creates its own ideal habitat.

Commercial forest monocultures also encourage the mass reproduction of butterflies and moths, such as nun moths and pine loopers. What usually happens is that viral illnesses crop up towards the end of the cycle and populations crash.

The storms pummel mature trunks with forces equivalent to a weight of approximately 220 tons. Any tree unprepared for the onslaught can't withstand the pressure and falls over. But deciduous trees are well prepared. To be more aerodynamic they cast off all their solar panels. And so a huge surface area of 1,200 square yards disappears and sinks to the forest floor. This is the equivalent of a sailboat with a 130-foot mast dropping a 100-by-130 foot mainsail.

Why do tree grow into pipes in the first place?... What was attracting them was loose soil that had not been fully compacted after construction. Here the roots found room to breathe and grow. It was only incidentally that they penetrated the seals between individual sections of pipe and eventually ran riot inside them.

Sometimes, especially in cold winters, the old wounds can act up again. Then a crack like a rifle shot echoes through the forest and the trunk splits open along the old injury. This is caused by differences in tension in the frozen wood, because the wood in trees with a history of injury varies greatly in density.

the DevOps Handbook

is an excellent book by Gene Kim, Jez Humble, Patrick Debois, and John Willis (isbn 978-1-942788-00-3). As usual I'm going to quote from a few pages.

Make infrastructure easier to rebuild than to repair.

The average age of a Netflix AWS instance is twenty-four days.

Interrupting technology workers is easy, because the consequences are invisible to almost everyone.

In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures.

Over the following year, they eliminated testing as a separate phase of work, instead integrating it into everyone's daily work. They doubled the features being delivered per month and halved the number of defects.

Bureaucracies are incredibly resilient and are designed to survive adverse conditions - one can remove half the bureaucrats, and the process will still survive.

When we have a tightly coupled architecture, small changes can result in large scale failure.

Our deployment pipeline infrastructure becomes as foundational for our development processes as our version control infrastructure.

If we find that unit or acceptance tests are too difficult and expensive to write and maintain, it's likely that we have an architecture that is too tightly coupled.

Any successful product or organization will necessarily evolve over its life cycle... eBay and Google are each on their fifth entire rewrite of their architecture from top to bottom.

... which can lead to the unfortunate metric of mean time until declared innocent.

The principle of small batch sizes also applies to code reviews.

80% of MTTR (mean time to recovery) is spent trying to determine what changed.

High performing DevOps organizations will fail and make mistakes more often... If high performers are performing thirty times more frequently but with only half the change failure rate, they're obviously having more failures. [Roy Rapoport, Netflix]

Spiders repair rips and tears in the web as they occur, not waiting for the failures to accumulate. [Dr Steven Spear]

NDC Does C++ Countdown!

It was my pleasure to run a small workshop style session at the excellent NDC-London conference. I ran a fun C++ game which parodies the popular UK TV gameshow Countdown.

• In the TV version contestants take turns picking 9 random vowels/consonants and finding the longest word in 30 seconds.
• In my version contestants take turns picking 7 random tokens from 5 categories: (keywords, identifiers, operators, punctuators, literals) and writing the shortest C++ program using all 7 tokens in 8 minutes.

Contestants write their code in customized cyber-dojo sessions which automatically:

• checks which tokens have been used
• tells you the size of the program
• allows everyone to see all the submissions in the review

The rules:

• tokens must be the correct type; eg you cannot write "." or ... for a dot operator
• whitespace does not count towards the program's size
• additional tokens are allowed
• the program must compile
• the program is not executed
• warnings are allowed

In one round Phil Nash selected these 7 tokens:

const vector tokens =
{
    ".",                  // operator
    "switch",             // keyword
    "snafu",              // identifier
    ",",                  // punctuator
    "\"salmo\"",          // literal
    "goto",               // keyword
    "!",                  // operator
};

and the winning solution (54 characters long) was:

union X { X* x; };
X snafu() {
  l: switch (X().x,!"salmo"); goto l;
}

In another round Hulgar Frydrych selected these 7 tokens:

const vector tokens =
{
    "catch",              // keyword
    "->",                 // operator
    "[",                  // punctuator
    ";",                  // punctuator
    "--",                 // operator
    "foobar",             // identifier
    "operator",           // keyword
};

and the winning solution (53 characters long) was:

class c {
  c operator->(){ 
    foobar:
    try{
    }
    catch(c x[]){
        x--;
    }
  }
};

Can you create shorter versions?

the design and implementation of cyber-dojo

At the excellent Agile on the Beach conference in Cornwall I did a presentation outlining some of the history, design and implementation of cyber-dojo. The video has just gone live on youtube.

Continuous Delivery

Is an excellent book by Jez Humble and Dave Farley. As usual I'm going to quote from a few pages...

Software delivers no value until it is in the hands of its users.

The pattern that is central to this book is the deployment pipeline.

It should not be possible to make manual changes to testing, staging, and production environments.

If releases are frequent, the delta between releases will be small. This significantly reduces the risk associated with releasing and makes it much easier to to roll back.

Branching should, in most circumstances, be avoided.

Dashboards should be ubiquitous, and certainly at least one should be present in each team room.

One of the key principles of the deployment pipeline is that it is a pull system.

A corollary of having every version of every file in version control is that it allows you to be aggressive about deleting things that you don't think you need... The ability to weed out old ideas and implementations frees the team to try new things and to improve the code.

It should always be cheaper to create a new environment than to repair an old one.

The goal of continuous integration is that the software is in a working state all the time... Continuous is a practice not a tool... Continuously is more often than you think.

The most important practice for continuous integration to work properly is frequent check-ins to trunk or mainline.

Ideally, the compile and test process that you run prior to check-in and on your CI server should take no more than a few minutes. We think that ten minutes is about the limit, five minutes is better, and about 90 seconds is ideal.

Enabling developers to run smoke tests against a working system on a developer machine prior to each check-in can make a huge difference to the quality of your application.

Build breakages are a normal and expected part of the process. Our aim is to find errors and eliminate them as quickly as possible, without expecting perfection and zero errors.

Having a comprehensive test suite is essential to continuous integration.

You should also consider refactoring as a cornerstone of effective software development.

Building Microservices

Is an excellent book by Sam Newman. As usual I'm going to quote from a few pages...

Because microservices are primarily modeled around business domains, they avoid the problems of traditional tiered architectures.

Microservices should cleanly align to bounded contexts.

Another reason to prefer the nested approach could be to chunk up your architecture to simplify testing.

With an event-based collaboration, we invert things. Instead of a client initiating requests asking for things to be done, it instead says this thing happened and expects other parties to know what to do. We never tell anyone else what to do.

We always want to maintain the ability to release microservices independenty of each other.

A red build means the last change possibly did not intergrate. You need to stop all further check-ins that aren't involved in fixing the build to get it passing again.

The approach I prefer is to have a single CI build per microservice, to allow us to quickly make and validate a change prior to deployment into production.

No changes are ever made to a running server.

Rather than using a package manager like debs or RPMs, all software is installed as independent Docker apps, each running in its own container.

Flaky tests are the enemy. When they fail, they don't tell us much... A test suite with flaky tests can become a victim of what Diane Vaughan calls the normalization of deviance - the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.

All too often, the approach of accepting multiple services being deployed together drifts into a situation where services become coupled.

Most organizations that I see spending time creating functional test suites often expend little or no effort at all on better monitoring or recovering from failure.

Developer on Fire

Recently I was honoured to be interviewed by Dave Rael, the man behind Developer on Fire.
Click the image below to listen :-)

the design and evolution of cyber-dojo

I've talked about the design and evolution of cyber-dojo at two conferences this year. First at NorDevCon in Norwich and then also at Agile on the Beach in Falmouth. Here's the slides.

Design and Evolution of cyber-dojo from Jon Jagger

cyber-dojo new release

The new release of cyber-dojo is out :-)

• the setup start-points are now decoupled from the server and can be customized.
• the manifest.json format has changed accordingly.
• the instructions for running your own cyber-dojo server are new.

cyber-dojo traffic-lights

My friend Byran who works at the awesome Bluefruit Software in Redruth has hooked up his cyber-dojo web server to an actual traffic-light! Fantastic. Check out the video below :-)



Byran writes

It started out as a joke between myself and Josh (one of the testers at Bluefruit). I had the traffic lights in my office as I was preparing a stand to promote the outreach events (Summer Huddle, Mission to Mars, etc...) Software Cornwall runs. The conversation went on to alternative uses for the traffic lights, I was planning to see if people would pay attention to the traffic lights if I put them in a corridor at the event; we then came up with the idea that we could use them to indicate TDD test status.

Although it started out as a joke I am going to use it at the Summer Huddle, the lights change every time anyone runs a test so it should give an idea of how the entire group are doing without highlighting an individual pair.

The software setup is very simple, there is a Python web server (using the Flask library) running on a Raspberry Pi that controls the traffic lights using GPIO Zero. When the appendTestTrafficLight() function (in run_tests.js.erb) appends the traffic light image to the webpage I made it send an http 'get' request to the Raspberry Pi web server to set the physical traffic lights at the same time. At the moment the IP address of the Raspberry Pi is hard coded in the 'run_tests.js.erb' file so I have to rebuild the web image if anything changes but it was only meant to be a joke/proof of concept. The code is on a branch called traffic_lights on my fork of the cyber-dojo web repository.

The hardware is also relatively simple, there is a converter board on the Pi; this only converts the IO pin output connector of the Raspberry Pi to the cable that attaches to the traffic lights.

The other end of the cable from the converter board attaches to the board in the top left of the inside the traffic lights; this has some optoisolators that drive the relays in the top right which in turn switch on and off the transformers (the red thing in the bottom left) that drive the lights.

I have to give credit to Steve Amor for building the hardware for the traffic lights. They are usually used during events we run to teach coding to children (and sometimes adults). The converter board has LEDs, switches and buzzers on it to show that there isn't a difference between writing software to toggle LEDs vs driving actual real world systems, it's just what's attached to the pin. Having something where they can run the same code to drive LEDs and drive real traffic lights helps to emphasise this point.

cyber-dojo update

Between my recent fishing trips I have been working hard on a new cyber-dojo architecture.

• pluggable default start-points so you can now use your own language/tests and exercises lists on the setup page
• a new setup page for your own pluggable custom start-points
• once I've got the output parse functions inside the start-point volumes I'll be switching the public cyber-dojo server to the new image and updating the running-your-own-server instructions
• I've switched all development to a new github repo which has instructions if you want to try it now.

River Tone chub/roach

Last week I broke my Daiwa Spectron spliced-tip rod which has served me for 25 years+. After some searching around I replaced it with a Drennan Acolyte Ultra 14ft. I christened it yesterday with my heaviest ever bag on the Tone; 80lb+ of chub and roach with a few skimmer bream, one roach-bream hybrid, and several small perch. All caught on waggler, red/yellow/white maggot, on 1.7lb hooklength, and a size 20 Kamasan B560. Wooohooo.

Verdal salmon

Once a year a spent a glorious week fishing for salmon on the River Verdal in Norway. This year, despite the low water, I caught this cracker. A huge thanks to John Oldren who owns the beat. You will not find a more hospitable host. Also to Gary Scott for being super helpful as always.

Tulchan salmon

One of my favourite salmon beats is Tulchan on the river Spey.
On my most recent trip I caught two, including this beauty from Hunter's Pool.

the story of cyber-dojo (so far)

The video of my ACCU 2016 talk, the story of cyber-dojo (so far), is now up on the ACCU youtube channel. Enjoy!

pair programming keynote

The video of my NorDevCon endnote on Pair Programming is now up on InfoQ . Here's the slides too.

NorDevCon 2016 pair programming from Jon Jagger

Wooohooo - River Wye Springer

Woooohoooo. I've caught my first River Wye Springer. Thirty-four inches long, nineteen inches fat, eighteen pounds exactly. I caught it on the Monument pool of the Tunnel beat on a green and yellow devon minnow (a technique I'd never used before). Huge thanks to my friend Steve Roberts for skilfully guiding me.

Before setting off for this fishing trip I'd been stumped on a cyber-dojo fault which eluded me for the best part of a day. After the trip, I cracked it within five minutes. So you see, fishing fixes faults!

nordevcon cyber-dojo presentation

It was a pleasure to speak at the recent norfolk developers conference. My talk was "cyber-dojo: executing your code for fun and not for profit". I spoke about cyber-dojo, demo'd its features, discussed its history, design, difficulties and underlying technology. Videos of the talk are now on the infoq website. The slide-sync is not right at the start of part 2 but it soon gets corrected.

• part 1
• part 2

code and test oppose each other

I was reading one of Michael Feathers excellent blog posts on Symbiosis. His writing really resonates with me. As I get older I feel I'm starting to get a handle on thinking about things more dynamically and less statically. Looking back, if I had to pick the one thing that helped me the most on this road I would say its Le Chatelier's Principle, which I paraphrase as

Stable systems tend to oppose their own proper function.

As I recall, Le Chatelier was a chemist and his principle is worded in the context of chemical reactions. The same fundamental "system of opposition" is also described in Walter B. Cannon's classic book The Wisdom of The Body which I first learned about in Jerry Weinberg's book General Principles of Systems Design (p 177). I'd like to try to explain what "oppose their own proper function" means using an example from the body. It's called the Glucose Cycle.

---

If you eat a donut your blood sugar (glucose) increases...

If this increase continues unchecked you get hyper-glycemia and you die very quickly. Fortunately this does not happen because the mammalian body is the result of millions of years of destructive testing! Cells in the pancreas detect the glucose increasing and start producing insulin. The liver and muscles detect the insulin increasing and start converting glucose into a stored form (called glycogen). This naturally reduces the amount of glucose and you don't get hyperglycemia.

However...

If this decrease continues unchecked you get hypo-glycemia and you die very quickly. Fortunately this does not happen either because other cells in the pancreas detect the glucose decreasing and start producing glucagon. The liver and muscles detect the glucagon increasing and start converting the glycogen back to glucose.

These two effects work in opposition to each other regulating the blood stream glucose.
All this tremendous activity to keep something else constant.
I find it deeply beautiful and deeply paradoxical.

---

Bradford Keeney writes about this same paradox in his classic book The Aesthetics of Change. An example he uses is evolution. He writes about the battle between a predator and its prey goes beyond the 'mere' battle for food and territory. He describes a larger cybernetic picture, how the ongoing battle is itself a means or process of generating, maintaining, and stablizing an ecosystem. That evolution is always co-evolution as John Gall said.

---

This duality suggests that if you want to understand how codebases successfully change you should also understand how codebases successfully stay the same. To quote The Aesthetics of Change again: "Change cannot be found without a roof of stability over its head. Similarly stability will always be rooted to underlying processes of change".

In this light I see test driven development, as much more than simply specifying required behaviour (as important as that is). I see coding and testing working in opposition to each other naturally regulating each other. The ongoing 'battle' between coding and testing, between change and constancy, is itself a primary means of generating, maintaining, and stabilizing the development process.

My instinct tells me good software development is full of processes regulating each other via feedback like this. I'd like to make a collection of them. What's your favourite example? Please tell me. Thanks.