At the excellent Agile on the Beach conference in Cornwall I did a presentation outlining some of the history, design and implementation of cyber-dojo.
The video has just gone live on youtube.
Continuous Delivery
Software delivers no value until it is in the hands of its users.
The pattern that is central to this book is the deployment pipeline.
It should not be possible to make manual changes to testing, staging, and production environments.
If releases are frequent, the delta between releases will be small. This significantly reduces the risk associated with releasing and makes it much easier to to roll back.
Branching should, in most circumstances, be avoided.
Dashboards should be ubiquitous, and certainly at least one should be present in each team room.
One of the key principles of the deployment pipeline is that it is a pull system.
A corollary of having every version of every file in version control is that it allows you to be aggressive about deleting things that you don't think you need... The ability to weed out old ideas and implementations frees the team to try new things and to improve the code.
It should always be cheaper to create a new environment than to repair an old one.
The goal of continuous integration is that the software is in a working state all the time... Continuous is a practice not a tool... Continuously is more often than you think.
The most important practice for continuous integration to work properly is frequent check-ins to trunk or mainline.
Ideally, the compile and test process that you run prior to check-in and on your CI server should take no more than a few minutes. We think that ten minutes is about the limit, five minutes is better, and about 90 seconds is ideal.
Enabling developers to run smoke tests against a working system on a developer machine prior to each check-in can make a huge difference to the quality of your application.
Build breakages are a normal and expected part of the process. Our aim is to find errors and eliminate them as quickly as possible, without expecting perfection and zero errors.
Having a comprehensive test suite is essential to continuous integration.
You should also consider refactoring as a cornerstone of effective software development.
Building Microservices
Because microservices are primarily modeled around business domains, they avoid the problems of traditional tiered architectures.
Microservices should cleanly align to bounded contexts.
Another reason to prefer the nested approach could be to chunk up your architecture to simplify testing.
With an event-based collaboration, we invert things. Instead of a client initiating requests asking for things to be done, it instead says this thing happened and expects other parties to know what to do. We never tell anyone else what to do.
We always want to maintain the ability to release microservices independenty of each other.
A red build means the last change possibly did not intergrate. You need to stop all further check-ins that aren't involved in fixing the build to get it passing again.
The approach I prefer is to have a single CI build per microservice, to allow us to quickly make and validate a change prior to deployment into production.
No changes are ever made to a running server.
Rather than using a package manager like debs or RPMs, all software is installed as independent Docker apps, each running in its own container.
Flaky tests are the enemy. When they fail, they don't tell us much... A test suite with flaky tests can become a victim of what Diane Vaughan calls the normalization of deviance - the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.
All too often, the approach of accepting multiple services being deployed together drifts into a situation where services become coupled.
Most organizations that I see spending time creating functional test suites often expend little or no effort at all on better monitoring or recovering from failure.
Developer on Fire
Recently I was honoured to be interviewed by Dave Rael, the man behind
Developer on Fire.
Click the image below to listen :-)
Click the image below to listen :-)
the design and evolution of cyber-dojo
I've talked about the design and evolution of cyber-dojo at two conferences this year.
First at NorDevCon in Norwich and then also at Agile on the Beach in Falmouth. Here's the slides.
cyber-dojo new release
The new release of cyber-dojo is out :-)
- the setup start-points are now decoupled from the server and can be customized.
- the manifest.json format has changed accordingly.
- the instructions for running your own cyber-dojo server are new.
cyber-dojo traffic-lights
My friend Byran who works at the awesome Bluefruit Software in Redruth has hooked up his cyber-dojo web server to an actual traffic-light! Fantastic. Check out the video below :-)
Byran writes
Byran writes
It started out as a joke between myself and Josh (one of the testers at Bluefruit). I had the traffic lights in my office as I was preparing a stand to promote the outreach events (Summer Huddle, Mission to Mars, etc...) Software Cornwall runs. The conversation went on to alternative uses for the traffic lights, I was planning to see if people would pay attention to the traffic lights if I put them in a corridor at the event; we then came up with the idea that we could use them to indicate TDD test status.
Although it started out as a joke I am going to use it at the Summer Huddle, the lights change every time anyone runs a test so it should give an idea of how the entire group are doing without highlighting an individual pair.
The software setup is very simple, there is a Python web server (using the Flask library) running on a Raspberry Pi that controls the traffic lights using GPIO Zero. When the appendTestTrafficLight() function (in run_tests.js.erb) appends the traffic light image to the webpage I made it send an http 'get' request to the Raspberry Pi web server to set the physical traffic lights at the same time. At the moment the IP address of the Raspberry Pi is hard coded in the 'run_tests.js.erb' file so I have to rebuild the web image if anything changes but it was only meant to be a joke/proof of concept. The code is on a branch called traffic_lights on my fork of the cyber-dojo web repository.
The hardware is also relatively simple, there is a converter board on the Pi; this only converts the IO pin output connector of the Raspberry Pi to the cable that attaches to the traffic lights.
The other end of the cable from the converter board attaches to the board in the top left of the inside the traffic lights; this has some optoisolators that drive the relays in the top right which in turn switch on and off the transformers (the red thing in the bottom left) that drive the lights.
I have to give credit to Steve Amor for building the hardware for the traffic lights. They are usually used during events we run to teach coding to children (and sometimes adults). The converter board has LEDs, switches and buzzers on it to show that there isn't a difference between writing software to toggle LEDs vs driving actual real world systems, it's just what's attached to the pin. Having something where they can run the same code to drive LEDs and drive real traffic lights helps to emphasise this point.
cyber-dojo update
Between my recent fishing trips I have been working hard on a new cyber-dojo architecture.
- pluggable default start-points so you can now use your own language/tests and exercises lists on the setup page
- a new setup page for your own pluggable custom start-points
- once I've got the output parse functions inside the start-point volumes I'll be switching the public cyber-dojo server to the new image and updating the running-your-own-server instructions
- I've switched all development to a new github repo which has instructions if you want to try it now.
River Tone chub/roach
Last week I broke my Daiwa Spectron spliced-tip rod which has served me for 25 years+.
After some searching around I replaced it with a Drennan Acolyte Ultra 14ft.
I christened it yesterday with my heaviest ever bag on the Tone; 80lb+ of chub and roach with a few skimmer bream, one roach-bream hybrid, and several small perch. All caught on waggler, red/yellow/white maggot, on 1.7lb hooklength, and a size 20 Kamasan B560. Wooohooo.
Verdal salmon
Once a year a spent a glorious week fishing for salmon on the River Verdal in Norway.
This year, despite the low water, I caught this cracker.
A huge thanks to John Oldren who owns the beat. You will not find a more hospitable host.
Also to Gary Scott for being super helpful as always.
Tulchan salmon
One of my favourite salmon beats is Tulchan on the river Spey.
On my most recent trip I caught two, including this beauty from Hunter's Pool.
On my most recent trip I caught two, including this beauty from Hunter's Pool.
the story of cyber-dojo (so far)
The video of my ACCU 2016 talk, the story of
cyber-dojo (so far), is now up on the
ACCU youtube channel. Enjoy!
pair programming keynote
Wooohooo - River Wye Springer
Woooohoooo. I've caught my first River Wye Springer.
Thirty-four inches long, nineteen inches fat, eighteen pounds exactly.
I caught it on the Monument pool of the Tunnel beat on a green and yellow devon minnow (a technique I'd never used before).
Huge thanks to my friend Steve Roberts for skilfully guiding me.
Before setting off for this fishing trip I'd been stumped on a cyber-dojo fault which eluded me for the best part of a day. After the trip, I cracked it within five minutes. So you see, fishing fixes faults!
Before setting off for this fishing trip I'd been stumped on a cyber-dojo fault which eluded me for the best part of a day. After the trip, I cracked it within five minutes. So you see, fishing fixes faults!
nordevcon cyber-dojo presentation
It was a pleasure to speak at the recent norfolk developers conference.
My talk was "cyber-dojo: executing your code for fun and not for profit".
I spoke about cyber-dojo, demo'd its features, discussed its history, design, difficulties and underlying technology.
Videos of the talk are now on the infoq website. The slide-sync is not right at the start of part 2 but it soon gets corrected.
code and test oppose each other
I was reading one of Michael Feathers excellent blog posts on Symbiosis. His writing really resonates with me. As I get older I feel I'm starting to get a handle on thinking about things more dynamically and less statically. Looking back, if I had to pick the one thing that helped me the most on this road I would say its Le Chatelier's Principle, which I paraphrase as
Stable systems tend to oppose their own proper function.As I recall, Le Chatelier was a chemist and his principle is worded in the context of chemical reactions. The same fundamental "system of opposition" is also described in Walter B. Cannon's classic book The Wisdom of The Body which I first learned about in Jerry Weinberg's book General Principles of Systems Design (p 177). I'd like to try to explain what "oppose their own proper function" means using an example from the body. It's called the Glucose Cycle.
If you eat a donut your blood sugar (glucose) increases...
If this increase continues unchecked you get hyper-glycemia and you die very quickly.
Fortunately this does not happen because the mammalian body is the result of millions of years of destructive testing!
Cells in the pancreas detect the glucose increasing and start producing insulin.
The liver and muscles detect the insulin increasing and start converting glucose into a stored form (called glycogen).
This naturally reduces the amount of glucose and you don't get hyperglycemia.
However...
If this decrease continues unchecked you get hypo-glycemia and you die very quickly.
Fortunately this does not happen either because other cells in the pancreas detect the glucose decreasing and start producing glucagon.
The liver and muscles detect the glucagon increasing and start converting the glycogen back to glucose.
These two effects work in opposition to each other regulating the blood stream glucose.
All this tremendous activity to keep something else constant.
I find it deeply beautiful and deeply paradoxical.
Bradford Keeney writes about this same paradox in his classic book The Aesthetics of Change.
An example he uses is evolution. He writes about the battle between a predator and its prey goes beyond
the 'mere' battle for food and territory. He describes a larger cybernetic picture, how the ongoing battle
is itself a means or process of generating, maintaining, and stablizing an ecosystem.
That evolution is always co-evolution as John Gall said.
This duality suggests that if you want to understand how codebases successfully change you should also understand
how codebases successfully stay the same. To quote The Aesthetics of Change again: "Change cannot be found without a roof of stability over its head. Similarly stability will always be rooted to underlying processes of change".
In this light I see test driven development, as much more than simply specifying required behaviour (as important as that is).
I see coding and testing working in opposition to each other naturally regulating each other.
The ongoing 'battle' between coding and testing, between change and constancy, is itself a primary means of generating, maintaining, and stabilizing the development process.
simplicity
Back to quotes table-of-contents
From The Aesthetics of Change
From Thinking in Systems - A Primer
From The Systems Bible
From Patterns of Software
From Safer C
From The Tao of Business
From The ACM Turing Award Lectures
From The Way of the Leader
From Extreme Programming Explained
From Tao Te Ching
From Simplicity
From Simple and Usable
All simple and complex regulation as well as learning involve feedback. Contexts of learning and change are therefore principally concerned with altering or establishing feedback.
From Thinking in Systems - A Primer
Complex systems can evolve from simple systems only if there are stable intermediate forms.
From The Systems Bible
A complex system that works is invariably found to have evolved from a simple system that worked.
From Patterns of Software
In the modern era, we have come to favor simplicity over complexity, perfection over imperfection, symmetry over asymmetry, planning over piecemeal growth, and design awards over habitability. Yet if we look at the art we love and the music, the buildings, towns, and houses, the ones we like have the quality without a name, not the deathlike morphology of clean design.
From Safer C
A central and hard-earned engineering principle in older engineering areas such as mechanical engineering and civil engineering is that simplicity rules.
From The Tao of Business
Keeping things simple is an art. And, as with any art, simplicity needs to be cultivated.
From The ACM Turing Award Lectures
I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. [C.A.R. Hoare]
From The Way of the Leader
The art of leadership is an art based on simplicity, and all success is rooted in performance.
From Extreme Programming Explained
Simplicity supports courage because you can afford to be much more courageous with a simple system.
From Tao Te Ching
Simplicity in conduct, in beliefs, and in environment brings an individual very close to the truth of reality.
From Simplicity
An expert is someone who has succeeded in making decisions and judgements simpler through knowing what to pay attention to and what to ignore. Simplicity means focused effort. Simplicity is a unification around a purpose.
From Simple and Usable
Simple organization is about what feels good as you're using the software, not what looks logical in a plan.
a man called Ove
is the title of an excellent book by Fredrik Backman.
As usual I'm going to quote from a few pages:
He felt one should not go through life as if everything was exchangeable. As if loyalty was worthless. Nowadays people changed their stuff so often that any expertise in how to make things last was becoming superfluous. Quality: no one cared about that any more. Not Rune or the other neighbours and not those managers in the place where Ove worked. Now everything had to be computerised, as if one couldn't build a house until some consultant in a too-small shirt figured out how to open a laptop.
'They've bumped up the electricity prices again,' he informs her as he gets to his feet. He looks at her for a long time. Finally he puts his hand carefully on the big boulder and caresses it tenderly from side to side, as if touching her cheek. 'I miss you,' he whispers. It's been six months since she died. But Ove still inspects the whole house twice a day to feel the radiators and check that she hasn't sneakily turned up the heating.
'Now you listen to me,' says Ove calmly while he carefully closes the door. 'You've given birth to two children and quite soon you'll be squeezing out a third. You've come here from a land far away and most likely you fled war or persecution and all sorts of other nonsense. You've learned a new language and got yourself an education and you're holding together a family of obvious incompetents. And I'll be damned if I've seen you afraid of a single bloody thing in this world before now.' ...
'I'm not asking for brain surgery. I'm asking you to drive a car. It's got an accelerator, a brake, a clutch. Some of the greatest twits in world history have sorted out how it works. And you will as well.'
And then he utters seven words, which Parvaneh will always remember as the loveliest compliment he'll ever give her.
'Because you are not a complete twit.'
Men like Ove and Rune were from a generation in which one was what one did, not what one talked about.
Ove has probably known all along what he has to do, but all people are time optimists. We always think there's enough time to do things with other people. Time to say things to them. And then something happens and then we stand there holding on to words like 'if'.
'But serious, man. You do this every morning?' Jimmy asks cheerfully.
'Yes, to check if there have been any burglaries.'
'For real? Are there a lot of burglaries round here?'
'There are never a lot of burglaries before the first burglary,' Ove mutters and heads off towards the guest parking.
'There is no hope for these boys and girls,' the headmaster soberly explained in the interview. 'This is not education, this is storage.' Maybe Sonja understood how it felt to be described as such. The vacant position only attracted one applicant, and she got the boys and girls to read Shakespeare.
fishing and refactoring: A story with two endings
The Happy Ending
I'm salmon fishing on the River Spey, on the beautiful Tulchan beat,
concentrating on spey casting.
My phone is back at the hotel.
My laptop is back at the hotel.
I'm not thinking about coding.
I'm fishing and I'm only fishing.
Suddenly, from "nowhere" my subconcious pops an idea up into my consciousness.
It says, hey, you know that code you were working on 2 weeks ago...?
Where you had that nagging feeling there was a better, simpler solution?
I know you know what I'm talking about!
We'll I've been working on that for you.
And I've come up with this. Here's the idea. Tada. What do you think?
For a moment, I think about the idea. It seems a good one.
Of course, I can't try it out now because I'm
waist deep in the middle of the River Spey. I promise myself I'll look
into it when I get back to the hotel.
I carry on trying to catch a salmon.
Later, back at the hotel I try out the new idea. I re-run all the unit
tests and they pass. I make the change.
A few weeks later I'm fishing again. This time on the River Tay at Dunkeld. My subconscious pops up a new idea. I make a note of it. In the evening, back at the hotel I try the change. All the tests pass. The change is live.
A few weeks later I'm fishing again. On the River Verdal in Norway. My subconscious pops up with another new idea...
The Sad Ending
I'm salmon fishing on the River Spey, on the beautiful Tulchan beat,
concentrating on spey casting.
My phone is back at the hotel.
My laptop is back at the hotel.
I'm not thinking about coding.
I'm fishing and I'm only fishing.
Suddenly, from "nowhere" my subconcious pops an idea up into my consciousness.
It says, hey, you know that code you were working on 2 weeks ago...?
Where you had that nagging feeling there was a better, simpler solution?
I know you know what I'm talking about!
We'll I've been working on that for you.
And I've come up with this. Here's the idea. Tada. What do you think?
For a moment, I think about the idea. It seems a good one.
Of course, I can't try it out now because I'm
waist deep in the middle of the River Spey. I promise myself I'll look
into it when I get back to the hotel.
I carry on trying to catch a salmon.
Later, back at the hotel I think about the new idea.
I'm wary of making the change.
There are no unit-tests I can quickly run.
I'm remembering a few months back - that time when I tried an idea and broke
stuff and was castigated.
I'm fearful of making the change. I don't make it.
A few weeks later I'm fishing again. This time on the River Tay at Dunkeld. My subconscious gives me a new idea. I make a note of it. In the evening, back at the hotel I think about making the change. There are still no unit-tests. And I'm still remembering that time a few months back - when I tried an idea and broke stuff. Boy was I ridiculed. And if anything the culture is a bit worse since then. I can feel the fear. Once again I decide not to make the change.
A few weeks later I'm fishing again. On the River Verdal in Norway. This time, my subconscious doesn't pop up a new idea.
Subscribe to:
Posts (Atom)