fun with feedback frequency

I'm currently reading Brain of the Firm, by Stafford Beer. On page 36 he writes:

The outcome is startling ... ... the total system is dominated, not by the forward network, but by the feedback network ... The output signals will be of greater 'purity' than we had any right to expect.

The word feedback caught my eye.
Feedback is a key principle of complex adaptive systems.
Feedback is also one of the 4 values of eXtreme Programming.
Feedback also gets several mentions in the official Scrum Guide.
So I've been playing around with Feller's walk (named after William Feller) some more...

Feller's Walk

Each Feller's walk is 1000 steps long, and at each step I flip a fair coin.
As I walk I keep a running total (starting at zero):

• adding one each time I flip a head
• subtracting one each time I flip a tail

What will the total look like as the walk progresses?
Here's the plot of the step (x-axis) against the total (y-axis) during one simulated walk.

It starts at a total of zero on the far left and wanders along, edging upwards when a head is thrown, edging downwards when a tail is thrown, and ending at about -26 indicating that overall there were 487 heads and 513 tails (487-513=-26, 487+513==1000).

Apparently, most people's intuition is that the total will hover around zero.
But it doesn't.
You cannot rely on randomness to correct the problems that randomness creates.
Without constraints variance will accumulate.
If the total happens to be +24 the coin isn't going to throw more tails for a while to try to balance things up.
Without feedback the coin has no memory.

To help see this I simulated 5000 walks and plotted the frequency of the total (y-axis) against the total (x-axis) at various points along the walks:

• after 10 steps in gold
• after 100 steps in blue
• after 1000 steps in red
• I chop the x-axis to +- 2 standard deviations

step	freq[0]	max abs(total)	std.dev
10	1204	-10	3.19
100	405	36	10.15
1000	121	140	31.71

Over all the walks, at the 10th step:

• the total was zero 1204 times (5 heads, 5 tails, 5-5=0, 5+5=10)
• the total farthest from zero was -10 (0 heads, 10 tails, 0-10=-10, 0+10=10)
• the standard deviation of the total was 3.19

Over all the walks, at the 100th step:

• the total was zero 405 times (50 heads, 50 tails, 50-50=0, 50+50=100)
• the total farthest from zero was 36 (68 heads, 32 tails, 68-32=36, 68+32=100)
• the standard deviation of the total was 10.15

Over all the walks, at the 1000th step:

• the total was zero 121 times (500 heads, 500 tails, 500-500=0, 500+500=1000)
• the total farthest from zero was 140 (570 heads, 430 tails, 570-430=140, 570+430=1000)
• the standard deviation of the total was 31.71

The further along the walk I progress, the less likely my total is to be zero.
Without constraints variance will accumulate.

Once again, with feedback

Every Nth step, instead of flipping the coin, I look at the total and:

• if the total is negative (more tails than heads) I pretend I've flipped a head, and add one to the total.
• if the total is positive (more heads than tails) I pretend I've flipped a tail, and subtract one from the total.

This feedback increases the likelihood of the total staying nearer to zero.
The smaller the value of N, the greater the effect.

Estimating feedback effectiveness

Using the simulation I can measure how good (or bad) people's estimates are of how effective feedback is. For example, I can:

• run the simulation with some feedback (N==10 say) and display the results
• gather estimates with no feedback (N==never) of, eg
  • freq[0], or
  • max abs(total), or
  • std.dev, etc
• run the simulation with no feedback and see how the actual results compare.

Or

• run the simulation with no feedback and display the results
• gather estimates of the least feedback needed to, eg
  • triple freq[0], or
  • quadruple max abs(total), or
  • half std.dev, etc
• run the simulation with various amounts of feedback and see how the actual results compare.

I've put the simulation source up on https://github.com/JonJagger/Fellers-1000-coin-tosses.
The simulation runs directly from this URL https://rawgithub.com/JonJagger/Fellers-1000-coin-tosses/master/fellers.html so you can try out various values of N if you want to before reading on...

A little feedback goes a long way

Here's 5000 simulations of a 1000 step walk with N==25.
That means, that every 25th step instead of flipping the coin, the simulation nudges the total towards zero as described above.

step	freq[0]	max abs(total)	std.dev
10	1379	-10	2.97
100	553	32	8.04
1000	387	84	15.18

The effectiveness of one feedback nudge every 25 steps amazes me.

• freq[0] tripled from 121 to 387
• max abs(total) shrank from 140 to 84
• std.dev halved from 31.71 to 15.18

Startling! Just as Stafford Beer said.
Let's hear it for feedback!

observing

Back to quotes table-of-contents

From The Psychology of Computer Programming

If egoless programming is used, everyone in the group will have the opportunity to examine the work of everyone else at some time, thereby tending to prevent the establishment of strong hierarchy.

From Toyota Production System

Sometimes I would spend all day watching the grandmother next door weaving. I came to understand the way the weaving machine worked. [Toyoda Sakichi]

Stand on the production floor all day and watch - you will eventually discover what has to be done. I cannot emphasize this point too much.

From The Way of the Leader

Listen carefully. Observe closely.

From Zen and the Art of Motorcycle Maintenance

By far the greatest part of his [the mechanic's] work is careful observation and precise thinking.

From Non Violent Communcation

Observing without evaluating is the highest form of human intelligence.

From Wit and Wisdom from Poor Richard's Almanac

Observe all men; thyself most.

From The Mind of War

He observed very carefully.

From Quality Software Management. Vol 2. First-Order Measurement

In the end, it's not the observation that counts, it's the response to the observation. That's why Zen masters teach patience in response.

What power corrupts most thoroughly is the ability to make meaning of observations.

No other observational skill may be more important to software engineering than precision listening.

The switch from cost observation to value observation is the strongest indication that an organization has made the transition from Pattern 2 [Routine] to Pattern 3 [Steering].

A fact becomes a feeling as soon as you observe it.

From An Introduction to General Systems Thinking

We drive more slowly at night to give us more time to observe potentially dangerous situations.

From Zen Mind, Beginner's Mind

The true purpose is to see things as they are, to observe things as they are, and to let everything go as it goes.

From The Book of Five Rings

You should observe reflectively, with overall awareness of the large picture a well as precise attention to small details.

From Pragmatic Thinking and Learning

Much of perception is based on prediction.

From General Principles of Systems Design

Complexity is a relationship between system and observer.

what should all tests passing look like?

I was fixing a fault in cyber-dojo the other day and I happened to be looking at the starting test code for C++. I'd written it using an array of function pointers. At the end of a practice the array might look like this (in main I simply iterate through the array and call each of the functions):

typedef void test();

static test * tests[ ] =
{
    a_recently_used_list_is_initially_empty,
    the_most_recently_added_item_is_always_first,
    items_can_be_looked_up_by_index_which_counts_from_zero,
    items_in_the_list_are_unique,
};

I imagined it written like this instead:

int main()
{
    a_recently_used_list_is_initially_empty();
    the_most_recently_added_item_is_always_first();
    items_can_be_looked_up_by_index_which_counts_from_zero();
    items_in_the_list_are_unique();
}

I wondered why I was putting the function pointers into an array at all. The most obvious thing I lose by not using an array is that I can no longer print out the number of tests that have passed. I thought about that a bit. I started to wonder what benefits I actually get by being told how many tests had run. It's a 100% quantity and 0% quality measurement.

The most obvious benefit I can think of is that, after writing a new test, I can verify it has run by seeing the number of tests increase by one. The problem is I don't think that's a benefit at all. I think that interaction could easily encourage me to not start with a failing test.

If all tests passing produces any output I will look at the output and start to rely on it. That doesn't feel right for a supposedly automated test.

If each passing test introduces any new output, even if it's just a single dot, then I'm in danger of falling into a composition trap. Those dots don't make a difference individually, but sooner or later they will collectively.

If the code I'm testing doesn't produce any output when it runs (which is very likely for unit-tests) then is it right for my tests to introduce any output?

If I want to time how long the tests take to run shouldn't that be part of the script that runs the tests?

I think there is something to be said for all tests passing producing no output at all. None. Red means a test failed and the output should be as helpful as possible in identifiying the failure. No output means green. Green means no output.

Thoughts?

visibility

Back to quotes table-of-contents

From The Secrets of Consulting

Most of what a plant is doing is out of sight.

From The Toyota Way

Extra inventory hides problems... Ohno considered the fundamental waste to be overproduction, since it causes most of the other wastes… big buffers (inventory between processes) lead to other suboptimal behaviour, like reducing your motivation to continuously improve your operation.

From Quality Software Management. Vol 4. Anticipating Change

Without action things will only get less visible over time.

A measurement system helps makes the software product visible.

Software is invisible only when we have not developed the correct engineering measurements. A hundred years ago electricity was considered invisible. We only knew of its existence when it shocked us.

From Leverage Points

They [feedback loops] may not be very visible. But their presence is critical to the long-term welfare of the system. One of the big mistakes we make is to strip away these "emergency" response mechanisms because they aren't used often and they appear to be costly.

From Situated learning - Legitimate peripheral participation

A window's invisibility is what makes it a window, that is, an object through which the outside world becomes visible.

From Implementing Lean Software Development

The behaviours that ranking systems encourage are competition, hiding information so as to look good, and hiding problems so as not to look bad.

From Toyota production system

The vicious cycle of waste generating waste hides everywhere in production.

From The Silent Language

Culture hides much more than it reveals, and strangely enough what it hides, it hides most effectively from its own participants.

From Brain Rules

Vision is by far our most dominant sense, taking up half of our brain's resources.

From Mending a bike puncture

Air, like software, is invisible. To get the hole to reveal itself I switched to a different medium, from air to water.

From What Did You Say?

We structure our world so we will not receive feedback that threatens our view.

From General Principles of Systems Design

Regulation is invisible - when it works.

From the DevOps Handbook

The principle of small batch sizes also applies to code reviews.