When I go to Wellington in January for my daughter's wedding, I must try Dave's favourite watering hole, its a pity he wont still be there so I can buy him a Mac's Sassy Red: for being prepared to dig in.
Now remember we were talking about IT education in schools and the question of how to replicate an experiment. The normal practice is for a series of trials to be run, and where success is achieved to attempt to replicate success in other schools. My argument is that this fails in the main for two reasons: (i) the trial schools get more attention and often pre-select themselves and (ii) the context (social, historical etc) of each school is different in subtle (or less than subtle ways) and this disturbs the assumed contextual homogeneity of the experiment and roll out process. This isn't just schools by the way, it also applies in marketing and other areas.
100% agreement. Over the last decade I have looked at hundreds of attempts to do this kind of thing in multiple areas, and there's some great stuff on Xerox's failure to transfer learning and good practise between two buildings across a street in The Social Life of Information
My argument was that you could never replicate an outcome and Earl agrees with that;
The PROBLEM is that outcomes are exactly what we want to replicate. Witness Dave's pointer to the Balle-Argentee method. There's a level at which I don't care how you get to universal literacy as long as we GET there. In a chemistry class we CAN replicate both process and outcome (most of the time - hence the fudge factor) but not in social sciences and sure as hell not in learning.
My wife is a speech therapist/ researcher who caused some deep consternation at a conference a few years ago by making the case that a speech therapist has no idea how they achieve what they achieve, partly because every single case is unique while therapists have only a limited range of tactics they can use to influence a child (or brain damaged person) towards more competent communication.
They can't actually explain how it works, but it seems to work anyway and its not magic. I agree, getting a child to use a broken or never constructed tool (language) to repair that tool is not magic, its a bloody miracle. The point being that the outcome is agreed (though not always by the child who may not be able to conceive that outcome - hence an inability to create it themselves) but the path to that result is different every single time. I think its on point, but I'm not sure.
the disagreement is over the replication of starting conditions. Now I want to maintain my position, while pointing out that the phrase I used is an abbreviate of a complex set of arguments so misunderstanding is possible. Why do I do so?
For starters, it is very unlikely that each new school is going to be fully aware of what happened elsewhere, and there is no particular need for this to be the case. Even if they are aware they would not be able to replicate the path of the experiment.
Hmm. But their reason for wanting to replicate the experiement is because they know that it succeeded somewhere else, inverse case being absurd.
Here Earl's reference to Lorenz's discovery that minor changes have major impact on results contradicts his second argument as to ritualisation.
My turn not to be clear. the point about ritual is that it is comforting to the participant and if you have a "successful" ritual replication of outcome is relatively simple and when it fails you can avoid blame because "it always worked before". I'm not advocating for ritual at all, only for the natural tendency to migrate in that direction. More Balle-Argentee.
It will take no conscious effort not to follow the same path, it will happen anyway. The intention is not to achieve the same outcome, [EM See above] but allow a contextual solution to emerge in each situation. We are not condemned just to be active and constructive, although that is no bad thing; we can learn.
True, apparently, but I'm not at all sure what the hell it is that we learn. A speech therapist learns how to use poorly understood tools to achieve more or less consistent outcomes in myriad distinct cases. (I have sneakign suspicion that nobody is taking into account the cognitive effect of paying supportive, positive, non-judgemental attention to a communication-disabled child and showing that their achievements matter to the adults)
BTW, have a look at NZ Ministry of Education material on their professional development funding programme – at a meeting in Auckland last week I heard Douglas Harre from the Ministry talk about creating opportunities for teachers to learn for themselves the process of learning about ICT in education. Its actually inspirational, both from an organisational perspective (a ministry INSISTING that teachers take their development in their own hands) and in the outcomes (hugely various but all like a Lorenz attractor, forming a pattern around the objective if integrating ICT as an effective tool in enabling kids to learn - something - anything)
They appear toi be saying something along the lines of "if you learn how to do this from start to finish, you wont need to borrow someone else's ideas, you might be able to see for yourself what is needed by paying attention and having the experience and confidence to build your own model in your own situation and environment"
Lets take another example to illustrate the point. I observe my parents as a child, and also other parents, when I am a parent myself. From that and other learning I know that some basics at the starting point of a child's development are critical. If you say something you should always follow through, always be there when they need you etc. etc. I don't believe that all children will end up the same, but I can learn. I don't have to always start from scratch.
So trying to exactly replicate the starting conditions might well be wasteful, it is in anyway impossible. However in the context of experiments and transfer of learning (the objectives of the post) one would not be attempting an exact replication, but rather to say something along the lines of XYZ put these things in place, had these resources and took these actions and things worked for them. Having done that, you move on work out the degree to which you can replicate those starting conditions, or where, given your local context to have to provide a substitute. You then take action and monitor, reinforcing good patterns, disrupting the bad ones.
Now here is where we diverge. As the Balle-Argentee case makes clear, it is exactly this process that leads to the destruction of the Silver Bullet. How do we know which parts of the starting conditions can safely be traded out and which not? The original team doesn't (their experience is both particular and no longer true), and neither do I because the conditions under which I am working are different (location, people, problem mix etc). And then I have to decide whether, given the differences, a particular pattern is "good" and another is "bad". The fact is that I don't, and can't, KNOW.
At every stage I am forced to make decisions based on grossly inadequate understanding of what I am doing (actually, it seems to me that, with enough diversity in the team we can make fairly good decisions due to some emergent product of convergent intuition) and the effects of those decisions. Never mind me, go and read Nicholas Taleb
Now I summarised that quickly in the phrase: In any complex system you can never replicate outcome, but you can replicate starting conditions. As a summary I stand by it, hopefully now I have fleshed out the detail a bit hopefully Earl may modify his criticism!
Modified, yes. although I don't know if we are any more aligned on this (just this) point. Johnnie, Jon H, any comments?
If not, then it is back to the Sassy Red.
Good idea, beer is the universal solvent, deconstructing our certainties and returning our knowledge DNA to the puddle of combinatory possibilities from which all great ideas (and some half assed one*) spring; the endlessly replicable initial condition.
*The vast majority of half-assed ideas comes from brain-storming sessions in marketing departments of major corporations.
The essence of any complexity based intervention is the option of a safe-fail experimental process in which you amplify or dampen emergent patterns to evolve to a desirable, but not definable future. In a complex system the past cannot replicate except by accident. So outcome based replication, whether you want it or not is not possible in theory, and does not work in practice in human systems where you are dealing with issues such as repeating success from one school to another. That is not the same as a professional using medical or therapeutic techniques.
Posted by: Dave Snowden | October 19, 2007 at 07:08 AM