I was recently invited to join a panel discussion at Realising Ambition’s annual Learning Event in Birmingham on the 1^st and 2^nd June which was focused on the concept and worth of Standards of Evidence. The UK-wide £25m Big Lottery Fund programme, which aims to replicate evidence based services to prevent children and young people from entering the criminal justice system, partly based its initial recruitment of projects, on their adherence to Dartington Social Research Unit’s standards. However, I got the distinct impression from programme partners in the run up to the event that enthusiasm for this approach might be starting to wane even if in other circles such standards are increasingly flavour of the month.

For me, Realising Ambition’s more reflective position is a welcome development as I have always been something of a sceptic when it comes to standards, regardless of whether they relate to evidence or otherwise. That’s not to say I don’t think things should be ‘good’ or even ‘excellent’. It’s just that often what is good for the goose is not always good for the gander. I chose to illustrate this point through a recital of my favourite scene from Quentin Tarrantino’s film Pulp Fiction in which Jules (Samuel L. Jackson) and Vincent (John Travolta) discuss the finer details of European travel:

Jules: So, tell me again about the hash bars.
Vincent: Okay, what you wanna know?
Jules: Hash is legal there, right?
Vincent: Yeah, it’s legal, but it ain’t a hundred percent legal. I mean, you can’t walk into a restaurant, roll a joint and start puffing away. You’re only supposed to smoke in your home or certain designated places.
Jules: And those are hash bars?
Vincent: It breaks down like this: it’s legal to buy it, it’s legal to own it, and if you’re the proprietor of a hash bar, it’s legal to sell it. It’s legal to carry it, but that doesn’t really matter ’cause get a load of this, all right? If you get stopped by the cops in Amsterdam, it’s illegal for them to search you. I mean, that’s a right the cops in Amsterdam don’t have.
Jules: [laughing] Oh, man! I’m going, that’s all there is to it. I’m fucking going.
Vincent: Yeah baby, you’d dig it the most. But you know what the funniest thing about Europe is?
Jules: What?
Vincent: It’s the little differences. I mean, they got the same shit over there that they got here, but it’s just there it’s a little different.
Jules: Example?
Vincent: All right. Well, you can walk into a movie theater in Amsterdam and buy a beer. And I don’t mean just like in no paper cup, I’m talking about a glass of beer. And in Paris, you can buy a beer at McDonald’s. And you know what they call a Quarter Pounder with Cheese in Paris?
Jules: They don’t call it a Quarter Pounder with Cheese?
Vincent: Nah, man, they got the metric system. They wouldn’t know what the fuck a Quarter Pounder is.
Jules: What do they call it?
Vincent: They call it a “Royale with Cheese.”
Jules: “Royale with Cheese.”
Vincent: That’s right.
Jules: What do they call a Big Mac?
Vincent: A Big Mac’s a Big Mac, but they call it “Le Big Mac”.
Jules: “Le Big Mac.” [laughs] What do they call a Whopper?
Vincent: I dunno, I didn’t go into Burger King. But, you know what they put on French fries in Holland instead of ketchup?
Jules: What?
Vincent: Mayonnaise.
Jules: God damn!
Vincent: I seen them do it, man, they fucking drown them in that shit.
Jules: That’s some fucked up shit.

Now let’s face it, McDonalds is a company that has made a pretty decent return out of possibly the most standardized approach to replication in the history of commerce. Yet it still has to adapt, to accommodate different measurement systems, different languages and, increasingly, different and changing tastes. The standards that apply in one location, don’t necessarily apply in another and, more to the point, that’s what keeps us interested. Jules and Vincent are only discussing the names of McDonalds’ signature sandwiches because they are different. That’s what makes them remarkable and worthy of consideration as to why they may not be the same.

So why would it be any different with the design of social programmes or the evidence we collect to demonstrate programme impacts. Even when validated by the most robust Randomised Control Trial, often regarded as the gold standard of evaluation techniques, we know that conformity to a prescribed set of programme components will not necessarily produce the same results in every location because in the real world we cannot control every variable. Staff may perform the same roles but they will not have the same personalities; projects might be delivered at the same time of day but that might not always be the ‘right’ time of day; participants might be referred by a partner in one context that simply doesn’t exist in another; in one location it might be sunny whereas in another, like Manchester, it’s more likely to be raining!

Rather than assessing programme fidelity with a model that has been found to ‘work’ in another context then, evaluation might be better focused on identifying the variations and associated adjustments that work better in other contexts and which might facilitate continuous service improvements. What this also points to is the senselessness of uniformly privileging one ‘type’ of evidence over another. Is it always better to assess success on the basis of a comparison of outcomes between those that are involved in a programme and those that are not? Might we sometimes gain more insight and be able to adjust provision in order to improve outcomes by using other methods such as participatory action research or real-time analysis of monitoring data? Is a validated questionnaire always more helpful in understanding someone’s state of mind than a friendly chat or even a role play exercise? I suspect not, since if the goal is to improve and achieve better outcomes then the focus should be on collecting evidence that will help achieve that end even it won’t ‘prove’ the approach will work in another time and place… since as Ronald found out, that will never happen.

Written by Tim Crabbe (Chief Executive)