What works or what may work, and bets on whether it will
March 12, 2013
Opinion piece: by Michael Bassey AcSS, Emeritus Professor of Education of Nottingham Trent University:
On 4th March the Cabinet Office issued this press release:
LAUNCH OF THE WHAT WORKS CENTRES
A new initiative will build on existing evidence-based policy making to guide decision-making on £200 billion of public spending. National government and local public services will benefit from access to a world leading network of centres providing robust, comprehensive evidence to guide decision making on £200 billion of public spending, announced the Chief Secretary to the Treasury, Danny Alexander, and Minister for Government Policy, Oliver Letwin, today.
Should we say “hooray” and welcome what has seemed for too long to be ‘policy-based evidence’ giving way to evidence-based policy? Or should we recall what Davies, Nutley and Smith said in their What Works: Evidence-based policy and practice in public services (Policy Press, 2000):
The rise of evidence-based policy reached its apotheosis in 1997 when a Labour Government was elected with a philosophy of what matters is what works.
Or perhaps note that the Campaign for Social Science of the Academy for the Social Sciences has for the past two years been trying to “inform and influence public policy with social science” with an impressive series of booklets (seven so far) entitled Making the Case for the Social Sciences. Seems a pity, but typical, that politicians Alexander and Letwin ignore the significant work of academics.
But my purpose here is not to denigrate this government initiative, but to encourage it – yet with a methodological caution.
In 2001 I had a paper in the Oxford Review of Education (Vol 27, No 1, 5-22) entitled ‘A Solution to the Problem of Generalisation in Educational Research: fuzzy prediction’. In the conclusion to the paper I wrote:
Educational research shares with research in the other social sciences the problem that, because it is social, i.e. about human beings, it inevitably embraces a multitude of variables. This precludes the making of scientific generalisation. However by invoking the principle of fuzziness, that there is a class of statements which are imprecisely probable, it is possible to develop the idea of fuzzy generalisation.” While the natural scientist usually feels able to say “x in y circumstances results in z”, the social scientist is wiser to say “x in y circumstances may result in z”.
This is relevant to the politician’s request to be told ‘what works’. I have long insisted that all that can be said to the politician is ‘what may work’. Good research provides such fuzzy prediction – and hopefully is able to spell out the circumstances in which this has a high probability of being the case. But the politician is entitled to ask what is the likelihood that something ‘may work’. I tackled this in my paper with what I called ‘best estimate of trustworthiness’ (BET):
The difficulty with the fuzzy prediction if x in y circumstances then z may occur is that logically it must also be true that if x in y circumstances then z may not occur. The potential user of such a research conclusion will want some idea of how likely it is that z may occur. Rather tentatively I would like to put forward the following suggestion.
Suppose that researchers, having worked on the topic and read the relevant literature, from their professional experience hazard a guess as to the likelihood of z occurring. Call this the best-estimate-of-trustworthiness, or BET and give it a fuzzy form.
A BET could be in the form ‘very likely’, ‘likely in 9 out of 10 cases’ or a paragraph of prose. Making such a BET takes the researcher beyond the empirical evidence of a research project and into the realm of tacit and professional knowledge. It requires the courage to ‘put one’s head above the parapet’. But it should be of value to the politician and help ensure that new policies are less likely to trample on the minorities for whom “z” does not apply.
This is not academic pedantry. Because something works in some situations, the politician should be wary of legislating for it to be done in all situations.
It would be pedantic to insist that the new post is called ‘What May Work’ national advisor: but an appreciation of the inherent fuzziness of ‘what works’ would be wise.
My colleague Andrew Pollard has taken this further. He urges that instead of expecting policy to be evidence-based, it should be seen as evidence-informed. Recognising that policy making is influenced by many factors other than evidence such as party ideology, personal ambition, and opportunism, this may be as much as we can hope for.
Andrew also cautions about my use of ‘fuzziness’. He points out that politicians might think a ‘fuzzy’ conclusion means that the research methodology rather than the research outcome was fuzzy – and this would do a great disservice to researchers. So, I am content to keep this concept, which I hold to be important, to academic discourse, and encourage colleagues, when describing ‘what may work’ to urge politicians, if they take note of research findings, to recognize that their policy decisions will at best be evidence-informed rather than evidence-based.
• Michael Bassey is an emeritus professor of Education of Nottingham Trent University, a former president of the British Educational Research Association, and an academician of the Academy for Social Sciences. His latest book Convivial Policies for the Inevitable: global warming, peak oil, economic chaos (Book Guild 2012) sees the future as fuzzy and frightening.