Sean Redmond
Civil Servant in the Department of Justice seconded to the School of Law, University of Limerick; Head of Research Evidence into Policy, Programmes and Practice (REPPP) research team
No one advocates for policy or social programmes with an absence of evidence. However, greater appreciation of the complexities of social policies and programmes presents new demands for the application of scientific evidence to resolve real-world problems.
It is unnecessary to append the word ‘evidence’ to social programmes nowadays. Its ubiquity makes any ambition toward evidence-based policy a So what? claim, at least in normative terms. Evidence-based programmes (EBPs) are like apples in pie: why wouldn’t you want it?
EBP as a paradigm has its origins in social and welfare programmes in the US,1 took hold in the 1980s and became a social-science high-water mark in the 1990s and early 2000s. At its peak, advocates determined a hierarchy of evidence for social programmes, with randomised control trials (RCTs) at its zenith. RCT is a method that randomly assigns research subjects to control or treatment groups, thus controlling for possible external influences and establishing a counterfactual baseline to determine the true effect of an intervention. In youth justice, the hierarchy of evidence has been used to recommend whether programmes should be supported by government funding.2
On closer examination, however, some observers questioned the veracity of claims of public investment being skewed toward EBP. This commentary, with some merit, holds that EBP is still quite peripheral in reality, limited to programme-islands based on professional discipline or appended to business as usual as hermetic entities but external to the silos they were intended to disrupt.
Other commentators say the evidence to support EBP is limited: that RCTs are only one type of evidence, albeit with experimental surety, that should be considered alongside other important sources (e.g., craft knowledge from front-line professionals and citizens’ lived experience). The reasoning suggests that multiple data points make for better judgements, and that the effects of EBP are determined significantly by context.
A recent evaluation undertaken by Research Evidence into Policy, Programmes and Practice (REPPP), of Ireland’s first bail supervision programme for young people,3 was careful to parse out the ‘enabling conditions’ shown to be instrumental in assuring optimal performance. While RCT probably offers the highest standard of causal claim for attributing a particular outcome to a particular programme of intervention, its inability to explain why and how – a good example of the ‘black-box effect’ – can impoverish policymaking because of the need to generalise learning from programme experience.
This need is particularly stark in more complex contexts, where theory development arising from thematically testing programme hypotheses can shed light on ‘stuck’ problems. Moreover, this limited knowledge exchange, which shares only results-related information from EBP without showing the mechanisms under the bonnet risks a dependency relationship between policymakers and EBP providers.
If epistemic communities have been elevated to arbitrating what is important to research and the rules for determining the threshold for viable evidence, and retain why and how knowledge regarding programme performance, legitimate concerns have been raised about who governs science.4
Thankfully, a more mature relationship appears to be developing between policy decision-making and science in the area of youth justice in Ireland. Here, REPPP and youth-justice decision makers are developing a ‘policy-informed evidence’ discourse recognising that values and science are not easily separable.5 Tension will always exist from science speaking truth to power, however uncomfortable, by way of rigorous methods and procedures. But there is a compelling argument that the values informing priorities for science be distributed more appropriately across the political system and civic society.
Policy-informed evidence has led to some interesting scientific requests to REPPP by policymakers. One request, also seed-funded by the Irish Research Council,6 refers to mining routine data held in the youth justice system in Ireland to determine its capacity to yield outcome-related information, aligned to policy aspirations for young people. Lots of institutional energy is dedicated to data collection, but it is questionable to what degree the data can provide more than evidence on investment levels (inputs) and programme activity (outputs), as opposed to whether the activity itself is associated with better outcomes for society.
This research, which included a comparative study of jurisdictions considered to be youth justice leaders, suggests that Ireland is in good company with its data limitations. The research also, importantly, provides practical fixes to improve the capacity for outcomes-based evidence and aspirations toward youth justice policy reform.7
Another request refers to co-designing a more effective model of relationship-building with young people and front-line youth professionals. From a values perspective, there is a clear imperative. One of the largest Department of Justice investments in the youth justice space is the national Garda Youth Diversion Project (GYDP) programme, which operates in collaboration with voluntary youth organisations.8 GYDP was allocated over €18 million in 2020.9 Approximately 75%–80% of its budget is spent on professional time (i.e., salaries) by collaborating youth organisations. At least 60% of professional time is spent on building effective relationships with young people to bring about positive change.
Professionals, young people, families, and communities all value relationships. However, for an estimated investment of at least €8 million a year dedicated solely to relationships, very little is known or documented, in Ireland or internationally, about the most effective exchanges, risking quality by geography.10 Better understanding of what is known about effective change relationships has huge potential for informing other human service programmes sponsored by the state.
Relationships and relational practices are hugely intriguing to science as a way to help young people navigate adversity. Research in this area offers the prospect of modelling and scaling effective interactions bottom-up, from micro individual exchanges between youth professionals and young people to the macro design of targeted youth programmes. Over the last two years, REPPP has been co-designing action research with front-line professionals after a wide-ranging systematic evidence review. REPPP’s approach has incorporated design thinking, where user knowledge is integral to discovery – not just intelligence to help implement evidence-based programmes.
Big science can confidently predict general patterns of youth offending, aligning Ireland to international trends which indicate that children who commit crime generally grow out of it as they mature into young adults. The evidence supports a light-touch policy, keeping children out of the criminal justice system if possible. Indeed, over the years, thousands of tax-paying, law-abiding adults have probably benefited from this often-unsung, evidence-informed policy which ensures they are not burdened forever with a criminal conviction.
Far less is known about those who do not follow this trend. Referred to as the ‘right-hand tail’,11 a very small group of children are considered to be responsible for over half of all youth-related crime. Another policy-informed evidence request to REPPP is to better understand the contexts of children involved in crime networks, particularly collaborations with adult-network actors.
Programme responses, though receiving increasing international attention, remain under-developed and limited. REPPP’s application of social-network analysis to determine patterns and flows in criminal networks, and its qualitative examination of networks using our own methodological innovation Twinsight12 to disclose narratives of networks actors and their relationships, led to an evidence base sufficient to fund and trial two intervention programmes in 2020. This research, which takes us into the realms of complex youth-justice design, is attracting significant international interest.
The phrase wicked problem, originally coined by Horst Rittel in 1973,13 is creeping into mainstream discourse on programme planning. Wicked problems contrast with ‘tame problems’, which are linear and bounded, play by the rules, and require only scientific, technological, or engineering solutions, for example designing a railway system.
Wicked problems are complex, regressive, and subject to the vagaries of external systems. They have competing interests advocating diverse diagnosis and remedies. Such complexity leaves programme planners with options that are not solely right or wrong but good or bad, dealing with infrastructural legacies of bad historical decisions and ongoing exposure as plans roll out, due to limited opportunities to lab-test solutions and plan their application before hitting the ground in real-life.
It was not prescience but post-1960s scepticism that led to the name wicked problem. However, loosely comparing Rittel’s ten properties of wicked problems to the presentation of the Covid-19 pandemic, the fit is compelling and sobering. Plans and evidence-based programmes, however well substantiated, are brittle in the face of the turbulence and unique challenges of wicked problems. As the boxer Mike Tyson said, ‘Everyone has a plan till they get punched in the mouth.’
Nevertheless, plans and programmes are widely demanded when the most effective way to chart a bumpy trajectory ‘north’14 to resolve or survive a wicked problem is probably competent stewardship. Facing persistent instability and significant knowledge gaps, but with a need to forge through the fog, would you put your trust in quality-assured route planning offered by Google Maps, or in a policy version of Amelia Earhart drawing on her scientific and craft knowledge to muddle through?15 I know which one I would choose.
At the request of policymakers, REPPP works with coalitions of stakeholders to understand and work through real-life wicked problems in local communities in Ireland using Rittel’s framework.16 The problems encountered concern community safety, perceived trust gaps between state and local community action, and embedded intergenerational drug economies. The problems may not be at pandemic level, but they are no less important to a citizen’s lived experience.
The wicked problem challenge has demanded a far more dynamic application of evidence to supporting real-time judgement and navigable, reflexive strategies for improvement. For more information on the work of the Research Evidence into Policy Programmes and Practice initiative, funded by the Department of Justice, please visit https://ulsites.ul.ie/law/node/106531 or contact Dr Sean Redmond at sean.redmond@ul.ie. Follow us on Twitter: @REPPP4.
ENDNOTES
Copyright © Education Matters ® | Website Design by Artvaark Design