Marsha Orgill, Sara Van Belle, Woldekidan Amde, Aku Kwamie, Thubelihle Mathole, Gaëlle Vareilles, Emilie Robert, Valeria Campos da Silveira and Bruno Marchal

 

The authors, members of the SHAPES cluster group on ‘theory-driven inquiry’ attended the 1st International Conference on Realist Approaches to Evaluation and Synthesis   in Liverpool, UK (27^th – 30^th October 2014). This was the first formal realist conference ever held, hosted by the new Centre for the Advancement of Realist evaluation and Synthesis of the University of Liverpool (CARES). Here are their notes (including the deep insights that emerged during a wonderful Italian dinner!).

This first conference was a quaint, well-organised get-together with approximately 180 people in attendance. The premises of the old New Royal Infirmary on the University’s campus provided a highly conducive context to meeting almost everyone there. A mix of researchers from a range of countries and disciplines made for lively cross-country and cross-disciplinary discussions. It is clear that realist evaluation and especially realist synthesis is currently making big waves in the UK, Canada and Australia, with smaller waves rippling through Ghana, Spain, France, Switzerland, Sweden and South Africa.

The conference had a clear methodological focus on realism and its application in realist synthesis and realist evaluation. The aims were indeed to provide a forum for networking and knowledge sharing among international researchers interested in advancing realist methodology in the health and social sciences, and to stimulate researchers and practitioners to present their realist research, discussing their successes and challenges. Opportunities for training on realist methodology were provided. The conference was also ‘philosophical’ in its approach, devoting time to considerations of realism (“how real is real”?), meanings of “what works” (what has worked, what can work, what will work?), and the easy conflation of contexts, mechanisms, moderators and mediators…

In addition to these theoretical and methodological leanings, presentations were also content-rich, dealing with realist research on a cross-section of social challenges, such as helping communities to use sport to promote learning in school children, organising effective management training programmes, synthesising knowledge for policy makers on reducing violence amongst young people, mental health services in correction sub-systems, and integration of care delivery platforms for the elderly. Despite the conference’s compact size, it was often difficult to choose between comparably exciting parallel sessions.

Well-known names in realist evaluation were in attendance, including Ray Pawson and Nick Tilley (authors of the seminal text ‘Realistic Evaluation’), Trisha Greenhalgh, and Gill Westhorp. Seasoned presenters as they are, they captured the imagination of the audience with insightful knowledge, creative presentations – and a good dose of humour. Nick Tilley talked about Policy Makers and Realist Evaluation and explained how he managed to find himself in a Department of Engineering at University College London. He was clearly fascinated by engineers’ ‘archetypal realism’, the fact that their systematic analysis of failure lead them to be innately theory-driven – perhaps this could be a lesson for policy-making (shifting from ‘evidence-based’ towards ‘tested-and-testing-theory-led’)? Trisha Greenhalgh dealt with challenging issues in realist research, such as how it relates to positivism and constructivism. She also challenged realist researchers to think whether they should join the campaign for real EBM. On a lighter note, Ray Pawson was the quiz master of the first ever realist evaluation quiz during the conference diner.  As he attended the various sessions of the conference, it was quite insightful as well as a great learning opportunity having him in the audience. As he put it during Melanie Ehren’s presentation Wrangling with realist synthesis, “this is a direct on-spot supervision”!

 

There were also a number of plenary sessions that provided insights into how policymakers and research commissioners think about realist research. In his talk Research impact: can realist research make a difference, Kieran Walshe offered a view behind the scenes of the research commissioning processes of the health services and delivery research programme of the National Institute of Health Research.  Similarly, Mike Kelley, Director of the Centre of Public Health at NICE (National Institute for Health and Care Excellence – UK) used Greek mythology (Dionysus being the god of  disorder and complexity!) to talk about the bulk of behaviour change interventions to date being based on psychology, and the potential for integrating more sociological theory to help us understand how altering the structures of social practices are equally important mechanisms in changing behaviours.

 

Some notes and lessons learned from the perspective of well-read newcomers to realist evaluation:

• While it is good to develop your original programme theory (PT) empirically with people on the ground, one must build the PT on existing social science theory and move beyond the superficial explanations.
• Mechanisms are at the heart of the realist explanation. In the words of Pawson and Tilley, they are made up of ‘reasons and resources’ – it is thus the combination of the resources of the programme as well as the underlying reasoning of the actor implementing the programme that need to be explained in realist evaluation. Mechanisms were discussed as latent, lying in wait for an awakening (e.g. we all have the latent capacity for happiness, and when your child passes a school grade the happiness mechanism is triggered).
• There are four ways to think about mechanisms: forces (physical, psychosocial, institutional); interactions (transfer of something from a party to the other, changed state); power and liabilities of things; processes (sequences where later elements depend on earlier ones).
• In RE, short-term outcomes are regarded as equally valuable as measuring long-term outcomes. Indeed, short-term outcomes can be part of the mechanism or of the context of the next link of the causal chain and need to be unpacked as such.
• The ‘fake handbags’ and ‘horses to courses’ were perhaps the two most used and memorable expressions at the conference. The former is a reference to the need to remain true to the principles of realism and be wary of unwarranted labelling of evaluations and synthesis as realist, and the latter a recognition of diverse ways in which we can apply realist evaluation.
• It is imperative to engage with extant literature to better understand and analyse the psychological and social mechanisms underlying the causal processes. This will also give more credibility to our research and allows us to make a worthy contribution to the body of knowledge.
• Globally, almost everybody faces the challenge of short-term funding while being expected to measure long-term impacts and document lengthy and complicated causal pathways.

 

From the notes of some seasoned realists

• Realist evaluation starts from an explicit embracing of complex causation – after all, much of the world that is of interest to us is social in nature. Several delegates dealt with the issue of complexity, presenting their thoughts on the Context-Mechanism-Outcome configuration, or on how perhaps simple rules can capture the essence of the change mechanism. However, the word ‘complexity’ is still understood in different ways and there isn’t yet a consensus on whether the concepts of complex adaptive system theories can be transposed wholly to the social world.
• There was some tension between the purists and the pragmatists. For the latter, a definitive definition of ‘mechanism’ does not matter much, as long as the study manages to produce a plausible explanation of what was observed. The former argue that realist evaluations should not become mere pragmatic mixed-method studies. They argue that refining well-built programme theories or middle-range theories is at the core of realist inquiry, and these should be built on the basis of both empirical research and theories (from social sciences, psychology, political sciences, organisational studies, etc.).
• Another latent bone of contention reflects a longstanding debate in the social sciences: is it agency or structure that explains how societies evolve and programmes are implemented (or not)? There has always been some critique that some realist evaluators focus too much on the individual as an agent of change, seeking explanations in individualistic theories. Others focus mainly on context as a facilitator or barrier to programme success. Informal discussions pointed to Archers’ view on the agency-structure interaction (morphogenesis), to relational sociology or David Byrne’ s work as sources of inspiration in developing realist explanations that focus on the continuous interaction between agents and structure, and that is indeed the realist’s perspective.

 

The presence of many PhD students using the realist approach for their doctoral research is encouraging. It means that in a few years from now, well-trained researchers will be able to further use, train, and reflect on the realist approach. The Thematic Working group SHAPES, and its theme Theroy-driven inquiry will contribe to building this capacity. Currently action plans are being drawn, and we expect to be able to set up a number of exchange platforms, a buddy system and other processes to allow learnng, developing and implementing realist and other theory-driven inquiry approaches to Health Policy and Systems Research. Let us know if you want to go with us on this fascinating journey!

 

 

Affiliation of authors

Marsha Orgill: Health Economics Unit & Health Systems Division, School of Public Health & Family Medicine, University of Cape Town, South Africa (ms.orgill@uct.ac.za)

Sara Van Belle: Department of Public Health, Institute of Tropical Medicine (svanbelle@itg.be)

Woldekidan Made: School of Public Health, University of the Western Cape, Cape Town, South Africa (woldsha@gmail.com)

Aku Kwamie: School of Public Health, University of Ghana, Accra, Ghana (aku.kwamie@wur.nl)

Thubelihle Mathole: School of Public Health, University of the Western Cape, Cape Town, South Africa (tmathole@uwc.ac.za)

Gaëlle Vareilles: French School of Public Health, Rennes, France & International Federation of Red Cross and Red Cerescent Societies (gaelle.vareilles@ifrc.org)

Emilie Robert: Centre de recherche du CHUM, Université de Montréal, Canada  (emilierobert.udem@gmail.com)

Valeria Campos da Silveira: Department of Public Health, Institute of Tropical Medicine (vcdsilveira@itg.be)

Bruno Marchal: Department of Public Health, Institute of Tropical Medicine & School of Public Health, University of the Western Cape, Cape Town (bmarchal@itg.be)