Caroline Fiennes: Why we looked at every relevant study instead of doing a new - but limited - one

The charity sector should do more systematic reviews to find out what is reliably known already, writes our columnist

Caroline Fiennes
Caroline Fiennes

Any single piece of evaluation research, designed to understand the effect of an intervention, has limitations. It will examine the effect of a particular intervention on some particular outcomes in a particular group of people, referred to in research as "populations", at a particular time. That's fine, but it inevitably limits the value of the research for organisations using, say, the same intervention on a different population. Studies also vary in their robustness - the chance that their answer is wrong - and even a good study can get a weird result by fluke.

Better, then, to look at multiple impact studies when designing programmes or making funding decisions. This is what the Blagrave Trust, a foundation, recently asked my organisation, Giving Evidence, to do on the subject of outdoor learning, one of its funding areas; the report will be published next month. With University College London, we looked for every relevant study of outdoor learning published over the past 10 years. Systematic reviews such as this enable people to stand on the shoulders of myriad giants - and see a long way.

The first stage is deciding precisely what you're interested in - location, types of intervention, populations, study designs and so on - and how you will find the material. That scope and method gets written up and published so that anybody can check it or repeat it.

Within the turf we defined, we found 15 systematic reviews of outdoor learning from various countries. Collectively, they cover a huge terrain: one of them analysed 150 studies, another 96 studies, another 58. Their combined insights are much less open to bias or irrelevance than any individual study, and they show the patterns much more clearly.

We also looked for primary evaluations (studies of people, rather than studies of studies) done in the UK, both in the academic literature and by "crowdsourcing" through the project's steering group. We found 58. The smallness of the number surprised some people - there are several entire journals about outdoor learning, though a study of them found that only 11 per cent of the research they report assesses outcomes and effects. (That isn't to deride the rest: they just had different purposes.)

Two features of the 58 UK primary evaluations are striking. First, they're spread pretty thinly, across many interventions, age groups, outcomes and so on. That spread limits their reliability because any of them might be flukes. Second, they don't score well for quality. We assessed them on the five-level scale that Project Oracle uses for evidence to underpin youth work in London. Only about half make it onto the scale at all, only one makes it to Level 3 and none gets higher than that.

A central tenet of medical research is to ask an important question and answer it reliably. Systematic reviews like this can show what's reliably known already and guide future research by highlighting important remaining questions so that we can focus our finite research resources on answering them properly. The charity sector should use systematic reviews much more: they're often much better than the standard approach of commissioning one new, fresh, but limited study.

Caroline Fiennes is director of Giving Evidence and author of It Ain't What You Give

Before commenting please read our rules for commenting on articles.

If you see a comment you find offensive, you can flag it as inappropriate. In the top right-hand corner of an individual comment, you will see 'flag as inappropriate'. Clicking this prompts us to review the comment. For further information see our rules for commenting on articles.

comments powered by Disqus