Caroline Fiennes: Do we only publish graphs that go up? I suspect so

Charities should take a lead from the medical profession and investigate whether its published research is misleadingly positive. If they don't, it's their beneficiaries who will miss out, writes our columnist

We can't make evidence-based decisions if the literature is incomplete or biased, writes Caroline Fiennes
We can't make evidence-based decisions if the literature is incomplete or biased, writes Caroline Fiennes

It's hard to make evidence-based decisions if much of the evidence is missing or ropey. So it's disastrous that the results of many medical clinical trials are missing, preventing doctors from using them.

It's thought that fully half of all clinical trials remain unpublished. It's not difficult to guess which half. Apparently, trials published by pharmaceutical companies are four times more likely to show that the drugs have a positive effect than identical trials conducted independently. So why is that?

Well, trials themselves don't lie. Magically, however, the negative ones don't get published. This publication-bias costs lives, yet is perfectly legal. Dr Ben Goldacre, author, broadcaster, campaigner and debunker of bad science, says that the near-fatal effects of a drug trial at Northwick Park Hospital in Harrow, north-west London, a few years ago – when all the men in the trial ended up in A&E with multiple organ failure – could have been predicted from results that were known but not published.

So we should all applaud the AllTrials campaign, initiated by Goldacre and seed-funded by the science author Simon Singh, to ensure that the results of all trials are published. Goldacre and Singh take scientific integrity seriously: both were accused of libel for highlighting bogus claims, refused to recant, endured horrible, long legal trials – and won.

Does the charity sector have the AllTrials problem? Do we withhold some of our monitoring and evaluation research and publish only unrepresentatively positive and misleading material? I suspect so. I did it myself when I was a charity chief executive: graphs that go up made for good meetings, so we published them; graphs that go down made for terrible meetings, so we didn't. I don't believe we were alone.

Monitoring and evaluation is research. It's not always framed as that – it's often seen as compliance with the requirements of funders; but it's there to investigate possible causal links. Whether and when does intervention X lead to outcome Y? Do breakfast clubs improve children's learning? Does teaching prisoners to read help them get jobs when they are released? These are research questions.

Ideally, that research would not be private but would be published to constitute evidence-based practice, just as clinical trials guide doctors. Any other charity could use it to decide whether the intervention might work in its context and whether it should replicate it.

But we can't make evidence-based decisions if the literature is incomplete or biased; and, as ever, it's our beneficiaries who miss out.

Research-withholding and publication-bias are commonly studied in medicine to establish whether and where there's a problem and work can be targeted to fix it. But to my knowledge, neither has ever been studied in our sector. Not once. One study (itself unpublished) in Canada that was looking at something else found – shockingly – that the proportion of research carried out by charities that was published was 2 per cent.

Investigating the withholding of research and publication-bias is neither difficult nor expensive. It's time we knew whether we too need a fix.

Caroline Fiennes is director of Giving Evidence and author of It Ain't What You Give, It's The Way That You Give It

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Register
Already registered?
Sign in
Follow us on:

Latest Jobs

RSS Feed

Third Sector Insight

Sponsored webcasts, surveys and expert reports from Third Sector partners

Markel

Expert hub

Insurance advice from Markel

How bad can cyber crime really get: cyber fraud #1

How bad can cyber crime really get: cyber fraud #1

Promotion from Markel

In the first of a series, we investigate the risks to charities from having flawed cyber security - and why we need to up our game...

Third Sector Logo

Get our bulletins. Read more articles. Join a growing community of Third Sector professionals

Register now