It's time to stop relying mainly on opinion

Analysis of the voluntary sector has long been hampered by a poor evidence base, writes Nick Seddon, and it's about time we got the basic boring stuff right.

For most of us, most of the time, talking about the third sector involves pointing to details of an immense landscape that few of us even vaguely know our way around.

Not that this is surprising. To put it bluntly, no one can say exactly how many charities there are in this country, let alone how many bodies there are in the third sector as a whole. And no one has any idea how much money is involved. On the basis of registered main charities alone - there are 169,000 of them with a total income of more than £42bn - there is more money flowing into the sector than was spent on the NHS when Labour came to power.

But this doesn't take into account unregistered charities, many of which, like universities and housing associations, are major financial and human resources, or the hundreds of thousands of organisations in the third sector that aren't charities at all. Given that it's such a major part of the economy, it's clear the third sector is poorly served by its evidence base.

Changing dimensions

A key strategy for understanding the sector has been to use surveys. These can be handy barometers for gauging trends in opinion, but they aren't exactly what you'd call hard data. That the recent nfpSynergy survey found 71 per cent of respondents agreeing that what it means to be a charity is "increasingly blurred" may usefully alert us to rising anxiety in a frenetic sector that doesn't have, as Karl Wilding, head of research at the NCVO, puts it, "even the seven-digit charity number as a unifying point of reference". But it doesn't tell us anything about the sector's changing dimensions.

Invoking precise percentages also beguiles us into believing that these surveys are more accurate and based on more robust methodologies than they actually are. The Charity Commission's survey has caused particular consternation. While it seemed to confirm the concerns of many, such as me, who have subsequently repeated the findings with almost mantric glee - 88 per cent of charities aren't achieving full cost recovery, 67 per cent of large service delivery charities get 80 per cent or more of their income from the state - these findings were skewed because they came from a self-selecting, unweighted sample with a 2 per cent response rate, mainly smaller charities not engaged in statutory service delivery. The figures are, says Stephen Bubb, chief executive of Acevo, "at best misleading". To a statistician, they're all but invalid.

Stop relying on opinions

We have to ask how the sector is best served by all this polling. And even if we were using the Rolls Royce of surveys, we shouldn't allow the proliferation of such material to make us complacent about the paucity of real evidence. It's time we stopped relying for our opinions on - well, on opinions.

"It is astonishing," says Bubb, "that a sector of such importance to economic and social wellbeing is so badly served by its data and statistical base." More and more people are beginning to say the same. It's not that there's no data - far from it. The NCVO's UK Voluntary Sector Almanac, for example, offers authoritative data that is particularly valuable because it enables longitudinal comparisons - comparisons over a number of years. However, as the NCVO acknowledges, an enormous swathe of the sector - housing associations, religious bodies, quangos and more besides - falls outside its remit. There are also many other umbrella bodies, funding bodies, academic departments and charities with large research departments, not to mention government departments; but all are, in Wilding's words, working like sole traders, operating in isolation from one another.

Our evidence, and therefore our understanding of the sector, is disparate and disjointed. Wilding puts it neatly: "We would argue that we don't have an evidence culture or infrastructure." To remedy this, we need to get the basic, boring stuff right, as he points out: "Among other things, we need to harmonise the definitions and classifications in use by different research bodies." It's not much of a headline slogan, but taxonomy matters. If we're to describe the changing characteristics and dynamics of the third sector, particularly its economic and organisational dimensions, we need to aggregate and synthesise the diverse sources and approaches.

Accordingly, momentum is gathering for a single body to fulfil such a function. Even government is starting to show interest. "The suggestion of a single responsible body is an idea the Office of the Third Sector is actively looking into," says a spokesman for the Cabinet Office. "We are consideringthe ideaas part of our broader work on building the evidence base in partnership with the sector."

One of the models talked about is the Institute for Fiscal Studies, an independent economic research organisation. In a recent poll by Third Sector - another self-selecting survey - 77 per cent of respondents said such a body was needed. There would be merits to this. However, as Martin Brookes, director of research at New Philanthropy Capital, observes: "While the IFS takes a very outward-looking perspective towards policy makers and the media, the analogy works only to a limited extent. The IFS doesn't produce basic data. It's an applied research body that processes data produced by others, such as the Office for National Statistics." So we still need the number crunchers.

Taking the lead

This is where GuideStar could come into play. One of the driving forces behind the online research institute was to provide a common information platform that could be used as the evidence base for research and policy analysis, according to Les Hems, GuideStar's director of development. He says: "We're actively working to hold data on all third sector organisations, to offer more extensive financial, economic and classification information." Currently supported by the Treasury, the Home Office and the Charity Commission, plus private donations and grants, GuideStar would need to increase capacity substantially to become the sector's chief boffin hub, but there's much to be said for identifying an existing organisation that's doing a good job and reinforcing its role.

As well as boasting a sophisticated search facility for the public, organisations are encouraged to post their own impact reports and general information about themselves, a facility that Cancer Research UK, to take one prominent example out of 12,000, has used to good effect.

A similar principle is propelling a forthcoming NPC project, an online wiki library on which charities will be able to post outcomes and effectiveness data. Brookes sees this as a way to 'turbo-charge' the flow of information. Mapping the changing landscape is not about ordering it, but understanding it - and, as some are beginning to realise, the sector needs to take the lead. "We need to be able to supply data but maintain our independence," says David Brocklebank, GuideStar's chief executive. "The third sector should embrace this as a way to engage with the public."

One day we may even be able to base our opinions on facts.

Who keeps the facts and figures?

Charity Commission

The commission is the regulator and registrar for charities in England and Wales, and it holds data on all registered charities. The commission does not publish statistical reports, and the level of granularity and sophistication of the information that is available is disappointing.

National Council of Voluntary Organisations

Using data collected by GuideStar, the NCVO publishes an impressive overview of the voluntary sector and its work, the UK Voluntary Sector Almanac. A major strength of the almanac is that it has been running for more than 10 years, enabling us to chart changing trends. Because it limits its research to what it calls the voluntary and community sector, thereby excluding some of the biggest organisations, such as government bodies with charitable status, the downside from a wider perspective is that it doesn't - and doesn't claim - to speak more broadly for the third sector.

Charities Aid Foundation

CAF publishes an excellent annual report, Charity Trends, which provides detailed statistics and analysis of current trends for the top 500 fundraising charities, relating to areas such as workforce and voluntary and statutory income streams. Using NCVO data, it attempts to contextualise this information in a wider sector catchment. A snip at £164.


GuideStar UK has been dubbed the "Bloomberg screen of philanthropy" because of its quest to push donations into charity by increasing the range and transparency of the information available. Although it wouldn't claim to be comprehensive, this web-based data-capture institute provides the broadest and deepest information for people wishing to understand the third sector. Its new intelligence services for lawyers and local government commissioners provide unrivalled information about organisations, but sadly you have to pay for these.

But research must be useful to the sector

Martin Brookes and Joe Saxton argue that a one-stop academic shop is not the silver bullet.

Discussion about the evidence and research base for the sector is in vogue at the moment. Not only is a giving and philanthropy centre of excellence in the pipeline, but an equivalent to the Institute of Fiscal Studies for the sector is also being proposed.

There is no doubt that more research is needed. There is a desperate shortage of good research, of all types, that looks at or is aimed at the sector. One should therefore welcome initiatives to improve the evidence and research base, and these are absolutely needed. But to go from realising there are some serious gaps in the sector's collective knowledge to deciding the answer must be a one-stop academic shop is a flawed step.

The idea that an 'IFS of the sector' represents some sort of silver bullet is mistaken. Replicating the IFS would be virtually impossible and the attempt would show a misunderstanding of the institute itself and the world in which it operates.

Our belief is that research of all types is needed and any strategy should encourage a 'mixed ecology' of research. This includes refereed academic papers, applied research such as that carried out by nfpSynergy, 'sub-sector' research by New Philanthropy Capital, collections of data from charities, 'think piece' research by research institutes and evaluations produced for grant makers and published. The basic problem is less the lack of a single centre of excellence than a poor quantity of research and a poor flow of information about what research is available. An online 'results library' of the kind NPC has proposed would provide one effective route to better research dissemination.

But the sector is both part of the problem and part of the solution. The sector itself needs to show a greater appetite for research. As part of this, it needs to be more self-critical and questioning. So although it is important what politicians or funders or academics think, research should above all be driven by and useful to charities, social enterprises and voluntary and community groups.

Martin Brookes is director of Research at New Philanthropy Capital and from August will be a council member at the Economic and Social Research Council.

- Joe Saxton is founder of sector research consultancy nfpSynergy.


Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in

Before commenting please read our rules for commenting on articles.

If you see a comment you find offensive, you can flag it as inappropriate. In the top right-hand corner of an individual comment, you will see 'flag as inappropriate'. Clicking this prompts us to review the comment. For further information see our rules for commenting on articles.

comments powered by Disqus