There is a huge amount of data in the world: the IT company IBM estimates that 2.5 quintillion bytes of data are created every day.
Using and analysing large data sets, or so-called 'big data', is key to staying competitive in the private sector, but charities are also increasingly looking to data to help them become more effective and to improve outcomes for their beneficiaries.
Definitions of 'big data' vary, but Simon Rogers, data editor at Twitter, has defined it as "one byte more than you are comfortable dealing with".
Duncan Ross, founder and director of the charity Datakind UK, which matches data science volunteers with social organisations to help the latter gain insights from their data, says that big data involves using diverse sources, some of which might be very large, and employing mathematical techniques to extract new information from them. Ross says charities should not be put off by the word 'big' - this could refer to the importance of the data, rather than its size.
Ross says big data could bring significant benefits to charities. "Decisions in all sectors have typically been made by people who have grown up in the organisations and are based on their experience," he says. "But what study after study has shown is that experience does not compare with experience plus data. It's crucial to find data to show that what you are doing works. But it is not only about supporting your work with evidence; it is also about finding out how to do things more effectively and how to do more with the resources you have."
Using data effectively should be part of a charity's mission, he says: "Charities have a duty to be as effective as possible with the resources available. Anything that is not is a betrayal of the trust in charities. It is easy to keep on doing things you do simply because that's what you've always done. Data allows you to see if that is the right thing to do."
Charles Nall, director of finance at Moorfields Eye Hospital and a former finance director at the Children's Society, believes that if charities do not start making use of data, they could find themselves at a disadvantage because other organisations might fill this void. He says: "In future, it might be a think tank or a technology company brought in by government to analyse social issues. That could apply a different perspective and set of algorithms to constructing these interventions."
Karl Wilding, director of public policy at the National Council for Voluntary Organisations, shares the view of the philanthropy commentator Lucy Bernholz that "data is the new fuel for social change". Wilding says: "Money is one of the things we need, but it is not the only thing. Data will sit alongside money as a way of achieving better outcomes."
But he also strikes a note of caution, arguing that data is not a replacement for strategy. "Data is a tool for decision-making; it does not make decisions for us," he says. "We have to be realistic about the sector's knowledge, skills and capacity. If we want to help the sector take advantage of the opportunities, we need to strengthen its capability to work with data."
The sector, Wilding says, needs to hear examples of small organisations using their data to improve their outcomes, because most examples come from the private sector. He cites Tesco's Clubcard initiative, which collects data on its customers' spending habits and uses it to inform the business.
Wilding says it is early days for the sector and big data, but adds that as more open data becomes available, the potential for charities to use it will grow. For example, the government has asked councils in England to publish information on spending of more than £500 in an open format online. The NCVO has started to look at how it can use this data to gain a greater understanding of how much councils are spending on the voluntary sector. In future, it hopes to use the local authority data alongside that of other funders, such as foundations, to see where they are spending money and where the need is greatest. Wilding says: "If you overlay data on need, it starts to build a more complex understanding of what the funding ecosystem looks like."
But Wilding recognises there are barriers that stop charities from using and analysing their data - most notably the lack of skills and resources. However, initiatives such as Datakind and the Open Knowledge Foundation, which runs courses to help people gain data skills, are increasingly helping charities get to grips with their data. And the Nominet Trust, which invests in technology schemes aimed at solving social problems, has developed Data Unity, an open-source web tool that lets organisations explore and visualise data and share it with others.
Wilding believes some charities are deterred by data projects because they are concerned about what the data might show. "Maybe they worry that they are not doing as well as others," he says. "Sometimes it is difficult to collect data about some outcomes - softer outcomes, for example."
He says it is important for the NCVO and other sector bodies to be clear about why the sector should use and be open about data and to provide examples of how this is already happening. "The worst thing would be to tell the sector that it must do open or big data," he says. "What we've got to say instead is 'you've got some problems: this is how data can help you to solve them'."
Case study: Keyfund
Keyfund is a Newcastle-based charity that helps groups of young people design, plan and deliver their own projects. It aims to develop 12 key skills, including problem-solving and negotiation.
The charity, which works with disadvantaged young people in the north east, started developing its data strategy about five years ago with the aim of using it to improve, rather than simply prove, its impact.
The charity collects data on up to 6,000 young people that use its programme each year, adding demographic data and measuring how they develop, using the young people's own assessment of the skills they acquire, where they can do better and how the process in general can be improved.
Hannah Underwood, chief executive of Keyfund, says the process has given the charity the confidence to innovate and make difficult decisions about its work.
Underwood says some funders were starting to question whether leisure projects, rather than enterprise or environmental ones, were the best use of their money.
But analysis of its data showed not only that young people gained just as much from leisure projects, but that these had an even more positive impact on young people from deprived areas than other schemes.
Underwood says the charity has also investigated whether there was any correlation between the gender of the person working with a group of young people and the group's success. It found that girls from an Asian background progressed better when supported by females.
"You've got to be brave enough to ask difficult questions about yourself and about your organisation," she says.
Underwood is leading the charity's data analysis work. "If charities want to take this process seriously, then it needs to be driven from the top," she says.
Case study: Macmillan Cancer Support
The cancer charity has been analysing NHS data sets, including cancer registration and mortality, to create a desktop tool for commissioners to plan cancer care in advance in any area of the UK.
The model developed by the charity's team of data scientists predicts how many people will be diagnosed with cancer, what types they will have, how many will die and what their needs will be in one to three years' time. It has been under test since December and will be launched next month.
Mike Hobday, director of policy and research, says it will enable commissioners to gain a rapid understanding of need in their areas and allow them to plan services for their populations, rather than provide for whoever turns up.
"Charities can be entrepreneurial in terms of saying where significant interventions can be made using big and open data because we are not caught up in the daily hurly-burly of the NHS," he says.
Hobday says Care.data, the NHS data-sharing scheme, would be "massively beneficial" to Macmillan's data work because cancer patients are increasingly treated in primary care. But the scheme to extract patient data stored in GP surgeries and place it in a central set of databases has been delayed until the autumn because of resistance to it. "We've not been very good at talking about the sorts of insights that can come out of it and how it brings benefits to patients," he says.
Macmillan started its work on big and open data about four years ago. Working in partnership with the NHS in Manchester, it explored why some cancer patients were more expensive to treat than others, despite having the same level of illness and outcomes. Hobday says it came down to the quality of support: it was nurses who were helping to get patients home quicker.