Impact measurement in the third sector is poorly understood, too expensive for smaller organisations and too focused on the needs of funders, according to a report by the Third Sector Research Centre published today.
The report, Impact Measurement Practice in the UK Third Sector: a review of emerging evidence, says that many organisations lack the skills to use impact measurement effectively, that its practice is inconsistent across different sizes and sectors of charity, and that organisations have "uneven access" to support and guidance.
The report was written by Jenny Harlock, a research fellow for the TSRC at the University of Birmingham. It is based on a review of available research into impact measurement.
"Impact measurement-related activities have increased across the sector in the UK in recent years; however, there are inconsistencies in levels of practice and understanding among third sector organisations" the report says. "Practice appears to be concentrated among larger, higher capacity organisations, with smaller organisations undertaking impact assessments in more limited ways."
It says that "funders and commissioners’ demands for impact data" appear to be the main motivation for many organisations to carry out impact measurement.
"There are growing concerns that funders’ and commissioners’ requirements are shaping and dominating approaches to impact measurement in the third sector over the needs of service users, beneficiaries and organisations themselves," the report says.
It says the review has "raised questions over the quality and robustness of impact measurement practice". There are question marks, it says, over impact measurement tools, staff skills and training, the resources available and review processes.
It also expresses concern about a lack of comparability between sectors and "selective presentation of results by organisations".
The report highlights problems in the research it has used as the basis for its conclusions. It says much of the existing research is "boosterist" – designed to encourage impact measurement rather than measure its effectiveness. It also says that the concept of impact measurement itself is not clearly defined and that different academic papers on the subject give different definitions.
The report says much of the evidence reviewed was not "robustly designed research projects in good quality, peer-reviewed journals" but was instead "pieces from the trade press, policy documents or pieces published by particular bodies with an interest in this area".