Monitoring performance: Give us the tools and we'll measure the outcomes

Charities often struggle to measure the effect their work has and the progress made by vulnerable clients. But there is a range of tools that they can use to assess and record the impact of their interventions, providing them with evidence that can be used to persuade funders that the work is achieving its aims. Andy Ricketts reports.

Homelessness charity Thames Reach seen here at work on the streets. (Credit: Michael Walter/Troika)
Homelessness charity Thames Reach seen here at work on the streets. (Credit: Michael Walter/Troika)

It's a familiar problem. Funders want hard facts, but charities often work with vulnerable people with chaotic lives, and measuring the success of charitable projects can be fraught with problems.

But a new type of management information tool could provide effective ways of measuring 'soft outcomes' - the kind of progress that is hard to pin down, but is progress nonetheless.

"Back in 2001, few people could describe what an outcome was," says Kevin Ireland, executive director of the London Housing Foundation. "And there was certainly no way of measuring outcomes in the homelessness sector."

The foundation, which supports voluntary agencies tackling single homelessness in London, set to work on the problem with a pair of consultants who would later form Triangle Consulting.

The two organisations came up with the Outcomes Star. It enables key workers to assess periodically the position of their clients on a sliding scale of one to 10 in 10 key areas, such as managing money and social relationships. The total can be compared periodically to measure progress.

"The real difficulty in the social work field is finding ways of showing that you have achieved your aims," says Ireland. "Working with people with particular vulnerabilities, you often find they do not come out with a shiny flat or a new job, but you have got some way down the line. Often the most valuable work is moving people along and maybe helping them to realise that they need help.

"We were saying to the agencies that if you do not get your act together, the funders and commissioners might come along and say 'this is what we want you to produce' unless you say 'this is what we do, and here's the proof'.

"The star gives practical steps that the client can understand. It gives them more control and allows the journey to be shared."

The tool charts steps to self-improvement, and leaves aside goals such as a job and a flat, which for many clients at certain points are little more than a pipe dream.

The London Housing Foundation is not the only organisation attempting to crack this particular nut. In Norfolk, voluntary organisations led by umbrella group Voluntary Norfolk went to the research centre at City College Norwich with the aim of coming up with a way of measuring these 'soft outcomes' - the hard-to-pin-down behavioural variables that contribute to a person's overall wellbeing.

With the help of a grant from the Big Lottery Fund for two-and-a-half years from September 2003, the organisations came up with the 'Soul Record' - short for Soft Outcomes Universal Learning.

Soul Record has typically been used more broadly than the Outcomes Star and its owners say it can be adapted to work with more or less any type of beneficiary.

The tool is a questionnaire that asks people to rate themselves on a scale between one and six on a list of statements such as "I am a confident person".

Clayton Anderson, Soul Record project manager at the research centre, says the tool is valuable in assessing the progress made by clients.

"People can always assess the state of clients when they come through the door, but they cannot measure how people feel about their progress along the way," he says.

Organisations wanting to use Soul Record send staff on a training course to learn how to use it effectively. It costs £150 per person.

According to Anderson, Soul Record should be used at least twice in the course of an activity to establish comparison points, whether the programme lasts one week or a year. Experience shows that people will mark themselves more harshly the second time they use it.

Anderson says: "Best practice is to have at least three attempts over the course of an activity. The scores can drop and people will often mark themselves lower at the second attempt."

Individual organisations can work with the centre to modify the questionnaire to fit their project, says Anderson. For example, a version that was applied with people with autism used weather symbols rather than numbers to describe feelings.

Projects in East Anglia have used the tool to back up applications for funding. "People have acquired extra funding as a result of using it," he says.

Anderson recognises that measuring soft outcomes has its limitations. "It is fair to say it is not a scientific thing and won't be 100 per cent accurate," he says. "It measures how people feel about their situations." But he rejects the argument that variations between client groups mean it is not useful. "Soul Record offers accurate results in the sense that this is how people are feeling at the time," he says.

The research charity New Philanthropy Capital began work in January 2007 on developing a way to measure wellbeing among young people. The tool is still under development and will not be ready for general use until 2009, but has already proved a useful way of measuring outcomes, according to Lucy Heady, quantitative analyst at NPC.

"The motivation was to develop a tool that could capture the outcomes that all charities working with children want to achieve," she says. "Capturing exam results or truancy rates is one thing, but we wanted to capture the impact of an intervention.

"We also wanted to compare charities to see where certain measures have had an impact. A charity working in schools might have an impact on school-related outcomes, but it would be interesting to see how that relates to the rest of children's lives - in their mental health, for example."

The tool uses 40 questions to measure wellbeing in 10 core areas, such as physical and psychological health, behaviour and family. The number of questions could rise to about 50, says Heady.

NPC's model will be free for organisations to use if they agree to submit the data they collect to NPC and sign up to a code of conduct about how they use it.

Measuring soft outcomes has made some organisations uneasy. Ireland admits that some people are worried about using a system that compares degrees of success between different types of charities working across a range of client groups.

"There is a certain concern among some organisations about the implications of this," he says. "One organisation recently said it feared that if all of us used the Outcomes Star we would lose what is special about our organisations. We would all lose our individuality."

Other concerns have also been expressed about using results in league tables, such as those used to compare schools. But no matter where organisations stand, it seems inevitable that more tools of this kind will be produced and put into use.

CASE STUDY - Victim Support

Victim Support began using Soul Record with clients in 2005. The charity, which provides support to people affected by crime, initially used the system to measure its success in its work with young people.

"We work with victims and witnesses of crime and we needed a way to assess that," says Brian Butcher, the charity's east of England regional manager. "You have got to be able to prove the success of what you do."

He says the charity has cited the fact that it is using the tool in funding applications and seen a positive response.

"We have made bids to run training days for the likes of the local criminal justice board, and in those we have said that we will use Soul Record to prove that we have achieved the learning outcomes it is looking for," says Butcher.

The charity has also found it was a useful way of measuring the success of courses it ran for staff and volunteers.

Using the system has helped change the way the charity engages with its clients and made working methods more evidence-based, says Butcher.

"It has been really useful in opening up a dialogue with the young people."

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in

Before commenting please read our rules for commenting on articles.

If you see a comment you find offensive, you can flag it as inappropriate. In the top right-hand corner of an individual comment, you will see 'flag as inappropriate'. Clicking this prompts us to review the comment. For further information see our rules for commenting on articles.

comments powered by Disqus