The telephone poll of 1,000 people by children's charities NCH and Chance UK also found that 17 per cent of people object to being vetted by the Criminal Records Bureau.
It found that 69 per cent of men do not volunteer to work with children at all, with nearly 68 per cent of that figure blaming lack of time.
NCH director of children's services Hugh Thornbury said 80 per cent of the charity's volunteers were women, but it was desperate to use Volunteers' Week to recruit more men to its mentoring services for young boys with behavioural problems.
He said: "In recent years there has rightly been a lot of publicity about the wrong men working with children, but this has made men in general feel more anxious about doing it."
He said NCH planned to work on improving its communications about what volunteering roles needed filling and why CRB checks were necessary.
He said "subtleties of language" were important. "Phrases like ‘working with vulnerable children' can be off-putting to men, whereas mentioning things like ‘challenging young people' appeals much more to them," he said.
Meanwhile, the Department of Health has marked Volunteers' Week with the announcement that 300 projects and 20,000 volunteers will this year benefit from its Opportunities for Volunteering scheme.
The scheme is an annual multi-million pound programme aimed at involving more volunteers in health and social care.
This year, the department has set aside £6.7m for a range of projects including an Age Concern advocacy service for frail older people in Buckinghamshire, and a project in Lewisham, south London involving young people and the crime prevention charity Nacro.
Speaking at an event yesterday to mark the 25th anniversary of the scheme, care services minister Ivan Lewis, said: "As a health minister, I see the huge difference that volunteers make in health and social care."
Girlguiding UK has publishing a survey of senior business figures highlighting how employer-supported volunteering schemes can benefit a company's image and the morale and skills of its staff.
Of the 500 senior managers interviewed, 87 per cent of those whose businesses have ESV schemes said the schemes promoted a positive image of their business. More than 80 per cent said they boosted staff morale and 94 per cent agreed that employees who volunteer bring a wider range of skills and experience to their jobs.
The survey also found that only 32 per cent of businesses have a formal ESV policy, but of those that do not have one, 58 per cent say they would consider having one if enough staff asked for it.
Girlguiding UK chief guide Liz Burnley said guiding volunteers received 150,000 working days' worth of training a year. She said: "This brings huge value to businesses, giving volunteers practical skills such as risk assessment, first aid and child protection, as well as helping to develop ‘soft skills' such as leadership, time management, team-building and diversity awareness."
Volunteers' Week also sees the launch of an expanded version of Volunteering England's website. The extra content includes information on legal issues, sport and public sector volunteering, as well as good practice case studies and documents such as interview guides, induction checklists and role descriptions.
The charity says additions make the website "the single most useful resource for anyone involved in volunteering."
Jane Heath, VE's assistant director of outreach services, said: "We aim to inspire those looking to volunteer, and to arm volunteering professionals with the necessary tools to carry out their jobs as effectively as possible."