David Harker, chief executive of the policy and support body for Citizens Advice bureaux, said his members were faced with a "byzantine level of bureaucracy" which represented "a clear and present danger to many bureaux".
More than 240 local bureaux, which give advice on issues such as welfare and debt, have contracts with the government to run the community legal service, the successor to Legal Aid, which are worth £25m.
But they are reeling from a new auditing regime that claws back grants if bureaux are judged not to have met the target of 1,100 hours of work with eligible clients.
Harker said the level of bureaucracy could force many bureaux to pull out of the community legal service, one of the main planks of the Government's attempts to combat social exclusion. A declining number of law centres and solicitors in private practice participate in the reformed service.
"My fear is that CABs and other advice agencies may do what private solicitors have done and walk away, leaving large areas of the country where people can't get access to legal advice," he said.
Under the system introduced in April, advice workers have just 15 minutes to write a letter and cannot count the time spent opening mail towards their target hours. Home visits to clients may be retrospectively disqualified.
One bureau in Bradford has been told it has only fulfilled 75 per cent of its contract in the first half of the year, and so its grant for the rest of the year has been halved.
A further five bureaux in the North of England have also been penalised.
"This is absolutely devastating for a small voluntary organisation," said Harker It is money that has already been spent. Is this an example of the Government's commitment to working with the voluntary sector?"
Harker discussed the issue with Lord Chancellor Lord Falconer, whose Department for Constitutional Affairs is responsible for the community legal service, during last week's Labour Party conference, but there has been no commitment to change the regime.
A spokesman for Advice UK, which represents 1,000 independent advice centres, many of whom carry out contracts for the community legal service, said: "Many members are seriously considering handing them back. The system is not functioning well."
A Department spokeswoman said the auditing system was new and voluntary organisations were not used to them. "But we need to get value for money," she said.