Cameron held talks with CSV, the Prince's Trust and the Duke of Edinburgh's Award to discuss his plans.
But Andy Forster, head of policy at Volunteering England, which wasn't invited to the talks, said: "Volunteering is freely given. By introducing a compulsory element, it is no longer volunteering. It does not match the Compact definition of what volunteering means.
"We do not have a problem with, for example, the idea of compulsory citizens' service. But you do need to be careful when you start using words interchangeably, and in this context volunteering is the wrong word."
Forster, who is also a trustee of the Russell Commission, added that the Conservative leader's proposals lacked originality. "There's not really anything new in what he's saying," he said.
However, Dame Elisabeth Hoodless, chief executive of CSV, praised Cameron's idea.
"It's a very exciting vision," she said. " There are a number of aspects to his proposals that are distinct from those made by the Russell Commission.
"Cameron is very keen that young people should have the opportunity to meet people they might not normally come into contact with. He also wants to use volunteering to promote social cohesion."
Hoodless said CSV's own research showed that 86 per cent of young people would volunteer if their friends did so too.