"I really think you have an image problem and a name problem," he said. "When the public hears the words 'voluntary sector' they think of people doing coffee mornings."
He added that while he didn't in any way denigrate the worth of volunteers' efforts, the sector needed to find a way of differentiating between that image and "the work that huge organisations perform to deliver public services".
Milburn, former Health Secretary and an architect of Labour's next election manifesto, used the conference to outline his vision for the future relationship between the voluntary sector and the Labour government.
He envisaged more voluntary sector provision of public services, and better consultation with the people those services are there to help.
To achieve these, he proposed four key changes.
Firstly, volunteering should become part of the secondary school curriculum, with all teenage pupils required to complete a period of "national community service".
Additionally, new legislation might be needed to boost local control of services. The legal model that paved the way for NHS Foundation Hospitals should be adapted for leisure services, schools and colleges, he argued.
Thirdly, the sector needs a level playing field with the public and private sectors, and he expressed support for longer term contracts, a voluntary finance initiative, and an automatic right for the sector to bid for public contracts.
"The government could go further still. The sector needs a seat at the planning table - in local authorities, strategic health authorities and in local strategic partnerships," Milburn said.
But in order to ensure it gets this recognition, he concluded, the sector "needs to get its house in order in terms of how it is organised and how it is governed", as a bigger role in public service delivery demands greater accountability.
"The last thing the voluntary sector needs is an Enron-style scandal just at the point when it is becoming such a key part of service delivery in this country," he said.