A newly published independent study into local governance found that black and minority ethnic groups felt excluded from the Compact process, and that many did not even know of its existence.
The report, Contract or Trust, funded by the Joseph Rowntree Foundation, examined how the Compact was working in 12 areas around the UK.
The researchers found that the ways in which organisations, such as volunteer bureaux, rural councils for voluntary services and councils for voluntary services, worked tended to discourage the involvement of people from black and minority ethnic groups.
Professor Gary Craig, who co-led the research, said: "The institutional barriers existing within mainstream agencies look like institutional racism."
These barriers include modes of communication - such as town hall meetings and even the language used. Organisations should be more creative in finding ways to involve BME groups, said Craig.
"If you think that a formal committee meeting in the town hall is the only way of communicating with the voluntary sector then you're living in the dark ages.
"One council for voluntary services developed an email list for its members," he said. There were some examples of good practice in this area, but not enough, he added.
Angela Sarkis, strategic adviser at the Council for Ethnic Minority Voluntary Organisations, agreed, saying: "These umbrella bodies have been drawing down resources from central government and local authorities because they say they have a relationship with BME groups. The Compact process has blown this idea wide open. BME groups don't get any real representation from those organisations at all."
Many minority groups felt that they were being used as cheap service providers to help deliver government racial inclusion targets, without actually having any say in policy, said Craig.
"They feel that they are black service providers to black people, taking the heat off local authorities, which should have been providing these services," he said.
BME organisations were more likely to be hit by cuts than other organisations, the survey found.
Sue Wigley, director of National Association of Councils for Voluntary Service, said that the under-involvement of BME groups in Compact development was a priority for the organisation.
"However, there is much good work going on across the country, with many councils for voluntary service actively engaged in innovative practice," she said.
"Our role is to encourage this practice everywhere, and to that end we have established an Equalities Advisory Group to advise us on our work."
The research also found that many small groups were too anxious about losing funding to criticise local authorities, although the Compact is meant to protect the right to criticise.