The Joint Council for the Welfare of Immigrants hopes charities can scupper Brown's intentions, revealed in a speech on Britishness last week, by refusing to take volunteers.
It would like to see a repetition of the united front the sector put up against the 2005 Immigration and Asylum Regulations, which contained a requirement for failed asylum seekers unable to return to their countries of origin to do community work.
"The sector has refused to cooperate with those proposals," a JCWI spokeswoman said. "Because it's used as a non-custodial sentence for criminal offences, community service has very negative connotations."
But the council's proposal was undermined last week when volunteering charity CSV issued a statement expressing support for Brown's idea.
Dame Elisabeth Hoodless, executive director of CSV, said: "We welcome the recognition that participation in the community helps immigrants integrate into society."
The JCWI spokeswoman expressed disappointment with CSV. She said: "We're very concerned to hear CSV coming out with that."
A spokesman for CSV defended its position, saying it would be interested only in taking on volunteers on a non-compulsory basis.
In his speech, Brown argued that community work would expose immigrants to broader aspects of British society. He said: "It's right to consider asking men and women seeking citizenship to undertake community work that introduces them to a wider range of institutions and people."