Failed asylum seekers who cannot go home because they are medically unable to fly, or because their home country is too dangerous, would have to carry out community service in return for the "hard-case support" they receive.
As the amendment received House of Lords approval on 15 June, Lord Rooker suggested that the Government would look to the voluntary and public sectors to implement the plan.
But at least four charities have baulked at the idea.
"There is a world of difference between people who are forced to do community service and those who do so voluntarily," said Jason Tanner, spokesman at CSV. "We support volunteering, not people being treated like criminals when they aren't criminals."
Tauhid Pasha, a director at the Joint Council for the Welfare of Immigrants, said: "To force such people to work for the support they receive is humiliating and potentially unworkable."
Deborah Singer, co-ordinator of the Refugee Women's Resource Project at Asylum Aid, said: "We wouldn't consider offering to organise these proposals."
A Refugee Council spokesman said: "We would rather the Government didn't implement this plan."
Only one organisation offered the Government any support. Steve Rylance, communications manager at Refugee Action, said: "We wouldn't rule out assistance, because our involvement might mitigate some of the harsher aspects of the legislation."
The bill will be subject to a Commons vote in July or September.