Addressing charity chief executives and chairs at a breakfast meeting last week, Clark said he was surprised at the sector's passivity over the second Olympics raid on the National Lottery, which will cost arts and heritage charities £100m, according to some estimates.
He also criticised charity leaders for their relative silence on the effect that Gordon Brown's cut in the basic rate of income tax will have on Gift Aid.
The cut, which was announced in this year's Budget, could lose charities as much as £70m.
"In any other sector there would have been an outcry," said Clark. "Sometimes I think you give politicians an easy ride."
Clark said the behaviour of the sector was all the more startling given its strong association with campaigning. "I'm slightly surprised by what is a campaigning sector," he said. "You don't get anywhere by being polite."
Kevin Curley, chief executive of Navca, the umbrella body for local voluntary sector bodies, said: "I think it's a fair criticism."
Navca was one of several third sector groups, including the NCVO, to give an immediate welcome to culture secretary Tessa Jowell's announcement in March on how the Government intended to meet the rising costs of the Olympics.
But he added: "Some local authorities can be very punitive. The price of speaking out could be exclusion from a seat at the policy table or even a reduction or removal of funding."
Helen Donohoe, head of campaigning at the NCVO, said: "I take his point." But she added that the sector campaigned in different ways, some of which weren't visible to the public eye.
Debra Allcock Tyler, chief executive of the Directory of Social Change, said: "I agree with him that charities don't shout enough. The reason they don't is that there are lots of instances in which they do make a noise, but then lose funding. They're afraid to bite the hand that feeds them."