Bubb warned that there would be "considerable anger" within the voluntary sector if money were siphoned away from good causes.
"It is simply not fair on thousands of voluntary and community organisations that their applications should miss out due to Government mistakes," he said.
Liberal Democrat peer Julia Neuberger has accused sector umbrella bodies of not being forceful enough in telling ministers to keep their hands off money earmarked for good causes.
In her Third Sector column this week, she says the sector should be shouting louder on the Olympic issue, and she questions whether it has been tough enough in its response.
Luke FitzHerbert, senior researcher at the Directory for Social Change, agreed. "The voluntary sector umbrella bodies have taken this raid very quietly," he said. "The NCVO could make it clear to the Government that it will find it hard to support all of its partnerships if it steals the sector's money.
"It has not made the connection very strongly. The Government is keen for us to deliver services on the cheap, but not keen on putting its resources behind helping us to be a strong and independent alternative to official Government policies."
An NCVO spokesman denied the organisation had campaigned weakly, saying it had been lobbying ministers and writing to the press on the threat the games posed to sector funding for three years. But he added: "It's hard to gain a huge amount of coverage on an issue that is still under discussion."
Trevor Watkins, head of trusts at Leonard Cheshire, said individual charities were frightened to speak up because it might jeopardise future lottery bids.
"The umbrella groups probably should have made more noise, but I rather suspect that the raid will happen anyway," he said. "Culture secretary Tessa Jowell will do what she wants."
A spokeswoman for the Department for Culture, Media and Sport said an announcement would be made in the new year on how the £900m games shortfall would be met.
- See Opinion, page 16.