John Armitt, chairman of the Olympic Delivery Authority, said last week that the Government would have to bail it out after costs continued to spiral.
Boris Johnson, the Mayor of London, has said taxpayers in the capital will not contribute "a penny more".
Voluntary organisations are already expected to lose £250m in lottery funding to London 2012.
"The Government has made a commitment that there will be no further raids on lottery funds to pay for the Olympics," said a spokeswoman for the NCVO. "We expect it to uphold that commitment.
"We will continue to police the Government's commitment to prevent any more funding diversions."