Olympics minister Tessa Jowell has met sector groups associated with the four lottery distributors but has so far declined to give this promise.
"We are all worried that there may be more calls on lottery money, not just between now and 2012, but also after 2012," said Robin Simpson, chief executive of the Voluntary Arts Network.
"The chances of costs rising are high. Ministers have said there are no plans to take extra lottery money, but that's not the same as saying no more will be taken."
The network is also pressing Jowell to provide a specific pot of money to support the Cultural Olympiad, intended as a formal recognition of the importance of the voluntary sector in the arts, and another for voluntary arts groups in Scotland, Wales and Northern Ireland, which have suffered most from the lottery diversion.
Pete Moorey, parliamentary and media manager at umbrella body the NCVO, said the Government should make a statement in the Commons that there will be no further diversion at any time.
"We cannot be in a position where the Government keeps coming back to the lottery if there are further overspends," he said. "If it does, we will fight it every step of the way."
Jowell's negotiations with the sector have been held in the run-up to the Government's formal request to Parliament for approval for the £1.4bn that is to be transferred from lottery distributors to the Olympics.
The debate was meant to take place last week but now may not be heard until the new year.
Two weeks ago, a House of Lords committee on Olympic funding reported that the Department for Culture, Media and Sport had not provided any alternative funding options in the event that costs overrun.
It also reported a DCMS statement that there was "no prospect" of further funding from the Government or the Mayor of London. The DCMS was also quoted as saying that, because it was not possible to describe theoretical benefits from future lottery applications that have not yet been submitted, its impact assessment expected the loss of funds to arts, sports and heritage charities to have "no differential impact".
The department added that diverting funds from lottery distributors to the Olympics represented a transfer "from one good cause to another".
How Olympics cash will be spent
- Preparation of the site, including demolition and the construction of bridges, highways and other infrastructure
- Construction work, including the Olympic Stadium and warm-up track, pools and other venues and circuits
- Logistics: the organisation and movement of workers, equipment and materials on and off the site
- Transport: contributions to the cost of constructing new infrastructure and transport operations
- Village: infrastructure and other costs needed to build the Olympic Village
- Olympic Delivery Authority management costs, including the costs of operating the authority and its delivery partner.