The calculations, based on research by the Liberal Democrats, show that third sector groups that receive grants from Arts Council England, the Heritage Lottery Fund and Sport England will lose a total of £332.6m of funding to the 2012 Olympics.
Heritage charities will fare worst with a loss of £116m, while sports charities will lose £112m and arts charities £105m.
Anthea Case, chair of Heritage Link, said: "We have asked the Government to publish more detail on Olympics-related ticket sales, and it's clear that these have much more impact than the Government acknowledges."
Of the £750m to be raised by special Olympics lottery games, it was initially expected that 59 per cent would be cannibalised from normal lottery sales as players switched over. However, the estimated cannibalisation has now risen to 77 per cent. For the Heritage Lottery Fund, this rise of 18 percentage points represents £22.5m - more than the cost of its Heritage Initiative, which has helped 1,400 UK communities care for local landmarks.
"The Government must make a clear commitment that no more lottery funds will be diverted to the 2012 Olympics, either directly or through these more devious means," Case said.
Robin Simpson, chief executive of the Voluntary Arts Network, said small voluntary groups would suffer most. "ACE achieved a good settlement in the Comprehensive Spending Review, but the only funding route most small groups have is through open-access lottery funds," he said.
An ACE spokesman said the Olympics were a fantastic opportunity for UK arts, including the voluntary and community sector. "We hope they will play an active part in the Cultural Olympiad," he said.
A spokeswoman for Sport England confirmed that sales of the Olympic scratchcards were selling "ahead of projection" and therefore affecting lottery income earlier than expected.
"We will have to be innovative if we want to realise our ambition of delivering a world-class community sport system and increasing sports participation," she said.
Derek Mapp, chief executive of Sport England, resigned last week after a row with James Purnell, the DCMS minister, over funding priorities. Mapp criticised the Government after the second raid on the lottery was announced in March.