The sum will deal a substantial blow to arts and heritage charities in particular. They already face a loss of up to £100m over the next four years because of the Government's second raid on lottery distributors in March.
The impact of new games such as Lucky Dog scratchcards, which were launched specifically to raise money for Olympics projects, was revealed during an Olympics debate in the House of Lords last week.
Lord Evans of Temple Guiting, the Government's spokesman for culture, media and sport, said: "Non-Olympics good causes may lose on average about 5 per cent of their income as a result of sales diversion."
In April, the National Lottery announced it had raised £20bn of good cause money since it started in 1994. This amounts to an average of £1.54bn a year - 5 per cent of which is £77m.
Lord Evans added: "When the Olympic Park is sold, money will come back to the lottery." The Government has promised to release details on the return of profits to lottery distributors soon.
Conservative peer Lord Kenneth Baker called the second Olympics raid "an act of larceny".
He pressed the Government to give similar tax breaks to those enjoyed by donors in the US to private sector philanthropists who are donating money to bridge the funding shortfall for good causes.
"Every $60 given to a cultural organisation in America is worth $100," said Baker.