The Department for Culture, Media and Sport wrote to members of Navca in response to its campaign to prevent lottery money being diverted to the games.
The letter says: "From time to time, the Government sets the priorities among the good causes. We have in the past allocated a share of lottery money for projects like Tate Modern that ultimately led to major national investment."
Recipients are angry at how the letter asserts ministers' control over the lottery and fear it proves the Government has abandoned the additionality principle - the idea that money raised by the lottery should not go to anything normally funded by government.
Moira Cunningham, chief executive of Compact-support body Tameside Third Sector Coalition, said: "This is saying that the Government doesn't care what we think because it has the power to do as it likes with lottery funds."
Bill Albert, chief executive of the Norfolk Coalition of Disabled People, said: "This reply is insulting, because the Government is saying it will do what it wants." Kevin Curley, chief executive of Navca, described the response as "cynical and dismissive".
A spokesman for the DCMS said the Government aimed to minimise the impact on community groups of using lottery money to fund the Olympics. He added that an announcement on the games would be made by the end of next month.
Last week, Labour MP Lindsay Hoyle registered the second early day motion calling on the Government to protect community groups by not using lottery money to pay for the Olympics. It was supported by 20 MPs.
- See Letters, page 17.