Culture and leisure trusts are worried that local authorities could see them as soft targets for spending cuts.
Many of the charities were set up by councils and still rely heavily on them for funding.
Craig McAteer, chair of the Sports and Recreation Trust Association, known as Sporta, which represents more than 100 culture and leisure trusts with a combined turnover of £750m, said: "We are concerned that discretionary services such as leisure and community centres might be seen as soft targets by hard-pressed councils.
"Any closures or cutbacks in service would be a huge loss to communities around the UK."
Brian Leonard, the newly appointed chief executive of Sporta, said he was not convinced the Cabinet Office regarded the trusts as part of the third sector and that their role in establishing the big society was in danger of being overlooked.
He said many trusts were based in working class areas and offered alternatives to more expensive private health clubs.
"It is important that people talk to the trusts because they are community facilities open seven days a week, but they tend to be ignored," said Leonard.