The 'charity choice' idea was the most controversial of 16 proposals the party put forward in Sixty million citizens, its consultation document on the voluntary sector launched by leader Iain Duncan Smith.
The consultation process drew to a close at the end of August, and the early indications are that the lottery suggestion could alienate many in the voluntary sector and cost the party votes.
Its critics include NCVO, which says lottery money should be distributed according to need rather than popularity. "We are not in favour of proposals to give players a greater say over which causes should benefit," said Belinda Pratten, a policy officer in charity law at NCVO.
"Lottery funding aims to help those at greatest disadvantage in society and to improve the quality of life in the community."
She said the Tories were in danger of "over-egging" the impact of the lottery on charities when they should be placing more emphasis on Gift Aid. "Care should be taken not to promote playing the National Lottery as an effective way of giving to charity," she said.
Pratten added that the document "reflects a strong commitment and interest in the voluntary sector, but it needs to be put in a wider policy context".
The issue of lottery money distribution drew a mixed reaction on the online part of the consultation, which the Tories ran for the first time.
One contributor suggested it would lead to all the money going to large charities, such as the NSPCC and RSPCA, that had the funds to produce emotive appeals.
The Conservatives' policy analyst Peter Franklin said nothing in the document was a done deal. "We wouldn't want to enact anything that large sections of the sector would not support," he said.
But reforming the lottery retains support in the party ranks. "There is a feeling that lottery players should be trusted to decide how to spend their money," said Franklin. "I think people would be surprised at the support the less glamorous causes attracted."
He added that issues like asylum seekers should not be left to lottery boards. "If they are so important, they should be funded through central government rather than by short-term grants that are in the hands of unaccountable committees," he said.
Embarrassingly, the internet feedback generated only 19 responses. "If we do something like this in future we will do a lot more to promote it," said Franklin.
The party is expected to finalise its voluntary sector manifesto over the next year.