The research found that charities with annual incomes below £5m lost £1 in VAT for every £50 of income, while those above the £30m mark only lost £1 in every £80.
Tony Jaras, senior VAT manager with Deloitte, said: "The disparity must to some extent arise from the fact that smaller charities lack the resources to manage their VAT burden as effectively as larger ones.
"Our survey suggests that many charities just can't devote enough staff time to this, or find the money for professional advice."
The poll revealed that irrecoverable VAT cost the 200 charities that took part in the survey more than £60m a year.
This compares with the Charities' Tax Reform Group estimate of £500m for the charity sector's total irrecoverable VAT burden. But the Deloitte survey includes VAT relating to exempt business supplies and non-business charity activities, whereas the CTRG figure takes into account the latter only.
Fifteen of the organisations that took part in the survey lose more than £1m a year in VAT, but the average is less than £100,000.
Twenty-four charities recover no VAT at all, 40 recover up to £50,000 and 23 recover more than £500,000 a year.
One respondent to the survey said: "People would go ballistic if they knew how much of their donations went to the Treasury."
Another said there was an ethical dimension to the question of a tax system that treated charities as the final consumer for VAT purposes but designated arms manufacturers and tobacco companies 'tax neutral'.
The survey also found that a significant number of respondents did not devote sufficient resources to ensuring full compliance with VAT law, and have incurred "assessment, warning or penalties" as a result.
Helen Donoghue, director of the CTRG, said: "The survey does not give sufficient information to make any revised assessment of the total VAT bill of the sector. The best current estimate is that the sector pays between £400m and £500m a year in VAT.
"Although the impression given by the survey is that smaller charities are hit harder than larger ones, it is the service-providing charities - both large and small - that are seriously hit by VAT. Even grant-making trusts are affected because they are, in effect, funding other charities' tax bills."