The statistics, which are based on the largest multi-year study of giving behaviour in the UK, reveal a "worrying" downwards trend, according to CAF.
UK Giving 2019 says the proportion of the British public who either gave money to charity directly or sponsored a friend or family member fell from 69 per cent in 2016 to 65 per cent in 2018.
The proportion who gave money directly to charity fell to 57 per cent from 60 per cent in 2017 and 61 per cent in 2016.
"With three years’ worth of data, we can now see a clear trend in people’s charitable giving and it is headed in a worrying direction," said Susan Pinkney, head of research at CAF.
But although fewer people report donating money, those who do give are giving more.
The median monthly amount given in 2018 was £20 and the mean amount was £45, which compares with £18 and £40 in 2016.
As a result, the overall amount given was £10.1bn in 2018, compared with £10.3bn in 2017 and £9.7bn in 2016.
The report also reveals that trust in charities has fallen below 50 per cent.
Just 48 per cent believe charities to be trustworthy, compared with 51 per cent in last year's report and 50 per cent in the previous one.
Twenty-one per cent do not believe charities are trustworthy, an increase from 19 per cent on the figures for 2016 and 2017.
"If people lack trust, that means they worry that their hard-earned money is not being well spent when donated to charities," said Pinkney.
"This is a challenge that the entire charity sector needs to tackle head on, finding ways to inspire people to give and demonstrating to them that their money is making a difference."
John Low, chief executive of CAF, said in his foreword to the report that the key measures of giving were yet again on a downward path.
He added: "It is incumbent on all of us to make the case for the work of charities in all their forms."
A spokesman for the Institute of Fundraising said it was "a challenging external environment for fundraising".
"Over the past year or so charities have done a huge amount of work to review their strategies, invest in new approaches,and respond to changes in data protection legislation, which might have meant charities have had to pause some of their fundraising activity," the spokesman said.
"The link between giving and asking is absolutely crucial. The research suggests that people are less likely to be approached for money, so it is not surprising that we see a correlation with a fall in asking to a reported drop in giving."
Sir Stuart Etherington, chief executive of the National Council for Voluntary Organisations, said the report vindicated a change in many fundraising strategies that would pay off eventually.
"Fewer people giving larger sums reflects a year in which many charities were scaling back recruitment of new donors and focusing instead on cultivating their relationships with existing supporters," said Etherington.
"This is the result of a positive change in charities' fundraising strategies away from some of the approaches of the past.
"You see this reflected in fewer people saying they were approached for donations or seeing a charity appeal on television or on their doormats.
"While it means less income growth in the short term, charities believe having a core of more engaged and satisfied donors will lead to stronger support in the long term."
The report also says that the number of people taking part in charitable activity or social action, such as signing a petition, in the previous four weeks has fallen for the third year running, from a peak of 68 per cent in 2016 to 64 per cent.
Men are twice as likely as women to say that they rarely or never give to charity (26 per cent against 13 per cent).
And rates of volunteering have remained stable, as have rates of people donating goods to charitable causes such as charity shops.
The UK Giving 2019 findings are based on monthly interviews and include a yearly total of more than 12,000 individual interviews.
The report is available here.
The findings of the report confirm the results of Third Sector’s analysis of the most recent accounts of the top-10 fundraising charities, which showed that although overall donations had grown in the past three years on average, legacy growth had masked the fact that other forms of donation were stalling.