And community development finance institutions now have nearly £400m available to lend, up from £220m in 2003.
The CDFA said the growth indicated that the sector had "established itself as a credible force for social change".
CDFA chief executive Bernie Morgan said: "The results from the report prove that the investment from the Government, banks and others is really paying off."
However, the association reiterated its warning, first made a year ago, that the proposed scaling back of the DTI's Phoenix Fund in April 2006 would seriously affect the sector's viability.
The Phoenix Fund has put more than £40m into CDFIs since its launch in 2000. It was the largest contributor to covering CDFIs' operating costs in 2003/04 and the second largest provider, after banks, of capital for onward lending and investing.
The Government has indicated that the mantle for supporting the sector should be taken up by regional development agencies and the devolved administration in Wales. But according to the report's author, CDFA deputy chief executive Sarah McGeehan, progress on this is frustratingly slow.
"Regional support is growing and some RDAs have emerged as strong partners of the sector," said McGeehan. "But we are worried there will be casualties among sound organisations in some areas because they are at the mercy of changing regional policies. Access to continuing funds remains the top priority for our members."
The report - called Inside Out - The State of Community Investment Today - said that CDFIs expected their funding sources to shift over the next three to five years, with a reduction in reliance on grants and more generated income from investment portfolios, support and training.
Most CDFIs tend to specialise in one market and, in a shift from 2003, 61 per cent are now serving micro-enterprises. Overall, CDFIs are more likely to provide loans for start-up businesses, while social enterprise lenders are more likely to provide finance for established businesses.
CDFIs have financed more than 9,500 businesses, individuals and initiatives, such as the East Anglian schools roadshow MiniMonsters, that are unable or unwilling to access finance from commercial sources. During 2004, they wrote loans totalling nearly £18m, almost double 2003's figure.
The loans are for purposes such as the purchase and renovation of buildings, working capital, equipment purchase and gap financing. Individuals can get personal loans for emergencies, home improvements, debt consolidation, meeting the costs of returning to work and buying domestic equipment.
- CDFIs now provide £147m in loans and investments
- The value of loans written in 2004 almost doubled, to nearly £18m
- 6,523 loans have been made to date
- CDFIs have nearly £400m available to lend - up 80 per cent on 2003
- The average loan to micro-enterprises is £5,200, to small businesses £28,000, to social enterprises £50,000 and to individuals £500.