It is important to ensure that you have a good relationship with your web developers. If your agency has supplied you with a good content management system, there is no reason why you shouldn't use it for many years, making it a sound long-term investment. Some charities have already had the same agency and CMS for seven years or more, making the original spend very worthwhile.
But even an organisation that has used the same CMS for years is going to want to add to the site and develop new functionality to meet the changes in the organisation and the way the web is used. You should ask your developers to refresh your design every three years or so.
Once an agency has set up a site for you, you should be able to gauge that agency's strengths and weaknesses. It might have great designers but less strong coders; it might be great at customer service but slow on delivery.
Whatever the case, you need to work with the agency to make life easier. If you require something that is beyond its abilities, ask if there is someone else who could help. If it is slow at turning things round, make sure you set fixed deadlines and ample lead time.
Most of all, don't push the agency beyond its tolerance when trying to get free or low-cost work done. Most agencies are made up of nice people who will try to help charities, but they have to earn a living too.
The biggest failure rate comes from expecting too much for too little. You will get away with it for a while, but in the end it will sour the relationship and waste the investment. You will end up starting from scratch with a new agency and a new CMS rather than making the most of the one you have.