Doctors writing in the British Medical Journal this month said: "Elderly people should not be stigmatised as being impaired or inevitably disabled".
The UK is the only country to use the same symbol for older people and those with disabilities or visual impairments.
The road sign, depicting a man with a stick leading a woman with a hunched back, was designed more than 20 years ago in a children's competition.
Charities for older people have labelled the sign "offensive", and say it establishes and reinforces a negative stereotype of older people as frail and vulnerable.
Help the Aged said it has been campaigning to change the sign for several years after receiving complaints from older people's groups, and met with the Department of Transport in 2002 to discuss replacing the symbol. "We really need to move away from this image of older people as hunched, incapacitated figures," said a spokeswoman.
Age Concern said it had also received letters and phone calls from people who found the sign offensive. "Many older people are fit and healthy and we would like the sign to be different."
The charities have called for the sign to be replaced with an internationally-recognised road sign alerting drivers to the presence of vulnerable pedestrians.