One group likened the move to inviting the British National Party to speak at an anti-racist rally.
Christian Aid, Friends of the Earth and the Burma Campaign are concerned about the company's presence in Burma rather than the health risks of smoking.
"How any company that collaborates with a military regime can be interested in corporate social responsibility is a joke," said Craig Bennett, corporates campaigner at Friends of the Earth.
"This conference raises questions about whether the interest of these companies in this area is simply a way to stave off government regulation and counter accusations of mal-practice by campaigning groups."
The three organisations are all putting their names to a letter being sent to delegates at the conference on Thursday in London, which is organised by Ethical Corporation, a magazine which focuses on improving corporate ethical standards.
The event includes speakers from companies such as Shell, Premier Oil and Anglo American as well as organisations such as Amnesty International and WWF.
"These conferences can provide a platform to companies whose commitment to corporate social responsibility is frankly questionable," said Daniel Graymore, policy officer at Christian Aid.
Tobias Webb, editor and publisher of Ethical Corporation, hit back by labelling the charities as "naive and ill-informed".
"The conference is not presenting a platform for corporate greenwash," he said. "We've invited a speaker on to a panel discussion about how controversial companies are attempting to be more accountable, which is both relevant and appropriate."
"In no way are we saying or portraying these companies as being socially responsible," he said. "However, if all companies spent as much money as British American Tobacco investing in corporate social responsibility, then it would be a big step forward."
British American Tobacco declined to comment.