Brooks Newmark has resigned as Minister for Civil Society over allegations due to be published in a Sunday newspaper.
Newmark, who was only appointed to the role in July, has been replaced by Rob Wilson, the Conservative MP for Reading East.
A spokesman for Number 10 declined to comment on the nature of the allegations, which are due to appear in the Sunday Mirror newspaper tomorrow.
A statement from Number 10 said: "The Prime Minister has accepted Brooks Newmark's resignation from his role as Minister for Civil Society. He will be replaced by Rob Wilson MP."
The statement from Newmark said: "I have decided to resign as Minister for Civil Society having been notified of a story to be published in a Sunday newspaper.
"I would like to appeal for the privacy of my family to be respected at this time. I remain a loyal supporter of this government as its long term economic plan continues to deliver for the British people."
Newmark's short spell in office will be remembered for his comments that charities should "stick to their knitting" and stay out of party politics.
Wilson, who is 49, has been MP for Reading East since the 2005 general election, and has held junior roles including parliamentary private secretary to Chancellor George Osborne between October 2013 and July this year.
He was shadow minister for higher education between 2007 and 2009, when he was appointed as a whip with responsibility for transport and health. This will be his first ministerial role.
He complained to the Charity Commission in January about the Family and Childcare Trust after it used hashtags on Twitter that were being promoted by the Labour Party. The commission subsequently cleared the charity.
Wilson has written books including 5 Days to Power: The Journey to Coalition Britain, which is about the formation of the coalition government and was published in 2010, and The Eye of the Storm: The View from the Centre of the Political Scandal, which was published in July.