A group of local residents has asked the Charity Commission to reorganise the board of the Livesey Trust building in south London, which was shut down last year, so that Southwark Council will no longer be the sole trustee.
The Friends of the Livesey Museum, consisting of local residents, have written to the commission saying the trustee board should comprise between three and five councillors and an equal number of local residents.
"The appointment of the council as the sole trustee creates a conflict of interest between its statutory duties and its role as trustee," the letter says.
The group has campaigned since early last year to prevent the council selling the building, which previously housed the Livesey Museum for Children, and which councillors did not realise was held in trust. It is standing empty and the council is carrying out a consultation to identify a new use.
The group said that if the commission did allow the council to remain as sole trustee, it should ensure the building could not be sold without approval from local residents.
Councillor Lewis Robinson, executive member for culture, libraries, leisure and sport at Southwark Council, said a decision on the Livesey Trust would be taken once the consultation was complete.