A petition to tear down the building, where slaves were once sold, has nearly reached 115,000 signatures, just around 36,000 shy of its 150,000 goal.
"Most African Americans in the city wish for it to be taken down and replaced with something more positive about African American history," the petition reads. "The Market House building is a reminder of slavery and fuels white supremacy."
VIDEO: Market House in Fayetteville set on fire during protests
Robert Taber, an assistant professor of history at Fayetteville State University, said historical records show slaves were sold at or near the Market House.
"We have the records and it happened fairly often, and I think it's a real mistake to downplay the realities of that," Taber said.
The Market House is one of the oldest structures in the city that was built and designed by African Americans.
Taber told ABC11 there was even a free black community, before the Civil War, who would go to the building and hold independent day festivals.
During Jim Crow, people labeled it the "Old Slave Market" on plaques and maps to demean and intimidate the black community.
"This is not the first time there's been discussions over the Market House's identity, what it means to people and all that kind of thing," Taber added.
The recent protests over police brutality, the attempted burning down of the building, and the current conversation of modern-day racism are all reasons why this petition was created two weeks ago.
$5,000 reward offered for information about people involved in Market House fire
The creators of the petition said it's time bring down the symbol of hate and create something that can unify Fayetteville.
Taber said it's up to the city and community to come to a conclusion on whether the building can be repurposed for a better message or be completely torn down.
"The question then becomes - can we apply enough of an antidote or anti-venom to have it stop being poisonous or is it just time to move on," said Taber.