The decision came down during the quarterly meeting on Saturday.
Stay on top of breaking news stories with the ABC11 News App
The trustees elected to restore the building to its original name - the Classroom Building.
The committee's report noted Carr's donation of 62 acres for the college which now makes up present-day East Campus.
The report also noted Carr's time as an active proponent of white supremacy.
Carr was a member of the initial Ku Klux Klan and ran for the U.S. Senate in 1900 on a white supremacist platform.
"The white supremacist actions that Carr pursued throughout his life, even when considered in light of the time in which they were held, are inconsistent with the fundamental aspirations of this university, and removing the name will be a powerful statement that lifts up our values as a diverse and inclusive institution," the committee wrote.
The report was in response to the request from Duke's history department, located in the Carr building, to have the name removed.
President Price sent an email to the Duke community informing them of the decision:
Our campus is first and foremost an inclusive community of people, not of classrooms and buildings," Price wrote. "With each new student or faculty member who arrives here, with each new discovery made or perspective shared, this community grows and evolves to better meet the challenges of its time. The renaming of the Carr Building represents one such evolution, at once a reflection of how our world has changed and a demonstration that our values and bonds will endure far longer than mortar or stone.
There is no word on when the name will be physically removed from the building.
"There will be a marker in the building that indicates why it was named for Carr in the first place, in 1930, and why that name was taken down ninety years later," said Duke Vice President for Public Affairs Michael Schoenfeld.
The board also endorsed the committee's recommendation to "to present educational and historical information on Julian Carr in order to preserve the record on Carr's contributions to Trinity College and help the community understand his complex legacy. The committee thus recommends that the university display information inside the Carr building that outlines Carr's connection to Duke and his legacy in the wider world; the marker should state why the university chose to name the building in his honor in 1930, and why it chose to remove his name nearly ninety years later."
According to a release sent by Duke, the committee held three meetings and solicited input from members of the university community.