Hess also has withdrawn from all SEC games for the rest of the season, SEC spokesman Craig Pinkerton confirmed in an email to The Associated Press.
Hess' termination from the ACC comes after he was accused of directing an ethnic remark at Atlanta hotel magnate Mit Shah during Sunday's Louisville-Wake Forest game in Winston-Salem, North Carolina.
Shah, the former chairman of the Wake Forest University board of trustees said on his now-private Twitter page after the game: "Karl Hess to me at the Wake-Louisville game tonight..."When I'm older, I want to sit in your seat & watch your Egyptian ass ref a game" #Wow".
Now the CEO of an Atlanta-based hotel management company, Shah is of Indian descent and was born in New Jersey, but he grew up in Winston-Salem.
Hess, who has worked ACC games for more than 25 years, told ESPN that the comment was made in jest.
"It's a sad day and is devastating," Hess told ESPN on Friday. "But I'm responsible."
"I wasn't trying to deliberately hurt anyone," he added. "That's not my character. I goof around a lot, and there was no intent to hurt anyone."
Hess had been scheduled to work an AAC game between Houston and Memphis on Sunday, but he has chosen to step away from his remaining assignments with the league. Hess, who also works for the Big 12 and Big East, might do the same for those leagues as well.
Referees are independent contractors for leagues, and Hess told ESPN earlier in the day that he hadn't heard from any league other than the ACC regarding his status.
Big East supervisor of officials John Cahill told ESPN that the league is doing its due diligence and conducting an independent investigation into Hess.
A source told ESPN.com's Andy Katz that Hess' termination was the result of accumulated incidents in the ACC.
From February 2012 until last month, Hess did not work any NC State home games in Raleigh, North Carolina. That followed an incident in which Hess ejected Wolfpack legends Tom Gugliotta and Chris Corchiani, who were attending the game as spectators, due to "excessive demonstration on several calls." The ACC publicly reprimanded him for the incident.
Hess worked the 2007 national title game and has also officiated in five other Final Fours -- in 2003, 2006, 2008, 2009 and 2013.
Information from ESPN.com's Andy Katz and The Associated Press was used in this report.
Racial Remark Costs Ref Job With ACC
ESPN college basketball Insider Jeff Goodman discusses the ACC's decision to cut ties with veteran referee Karl Hess.