"I didn't think it was real. I thought I was seeing things," he said.
There was a sign ahead that used profanity and mocked cyclists ahead of the Ironman race.
It said drivers can expect delays June 3 because of "a** holes on bikes."
"Seeing the signs like that, it just sends the wrong message," said Caffrey, who said he has competed in two Ironman races.
Caffrey says some drivers already get agitated with cyclists and the message was instigating tensions.
"It's dangerous. We're up against thousands of pounds of cars and you got to stay as safe as possible," he said. "At the end of the day, we're coming home to our families. We want to stay safe and we're drivers too. Everyone's got to understand the rules out there, and why we're at where we're at on the road and how we ride."
Another sign along US-64 near Jordan Lake referred to riders as "idiots."
An NCDOT spokesperson said the signs do not belong to their agency. ABC11 reached out to the Ironman organization to see if the signs belonged to them.
The signs have since been fixed.