David Kurakula told police he was walking with the 3-year-old towards his pre-school when two men in a white pickup truck cruised slowly by him and then out of sight.
A few minutes later, he saw the same two men wearing dark hooded sweatshirts walking towards him.
The men told the victim it was a robbery. One of them started searching through his pockets while the other man started trying to take the child out of the safety seat in the stroller.
"I was really terribly scared – I thought my grandson is really taken away," Kurakula told Eyewitness News.
Kurakula started yelling for help and the men took off.
"My grandson was terribly scared. He was shivering, yelling, 'Ta-Ta', and I was screaming, help me, help me!” said Kurakula.
All the robbers got was some paper towels that Kurakula had in his pockets.
The robbers didn't show any weapons and neither the man or the infant were hurt.
Cary police say they don't think the men were trying to kidnap the child and are calling it a strong arm robbery. They're looking for the white truck and the suspects who were in their 30s.
Still, officers said they were surprised by the mugging which is virtually unheard of in Cary.
"Very brazen, not a normal thing for Cary … frightening would be a way to put it," offered Capt. Don Hamilton.
Officers had a tip for anyone else in the same situation.
"Don't try to be a hero. Don't take anything upon yourself. If they're looking to rob something from you just give 'em the property, don't try to fight back," said Hamilton.
