One of them lives in North Carolina.
Charles Bailey is now using his troubled past to help keep today's children safe from predators.
At age 59, he still carries the pain first inflicted at age 10.
"[After] the first time I was raped," he said. "I didn't feel I fit anywhere. I didn't feel connected to my peers anymore. I felt different."
Bailey says it was the fall of 1961 when New York priest Thomas Neary began sexually abusing him.
"The first time he raped me was in my own bedroom," Bailey recalled. "He laid all the groundwork."
Bailey says Neary did that by telling his parents he was preparing their son for the priesthood, and that took intense counseling.'
"He says counseling Chuck - he called me Chuck - to become a priest is going to be gut wrenching and it's going to be hard. He's going to be sick. He's going to be crying. He's going to be inconsolable sometimes. Just ignore him," recalled Bailey.
Bailey tried to shut out the horrors of his past - not telling anyone about his secret for 40 years. But one day he told his wife.
"I just came out. I could no longer hold it back. The horror on her face was amazing. She dropped everything and came over, sat with me, held me, and we wept for a couple of hours as I tried to tell her as much as I could get out," he recalled.
Now, Charles Bailey is standing shoulder to shoulder with other men who are victims of abuse just like himself. They're telling their stories as part of an Oprah Winfrey show series.
Bailey has also written a book called In the Shadow of the Cross - a raw account of the alleged abuse at the hands of Neary.
In the book, Bailey recounts the hell of his childhood.
"I was so upset I would cry myself to sleep at night - just every night," he said.
Bailey and his wife Sue have dedicated their lives to helping abuse victims. But, he says he still battles with extreme guilt - wishing he could have done more back then.
"The 59-year-old man probably would have done something, but the 10-year-old boy was not capable. If I could have just gathered the strength, I just think of all these men I could have saved," he said.
Thomas Neary died in 2001, so Bailey's claims were never investigated. Although, other victims have contacted him saying they too were abused by Neary.
Part two of the Oprah special will air next Friday.