Donald Baldwin Jr. is jailed on $200,000 bond.
The victims' family says early Wednesday morning a naked Baldwin knocked on the window of the apartment the two families share at Park Creek Apartment in Raleigh.
"He see the guy right here at the corner, he was naked," the victims' uncle said. "He was like that, and they tell the guy, 'You need to get out. You need to get out. What are you doing here?'"
They thought he left.
One of the men took his wife to work, leaving their sons sleeping in a bedroom.
They say Baldwin, 31, came into the apartment without the other family waking up and got into bed with the boys --ages 6 and 8.
"The boys, they were sleeping," the uncle said. "This guy, he come to the bed and starts touching my nephew so he wakes up, wakes up and tried to fight with the guy, 'Don't touch me, Don't touch me!'"
The uncle said he had never seen the man before, but Baldwin lives in the Park Street Apartments on Chris Street. He's lived there for years.
Baldwin's father has also lived in the apartment for years but didn't want to talk about the incident.
However, neighbors spoke out about what happened.
"Didn't seem like he would hurt a fly," neighbor Daniel Taft, Sr. said. " I feel if it was my place they probably would've taken him out in a body bag."
Instead, the victims' family kept Baldwin in the apartment until police arrived.
Baldwin is also charged with possession of cocaine. It is unclear if he was high when police say he committed the crimes.