ABC affiliate WPVI spoke exclusively with his family, who shared a photo of Davoris Carter.
They say Davoris attended a school in Northeast Philadelphia for students with special needs, and had both a school chaperone and a family member accompany him on the trip.
However, when the family member stepped away to go get food, Carter went into the water with some other students near Morey's Pier, while staff watched from the shore.
There were no lifeguards on duty, which is typical for a weekday at this time of year.
His grandmother, Kenya Pippen, told WPVI she wants answers and accountability.
Three other students also needed to be rescued and were aided out of the water by a Wildwood police officer who got to the scene first.
Search underway for missing 14-year-old swimmer in Wildwood
The search for Carter was suspended after several hours.