Detectives with the Cumberland County Sheriff's Office said the child hid the weapon in his book bag before leaving his house Tuesday morning and managed to keep it hidden until after lunchtime.
Deputies say the incident was reported by school officials after the student threatened others with it. However, no one was injured.
Authorities were called to the school around 1:30 p.m.
When school let out a short time later, parents were shocked to learn about the incident.
"I'm upset," parent Pamela Murphey said. "I'm going to call the board. This doesn't make any sense."
"I'm in shock ? my children are too, they are scared," parent Annie Williams said.
Michelle Pineey's son Ben said the student told him during lunch what he planned to do.
"He said he was going to kill him," Ben Coles said. "He told me when I got to the cafeteria."
Pineey said she was stunned by her son's revelation.
"Where are these kids getting their guns?" she asked. "It's just unimaginable."
The school's principal said the second grader was not a troublemaker, but didn't answer deputies' questions.
"It was my understanding he said all of, 'I don't know, I don't know,' he is not a behavioral student, don't have any disciplinary issues with him," Principal Cathy Tearry said.
According to school policy the juvenile can be suspended for up to 10 days, and be reassigned to an alternative school.
The child was detained by the Cumberland County Sheriff's Office Tuesday, but now is being held at a juvenile facility.
The sheriff's office was responsible for the initial investigation, but the case was turned over to the Office of Juvenile Justice Cumberland County.
Because the boy is a minor, the DJJ with input from Juvenile Court judges, will make a final decision about the case.