Deputy Chris High of the Halifax County Sheriff's office said the man appeared to be watching the houses.
Deputy High spoke to the man who identified himself as 36-year-old Anthony Leon Ellis.
He then arrested Ellis - who had outstanding federal warrants for firearms violations issued by the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF). Ellis is also charged with being a fugitive in North Carolina, which is a felony.
Ellis has been placed in the Halifax County Jail without a bond.