The FBI and Transportation Security Administration say the drones were intercepted both in the air and in the hands of operators near venues hosting matches, including one recently confiscated in Atlanta.
It's unclear how many drones have been seized in the Los Angeles area.
Experts warn that even small drones can pose a significant risk when flown over large crowds gathered for international soccer's biggest event.
"The number is quite shocking actually," said retired Col. Bill Edwards, director of operations at ENSCO, who works in operational security.
Edwards said many operators appear to be unaware of the law rather than acting with malicious intent.
"In this case what we're seeing are what I would classify as 'careless and clueless' drone pilots ... probably not nefarious, just people trying to get pictures, trying to get video of their favorite players and games," Edwards said.
Still, authorities emphasize that flying drones over restricted areas such as SoFi Stadium or stadiums in Seattle is illegal and prosecuted as a criminal offense.
Security officials say the threat stems partly from how quickly drones can move and how little time authorities have to respond.
"These platforms are small, and they fly fast, but our decision advantage timeline is really about 90 seconds to make a decision," Edwards said.
Concerns about drones have also been heightened by recent incidents involving planned attacks - like the one targeting the recent UFC fights at the White house - using similar technology.
"It was all orchestrated around specifically starting that attack with a small drone explosive, so that's really a potential danger," Edwards said.
Posted signs around World Cup venues and designated fan zones warn that launching or operating drones is prohibited. Federal law enforcement agencies are also monitoring airspace from secure, off-site locations to detect and respond to violations.