Many say they don't want the facility anywhere near them after a flier went out to residents last week.
Immigration Customs and Enforcement - or ICE as it's commonly referred too - already has a facility on the other side of town. ICE's contract for the space runs out at the end of 2012 and it's looking for a new location.
The Town of Cary says it is just office space. No one is kept there overnight and it's not a jail, but neighbors near the Highway 55 location say they simply don't want it in their backyard.
"No one is arguing the need for this function in our government and we value that function, just not in such close proximity to our homes," said Mark Howard with West Park Homeowners Association.
Federal officials have not confirmed they plan to relocate to the supermarket building and the Town of Cary says no permits have been applied for.
Still, the West Park community plans a meeting Tuesday night and we're told representatives from the Town will be there.
For more on what Cary is saying, click here to visit the Town's website.