The jobs will end in the middle of November.
The 500 jobs represent about one-third of the work force at call center.
Those affected all work for one specific line of business, but Xerox is not saying what client is leaving.
Workers told ABC11 that it's the call center operations for Sprint, but the company has not confirmed that.
The official word from Xerox is that they plan to hire back about half of the people who are laid off, and will begin making those hires the same week as the layoffs happen.
The layoffs are clearly upsetting to folks who call Cary home.
"It's hard for a lot of people to find a job, and when you have that many people, skilled workers with good salaries that are out of the workforce, it's a lot more people struggling," said Cary resident Mary Morris.
As Xerox is cutting jobs in Cary, MetLife is promising to bring in more than a thousand over the next few years.
ABC11 looked at employment numbers going back one year to see just how many jobs we've gained and how many we've lost. The numbers tell a surprising, and unnerving story.
A survey of employers shows a huge gain in jobs in the second half of last year about 60,000. However, that number was less than 6,000 in the first half of this year.
A survey of the labor force is even more stark. The back half of 2012 added 53,000 people. The first of half of 2013 lost 90,000.
Employment numbers in that survey told the same story. A big jump at the end of last year, and a big drop at the start of this year.
The story that the state wants to tell is of the state being on an upswing.
"North Carolina is becoming a great recruitment for young talent," said Gov. Pat McCrory during a ribbon cutting at new Raleigh business Thursday. It's becoming successful. It's the Carolina comeback."
The numbers, however, don't bear that out.
Economists say they do expect things to get better in the next year or two.