This comes after the release of a secretly recorded video from a campaign fundraiser.
On the recording, Romney can be heard calling the ABC talk show "high risk." He also talked about the political affiliations of the hosts saying four of the hosts were sharp-tongued and not conservative.
A spokesperson for Romney said Ann Romney will be joining the ladies of "The View" instead.
Barbara Walters made the announcement during the show Monday and left the invitation open to Romney to come to the program if he changes his mind.