The same survey showed those fans also think Minnesota Vikings star Adrian Peterson should only be allowed to play again if he is found not guilty of felony child abuse.
Only 32 percent said Goodell should lose his job, with 66 percent saying he shouldn't.
Goodell initially suspended Ray Rice for two games after the Baltimore Ravens running back was charged with assaulting his then-fiancee. The commissioner defended the punishment at first before admitting more than a month later that he "didn't get it right."
When a video of the assault later surfaced, Goodell suspended Rice indefinitely, saying the images constituted new evidence. Rice was released earlier that day by the Ravens.
The poll shows strong support for keeping Rice off the field for at least some period of time. Forty-three percent of fans say Rice should never be allowed to play again. Just 7 percent say he should be able to play now, with 49 percent saying he should be permitted to return after missing more time.
Opinions differed by gender and race. Slightly more than half of women say Rice should never be allowed to play again, compared with 37 percent of men. Just 19 percent of black fans say he should receive a permanent ban, while 46 percent of white fans support that.
Respondents were more receptive to the idea of Peterson returning to the field. Peterson is currently on paid leave while he faces child abuse charges.
Fifty-four percent of fans say he should be allowed to play again if he is found not guilty, and another 29 percent say he should be able to return regardless of the case's outcome. Only 15 percent say he should never play again.
Answers to this question also varied by gender and race. Thirty-four percent of men say Peterson should be allowed to return under any circumstances, compared with 22 percent of women. And 45 percent of black fans say he should be able to return no matter the verdict, while only 25 percent of white fans say that.
The poll suggests that the recent spate of highly publicized domestic violence cases has made a small dent in the NFL's popularity. An AP-GfK Poll conducted in January found that 49 percent of respondents considered themselves fans of pro football. That number dropped to 43 percent in the current poll.
In January, 19 percent of respondents said their interest in the sport had increased in the previous five years, with 12 percent saying it had decreased. This time, 12 percent say it has increased while 15 percent say it has decreased.
Of the group with less interest, 42 percent say the recent domestic violence arrests have been an extremely or very important factor in that drop.
The AP-GfK Poll was conducted Sept. 25-29 using KnowledgePanel, GfK's probability-based panel designed to be representative of the U.S. population. It included online interviews with 1,845 adults, including 836 NFL fans. The margin of sampling error is plus or minus 2.5 percentage points for all respondents and plus or minus 3.8 percentage points for the NFL fans.
Information from The Associated Press was used in this report.
NFL Details Attempts To Obtain Ray Rice Video
ESPN.com senior writer Don Van Natta Jr. breaks down the NFL's four attempts to obtain the video of Ray Rice and his then-fiancee, Janay Palmer, inside an elevator at the Revel casino.
Adrian Peterson Not Considering Guilty Plea
ESPN legal analyst Lester Munson discusses Adrian Peterson's expected not guilty plea on child abuse charges.