"I haven't missed a game due to injury in 20 years, since my junior year of high school. I consider myself very fortunate for that," Brees told ESPN Insider Ed Werder. "[That's] another reason why this decision was so tough.
"Bottom line is Luke McCown gives us the best chance to win this week. I have to be honest with myself and do what is in the best interests of the team. I have total confidence in Luke, as does our team."
The Saints will start McCown against the Panthers, and Garrett Grayson will serve as the backup.
Brees was optimistic earlier in the week that he would be able to play despite suffering a bruised rotator cuff in his throwing shoulder Sunday. Payton, however, said Brees and the team agreed that the strength and velocity weren't where they needed to be.
Payton said he still expects Brees back "sooner than later" and that they will continue the rehab process next week.
"I think just where he's at in the rehab and the strength, I think he felt -- I think we all felt -- like it would be too early, him playing this weekend," Payton said. "And I would say next week then continue the process without trying to put an estimation on his return.
"He felt like he made progress throughout the week, and yet it's still not where it needs to be or where we would feel comfortable with the strength that he needs. But he did get better each day, and we'll just see where it's at next week."
Brees, 36, has never missed a regular-season game due to injury in his 10 years with the Saints, which began after he recovered from major shoulder surgery in 2006.
The only other quarterback to start for New Orleans since 2006 was Mark Brunell, who started in Week 17 of the 2009 season after the Saints had clinched the 1-seed in the NFC.
Payton said Brees tested out his arm in the mornings as part of his rehab but didn't throw in the framework of practice. Brees participated in practices on a limited basis during the week without throwing.
"I don't think it was a difficult decision today. I think actually it was the opposite for just where he's at," Payton said. "His focus and preparation throughout the week was very much like he was starting. And even out on the practice field today as Luke's taking the reps, he's on top of what we're doing by game plan.
"He practiced both yesterday and the day before. And yet I think this morning as he was throwing, he just felt like the velocity still wasn't there."
Sources told ESPN NFL Insiders Chris Mortensen and Werder earlier this week that Brees' MRI did not show any tearing in the rotator cuff -- a diagnosis that was confirmed by orthopedic surgeon Dr. James Andrews.
McCown, who has a 2-7 career record as a starter, said he could see the disappointment in Brees, especially knowing what a competitor he is.
"He doesn't give up reps, much less games, obviously," McCown said. "To see kind of the air of disappointment in his face ... as a friend I understand that and as a competitor I understand that. At the same time, he's been tremendously supportive."
McCown hasn't started a game since Week 2 of the 2011 season, when he was with the Jaguars in a 32-3 loss to the Jets. He last completed an NFL pass in Week 12 of that season.
The Saints have stuck with McCown, 34, as their backup for three years because they trust him based on both his experience and skill set. He completed 24 of 35 passes for 323 yards, three touchdowns and zero interceptions this preseason, often lining up with the first-string offense.
Teammates and Payton have expressed confidence in McCown all week, saying the game plan won't change much with him behind center.
"I never got into football hoping to be a backup," McCown said. "I started and played 45 straight games in college [at Louisiana Tech]. You never approach something saying, 'I hope I don't play today.' So it's an opportunity that I'm excited about. But I'm excited because it's a chance to help our team win."
With Josh McCown returning as the starter for the Cleveland Browns this weekend, it will mark the first time that the brothers will both start an NFL game since Week 14 of the 2007 season, according to ESPN Stats & Information.
With Brees ruled out, Las Vegas sportsbook operator CG Technology bumped the Panthers from 6.5-point favorites to a 9-point margin against the Saints. Most Nevada books had kept the line off the board with Brees' status uncertain.
ESPN Staff Writer David Purdum contributed to this report.
Brees out against Panthers
ESPN Saints reporter Mike Triplett explains why Saints quarterback Drew Brees will not play against the Panthers.