Olsen suffered the concussion on a helmet-to-helmet hit by Washington Redskins defensive end Ryan Anderson in this past week's embarrassing 29-21 loss that led to the firing of head coach Ron Rivera two days later.
Anderson was ejected and is expected to be fined by the league.
Olsen said earlier this week he was feeling better, but he didn't practice all week. He will be replaced by Ian Thomas and Chris Manhertz.
Thomas stepped up last season as a rookie when Olsen was out with a foot injury. The fourth-round pick out of Indiana had 36 catches on 49 targets for 333 yards and two touchdowns. He has only five catches for 28 yards this season, but interim coach Perry Fewell is not concerned about the tight end position with Olsen out.
"I feel very good about it,'' Fewell said. "Those guys have been pretty consistent this week. Those guys have really busted their heinie this week."
Olsen was third on the team in receptions with 48 catches for 552 yards and two touchdowns.
Ninkovich: Matt Ryan will get it done vs. Panthers
Rob Ninkovich is going with Matt Ryan and the Falcons against the Panthers.