Today, he says he stays awake at night thinking about how democracy can flourish in our constantly evolving world.
Watch "Juneteenth: Together We Triumph - A Soul of a Nation Special Event" FRIDAY at 9 p.m. ET and next day on Hulu.
"We're now seeing state legislatures try to further suppress the vote or allow partisan politics to infect voting results and election results. That troubles me," Obama told "Good Morning America" host Michael Strahan in an exclusive interview.
Republicans in favor of measures requiring voters to identify themselves say these laws protect against people voting more than once, voting on behalf of someone else, or voting when they do not have the legal right to do so. Others who agree with Obama say voter fraud is not nearly common enough to warrant these laws, which disproportionately affect minorities including people of color.
Obama said he also spends a lot of time thinking about income inequality. "How can we increase minimum wages? How can we ensure that if you work hard in this country, you can make it?" he said of the questions that keep him up at night. "How does our democracy get refreshed and updated so that it continues to work effectively as we become a more diverse society?"
Still, Obama is optimistic democracy can prosper.
WATCH | What to know about the Juneteenth flag
Ahead of the Juneteenth holiday, which celebrates the emancipation of all enslaved people in the U.S, the former president said he trusts in a new generation of people fighting for equal rights. He met with some of these next-generation leaders, talented young athletes and students in one of Washington D.C.'s poorest neighborhoods, to discuss their ambitions. He also encouraged all Americans to reject hopelessness and work toward unity.
Even after the U.S. Capitol was attacked by insurrectionists who falsely believed then-President Donald Trump had won re-election, Obama said he believes "there's a way back."
"We've gotta do a better job of reducing the influence of those who try to inflame division and traffic in conspiracy theories for their own benefit," he said. "If we can't agree on basic facts then it's very hard for us to negotiate and compromise in a way that is constructive."
In May, the House of Representatives passed a bill to create an independent investigation into the January insurrection, but it was blocked in the Senate despite six Republican senators breaking ranks to vote in favor of the investigation. House Republicans argued the investigation would be used as political ammunition against them into the 2022 midterm elections.
"I do not believe the additional, extraneous commission that Democratic leaders want would uncover crucial new facts, or promote healing," Senate leader Mitch McConnell said. "Frankly, I do not believe it is even designed to do that."
Five people, including Capitol Police officer Brian Sicknick, died during or after the riot, and 140 police officers were injured. At least 465 of those who breached the Capitol have been arrested.
Meanwhile, many voters across the country still question the 2020 election results that spurred that attack -- despite repeated debunking of false voter fraud claims, several recounts confirming President Joe Biden's win, and failed lawsuits on behalf of the Trump campaign.
"How we change that dynamic is gonna require, I think, a level of the public paying attention and insisting and demanding their elected officials and people in the media just, you know, saving our democratic habits and practices in a way that we have not seen," Obama said, adding that the rhetoric delegitimizing the process is a "threat to democracy."
Obama acknowledges how far political camps in our country have drifted apart from each other.
"If you are fighting for dignity and human rights and inclusion and opportunity, there's always gonna be resistance in a big, complicated society like ours. The question is are we able to maintain a process in which we listen to other voices?" he said. "If we lose, we say, 'As long as the process was fair, then I guess we gotta go out there and persuade more people to our side.'"
"There's a give and take," he added. "In our world of hot takes and Twitter, [there's an effort to] demonize the other side as much as possible. Sometimes, we think somehow the compromise and listening and working together is wrong."
Our country is only becoming more diverse, and our task is to make sure democracy works for everybody, Obama said.
"The American dream has been a reality for some and a myth for others," he said. "[There are] a lot of communities around the country where the kids who grew up here may formally be free, but structurally, because of poverty, because of schools that aren't working, because of substandard housing ... It requires so much more effort for them to live out that American dream. ... Our job is to make sure that it's not a myth, and right now, for too many, it still is."
His point is made clear in the very place where the former president met with ABC News: the Anacostia neighborhood of Washington, D.C., where the area's population is 94% Black and where 38% of its residents live at or below the poverty level.
"Something's happening in our society that prevents them from ascending to the highest office in the land," he said of underprivileged communities like Anacostia. "The same is true for African Americans; the same is true for Latinos; and [for] the first Americans, Native Americans. ... The odds are stacked in ways that prevent a lot of young people from realizing their potential, and we can do something about it."
Anacostia's Barry Farms neighborhood was once a tobacco plantation and later the first settlement for freed, formerly enslaved people in the city. In 1943, it became the Barry Farm Housing Projects and in the 1970s, the Barry Farm Community Summer Basketball League was formed.
In 1996, former resident Miles Rawls restarted the program when he returned from military service. Today, it is known as the George Goodman League and operates in partnership with the District of Columbia on the grounds of the Barry Farms Recreation Center, in an effort to keep history alive and people off the streets.
The league in the southeast corner of the city has attracted passionate ball players from around the region, and even the occasional NBA player, such as Kevin Durant, Gilbert Arenas and John Wall. This year, the league will celebrate its 25th anniversary.
"Back in those days -- the late '90s, early 2000s -- every gangster, hustler, troublemaker, you name it, was in here watching the games," Rawls said. He had one rule, he said: "No beefs in the gates... This was a crime deterrence."
Seven years ago, Rawls, along with Marcus Chester, decided to expand the league, subsequently starting the Junior Goodman League. Today, it fields youth teams of third through 12th graders.
But due to the pandemic over the last year, the courts were closed.
"Throughout it all, I think they have learned to roll with the punches. Life comes at you with different things and you have to learn to deal with them," said Chester, who runs the Junior League.
Now, he says the program is back in action. Earlier this month, Obama surprised a group of athletes from the league as well as players from area schools.
Seventeen-year-old Rakisha Ballinger, who played pick-up with the Goodman League, is graduating from high school this year. She told Obama she will be attending Wagner College in New York to play basketball and study nursing. Electra Allembert, 17, said she doesn't yet know where she wants to go to college, but told the former president she wants to study law.
"I was like a lot of teenagers. I made mistakes," Obama told the students. "The reason I tell young people that is because I want them to have a sense that even if you make some mistakes, you still have so much time to reorient yourself and to go down a path that can lead to success."
Obama added that being on a basketball team like the Goodman League, in any sport, helps players "start feeling like it's all part of something bigger than just you. ... That's true in sports, [that] is true in life."
The former president said he takes hope from "the long view" and "recognizing that resilience, determination, the ability to deal with setbacks and disappointments... carry us forward."
"No one has exhibited that more historically in this country than African Americans," he said. "The March on Washington happened during my lifetime. That's not ancient history. In big parts of the country, segregation was still operative when I was alive. ... What seems like stuff we now take for granted, that's just a generation old."
Now, the next generation has taken on the mantle of furthering equality. When asked what a young Obama might be doing at this time of social change, he said he hopes he'd "be out there with those young people ... trying to steer that energy into constructive, practical change."
"I could not be prouder of the young activists who responded to the George Floyd murder and so many of the other incidents that we've seen over the last several years," he said. "You had a level of white participation in those protests that you never saw, even at the height of the Civil Rights Movement back in the '60s. ... The key now is to translate that righteous anger and frustration and sadness into concrete policies."
Sixteen-year-old Machi Chester, who has been playing with the Junior Goodman League since it launched seven years ago, says he was one of those people who went to the Floyd protests.
"Just to come together as a community was -- it was a good thing," Chester said.
Obama acknowledged that it can be "tempting" to consume the news sometimes and "feel despair."
"When we examine the history of race relations in this country ... what's happening with climate change, it is tempting sometimes to say, 'My goodness, humans are not going to get their act together,'" he said.
Hopelessness can be a "refuge," he said, from the harder work of pushing for change.
"The harder thing to do ... is to look at that stuff squarely, not deny it, not pretend there's no racism out there, not pretend that humans aren't cruel and selfish sometimes. But to say, 'Despite those truths and that fact, that's not immutable. That's not something that can't be changed,'" he said.
"We're not all gonna live in a perfect kumbaya society but we can make it better by working and by reaching out, and by assuming the best in each other. And if enough of us do that, we move that boulder up the hill," he said.
In his new book "A Promised Land," he writes about how often people believe change comes from one charismatic leader, but he says that's not entirely true.
"When we have a bunch of people working together, that's when change happens. That's one of the things I really admired about so many of the protests that happened this summer."
"When you look at our history, it happens from the bottom up. It happens because of [the] multiplicity of voices. Because none of us alone have all the answers," he said. "A thousand people who are organized, they can change the law in a city. A couple thousand folks who mobilize, they can have an impact at the state level. And a handful of people who are determined, they can change a neighborhood."
Obama said he misses doing that work while he was in office. However, he said, "I don't miss the pomp. I don't miss the attention."
Now, when he's not traveling, the two-term president spends his time between his homes in Washington, D.C., and Massachusetts. He said he feels joy from his daughters, Sasha, who just turned 20, and Malia, 22.
"When I see the two of them and the grace and kindness and sense of responsibility they have to each other and to the wider world ... of all the things I have accomplished in my life ... my partnership with Michelle, resulting in them, that makes everything worth it," he said.
In honor of Juneteenth, we're telling stories of what Black freedom means today, from a 94-year-old's quest for a national holiday to the fight for reparations to cultural celebrations. Click here for more stories from your city and around the country.