AAA estimates 39.1 million people will travel by car over the holiday, even as drivers feel the pain at the pump.
"We haven't seen anything below $4 in a while," said Kas Mosby, who was traveling from South Carolina to Virginia along Interstate 95, one of the East Coast's busiest travel corridors.
For many, rising costs have forced travelers to be more strategic about when and where they fill up. Drivers stopping in Fayetteville along I-95 said they were surprised to find some of the lowest prices in the region.
"As I was getting off the highway, I was like, 'Oh my gosh, $3.94,'" Mosby said. "We need to fill up now, because if you go down a little further, it's like $3.99 and then it jumps to $4.14."
According to AAA, as of May 22, the average price for a gallon of gas in Fayetteville was $4.15, compared to $2.83 at this time last year.
Even with higher prices, many travelers say canceling holiday plans isn't an option.
"Oof, gas is expensive," said Dekyhia McIntyre, who was traveling to South Carolina. "I just wanted to get away from up north, kind of go to the South for the weekend. (I've) been missing it a little bit."
Experts recommend drivers slow down to conserve fuel, avoid excessive idling and leave before 10 a.m. on Memorial Day to avoid the heaviest traffic.