Freeman blooped a two-out single off Brian Duensing (0-1) to bring home Jason Heyward, handing the Twins their seventh straight loss.
Trevor Plouffe and Ryan Doumit had RBI singles in the eighth against the depleted Atlanta bullpen, giving the Twins a 4-3 lead. They were one out away from snapping their slide when Gattis lined an 0-1 pitch from Glen Perkins deep into the left-field seats for his ninth homer of the season.
Craig Kimbrel (1-1) worked a scoreless 10th for the win.
Brian McCann homered and drove in two runs for the Braves, while Tim Hudson pitched five solid innings around a 1-hour, 26-minute rain delay — but they were afterthoughts after the dramatic comeback.
Gattis, who surprisingly made the team as a non-roster player and has been a key contributor, kept the game going with his first career pinch-hit homer. Duensing got the first two hitters in the 10th, but Jason Heyward doubled off the wall in left-center. Justin Heyward was intentionally walked before Freeman came through on a 3-2 pitch.
The Braves bullpen, which lost two key relievers to season-ending surgeries in the past week, surrendered the lead in the eighth. Luis Avilan retired only one batter, giving up a hit and a walk.
Plouffe tied the game at 3 with a slow roller off Cory Gearrin that got through the hole between first and second. Doumit followed with another hit to right, this one much sharper, to put the Twins ahead.
Hudson gave up five hits and two runs while striking out five. He bounced back from two rough starts, having allowed 11 runs in 8 2/3 innings in losses at San Francisco and Arizona.
Anthony Swarzak pitched four solid innings in relief of Mike Pelphrey, who was lifted after a fierce storm suddenly popped up with the Twins batting in the third. The umpires wasted no time calling for the ground crew, which struggled to get the tarp over the infield amid blinding rain and whipping winds.
Then, as the storm began to break almost as quickly as it appeared, a power surge knocked out the lights and video board at Turner Field. The board came back on in a matter of seconds, but the lights took much longer to fire back up. Finally, the game resumed when the field was fully lit.
Hudson returned to the mound for the Braves and quickly gave up an RBI single to Justin Morneau, tying the game at 2.
Leading off the fourth, McCann lined a pitch off Swarzak into the right-field seats that put the Braves back on top. Anthony Varvaro threw two scoreless innings, but the bullpen that lost Jonny Venters and Eric O'Flaherty couldn't finish the job.
The Braves jumped ahead 2-0 in the first against Pelfrey. Heyward singled and came around to score on Freeman's two-out double. McCann followed with a single that brought home Freeman.
Minnesota made it 2-1 in the second on Pedro Florimon's RBI single.
NOTES: The Twins will start RHP Samuel Deduno against the Tigers on Friday. He will take the spot previously held by LHP Pedro Hernandez, who was demoted to the minors on Monday. “He can wing it,” manager Ron Gardenhire said of Deduno. “You're going to get a guy who's going to put fear in people's hearts.” Deduno starred for the Dominican Republic in the World Baseball Classic before sustaining a groin injury pitching in the final. He spent six weeks recovering before joining Minnesota's Triple-A team at Rochester. ... Braves RHP Jordan Walden (shoulder inflammation) played long toss before the game but isn't likely to be activated Monday, when he is eligible to come off the disabled list. Manager Fredi Gonzalez believes Walden will need a couple of minor-league appearances before he returns to Atlanta.