Berwick slipped to a disappointing 3-2 away defeat at Annan on Saturday when they conceded the losing goal with effectively the last touch of the game.
Berwick bounced back from the loss of an early goal with two strikes from Darren Lavery giving them a 2-1 interval lead.
But the home side hit back after the break. They equalised with a strike from 35 yards out before snatching the winner four minutes into injury time.
The defeat left Rangers second from bottom of the SPFL 2 table with only two points out of the last possible 18 making their home games this week against Montrose and Albion Rovers must-win fixtures.
Berwick welcomed back goalkeeper Paul Grant for the trip to Galabank after he missed the two previous cup ties, whilst Kenny O’Brien was preferred up front to Scott Dalziel, who was on the bench.
They got off to a poor start when Martin McNiff headed the home side in front after only six minutes. McNiff headed in from the right hand side of the six yard box, the ball squeezing past goalkeeper Grant in the bottom corner.
Lee Currie shot over for Berwick, whilst Andrew Mitchell did likewise at the other end. Josh Todd also went close for Annan, but Berwick striker Darren Lavery turned the game around for Rangers before half-time, netting twice in the space of two minutes.
He equalised with a free kick after 40 minutes and put the visitors in front after 42 with a shot from outside the box. Both goals found their way into the net at the bottom corner.
Lavery almost completed a quick-fire hat-trick when he headed narrowly wide, whilst Mitchell was denied by a close-range save from Grant just before the half-time whistle.
Early in the second half Steve Sloan shot wide for Annan, before they equalised midway through the second half when Scott Davidson fired in from distance after 69 minutes, having been set up by Todd.
Peter Watson headed over as the home side looked to finish with a flourish.
The game looked as if it was heading for a draw, with substitutes Ally Love and Scott Dalziel both firing over at either end, but with what was effectively the last touch of the ball, Kieran Brannan headed home deep into stoppage time to clinch all three points for Athletic.
Annan: Mitchell, Mitchell, Bradley (Love 58), Sloan, Watson, Swinglehurst, Todd (Flynn 82), Jardine, Hopkirk, Davidson (Brannan 88), McNiff.
Berwick: Grant, Jacobs, Hoskins, Dunlop, Tulloch, Notman, Gray (Carse 81), Janczyk, Currie, Lavery (Gielty 78), O’Brien (Dalziel 73),
Referee: Don Robertson.