John Crawford put the Streamers ahead twice in normal time, only for Gretna to equalise. And with the score at 2-2 Mark Walters saved a penalty in the 90th minute to take the tie into extra time.
Conor Devaney fired Coldstream ahead again, but Gretna levelled for a third time and there were chances at both ends before the tie went into penalties.
Jonny Simpson and Richard Baxendale missed from the spot as Gretna scraped through.
It was cruel on Coldstream, who had started the match brightly.
An accurate through-ball from Gary Wales set John Crawford racing clear and newly-crowned East of Scotland player of the year slotted past the advancing Gretna goalkeeper.
But Gretna equalised midway through the first half when their winger cut inside to hit an unstoppable shot into the top corner.
The home side went in front when a Richard Baxendale mistake was punished. The ball was immediately crossed into the danger area and despite an outstanding save from goalkeeper Mark Walters the rebound was hit hard and low to give the Lowland League side the lead.
But in a pulsating tie, Coldstream levelled again. Greg McCrudden played the ball to Conor Devaney, whose reverse pass was hit first time into the corner by Crawford.
With extra time looming, there was more drama. Gretna had the chance to win the tie in the last minute of normal time when Jake Birkett conceded a foul on the edge of the box. The referee awarded a penatly, Walters dived low to his right to claw away the spot-kick.
At 2-2, the match then went into extra time.
Coldstream took the lead when the Gretna goalkeeper was penalised for picking up a pass back. From the resultant free-kick, Devaney’s strike took a deflection on its way into the net.
But the home side equalised again when a bouncing ball not dealt with properly and Gretna’s midfielder sent a rocket of a shot past Walters for the leveller.
Coldstream couldn’t do enough when it came to the lottery of penalties, but they can be proud of this performance.
Coldstream: Walters, Allan, Baxendale, Ainslie, Birkett, McCrudden, Douglas, Wales, J. Simpson, Devaney, Crawford. Subs: Peoples, Powling.