Duns came from two down to lead 3-2 thanks to a double from Michael Robinson, but after having a penalty claim turned down the home team let Ormiston come back to grab an injury-time winner.
Carl Hudson’s men never really got started in this East of Scotland Qualifying League game and could have been 4-0 down at half time.
Only by good fortune and some fantastic goalkeeping by Scott Lumsden did they find themselves trailing 2-0 at the interval.
But the second half was a different story, kick-starting a Duns revival.
Goals from Kieran Lee and a double from recent signing Michael Robinson put the home side 3-2 up.
With the game on a knife edge, two penalty decisions at either end proved crucial. Both went against Duns.
The first came when the home side were on the attack.
Daniel Pattenden appeared to be brought down by the visiting goalkeeper when he was about to score, but the referee waved away the Duns appeals to award a spot kick.
At the other end, however, he did award a penalty to Ormiston, who scored to make it 3-3.
Deep into injury time Ormiston fired in the winner.
Duns: Lumsden, Ainslie (R Burns), A Renton, Dixon, G Renton, McGregor (Bolton), Weatherburn, Lee, Robinson, S Burns (Pattenden), Waddell.