Berwick paid the penalty for having a man sent off when they were beaten by Elgin, whose victory lifted them off the bottom of the table.
Rangers had central defender Jonny Fairbairn dismissed after 30 minutes when he was shown a straight red card for a last man foul on Craig Gunn.
The visitors then went on to secure the victory when Archie McPhee slammed home a second half penalty, with Daniel Moore adding a second in injury time.
Berwick, with only one point from their last two games, made three changes from the side which had lost to Annan the previous Saturday, with Kerr Young, Ross Gray and Craig Dargo all handed starts.
After only four minutes Darren Lavery saw goalkeeper Ross Laidlaw tip a low shot away for a corner and in the 12th minute Stevie Notman was unlucky when he saw a glancing header from a Lee Currie free kick on the right slip narrowly wide.
Fairbairn then had a header from a Willis corner which was directed straight at Laidlaw, whilst at the other end Daryl McHardy lifted the ball over the crossbar.
A Currie free kick from distance was deflected straight into the gloves of Laidlaw, but Rangers suffered a setback after half an hour when central defender Fairbairn was shown a straight red card for a last man foul on Craig Gunn as he cut across to intercept a long ball through the middle.
In the 37th minute Dargo chipped the keeper but Ally Gilchrist got back to clear the ball off the line.
Down to ten men, Notman dropped back to replace Fairbairn in central defence and he made a good covering header to clear the danger from a cross into the box.
Early in the second half Currie clipped a first time shot which was tipped away for a corner. Berwick were prone to conceding free kicks on the edge of their own box, but Elgin failed to take advantage.
Gunn pulled a shot wide from outside the box but it never troubled the home goal.
Bald made a good point-blank save to deny Archie McPhee from close range following a deep cross from the left, but in truth Elgin never really stretched Berwick despite their numerical advantage.
In the 68th minute a McPhee curling shot from the right was saved by Bald, but he finally got the better of the Berwick keeper after 70 minutes when he rifled home from the penalty spot after Devon Jacobs brought down Craig Thomson in the box.
Rangers immediately substituted Jacobs for striker Blair Henderson and went three at the back to try and force the game.
As Elgin pressed forward Thomson cut in from the left but his shot from 20 yards out was hit straight at Bald.
Late in the game there were chances at both ends, with Paul Willis heading against the Elgin crossbar from a Currie free kick, but it was the visitors who finished on a high with Daniel Moore netting from close range after they broke down the left and cut open the home defence.
Berwick: Bald 7, Jacobs 6,( Henderson 72), Young 6, Fairbairn 4, Drummond 6, Notman 7, Willis 7, Currie 5, Gray 6, Dargo (5, Russell 82), Lavery 5.
Bookings: Young 12 dissent; Fairbairn (red) 31 foul; Jacobs 69 foul;
Elgin: Laidlaw 7, Cooper 6, McHardy 6, Gilchrist 7, Duff 6, Nicholson 7, Thomson 7, Moore, Wyness (Sutherland 66), Gunn (McLean 82), McPhee.
Bookings: Nicholson 33 foul; McPhee 35 mins foul;
Referee: Don Robertson.