Kieran Lee put Duns ahead with only nine minutes on the clock, firing in a low shot from just inside the box and the Dingers had the ball in the net again but Steven Burns was ruled offside.
There could have been goals at either end before the home side levelled the scores on 39 minutes and 1-1 was about right at half-time.
Duns conceded a second goal seven minutes after the break but battled back to level the scores again when Steven Burns sent in a great cross from the right for Junior Pattenden to head home.
The visitors then looked the more likely winners at that point but they conceded again and allowed Kelso to take control of the game from that point onwards.
The game was put beyond the Dingers when Kelso added a fourth goal on 83 minutes
Duns: Wales, R Burns, A Renton, Dixon, G Renton (Falconer 86), McGregor (Bolton 75), Weatherburn, Lee, S Burns (Walker 67), Pattenden, Wadddell.