With two minutes to go the Pirates were trailing to a strike from Dennis Lawrence following a needless foul.
Then Walker turned in a cross and, well into the third minute of added time, he grabbed the winner as the ball was played in from a corner.
Wrexham thought they had the game won after capitalising on that needless home foul by Craig Disley 11 minutes into the second half. He had tracked Mark Jones for 30 yards before getting in a tackle from behind to pick up a yellow card.
There was no doubt about that, but there were questions about Rovers defence as Darren Ferguson's long free-kick was headed down by Danny Williams for Lawrence to supply the finishing touch.
The late win came as a boost for caretaker manager Paul Trollope, who is seeking the top job.
Under their young caretaker Rovers have rediscovered a zest for the game, but it was the experience of Wrexham's Jon Walters that so nearly cost them.
Moving out to the right he constantly out jumped defender Robbie Ryan and produced a cross to the far post which saw Juan Ugarte just lose out in a tussle with goalkeeper Scott Shearer.
Wrexham have taken back last season's scoring hero Ugarte on a month's loan from Crewe but he was substituted at half-time.
Rovers were always looking for the lively running from deep of Junior Agogo and some useful crosses from Ali Gibb on his right.
The best of those came just before the break when Agogo jumped well in the centre of goal. But the ball caught him on the cheek, not the forehead.
Strangely Rovers' late revival came after both Agogo and Gibb had been replaced.