The home side dominated the first-half and if it had not been for a string of superb blocks from shot-stopper Steve Phillips, Chris Casper's side would have been well-placed to snatch their seventh successive League Two win.
But Haldane's poachers strike in the 73rd minute, and then a breakaway goal from strike partner Walker with four minutes to go, sealed the visitors victory at Gigg Lane .
Left-back Tom Kennedy saw a long-range left-foot shot saved well-saved by Phillips, who got down low to smother the ball.
Midfielder Dwayne Mattis fired wide in the 26th minute as Bury pressed and the impressive Marc Pugh was also denied by Phillips, who produced a fingertip save to nudge the ball over the cross bar after a goalmouth scramble which also saw Andy Bishop denied from close-range.
Pugh set-up sub Jake Speight three minutes before the break, but he should have done much better despite a tight angle and dragged his shot across the face of goal.
At the other end Haldane fired wide from the edge of the box as the visitors had three chances in two frantic minutes before half-time. Walker had a shot brilliantly saved by Bury keeper Kasper Schmeichel and then Chris Carruthers saw the Manchester City loan ace tip his 30-yard thunderbolt over the bar as referee Andy Penn looked to blow his whistle for half-time.
Rovers skipper Stuart Campbell fired wide of the target two minutes after the resumption, before Speight saw a fierce left-foot shot turned round the post by Phillips midway through the second half.
Paul Scott went desperately close to making the breakthrough. The big defender had a close-range left-foot shot blocked by Phillips in the 69th minute.
But Bury's lack of a killer instinct in front of goal proved costly as Haldane's strike from Ryan Green's cross and Walker 's late strike sunk the home side's hearts having only lost 1 of their last 10 matches.