The 20-yard strike from the substitute completed a fine comback from Rovers who had fallen behind to Gary Alexander goal just before the break - but Rovers came storming back and levelled early in the second half through Craig Disley and the game looked to be heading for extra-time until Haldane's last gasp winner set up a final against Southend United.
In a full-blooded cup clash Orient started well and Alexander almost put the O's ahead on ten minutes.
The O's skipper, making his first start since an ankle operation in October, saw his close-range shot blocked by Rovers keeper Ryan Clarke.
But Rovers threatened on the break and Disley raced clear of the home defence and it needed a brave save from keeper Glen Morris to keep the scoresheet blank.
Former Orient striker Lee Thorpe then rattled the bar with an effort from 25 yards but it was the home side who edged ahead on 43 minutes when Lee Steele danced down the right and crossed for Donny Barnard to touch the ball into the path of Alexander who swept it home.
Rovers hit back at the start of the second half and James Hunt's through ball carved open the Orient defence allowing Disley the chance to race clear and lob home.
After that Orient had plenty of pressure but paid the price for failing to create many clear-cut chances as Haldane raced onto a loose ball in stoppage time to fire Rovers into the final.