Both sides started well and on two minutes a fine interchange between Rovers' duo Jamie Forrester and Richard Walker ended with Forrester's shot being brilliantly tipped over by Jamie Young. A minute later Rovers keeper Scott Shearer was forced to save a Drewe Broughton header after a cross from Andrew Burgess.
On five minutes Broughton should have done better when left unmarked but he shot well wide.
On 20 minutes Lewis Haldane took a pass from Craig Hinton but his effort was smothered by Young, while five minutes later a brilliant point-blank save by Shearer denied Alun Armstrong.
The game erupted in the 37th minute when Rovers skipper James Hunt made a two-footed tackle on Armstrong, who retaliated by pushing Hunt over and both players were deservedly sent off.
Rovers took the lead on 40 minutes when Stuart Campbell took his free-kick on the left and saw the ball go past the entire defence and into the net.
Rushden levelled on 44 minutes when Broughton headed home a Burgess cross from the right.
Rovers regained the lead in the 49th minute when Young failed to hold Campbell's corner and Steve Elliott volleyed home from eight yards out.
Rushden almost equalised in the 62nd minute when Gary Mills saw his fierce 25-yard drive go inches wide.
Rovers Jamie Forrester should have done better in the 80th minute but he shot well over when he was in the clear.
Rushden levelled on 84 minutes when Rovers failed to clear substitute Lee Tomlin's cross and Broughton turned to score his second of the game.
Rovers then grabbed their winner in the 91st minute when Forrester turned in a cross from the impressive Walker causing misery to Rushden who had a pointless Christmas period.