The League Two visitors had outplayed the Championship leaders until Aaron Lescott was sent off for a foul on Arturo Lupoli.
A yellow card appeared the most likely outcome, but to everyone's surprise referee Andy D'Urso produced red to incense the Rovers camp.
Rovers were dominant in the first half and a two-goal interval lead would certainly not have flattered them.
There were early warning signs for lacklustre Derby as ex-Ram Steve Elliott headed straight at Stephen Bywater and a slip by Darren Moore almost let in Rickie Lambert.
And Rovers, cheered on magnificently by almost six thousand travelling fans, gained in confidence to build up a head of steam and create several clear chances.
A dreadful back-pass by Richard Jackson which almost resulted in an own goal was a further boost for the Pirates, who went desperately close in the 32nd and 34th minutes.
Bywater did brilliantly to prevent Richard Walker's header from striking the top corner of the net after Lambert delivered a right-wing cross.
And Bywater, with as much luck as judgment, came to the rescue again two minutes later when Sam Igoe's close-range header struck his legs when a goal had looked certain.
A shot from Giles Barnes finally warmed Steve Phillips' fingers in the 35th minute, but the Rovers defence was showing why they had kept clean sheets in their previous four games.
Rovers were quick to retaliate with Lambert mis-kicking horribly with the goal at his mercy from Andy Sandell's pass.
And Sandell, who was giving Jackson a torrid time, had the ball in the net with a 41st minute header which was ruled out for an infringement.
Derby no doubt received a half-time roasting and they bucked their ideas up to force a couple of corners at the start of the second period.
And they almost took a 56th minute lead when Moore's header from Seth Johnson's corner was booted off the line by Lescott.
The tie was really warming up and Rovers had another great chance on the counter-attack as Sandell sprinted clear of Moore to strike a shot too close to the relieved Bywater.
Derby continued failure to get hold of the game with any conviction added to the increasing frustration of the home supporters, who had expected an easier passage into the fifth round.
And it was the visitors who threatened again in the 65th minute with the impressive Lambert turning Sandell's left-wing cross inches past the far post.
Derby got their unexpected tonic in the 70th minute with Lescott's red card and Rovers quickly re-organised, with defender Ryan Green replacing Walker two minutes later.
The Rams brought on Peschisolido for Lupoli 16 minutes from time and it was the much-travelled Canadian who broke Rovers' hearts in the 82nd minute.
For once the Rovers defence had switched off when Seth Johnson nodded the ball forward for Peschisolido to sprint clear and drive a left-foot shot past Phillips from 12 yards.
Rovers still refused to lie down and forced three late corners before referee D'Urso blew the final whistle on their FA Cup campaign.