Colchester boss Aidy Boothroyd wants the player to get some match action as he recovers from injury.
"Pat is going out to get match fit, because reserve team football can only get you so far," Boothroyd told the club's official website.
"Bristol Rovers asked if they could take him for a period of time, but this is very much a temporary measure and I hope that he can go there and get ready to go for the second half of the season."
As part of the deal, Baldwin will be prevented from appearing for the Pirates when they play Colchester on December 5.