Defender Paul Scott scored the only goal of the game, three minutes before half-time, enough to seal the points for the home side and a result which also ended a run of seven games without a win.
Caretaker boss Chris Casper can have few complaints from the Shakers who, having lost manager Graham Barrow, have something to smile about for a change.
Bury dominated the early exchanges and came close to scoring after 15 minutes, but striker Allan Smart, back in the side after serving a three-match ban, saw a shot saved by Rovers keeper Scott Shearer.
Junior Agogo then went close at the other end, but his curling shot was easily saved by Bury keeper Neil Edwards.
Bury skipper Dave Challinor saw a header from a Simon Whaley corner fly wide mid-way through the half, before a minute later Agogo came close to scoring but, having been left unmarked at the back post, he somehow managed to drag his shot wide of the target.
Whaley saw a powerful shot charged down in the 28th minute and a minute later saw a long-range effort fly wide of the target.
Midfielder Dwayne Mattis, last year's player of the season, fired wide from 20 yards and then four minutes before the break.
However, the home side's pressure told and Scott found the back of the net with his sweet left-foot volley from the edge of the box.
Bury started the better in the second half and, after some sustained pressure, Whaley was unlucky to see a free-kick charged down from a dangerous position.
Midfielder Dave Buchanan was unlucky not to score. The youngster saw two fierce left-footed long-range efforts fly agonisingly wide of the target, before Challinor had a header from a Kennedy corner cleared off the line by former Stockport team-mate Aaron Lescott.
Agogo could have earned his side a share of the spoils with a minute to go, but he shot wide from close range.