After taking a point at Leicester City a week ago he was within seconds of victory against Bristol Rovers.
Two points were snatched away from him with the last kick of the game when Chris Lines swept in Rovers' equaliser.
Until then it had seemed an inspired move by Byrne would bring Swindon their third derby win in a row over their West Country rivals.
He had given Ben Joyce his first start of the season but it was not working out for the 19-year old and, after half-an-hour, Byrne sent on Jon-Paul McGovern to replace him.
With his pace and determination down the right Swindon were suddenly energised.
They had gone behind to Rickie Lambert's 17th goal of the season, a stunning side-footed volley which completely deceived keeper Phil Smith but that was soon to change.
McGovern took a corner on the right, it was headed down by Billy Paynter and Patrick Kanyuka, who had come up for the set-piece, knocked it in.
He had been inches away from scoring from a similar corner minutes earlier but for the second time defenders had failed to pick him up.
With his goal to build on, Swindon set off at a furious pace in the second half and Barry Corr put them ahead seven minutes after the break with Paynter again claiming the assist with a shrewd header.
For 20 minutes it seemed all Swindon as nothing went right in front of goal for Rovers.
Joe Kuffour, whose first League games had been for Swindon as a loanee from Arsenal, set up two great chances missed by Craig Disley and Sean Rigg.
Lambert then hit a post for the second time with a free-kick and Rovers' last chance seemed to have gone when Smith brought off a smart save to deny Craig Hinton from Lambert's throughball.