Lee Naylor thought he'd won it for Wolves with a late strike that took a heavy deflection, but John had other ideas and finished off a great run in the 90th minute.
It was a blistering start to the game as both teams had great opportunities to take the lead inside the first 45 seconds.
First Colin Cameron was put through by Seyi Olofinjana but placed the ball just wide of Luke Steele's post from 12 yards.
And then, at the other end, Shaun Goater and Lloyd Dyer combined to set up Dele Adebola who was just off target when he only had Michael Oakes to beat.
The fast pace continued and City could have gone ahead on two minutes when Dyer crossed for Goater, who saw his shot blocked by Jody Craddock. It bounced clear for Michael Doyle but he dragged his effort wide.
Joleon Lescott fired wide for Wolves in the fifth minute before Adebola saw a low shot held by Oakes.
Olofinjana cracked one from the edge of the box on 11 minutes but Steve Staunton took the sting out of the shot and made it a simple save for Steele.
After 15 minutes, Doyle conceded a free-kick on the corner of City's box and it led to the opening goal.
Naylor fired in a sweet cross and Clarke was left unmarked to head home his fifth league goal of the season.
It might have been two just a minute later when Kenny Miller released Ki-Hyeon Seol on the right wing but his strike was blocked by Marcus Hall.
Coventry drew level on 18 minutes when McSheffrey drove into the box past Naylor and the Wolves full-back tripped him to concede a penalty.
The City striker picked himself up to blast the spot-kick past Oakes for his third penalty goal in the last three matches at Highfield Road.
The frenetic start could not continue but it was the home team who looked most likely to edge ahead.
Claus Jorgensen headed over from a Doyle corner before Goater was denied two clear-cut chances.
Adebola and Richard Duffy picked out the striker with crosses but the former's was just too long while the latter's was cut out by a last-ditch challenge by Craddock.
Wolves were on top in the second half and were certainly dominating possession but Coventry still looked more like scoring.
Seol came close, however, for the visitors when Hall sliced a cross into his path but Steele pulled off a great save to deny the South Korean.
But Coventry could have won it when Duffy retrieved a corner and crossed for McSheffrey but he headed against the crossbar.
Then Doyle set Adebola free but he was denied at point-blank range by Oakes.
Andy Morrell was thrown on by Micky Adams and almost scored with his first touch but Oakes was equal to it.
But it was the visitors who snatched the lead when Naylor cut in from the left and saw his shot take a massive deflection which sent it past the helpless Steele.
Substitute John grabbed a dramatic equaliser when he cut past two defenders before toe poking into the bottom corner in the last minute.