Weber's second try came after a breath-taking buildup, with replacement back Alex Nankivell bursting out of defence and fullback Solomon Alaimalo punching a hole through midfield before giving off to the pint-sized number nine in the 70th minute.
The Chiefs, who also had tries from flanker Mitch Karpik and Anton Lienert-Brown, held on in a nail-biting finish to keep their slender finals hopes alive.
The Chiefs are in 11th place, three points outside the top eight playoff places.
"Now, we've got a hell of a lot of fight in us. We were down with about 15 (minutes) to go, I knew we still had it in us," said stand-in skipper Weber, who grabbed the Chiefs' first try in the 26th minute.
The Sharks had their sixth loss for the season but grabbed a bonus point to leapfrog the Jaguares and Bulls into top spot in the tight South African conference.
Weber's winner came only seconds after first-five Marty McKenzie had squandered a chance to put the Chiefs in front.
After Lienert-Brown shrugged off a tackle to dart over the line near the left corner in the 68th minute, trimming the Chiefs' deficit to a point, McKenzie skewed his conversion kick left of the posts, bringing a groan from the terraces.
Weber ensured it would not matter, however, and the Chiefs' fourth win of the season lifted them off the bottom of the New Zealand Conference into fourth with 23 points, six behind the third-placed Highlanders.
The Sharks dominated territory and possession but could do little with it in the first half and went into the break 17-13 down after Marty McKenzie converted Karpik's 35th minute try.
Momentum swung back to the visitors when hooker Nathan Harris was given a yellow card in the 56th minute for a cynical ruck infringement deep in the opponents' 22.
The Sharks rumbled to the line and lock Ruan Botha stretched an arm out under a pile of players to plant the ball on the chalk.
First-five Curwin Bosch slotted the conversion and a penalty minutes later to give the Sharks a six-point lead at the start of the final quarter as the Chiefs' discipline dissolved.
All Blacks midfielder Lienert-Brown steadied the home side with their third try, however, before the pint-sized Weber led from the front to nudge his side ahead.