The Waratahs have conceded more than 100 points in two record losses to start the Super Rugby AU season after a 41-7 thrashing by the Queensland Reds last week and a 61-10 humbling at the ACT Brumbies on Saturday.
New Zealander Penney said his young side were "shattered" and "broken" after a masterclass in ruthless rugby from the nine-try Brumbies at Canberra Stadium.
"I love these boys and I'm doing my best," Penney said. "If other people think I'm not the right person, I'm not going to stand in the way.
The Waratahs have suffered more than most from the financial crisis in the game caused by the COVID-19 pandemic and lost a squad's worth of internationals in the last two years.
The Waratahs play Perth-based Western Force in their first home match of the season next week.