Muhyiddin Yassin, backed by the corruption-tarnished former governing party, was sworn in by the king.
Mr Mahathir, 94, who has dominated Malaysian politics for decades, described the appointment of Mr Muhyiddin as illegal and a betrayal.
He vowed to seek a vote in parliament to challenge Mr Muhyiddin's support.
Mr Mahathir, who was the world's oldest elected leader, had returned to power in 2018 in a coalition with his old rival Anwar Ibrahim.
