The Welshman took charge of the team at the start of 2017 and guided Fiji to the World Series title in 2018/19, a Commonwealth Games silver medal on the Gold Coast and Olympic gold in Tokyo.
He was offered a contract extension through to the 2024 Paris Olympics but said he wanted to pursue coaching opportunties in the 15-a-side game.
"It is with a heavy heart that I have to inform Fiji Rugby and the people of Fiji that I will not be accepting the contract offer to coach the Fijian 7s Team to the next Olympic Games," he said.
"I have a burning ambition to challenge my coaching again in the 15s environment and know that I must immerse myself in this if I'm truly to be measured and to progress in my coaching career.
"I'm an individual who is driven by the challenge of the sport that I love, and this is what brought me to Fiji in the first place, the chance to coach the best 7s team in the best rugby nation in the world.
"My family and I have lived the sporting experiences in Fiji that I never thought would have been possible and I cannot express my gratitude enough to the FRU Board and everyone for supporting us on our journey."
Fiji Rugby Union CEO John O'Connor said the Board had endorsed Baber to continue as head coach but they respect his decision to pursue a new challenge.
"We totally respect the decision made by Gareth and thank him for his service to Fiji Rugby and to Fiji. We are all proud of Gareth's achievements, culminating in the gold medal defence at the Tokyo Olympics."
Fijiana 7s head coach Saiasi Fuli will oversee the Fiji men's teams preparation for the opening round of the World Rugby Sevens Series in Dubai on November 26-27, alongside his duties with the women's team.
Fiji 7s captain Jerry Tuwai will also step into a player-coach role with the men's team.
"While coach Fuli is committed to the Fijiana 7s programme until the World Cup in South Africa, he has also agreed to assist in preparing the Fijian 7s Team and we are thankful and grateful to him for agreeing to take up the challenge of preparing the Fijian 7s Team with the support of Jerry Tuwai."
Gareth Baber's contract was due to expire at the end of the year but the FRU Board approved his request for an early release on November 30.
However, the former Wales and Hong Kong 7s coach will continue to provide support to the Fiji team and to Saiasi Fuli in the lead-up to the Dubai 7s.
Baber said he still had a lot to learn in the 15-man game but he would love to be a part of the Fijian Drua or Flying Fijians coaching staff in the future.
He believed a Fijian was capable of succeeding him with the Fiji 7s team, while CEO John O'Connor said Saiasi Fuli was the only local coach with the necessary experience to be considered for the job.
Fuli was an assistant coach to Gareth Baber in the 2017/18 season and is also a former Fijian Drua assistant coach.
The head coaching role will be advertised globally tomorrow and applications close on November 13.