The 101 migrants have been under government care in Port Vila since November, after they were freed from their traffickers by police.
A representative for the men, Shahin Khan, said a group of 16 migrants left Port Vila on a flight to Bangladesh Wednesday morning.
The International Organisation for Migration is paying for and arranging their flights.
Some of the men - who were brought to Vanuatu over the past two years - are refusing to return to Bangladesh.
Bangladesh officials have said around 70 men will return to Bangladesh in different phases, with the remaining victims to stay as witnesses in the trial four people charged in connection with their trafficking.
Mr Khan said 11 victims had refused to sign forms expressing willingness to return to Bangladesh and were now facing pressure by Vanuatu police to sign.