The International Paralympic Committee announced that Samoa, Kiribati, Tonga and Vanuatu will no longer compete at the Games which are due to start on Tuesday 24 August.
Spokesperson for the International Paralympic Committee, Craig Spence, told Inside the Games all four nations were left financially hamstrung by Australia's two-week hotel quarantine rule on international travellers.
"We have a number of nations from the Pacific Islands that will be unable to travel... they would need to travel from their country through Australia where there is strict rules at the moment," he said.
"They would have to quarantine at their expense for two weeks before flying here, they would then need to quarantine on the way back, so effectively they would need to pay for four weeks extra at a hotel... and they just do not have the resources to be able to accommodate that expense."
President of the Vanuatu Paralympic Committee, Margaret Macfarlane, told RNZ Pacific the decision to withdraw Vanuatu from the 2020 Games was made collectively last month based on concerns about the Delta virus.
"The committee, our athletes, our coaches, we assessed the risks, it just seemed at the time to be an extremely risky thing to undertake," she said.
Margaret Macfarlane said uncertainty that any travel back to Vanuatu could entail the risk of additional quarantine periods was another factor.
"We only have two flights a week in and out of Vanuatu and this puts enormous timing and operational constraints on finding safe routes to Tokyo and then back," Ms Macfarlane said.
She said on top of all of this was the huge financial cost.
"Vanuatu Paralympics is self-funded. We have no core funding from any agency and so our finances were very constrained," she said.
Ms Macfarlane said even with all these considerations it was still a tough decision.
"It was a terrible decision and a terribly hard one to make because we had over the last three or four years strong support from the Oceania Paralympic Committee, it really brought Vanuatu Paralympic Committee to a very high level of operations, but we had to make that decision," she said.
"I think these are similar reasons to why the other committee members were forced into withdrawing as well. It was just so difficult to find that really narrow window of opportunity to either go or come back on specific dates."
Three athletes had qualified for Vanuatu with Elie Enock set to compete in discus, shot put and javelin, Marceline Moli ready to participate in long jump and Ken Kahu, Vanuatu's first-ever prequalified Para athlete, competing in the javelin event.
Margaret Macfarlane said the athletes are now focused on national games and the Commonwealth Games scheduled to take place in Birmingham in July 2022.
"We are refocusing and we are looking at a national games here in Vanuatu in November and then of course we have the Commonwealth Games in July 2022 in Edinburgh, and we're refocusing now our efforts on that."
But Ms Macfarlane said even these plans have been affected by the withdrawal from Tokyo because it was also the pre-qualification event for Birmingham.
"Because we haven't got athletes there, the pre-qualification pathway is going to be quite difficult, so at the moment the doors are closed but we're hoping we might be able to wedge it open somehow."