Keith Mills told BBC radio that the delayed Olympics, scheduled for July-August, were looking unlikely to happen in his opinion due to the COVID-19 pandemic.
"If I was sitting in the shoes of the organising committee in Tokyo, and thankfully I'm not, I would be making plans for a cancellation," he said.
"I'm sure they have plans for a cancellation but I think they will leave it to absolutely the last minute in case the situation improves dramatically and in case the vaccines roll out faster.
"It's a tough call and I wouldn't like to be in their shoes."
Japanese Prime Minister Yoshihide Suga has vowed to forge ahead with preparations for the Games in the face of growing public opposition as Japan battles a surge in coronavirus infections.
Recent media polls showed close to 80% of Japanese believe the Olympics, already postponed by a year because of the pandemic, should be delayed again or cancelled entirely.
World Athletics head Seb Coe, who was the chairman of the 2012 organising committee, told Sky News he did not think the Games would be cancelled but they would be a challenge and a very different experience.
"Of all the countries on the planet that really has the fortitude, and resilience and the street-smarts to see this through, it is actually Japan," he said.
"I wake up as a federation president really grateful that it is Japan that's dealing with this and not some other places that I can think of. So I'm sure we will be there.
"I think there will be big issues around crowds and distancing the athletes. Just think about the village, you've got 10,500 athletes and probably another 7,000 support staff in there and they're all probably wanting to eat at roughly the same time," he added.
"I think the Games will take place but they will look different."