The Japanese government and International Olympic Committee decided in March to postpone the 2020 Olympics by a year due to the global pandemic.
But as Japan grapples with a third wave of infections, the NHK poll shows 32% of respondents wants the Games to cancelled entirely.
Only 27% said they should go ahead.
According to an NHK poll in October, 40% said the Olympics should be held as planned with only 23% favouring cancellation and 25% preferring further postponement.
While Japan has avoided the vast number of infections and deaths seen in many other countries across the world, a recent resurgence in cases has forced the government to re-introduce steps such as requesting bars and restaurants to close early.