Fans from overseas had already been banned from the Games but it has been confirmed that spectators from Japan will be allowed.
Capacity will be set at 10,000 spectators, provided it does not exceed 50 percent of a venue's capacity.
The Olympics are scheduled to begin on 23 July, while the Paralympics follow a month later, from 24 August.
Spectator numbers for the Paralympics will be confirmed by 16 July, said a joint statement from the International Olympic Committee, the International Paralympic Committee, the Tokyo 2020 organising committee, the Tokyo metropolitan government and the government of Japan.
The decision to allow spectators comes despite the release of a report last week by Japanese medical experts that said holding the Games without spectators was the "least risky" and most desirable option.
"There are so many cases, domestically and internationally [of] sports events with spectators," said Tokyo 2020 organising committee president Seiko Hashimoto.
"By exercising thorough measures and based on the government criteria, we believe we can hold the Games with spectators.
"The entire world is facing the same issues and we have to work together to overcome them."
Delegates and sponsors will be classed as organisers, and therefore will not be included in the 10,000 spectator limit.
Should there be a rapid increase in Covid-19 infections and impact on Japan's healthcare systems, the five parties will consider further restrictions.
This could include further reducing spectator numbers at venues.
There are fears that the Games may trigger a surge in coronavirus infections in Japan.
As a result, Prime Minister Yoshihide Suga last week encouraged the Japanese public to watch the Games on television, rather than attending events in person.
On Sunday, a state of emergency in Tokyo was lifted, five weeks before the start of the Olympics.
Emergency Covid-19 restrictions had been in place in Japan's capital and other prefectures since late April amid a surge in infections.
However, some "quasi-emergency" measures will remain in some areas, including Tokyo, until 11 July.
On Saturday, a Ugandan athlete because the first to test positive for Covid-19 on arrival in Japan.