Video footage showed a plume of black smoke filling the sky, with huge flames coming out from the roof of the building.
President Cyril Ramaphosa visited the scene and called it a "terrible and devastating event".
A 51-year-old man was being held and questioned by the authorities, police said.
The blaze, which began shortly after 6am local time came the day after Archbishop Desmond Tutu's state funeral at St George's Cathedral, near Parliament.
Ramaphosa said news of the fire was a "terrible setback to what we were basking in yesterday" and added that Archbishop Tutu would also have been devastated.
He said the building's sprinkler system had not functioned properly, and praised firefighters for responding to the fire in minutes. Without their intervention the National Assembly would have been reduced to "ashes", he said.
Dozens of firefighters battled the flames. Officials later said that the entire parliamentary complex was severely damaged.
Jean-Pierre Smith, a member of the Cape Town mayoral committee for safety and security, told reporters that the roof above the old assembly hall was "completely gone" and further damage inside the old chamber had not yet been evaluated.
"It is not possible to see whether it's damaged. We hope it is not because it has so many historical artefacts, but you can't gain access to it without breaking the doors down and we don't want to do that," he said.
He also said that parliament's fire alarm only rang when firefighters were already on site.
Officials said the fire started on the third floor offices and quickly spread to the National Assembly (the Parliament's lower house) chamber, reports the BBC's Nomsa Maseko in Cape Town.
No injuries have been reported and there's no indication of what may have started it, our correspondent adds.
Fire and rescue service officials said it could take several more hours to put the fire out completely because of the carpets and wooden floors in the building.
The Parliament is not currently in session because of the holidays.
The Houses of Parliament in Cape Town are made up of three sections, with the oldest dating back to 1884. The newer sections built in the 1920s and 1980s house the National Assembly. Meanwhile, the government is based in Pretoria.
It is the second fire at the parliament in under a year. In March there was a fire caused by an electrical fault.
Last year, a fire ravaged part of the University of Cape Town's library, which was home to a unique collection of African archives.