A drone laden with explosives struck the building, injuring six of his bodyguards in an apparent assassination attempt, officials said.
Mr Kadhimi called for "calm and restraint from everyone".
The attack, which comes after violent unrest over recent election results, was condemned by both the US and Iran.
Security sources say three drones were used in the attack, launched from near Republic Bridge on the River Tigris, but two were shot down.
