Locals said he had done very little to help Cobargo in New South Wales (NSW), where two people died earlier this week and many lost their homes.
The PM said he was "not surprised people are feeling very raw".
NSW's premier has declared a week-long state of emergency in response to the escalating bushfire threat.
Since September, bushfires have killed 18 people and destroyed more than 1,200 homes across NSW and neighbouring Victoria. At least 17 people remain missing after fires this week alone.