Grant said he had always endured racism in his career but it had escalated after he covered the King's Coronation for national broadcaster ABC.
The veteran Aboriginal journalist had spoken during the coverage about the impact of colonisation on his people.
The ABC has called for the "grotesque" abuse against the host to stop.
But Grant also accused his employer of an "institutional failure" to protect or defend him.
Grant has won several journalism awards over a four-decade career and in 1992 he became the first Aboriginal prime-time host on Australian commercial TV.
But on Friday, he announced he was indefinitely stepping away from his roles hosting the ABC's flagship Q+A panel discussion show and writing a weekly column online.
"Racism is a crime. Racism is violence. And I have had enough," the Wiradjuri man wrote.
"I want no part of it. I want to find a place of grace far from the stench of the media."
Mr Grant said he was invited to be part of the ABC's Coronation coverage specifically to talk about the legacy of the monarchy.
During the segment, he said the symbol of the Crown "represented the invasion, the theft of land - and in our case - the exterminating war", referring to a period of martial law in 1820s New South Wales that was used to justify the killings of Wiradjuri people.
The discussion divided people online and some people made formal complaints to the ABC about its appropriateness.
On Friday, Grant accused some "people in the media" of distorting his words and depicting him as "hate filled", inflaming racist abuse against him.
He said he apologised if his own comments had caused offence but that the "hard truths" were spoken out of love for Australia.
"No-one at the ABC... has uttered one word of public support. Not one ABC executive has publicly refuted the lies written or spoken about me," he wrote.
In a statement, ABC News director Justin Stevens described Grant as "one of Australia's best and most respected journalists" and said his treatment had been "abhorrent".
Stevens did not address the frustrations Grant aimed at the organisation, but said "the ABC stands by him".
Of the Coronation segment, the news director added that it was "regrettable" that it had elicited "a strong response from some viewers".
"Any complaints, criticism - or vitriol - regarding the coverage should be directed to me, not to him," he said, adding the ABC would continue to refer threats to police.
Grant's announcement has triggered an outpouring of tributes from peers across the media industry.
"Stan Grant is an Australian icon, a serious journalist, a leader in this country. This is a sad and disgraceful result," newspaper columnist Sean Kelly wrote.