Prime Minister Malcolm Turnbull last year announced two deals between the US and Australia, with the latter resettling refugees from Costa Rica while the US took in refugees from Manus Island and Nauru.
Neither government had publicly linked the two deals until Tuesday, when Mr Dutton told Sky News that Australia would not resettle refugees from Costa Rica until the US had reassured the Government over the second deal.
He added that he had no problem with people describing the two deals between the US and Australia to resettle refugees as a people swap, prompting Foreign Minister Julie Bishop to dismiss the characterisation.
The Immigration Minister today dismissed the response to his comments as a "storm in a tea cup".
Mr Dutton told 2GB he had been clear the two deals were not linked.
"In the end, people can use whatever language they want," he said.
"I'm not going to get bogged down in nuance and discussion
"I'm a pretty frank speaker and I've clear that it's not a people swap. They are two separate arrangements."been
Mr Dutton also accused Labor of playing "word games" after his Opposition counterpart Shayne Neumann called for clarification on the issue.
Mr Neumann told Sky News today the Government needed to release the terms of the agreements.
"Was Peter Dutton going solo?" he asked.
"Was he retrospectively putting demands on the Americans or was he retrospectively changing the terms of the agreements?
"We don't know. The Government needs to clarify."