Under plans unveiled by the Trump administration on Tuesday, almost all people staying in the US illegally can be subject to deportation.
Mexican Foreign Minister Luis Videgaray said his country could not "accept unilateral decisions imposed by one government on another."
The row comes as two top US officials are preparing to visit Mexico.
The White House said Secretary of State Rex Tillerson and the head of Homeland Security, John Kelly, would talk to Mexican officials and "walk through" the implementation of the new immigration guidelines.
They include plans to enforce an existing provision of the US Immigration and Nationality Act that allows authorities to send undocumented migrants back to Mexico, regardless of where they are from.
It is unclear whether the US has authority to force Mexico to accept foreigners.
But Mr Videgaray said on Wednesday: "We are not going to accept that because we don't have to and it is not in the interest of Mexico."
Mr Videgaray also warned the US about treatment of Mexican citizens.
"The Mexican government will not hesitate in going to international organisations, starting with the United Nations, to defend human rights, liberties and due process for Mexicans abroad according to international law."
Mr Tillerson and Mr Kelly were scheduled to discuss a range of issues with President Enrique Pena Nieto on Thursday, but Mr Videgaray said the new changes to policy would now dominate the discussion.
White House Press Secretary Sean Spicer said on Wednesday that he expected a "great discussion".
An estimated 11 million undocumented immigrants live in the US, many of them from Mexico. President Donald Trump made immigration and border control a key part of his campaign.
The new priorities are broad enough to apply to almost any illegal immigrant, including anyone who has been charged with a crime, misrepresented themselves, poses a risk to public safety, or "have abused any program related to receipt of public benefits".
The new guidelines also allow Border Patrol and Immigration and Customs Enforcement to deport people immediately.
Expedited deportations can now be used against undocumented immigrants who are unable to prove they have been in the country for more than two years, anywhere in the US.
Previously, expedited removals were applied to people who had been in the country for less than two weeks and were within 100 miles of the border.