During a French naval mission to the small uninhabited islands in January, France reportedly took action to re-enforce its sovereignty, when officers landed, polished a plaque and painted the French flag on a rock.
Johnny Koanapo, who has represented Vanuatu's claim in talks with France, said the unilateral move complicated the negotiations.
"We, the people of Vanuatu, will not take this lightly. We are very concerned and we want France to give a proper explanation," he said.
Vanuatu's foreign minister Ralph Regenvanu summoned the French representative in Port Vila last week over the matter and a letter was being drafted seeking an explanation to send to the French president, according to Mr Koanapo.
The MP represents the island of Tanna, whose chiefs had historically visited Matthew and Hunter for rituals and other ceremonial activities.
He said France's presence had scared his people off visiting as they used to do.
"When France effectively occupied the territory, not giving it back to us after the decolonisation period, it made it difficult. So for our old people who used to go down, they no longer go down.
"It's disrespecting our people, disrespecting our culture, which is not good," Mr Koanapo said.
While Vanuatu had always acknowledged the islands were disputed, it was still waiting for France to be "definite" about its position, according to Mr Koanapo, who was Vanuatu's chief negotiator during the first talks in Sydney.
He said it was important for the two sides to get to the table, although he feared the latest "nonsense" would complicate matters.
"Dialogue and negotiations (are) part of the diplomatic approach which any country must take in order to resolve a situation, so we haven't lost faith in negotiations because this is where we should be able to find a solution."
"We had agreed we would hold the negotiations last year in June . We were waiting for France to come back. France never came back so I hope they take this seriously because it affects our identity," he said.
"It's hypocritical on the part of France to be talking about the indigenous culture, the indigenous peoples and not allowing Vanuatu to go to the two islands to perform its cultural activities as our old people have always been doing," Mr Koanapo added.
With talks apparently stalled, Vanuatu is pressing for a meeting between the two sides at the end of the month.
According to reports, Vanuatu is planning commemorations to mark the anniversary, this weekend, of Vanuatu's own flag-raising expedition on Hunter Island in 1983.