Tokyo says the fleet included six coastguard ships, three of which appeared to be armed.
Japan has summoned Chinese diplomats to protest.
Over recent years, Beijing has become increasingly assertive about its rights in waters it believes are Chinese.
Testing Japan's resolve
In this latest incident, Chinese boats sailed close to the Senkaku Islands, which are administered by Japan but also claimed by China. Beijing calls them the Diaoyu Islands.
