The alleged assailant was said to have stabbed a woman, before firing a hunting rifle to kill two policemen in the city of Nakano, Nagano prefecture. A fourth death was later confirmed.
BBC reports the suspect is the son of the city assembly's speaker, local media say.
Gun violence remains extremely rare in Japan, despite the killing of ex-Prime Minister Shinzo Abe in July last year.
According to Japanese news agency Kyodo, police received a call at around 4:25pm local time about a man who had chased and then stabbed a woman.
An eyewitness working in a nearby field told Kyodo that the man's attack on his first victim had been carried out with a blade around 30 centimetres long.
His motive was not clear. When the witness asked the suspect why he had stabbed the woman, he was said to have replied: "I killed her because I wanted to."
The suspect - who was reportedly wearing a camouflage uniform, a hat, sunglasses and a mask - then allegedly shot police officers who responded to the emergency.
It was not immediately clear how the fourth person - described as an elderly woman - had died.
The suspect reportedly barricaded himself for hours inside the home of his father, Masamichi Aoki, before being detained.
Abe was Japan's longest-serving prime minister and his death profoundly shocked a country where handguns are banned and incidents of political violence are almost unheard of.
In 2014, there were just six incidents of gun deaths in Japan, compared with 33,599 in the US.
People have to undergo a strict exam and mental health tests in order to buy a gun. Even then, only shotguns and air rifles were allowed.
Armed police officers prepare to storm near the house where a suspect has barricaded himself with a hunting gun in Nakano City, Nagano prefecture on 26 May. Photo: Hiroto Sekiguchi / The Yomiuri Shimbun via AFP