Takumi Minamino, Kaoru Mitoma and Maya Yoshida were all denied by Livakovic before Mario Pasalic coolly sent Japan keeper Shuichi Gonda the wrong way to set up a last-eight date for the Croatians with Brazil or South Korea on Saturday morning (NZ time).
Reuters reports three times in the knockout rounds in Russia four years ago Croatia came from behind to advance after extra time before losing to France in the final.
They once again showed their resilience and patience by coming from a goal down to tame a lively Japan team who had beaten Germany and Spain in the group stage.
Daizen Maeda opened the scoring for Japan in the 43rd minute and Ivan Perisic headed the equaliser 10 minutes after the break but the teams could not be separated over the remainder of the game.
For Japan, this was last-16 heartache for the fourth time after a loss to Turkey in 2002, exiting on penalties against Paraguay in 2010 and giving up a 2-0 lead to lose to a stoppage- time goal against Belgium four years ago.
"Every day, for four years, we worked hard to break this barrier (by reaching the final eight) but we couldn't get the result we wanted," captain Yoshida said.
"It's really hard to take."
After all their hard work in topping a difficult group and matching the Croatians for 120 minutes, Japan crumbled in the shootout with only Takuma Asano converting his penalty.
Nikola Vlasic and Marcelo Brozovic put Croatia 2-0 ahead and even though Marko Livaja hit the post with his spot-kick, Pasalic was able to seal the deal with the fourth penalty.
The game was more open than anyone had reason to expect, with Croatia ratcheting up the physicality to gradually take control of midfield and Japan trying to hit them on the break.
Perisic was put through on goal in the eighth minute but Gonda pushed away his shot and Bruno Petkovic also had a one-on-one, only to tap an attempted pass tamely through the Japanese goalkeeper's legs.
Japan showed far more adventure than they had in the first half of their group matches and Shogo Taniguchi, Daichi Kamada and Maeda all had chances to open the scoring before the latter broke the deadlock in the 43rd minute.
Ritsu Doan took the ball from a short corner and curled it into the box where Bruno Petkovic deflected it back across the six-yard area under close attention from Yoshida and Maeda pounced to tuck it into the net.
The equaliser came 12 minutes later with a cross from the same flank, Dejan Lovren lofting a beauty into the box and Perisic getting in front of his marker to head the ball powerfully into the bottom right corner of Gonda's goal.
That brought Japan out of their shells and Wataru Endo was soon firing in a long-range effort which Livakovic tipped over the bar.
Luka Modric had a sweetly struck shot from range turned over by Gonda in the 63rd minute and substitute Ante Budimir steered header wide of the post.
Gonda again had to be at his best to keep out a rocket of a shot from Perisic in the 77th minute, while at the other end Japan continued their raids but without carving out any clear-cut chances.
Extra time was a little more ragged although a thrilling run and piledriver of a shot from Japan substitute Kaoru Mitoma brought a fine save out of Livakovic just before the break.