Jordan Petaia, Folau Fainga'a and Pete Samu scored tries in the last quarter as Australia recorded their first win over England since knocking the co-hosts out of the 2015 World Cup.
Samu's try took Australia to a 30-14 lead but England debutants Henry Arundell and Jack van Poortvliet scored tries in the dying minutes to make the scoreline more respectable.
It was quite a turnaround for the Wallabies, who must have thought they were cursed after the first 34 minutes of the match.
They lost flyhalf Quade Cooper to a calf injury in the warm- up, fullback Tom Banks with a broken forearm after a nasty fall 21 minutes in, and Swain to a red card for a headbutt just after the half hour mark.
The Wallabies edged ahead from Lolesio's third penalty four minutes after the break but England prop Ellis Genge rolled over the line on the back of a maul five minutes later to inch England back in front at 11-9.
Australia hit back after 64 minutes, putting together their first sustained attack of the match then spreading the ball wide for winger Petaia, on for Banks, to finish well in the corner.
England number eight Billy Vunipola was sent to the sin bin in the 67th minute for a high tackle on Michael Hooper and Australia had scored two more tries before his return.
Hooker Fainga'a danced away from a rolling maul to touch down in the 69th minute and loose forward Samu crashed through a series of tacklers to take Australia's lead out to what proved to be an unassailable 16 points.
The series continues over the next two weeks with tests at Brisbane's Lang Park and the Sydney Cricket Ground.