The animal, weighing 20 to 25 kilograms (44 to 55 pounds) and named Whollydooleya tomnpatrichorum, is a distant and bigger cousin of Australia's largest living flesh-eating marsupial, the Tasmanian Devil.
It is the first creature to be formally identified from a range of strange new animals whose remains have been found at a fossil site in remote northwestern Queensland.
"W. tomnpatrichorum had very powerful teeth capable of killing and slicing up the largest animals of its day," said University of New South Wales professor Mike Archer, the lead author of a study into the find published in the Memoirs of Museum Victoria.
The late Miocene period between 12 and five million years ago, when Australia began to dry out and megafauna began to evolve, is one of the least understood in the vast continent's past, he added.
Fossils of land animals from this time are extremely rare.
"Fortunately, in 2012, we discovered a whole new fossil field that lies beyond the internationally famous Riversleigh World Heritage Area fossil deposits in northwestern Queensland," said Archer.
With a grant from the National Geographic Society, Archer and his colleagues began to explore the "New Riversleigh" site in 2013 and the species' highly distinctive molar was one of the first finds.
"New Riversleigh is producing the remains of a bevy of strange new small to medium-sized creatures, with Whollydooleya tomnpatrichorum, the first one to be described," said Archer.
"These new discoveries are starting to fill in a large hole in our understanding about how Australia's land animals transformed from being small denizens of its ancient wet forests to huge survivors on the second most arid continent on Earth."
Team member Suzanne Hand said medium- to large-sized Australian Late Miocene creatures have previously been known from fossil deposits in the Northern Territory.
"But those deposits give almost no information about the small to medium-sized mammals that existed at the same time, which generally provide more clues about the nature of prehistoric environments and climates," Hand said.
Related news: