The office in the Northern city of Kaesong was opened in September as the two Koreas knitted closer ties, but the South's vice unification minister Chun Hae-sung told reporters Pyongyang had "notified the South they are pulling out of the liaison office".
The decision had been taken "in accordance with an order from an upper command", he said, adding: "They said they didn't care whether we stayed at the liaison office or not."
The South's President Moon Jae-in was instrumental in brokering talks between the nuclear-armed, sanctions-hit North and Washington, Seoul's key security ally.
Moon has long backed engagement with the North to bring it to the negotiating table, and has been pushing the carrot of inter-Korean development projects, among them the restarting of an industrial zone also in Kaesong and lucrative cross-border tourist visits by Southerners to the North's picturesque Mount Kumgang.
But the sanctions currently in place effectively block their resumption, while a preliminary study for a plan to renovate the North's decrepit rail system was repeatedly delayed.
Questions were even raised over whether supplies provided to set up the liaison office were a sanctions violation.
The failure by Kim and Trump to reach agreement in Hanoi last month on walking back Pyongyang's nuclear programme in exchange for relaxation of the measures against it has raised questions over the future of the wider process.
In Vietnam both sides expressed willingness to talk further, but it has since emerged that Washington presented Kim with a wider definition of what it regards as denuclearisation.
A senior Pyongyang diplomat told reporters last week that the North was considering suspending nuclear talks with the U.S.
Analysts said Friday's decision could be a sign Pyongyang felt Seoul was unable to exert sufficient influence on Washington.
"With the pull-out, the North is pressuring the South to do more as a middle man between Pyongyang and Washington after it didn't get the resumption of the Kaesong industrial complex and Mount Kumgang tours," said Yoo Ho-yeol, professor of North Korean studies at Korea University.
"It could be seen as either pressure, or a warning," he told AFP. "Internally, Pyongyang could use the withdrawal as a propaganda message to its people that it is taking a lead when it comes to inter-Korean relations."
The North has recently summoned several of its top diplomats around the world back to Pyongyang.
Cheong Seong-chang, a senior researcher at the private Sejong Institute, said that move and Friday's pull-out could signal "that the North is considering a shift in denuclearisation strategy and foreign policy".
It was "hard to rule out a hardline statement", he added.
In his New Year speech -- a key political event in the North -- Kim said without giving details that Pyongyang might see a "new way for defending the sovereignty of the country and the supreme interests of the state" if the US persisted with sanctions.
'Round-the-clock consultation'
Seoul sought to keep the door open to more contact.
"We regret the North's decision," vice minister Chun said. "Though North Korea has pulled out, we will continue to work at the liaison office as usual."
The facility opened three months after Kim signed a vague pledge at his first summit with Trump in Singapore to work towards "denuclearisation of the Korean peninsula", and shortly before Moon went to Pyongyang for his third summit with Kim last year.
It stands in a city that was initially part of the South after Moscow and Washington divided Korea between them in the closing days of World War II, but found itself in the North after the 1950-53 Korean War, which ended in an armistice rather than a peace treaty.
The four-storey building includes separate Northern and Southern offices and a joint conference room.
When it opened Seoul's unification ministry said it would become a "round-the-clock consultation and communication channel" for advancing inter-Korean relations, improving ties between the US and the North, and easing military tensions.
But the Hanoi summit took place without the usual several rounds of preliminary negotiations between lower-rank officials, and broke up without even a joint statement.
A top security adviser to Moon, Moon Chung-in, told AFP last week that Pyongyang needed to take "actual action" on denuclearisation to persuade the US to grant concessions.
The South's presidency held an emergency meeting of the National Security Council after the North pulled out of the liaison office on Friday.