The announcement marked one of Russia's most significant retreats and a potential turning point in the war, now nearing the end of its ninth month.
In televised comments, General Sergei Surovikin, in overall command of the war, reported to Shoigu that it was no longer possible to keep Kherson city supplied.
"Having comprehensively assessed the current situation, it is proposed to take up defence along the left (eastern) bank of the Dnipro River," said Surovikin, standing at a lectern and indicating troop positions on a map whose details were greyed-out for the TV audience.
"I understand that this is a very difficult decision, but at the same time we will preserve the most important thing - the lives of our servicemen and, in general, the combat effectiveness of the group of troops, which it is futile to keep on the right bank in a limited area."
The news followed weeks of Ukrainian advances towards the city and a race by Russia to relocate more than 100,000 of its residents by ferrying them to the opposite side of the river.
Kherson is the main city of the region of the same name - one of four Ukrainian regions which President Vladimir Putin proclaimed in September he was incorporating into Russia "forever", and which the Kremlin said had now been placed under Moscow's nuclear umbrella.
Shoigu told Surovikin: "I agree with your conclusions and proposals. For us,the life and health of Russian servicemen is always a priority. We must also take into account the threats to the civilian population.
"Proceed with the withdrawal of troops and take all measures to ensure the safe transfer of personnel, weapons and equipment across the Dnipro River."
The announcement had been anticipated by Russia's influential war bloggers, who described it as a bitter blow.
"Apparently we will leave the city, no matter how painful it is to write about it now," said the War Gonzo blog, which has more than 1.3 million subscribers on Telegram.
"In simple terms, Kherson can't be held with bare hands," it said. "Yes, this is a black page in the history of the Russian army. Of the Russian state. A tragic page."
Compounding the sense of Russian disarray in Kherson, Moscow's number two official there, Kirill Stremousov, was killed in a car crash on Wednesday.
Stremousov was one of the most prominent faces of Russia's occupation. Ukraine viewed him as a collaborator and a traitor.
In a video statement only hours before his death, Stremousov denounced what he called Ukrainian "Nazis" and said the Russian military was in "full control" of the situation in the south.