"The smugglers deliberately pushed the migrants into the waters since they feared that they would be arrested by the authorities once they reach the shore," an IOM emergency officer in Aden told AFP.
The International Organization for Migration (IOM) said it had found on Wednesday the shallow graves of 29 migrants on a beach in the southern province of Shabwa after they had been buried by survivors.
Then they "simply went back to where they came from to pick more migrants and try to smuggle them into Yemen, again".
IOM says around 55,000 migrants have left the Horn of Africa headed for Yemen since the start of the year, most aiming to find work in Gulf countries.
The journey is especially hazardous at this time of year due to strong winds in the Indian Ocean.
The IOM officer said there were "many women and children among those who died and those who are still missing".
An IOM statement said its staff had provided urgent care to surviving migrants who had stayed on the beach.
It estimated the average age of the migrants on the boat at around 16.
The survivors told the IOM that a smuggler had pushed them to the sea after seeing people who looked like officials, the IOM's Yemen mission head Laurent de Boeck said.
"They also told us that the smuggler has already returned to Somalia to continue his business and pick up more migrants to bring to Yemen on the same route," he said.
"This is shocking and inhumane."
Despite a war that has cost more than 8,000 lives since March 2015 and brought the country to the brink of famine, Yemen continues to attract people fleeing the Horn of Africa.
Several refugee camps in southern Yemen host Somali refugees.
Already one of the Arab world's poorest countries, the country has been rocked by years of conflict between Shiite Huthi rebels and the government of President Abedrabbo Mansour Hadi, backed by a Saudi-led coalition.
In March, a helicopter opened fire on a boat carrying over 140 passengers off Yemen's Red Sea coast, killing 42 civilians and wounding 34.
A confidential United Nations report seen by AFP in July said the attack was almost certainly carried out by the Saudi-led coalition, in an operation it said was a violation of international humanitarian law.
It said coalition forces were the only parties to the conflict able to operate armed utility helicopters in the area.