Nong Thi Nghiep, chairwoman of Duyet Trung Ward in Cao Bang Town, said it happened at 2 a.m.
Those killed included a 29-year-old woman, a five-year-old girl and a two-year-old boy who were crushed under the debris.
Another sustained minor injuries and was taken to a local hospital.
Heavy rains washed rock and soil down a mountainside and onto the three-meter-high embankment built by locals, Nghiep said.
The embankment fell on the house, knocked down its roof and walls.
Due to the influence of cold air, the northern mountains have been hit by heavy rains since April 28.
At least 1,635 houses and more than 2,500 hectares of rice and other crops have been damaged so far.
Cao Bang has received rainfall of up to 80mm, which has damaged 127 houses and 51 hectares of crops, mainly in Bao Lac and Quang Hoa districts.