The bodies of two of them were found at the scene while the third was washed away and his body was found a few hours later.
The tragic accident occurred on Saturday evening when the boat had traveled around two kilometers from Diep Son Island and a dozen passengers fell into the water.
Tran Chi Linh, who saw and tried to rescue the passengers with his fishing vessel, said: "My friend and I threw a rope and used it to pull people up. Everyone was panicking. I am very sad I could not save all of them."
The bodies of the victims have been handed over to their families.