Dozens of officers blocked off both ends of a section of Tran Quang Khai Street in District 1, a 15-minute drive from Ben Thanh Market, before raiding Thanh Vinh Restaurant.
In so-called VIP rooms, dozens of waitresses in skimpy clothing were found flirting with drunken male customers, they said.
The police said the restaurant has committed many violations like offering sex services.
Another group of officers raided a nearby hotel and caught two women having sex with guests. The women confessed they were waitresses at the restaurant and sent to the hotel to have sex with drunk customers for VND2 million ($86).
Three restaurant managers were arrested for brokering sex services for which they received a VND1 million ($43) bonus each time they had a client.
The raid was part of an ongoing campaign to crack down on dubious and often illegal activities by restaurants and bars, particularly in the tourist area.
Official data estimates there are 3,000 sex workers in Saigon. A study by the International Labor Organization suggested that sex workers are among the most vulnerable in Vietnam as they have to deal with regular police raids, risks of sexually transmitted diseases, and persistent fear of theft and violence.
Sex workers and their customers are not considered criminals under Vietnam laws, but those brokering the service can face jail terms.