The incident happened off the coast of Mamutik Island, one of Sabah's most visited diving locations at 2:20 p.m. on Saturday, The Star reported.
The victim was found unconscious and he was brought ashore by lifeguards.
The lifeguards administered cardiopulmonary resuscitation (CPR), but were unable to save him, Daily Express Malaysia reported.
The victim was taken to a local hospital for emergency care, where he was pronounced dead by a doctor.
According to initial investigations, he and seven other Chinese tourists were visiting the island together to go diving.
Police suspected the victim had drowned accidentally as investigators found no signs of criminality or foul play surrounding the incident.
An autopsy will be carried out to identify the cause of death.
Malaysia has recently become a top destination for Chinese tourists following a new visa waiver policy.