"At 7:14 a.m. (2214 GMT), four Chinese nationals were rescued but we're still searching for the remaining 18 members," a coast guard spokeswoman told AFP.
"A plane is already at the scene and two ships are on the way," added the official, who declined to be named.
The ship issued a call for help around 11:15 p.m. local time on Tuesday, the coast guard said.
Its location, around 110 kilometers (65 miles) west of the remote and uninhabited Danjo islands in far southwestern Japan, has made reaching the site difficult.
The accident came as a cold snap hit much of Japan, with heavy snow falling in some areas and daytime temperatures in some of the islands nearest the rescue site reaching just three degrees centigrade (37 degrees Fahrenheit).
The coast guard identified the vessel as a 6,651-ton cargo vessel named Jintan and registered in Hong Kong.
The crew on board included 14 Chinese nationals and eight people from Myanmar, the coast guard said.