The Liberal Democratic Party (LDP), which has governed Japan almost continuously since 1955, has been convulsed by a series of corruption scandals in recent years, including one over kickbacks to lawmakers that sank Ishiba's predecessor.
Just 26% of voters polled by the Asahi Shimbun daily expressed support for Ishiba's Cabinet, a dramatic fall from the 40% who felt similarly in the same survey in February.
A separate survey by the Yomiuri Shimbun daily put support for the administration at 31%, down from 39% in February.
Both were the lowest since Ishiba took office in October.
Ishiba, 68, has come under fire for handing out gift vouchers each worth ¥100,000 (US$670) to 15 new LDP lawmakers - a move he has defended as legally sound and not a political donation.
Grilled by opposition MPs in parliament, Ishiba apologized on Friday "for causing trouble and worry to many people" over the voucher scandal.
Ishiba said the vouchers - which he paid for personally - were intended as a token of appreciation, not a donation, for the families of lawmakers who took office for the first time after last year's general election.
On Monday, Ishiba apologized again, saying "there was a discrepancy between the popular conception" and his practice of sending gift vouchers.
The Asahi poll showed that 75% of people thought the gift voucher distribution was problematic, against 23% who thought it was not.
However, 60% thought Ishiba should stay as the premier, compared to 32% who wanted him to step down, according to the poll of 1,137 voters.
In the Yomiuri poll, 75% saw the gift vouchers as a problem, against 19% who did not.
Japanese media reported that flagging support could provoke attempts within the LDP to pressure Ishiba to step down ahead of an election in July for parliament's upper house.
Ishiba held a snap general election in October after being selected as prime minister, but voters angry over corruption within the LDP and inflation dealt him a blow.
They deprived his coalition of a majority in parliament's powerful lower house, in the worst election result in 15 years for the LDP, which has governed Japan almost continuously since 1955.
Since the election, the party and its junior coalition partner Komeito have needed opposition support to pass legislation.
A third poll by the Mainichi Shimbun showed 23% of voters supported Ishiba's Cabinet, down from 30% in February, with 78% calling the vouchers a problem.