Big firms also plan to cut capital expenditure plans in the current fiscal year, underscoring the challenges the BOJ faces in aggressively printing money to nudge risk-shy companies into boosting spending.
To keep the economy afloat, Abe may have to delay yet again a sales tax hike scheduled for next year.
The weak reading will also keep the BOJ under pressure to loosen monetary policy further when it reviews its quarterly forecasts later this month, some analysts say.
"This data confirmed the very cautious stance of Japanese firms reflecting the market volatility since January. There's no signs of corporate sentiment bottoming in coming months," said Mari Iwashita, chief market economist at SMBC Friend Securities.
"There's more than a 50 percent chance the BOJ will consider easing policy further this month."
The headline index gauging big manufacturers' sentiment stood at plus 6 in March, half the level seen three months ago and worse than a median market forecast of plus 8, the survey showed.
It was the lowest reading since June 2013, as exporters felt the pinch from sluggish emerging market demand and yen rises that erode their competitive advantage overseas.
Big non-manufacturers' sentiment dropped to plus 22 from plus 25 three months ago, deteriorating for the first time in six quarters, as spending by overseas visitors moderated.
Sentiment soured for manufacturers and non-manufacturers big and small for the first time since June 2014, when Japan was reeling from the blow of a sales tax hike in April of that year.
Both big manufacturers and non-manufacturers expect sentiment to worsen in the coming three months, underscoring companies' concern over the murky global economic outlook.
"The slowdown in emerging economies, particularly that of China, has affected business sentiment. The impact was big for big manufacturers with direct exposure to global economic conditions," a BOJ official told reporters in a briefing.
The yen's rise is also expected to hurt profits.
Big manufacturers based their business plans on the assumption the dollar would average 117.46 yen in the current fiscal year, higher than recent levels around 112 yen, suggesting that corporate profits could be revised down ahead.
In a sign uncertainty over the global outlook is hurting companies' spending plans, big firms plan to cut capital expenditure by 0.9 percent for the current fiscal year from year-before levels.
Japan's economy shrank in October-December due to weak exports and consumption. Many analysts now expect it to have contracted again in January-March, which would put the country back into recession, commonly defined as two straight quarters of negative growth.
The gloomy survey will add pressure on the BOJ to do more to prop up growth, though many central bankers are reluctant to ease again soon with financial institutions still struggling to digest the impact of January's controversial decision to adopt negative interest rates.
The tankan showed financial institutions' sentiment deteriorating sharply in March, most likely due to the pain they are feeling from negative rates and dwindling profit margins.
The government, for its part, is already considering compiling a fresh fiscal stimulus package and delaying next year's sales tax hike to prevent external factors from disrupting a fragile economic recovery.