Accountants warn Labor's plan to end cash handouts for share investors will push more people onto the aged pension, but social groups say the change will help fund key services.
Labor wants to reform the system that allows share investors, who technically don't earn a taxable income, to get government cash refunds for share dividends.
"If millionaires are getting hundreds of thousands of dollars of taxpayer money, that's not a welfare system I'm going to back," Opposition Leader Bill Shorten told reporters on Wednesday.
But the Institute of Public Accountants says the system has helped keep retirees off the aged pension.
"Australia should be looking at every avenue possible to reduce long-term reliance on government-funded pensions," chief executive Andrew Conway said.
"This proposed measure will deter entrants from investing in their future self-funded retirement."
Labor says about 14,000 full pensioners and 200,000 part pensioners will be affected by the change.
The Australian Council of Social Services said only 16 per cent of people older than 64 pay income tax, and many who don't are quite well off.
"We have a choice. We close gaps like these in the tax system, or we charge people more for services like aged care and home care," chief executive Cassandra Goldie said.
Malcolm Turnbull said Labor's plan will hit retirees who rely on the cash payouts because they technically don't earn a taxable income on their share portfolios.
"Have a word to your parents and they'll tell you how they feel about it," the prime minister told reporters on Wednesday.
"Shorten's budget plans are completely out of control and now he's robbing pensioners to pay for his incompetence."
The original scheme was introduced under Paul Keating to make sure company profits weren't taxed twice - once with corporate tax and again via personal income tax.
But changes under John Howard in 2000 allowed investors to get a cash refund from the government if their tax imputation was more than the tax they owed.
Labor will abolish these refunds - which cost $8 billion a year - if elected.