Here’s a mystery of the modern economy: Growth is picking up, the job market is improving and most government data show the economy on the mend. Yet the parts of the private sector that ought to be enjoying a robust recovery aren’t.

The sliding fortunes of Walmart (WMT) may best represent this recovery gap. Overall, retail sales rose 4.2% in 2013, or about 2.7% after accounting for inflation. And consumer confidence surveys show Americans on the whole feel considerably better now than they did a year ago. That ought to indicate good times for the nation’s biggest retailer.

Yet Walmart is struggling with weak sales and an underperforming stock price. The company recently cut its profit outlook, with analysts polled by S&P Capital IQ expecting just a 2.1% gain in sales when Walmart reports its quarterly earnings on February 20. That’s for a company that has consistently outcompeted nearly every other retailer except, perhaps, Amazon. Walmart’s stock has suffered, rising just 4% during the past year, while the S&P 500 index rose 17% during the same timeframe.

Walmart, though known as a discounter, may be too expensive for millions of shoppers finding themselves more pinched — not less — as the pace of the so-called recovery accelerates. “Their consumer is shifting downward,” says Joe Brusuelas, chief economist for financial-data firm Bloomberg LP. “The competition for Walmart is changing. It’s now dollar stores.”

While employers added about 2.2 million jobs in 2013 — pushing the unemployment rate down from 7.9% to 6.7% — other changes made life harder for lower-income Americans who form Walmart’s customer base. At the start of 2013, Congress repealed a payroll tax cut that had been in effect for two years, effectively taking about $80 per month from the typical household budget. In November the government cut back on food-stamp benefits, which had also been beefed up during the recession. And effective January 1 of this year, Congress zeroed out enhanced jobless benefits that had been in effect since 2008, leaving less money for 1.3 million unemployed people already struggling to get by.

The net result of those pullbacks is that disposable income — which includes government transfer payments — flatlined toward the end of 2013, as this chart shows:

Walmart is hardly the only retailer struggling. Mass-market chains such as J.C. Penney (JCP), Best Buy (BBY), Target (TGT), the Gap (GPS) and even mighty Amazon (AMZN) turned in disappointing results for the last three months of 2013, indicating a kind of retail recession that lingers long after the official recession ended in the middle of 2009.

So who’s doing well enough to pull retail sales numbers up to relatively healthy levels? Mostly high-end merchants such as Nordstrom (JWN) and Michael Kors (KORS), luxury automakers such as BMW and Mercedes, upscale appliance manufacturers including General Electric (GE) and even yacht manufacturers. As the New York Times noted in a recent article, demand is much stronger for GE’s top-of-the-line dishwashers and refrigerators than for cheaper, mass-market models. And a modest boom in home remodeling is being driven by a small portion of homeowners who have both cash and home equity, a combo many mortgage holders can only envy.

There’s not much real-time data that breaks down spending by income category, though there are plenty of proxy measures showing the top 10% or perhaps 20% of earners may account for the whole increase in retail sales during the past couple of years. “It’s a bifurcated society,” says Brusuelas. “The upper end still has access to credit. They’re making strong income gains and doing fine. But there’s been a fairly steep downturn among middle- and lower-income groups.”

Here’s another chart that shows the breakdown of income through 2011, the latest data available:

What’s alarming for retailers such as Walmart is the sharp drop in income in the middle tier of earners. In 2001, taxpayers earning $100,000 or less accounted for 60% of total income; in 2011, they accounted for less than 50%. In basic terms, that means there are fewer middle-income families with money to spend, leaving retailers — and the overall economy — more dependent on a smaller group of high-income consumers.

Walmart will survive, and perhaps adapt. Maybe it will chase shoppers downmarket, which would probably cut into profitability. Or it might go after wealthier shoppers, drifting away from its core business. It could also stay right where it is, doing no better than the mainstream Americans it has long catered to.

Rick Newman’s latest book is Rebounders: How Winners Pivot From Setback To Success . Follow him on Twitter: @rickjnewman .