When restaurant companies like Zoes Kitchen and El Pollo Loco became public a few years ago, Wall Street had high expectations for these ambitious "growth darlings." But despite impressive beginnings, a fall-off in traffic has pressured their shares, and low gas prices and a strengthening job market over the past year are not enough to bring them back to life.

Noodles & Company (NDLS) priced its IPO in 2013 at $18 per share and jumped over 100% in its first day of trading, and rose to almost $47 by the end of that year. However, the chain -- which sells noodles of all cuisines from Mac and Cheese to Pad Thai -- now trades at under $12.

Sandwich seller Potbelly (PBPB) priced its IPO at $14 a share in October 2013 and rose 120% to over $30 in its first day of trade. It's down to $11 today. It's a similar story at El Pollo Loco (LOCO). The Mexican style restaurant priced its IPO in June 2014 at $15 a share and surged 60% to over $24 on its first trading day. It now languishes at $12.

At the ICR Conference in Orlando, Fla., El Pollo Loco CEO Steve Sather told Yahoo Finance that expectations had been very high for these chains when they went public. "There was a euphoric state where a lot of focus was on the fast casuals and really seeing who was going to be the next Chipotle at that time," he said.

Part of the recent selloff in this group can be explained by a shift in investor appetite. Investors are reducing their risk exposure amid market volatility, according to Maxim Group analyst Stephen Anderson. And that's impacting the more speculative, growth-oriented restaurant concepts.

But recent lackluster results have also put downward pressure on the group. "Especially in this environment, investors are only paying up for performance," said Piper Jaffray analyst Nicole Miller Regan. Tepid sales at existing locations have also thrown into question the unit growth trajectories of these names that had supported higher valuations based on a growth thesis.

Last quarter, El Pollo posted anemic comparable store sales growth, up 0.6%, led down by a drop in traffic. Yet Sather remained positive on new restaurant openings. "Our new restaurant pipeline is strong and growing stronger, and we continue to be excited about hte long runway of growth that we believe is ahead of us in both new and existing markets," he said in the company's press release. Management of the chain, which currently has over 400 stores in the Southwest, sees unit growth of 8% to 10% annually; the company aims to eventually have 2,300 stores across the U.S.

Sather said he sees a recovery in same-store sales to the low-single-digits and emphasized the brand's positioning that emphasizes healthy food at affordable prices.

Noodles & Co., meanwhile, saw its comparable sales fall 0.9% in its latest quarter. Chairman and CEO Kevin Reddy -- who was previously COO at Chipotle after starting out at McDonald's (MCD) -- remains bullish on the company's growth prospects. "We've been a little slower than the historical norms have been, but we're doing all the right things to get back to that," he said. Reddy noted a more competitive environment but said he feels confident in Noodles' positioning with families.

Even companies that have been sporting better results, like Mediterranean-inspired chain Zoes Kitchen (ZOES), have seen their stocks decline. The company posted 4.5% comparable-store sales last quarter and sees its current store count of over 160 stores, 34 of which were opened last year, growing tenfold to 1,600 in the long term.

Yet Zoes hasn't budged much since its IPO in April 2014, when it priced at $15 a share and jumped to just under $25. While the stock has remained stable at $26 and has earned a higher valuation (trading at 25 times EBITDA versus 9 times for Noodles and El Pollo Loco), it hasn't been able to break out. "The concept is still really resonating with the consumers," said CEO Kevin Miles.

This market response follows that of burger grill, The Habit Restaurant (HABT), which rose almost 120% in its first day of trading in November 2014, from its $18 pricing to nearly $40. It's now near its offering price of $20 despite posting solid low-single digit comparable-store sales with unit growth potential to boot.

The bottom line: Until investors see higher employment or lower gas prices translate into robust sales at these unit-growth chains, their stocks will remain pressured amid a more turbulent macro environment.