Financial models once promised a clean world. Prices reflect information. Risk is rewarded. Deviations are noise. Then Robert Shiller walked in with data that refused to behave. Markets, he showed, swing more than fundamentals can justify and those swings build on stories people tell each other. The conclusion is not that finance is chaos, only that it’s human. That human element is where models crack.
🧩 What “Irrational Markets” Means — a short primer
Shiller’s central claim is disarmingly simple. Markets often move farther and faster than underlying cash flows or productive capacity could possibly warrant. The engine isn’t a neat equilibrium mechanism smoothing all shocks. It’s collective psychology reinforced by feedback loops. Prices drift from fundamentals because people drift from pure rationality.
Set this against the strongest version of the Efficient Market Hypothesis. EMH says prices fully reflect available information and deviations from fair value are neither persistent nor predictable. If prices move a lot, it must be because the information set changed a lot or because risk premia did. Shiller’s career has been a long, empirical test of that assertion. His answer is a qualified no.
“Irrational” in his telling is not a moral label. It’s a diagnostic one. Investors are not calculating machines. They use narratives, imitate peers, fear losses more than they value equivalent gains, and extrapolate recent experience. In aggregate, those tendencies can create self-reinforcing cycles that make prices wander. Sometimes that wander is harmless. Sometimes it is a bubble.
A working definition helps. An irrational market is one where the variance of prices exceeds the variance implied by subsequent fundamentals in a way that cannot be explained by risk compensation or information updates alone. That definition turns an intuition into something you can test.
🟦 How Shiller Proved It — the excess‑volatility challenge
In his 1981 American Economic Review paper, Shiller asked a pointed question: do stock prices move too much to be justified by subsequent changes in dividends? He compared the volatility of stock prices to the volatility of a “perfect foresight” price computed from the present discounted value of realized future dividends. Prices were far more volatile than that anchor. The empirical gap was not a rounding error. It was a gulf.
For a generation taught that markets are efficient, this was a formal blow. If price equals the expected present value of future dividends under rational expectations, then the price should be less volatile than the ex post perfect-foresight value. The data said otherwise. You could still claim time-varying risk premia or changing discount rates, but the bar for rational explanations rose.
A quick methodological note matters because Shiller’s leverage is subtle. He’s not assuming perfect forecasting. He uses a variance bounds test. If price equals expected value of future dividends, the variance of price should be bounded by the variance of that realizable stream. Observed prices violate the bound. This is not an exact map of irrationality, only a strong signal that something beyond a clean EMH story is at work.
The criticism arrived quickly. Perhaps dividends are a poor proxy for total cash flows because of buybacks. Perhaps the discount rate moves. Perhaps learning about growth is lumpy. All true as partial defenses. None erases the finding that prices routinely swing far beyond what subsequent cash flows can explain.
🟦 From Paper to Popular Phrase: “Irrational Exuberance” and the anatomy of bubbles
Two decades later, Shiller turned a technical debate into a public conversation. Irrational Exuberance popularized the idea that asset markets are driven by narrative contagion, credit conditions, and valuation extremes. It wasn’t just the stock market. Housing, too, could disconnect from incomes and rents when beliefs spread faster than facts.
His favored signal for public debate was CAPE — the cyclically adjusted price‑to‑earnings ratio that compares price with a 10‑year average of real earnings. CAPE does not predict the day a party ends. It sets the mood music. High CAPE regimes are historically associated with lower subsequent long‑run returns and more fragile market structures. In the late 1990s dot‑com surge, CAPE screamed caution while stories about the “new economy” overwhelmed skepticism. In the mid‑2000s housing boom, the narratives shifted to real estate, but the pattern rhymed.
Shiller’s rhetorical move was to blend data with the sociology of markets. People talk. Stories spread. Media echo certain narratives and downplay others. Credit availability extends reach. The result is not an inevitable crash, only heightened vulnerability. A modest shock becomes an amplifier rather than a speed bump because valuations have climbed and belief is brittle.
CAPE became a shorthand in public discourse precisely because it translated a complex critique into a single gauge. You could argue with the calibration. You could not ignore the history it summarized.
💡 Why humans break models: behavioral mechanics
Kahneman and Tversky’s prospect theory provides the mechanism behind Shiller’s observation. People evaluate gains and losses relative to a reference point, not in absolute wealth. They dislike losses more than they like equivalent gains. They overweight small probabilities and underweight large ones. In markets, these tendencies show up as the disposition effect, where investors sell winners too quickly and hold losers too long, and as overreaction or underreaction to news depending on context.
Anchoring and availability bias add texture. Investors latch onto salient prices and recent events when forming expectations. They mistake a vivid story for a likely one. When your neighbor doubles their money in tech stocks, it becomes hard to sit out a bubble cycle. When headlines turn dark, the same mechanisms make risk feel unacceptably large. Models that assume smooth updating of beliefs miss these jolts.
Market dynamics translate psychology into price. BlackRock’s work on investor sentiment shows how positioning and flows can push prices away from fundamentals for sustained periods. When risk appetite rises, inflows amplify momentum, which convinces more investors to join. The feedback loop gains strength because the price confirmation seems like proof. J.P. Morgan’s long‑run guides show that such valuation cycles and regime shifts recur. The pattern isn’t random, it’s human.
None of this implies chaos. It implies path dependence. Small shocks matter when the crowd is leaning one way. Behavior shapes the path that fundamentals travel.
💡 Why this matters now: valuations, regime shifts and practical risk
In an era of low interest rates, large passive flows, and instantaneous information, behavioral drivers have become both more visible and more consequential. Liquidity can appear abundant until it vanishes. Social media accelerates narrative contagion. Algorithmic strategies can unintentionally hard‑wire feedback loops. When valuations stretch, these dynamics add fragility.
Institutional perspectives sharpen the point. The J.P. Morgan Guide to the Markets lays out how valuation metrics like CAPE correlate with long‑run returns while also documenting that regime shifts — inflation cycles, policy pivots — alter the payoff to risk. BlackRock emphasizes that sentiment and positioning data often lead or lag fundamentals in ways that matter for allocation. The practical takeaway is not to worship a single metric. It’s to recognize how signals interact.
The stakes extend beyond portfolios. When valuation signals collide with momentum and cheap credit, we move from model failure to macro risk. Household balance sheets lean into stories. Corporate leverage tracks valuation windows. Policymakers end up managing bubbles as much as growth.
Check how disciplined your portfolio really is.
⚙️ Common misconceptions and fair criticisms
Shiller’s name sometimes gets dragged into a caricature — the man who always warns and occasionally gets lucky. It misses the nuance. He issued probabilistic warnings, not calendarized prophecies. High CAPE meant low expected returns over a decade, not a crash next quarter. Irrational exuberance described vulnerability, not fate.
The fair criticisms focus less on the basic insight and more on measurement. Aswath Damodaran has argued that CAPE needs adjustments to account for shifting accounting standards, elevated profit margins, and the relationship between interest rates and earnings. The Financial Times coverage has made similar points in plain language. A high CAPE in a world of structurally lower real rates and higher margins may not carry the same message as in 1901.
These critiques are healthy. They prevent a useful ratio from becoming an oracle. They also remind us that extrapolating twentieth‑century averages into a transformed economy requires humility. Behavioral finance can explain why prices swing. It does not absolve us from updating the tools we use to measure those swings.
The misconception on the other side is that because CAPE is imperfect, the entire behavioral thesis collapses. That’s a non sequitur. You can be skeptical of a single gauge and still accept that sentiment, narratives, and flows create excess volatility.
🟦 Case studies and data: what the numbers actually say
Start where the debate began. The 1981 excess‑volatility finding showed that the variance of stock prices exceeded the variance of perfect‑foresight dividend streams. Decades later, that result survives adaptation for buybacks and alternative cash flow measures. The magnitude moves. The direction remains.
Consider the dot‑com episode. CAPE climbed above 40 in 1999. Earnings growth did not justify that altitude. The story did — a narrative of a reinvented economy where profits were optional and scale was destiny. Flows into tech funds accelerated. IPOs priced with minimal operating history. When the cycle turned, the correction wasn’t just about fundamentals catching up. It was about belief unwinding and funding conditions tightening.
Housing in 2006–07 offered a different mechanism with the same DNA. Shiller’s home price indices detached from rent and income anchors as credit standards loosened and securitization scaled the story nationwide. Prices stabilized for years despite questionable underwriting because the narrative of ever‑rising real estate was self‑reinforcing. When it broke, leverage magnified the adjustment.
Institutional corroboration fills in the modern picture. J.P. Morgan’s guides trace CAPE swings across a century and map them to realized returns. The pattern is one of regimes, not precision timing. BlackRock’s sentiment research shows how positioning can diverge from fundamentals for extended periods, only to snap back violently when a catalyst arrives. These aren’t isolated anecdotes. They are artifacts of market structure and human behavior repeating at different scales.
The lesson hides in plain sight. Data alone rarely change minds at the peak of a narrative. Minds change when price moves force them to.
🟦 Counterarguments and alternative frameworks
Refined efficient‑market views haven’t vanished. Proponents argue that apparent excess volatility can be reconciled with rational learning, time‑varying discount rates, or changes in risk premia tied to macro uncertainty. There is truth here. Discount rates do move. The state of the world shifts. A clean dividend model can miss a lot.
Damodaran’s practical critique improves the conversation. Adjust CAPE for accounting changes. Normalize for interest rates. Consider profit margin cycles. Treat any long‑run ratio as a probabilistic input, not a command. The Financial Times has pressed the same point for a broader audience — valuation is a warning label, not a detonator.
Where do these alternatives succeed? They reduce false alarms. They sharpen the use of valuation in allocation. They explain periods where high multiples persist without disaster. Where do behavioral accounts still hold the edge? In explaining the pattern of overshoot and reversal that appears too often to be driven only by risk premia.
A compact comparison helps.
| Framework | What it explains best |
|---|---|
| Strict EMH with time‑varying risk premia | Persistence of high valuations without immediate crashes and the role of discount rate shifts |
| Behavioral finance (Shiller, prospect theory) | Overshoot, herd dynamics, momentum reversals, and divergence of price from subsequent cash flows |
Taken together, the message is not to pick a team. It is to know when each lens adds clarity.
🟦 Practical takeaways — what investors, journalists and policymakers should do
You do not cure human markets. You manage them. Investors should combine valuation gauges with flow and sentiment indicators, then plan for regime shifts rather than point forecasts. Journalists should avoid false precision and use probabilistic language. Policymakers should focus less on bubble hunting and more on building buffers when credit and belief sprint together.
A short toolkit makes this concrete.
- Use adjusted CAPE as a context setter, not a trigger. Blend it with rate‑sensitive metrics and margin analysis.
- Track sentiment and flows — positioning, fund flows, options activity — to gauge feedback risk.
- Stress‑test portfolios for valuation compression and liquidity shocks. Diversification and pre‑set rebalancing beats ad hoc heroics.
- Employ simple risk rules like stop‑loss bands or volatility caps where appropriate to prevent narrative capture.
- For policymakers: lean on macroprudential tools when exuberance meets leverage — loan‑to‑value limits, countercyclical buffers, and transparency in securitization.
Two small habits deliver outsize benefit. First, write down your base case and the story that would make you abandon it. Second, schedule rebalancing on the calendar so you don’t negotiate with yourself when prices are loud.
Want a quick gut check? Map today’s allocation against a range of 10‑year expected returns derived from valuation and rates. If you wouldn’t buy your own portfolio at today’s prices, you’ve learned something useful. Check how disciplined your portfolio really is.
🟦 Final synthesis — a modern verdict with a human note
Markets are not random in the way dice are random and not rational in the way textbooks wish. They are social machines that turn beliefs into prices, then use those prices to validate new beliefs. Shiller didn’t reject markets as a field of study. He widened it. He gave us permission to say that psychology matters without surrendering to mysticism.
The verdict is modern humility. Use models as maps, not as territory. Treat valuation as a weather report. Pay attention to the stories that spread and the leverage that rides them. Expect regimes to change and plan for what you will do when they do. That mix won’t eliminate drawdowns or banish bubbles. It will make you a steadier participant.
Living with uncertainty is not a failure of finance. It’s the job. Shiller’s contribution is to make that job feel honest.
📚 Related Reading
– Risk vs. Return: Why Valuation Still Matters When Markets Forget https://axplusb.media/risk-vs-return
– Volatility and Regimes: A Field Guide for Long‑Horizon Investors https://axplusb.media/volatility-and-regimes
– Investor by the Numbers: Sentiment, Flows, and the Stories We Trade https://axplusb.media/investor-by-the-numbers