Dr. Frankenstein’s Benchmark: The S&P 500 Index and the Observer Paradox

The most popular understanding of the U.S. stock market is the S&P 500 Index (SP5) of leading U.S. companies. Introduced in 1957, it is the most widely cited index, and more money is managed to it, by far, than any other benchmark. As such, SP5 has become a proxy for the health of U.S. investors, corporate America, and the overall economy. While SP5’s growth and prominence over the past half-century is well known, few market observers stop to ask basic questions: Why was it created? What need did it fill? What has happened since? And most importantly, is it still fit for purpose? That last question is the focus of this essay.

Nearly seventy years after its creation, SP5 may be fit for purpose, but it is clearly no longer the narrow one of the 1950s. While SP5 was initially a tool for measurement and understanding, it has over time become a central actor in shaping investor behavior. The widespread use of SP5-based products now actively influences the market SP5 was intended merely to observe. Though not a true observer paradox in the quantum physics sense, the feedback loop between measurement and subject has become pervasive. The consequences have been many, but perhaps the most significant one has been the widening gap between investment in the stock market and actual ownership of businesses through the stock market. This phenomenon, in turn, is a distinguishing characteristic of “financialization,” a criticism leveled against Wall Street in recent decades for focusing on generating financial returns regardless of, or even in opposition to, outcomes in the real economy.

The personification of an impersonal system is not unusual in a complex society. In the case of SP5, I would suggest that Dr. Frankenstein’s creature is the right character. Mary Shelley’s story, Frankenstein from 1818, is a tale of how scientific progress can diverge wildly from the plan.¹ SP5 tracks that narrative all too well.

Bringing Order to Chaos

It did not start out this way. When SP5 was launched, it was a vast improvement over the numerous industry and early market averages that had appeared in the late nineteenth and early twentieth centuries. Those primitive measures struggled with stock splits, stock dividends, and dividend payments. Over time, they tended to become distorted. As one critic noted in 1957 at the launch of SP5, even if all the constituent elements of those older indices went to zero, the index could still have a positive value. More importantly, these earlier market measures, often associated with media platforms, were mostly indices of price, not size. They did not capture what we now call market capitalization but what was referred to at the time as market value. While indices of value existed, due to calculation limitations, they were neither broad nor timely. The most advanced Standard & Poor’s products prior to 1957 were a value-weighted index of 233 stocks calculated weekly, and one representing ninety companies calculated daily after the market close. Both had been launched in the 1920s.

By contrast, the new index was calculated hourly during the trading day. And with the five hundred largest NYSE stocks (out of nearly 1,100 on the exchange at that time), it covered more than 90 percent of the market’s value. Reflecting the structure of the economy and the market as it existed then, the new index had 425 industrials, twenty-five rails, and fifty utilities. (At that time, banks did not trade on the NYSE. Forty financials were added to the index starting in 1976.)

As the media around the introduction emphasized, SP5 was a technological development, taking advantage of emerging computation capability and communication coordination to provide a comprehensive value-weighted index measure on an hourly basis.² The result gave investors and the financial media an intraday, easy-to-understand, and non-distorted answer to the question of how the overall stock market was doing in absolute and percentage terms: up, down, sideways. SP5 was designed to and initially did serve as a transparent reflection of the reality of the underlying securities trading on the NYSE, the country’s largest exchange by far.

In its earliest incarnation, SP5 also played a bit role in the emergence of modern finance. Starting in the late nineteenth century, economists had sought to bring to human affairs the measurement, rules, and formulas that had so advanced the natural sciences: hence the rise of what has come to be called the “social sciences.” Human behavior, it was understood, was no different than any other natural system. It was just a matter of getting the right data and figuring out the formulas that explained it. While the science of finance was late to the party, by the middle of the twentieth century, academics were hard at work bringing it up to then-current scientific standards, including the creation of measurement tools, such as SP5. This tale was well told by financial journalist Peter Bernstein in his must-read Capital Ideas (1991). It is countered somewhat—or at least placed in historical context—by my own Getting Back to Business (2018).

To be clear, SP5 has been wildly successful in a traditional sense. It is “the” serious benchmark for measuring and understanding the broad U.S. stock market. (The Dow Jones Industrial Average is older, but it is analytically of little use. That’s one of the reasons SP5 was created.³ The popular Nasdaq-100 is more recent and leans heavily toward the new economy.) Most other major market indices are similar to SP5 in design, including the leading “growth” and “value” benchmarks used by institutional investors and licensed by another index vendor. But SP5 is no longer just a yardstick. It has taken on a life of its own and changed the whole nature of stock investing in ways that most investors have not paused to consider.

“It Lives!”: The Creature Comes Alive

The creature’s first sign of life independent of its components came in the form of estimates for what SP5 itself would “earn.” These new “top-down” estimates took advantage of the growing availability in the postwar decades of greater and more detailed macroeconomic data. That data permitted market participants, especially brokerage strategists, to begin forecasting SP5’s expected change in profits per unit of the index compared to the prior year’s actual results. The move coincided with the emergence of macrolevel market investment strategies.

By the 1970s, Wall Street forecasts for individual companies and the market as a whole became sufficiently ubiquitous that a business sprung up to aggregate the estimates and provide “consensus” forecasts. I/B/E/S (Institutional Brokers’ Estimate System) was founded in 1976. Today, forward estimates for SP5 earnings are common. There were twenty-two on the Bloomberg terminal the last time I checked. Investors and traders can easily track how the consensus based on these estimates rises (and sometimes falls) during the course of the year as strategists revise their numbers. It is worth noting that the two sets of calculations—the top-down ones based on macroeconomic data, and the bottoms-up ones tallied from individual company forecasts—can and regularly do differ. Whether or not the forecasts coincide, the top-down projections for the index have become distinct from the specific prospects of the underlying companies. They may be directionally parallel, but they are analytically separate.

Once SP5 had its own estimates no longer directly linked to the underlying constituents, its evolution to sentient being progressed rapidly when these forecasts were used to calculate the market’s P/E ratio, the basic valuation measure of any stock. Assigning the market a P/E number was akin to Dr. Frankenstein throwing the switch and giving his creation life. If it has a P/E, it breathes. It can be cheap, it can be expensive, it can be fairly priced. It has a character. (That character is not completely transparent. The forward estimates are usually for “operating earnings,” after all the bad stuff is taken out. Actual reported earnings are historically up to 10 percent lower than those operating earnings: that is, forward P/Es are regularly understated.) However analytically useful, the use of valuation metrics like the P/E ratio further blurred the distinction between a constructed aggregate and a distinct investment asset.

Combining earnings estimates and P/E ratios, the creature has also developed hopes and dreams. These are known as “price targets.” As an institutional investor, I struggle with the very idea of a price target, but market strategists spend a lot of time on getting SP5’s bullseye right. What is more important, however, is that the price target exercise is nearly fully attenuated from the current valuation or future prospects of the companies that make up SP5.

Earnings growth from the index—note the semantics: earnings are now attributed to the index itself—has been generally consistent, with a relatively low degree of variance except in times of crisis, such as Covid, the global financial crisis, etc. In contrast, the valuation—the P/E multiple—can vary widely. Not surprisingly, there are now numerous, necessarily top-down models that try to determine the right valuation multiple for SP5. The “Fed model,” created by economist and strategist Ed Yardeni in the 1990s, notes the relationship between the market’s valuation and interest rates, especially the rate on the ten-year Treasury bond. This is not surprising as that bond is the basis for discount rates of future earnings or dividends. There should be a relationship.

Yale’s Robert Shiller assesses the valuation of SP5 earnings in terms of long-term historical P/E ratios adjusted for inflation. The result is the CAPE model, the market’s cyclically adjusted P/E. There are others, such as market capitalization to GDP ratio, a measure favored by Warren Buffett. I have my own much more subjective approach, a very distant derivative of Francis Fukuyama’s underappreciated Trust: The Social Virtues and the Creation of Prosperity (1995) with a heavy dose of Michael Jensen’s agency theory. The resulting simple algorithm is that high-trust societies trade with high multiples. Low-trust societies trade with low multiples. The reader can decide where we currently are or should be on that spectrum. These approaches all share one characteristic: they effectively view SP5 as an independent entity, not just a calculation derived from something else.

In the past five years or so, the direct relationship between SP5 and the underlying securities has become even more distant due to only a handful of companies (first the faangs, and now the Mag7) driving the index. What good is the Yardeni or Shiller model if only a half dozen stocks account for much of the market’s movement? By the time you read this, there may be a new acronym. In this environment, market strategists find themselves valuing—putting a prospective P/E multiple on—something that is supposed to be a broad measure of business activity when it is, in fact, increasingly narrow. SP5 was first an entirely neutral index, then it became a forecastable macroeconomic entity, and now it is the tail of the dog.

Initially just an idea, SP5 first took on corporeal form in August 1976 as an index fund, the Vanguard First Index Investment Trust, that was created specifically to track SP5. The age of so-called passive index investing had begun. Many other products followed suit. In recent decades, these index funds have been joined by ETFs. The well-named SPY (pronounced “spider”) debuted in 1993. Options contracts on SP5 began trading on the CBOE and CME in the early 1980s. The popular mini futures were introduced in 1997. These are real securities, not just a measure of stocks.

S&P Global estimates that, as of the end of 2023, total assets indexed (~$7 trillion) or benchmarked (~$6 trillion) to SP5 amounted to $13 trillion. That equaled just under one-third of the market value of $40 trillion of SP5 at that time. Another ~$3 trillion (notional value) of derivatives were based on SP5.⁴ All told, up to 40 percent of the index’s value was in SP5-based or SP5-oriented products. More recent figures, through the end of 2024, suggest a similar percentage of index assets are managed to the index itself.⁵ Think about that. While SP5 still measures the market, models based on it have become the largest single investment in the market.

As a youngster, SP5 was pliant, keeping to its original mission of measuring the market, not moving it. No longer. At its current size, our lumbering giant knocks heavily into the furniture as it moves through our portfolios. It is so large that inclusions and exclusions lead to material share price swings in the affected securities. There is a voluminous academic literature on the impact on security prices when there are changes in the index. Traders keenly participate in the “better to buy” and “better to sell” activity upon those inclusions and removals.⁶

Given the stakes, the rules concerning inclusion and exclusion are of more than passing interest. For SP5, they are transparent and available on the index provider’s website, under the “Index Governance” section:

S&P Dow Jones Indices considers information about changes to its indices and related matters to be potentially market moving and material. Therefore, all Index Committee discussions are confidential. S&P Dow Jones Indices’ Index Committees reserve the right to make exceptions when applying the methodology if the need arises. In any scenario where the treatment differs from the general rules stated in this document or supplemental documents, clients will receive sufficient notice, whenever possible.⁷

The rules make an explicit exception for one large Omaha-based conglomerate with multiple share classes. And the committee used its discretion several years ago to include a battery-powered shooting star that otherwise would not have qualified at the time. Such exceptions are practical in the management of a complex system, but they also highlight the challenge of SP5 remaining truly neutral.

“More, Far More, Will I Achieve”:
The Creature Gains Momentum

Weighting a broad index by market value was a material improvement over the mostly equal-weighted or simple price-weighted measures common before SP5. And it still is. But it is not perfect, and the emergence of gargantuan financial products based on SP5 exaggerates those imperfections. The daily reweighting of the index should be a simple exercise in observation. But it’s not so simple when trillions of dollars are in funds tracking or competing with the index. When there are net inflows or outflows to these products, the index providers buy and sell the constituent stocks (or market futures) without regard to valuation, without regard to dividends, without regard to anything. They have to.

The result is that SP5 as an index exhibits the characteristics of a momentum product, buying what’s going up and selling what’s going down. In doing so, it accentuates those very trends. Modern finance likes to decompose stock performance into a long list of “factors” separate from actual business conditions. In eighteen of the past twenty years (through 2024), SPY was “overweight” to the Barra momentum factor compared to the ETF of the equal-weighted S&P 500 Index.⁸ And the exposure to the momentum factor increased in recent years as the market has become more concentrated. To be clear, any market-cap-weighted index will tilt toward momentum by nature: growing constituents necessarily get larger weights. It only stands to reason. (By design, equal-weighted indices have the opposite characteristic.)

The point here is that SP5 as an investment product left the strictly passive approach long ago. It can still claim to be rules-based, however, and that is good enough for most nonprofessional investors in SP5. And there’s nothing wrong with momentum investing as a conscious selec­tion by an investor. But whether Main Street investors fully appreciate the consequences of SP5 being both a measuring device and a momentum-oriented financial product at the same time remains unclear. They are very different propositions.

Semantics are not the only victim of SP5’s simultaneous success as a yardstick and as a financial product. So is price discovery. With so much money flowing into index funds over the past several decades, the burden of determining the fair, negotiated price of a security has been falling on a shrinking pool of active participants. The academics are still working out the impact of this boom on the widely assumed market “efficiency.” Suffice it here to note that the structural distortion of the market’s long-term “weighing machine” by this near-term “voting machine” facet (to stretch Benjamin Graham’s celebrated metaphor) should bode well for diligent individual security selection in the decades to come. Like Vladimir and Estragon, critics of index investing have been awaiting this outcome for years, but it has yet to appear in an unambiguous fashion.⁹

Similarly, index investing has been promoted for years as the low-risk investment option. The old saw from a half-century ago went that no brokers ever got fired for buying IBM on behalf of their clients. Today’s financial advisers assume that putting clients in broad index products for all intents and purposes shields them from litigation. Well, no good deed goes unpunished. Pushing everyone into the same “low-risk” posture creates enormous systemic risk. As Keynes famously noted ninety years ago, “there is no such thing as liquidity for the community as a whole,” particularly when that community constitutes up to 40 percent of the market by value and, essentially, owns the same asset.

Dr. Frankenstein has another reason to be proud. Not only has his creation come to life and grown large, it has also become part of the core finance curriculum at some very fine universities. Modern Portfolio Theory (MPT) was, by the admission of its creator Harry Markowitz, impractical until reference to a manageable market index replaced unmanageable covariance tables of every security versus every other security. That breakthrough came with the Capital Asset Pricing Model (CAPM) in the mid-1960s.

The CAPM replaced covariance with a simple measure of sensitivity—popularly known as beta—to the overall market, usually rendered by SP5. Beta is a way to give a modest amount of character to each stock in the SP5 vis-à-vis the broader market. The creation of the CAPM as a single factor model opened the door to more complicated, multifactor formulas, and to the entire world of factor and quantitative investing. That development has not gone unnoticed by the Nobel Committee. But without a common, workable benchmark against which to identify and measure all of these factors, investing would look dramatically different than it does today.¹⁰

And it may look dramatically different going forward due to the increase in SP5’s concentration over the past decade. As of the end of 2024, the top ten names in the index represented 37 percent of its total value. That is materially greater than a previous peak in concentration, around 25 percent, during the internet bubble in 2000.¹¹ I’m not passing judgment on today’s dominant companies or their valuation. But as a practical matter, these leading names are no longer sensitive to market movements in the traditional sense of beta and the CAPM. Instead, the market is sensitive to their movements. One need look no further than how the market reacts to Nvidia’s earnings or material news that affects the leading AI chip maker. When the CAPM causation is reversed—or at a minimum distorted—the formulas derived from it are not likely to work particularly well.

This is a distortion that goes well beyond the academy. For instance, the CAPM in various forms and beta in particular have come to be used by many regulated utilities to determine the allowable Return on Equity (ROE) for their asset bases. That is, if the formula is distorted, so too is your utility bill. Elements of the CAPM are also used to price and manage a wide variety of financial products. The math of those formulas still works. What is at stake is the meaning of that math if an element of the underlying financial architecture, the supposedly purely observational and not participatory component, is violated.

“I am Fearless and Therefore Powerful”:
Financialization as the Creature’s Final Form

SP5 has evolved from its original mandate into an enormous, multifaceted investment ecosystem. While buying an SP5-based index fund or ETF provides easily measurable financial returns, there are no direct goods or services entailed for the economy or the investor. One might say the same thing about an investment in the shares of the Coca-Cola company. Coca-Cola shares provide financial returns, not carbonated beverages. But the owner of Coca-Cola shares is a direct owner of the company and probably has a pretty good idea of what it makes. In contrast, the index investor eschews the direct ownership relationship. Yes, a share of SPY has an indirect claim on a few Coca-Cola shares, but the connection to any single company is mediated by the structure of the index. This attenuation of the connection between company owner and company has led to the current controversy surrounding proxy voting by index product providers.

This separation is emblematic of the broader “financialization” of investment that has been noted and criticized in recent years, not least by this journal. The vast ownership of SP5-based products could not constitute a better example of the divorce between direct business ownership through the stock market, on the one hand, and a Wall Street creation on the other. And now that we have entered a period of enhanced investment in infrastructure and some form of capital-consuming national industrial policy, the allocation of enormous amounts of assets to a diffuse financial entity with little direct impact on the productive economy is not particularly helpful.

One could make the same criticism of any pooled investment product which brings together capital from many investors and distributes it among a variety of companies in order to benefit from basic diversification. But the unprecedented scale of broad-market index investing, and the pointed narrative around passive investing, distinguishes those products from actively managed, narrower pools of invested capital with specific mandates (i.e., growth, value, small caps, dividends, specific sectors, stated geographies, and new technologies). The latter allocate capital in an intentional manner, engage in intentional price discovery, and (are supposed to) vote as proxies—a critical measure of business ownership—with equal intention.

While investment through a pooled intermediary necessarily creates agency issues as well as some degree of ownership attenuation, these delinkages are nowhere near as large or serious as the disconnect that comes with a broad-market passive investment. Some investors no doubt want exactly that: no business ownership entanglements (such as proxy voting), no value determination, and commitment-free financial returns. And the market can and should accommodate them. It is a matter of extent. With up to 40 percent of SP5’s value invested in SP5 and SP5-related products, the pendulum has clearly swung too far.

Is there really a problem here? The vast majority of investors seem quite content with Dr. Frankenstein’s creature as it is. Indeed, they see no monster at all. SP5 still functions as a measurement tool, even if it is no longer purely observational. And it continues to offer diversification, though less so than in the past. Financial products based on it are readily available and attractively priced. Most investors in broad index funds generally know what they are getting, even if they are not entirely aware of the nuances of being in a momentum-leaning product. While SP5 and similar funds could benefit from additional momentum descriptors, that is a quibble.

The issue of financialization bothers a segment of the policy community. There’s less evidence that it bothers the general population. To judge by the popular fervor in GameStop stock several years ago and memecoins more recently, the widening separation between investment and sober asset ownership is simply not a problem for those investors treating stock investment as one more video game in the “simulation” called life. From a regulatory perspective, the SP5 ecosystem comports with all existing rules. And despite the clear risk of overcrowding, current fiduciary practice leans toward passive investment in broad-market products, such as SP5 for retirees.

The Sunshine Solution

So, what’s to be done? While there is no clear policy solution to what SP5 has become, simply asking the question of whether a seventy-year-old tool is still fit for purpose is an appropriate first step.

At a minimum, investment analysts should be more familiar with equal-weighted or fundamentally weighted indices. They exist; they are just not as prominent as the large market cap-weighted ones. Retail investors in particular should have a better understanding of what it means to “own the market” through an SP5-like product. The financial media should show greater appreciation for the daily moves in a momentum-biased investment product double billed as a measurement device.

The academy and regulators need to acknowledge SP5 as a complex ecosystem, not just a sideline observer. They can then begin to ask whether or not a joint measurement tool and investment product distorts capital allocation. Retirement plan sponsors and trustees need to appreciate that an attitude of “all things passive” creates as much risk as it removes. Relentlessly pushing retirees into broad index products may be the low-cost option, but it is not at all clear that it is the low-risk solution.

Seventy years on from the creation of SP5, the challenge is less regulatory or even educational than it is philosophical. SP5 solved a problem at the time of its creation, but in doing so, it has created an even more significant challenge for the current generation of investors. That it has distorted the market’s price discovery mechanism is a well-aired argument.

What is less appreciated is the consequence of the underlying separation of investment from business ownership. That separation has built up gradually over many decades, with theories of numbers, such as MPT, pushing aside the theory and reality of business ownership as a motivating factor to become a minority investor in publicly traded businesses. It is time to come to terms with the impact of the observer paradox on the stock market and to acknowledge that SP5 has become so intertwined with the market that it obscures for many the genuine nature of owning stock in America, with its essential components of risk and reward.

In his 1974 comedy, Young Frankenstein, Mel Brooks turned the dark, solitary monster loping across the Artic ice in Shelley’s original tale into an altogether different beast: actor Peter Boyle in a well-appointed bedroom peering at the Wall Street Journal through reading glasses, while Madeline Kahn chirps off screen. That was the humorous Hollywood ending to this experiment to conquer nature.

But investors need to pay attention and think seriously about what they are getting and not getting when using SP5, whether as a measurement tool, a forecasting platform, an investment product, a mechanism of capital allocation, or an academic device. Recall that Shelley’s original story was a cautionary tale about Enlightenment hubris. Its subtitle, The Modern Prometheus, should remind investors that there is a price to pay for their presumption of a simple and singular solution to the challenge of investment decision-making under conditions of uncertainty.

This article originally appeared in American Affairs Volume IX, Number 3 (Fall 2025): 99–111.

Notes