Disagreement is the cornerstone of the capital markets, for if all agreed on what a share of stock was worth, few would buy or sell it. Put simply, difference of opinion is essential for generating trade and making markets efficient. Without disagreement, market participants would have less incentive to acquire costly information.
Some have predicted that generative artificial intelligence (AI) will give rise to an “algorithmic monoculture” as investors increasingly rely on a small number of large language models (LLMs) trained on the same underlying data. But there is reason to question this convergence story. We test it in a pre-registered study of how six open-weight LLMs forecast earnings per share (EPS) from 8,476 SEC filings made after the models’ training cutoffs.
The Study
To ensure that our results are not contaminated by the LLM observing the human-analyst forecast in its training data, we limit our sample to filings between September 1, 2024 and April 30, 2026, a period that begins after the last cutoff date of the models used in our study. We test six open-weight models: Meta’s Llama 3.1-8B and 3.3-70B, Mistral-7B, Mixtral-8x7B, Alibaba’s Qwen2.5-7B, and Google’s Gemma-2-9B.
The conventional measure of analyst dispersion uses forecasts outstanding on the filing date, most of which were made before the analyst could have read the filing, whereas the models forecast from the filing. We employ two comparisons to address this timing difference: one against human analysts who issued or reconfirmed an earnings forecast within 10 trading days after the filing, and one comparing how the AI model or human analyst revises its forecast in response to the same filing.
The Models Disagree, a Lot
The pre-registered prediction was that LLMs would disagree less than analysts. They do not. Model forecasts are far more dispersed than analysts’ forecasts. Against analysts who issued or reconfirmed a forecast within 10 trading days after the filing, the cross-model standard deviation is 5.7 times larger (geometric mean). The models’ revisions in response to the same filing are 8.9 times more dispersed than analysts’ revisions.
But Mostly Noise
But most of AI disagreement is not informed. When each model is queried repeatedly, more than 90% of the variance in the panel’s answers is sampling noise within a model rather than disagreement between models. To examine how much the earnings estimate is driven by information, we replace the filing with a random scramble of its words. We find that this changes cross-model dispersion by only 16%, and the scrambled filing yields forecasts about as accurate as the real one. We show that the models do read the filing, because their forecasts are correlated with the future earnings of the firm that filed the SEC disclosure. But what they take away from the filing moves their forecasts around without moving the models closer to the truth. Specifically, their revisions point toward the realized number only about a third of the time, and two models revising the same forecast agree on the direction about as often as a coin flip would. By contrast, human analysts behave very differently: Following a filing, human analysts disagree less, but machine disagreement remains largely unchanged.
Where the Disagreement Comes From
We drill deeper into what drives difference of opinion between LLMs. We find that this sort of disagreement is caused mostly by which information they select rather than how they interpret it. In particular, we ask each model to summarize a filing and then other models to generate a forecast based on that summary. We then examine how much of the variance in the forecast is explained by the summary vs. the choice of forecasting model. We find that the summarizer explains about half of the variance and the forecaster about one-eighth of the variance. In light of the fact that summaries are among the most common ways in which AI tools present disclosures to investors, one noteworthy finding is that forecasts from summaries are about 3.5 times as dispersed as forecasts from the full text, and less accurate.
Rethinking Monoculture
As measured by belief homogeneity, we find little evidence of algorithmic monoculture, as LLMs disagree far more than analysts. We consider specifically whether single forecasts have errors less correlated than analysts’ errors and find that they do, but the difference is mostly sampling noise; once each model’s draws are averaged, the models’ errors are about as correlated as human analysts’ errors. Moreover, the models appear to have an optimistic bias of about a quarter of realized EPS on average. An investor who queries one model once receives a forecast that is mostly noise. One who aggregates many queries receives a common component, which shows some bias.
Implications for Capital Markets Regulation
If AI disagreement were informed and converged on an alternative estimate of value, we might worry that investors relying on these models could lose the diversity of views on which price discovery depends. But given our findings that AI disagreement is mostly noise, we think the greater risk is that investors relying on these tools may find themselves losing to better informed counterparties. The risk is not that investors converge to the exclusion of contrary points of view, but that they treat a noisy, optimistically biased output as informed analysis. We find that the risk is greatest for an investor who queries a model once, and the remedy is not to query more often given that repeated queries reduce variance but not bias. One possible implication of this result is for regulators to focus on the presentation of AI-generated analysis to investors and the use of investing tools that are premised on LLM recommendations, and to consider how much the output varies from query to query and whether there is any form of systematic bias.
Our results also shed light on a practical litigation question: Might an AI model’s reading of a disclosure serve as evidence of how a reasonable investor would read it? Our findings suggest some caution here. In our data, a single model’s reading is one draw from a diffuse and highly variable distribution. We find that different models’ readings differ mainly in what they choose to notice, and neither tracks how the disclosure bears on realized earnings.
Caveats
To be clear, our findings are derived from experiments with small open-weight models; frontier commercial models, and professional systems customized for financial analysis, may behave differently. Under our experimental protocol, we find that LLMs do not form a belief monoculture. They produce disagreement that is large but consists mostly of noise.
Joshua Mitts is the David J. Greenwald Professor of Law at Columbia Law School, Moran Ofir is a professor at the University of Haifa, and Ariel Shillo is a judicial law clerk at the Supreme Court of Israel. This post is based on their recent paper, “Artificial Disagreement: How AI Models Interpret Disclosures,” available here.
Sky Blog