LLM Forecasting Benchmarks Are Broken: Hindcast Proves It

For years, the AI industry has measured LLM forecasting ability by replaying resolved prediction market questions and checking if the model's probability was accurate. The Hindcast paper, published July 15, 2026 on arXiv, now reveals that this entire evaluation paradigm leaks the answer through two unavoidable channels: retrieval-augmented models can surface post-event news articles, and each new training run incorporates data closer to the event date, turning forecasting into a glorified lookup test.

Hindcast paper (arXiv, July 2026) shows that backtesting LLMs on resolved prediction market questions is invalid due to two types of data leakage: retrieval contamination and training data recency.
Retrieval-augmented models can fetch post-outcome news articles; newer models are trained on data that includes the event itself, making the test a lookup, not a forecast.
The paper estimates that published LLM forecasting benchmarks overstate accuracy by at least 30%, eroding trust in every major evaluation to date.
Metaculus, Polymarket, and other platforms that use backtesting as a proxy for forecasting skill must redesign their evaluation protocols or risk misleading institutional clients.

Why Does Backtesting Leak the Answer for LLMs?

According to the Hindcast paper published on arXiv on July 15, 2026, the root cause is that backtesting — the standard method of replaying resolved questions and grading the probability the system would have assigned — was designed for human forecasters, not LLMs. Humans cannot retroactively read news articles from after a question was resolved, and their training data does not include the event itself. LLMs are different. A retrieval-augmented model can pull up a news report written after the outcome was known, turning the forecast into a lookup. Even without retrieval, each new model generation is trained on data that includes the event, so a question that was in the future for last year's model is now in the training set for this year's model. The paper demonstrates that this contamination is not marginal — it systematically inflates accuracy scores by over 30% across multiple benchmarks.

Who Is Most Exposed by This Revelation?

Metaculus and Polymarket are the most exposed. Both platforms have published or hosted research claiming that LLMs can forecast at superhuman levels using backtesting. The Hindcast paper directly undermines those claims. According to Forecasting Research Institute, which has partnered with Metaculus on several studies, the contamination problem was known anecdotally but never systematically quantified. Now that the paper provides concrete evidence, any platform continuing to use backtesting without correction risks losing credibility with institutional users such as hedge funds, government agencies, and academic researchers who rely on accurate forecasting metrics. The paper also implicates labs like OpenAI and Anthropic, which have touted their models' forecasting abilities based on these flawed benchmarks.

LLM Forecasting Benchmarks Are Broken: Hindcast Proves It

What Are the Two Specific Leakage Channels?

The Hindcast paper identifies two distinct channels. First, retrieval leakage: when an LLM uses a search or retrieval tool, it can access news articles written after the event resolved. The paper shows that for a question like "Will Russia invade Ukraine by March 2022?", a model retrieving in 2025 can pull up a 2022 article confirming the invasion, then assign a 95% probability, scoring perfectly. Second, training data recency leakage: each new LLM release is trained on data that extends closer to the present. A question that was unresolved for a model trained in 2023 is now resolved in the training data of a model trained in 2025. The paper estimates that this effect alone inflates accuracy by 15-20% for questions resolved more than a year before the model's knowledge cutoff. Combined, the two channels make backtesting essentially uninformative for comparing LLM forecasters.

Can the Industry Fix This Problem?

Yes, but the fix is expensive and slow. The Hindcast paper proposes using only questions that were unresolved at the time of the model's training data cutoff, and for retrieval models, blocking access to any document dated after the question's resolution date. This requires maintaining a curated dataset of questions with precise timestamps and a filtered retrieval corpus. According to the authors, this is feasible for a single evaluation run but impractical for continuous benchmarking at scale. The paper also suggests using synthetic questions that have never been resolved in the real world, but that approach introduces its own validity concerns. No major lab or platform has yet committed to implementing these corrections, and the paper warns that without them, all published LLM forecasting benchmarks should be considered invalid.

Dimension	Traditional Backtesting	Hindcast-Corrected Evaluation
Retrieval allowed?	Yes, without date filter	No, or date-restricted
Training data cutoff enforced?	No	Yes, strict cutoff before question date
Estimated accuracy inflation	30%+	0% (target)
Practical cost per evaluation	Low	High (curated dataset + filtered retrieval)
Adopted by major labs?	Yes (all current benchmarks)	None as of July 2026
Verdict	Broken for LLMs	Required for valid results

My thesis is simple: the Hindcast paper is the most important evaluation methodology paper of 2026 because it exposes that the entire forecasting benchmark industry for LLMs is built on sand. Every paper claiming GPT-4 or Claude can predict geopolitical events at superhuman levels is now suspect. In the short term, this is a credibility crisis for Metaculus and Polymarket, which have built products and services around these metrics. In the long term, it forces the industry to invest in proper evaluation infrastructure — curated, time-locked datasets with enforced retrieval filters — which will slow down progress reporting but produce honest results. The biggest winner is the Hindcast methodology itself, which could become the new standard. The biggest loser is any lab that has used backtesting to claim forecasting superiority without disclosing the contamination risk. I predict that within 12 months, at least one major lab (likely Anthropic or Google DeepMind) will publish a corrected benchmark using Hindcast-style controls, and that benchmark will show 20-40% lower accuracy than previously claimed.

By Q3 2027, Metaculus will implement Hindcast-style evaluation for all LLM forecasting contests, publicly acknowledging that previous results were inflated.
At least one hedge fund that uses LLM forecasting metrics from Polymarket will publicly disclose that it has adjusted its model selection criteria downward by 30% following the Hindcast paper.
OpenAI will publish a retrospective analysis of GPT-5's forecasting ability using corrected evaluation methods by Q1 2028, showing lower but still useful accuracy scores.

2022-2023
Early backtesting studies
First studies claim LLMs can forecast at superhuman levels using resolved prediction market questions.
2024
Anecdotal concerns surface
Data leakage concerns appear on AI safety forums but are not systematically addressed.
July 15, 2026
Hindcast paper published
First rigorous quantification of the leakage problem, estimating 30%+ accuracy inflation.
Late 2026 (expected)
Industry audits begin
Major labs begin internal audits of forecasting benchmarks; some begin developing corrected evaluation pipelines.

2022-2023: Early backtesting studies claim LLMs can forecast at superhuman levels using resolved prediction market questions.
2024: Anecdotal concerns about data leakage appear on AI safety forums but are not systematically addressed.
July 15, 2026: Hindcast paper published on arXiv, providing the first rigorous quantification of the leakage problem and estimating 30%+ accuracy inflation.
Late 2026 (expected): Major labs begin internal audits of forecasting benchmarks; some begin developing corrected evaluation pipelines.

Insight 1: The Hindcast paper is not just a critique — it provides a concrete methodology for fixing the problem, which is rare in evaluation methodology research.
Insight 2: The 30% inflation figure is a lower bound; for questions resolved close to the model's training data cutoff, the inflation could exceed 50%.
Insight 3: This paper will likely trigger a wave of retractions or corrections from labs that claimed superhuman forecasting based on backtesting.
Insight 4: The cost of proper evaluation (curated datasets, filtered retrieval) means that small teams without infrastructure will be unable to produce credible forecasting benchmarks, centralizing evaluation authority in a few well-funded labs.