When Likely Answers Are Wrong: LLM...

A new paper from researchers at arXiv (published June 25, 2026) systematically quantifies the relationship between sequence probability and correctness in large language models, revealing that the alignment is far from universal. The findings challenge a core assumption behind many popular decoding methods, including greedy search and beam search.

A new arXiv paper (June 25, 2026) systematically tests when sequence probability aligns with correctness across decoding methods and models.
Key finding: probability-correctness alignment is strong for deterministic tasks (e.g., math, closed QA) but weak or negative for open-ended generation.
Popular decoding methods that maximize probability (greedy, beam search) may harm performance on creative or ambiguous tasks.
The paper provides a framework for selecting decoding methods based on task type, with implications for LLM deployment and fine-tuning.

What Does the Paper Actually Measure?

According to the arXiv preprint, the researchers designed a controlled experiment to isolate the relationship between sequence probability (the conditional probability of a full output given a prompt) and output correctness. They evaluated multiple decoding methods—greedy search, beam search, top-k sampling, nucleus sampling, and contrastive search—across several model families (Llama 3, GPT-4, and Mistral). The tasks spanned deterministic domains (math word problems, factual QA) and open-ended domains (story completion, creative writing). The central metric was the Spearman correlation between sequence probability and a human-annotated correctness score. The paper reported that for deterministic tasks, the correlation was consistently positive and statistically significant (ρ > 0.6 across all models). For open-ended tasks, the correlation dropped sharply, often to near zero or even negative (ρ < 0.1 for story completion). This is the first systematic quantification I have seen that separates task type as a moderating variable.

When Likely Answers Are Wrong: LLM Probability-Correctness Gap

Why Do Popular Decoding Methods Assume Probability Equals Correctness?

Many widely used decoding methods—including greedy search, beam search, and even some contrastive methods—operate under the implicit assumption that shifting probability mass toward more likely outputs improves quality. The paper's authors explicitly state: "Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model." This assumption is baked into the design of beam search, which retains only the k most probable sequences at each step, and into the training of reinforcement learning from human feedback (RLHF), which often rewards higher-likelihood outputs. However, the paper's evidence suggests this assumption is only valid for a narrow subset of tasks. For open-ended generation, the model's probability distribution may reflect frequency in the training data rather than factual or creative correctness. According to the paper's analysis, the issue is that "the model's probability distribution over sequences is a poor proxy for correctness when the output space is large and underconstrained." This is a direct challenge to the design philosophy behind many current LLM pipelines.

Which Decoding Methods Survive This Critique?

Decoding Method	Probability-Correctness Alignment (Deterministic)	Probability-Correctness Alignment (Open-Ended)	Practical Verdict
Greedy search	Strong (ρ > 0.6)	Weak to negative (ρ < 0.1)	Good for constrained tasks only
Beam search (k=5)	Strong (ρ > 0.65)	Weak (ρ < 0.2)	Requires task-specific tuning
Top-k sampling (k=40)	Moderate (ρ ~ 0.4)	Moderate (ρ ~ 0.3)	Balanced but not optimal for either
Nucleus sampling (p=0.9)	Moderate (ρ ~ 0.45)	Moderate (ρ ~ 0.35)	Better for open-ended, still limited
Contrastive search	Strong (ρ > 0.7)	Moderate (ρ ~ 0.4)	Best overall, but computationally expensive
Verdict	No single method excels universally; contrastive search offers the best trade-off, but at higher cost.

What Are the Practical Limits of This Study?

The paper's methodology, while rigorous, has clear limitations. The evaluation was conducted on a single set of prompts (n=500 per task category) and used human annotators for correctness scoring, which introduces subjectivity for open-ended tasks. The paper acknowledges that "inter-annotator agreement for open-ended tasks was moderate (Fleiss' κ = 0.45), indicating that correctness itself is not a stable target." This means the negative correlation for open-ended tasks could partly reflect annotation noise rather than a genuine misalignment. Additionally, the study did not test very large models (beyond 70B parameters) or models trained with extensive RLHF. According to the paper, "We limited our experiments to models under 70B parameters due to computational constraints. It remains an open question whether larger models or those with more RLHF training exhibit different probability-correctness relationships." This leaves a significant gap for models like GPT-4 Turbo or Claude 3.5 Sonnet, which may have been trained specifically to align probability with correctness.

Who Gains and Who Loses from This Finding?

My thesis: This paper is a wake-up call for the LLM deployment community, but its impact will be felt most acutely by companies that have bet heavily on probability-maximizing decoding as a universal quality lever.

Short-term consequences: Developers building applications that mix deterministic and open-ended tasks (e.g., conversational agents that answer factual questions and then generate creative responses) will need to implement task-specific decoding selection. This increases engineering complexity and may push teams toward simpler, but suboptimal, single-method pipelines. Companies like Anthropic and OpenAI, which have heavily optimized their inference stacks around a single decoding method, may face pressure to offer more flexible APIs.

Long-term consequences: The paper points toward a future where LLM APIs expose not just temperature and top-p parameters, but also task-type hints that trigger different decoding strategies. Startups that build middleware to automate this selection—such as Portkey or Helicone—could see increased adoption. Conversely, providers that ignore this finding risk delivering subpar results on open-ended tasks, eroding user trust.

Who gains: Contrastive search methods (e.g., those proposed by Su et al. 2022) and their commercial implementations gain credibility. Research labs like Google DeepMind, which have explored contrastive decoding, are well-positioned. Companies that offer multi-strategy inference engines (e.g., Replicate, Fireworks AI) also benefit.

Who loses: Simpler APIs that only expose greedy or beam search without task-awareness lose appeal. Model providers that have not invested in task-adaptive decoding—potentially including some open-source model hosts—may find their outputs perceived as lower quality.

Concrete prediction: By Q1 2027, at least one major LLM API provider (likely OpenAI or Anthropic) will introduce a 'task_type' parameter that automatically selects the decoding method, citing this paper as motivation.

What Should Developers Do Now?

Audit your tasks: Separate your LLM use cases into deterministic (math, code with known tests, factual QA) and open-ended (creative writing, brainstorming, summarization with high variability).
Match decoding to task: Use greedy or beam search for deterministic tasks; use nucleus or contrastive search for open-ended tasks.
Expect API changes: Monitor for provider announcements about task-adaptive decoding. Be ready to adopt multi-strategy pipelines.

Predictions

OpenAI will release a 'task_type' parameter for its Chat Completions API by March 2027, allowing automatic selection of decoding strategy based on user-specified task category.
Anthropic will publish a follow-up study extending these findings to Claude 3.5 and 4 models, likely showing improved alignment due to their constitutional AI training, but still confirming the task-type dependency.
Hugging Face will add a task-aware decoding selector to its Transformers library by December 2026, making it easier for open-source developers to adopt this approach.

Article Summary

Sequence probability correlates with correctness only for deterministic tasks; for open-ended generation, the correlation is weak or negative.
Popular decoding methods that maximize probability (greedy, beam search) may harm performance on creative tasks.
Contrastive search offers the best overall trade-off but is computationally expensive.
The paper's limitations (moderate annotation agreement, model size cap) mean results should be replicated on larger, RLHF-tuned models.
Developers should adopt task-specific decoding strategies, and API providers are likely to add task-type parameters within 12 months.