Shannon Scaling Law: LLMs Hit a Noisy Channel Ceiling

A new preprint from arXiv proposes that LLM scaling laws, long assumed to be monotonic power laws, are fundamentally flawed. The authors model model parameters as channel bandwidth and training data as signal, showing that catastrophic overtraining is not an anomaly but a predicted consequence of exceeding channel capacity.

A new arXiv paper (2605.23901) proposes the Shannon Scaling Law, modeling LLM training as information transmission over a noisy channel.
It explains catastrophic overtraining and quantization-induced degradation as consequences of exceeding channel capacity, not training bugs.
This framework challenges the monotonic power-law assumptions behind current scaling efforts at OpenAI, Google DeepMind, and Anthropic.
Early adopters of information-theoretic early stopping and capacity-aware training will gain a competitive edge.

Why Do Existing Scaling Laws Fail to Explain Catastrophic Overtraining?

According to the authors of the arXiv preprint (2605.23901), current scaling laws—predominantly monotonic power laws derived from Kaplan et al. (2020) and Hoffmann et al. (2022)—fail to account for performance degradation under increased compute. The paper reports that catastrophic overtraining, where loss increases after a certain training duration, and quantization-induced degradation, where lower precision degrades performance beyond expected noise, are both observed but unexplained by existing theory. The Shannon Scaling Law reinterprets these phenomena as a model exceeding its channel capacity: when the signal-to-noise ratio drops below a threshold, information cannot be transmitted reliably, and further training adds noise, not signal.

How Does the Shannon-Hartley Theorem Apply to LLM Training?

The authors map model parameters to channel bandwidth and training data to transmitted signal, with loss corresponding to bit error rate. According to the paper, the channel capacity C = B * log2(1 + SNR) sets a hard upper bound on the rate of information that can be learned. When the model's parameter count (bandwidth) or training data (signal power) exceeds this bound, performance saturates or degrades. This is a direct application of Claude Shannon's 1948 theorem, which the authors claim has been overlooked in deep learning scaling literature. The preprint provides mathematical derivations showing that the Shannon Scaling Law reduces to the Kaplan power law in the low-SNR regime but predicts non-monotonic behavior in the high-SNR regime, matching observed phenomena.

Shannon Scaling Law: LLMs Hit a Noisy Channel Ceiling

Who Gains and Who Loses From This Theoretical Shift?

The winners are researchers and labs that adopt information-theoretic early stopping and capacity-aware training. The losers are those who continue to scale compute blindly.

Actor	Current Approach	Shannon Scaling Law Impact
OpenAI (GPT-5)	Compute-optimal scaling per Hoffmann	May hit capacity ceiling before expected; overtraining risk
Google DeepMind (Gemini 3)	Massive data+compute scaling	Quantization degradation predicted; capacity limits apply
Anthropic (Claude 4)	Constitutional AI + scaling	May avoid overtraining via early stopping heuristics
Smaller labs / academia	Resource-constrained training	Benefit from capacity-aware training; less waste
Hardware vendors (NVIDIA)	Sell more compute	Risk: scaling demand may soften if capacity limits are reached
Verdict		Smaller labs and Anthropic gain; OpenAI and DeepMind face disruption

What Concrete Predictions Does the Shannon Scaling Law Make?

According to the paper, the framework yields falsifiable predictions: (1) For a fixed model size, there exists an optimal training duration beyond which loss increases—a direct prediction of catastrophic overtraining. (2) Quantization to lower precision reduces effective SNR, and the law predicts a critical quantization level below which no further training improves performance. (3) The scaling exponent in the Kaplan law is not a constant but a function of the channel SNR, implying that different datasets and architectures will yield different exponents. The authors claim these predictions can be tested on existing models like LLaMA-3 and GPT-4-class systems within six months.

My thesis is that the Shannon Scaling Law is the most important theoretical advance in LLM scaling since the Chinchilla paper. In the short term, labs will scramble to validate or refute these predictions, likely leading to a wave of empirical studies. In the long term, this framework could reshape how we think about model capacity: instead of treating parameters as a resource to be maximized, they become a resource to be optimized within channel constraints. The biggest losers are OpenAI and Google DeepMind, whose entire scaling playbook assumes monotonic improvement with compute. The biggest gainers are smaller labs and researchers who can now design training regimes that avoid overtraining from the start. I predict that within 12 months, at least one major lab will publicly adopt an information-theoretic early stopping criterion derived from this framework.

Within 12 months, at least one major AI lab (likely Anthropic or a leading academic group) will publish a paper adopting the Shannon Scaling Law for early stopping.
Within 18 months, OpenAI or Google DeepMind will publicly acknowledge that their scaling efforts are hitting a capacity ceiling, citing this framework.
Within 24 months, the Shannon Scaling Law will be incorporated into standard LLM training textbooks and curricula, replacing the Kaplan power law as the default model.

Insight 1: The Shannon Scaling Law unifies previously anomalous phenomena (catastrophic overtraining, quantization degradation) under a single mathematical framework, making it a candidate for a new standard theory.
Insight 2: It directly challenges the compute-optimal scaling assumptions used by OpenAI, Google DeepMind, and Anthropic, suggesting that raw compute scaling has a hard ceiling.
Insight 3: The winners are smaller labs and researchers who adopt capacity-aware training; the losers are those who continue to scale compute blindly.
Insight 4: The framework is falsifiable and makes concrete predictions that can be tested on existing models within months.
Insight 5: This is the most significant theoretical advance in LLM scaling since the Chinchilla paper, with the potential to reshape the entire field.