- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes the "Information Abundance Paradox" (IAP): when task-relevant information is available in the training context, models can reduce loss by using it directly rather than encoding it parametrically, shifting learning from internalization to context-based inference. They provide various experimental evidence to support this hypothesis, spanning from a sweet spot of context length for achieving optimal performance (Figure 2) to gradient-ratio shifts between FFN and SA (Figure 5).
Critical strengths
- Clean isolation of the effect in SFT + causal mechanistic corroboration. The fixed 8-document budget with k-sweep varies only context relevance, and the resulting supporting pattern holds across 5 model sizes and 4 domains. It is also informative to see the FFN/SA gradient-ratio shift combined with module-restricted fine-tuning, which indeed provides a certain level of causal evidence for the mode-of-learning account, not just behavioral correlation.
- Rigorous, well-controlled pretraining evidence for an important, underexplored question. The sweep over 7 context windows × 4 scales is methodologically stronger than typical "scaling goes wrong" anecdotes and directly challenges the assumption that context length is a free scaling axis.
Critical weaknesses
I am expecting to see underlying mathematical analysis of why models shift to using context rather than parametric information, how this interacts with attention mechanisms, and why there is a sweet point of context length for achieving optimal performance (Figure 2). But unfortunately, this is not provided. The paper mostly stops at the experimental evidence level. Although it is interesting and somewhat informative to see the gradient norm shifts towards SA in Figure 5, it is still unclear what causes such a shift.
Questions for authors
- Why does language modeling loss (the training objective itself) follow a U-shape? More train-time context should not harm objective-aligned prediction?
- In Fig. 1, the long-context variants differ in training stage and data mix, not only context window. Can you rule out these alternative explanations for the Phi-3/OLMo-3 gaps?
3 comments
Thank you again for your careful review and valuable suggestions, which have helped us strengthen our work. We wanted to kindly follow up on our rebuttal and would appreciate hearing whether our clarifications address your concerns or if there is anything further we can explain.
Response to Rebuttal
Thanks for the rebuttal. As I have indicated in my review, this topic falls out of my expertise and I have low confidence in assessing the quality and novelty. But from my general impression , I learn something interesting insights from the submission and thus maintain a weak acceptance.
We thank the reviewer for their insightful review and for recognizing the clean SFT isolation, the causal module-restricted evidence, and the rigor of our pretraining sweep.
W1: No underlying mathematical analysis, unclear what causes the shift toward SA.
TL;DR: The paper already provides the analyses for theory (Sec. 4) and mechanism (Sec. 5). We now provide a new control (App. C.1).
Why models shift to context (Sec. 4, App. J). We formalize the parametric information frontier Ik(ρ), i.e. the minimum task information the weights must store, as the mutual information between the weights and the task I(W;τ) to reach risk ρ with context size k. Prop. 4.1 shows Im(ρ)≤Ik(ρ) for m>k, since longer context weakly enlarges the set of feasible solutions without requiring more information in the weights. When the context carries the task information, the weights no longer have to, which is the formal basis of the Information Abundance Paradox.
When the shift actually happens (Sec. 5.1). Our theoretical result in Sec. 4 is an achievability statement. Regarding realization, our synthetic study shows the shift happens selectively by finding that context addiction appears only in tasks where longer context lowers solution complexity (bitwise and string tasks), and not where it doesn’t (mod10, Caesar).
How this connects to attention (Sec. 5.2, App. I.3). Reading information from context is, by construction, done by attention, whereas storing it is associated with FFNs. When predictive evidence is in the window, the loss can be reduced by changing attention mass over those tokens, so gradient pressure moves to SA (Fig. 5). Fig. 6 then provides causal support for this, beyond correlation. SA-only updates lead to context addiction, while FFN-only updates preserve no-context robustness. App. I.3 further shows that k=8 training increases inference-time attention mass on context tokens relative to no-context training.
Why there is a sweet spot (Sec. 3.1, App. F). We believe the optimum comes from two competing forces. The rising side is the benefit of seeing more coherent text. The falling side is the reduced parametric internalization permitted by Prop. 4.1. App. F shows the optimum tracks the evaluation input length (PTB → 512, LAMBADA → 2048, WikiSPAN → 8192; MCQA/SuperGLUE → 2048), falling consistently at roughly 15-32 times the mean example length. Without informative distant context, neither the rise nor the fall appears (General Response E1, App. C.1).
We agree that a closed-form account of training dynamics for full transformers would be valuable, but it is an open problem well beyond the scope of our work. Our contribution is a formal achievability result, combined with controlled behavioral, gradient-level, and causal evidence that converge on the same mechanism.
Q1: Why does LM loss follow a U-shape? More context should not hurt objective-aligned prediction.
This is the evidence for the IAP, and not a contradiction. The U-shape is measured on held-out benchmarks with short inputs (Table 4), not on in-distribution long windows. Having the same objective does not mean having the same input distribution. A model trained with 32K windows learns to predict tokens using abundant earlier evidence. At test time that evidence is missing, so the model must rely on what it stored in its parameters, and Prop. 4.1 says it doesn't need to store as much parametrically.
Three observations support this: (i) the per-benchmark optimum moves to longer windows exactly as the benchmark examples get longer (App. F); (ii) the comparison is token-and-update-matched, uses the same data, initialization, and seed, and the trend holds under RoPE, ALiBi, and LongRoPE (App. B), ruling out positional-encoding and optimization artifacts; (iii) in a new control (General Response E1, App. C.1), same-length windows of shuffled chunks from different documents show no U-shape, so the degradation requires informative distant context.
Q2: Fig. 1 variants differ in training stage and data mix. Can you rule these out?
No third-party comparison can fully rule these out. This is why we label Fig. 1 a motivating observation (L48) and rest no claim on it. Every confound the reviewer raises (training stage, data mix, post-training) is removed by design in our own experiments (Sec. 3.1), where only the context window varies. The results in Sec. 3.1 are statistically significant across all four scales (App. G.2).
Still, we agree that our sentence at L52-53 is stronger than this evidence can support. We will rephrase it to say that Fig. 1 is consistent with the controlled study.
We hope this clarifies that the theoretical and mechanistic analysis the reviewer was looking for is already in the paper (Sec. 4, Sec. 5.1, Sec. 5.2). If so, we would be grateful if the reviewer would consider updating their assessment.