acceptodds
Under review as a conference paper at ICLR 2027

Information Abundance Paradox: Long Context Training Undermines Parametric Knowledge

Abstract

Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that longer contexts improve learning by exposing models to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant task-relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book QA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading. Our analysis suggests that this behavior arises when longer context provides a lower-complexity path to reducing loss. Consistently, context-rich training shifts gradient pressure from feed-forward modules towards attention modules, and causal interventions show that this shift increases reliance on context during inference. Taken together, these findings support the Information Abundance Paradox and challenge the view that near-infinite-context models are the inevitable endpoint of language modeling, even when high-quality long-context data is abundant.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rhtzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the "Information Abundance Paradox" (IAP): when task-relevant information is available in the training context, models can reduce loss by using it directly rather than encoding it parametrically, shifting learning from internalization to context-based inference. They provide various experimental evidence to support this hypothesis, spanning from a sweet spot of context length for achieving optimal performance (Figure 2) to gradient-ratio shifts between FFN and SA (Figure 5).

Critical strengths
  1. Clean isolation of the effect in SFT + causal mechanistic corroboration. The fixed 8-document budget with k-sweep varies only context relevance, and the resulting supporting pattern holds across 5 model sizes and 4 domains. It is also informative to see the FFN/SA gradient-ratio shift combined with module-restricted fine-tuning, which indeed provides a certain level of causal evidence for the mode-of-learning account, not just behavioral correlation.
  2. Rigorous, well-controlled pretraining evidence for an important, underexplored question. The sweep over 7 context windows × 4 scales is methodologically stronger than typical "scaling goes wrong" anecdotes and directly challenges the assumption that context length is a free scaling axis.
Critical weaknesses

I am expecting to see underlying mathematical analysis of why models shift to using context rather than parametric information, how this interacts with attention mechanisms, and why there is a sweet point of context length for achieving optimal performance (Figure 2). But unfortunately, this is not provided. The paper mostly stops at the experimental evidence level. Although it is interesting and somewhat informative to see the gradient norm shifts towards SA in Figure 5, it is still unclear what causes such a shift.

Questions for authors
  • Why does language modeling loss (the training objective itself) follow a U-shape? More train-time context should not harm objective-aligned prediction?
  • In Fig. 1, the long-context variants differ in training stage and data mix, not only context window. Can you rule out these alternative explanations for the Phi-3/OLMo-3 gaps?
3 comments

We thank the reviewer for their insightful review and for recognizing the clean SFT isolation, the causal module-restricted evidence, and the rigor of our pretraining sweep.

W1: No underlying mathematical analysis, unclear what causes the shift toward SA.

TL;DR: The paper already provides the analyses for theory (Sec. 4) and mechanism (Sec. 5). We now provide a new control (App. C.1).

Why models shift to context (Sec. 4, App. J). We formalize the parametric information frontier Ik(ρ)I_k(\rho), i.e. the minimum task information the weights must store, as the mutual information between the weights and the task I(W;τ)I(W;\tau) to reach risk ρ\rho with context size kk. Prop. 4.1 shows Im(ρ)≤Ik(ρ)I_m(\rho) \le I_k(\rho) for m>km>k, since longer context weakly enlarges the set of feasible solutions without requiring more information in the weights. When the context carries the task information, the weights no longer have to, which is the formal basis of the Information Abundance Paradox.

When the shift actually happens (Sec. 5.1). Our theoretical result in Sec. 4 is an achievability statement. Regarding realization, our synthetic study shows the shift happens selectively by finding that context addiction appears only in tasks where longer context lowers solution complexity (bitwise and string tasks), and not where it doesn’t (mod10, Caesar).

How this connects to attention (Sec. 5.2, App. I.3). Reading information from context is, by construction, done by attention, whereas storing it is associated with FFNs. When predictive evidence is in the window, the loss can be reduced by changing attention mass over those tokens, so gradient pressure moves to SA (Fig. 5). Fig. 6 then provides causal support for this, beyond correlation. SA-only updates lead to context addiction, while FFN-only updates preserve no-context robustness. App. I.3 further shows that k=8k=8 training increases inference-time attention mass on context tokens relative to no-context training.

Why there is a sweet spot (Sec. 3.1, App. F). We believe the optimum comes from two competing forces. The rising side is the benefit of seeing more coherent text. The falling side is the reduced parametric internalization permitted by Prop. 4.1. App. F shows the optimum tracks the evaluation input length (PTB → 512, LAMBADA → 2048, WikiSPAN → 8192; MCQA/SuperGLUE → 2048), falling consistently at roughly 15-32 times the mean example length. Without informative distant context, neither the rise nor the fall appears (General Response E1, App. C.1).

We agree that a closed-form account of training dynamics for full transformers would be valuable, but it is an open problem well beyond the scope of our work. Our contribution is a formal achievability result, combined with controlled behavioral, gradient-level, and causal evidence that converge on the same mechanism.

Q1: Why does LM loss follow a U-shape? More context should not hurt objective-aligned prediction.

This is the evidence for the IAP, and not a contradiction. The U-shape is measured on held-out benchmarks with short inputs (Table 4), not on in-distribution long windows. Having the same objective does not mean having the same input distribution. A model trained with 32K windows learns to predict tokens using abundant earlier evidence. At test time that evidence is missing, so the model must rely on what it stored in its parameters, and Prop. 4.1 says it doesn't need to store as much parametrically.

Three observations support this: (i) the per-benchmark optimum moves to longer windows exactly as the benchmark examples get longer (App. F); (ii) the comparison is token-and-update-matched, uses the same data, initialization, and seed, and the trend holds under RoPE, ALiBi, and LongRoPE (App. B), ruling out positional-encoding and optimization artifacts; (iii) in a new control (General Response E1, App. C.1), same-length windows of shuffled chunks from different documents show no U-shape, so the degradation requires informative distant context.

Q2: Fig. 1 variants differ in training stage and data mix. Can you rule these out?

No third-party comparison can fully rule these out. This is why we label Fig. 1 a motivating observation (L48) and rest no claim on it. Every confound the reviewer raises (training stage, data mix, post-training) is removed by design in our own experiments (Sec. 3.1), where only the context window varies. The results in Sec. 3.1 are statistically significant across all four scales (App. G.2).

Still, we agree that our sentence at L52-53 is stronger than this evidence can support. We will rephrase it to say that Fig. 1 is consistent with the controlled study.

We hope this clarifies that the theoretical and mechanistic analysis the reviewer was looking for is already in the paper (Sec. 4, Sec. 5.1, Sec. 5.2). If so, we would be grateful if the reviewer would consider updating their assessment.

Thank you again for your careful review and valuable suggestions, which have helped us strengthen our work. We wanted to kindly follow up on our rebuttal and would appreciate hearing whether our clarifications address your concerns or if there is anything further we can explain.

Reviewer_rhtzOpenReview ↗

Response to Rebuttal

Thanks for the rebuttal. As I have indicated in my review, this topic falls out of my expertise and I have low confidence in assessing the quality and novelty. But from my general impression , I learn something interesting insights from the submission and thus maintain a weak acceptance.

Reviewer_uif7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper investigates whether informative training context shifts language models from storing task knowledge in their parameters toward relying on contextual evidence. It reports non-monotonic downstream performance when varying pretraining windows from 512 to 32,768 tokens across four model sizes under matched token and update budgets. In supervised fine-tuning, increasing answer-supporting context generally improves performance with supporting evidence but reduces accuracy without context or with conflicting evidence. A theoretical proposition establishes that additional nested context cannot increase the minimum parametric information required to achieve a given risk. Synthetic experiments, gradient measurements, and module-restricted fine-tuning are used to investigate possible mechanisms.

Critical strengths
  1. The pretraining study provides a useful controlled observation. Section 3.1 matches training tokens, optimization steps, initialization within seeds, and hyperparameters across seven context windows and four model sizes. Appendix C.1 reports multiple training seeds, and Appendix B checks alternative positional encodings. These choices make the observed degradation more informative than comparisons between independently developed short- and long-context checkpoints. The results provide practical evidence that increasing the training window can hurt short-input performance under a fixed training recipe.

  2. The SFT study separates context relevance from simply adding more text. Section 3.2 varies the composition of eight contextual sentences and evaluates supporting, conflicting, and absent context. For Qwen3-4B, Table 16 reports that changing from k=0k=0 to k=8k=8 raises supporting-context accuracy from 81.2% to 87.8%, while lowering no-context accuracy from 54.4% to 49.1% and conflicting-context accuracy from 39.0% to 22.3%. This is a consequential robustness tradeoff. The module-restricted experiments also provide evidence that the choice of trainable modules affects that tradeoff.

Critical weaknesses
  1. The experiments do not adequately distinguish reduced knowledge internalization from changes in the conditions under which that knowledge is accessed. In Section 3.1, increasing the window while applying loss to every token substantially changes the distribution of available prefix lengths during training. For example, the proportion of positions with fewer than 512 preceding tokens falls from all positions at W=512W=512 to approximately 1/641/64 at W=32,768W=32{,}768. Yet Appendix E reports mean evaluation lengths of only 28–504 tokens. Reduced training exposure to short-prefix prediction is therefore a concrete alternative explanation for poorer short-input performance. The positional-encoding ablations do not remove this difference. The synthetic study has a related identification problem: Section 5.1 changes the number of test-time demonstrations together with the training condition. Its increasing supporting–conflicting gap could partly reflect exposure to more misleading examples at inference. The SFT results strengthen the evidence for learned context dependence, but do not by themselves establish that the same mechanism explains long-document pretraining. Proposition 4.1 correctly addresses an achievable information frontier; it does not establish that training selects solutions with less information in their weights.

  2. The proposed optimization mechanism rests on measurements that do not establish the claimed quantities. Section 5.1 treats the average parameter-gradient norm as a proxy for solution complexity. This quantity also depends on prediction residuals and optimization dynamics: a model that reaches low cross-entropy sooner can have smaller average gradients without learning a simpler function. The cited Why Neural Networks Find Simple Solutions: The Many Regularizers of Geometric Complexity (2022) defines geometric complexity through input derivatives of the model function, which is different from the statistic measured here. Similarly, Section 5.2 measures raw gradient ratios, whereas AdamW updates also depend on optimizer state. Figure 6 shows that restricting adaptation to different modules changes behavior, but it does not isolate the observed gradient-ratio shift as the mediator of ordinary training. Matching LoRA rank also does not necessarily match trainable parameter counts across FFN and attention modules. These results support a plausible mechanistic hypothesis, but the stronger claim that the mechanism has been established is premature.

Questions for authors
  1. Is there a pretraining comparison that preserves exposure to short-prefix prediction while varying access to informative distant context? Evidence that the degradation persists under such a control would substantially strengthen the internalization interpretation.

  2. In the synthetic experiments, how is the intended operation specified independently of the demonstrations, and have checkpoints trained with different demonstration counts been evaluated using a common test-time count? This would clarify whether Figure 4 isolates a learned change in strategy or also reflects changes in the evidence presented at inference.

  3. Are the SFT gradient ratios measured on LoRA factors or on effective weight updates, and are trainable parameter budgets matched in Figure 6? Is there evidence that the complexity relationship persists after accounting for training-loss trajectories? These details would determine how much mechanistic weight the gradient and intervention results can carry.

2 comments

We thank the reviewer for the careful review and for recognizing the usefulness of the pretraining and SFT experimental design.

W1a / Q1: Short-prefix exposure as an alternative explanation for Sec. 3.1.

TL;DR: Three results in the submission and a new control rule out short-prefix exposure.

Exposure alone cannot explain the curve's shape. Exposure to short prefixes falls monotonically from W=512. Yet MCQA (mean input 65 tokens) improves from 512 to 2048, a range in which matched-length exposure drops 4 times. Every optimum also sits 15–32 times above the mean eval length (App. F), not at the shortest window that covers the input, which is where an exposure account would put it.

Sec. 3.2 removes this confound by design. Context length is fixed, so the prefix-length distribution is approximately matched across kk (Table 15). Only relevance changes, yet the supporting-conflicting gap still grows significantly at all 5 sizes (App. G.3).

The module-level signature is shared across experiments. Pretraining shows the same FFN-to-SA gradient shift as SFT (Fig. 5b, significant at every scale). Positional undertraining does not predict this.

New control (General Response E1, App. C.1). Stitched windows match short-prefix exposure at every WW but remove informative distant context. Their loss stays flat, showing that reduced exposure alone cannot explain the degradation.

W1b / Q2: Synthetic identification.

The operation is specified by an explicit token in every sequence (e.g., ... 0 0 1 0 OP_MSB EQUALS ...), so the correct rule is defined independently of the demonstrations, and demonstrations in conflicting context setup contradict this token.

As requested, we re-evaluate all bitwise checkpoints with a common count of 128 demonstrations (General Response E2, App. C.2). With identical test inputs, the gap still grows with training context (0.03 → 0.57), so Fig. 4 reflects a learned change in strategy, not the evidence shown at inference.

W1c: Prop. 4.1.

We agree, and we do not claim otherwise. L292 states this as an achievability result. Realization of IAP is addressed empirically, which is the role of Sec. 5. Please also see General Response D1.

W2 / Q3: Mechanistic measurements.

TL;DR: All three measurements follow prior work and are corroborated by independent evidence.

Gradient norm. We follow prior work in using the training gradient norm as a proxy for solution complexity. Implicit gradient regularization shows that gradient descent implicitly penalizes the gradient norm [1], and this extends to SGD [2]. This penalty in turn controls the geometric complexity of the learned function [3]. Consistent with this, the drop appears only for context-addicted tasks (Table 9). We have made this explicit in the revised Sec. 5.1 (L330–332).

Raw gradients vs. AdamW. Ratios are computed on the LoRA factor gradients before clipping. Our claim concerns where the loss exerts pressure, which raw gradients measure directly, following the transformer dynamics work we build on [4, 5]. Adam largely normalizes per-coordinate update magnitudes, so update norm ratios would mostly reflect parameter counts and mask this signal. The gradient analysis is also not standalone, since its prediction is confirmed downstream, where k=8 models assign significantly more attention mass to context at inference (App. I.3).

Fig. 6 and parameter counts. Following standard LoRA practice, we fix the rank (r=64) across modules, so budgets are not matched. At fixed rank, FFN-only training has roughly 1.2-1.8 times more trainable parameters than SA-only. Yet, a capacity difference cannot explain Fig. 6. It would shift all conditions in the same direction, but we observe a crossover. SA-only training, despite having fewer parameters, is better under supporting context and worse under no and conflicting context. Moreover, all-module training, which has the most trainable parameters, falls between FFN-only and SA-only training at every size (e.g., 1.7B conflicting: 23.1 / 17.7 / 13.1). We agree that Fig. 6 provides causal support for distinct roles of the two modules, matching the direction of the gradient shift, but it is not a formal mediation analysis. We have softened the Fig. 6 caption accordingly.

In summary, the confound the reviewer raises is excluded by design in Sec. 3.2, inconsistent with the shape of the curve in Sec. 3.1, and contradicted by the new stitched-window control.

We hope these clarifications and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider updating their assessment.

Thank authors for the additional experiments, which address some parts of my initial concerns about . I decide to raise my rating from 2 to 3.

Reviewer_WZGDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper investigates how long-context training affects the learning behavior of language models. The authors propose the Information Abundance Paradox, hypothesizing that abundant task-relevant information in context may shift learning from parametric internalization toward contextualization. The hypothesis is investigated through controlled pretraining, SFT with varying context relevance, synthetic tasks, an information-theoretic formulation, and FFN/attention analyses.

Critical strengths

Strengths

  1. Important research question. Understanding how long-context training changes model behavior is timely and important.
  2. Intuitive hypothesis. I particularly appreciate the distinction between parametric internalization and contextualization. It provides a useful perspective on how models learn and utilize information.
  3. Thoughtful experimental design. The paper investigates the hypothesis at multiple levels, with useful controls over training conditions and context relevance.
  4. Interesting mechanistic analysis. The FFN/attention gradient analysis and module-restricted fine-tuning offer interesting evidence about the mechanisms of context reliance.
Critical weaknesses

Weaknesses

W1. Context reliance does not necessarily imply reduced parametric knowledge.

Figure 3 varies the number of task-relevant training documents (k∈{0,4,8}k\in\lbrace 0,4,8\rbrace) under a fixed eight-document budget. Increasing kk improves supporting-context performance but reduces no-context accuracy and robustness to conflicting context. Notably, the effects are asymmetric: from k=0k=0 to k=8k=8, no-context accuracy drops by only 1.7–6.3 points (not significant for Qwen3-8B, p=0.0645p=0.0645; Table 8), whereas the supporting–conflicting gap grows by 18–43 points. However, this does not distinguish (i) reduced knowledge internalization, (ii) unchanged knowledge with a stronger preference for contextual evidence, and (iii) train-test distribution mismatch. The experiments establish a behavioral shift more directly than an actual reduction in parametric knowledge.

W2. Context length and effective long-range information are not fully disentangled.

Section 3.1 varies the training window W∈{512,…,32768}W\in\lbrace 512,\ldots,32768\rbrace over long, coherent documents. While the token budget and optimization steps are controlled, longer documents do not necessarily provide stronger long-range predictive dependencies. Most next-token targets might still be predictable from local context. In addition, the 10B-token budget corresponds to 4 epochs over a 2.5B-token corpus (App. C.1), so data repetition may interact with window length. Thus, the non-monotonic trends in Figure 2 could partly reflect length-dependent optimization or training-evaluation mismatch rather than information abundance. The effective predictive value of distant tokens is not characterized.

W3. The theory does not yet explain the optimization mechanism.

Proposition 4.1 establishes Im(ρ)≤Ik(ρ)\mathcal I_m(\rho)\leq\mathcal I_k(\rho) for nested contexts with m>km>k. This concerns the minimum required parametric information, not the information actually encoded by SGD-trained models. Moreover, Section 4 is presented as formalizing the SFT results of Section 3.2, but the proposition requires nested inputs (X(k)=Tk,m(X(m))X^{(k)}=T_{k,m}(X^{(m)})), whereas Figure 3 replaces irrelevant documents with relevant ones under a fixed budget. The theory therefore does not cover the experiment it is meant to explain. In Section 5.1, average gradient norm is used as a proxy for solution complexity, but its relationship to learned function complexity and optimization preference is not sufficiently justified.

W4. The evaluation does not fully establish the long-context capability trade-off.

The pretraining evaluations focus on language modeling, SuperGLUE, and closed-book MCQA. These characterize certain aspects of performance but do not directly evaluate long-range retrieval, aggregation, and reasoning. Without dedicated long-context evaluations (e.g., LongBench or OOLONG), it is difficult to determine whether increased context utilization actually accompanies the observed degradation in context-independent performance.

I am open to raising my score if the concerns above are adequately addressed.

Questions for authors

Questions

Q1. Direct evidence for internalization (W1). Could the authors introduce controlled, previously unseen factual associations or synthetic rules, vary whether the corresponding information is supplied during training, and then measure closed-book retention? This could help distinguish reduced internalization from changes in inference-time context preference. How much of Figure 3's no-context degradation might instead arise from distribution mismatch?

Q2. Effective dependency length (W2). Could the authors quantify the predictive benefit of distant tokens, for example through the reduction in next-token NLL when extending the available history from L1L_1 to L2L_2? Alternatively, could they keep the nominal window LL fixed while varying the density or distance of task-relevant dependencies? This would help isolate information abundance from sequence-length effects.

Q3. Origin of the inflection points (W2–W3). Figure 2 shows different approximate optima for LM (≈8K) and SuperGLUE/MCQA (≈2K). App. E attributes this to evaluation-length distributions, but the evidence seems weak: the stated rule is 242^4–252^5 times the mean example length, yet SuperGLUE's ratio (2048/135 ≈ 15) already falls outside it, and the conclusion restates it as "one to two orders of magnitude," a much looser band. Moreover, PTB's optimum is at W=512W=512, the smallest window tested, which is a boundary rather than an inflection. Could the authors reconcile these numbers? Does the information-abundance hypothesis itself make any prediction about when an inflection should occur, or how it should change with model capacity and effective dependency length?

Q4. Interpretation of contextualization (W1, W4). Could the authors clarify the distinction between information being available, the model being able to retrieve and reason over it, and the model having internalized it parametrically? In particular, if reliable external information is accessible, increased contextualization may be beneficial rather than pathological. How should context addiction be interpreted in retrieval-augmented or tool-using settings?

5 comments

We thank the reviewer for the constructive review and for appreciating the internalization-contextualization distinction and our thoughtful experimental design.

W1 / Q1: Context reliance vs. reduced parametric knowledge.

Preference (ii) cannot explain a no-context drop. With no context at test, there is nothing to prefer. An unchanged knowledge explanation predicts flat no-context accuracy, yet it falls significantly in 4 of 5 sizes (Table 8).

Mismatch (iii) is held constant. Every condition, including k=0k=0, trains the models with 8 prepended documents, so the no-context test is equally out-of-format for all kk and cannot produce a trend in kk. Table 16 also quantifies this mismatch directly. A no-context-trained baseline differs from k=0k=0 by at most 3.7 points, which is smaller than the k=0→k=8k=0 \to k=8 drop at every size.

The asymmetry is expected. The gap compounds two opposing shifts (e.g., 0.6B: +26.1 supporting, −16.7 conflicting), while no-context accuracy reflects one. Still, the drop is substantial, as k=8k=8 no-context accuracy falls below even the untuned base model at every size (Table 16).

Our pretraining experiments already test Q1. In our pretraining experiments (Sec. 3.1), models are trained from scratch on identical data across WW, so all knowledge is previously unseen. Closed-book MCQA measures its retention with no test-time context and declines beyond 2K. Furthermore, Sec. 5.1 provides the synthetic-rule study the reviewer asks for, training from scratch on unseen rules while varying train-time context. The newly added control (General Response E1, App. C.1) addresses train-test length mismatch as an alternative cause.

W2 / Q2: Context length vs. effective long-range information.

We now run Q2's second suggestion (General Response E1, App. C.1), showing that both the gain and the degradation in performance require informative distant context.

W3: Theory.

Minimum vs. learned information. We agree with this distinction. Notably, Prop. 4.1 is already stated as an achievability result (L292), and realization is addressed empirically in Sec. 5. Please also see General Response D1.

Nesting vs. the SFT design. Thank you for catching this. Sec. 4 is meant to formalize the pretraining results of Sec. 3.1, where nesting holds exactly. The reference to Sec. 3.2 was a writing error, which we corrected in the revision.

Gradient norm. We follow prior work in using the training gradient norm as a proxy for solution complexity. Implicit gradient regularization shows that gradient descent implicitly penalizes the gradient norm [1, 2]. This penalty in turn controls the geometric complexity of the learned function [3]. Consistent with this, the drop appears only for context-addicted tasks (Table 9). We have made this explicit in the revised Sec. 5.1.

W4 / Q4: Long-context evaluation and interpretation.

Scope of the claim. Long-context capability is well studied, with many benchmarks for retrieval, aggregation, and reasoning over long inputs (Sec. 6). Our work targets an overlooked question and asks how the training context window shapes what the model internalizes as parametric knowledge. Hence, our evaluations focus on performance without long contexts. However, we still measure increased context use directly, and find that supporting-context accuracy rises with kk (Fig. 3).

Dedicated long-context benchmarks. LongBench or OOLONG would sit at floor for 20M–750M base models trained on 10B tokens, as both benchmarks require free-form generation and instruction following over long inputs, which these models are not trained for.

When contextualization is beneficial. We agree, and Sec. 7 already notes that whether the shift is beneficial depends on the application. Context addiction matters when context is absent or misleading, as in the case of noisy retrieval, which our conflicting condition simulates.

Availability, retrieval, and internalization. In Q4's terms, availability is what the window contains, retrieval is measured by supporting-context accuracy and attention mass analysis, and internalization by no-context and closed-book accuracy.

Q3: Inflection points.

Evidence in App. F. We present App. F as suggestive. We now state the band consistently as 15 to 32 times, covering all five observed ratios, and mark PTB’s optimum as a boundary.

What IAP predicts. IAP predicts that an inflection appears once training context supplies information that evaluation inputs do not. Its location therefore tracks evaluation length, and it vanishes when distant information is removed (General Response E1, App. C.1). The inflection persists from 20M to 750M (Fig. 2). However, we do not claim a capacity law.

We hope these clarifications and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider updating their assessment.

Thank you again for your thoughtful review and constructive feedback, which have helped us improve our paper. We wanted to kindly follow up on our rebuttal and new experiments, and would appreciate hearing whether they address your concerns or if any questions remain.

I thank the authors for the new experiments. The stitched-window control addresses W2 at the two sizes tested, and I also acknowledge that the from-scratch setting partly answers Q1: closed-book MCQA declines beyond 2K while LM loss is still improving, which a general capability loss would not produce.

My remaining concern is that this supports a narrower claim than the title. No-context accuracy measures whether knowledge is accessible under a short prompt, not whether it was encoded. A k-dependent access policy — k=8 models learning to read the answer off the documents — produces the same drop without knowledge loss, and the “nothing to prefer” reply only rules out a k-independent preference. One cheap test, in the spirit of my Q1: evaluate the k=8 models with 8 irrelevant documents, i.e. the k=0 format. Recovery would point to access; no recovery would support reduced internalization.

I will keep my rating at 2, and would move to 3 if that test supports the internalization reading, or if the claim is scoped accordingly in the title and abstract.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and valuable feedback. Reviewers found the question important and timely (WZGD), the pretraining sweep rigorous and well-controlled (rhtz, uif7), the SFT design a clean separation of relevance from added text (rhtz, uif7), and the module-restricted fine-tuning informative causal evidence (rhtz, WZGD). Below we report new experiments and discussions that address concerns shared across reviews. We have uploaded a revised version of the paper, with all changes marked in blue.

E1. New: stitched-window control (uif7 Q1, WZGD Q2).

TL;DR: Removing distant context while matching everything else eliminates both the gain and the degradation, so the effect comes from informative context, not from sequence length.

We address whether the Sec. 3.1 degradation stems from informative distant context or from a length-related confound, i.e., short-prefix exposure (uif7) or length-dependent optimization and data repetition (WZGD). We repeat our pretraining sweep at 20M and 55M with stitched windows, where each window of length WW is built from W512\frac{W}{512}-many shuffled 512-token chunks of different documents, separated by a special token. Length, tokens, epochs, steps, and short-prefix exposure all match coherent windows at every WW. Therefore, only informative distant context is removed.

Results (Fig. 8, App. C.1):

  • Stitched loss stays flat. At every WW, it is within 0.07 of the loss for W=512W=512, which remains inside seed variability.
  • Coherent loss is U-shaped. It improves by 0.31 up to 8K, then degrades by 0.44 (20M) and 0.30 (55M) by 32K, while stitched loss changes by only +0.02 and −0.03 over the same range.

Both the gain and the degradation therefore require informative distant context, as the Information Abundance Paradox predicts. Length-related factors are shared by both settings and thus cannot explain the difference.

E2. New: synthetic re-evaluation at a common demonstration count (uif7 Q2).

TL;DR: With identical test inputs, context addiction still grows with training context, so it is a strategy learned in training, not an artifact of the test-time demonstrations.

We now re-evaluate all bitwise checkpoints with the same 128 test-time demonstrations, so every model sees identical inputs. As shown in Fig. 9 (App. C.2), the supporting-conflicting gap still grows with training context (0.03 → 0.23 → 0.51 → 0.57), so Fig. 4 reflects a learned change in strategy, not the evidence shown at inference.

D1. Scope of the theory (rhtz W1, uif7 W1c, WZGD W3).

In the paper, we already acknowledge that Prop. 4.1 is an achievability result (L292), where more informative context permits storing less task information in the weights. Whether training selects such solutions is addressed empirically in Sec. 5, through synthetic selectivity, gradient allocation, module-restricted tuning, and inference-time attention.

We also corrected a writing error in Sec. 4, which formalizes the pretraining results of Sec. 3.1, not Sec. 3.2 (WZGD).

References

[1] Barrett, D. G. T., & Dherin, B. (2022). Implicit gradient regularization. arXiv:2009.11162.

[2] Smith, S. L., Dherin, B., Barrett, D. G. T., & De, S. (2021). On the origin of implicit regularization in stochastic gradient descent. arXiv:2101.12176.

[3] Dherin, B., Munn, M., Rosca, M., & Barrett, D. G. T. (2022). Why neural networks find simple solutions: The many regularizers of geometric complexity. arXiv:2209.13083.

[4] Zhang, B., Titov, I., & Sennrich, R. (2019). Improving deep transformer with depth-scaled initialization and merged attention. arXiv:1908.11365.

[5] Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2023). Understanding the difficulty of training transformers. arXiv:2004.08249.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.