acceptodds
Under review as a conference paper at ICLR 2027

Predictive Anchoring Shapes Contextual Dependence in Language-Model Pretraining

Abstract

We use knowledge distillation to study learned contextual dependence during language-model pretraining by varying teacher context regimes and composition, while holding the student's architecture and input window fixed. A frozen short-context teacher supplies a pressure to match locally conditioned predictions, alongside ordinary next-token learning. This short-context distillation (sKD) improves baseline predictions across WikiText-103 and SlimPajama-100M, while attenuating average contextual effects both where distant text helps baseline prediction and where it hurts. An equal-context distillation (eqKD) control, whose teacher receives the same causal input as the student, achieves lower overall loss than sKD but retains more of both helpful and harmful influences. We observe a tradeoff between helpful-context use and harmful-context attenuation in these contrasting distillations. Mixed-teacher distillation (mixKD) trains a single student using a fixed mixture of the two teacher distributions. mixKD achieves overall prediction loss close to eqKD, with lower loss in most, but not all, comparisons. It demonstrates substantially improved prediction on helpful positions compared with sKD, while retaining a prediction advantage over eqKD on harmful positions. It also shows greater average contextual benefit on helpful positions than sKD and a smaller average contextual penalty on harmful positions than eqKD. These findings hold across three seeds on validation and test data, at two evaluated short-context horizons on WikiText-103 and one on SlimPajama-100M. Equal-weight ensembles of separately trained sKD and eqKD students achieve lower overall prediction loss than mixKD, yet an exploratory post-test analysis finds that no fixed probability weighting of these student pairs simultaneously matches or improves on both mixKD's helpful-context benefit and harmful-context penalty. This study shows that similar overall prediction loss can accompany different learned contextual behavior, which can be shaped by teacher context regimes and composition.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language model pretraining, knowledge distillation, contextual dependence, multi-teacher distillation, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wrcmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

This paper studies how teacher context regimes shape contextual dependence during language-model pretraining. It compares short-context distillation (sKD), equal-context distillation (eqKD), and an equal probability mixture of their frozen teachers (mixKD), while retaining the student architecture and input access. Contextual effects are measured through full-versus-restricted observed-token losses on populations labeled by other-seed baselines. sKD improves overall prediction while attenuating both helpful and harmful contextual effects, and mixKD provides an intermediate tradeoff with loss close to eqKD. Student ensembles predict better, but the tested constant probability mixtures do not reproduce mixKD’s joint contextual response.

Critical strengths

The evaluations are comprehensive and carefully designed. Population labels are fixed across methods and obtained from other-seed baselines, which makes the differences between helpful and harmful contextual effects convincing.

Critical weaknesses
  1. The advance over prior work appears modest. Restricted-teacher supervision, long–short prediction alignment, and the long–short log-probability difference used here have already been studied [1]-[3]. The main additions are the helpful/harmful breakdown and the mixed-teacher comparison. These are useful, but the paper provides limited new explanation of when or why the observed tradeoff occurs.
  2. The significance of the ensemble comparison is unclear. mixKD has a different contextual-gain profile, but the equal-weight student ensemble has lower prediction loss overall and in both the helpful and harmful groups. Since contextual gain compares each model against its own short-context prediction, a larger gain does not by itself imply better prediction with distant context. Why this difference matters for understanding context use remains unclear to me.

[1] Induced Model Matching: Restricted Models Help Train Full-Featured Models.

[2] Long-Short Alignment for Effective Long-Context Modeling in LLMs.

[3] What is Wrong with Perplexity for Long-context Language Modeling?

Questions for authors
  1. Does a long–short alignment baseline show a similar tradeoff under the same helpful/harmful analysis? This would help assess what teacher mixing adds beyond prior work.
  2. What conclusion about context use does the mixture-versus-ensemble comparison support beyond differences in the models’ short-context prediction quality?
2 comments

We thank the reviewer for recognizing the clarity of the paper and the care taken in the evaluations. As you note, the main additions are the helpful/harmful breakdown and the mixed-teacher comparison.

On contribution and interpretation Comparing sKD and eqKD, which differ only in their respective teachers receiving short or full context (both at pretraining as well as distillation), reveals opposing strengths. eqKD predicts better on helpful positions, while sKD predicts better on harmful positions (and attenuates both helpful/harmful contextual effects more). These observations motivate mixKD, which recovers much of eqKD's helpful-position advantage while retaining a harmful-position advantage over it, at close overall prediction loss. These prediction contrasts repeat across all nine test settings (Tables 15-17). The key observations are that reduced contextual dependence can accompany both a predictive advantage and a predictive cost, and teacher mixing can change that two-sided tradeoff.

Section 6 proposes a "burden on distant information" interpretation. The short teacher supplies a learned local preference that remains fixed during student training. When the two teachers in mixKD disagree, the full-context-informed preference is counterbalanced by that local preference. In the mixture target, a weaker opposing preference may fail to overturn the local ordering while stronger ones survive. This pressure can restrain both helpful and harmful departures. While this interpretation explains only the supervisory pressure, not the exact emergent learned compromise, it motivates testing whether such supervision could improve resistance to distracting distant evidence. Such an application is untested at this point. In our current paper, helpful/harmful labels describe predictive effects, not semantic relevance.

Relationship to prior work IMM [1] uses induced matching, where a short-context reference is matched with student predictions averaged over longer histories sharing that suffix, rather than to each prediction individually. Its analysis shows that direct matching can prevent recovery of the true full-context predictor, even with unlimited data. Our finite-data study complements this by measuring both the benefits and costs of direct matching, and how teacher composition changes their balance. Applying induced matching within the same comparisons could test how it changes the two-sided tradeoff.

LSA [2] regularizes agreement between evolving predictions of the same model to improve length generalization. Our design compares two independently trained, frozen teacher regimes under the same KD weighting, then combines the supervision. Their contrasting strengths naturally motivate the two-sided characterization and mixture experiment, rather than only varying agreement strength. This does not mean that the two-sided tradeoff analysis cannot be useful for LSA.

LongPPL/LongCE [3] emphasize tokens whose predictions benefit from long context, for evaluation or training. We use the same contrast but also explicitly study positions where adding distant text makes predictions worse. This reveals what helpful effects are lost alongside harmful-effect attenuation, and how teacher mixing changes that balance. We are not claiming a new contextual metric.

On the requested LSA comparison, we have not done it in this analysis. In our understanding, there is no naturally equivalent mapping from teacher-mixture weights in mixKD to an LSA alignment coefficient; teacher composition and alignment strength are different controls. A valid comparison would need explicit coefficient-selection and context-construction choices. A specified LSA baseline could still be informative and could produce a similar profile, and we are not claiming that LSA could not have produced one.

On the ensemble and contextual gain We agree that mixKD's larger contextual gain does not establish better prediction. The equal-weight ensemble predicts better under both full and restricted input. Its larger restricted-input advantage on helpful and harmful positions explains its smaller contextual gain, as acknowledged in sections 5.3 and 6.

The ensemble was introduced as a control which tests whether an inference-time mixture of sKD and eqKD reproduces the contextual response of training-time mixture mixKD. As the data in sections 5.3 and appendix C.4.1 shows, mixKD's contextual response could not be reproduced by any of the tested fixed-weight combinations of the ensemble. This is not a claim of advantage, it only shows that they have a different helpful/harmful profile. Another resource distinction (and not a claim of superiority) is that mixKD evaluates one student per prediction, while the ensemble evaluates two.

Thank you for your detailed response and revisions. I appreciate your efforts, but I am still not fully convinced, particularly regarding the novelty and practical significance of the findings.

I will maintain my original rating of 2 (Weak Reject). I encourage you to further strengthen the work and submit the revised version to a future conference.

Reviewer_VSSvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • Problem: how does na objective incentivize a LM to use or ignore long context inputs
  • Setup: 204M decoder pretrined from scratch on WikiText-103 and SlimPajama-100M, with KD added to next-token prediction
  • Students differ only in teacher: sKD (frozen teacher sees only the last H tokens), eqKD (teacher sees the same context as the student), mixKD (fixed 50/50 mixture of the two teacher distributions)
  • Context use = loss change between an h-token suffix and the full prefix (Eq. 4). Seeing the full prefix can be helpful (improves CE) or harmful (worsens CE)
  • Findings: sKD attenuates both helpful and harmful context effects, eqKD retains more of both, mixKD falls in between (Table 1, Figs. 2-3). no sKD+eqKD ensemble can match mixKD's profile despite lower CE (Table 29, Fig. 4)
  • Takeaway: loss doesn't tell you how good models are at at using early context. the way the teacher is designed can encourage/discourage the utility of long context inputs.
Critical strengths
  • Interesting question of how teacher choice shapes student behavior
Critical weaknesses
  • The main finding is basically true by construction. The student is trained to match a teacher that sees only the last H tokens, and context dependence is then measured as how much the student's prediction changes between seeing only the last H tokens and seeing the full context. There is no evaluation at a different lengths or long-context tasks, so the results do not show anything beyond what the loss directly enforces. Actually the authors themselves acknowledge that uniformly scaling down the context dependence would reproduce the same helpful/harmful pattern (L2430-2434). So I don't know what the takeaway is here.

  • The claim that the mixed-teacher and equal-context-teacher students have similar loss is not clear. They have a gap of up to 0.011 nats, which is a lot for pre-training.

  • The closest prior work is not compared: Induced Model Matching (Muneeb & Ohannessian, NeurIPS 2024), which argues against the direct matching thing in this paper, n-gram data noising (Xie et al., ICLR 2017), and context-length curricula such as Shortformer (Press et al., ACL 2021) and sequence-length warmup (Li et al., NeurIPS 2022).

Also I don't know where to put this, but I think the entire paper is centered on one experiment that is not well-designed. The claims are much broader than what this experiments supports.

Questions for authors
  • Do you have any long-context or at least different context length evaluation? Like if the short-context teacher has a 64-token context do you evaluate at 32 and 128 tokens? Otherwise I don't understand how you make the claims in the paper.
  • Do you have any experiments that show that the effect of distillation is any different from standard regularization? Baselines are early-stopped but students are massively overtrained. So how is the comparison reasonable?
2 comments

We thank the reviewer for recognizing the question of how teacher choice shapes student behavior.

On the finding being expected by construction: We agree that a short-context teacher supplies an expected local pressure. However, sKD matches the student's full-input prediction to a separate frozen teacher, not to the student's own restricted-input prediction. So, setting the diagnostic cutoff h equal to the teacher horizon H does not make contextual gain the optimized quantity. The uniform-shrinkage example at lines 2430-2434 explains why opposite signed contextual changes alone do not establish semantic selectivity (which the paper does not claim). It does not determine the accompanying full-input prediction gains and costs, which also depend on the restricted-input prediction (Eq. 5).

Our contribution connects the two-sided helpful/harmful characterization with the mixed-teacher results. eqKD predicts better on helpful positions, while sKD predicts better on harmful positions despite its worse overall CE. mixKD recovers much of eqKD's helpful-position advantage while retaining a harmful-position advantage over it. An intermediate response is plausible, but the mixture identity does not automatically prescribe this learned compromise. In summary, our study measures the predictive costs and benefits of local anchoring and how teacher composition changes their balance.

On context lengths and claim scope: We report H=h=32,64 on WikiText-103 and H=h=64 on SlimPajama-100M, but have not evaluated fixed-H64 checkpoints at h=32,128 or established downstream evidence-use benefits. Cross-cutoff evaluation would test how the characterization extends across distance boundaries, but our claims concern the measured predictive and contextual tradeoffs, not semantic selectivity or generalization to arbitrary context lengths.

On similar overall loss: We used "similar" to describe proximity, not equality, statistical equivalence or a negligible difference (Table 1). The high-level point is that aggregate differences do not describe the contrasting helpful/harmful prediction differences.

On regularization and training duration: Both baselines and distilled students use minimum ordinary validation CE for checkpoint selection, excluding KD, and five-epoch early stopping (section 4). KD checkpoints were selected later as a consequence of the choice to compare using a shared-selection-rule, and thus, are not equal-exposure comparisons.

We completed a three-seed WT103 residual-output dropout (p=0.1) control on the same test targets and fixed helpful/harmful populations at h=64. At epoch 9 (which matches nominal training-token exposure with the baseline), mean full-input CE falls from 3.2317 to 3.1891. Validation-selected epoch-12 dropout improves it further to 3.1609.

Validation-selected dropout and sKD have close overall CE (3.1609 versus 3.1562), but dropout improves helpful-position prediction relative to baseline while sKD H64 worsens it. Both improve harmful-position prediction, with a larger gain under sKD. eqKD and mixKD predict better than dropout, overall as well as on both helpful and harmful positions. These comparisons repeat across seeds. This tested regularizer yields substantial gains without reproducing sKD's tradeoff, but other regularizers could produce a similar predictive and contextual profile.

Relationship to the suggested prior work: We have not evaluated these alternatives under the same protocol or established superiority over them. We will add the dropout results and expand section 2 to include these additional related works.

IMM is already discussed briefly in section 2. It matches a restricted reference to student predictions averaged over compatible longer histories. Its idealized analysis shows that direct matching can fail to recover the true full-context predictor. Our finite-data study complements this by measuring direct matching's predictive benefits and contextual costs. We are not claiming that mixKD removes that constraint.

Data noising connects corrupted text to smoothing between higher- and lower-order predictions. Both approaches make less-specific information useful during learning. Noising changes training examples; anchoring retains uncorrupted student inputs and supplies a frozen local predictive reference. We also test equal-context and mixed supervision.

Shortformer's staged training carries short-input-trained weights into a longer-input phase. mixKD instead retains both short- and equal-context predictions as simultaneous frozen supervision. Sequence-length warmup gradually increases learner input length to improve stability and efficiency. Our student window stays fixed across conditions, and teacher parameters and mixture weights remain fixed within each run.

Thanks for running the dropout comparison, which answers my question about regularization. I'm still not sure what the practical takeaway is though, since there's nothing at other context lengths or on downstream tasks. Keeping my score.

Reviewer_wfJ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a teacher's context window shapes what a student learns. Authors design the experiments following this process: Given the same student, which has fixed 1024 input tokens, authors construct two teachers whose context window is short (32/64) and full. They applied three objectives, sKD, eqKD and mixKD. sKD refers to the teacher with the short context and eqKD means the teacher with the full context (same as the student), and mixKD is the mixture of two teachers. They run the experiments using WikiText-103 and a 100M-token SlimPajama sample. Several findings are given based on the observation: 1. All three distilled students reach lower loss than the baseline. 2. The sKD relies less on distant text compared with the baseline, while eqKD keeps much more of this reliance. 3. MixKD’s overall loss is nearly the same as that of eqKD, but its reliance on distant text falls between that of sKD and eqKD.

Critical strengths
  1. The research question is reasonable. Whether two models at similar loss can differ in how they use distant text, is worth asking.
  2. The experiment design is thoughtful. Authors decompose the improvement into restricted-input improvement and contextual-gain change (Eq. 5), and the paper warns that average attenuation is not tokenwise contraction.
Critical weaknesses
  1. In my understanding, much of the headline pattern is expected by construction. A teacher that focuses on distant text will pull the students’ local performance, and a mixture will result in the student in between.
  2. The loss improvement look confounded with regularization. Training on multi epochs on nearly 100M tokens with no dropout. Baselines peak around epochs 8–9 and stop at 13–14, while distilled students peak at epochs 15–21 after roughly 1.8–2.3× the tokens (from Tables 5, 6). KD here may just be a regularizer that postpones overfitting, which would also explain why even the weaker short-context teacher improves CE. I would want a regularized baseline (dropout or label smoothing)
  3. Nothing show the measured differences matter. There is no downstream or evidence-use evaluation, even though the stated motivation is a document assistant that must use earlier relevant text and ignore distractors.
  4. A minor one. Generality is somewhat narrow. Only around 200M models are trained for many epochs using nearly 100M tokens is far from the single-epoch large-scale regime the "pretraining" framing suggests.
Questions for authors

Is there any probe, even a small retrieval or distractor task, showing that mixKD's contextual profile translates into better use of relevant distant text?

2 comments

We thank the reviewer for recognizing the value of the research question and the care taken in the experiment design. We agree that short-context supervision exerts an expected pressure toward locally conditioned predictions, but this pressure alone cannot determine the predictive and contextual outcomes in mixKD. This is illustrated by the different contextual responses of eqKD and mixKD despite similar overall prediction loss. More specifically, mixKD recovers much of eqKD's helpful-position prediction advantage while retaining much of sKD's harmful-prediction advantage. mixKD also has lower full-input CE than the mean of sKD and eqKD on both helpful and harmful positions in all 9 reported test settings (Tables 15-17). This shows a repeated pattern in what the student predicts well and how it responds to distant text, rather then merely confirming the direction of the short-context pressure. Unequal teacher weights are an untested extension, which would test if the unequal weighting provides a lever to control the balance between helpful/harmful positions.

A natural transition from the current findings in the paper is to test whether a similar effect is observed with relevant, irrelevant, or distracting distant contexts. Unfortunately, we currently do not have a successful evaluation showing that mixKD's contextual profile translates into better use of semantically relevant distant information. We agree that such an evaluation would be valuable for establishing practical implications. We briefly explored a synthetic entity-object probe, but the baseline model failed its task-capability checks even with nearby facts, perhaps because the models were not finetuned for following instructions. We did not pursue it further in favor of other pressing tasks needed to complete the study. This is briefly mentioned in the second paragraph of the AI use statement on page 10.

The document-assistant example motivates why contextual dependence is worth examining, but it is not an evaluated application (as stated briefly in the ethics statement on page 10). Our helpful/harmful populations describe only the observed-token prediction effects, not semantic relevance. Our key contribution using pretraining experiments is the relationship between teacher context regimes with the predictive and contextual response. Whether this relationship can be exploited to improve retrieval or distractor-resistance remains open, and not claimed by this submission.

On the topic of pretraining, we agree that the evaluated model sizes and data regime limit the scope of generalization. In this paper, "pretraining" refers to training base language models from fresh initialization on next-token prediction task (with auxiliary KD, when applicable), rather than task-specific or instruction finetuning. There are other published instances of multi-epoch language-model pretraining, such as "Scaling Data-Constrained Language Models" by Muennighoff et al. (2023).

On the question of regularization, we agree that it may contribute to KD's improvement over the CE-only baseline, and the paper does not claim otherwise. The first paragraph of Section 2 explicitly discusses this perspective. The baseline provides a common reference, and establishes that improved prediction can coexist even with attenuation of both helpful and harmful contextual effects under sKD. We do not attribute the prediction improvement to attenuation. The primary focus of the paper is to compare the various distillation setups sKD, eqKD, and mixKD. eqKD, in particular, provides an active distillation comparison without the short-context teacher restriction. So, a regularization explanation for improvement over baseline is compatible with this study's findings. All conditions use the same validation-CE selection and early-stopping rules, although their selected checkpoints reflect different training exposure (Section 4; Appendix B.3).

We are planning a limited additional teacher-free regularization comparison using dropout or label smoothing, and will share the results in a follow-up response if the analyses are completed and verified within the rebuttal period.

We completed three-seed runs on WikiText-103 for residual-output dropout (p=0.1), and evaluated the models with dropout disabled on the same test targets used in this paper and with unchanged helpful/harmful populations at h=64. At epoch 9, matching the baseline's nominal training-token exposure, dropout lowers mean full-input CE from 3.2317 to 3.1891. Validation selects epoch 12 in all three runs, reducing mean CE further to 3.1609. Thus, dropout's benefit is not solely due to its later selected checkpoint.

At their validation-selected checkpoints, dropout and sKD H64 have close overall CE (3.1609 versus 3.1562), but contrasting helpful-position outcomes relative to the baseline -- dropout improves these predictions, whereas sKD worsens them. Both improve harmful-position prediction, but sKD achieves a larger gain. eqKD improves both populations beyond dropout. mixKD also predicts better than dropout on both, recovering sKD's helpful-position deficit while retaining near-sKD harmful-position performance. These conditional prediction comparisons repeat across all three seeds.

The table reports descriptive three-seed mean full-input CE improvements over the same-seed original baseline, in nats per target, at validation-selected checkpoints. Positive values mean better prediction.

ConditionOverall improvementHelpful-position improvementHarmful-position improvement
Dropout p=0.1+0.0709+0.0941+0.0418
sKD H64+0.0756-0.0150+0.1851
eqKD+0.1386+0.1473+0.1267
mixKD H64+0.1481+0.1179+0.1886

These results show that teacher-free regularization produces substantial prediction gains without reproducing sKD's two-sided tradeoff. Teacher composition changes that tradeoff, as the mixKD comparison illustrates. The selected-checkpoint comparisons are not exposure-matched, and this result concerns the tested dropout configuration, not all regularizers.

Comments on the paper

1 comment

We have updated the manuscript and reproducibility kit following the points raised during the discussion.

Changes to the manuscript are highlighted in blue text for ease of identification. The revisions include the additional teacher-free dropout control, clarifications to the teacher/runtime/evaluation context notation (H, h, and h_eval), and expanded discussion of the related work raised in the reviews.

The reproducibility kit has also been updated to include the additional control, its analysis, and the corresponding manuscript/result mapping.

We thank the reviewers for the feedback, which helped us clarify both the experimental scope and the interpretation of the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.