- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2(−1)
Read review
Paper summary
This paper studies how teacher context regimes shape contextual dependence during language-model pretraining. It compares short-context distillation (sKD), equal-context distillation (eqKD), and an equal probability mixture of their frozen teachers (mixKD), while retaining the student architecture and input access. Contextual effects are measured through full-versus-restricted observed-token losses on populations labeled by other-seed baselines. sKD improves overall prediction while attenuating both helpful and harmful contextual effects, and mixKD provides an intermediate tradeoff with loss close to eqKD. Student ensembles predict better, but the tested constant probability mixtures do not reproduce mixKD’s joint contextual response.
Critical strengths
The evaluations are comprehensive and carefully designed. Population labels are fixed across methods and obtained from other-seed baselines, which makes the differences between helpful and harmful contextual effects convincing.
Critical weaknesses
- The advance over prior work appears modest. Restricted-teacher supervision, long–short prediction alignment, and the long–short log-probability difference used here have already been studied [1]-[3]. The main additions are the helpful/harmful breakdown and the mixed-teacher comparison. These are useful, but the paper provides limited new explanation of when or why the observed tradeoff occurs.
- The significance of the ensemble comparison is unclear. mixKD has a different contextual-gain profile, but the equal-weight student ensemble has lower prediction loss overall and in both the helpful and harmful groups. Since contextual gain compares each model against its own short-context prediction, a larger gain does not by itself imply better prediction with distant context. Why this difference matters for understanding context use remains unclear to me.
[1] Induced Model Matching: Restricted Models Help Train Full-Featured Models.
[2] Long-Short Alignment for Effective Long-Context Modeling in LLMs.
[3] What is Wrong with Perplexity for Long-context Language Modeling?
Questions for authors
- Does a long–short alignment baseline show a similar tradeoff under the same helpful/harmful analysis? This would help assess what teacher mixing adds beyond prior work.
- What conclusion about context use does the mixture-versus-ensemble comparison support beyond differences in the models’ short-context prediction quality?
2 comments
Thank you for your detailed response and revisions. I appreciate your efforts, but I am still not fully convinced, particularly regarding the novelty and practical significance of the findings.
I will maintain my original rating of 2 (Weak Reject). I encourage you to further strengthen the work and submit the revised version to a future conference.
We thank the reviewer for recognizing the clarity of the paper and the care taken in the evaluations. As you note, the main additions are the helpful/harmful breakdown and the mixed-teacher comparison.
On contribution and interpretation Comparing sKD and eqKD, which differ only in their respective teachers receiving short or full context (both at pretraining as well as distillation), reveals opposing strengths. eqKD predicts better on helpful positions, while sKD predicts better on harmful positions (and attenuates both helpful/harmful contextual effects more). These observations motivate mixKD, which recovers much of eqKD's helpful-position advantage while retaining a harmful-position advantage over it, at close overall prediction loss. These prediction contrasts repeat across all nine test settings (Tables 15-17). The key observations are that reduced contextual dependence can accompany both a predictive advantage and a predictive cost, and teacher mixing can change that two-sided tradeoff.
Section 6 proposes a "burden on distant information" interpretation. The short teacher supplies a learned local preference that remains fixed during student training. When the two teachers in mixKD disagree, the full-context-informed preference is counterbalanced by that local preference. In the mixture target, a weaker opposing preference may fail to overturn the local ordering while stronger ones survive. This pressure can restrain both helpful and harmful departures. While this interpretation explains only the supervisory pressure, not the exact emergent learned compromise, it motivates testing whether such supervision could improve resistance to distracting distant evidence. Such an application is untested at this point. In our current paper, helpful/harmful labels describe predictive effects, not semantic relevance.
Relationship to prior work IMM [1] uses induced matching, where a short-context reference is matched with student predictions averaged over longer histories sharing that suffix, rather than to each prediction individually. Its analysis shows that direct matching can prevent recovery of the true full-context predictor, even with unlimited data. Our finite-data study complements this by measuring both the benefits and costs of direct matching, and how teacher composition changes their balance. Applying induced matching within the same comparisons could test how it changes the two-sided tradeoff.
LSA [2] regularizes agreement between evolving predictions of the same model to improve length generalization. Our design compares two independently trained, frozen teacher regimes under the same KD weighting, then combines the supervision. Their contrasting strengths naturally motivate the two-sided characterization and mixture experiment, rather than only varying agreement strength. This does not mean that the two-sided tradeoff analysis cannot be useful for LSA.
LongPPL/LongCE [3] emphasize tokens whose predictions benefit from long context, for evaluation or training. We use the same contrast but also explicitly study positions where adding distant text makes predictions worse. This reveals what helpful effects are lost alongside harmful-effect attenuation, and how teacher mixing changes that balance. We are not claiming a new contextual metric.
On the requested LSA comparison, we have not done it in this analysis. In our understanding, there is no naturally equivalent mapping from teacher-mixture weights in mixKD to an LSA alignment coefficient; teacher composition and alignment strength are different controls. A valid comparison would need explicit coefficient-selection and context-construction choices. A specified LSA baseline could still be informative and could produce a similar profile, and we are not claiming that LSA could not have produced one.
On the ensemble and contextual gain We agree that mixKD's larger contextual gain does not establish better prediction. The equal-weight ensemble predicts better under both full and restricted input. Its larger restricted-input advantage on helpful and harmful positions explains its smaller contextual gain, as acknowledged in sections 5.3 and 6.
The ensemble was introduced as a control which tests whether an inference-time mixture of sKD and eqKD reproduces the contextual response of training-time mixture mixKD. As the data in sections 5.3 and appendix C.4.1 shows, mixKD's contextual response could not be reproduced by any of the tested fixed-weight combinations of the ensemble. This is not a claim of advantage, it only shows that they have a different helpful/harmful profile. Another resource distinction (and not a claim of superiority) is that mixKD evaluates one student per prediction, while the ensemble evaluates two.