- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a mismatch between nominal training sequence length and token-level context exposure in long-context continued pretraining. Under document-boundary masking, many target tokens in a long packed sequence are still predicted from relatively short same-document prefixes. The authors propose EXACT, which increases the loss weights of tokens with long effective left contexts and allocates the additional weight according to inverse bucket frequency. The method leaves the training data and model architecture unchanged. Experiments across Qwen and LLaMA configurations show consistent improvements on long-context benchmarks, complemented by ablations and evidence-sensitivity probes.
Critical strengths
A meaningful problem and a promising perspective. The distinction between available context length and the allocation of token-level supervision provides a useful perspective on long-context training. The paper suggests that increasing the training window alone may be insufficient if relatively little supervision is assigned to targets with long visible contexts. This insight has the potential to inform practical training recipes.
A simple and practical method. EXACT requires only context-length statistics and per-token loss weighting. It introduces no architectural changes or additional inference cost, making it straightforward to incorporate into existing continued-pretraining pipelines.
Consistent empirical results with useful controls. The improvements across multiple model configurations are encouraging. Keeping the data stream and training setup matched strengthens the comparison with standard continued pretraining. The weighting ablations and evidence-sensitivity probes also provide useful initial support for the proposed explanation.
Critical weaknesses
The mechanism is not yet sufficiently established. The paper measures effective context through the length of the visible same-document prefix. However, a target with a long visible prefix does not necessarily require distant information for its prediction. Consequently, the results establish the usefulness of the weighting rule more clearly than they establish that it works by correcting insufficient supervision of long-range dependencies. The evidence-sensitivity probes are suggestive, but do not fully bridge this gap.
Alternative explanations remain insufficiently controlled. Although EXACT preserves the underlying data stream, it changes the relative contribution of different tokens to the training objective. This may implicitly emphasize longer documents, particular domains, or later portions of documents. Comparisons that control for document length and domain are needed to distinguish the proposed exposure-allocation mechanism from these effects.
The conceptual insight needs stronger theoretical or hypothesis-driven validation. The methodological novelty of context-length-based inverse-frequency weighting is relatively modest. This is not inherently a weakness for a simple and effective method, but it places greater importance on establishing a clear and well-supported explanation. In particular, the paper does not yet show whether the initial degree of exposure imbalance predicts the magnitude of EXACT’s gains. Such a relationship would turn the current empirical observation into a more informative and potentially predictive training principle.
Some experimental details need clarification. More complete specifications of the evidence-sensitivity probes, training configurations, and statistical variability would improve reproducibility and make the mechanistic claims easier to assess.
Questions for authors
-
Controlling for document length and domain: Can the authors evaluate EXACT under matched or stratified document-length and domain distributions? Does the advantage persist when the relative training contribution of these factors is controlled?
-
Testing actual dependence on distant context: For the same target tokens, can the authors truncate distant context while preserving the local prefix and measure the resulting changes in loss or gradients? Does EXACT preferentially improve predictions that demonstrably depend on distant information?
-
Testing the predictive value of exposure imbalance: Across different corpora, packing policies, or training stages, does greater initial exposure imbalance predict larger gains from EXACT? Can the authors define an imbalance measure and test this relationship?
-
Clarifying reproducibility and uncertainty: Can the authors provide the exact answer-margin definition, probe construction and sample sizes, complete training settings, and results across multiple training seeds for representative configurations?