acceptodds
Under review as a conference paper at ICLR 2027

Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing

Abstract

Long-context adaptation is often treated as window scaling, but packed training with document masking creates a token-level supervision mismatch: most target tokens still receive short effective context. This paper introduces Effective-Context Exposure Balancing (EXACT), a supervision-allocation objective that upweights rare long effective-context targets according to their inverse frequency in the long tail. Across seven Qwen and LLaMA continued-pretraining configurations, EXACT improves all 28 trained-context and extrapolated-context comparisons on NoLiMa and RULER. On Qwen2.5-0.5B, EXACT improves NoLiMa by +10.09 in the trained setting and +5.34 under extrapolation, and improves RULER by +10.69 and +5.55. On LLaMA-3.2-3B, RULER improves by +17.91 and +16.11. Standard QA and reasoning performance is preserved, with a +0.24 macro change across six benchmarks. A distance-resolved probe shows that the gains arise when evidence is thousands of tokens away, while short-context cases remain stable. The results support a supervision-centric view: long-context adaptation depends on how strongly training supervises long-context predictions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context language models, continued pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFWPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a mismatch between nominal training sequence length and token-level context exposure in long-context continued pretraining. Under document-boundary masking, many target tokens in a long packed sequence are still predicted from relatively short same-document prefixes. The authors propose EXACT, which increases the loss weights of tokens with long effective left contexts and allocates the additional weight according to inverse bucket frequency. The method leaves the training data and model architecture unchanged. Experiments across Qwen and LLaMA configurations show consistent improvements on long-context benchmarks, complemented by ablations and evidence-sensitivity probes.

Critical strengths

A meaningful problem and a promising perspective. The distinction between available context length and the allocation of token-level supervision provides a useful perspective on long-context training. The paper suggests that increasing the training window alone may be insufficient if relatively little supervision is assigned to targets with long visible contexts. This insight has the potential to inform practical training recipes.

A simple and practical method. EXACT requires only context-length statistics and per-token loss weighting. It introduces no architectural changes or additional inference cost, making it straightforward to incorporate into existing continued-pretraining pipelines.

Consistent empirical results with useful controls. The improvements across multiple model configurations are encouraging. Keeping the data stream and training setup matched strengthens the comparison with standard continued pretraining. The weighting ablations and evidence-sensitivity probes also provide useful initial support for the proposed explanation.

Critical weaknesses

The mechanism is not yet sufficiently established. The paper measures effective context through the length of the visible same-document prefix. However, a target with a long visible prefix does not necessarily require distant information for its prediction. Consequently, the results establish the usefulness of the weighting rule more clearly than they establish that it works by correcting insufficient supervision of long-range dependencies. The evidence-sensitivity probes are suggestive, but do not fully bridge this gap.

Alternative explanations remain insufficiently controlled. Although EXACT preserves the underlying data stream, it changes the relative contribution of different tokens to the training objective. This may implicitly emphasize longer documents, particular domains, or later portions of documents. Comparisons that control for document length and domain are needed to distinguish the proposed exposure-allocation mechanism from these effects.

The conceptual insight needs stronger theoretical or hypothesis-driven validation. The methodological novelty of context-length-based inverse-frequency weighting is relatively modest. This is not inherently a weakness for a simple and effective method, but it places greater importance on establishing a clear and well-supported explanation. In particular, the paper does not yet show whether the initial degree of exposure imbalance predicts the magnitude of EXACT’s gains. Such a relationship would turn the current empirical observation into a more informative and potentially predictive training principle.

Some experimental details need clarification. More complete specifications of the evidence-sensitivity probes, training configurations, and statistical variability would improve reproducibility and make the mechanistic claims easier to assess.

Questions for authors
  1. Controlling for document length and domain: Can the authors evaluate EXACT under matched or stratified document-length and domain distributions? Does the advantage persist when the relative training contribution of these factors is controlled?

  2. Testing actual dependence on distant context: For the same target tokens, can the authors truncate distant context while preserving the local prefix and measure the resulting changes in loss or gradients? Does EXACT preferentially improve predictions that demonstrably depend on distant information?

  3. Testing the predictive value of exposure imbalance: Across different corpora, packing policies, or training stages, does greater initial exposure imbalance predict larger gains from EXACT? Can the authors define an imbalance measure and test this relationship?

  4. Clarifying reproducibility and uncertainty: Can the authors provide the exact answer-margin definition, probe construction and sample sizes, complete training settings, and results across multiple training seeds for representative configurations?

Reviewer_ZaqeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EXACT, a loss-weighting objective for long-context continued pretraining, which changes only the per-token loss weight. The key insight is that under packed training with document-boundary masking each target token is predicted from the prefix of its own document only, therefore most of the training signal falls on tokens with a short effective context even when the training window is long. The paper develops a weighting rule to move part of the supervision toward tokens with a long effective context. Experiments on seven base models demonstrate that the weighted objective improves NoLiMa and RULER over unweighted continued pretraining at trained and longer evaluation lengths, preserves short-context benchmark scores, and increases the effect of distant evidence on the answer margin.

Critical strengths
  • The paper identifies a concrete gap in packed causal training. Most of the loss falls on short contexts even in a long window.

  • The proposed objective is simple to apply.

Critical weaknesses
  1. The experimental setting does not test long-context adaptation. All seven base models were pretrained with a 32K or 128K context, while the continued pretraining uses 4K, 8K and 16K windows. In this setting, the training can reduce its existing long-context ability. The paper reports no score for the base checkpoint before continued pretraining. The main claim therefore lacks support in the setting the claim describes.

  2. The claimed distinction between data supply and supervision allocation lacks experimental support. Raising the loss weight of a token has the same expected gradient as sampling that token more often, so a data mixture that upsamples long documents moves loss mass into the long-context tail in the same way. The paper does not compare with a data mixture that upsamples long documents, which is the standard recipe the paper itself cites.

  3. The reported gains are not shown to exceed training noise. The sensitivity sweep shows the trained RULER gain on Qwen2.5-0.5B falling from +10.69 to +4.20 when α moves from 0.15 to 0.20. Such noise is huge and need more analysis.

  4. The claim of length-selective gains is not supported by the length-resolved results. Figure 4 shows gains on llama-8B at 2K to 4K, which lie below its 4K threshold, and LLaMA-1B gains more at 2K to 4K than at 5K to 16K.

Questions for authors
  • Does the continued pretraining change the RoPE base or any other position-scaling setting of the base model?

  • Could the authors report RULER per-task scores?

Reviewer_XFBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper observes that in packed causal lm training with document-boundary masking, most target tokens have short same-document left context even when the window is long. it proposes EXACT, which bucketizes tokens by effective left-context length, adds a fixed average extra loss weight to buckets above a threshold, and distributes that weight within the tail by inverse bucket frequency. in paired continued-pretraining runs on seven qwen and llama models with a 500m-token budget per stage, EXACT improves nolima and ruler at trained and extrapolated lengths, leaves standard qa roughly unchanged, and an ablation and an evidence-distance probe indicate that the gain comes from the effective-context signal.

Critical strengths
  • the observation is simple and the method costs nothing: a bucket lookup and a scalar weight per token, with the data stream, model and budget unchanged.

  • the ablation in table 2 is well designed. random same-mass reweighting hurts, packed-position weighting helps little, and upweighting by effective context helps a lot, which isolates the variable the paper claims matters.

  • the paired-run protocol and bootstrap intervals (tables 7, 8) make the main comparisons clean, and the comparison to Helm et al. (2025) in table 6 is the right prior-work baseline.

Critical weaknesses
  • the named contribution, inverse-frequency allocation within the tail, adds about one point over a uniform tail boost (table 2: +4.38/+5.12 vs +5.73/+6.54 on nolima). most of the effect is “upweight long-context tokens,” which is close in spirit to the loss weighting in longalign (Bai et al., 2024a) and length upsampling (Fu et al., 2024). the obvious data-side baseline, upsampling long documents within the same token budget so the exposure histogram shifts without reweighting, is not run, so the paper’s claim that allocation is distinct from data supply is asserted rather than tested.

  • the method is sensitive to its hyperparameters. in table 9, raising α from 0.15 to 0.20 cuts the ruler gains from +10.69/+5.55 to +4.20/+1.60, and moving τ from 1k to 2k cuts the average gain from +7.92 to +2.55. the sweep is only on qwen2.5-0.5b, and τ is set by hand per stage.

  • gains shrink with scale and budget. the largest numbers are on near-zero baselines (qwen2.5-0.5b nolima 9.04 → 19.13), the 7b nolima trained gain has a ci touching zero (+0.91, [-0.02, +1.76], table 7), and the relative lifts in fig. 4 are inflated by these low baselines (qwen-1.5b jumps from +15% to +90% at 12k because the baseline is 3.88). each configuration is a single run; the bootstrap resamples evaluation examples, not training seeds.

Questions for authors
  • how does EXACT compare to upsampling long documents at the same 500m-token budget, which changes the exposure histogram directly?
  • can you report at least one configuration with multiple training seeds?
  • does the gain persist at larger token budgets, or is exact mainly compensating for a short cpt stage?
  • since α and τ are tuned on qwen2.5-0.5b and the sweep shows a nolima/ruler tradeoff, how were they chosen for the larger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.