acceptodds
Under review as a conference paper at ICLR 2027

LongFocus: Focusing on Key Tokens for Efficient Long-Context Mid-Training

Abstract

Despite extensive progress in data selection for long-context mid-training, the objective typically remains a uniform token-averaged cross-entropy. Since only a small fraction of tokens substantially benefit from distant context, uniform averaging can dilute long-range supervision even within dependency-rich documents. Existing token-reweighting approaches address this mismatch, but their fine-grained token dependency scoring method introduces substantial training overhead. We therefore investigate whether a coarse context contrast can suffice when coupled with appropriate preprocessing. We propose **LongFocus**, an online objective that combines an inexpensive non-overlapping segmented reference with eligibility filtering and bounded rank-percentile upweighting for long-context mid-training. Specifically,LongFocus computes an inexpensive online dependency score, *SegRatio*, by measuring each token's relative loss increase from the full-context view to a coarse segmented view with no cross-segment attention. *PreSelect* restricts additional weighting by document length and full-view loss, while *Reweight* assigns bounded weights according to rank percentiles within the eligible set. We evaluate LongFocus on Llama-3-8B-Instruct, MoE-16B-A3B, and MoE-125B-A10B using a two-stage 64K-to-512K mid-training recipe, with evaluations before and after SFT. LongFocus improves Long-Avg over Vanilla CE in all matched model-stage comparisons. On MoE-125B-A10B, the post-SFT gains are 7.28 points (13.80% relative improvement) after 64K mid-training and 5.48 points (10.24% relative improvement) after the subsequent 512K stage. On Llama-3-8B-Instruct, it outperforms LongCE in Long-Avg at both post-SFT endpoints, with gains of up to 2.0 points (3.29% relative improvement), while reducing per-step training time and peak memory usage by up to 20.2% and 31.0%, respectively. Code is available at https://anonymous.4open.science/r/LongFocus-vsfnhyta.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context Language Modeling, Mid-training, Token Reweighting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WM62OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a token-weighting objective for long-context mid-training. The high-level idea is that most tokens can be predicted from nearby context; so identifying those small amount of tokens which really benefit from long context can provide better training signal (also speedup training and consume less GPU memory). The authors propose a cheaper solution, comparing to previous work such as LongCE, consisting of three steps:

  1. segment input into non-overlapping segments and calcualte each token's SegRatio score by comparing the loss when full context is available against only local context is available; 2) filter out those easier tokens whose full-context loss is too small; 3) re-weight kept tokens not by their raw score but based on their rank percentile.

The experiments were conducted on three model families (dense and MoE models with up to 125B total parameters) and using training contexts up to 512K. Results show than the proposed method is better than standard cross-entropy loss (treating all tokens equally) and LongCE.

Critical strengths
  • Strengths
    • the proposed method makes sense and is lighter (evidenced by faster training and lower GPU memory usage) than existing token weighting methods
    • The authors tested three base architectures, convering both dense and mixture-of-experts models; however, the details of MOE are not provided
    • The proposed method shows better performance than standard cross-entropy loss and other token re-weigthing technique (LongCE).
    • the authors open-source their implementation
Critical weaknesses
  • Weakness
    • The full comparison (against Vanilla CE and LongCE) is only done on dense model; but not on MOE (authors mention LongCE implementation runs out of memory)
    • Although the proposed method sounds interesting, it introduces a large number of hyperparameters, and the current ablation study is not very informative.
    • I think the presentation could be further improved by using simpler language (rather than introducing new terms without explanation), summarizing key findings (instead of listing results), reducing the bold formatting, and making tables more compact.
Questions for authors
  • Minor comments:
    • Line 51: dpendencies (Wu, Deng, Chen).
Reviewer_PaQEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of reweighing token loss in long range training contexts to improve downstream inference performance in such settings (e.g., RULER/HELMET). The authors propose a cheap heuristic that check the surprise of a model relative to a token with the full context and compares that to using on a small local context. When the difference between these two is large, it signals that the model should weight the loss for such tokens more heavily. The paper trains on 3 models (1 dense and 2 moe) and multiple training stages. Most of the comparisons are against vanilla CE loss.

Critical strengths

The paper focuses on an important problem of improving training signal for long-range dependencies. The method proposed appears computationally attractive relative to prior methods (LongCE) due to the choice of non-overlapping windows. The authors report reductions in training time and peak memory while retaining similar or slightly improved performance.

Critical weaknesses

Most of the comparison results in the paper are against vanilla CE loss. Table 1 llama-3-8b-instruct also compares against LongCE, but that is the only model that does. Generally, having a single comparison against prior baselines is not enough to determine the robustness of the method. Additionally, the authors bring up the Helm (token weighting) paper. This paper has several related training recipe that would also be useful to compare against.

My main concern is trying to unpack precisely why this method seems to outperform LongCE, considering there are many hyperparameters for such methods. While the authors provide some ablations, these are in relation to Vanilla CE, not LongCE. Both methods share a lot of similar components (short-context length, token weighting maximums, etc..).

Questions for authors

What are the MoE models used here? It is a bit strange to label llama 3 concretely and then give give just sizing but not the specific models for the MoE settings.

Do you have any explanation for why LongCE appears to do better on some tasks before SFT relative to LongFocus and the worse afterwards?

In the same vein, how reliable are the results over LongCE? Uncertainty estimates or multiple training runs comparing the two methods on the dense llama model would help make this more clear.

How were the hyperparameters for LongCE selected? Do the two methods have similar tuning budgets?

Reviewer_2F1EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examine the training of current models, long context capabilities by examining how to involve long context during mid training with respect to a two-staged coarse-to-fine segmentation-based method and cheap intermediate evaluation metric, introduced as SegRatio. Based on SegRatio, the authors investigate a two-stage intervention utilising the long-context utility signal: “Preselect” followed by “Reweight”. These lead to strong gains on Llama 3 8B, on standard long-context training baselines.

Critical strengths

The idea of proxying the utility evaluation and dependency of long-context with a windowed gain from a segment is a good idea, and builds nicely on the prior work from LongCE.

Rebuttal Edit: Based on the quality of the rebuttal, I have raised my score.

Critical weaknesses
  • The paper needs to be self-contained. Relegating key experimental details to the appendix is not an acceptable means of going over the page limit.
  • Figure text is generally unreadable. The authors need to work on the alignment of their Figures and their accompanying captions. Much of the figures have details that are not explained, the aspect ratio is off; consider using more of the margin by re-arrangement to present information more legibly. For example, Fig 1. The caption is not helpful, as the details in the figure are not explained, and the example is unreadable. If it is meant to be unreadable, make it clear; as the “reweight” chart’s legend is similarly unreadable, and clearly meant to be able to read. This is similar for the three subfigures in Fig 2. Fig 5 has much detail that is unexplained.
  • The introduction discusses terms that are not well-defined yet (what is a segment, what is meant by coarse), so it is difficult for this reader to really understand the contribution. Please introduce and gradually refine your exposition such that the reader can understand your work conceptually before going into details.
  • Although it may be difficult for the authors to show these results on additional models, testing a single backbone of Llama 3 8B is insufficient. It would be helpful to see gains across other open-weight model families (Qwen 3 would be the choice to work with, or at least explain why this was not possible/tried). The comparison on two other unattributed MoE models
  • Main results mostly call out specific table entry comparisons but there is little to indicate specific, in-depth understanding of what is happening. The text mostly reiterates the findings in the table without generalising or showing through any micro and mesh analyses any form of triangulation that would show why the results are as they are. (Most of Page 8 reads this way, so I didn’t learn anything as a result). As such, while the results are good, it is difficult to discern whether the claimed interventions are actually the contributing factor.
    • There are results in the main table that do not show your LongFocus as a stronger contributor, but these are not highlighted and explained away in the prose.
    • The strongest results are with respect to the large MoE architecture, but why this is isn’t explained.
    • The strong result for HELM-Cite for Llama 3-8B Instruct is also not examined nor differentiated from the other datasets and results.

Line specific critiques

  • L181, 234 A key statement is that you use rank rather than score, for calibration. This assumption should be tested against and discussed to justify your design decision.
  • L203 Does the \gamma exponent have to be larger than 1 or 0? Is there an upper bound for \gamma?
  • L459 I think your statement (“excessive long local …”) here is overly conclusive. It may be indicative, but you don’t have additional in-depth analyses that can show this is a conclusive cause.
Questions for authors
  • L196 How often does this constraint on the budget occur? How many samples does this affect (i.e., how important is this caveat to the overall selection)?
  • Table 3 - it would be helpful to know the confidence intervals here and how many runs the results in the table are based on. While I appreciate the synthetic \Delta_{CE} column it would be helpful to know whether the results are significantly different (and from the default LongFocus, not the Vanilla CE baseline; this is an ablation study, not a general performance comparison).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.