acceptodds
Under review as a conference paper at ICLR 2027

FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning

Abstract

Large language models can now process increasingly long inputs, yet their ability to effectively use information spread across long contexts remains limited. We trace this gap to how attention budget is spent during supervised fine-tuning (SFT) on long sequences: positional biases and learned attention sinks cause the model to allocate most of its attention to positionally privileged tokens rather than semantically relevant content. This training-time attention dilution (the starvation of content tokens in the attention distribution) weakens the gradient signal, limiting the model's ability to learn robust long-context capabilities. We introduce FocuSFT, a bilevel optimization framework that addresses this problem at training time. An inner loop adapts lightweight fast-weight parameters on the training context to form a parametric memory that concentrates attention on relevant content, and the outer loop performs SFT conditioned on this sharpened representation. Both loops apply bidirectional attention over context tokens while preserving causal masking for responses, reducing the causal asymmetry that gives rise to attention sinks and aligning inner-outer behavior. On BABILong, FocuSFT improves accuracy by up to +14pp across 4K–32K context lengths; on RULER, it raises CWE aggregation from 72.9% to 81.1% at 16K; and on GPQA with agentic tool use, it yields a 24% relative gain in pass@1. Attention analysis shows that FocuSFT reduces attention sink mass by 529 times and triples context engagement during training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context, Fine-Tuning, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WxacOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FocuSFT, a bilevel optimization framework designed to alleviate training-time attention dilution in long-context supervised fine-tuning. The method introduces an inner loop that adapts lightweight LoRA parameters to form a context-specific parametric memory, while the outer loop updates the base model using the adapted representations. Both loops employ bidirectional attention over context tokens while preserving causal masking for response generation. Experiments on BABILong, RULER, LongBench, and GPQA demonstrate improvements over standard SFT, particularly on long-context reasoning and information aggregation tasks. The paper also provides ablation studies and attention analyses to investigate the contributions of bilevel optimization and bidirectional context attention.

Critical strengths
  1. The paper investigates training-time attention dilution, an interesting perspective on why long-context fine-tuning may fail to improve effective context utilization.
  2. FocuSFT combines bilevel optimization, fast-weight adaptation, and bidirectional context attention into a coherent training framework without introducing additional inference-time computation.
  3. The method achieves substantial gains on BABILong (up to +14.2 percentage points) and RULER CWE aggregation (+8.2 points at 16K), including improvements beyond the 4K training context length.
Critical weaknesses
  1. The inner loop uses the same gold responses as the outer loop, making it unclear how much of the improvement comes from generalizable context learning rather than label-conditioned adaptation.
  2. The reported attention-sink reduction may partly result from different attention masks rather than learned improvements under identical inference conditions.
  3. Stronger long-context fine-tuning baselines and training-compute-matched comparisons are missing.
Questions for authors
  1. Since the inner loop adapts to the gold responses of the same examples used in the outer loop, could the authors clarify how this promotes generalizable context utilization? Would the method remain effective if the inner loop used context-only self-supervision or separate adaptation examples?
  2. Could the authors compare attention-sink mass and attention to relevant evidence under the same causal attention mask at inference time, with the temporary LoRA adapters removed? This would help distinguish learned improvements from the direct effects of bidirectional masking.
  3. Could the authors provide comparisons with stronger long-context fine-tuning methods and standard SFT under matched training-compute budgets, given the reported 1.71× training overhead? such as LongLoRA、LongAlign mentioned in related work.
  4. Does FocuSFT retain its improvements on other model families or model sizes? Even one additional backbone would help assess whether the findings generalize beyond Qwen2.5-7B.

If the authors can address the above questions, I will consider raising the score.

1 comment

Thank you for recognizing the training-time perspective, length generalization, and unchanged inference cost. We clarify how the supervised inner loop improves the deployed checkpoint and how the existing controls support this conclusion.

1. Gold-response supervision and generalization.

The inner loop uses the supervised task objective to make adaptation relevant to response generation. The resulting update is

phi_hat = stop_gradient(InnerSGD_K(theta, phi_0, x, y))
g_theta = grad_theta(response_CE(theta, phi_hat, x, y))

Only θ\theta receives the persistent AdamW update. The fast LoRA weights are reset for each training batch, excluded from the outer optimizer, and removed from the checkpoint. All headline evaluations use θ\theta alone, standard causal attention, and no adaptation or test-answer access. Thus, the observed test gains transfer through the persistent checkpoint, rather than through sample-specific adapters at inference.

Generalization is measured on held-out tasks and longer contexts: training uses sequences up to 4K, yet BABILong improves at 8K, 16K, and 32K, alongside RULER, LongBench QA, and GPQA. The inner objective provides task alignment during training; the outer updates accumulate its benefit in θ\theta. Context-only or separate-example adaptation would test a different objective and was not evaluated here. We will distinguish this observed performance transfer from the more specific interpretation of attention sharpening.

2. Attention measurements versus causal-inference performance.

The checkpoint-based attention extraction code runs no inner adaptation and applies each method's training mask: causal for Standard SFT and blockwise bidirectional context attention for FocuSFT. The 529×529\times sink reduction and 3.1×3.1\times context-engagement increase describe this training-mask setting. As specified in Appendix C, these are representative-sample diagnostics; aggregate statistics under a common causal mask are not reported. We will make the mask and measurement scope explicit in the captions.

The behavioral comparisons already use matched causal inference. The existing factorial ablation reports:

Training method4K8K16K32K
Standard SFT70.262.647.433.2
SFT+Bidir65.457.646.033.0
Causal Bilevel82.472.656.639.0
FocuSFT84.472.857.642.8

Causal Bilevel gains +12.2/+10.0/+9.2/+5.8 points with the same causal training mask as Standard SFT. FocuSFT adds +2.0/+0.2/+1.0/+3.8 points. Thus, the performance improvement transfers to ordinary causal deployment and does not depend on serving with a bidirectional mask. The factorial pattern supports the combined training design rather than sink reduction alone as a sufficient explanation.

3. Compute matching and stronger baselines.

The comparison holds the 3,000 examples, five epochs, batch size 32, and 469 persistent updates fixed. This controls data exposure while comparing the training rules. The measured wall-time ratios are 1.00/1.00/1.52/1.71 for the four rows above; we will use the precise wording "matched data exposure and persistent-update count." A longer-SFT run at equal wall time is a separate control not included in the current study.

LongLoRA emphasizes efficient context extension, while LongAlign studies long-context alignment data and training. Our setting keeps the training length at 4K and studies improvements in using, and generalizing beyond, that context. The factorial ablation addresses the components specific to this setting; direct LongLoRA/LongAlign comparisons are outside the reported experiments. The practical trade-off is explicit: 1.71 times training wall time, with deployment architecture and inference cost unchanged.

4. Other backbones and scope.

The experiments focus on Qwen2.5-7B. Within this backbone, the same training configuration transfers across synthetic reasoning, retrieval/aggregation, real-world QA, and agentic reasoning. In particular, BABILong improves from 33.2 to 42.8 at 32K, RULER CWE from 72.9 to 81.1 at 16K, and all four reported LongBench QA tasks improve. These results establish cross-task and length transfer; generalization to additional model families remains to be evaluated.

The central contribution is a training procedure that produces these gains in a standard causal checkpoint, with no adapters, extra gradient steps, or custom attention mask required at deployment.

Reviewer_sFZ3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FocuSFT, a training-time bilevel optimization framework intended to improve long-context utilization by mitigating attention dilution. An inner loop updates temporary LoRA adapters using response-token prediction loss, while an outer loop updates the base model conditioned on the adapted parameters. Both loops use bidirectional context attention and causal response attention. The adapters are discarded after training, and inference uses the base model with standard causal attention. Experiments on Qwen2.5-7B report improvements on BABILong, selected RULER tasks, LongBench QA, and agentic GPQA evaluation, supported by component ablations and training-time attention visualizations.

Critical strengths
  1. Substantial long-context improvements without additional inference cost. The reported BABILong gains are substantial: accuracy improves from 70.2% to 84.4% at 4K and from 33.2% to 42.8% at 32K, despite training exclusively on sequences of up to 4K tokens. Improvements on RULER aggregation and LongBench provide additional evidence of transfer beyond a single benchmark. Because the fast-weight adapters are discarded after training, these gains require no additional inference computation. This combination of length generalization and unchanged deployment cost is the strongest reason to consider acceptance.
Critical weaknesses
  1. The attention-mask definition leaves a consequential ambiguity about information flow in multi-turn training. Equation (5) permits all context tokens to attend to one another. If applied to an entire multi-turn sequence as written, an earlier context token could attend to a later tool response, allowing an earlier assistant prediction to access future information indirectly through subsequent transformer layers. Causal masking on response queries alone does not prevent this path. Figure 3 instead appears to depict blockwise bidirectional attention, leaving the actual implementation unclear. The authors should specify the turn-level masking rules and establish that earlier predictions cannot access later turns. This ambiguity prevents verification of the claimed causal information flow and makes the method difficult to reproduce faithfully.

  2. The evidence does not sufficiently establish the proposed mechanism or isolate its benefit from additional training compute. Appendix C states that the principal attention analyses use a single 4096-token sample, and the reported sink reduction is measured under training conditions. The paper does not establish that this behavior persists across held-out samples after removing the adapters and restoring causal attention at inference. Furthermore, FocuSFT requires 1.71× the training time of standard SFT, but no compute-matched SFT baseline is provided. Consequently, the results support the usefulness of the training recipe more strongly than the claim that alleviating attention dilution is responsible for its gains. Broader inference-time analysis and a compute-matched comparison would materially strengthen the evidence for the central contribution.

Questions for authors
  1. Does Equation (5) apply globally across all context turns, or is bidirectional attention restricted to individual context blocks? Please provide the exact masking rules or pseudocode and explain how they prevent information from later tool responses from reaching earlier assistant predictions indirectly across layers.

  2. Were the reported attention statistics measured with the adapted fast weights active? After removing the adapters and restoring standard causal attention, does the reduction in sink mass persist? This distinction is important for connecting the training-time visualizations to the deployed model’s behavior.

2 comments

Thank you for identifying where the multi-turn notation needs clarification and for recognizing the length-generalization results. The implementation uses blockwise bidirectional attention with causal ordering across blocks. This excludes the indirect future-context path raised in the review, as detailed below.

1. Exact multi-turn mask and cross-layer information flow.

Blocks are contiguous runs of the assistant-loss indicator (1 for assistant tokens, 0 for context). For a non-padding query and key, the exact rule is below. Allowed entries receive mask value 0; blocked entries receive negative infinity. Padding is blocked on both axes.

if key_block < query_block: allow
elif key_block > query_block: block
elif query_is_context: allow
else: allow only if key_position <= query_position

Context is bidirectional only within its own block. Assistant blocks are causal internally. Both types can see all earlier blocks, and neither can see a later block. This rule is used in both inner and outer forwards by build_attention_4d, which dispatches to build_glm_4d_mask_multiturn in the supplementary implementation. We will expand Eq. (5) with these block indices so that the mathematical definition directly matches the implementation and Figure 3.

For example, in C0,A0,C1,A1C_0,A_0,C_1,A_1, the later tool response C1C_1 is inaccessible to both C0C_0 and A0A_0. It cannot first enter C0C_0 and subsequently reach A0A_0. More generally, every attention dependency moves to the same or an earlier block. Inductively, hidden states in a block can depend only on that block and its predecessors at every layer. Within an assistant block, same-block dependencies are also position-causal. Tokenwise FFNs and normalization do not create cross-block paths. Thus, for fixed weights, stacking layers cannot create the future-context path raised in the review.

This guarantee concerns forward attention connectivity at fixed weights. The supervised fast-weight updates are confined to training: adapters are reset per batch and stripped from the saved checkpoint. All headline evaluations use only θ\theta with ordinary causal decoding and no inner adaptation.

2. What was active in the attention analysis?

The checkpoint-based extraction code loads saved checkpoints and runs no inner adaptation; adapted fast weights are not active in these forwards. It applies each method's training mask: causal for Standard SFT and blockwise bidirectional context attention for FocuSFT. The figures visualize attention under those conditions, with the representative-sample scope specified in Appendix C. We will state the checkpoint, mask, and sample scope together in the captions.

The 529×529\times ratio should therefore be read as a training-mask diagnostic. Aggregate causal-mask statistics are not reported. The direct evidence for transfer to the deployed model is the held-out performance measured after removing adapters and restoring causal attention.

The existing behavioral results already evaluate all methods under standard causal attention without adapters. BABILong accuracy at 4K/8K/16K/32K is:

Training method4K8K16K32K
Standard SFT70.262.647.433.2
SFT+Bidir65.457.646.033.0
Causal Bilevel82.472.656.639.0
FocuSFT84.472.857.642.8

Causal Bilevel improves by +12.2/+10.0/+9.2/+5.8 points while retaining causal masking throughout training and evaluation. FocuSFT then improves on Causal Bilevel at all four lengths. These results establish that the deployed gains do not require bidirectional inference or test-time fast weights, and support the combination of bilevel training and bidirectional context encoding. We will present the attention plots as diagnostics of the training design, alongside this direct behavioral evidence.

3. Training-compute control and scope.

The study holds 3K examples, five epochs, and 469 persistent updates fixed to compare the training rules under equal data exposure. The four rows cost 1.00/1.00/1.52/1.71 times Standard SFT; we will specify the budget as matched data exposure and persistent-update count. Equal-wall-time longer SFT is a separate comparison not included here. The measured trade-off is 1.71 times training wall time for gains of +14.2 points at 4K and +9.6 at 32K, with unchanged inference cost. Together with the explicit mask above, this clarifies both the implementation and the practical contribution of the training procedure.

Thank you to the authors for the detailed response and clarifications. The explanation of the blockwise attention mask addresses my concern about potential future information leakage. I also appreciate the clarification of the attention analysis and the training compute comparison. Although some concerns remain, particularly regarding the lack of a compute-matched SFT baseline, I find the responses helpful and am willing to raise my score to 3

Reviewer_Yj8POpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that long-context SFT suffers from training-time attention dilution, meaning positional bias and attention sinks absorb most attention mass, so gradients reflect a diluted view of the context. It proposes FocuSFT, a first-order bilevel scheme, and validates on a few datasets, showing an informative factorial ablation.

Critical strengths

A simple, low-cost training-time method with consistent gains and no inference cost. FocuSFT changes only the training procedure, i.e. an inner LoRA loop plus a mask, costs 1.71× training time, and leaves inference unchanged. It improves over standard SFT on every benchmark reported, and on an agentic benchmark. If the gains hold under stronger controls, the method would be easy to adopt.

Critical weaknesses
  1. The only trained baseline is standard SFT with the same number of outer steps, while FocuSFT does two extra inner gradient passes per sample on the same response tokens (1.71× compute). Standard SFT shows essentially no gain over the pretrained model (BABILong 70.2 vs about 70). The improvement could come from more effective optimization per sample, or a lookahead effect of computing the outer gradient. It need not come from sharper attention.
  2. The attention story is inconsistent. The 529× sink reduction is attributed to the bidirectional mask in Sec 4.4, yet the bidirectional mask alone lowers accuracy. So sink reduction by itself doesn't drive performance, which undercuts the dilution leads to poor learning premise.
Questions for authors
  1. How does FocuSFT compare with standard SFT at matched training compute, e.g. 1.7× epochs or steps, or a tuned learning rate, and with a lookahead optimizer that takes the same inner steps without the bilevel framing?
  2. What happens with an inner loop on context tokens only? Does the gain depend on fitting the response the outer loss then evaluates?
  3. Were the attention statistics (Figs 2–4c) computed under the bidirectional training mask or the causal inference mask? What are sink mass and context attention for the final θ\theta under causal inference, averaged over many samples rather than one?
  4. How does FocuSFT compare with Tandon et al. (2025) end-to-end test-time training, which also meta-learns an inner loop during training?
2 comments

Thank you for recognizing the practical value of the gains and unchanged inference cost. The existing factorial controls and the distinction between transient adaptation and persistent updates help clarify the contribution.

1. Extra optimization, compute matching, and lookahead.

Beyond the main Standard SFT comparison, the existing factorial ablation includes four trained configurations:

Training methodTime ratioBABILong 4K/8K/16K/32K
Standard SFT1.0070.2 / 62.6 / 47.4 / 33.2
SFT+Bidir1.0065.4 / 57.6 / 46.0 / 33.0
Causal Bilevel1.5282.4 / 72.6 / 56.6 / 39.0
FocuSFT1.7184.4 / 72.8 / 57.6 / 42.8

All use the same data exposure and 469 persistent updates, and all are evaluated with standard causal attention and no fast weights. Causal Bilevel improves by +12.2/+10.0/+9.2/+5.8 points without bidirectional masking. Adding bidirectional context attention then yields +2.0/+0.2/+1.0/+3.8 points. These controls locate the main benefit in the bilevel training component and show a beneficial interaction with the mask. The comparison matches data exposure and persistent-update count; longer-SFT, retuned-SFT, and matched-lookahead runs are not included. We will make the budget definition explicit.

The update makes the connection to lookahead precise:

phi_hat = stop_gradient(InnerSGD_K(theta, phi_0, x, y))
g_theta = grad_theta(response_CE(theta, phi_hat, x, y))

The inner steps update only transient low-rank FFN adapters; the outer AdamW step updates only θ\theta. The adapters are reset each batch, and no gradient is propagated through the inner trajectory. Thus, the extra inner passes change the point at which the persistent gradient is evaluated, rather than taking additional optimizer steps on θ\theta. This provides a precise optimization interpretation of the method. A lookahead-style interpretation is compatible with this update; the contribution is the particular transient-adapter SFT procedure and its measured transfer to adaptation-free inference.

2. Interpreting sink reduction and the factorial results.

The factorial result distinguishes changing attention access from learning effectively with that access. Bidirectional context attention alone does not improve accuracy, whereas its combination with bilevel training outperforms Causal Bilevel at every reported BABILong length. This is consistent with the design: the inner loop fits task-conditioned fast weights under the same mask used by the outer loop. Lower sink mass alone is therefore not the criterion of success; held-out causal-inference performance is. We will sharpen this distinction in the mechanism discussion. Standard SFT's near-flat BABILong result motivates studying the update rule in this configuration rather than implying a general failure of SFT.

The checkpoint-based attention extraction code runs no inner adaptation, using the causal mask for Standard SFT and the blockwise bidirectional training mask for FocuSFT. The 529×529\times reduction characterizes those training-mask conditions. Appendix C specifies the representative sample; many-sample common-causal-mask statistics are not reported. We will label this measurement setup explicitly. All accuracy comparisons above already use the same causal inference procedure, directly establishing the performance transfer to deployment.

3. Context-only inner supervision.

Response-token supervision makes the inner update task-aligned: it adapts representations toward the objective that the outer update must learn. Context-only adaptation would optimize a different objective and was not evaluated here. Since fast weights are reset every batch and removed from the checkpoint, the reported held-out gains are carried by θ\theta. Response-conditioned gradient shaping is a compatible interpretation of this training process; we will distinguish this broader optimization effect from the specific attention-based interpretation.

4. Relation to Tandon et al. (2025).

TTT-E2E meta-learns an initialization for continued next-token learning on the context at test time. FocuSFT uses supervised response-conditioned adapters during SFT to improve a checkpoint deployed without adaptation. Thus, both use an inner/outer structure, but optimize for different deployment procedures: context-dependent test-time learning versus a fixed causal checkpoint. We will make the training-time meta-learning connection explicit in related work. Our contribution is the transient-adapter SFT recipe and its adaptation-free downstream gains; a direct empirical comparison with TTT-E2E is not part of the reported study.

I thank the authors for the rebuttal. For q3, I think the answer confirms my concern in that the attention statistics are from just one sample and different masks for the 2 methods. Therefore they cannot show that the model has learned and sink reduction should not be presented as the mechanism. For q1 and q2, I believe they are still open. To answer them we need sft at the matched compute, a lookahead control and the inner loop trained on context tokens. Therefore, I am keeping my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.