- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes FocuSFT, a bilevel optimization framework designed to alleviate training-time attention dilution in long-context supervised fine-tuning. The method introduces an inner loop that adapts lightweight LoRA parameters to form a context-specific parametric memory, while the outer loop updates the base model using the adapted representations. Both loops employ bidirectional attention over context tokens while preserving causal masking for response generation. Experiments on BABILong, RULER, LongBench, and GPQA demonstrate improvements over standard SFT, particularly on long-context reasoning and information aggregation tasks. The paper also provides ablation studies and attention analyses to investigate the contributions of bilevel optimization and bidirectional context attention.
Critical strengths
- The paper investigates training-time attention dilution, an interesting perspective on why long-context fine-tuning may fail to improve effective context utilization.
- FocuSFT combines bilevel optimization, fast-weight adaptation, and bidirectional context attention into a coherent training framework without introducing additional inference-time computation.
- The method achieves substantial gains on BABILong (up to +14.2 percentage points) and RULER CWE aggregation (+8.2 points at 16K), including improvements beyond the 4K training context length.
Critical weaknesses
- The inner loop uses the same gold responses as the outer loop, making it unclear how much of the improvement comes from generalizable context learning rather than label-conditioned adaptation.
- The reported attention-sink reduction may partly result from different attention masks rather than learned improvements under identical inference conditions.
- Stronger long-context fine-tuning baselines and training-compute-matched comparisons are missing.
Questions for authors
- Since the inner loop adapts to the gold responses of the same examples used in the outer loop, could the authors clarify how this promotes generalizable context utilization? Would the method remain effective if the inner loop used context-only self-supervision or separate adaptation examples?
- Could the authors compare attention-sink mass and attention to relevant evidence under the same causal attention mask at inference time, with the temporary LoRA adapters removed? This would help distinguish learned improvements from the direct effects of bidirectional masking.
- Could the authors provide comparisons with stronger long-context fine-tuning methods and standard SFT under matched training-compute budgets, given the reported 1.71× training overhead? such as LongLoRA、LongAlign mentioned in related work.
- Does FocuSFT retain its improvements on other model families or model sizes? Even one additional backbone would help assess whether the findings generalize beyond Qwen2.5-7B.
If the authors can address the above questions, I will consider raising the score.
Thank you for recognizing the training-time perspective, length generalization, and unchanged inference cost. We clarify how the supervised inner loop improves the deployed checkpoint and how the existing controls support this conclusion.
1. Gold-response supervision and generalization.
The inner loop uses the supervised task objective to make adaptation relevant to response generation. The resulting update is
Only θ receives the persistent AdamW update. The fast LoRA weights are reset for each training batch, excluded from the outer optimizer, and removed from the checkpoint. All headline evaluations use θ alone, standard causal attention, and no adaptation or test-answer access. Thus, the observed test gains transfer through the persistent checkpoint, rather than through sample-specific adapters at inference.
Generalization is measured on held-out tasks and longer contexts: training uses sequences up to 4K, yet BABILong improves at 8K, 16K, and 32K, alongside RULER, LongBench QA, and GPQA. The inner objective provides task alignment during training; the outer updates accumulate its benefit in θ. Context-only or separate-example adaptation would test a different objective and was not evaluated here. We will distinguish this observed performance transfer from the more specific interpretation of attention sharpening.
2. Attention measurements versus causal-inference performance.
The checkpoint-based attention extraction code runs no inner adaptation and applies each method's training mask: causal for Standard SFT and blockwise bidirectional context attention for FocuSFT. The 529× sink reduction and 3.1× context-engagement increase describe this training-mask setting. As specified in Appendix C, these are representative-sample diagnostics; aggregate statistics under a common causal mask are not reported. We will make the mask and measurement scope explicit in the captions.
The behavioral comparisons already use matched causal inference. The existing factorial ablation reports:
Causal Bilevel gains +12.2/+10.0/+9.2/+5.8 points with the same causal training mask as Standard SFT. FocuSFT adds +2.0/+0.2/+1.0/+3.8 points. Thus, the performance improvement transfers to ordinary causal deployment and does not depend on serving with a bidirectional mask. The factorial pattern supports the combined training design rather than sink reduction alone as a sufficient explanation.
3. Compute matching and stronger baselines.
The comparison holds the 3,000 examples, five epochs, batch size 32, and 469 persistent updates fixed. This controls data exposure while comparing the training rules. The measured wall-time ratios are 1.00/1.00/1.52/1.71 for the four rows above; we will use the precise wording "matched data exposure and persistent-update count." A longer-SFT run at equal wall time is a separate control not included in the current study.
LongLoRA emphasizes efficient context extension, while LongAlign studies long-context alignment data and training. Our setting keeps the training length at 4K and studies improvements in using, and generalizing beyond, that context. The factorial ablation addresses the components specific to this setting; direct LongLoRA/LongAlign comparisons are outside the reported experiments. The practical trade-off is explicit: 1.71 times training wall time, with deployment architecture and inference cost unchanged.
4. Other backbones and scope.
The experiments focus on Qwen2.5-7B. Within this backbone, the same training configuration transfers across synthetic reasoning, retrieval/aggregation, real-world QA, and agentic reasoning. In particular, BABILong improves from 33.2 to 42.8 at 32K, RULER CWE from 72.9 to 81.1 at 16K, and all four reported LongBench QA tasks improve. These results establish cross-task and length transfer; generalization to additional model families remains to be evaluated.
The central contribution is a training procedure that produces these gains in a standard causal checkpoint, with no adapters, extra gradient steps, or custom attention mask required at deployment.