acceptodds
Under review as a conference paper at ICLR 2027

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

Abstract

A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention. It works for any KV cache policy, runs on a moderate hardware budget (e.g., a single Nvidia A100 GPU with 40 GB RAM), and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism). We also provide an efficient implementation of H2O sparse attention (the leading policy in our experiments) with dedicated scaled dot product attention kernel support. A new open source library for long-context inference and fine-tuning will be released with the publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long context inference, post training, sparse attention, key-value caching, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9MrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for fine-tuning language models with KV cache eviction active during training. It combines nested activation checkpointing, CPU offloading, and delta encoding of cache states to reduce GPU memory use. It also provides an optimized H2O implementation. Experiments on HELMET tasks compare this approach with full-attention fine-tuning followed by sparse-attention inference.

Critical strengths
  1. The paper addresses a practical problem: fine-tuning a model under the same cache eviction conditions used at inference. The proposed training procedure makes this possible with limited GPU memory, which is useful for long-context adaptation.

  2. The combination of nested checkpointing and delta encoding offers a way to avoid storing a full cache state for every chunk. The paper explains the implementation in detail and evaluates several cache policies at 64k and 128k context lengths.

Critical weaknesses
  1. OOMB is closely related to this work, as it also combines chunk-wise training, activation recomputation, CPU offloading, and sparse attention. The paper discusses the differences, but does not provide a direct experimental comparison. Table 5 reports training time without corresponding peak GPU memory, CPU memory usage, or measurements across context lengths. This makes the practical benefits over existing memory-efficient training methods difficult to assess.

  2. The main performance conclusion depends strongly on the evaluation metric. The main table replaces HELMET’s SubEM with MatchFirstWordOrPhrase, while Table 8 shows that the ranking reverses on several tasks under SubEM. For example, at 64k context with normalized H2O and a chunk size of 2048, the proposed method scores 50.0 on NQ and 55.3 on HotpotQA, compared with 58.8 and 68.0 for the SP baseline. The authors reasonably explain that irrelevant extra output can inflate SubEM scores. However, the current results do not clearly separate better answer accuracy from better answer formatting and stopping behavior. A comparison using a common answer extraction rule or controlled output format would help clarify how much the method improves the use of long-context information.

Questions for authors
  1. Can you provide a comparison with OOMB, or another closely related memory-efficient training method, under comparable settings? Training time, peak GPU memory, and CPU memory usage across context lengths would help clarify the practical benefits.
  2. Does the performance advantage remain with a common answer extraction rule or controlled output format? Reporting answer accuracy and stopping behavior separately would help explain the differences between Tables 1 and 8.
Reviewer_AiqDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses how to fine-tune transformer LMs with a sparse-attention KV-cache eviction policy already in place, so that the model co-adapts with the policy used at inference, while keeping GPU memory comparable to that of sparse-attention inference. Its main contributions are: (i) a policy-agnostic training scheme that records eviction decisions during the forward pass and replays them, without differentiating through the policy; (ii) nested activation checkpointing over layers and context cells with CPU offloading, combined with an autograd mechanism that exploits the linear recurrence of scatter/gather updates between consecutive KV buffers to store only deltas instead of full buffers, reducing per-cell saved KV-buffer memory in the autograd graph by a factor of k; and (iii) a faster H2O implementation that computes the required summed attention weights alongside a FlashInfer kernel, together with two H2O variants (selecting cache slots independently for each sequence in the batch, and a normalized score). The authors validate the method by LoRA fine-tuning Qwen3-4B-Instruct on 12 HELMET tasks at 64k/128k context (N_C = 32k; S ∈ {1k, 2k} for the QA tasks) under up to five cache policies. Training with the policy in place ("us") outperforms sequence-parallel training followed by sparse inference ("sp") on most tasks under MatchFirstWordOrPhrase/Accuracy, which the authors attribute mainly to "sp" producing over-long, unterminated outputs under sparse inference.

Critical strengths

1. A policy-agnostic, memory-bounded fine-tuning procedure. By replaying logged eviction decisions, nesting activation checkpointing over layers and context cells, and storing only scatter deltas of KV buffers in the autograd graph, the method backpropagates through very long sequences under, in principle, any KV selection or compression policy (demonstrated with lastrec and H2O variants). GPU memory per autograd call is on the order of one KV-cache buffer, independent of context length, while a training step is only about 30% more expensive than sequence parallelism at 128k with S = 2048 (Table 5). This is consequential because it removes a major practical barrier to training under the same memory constraints used at inference, without requiring policy-specific CUDA kernels (unlike OOMB, as discussed by the authors).

2. A clear, deployment-relevant train/inference mismatch. Under sparse inference, SP-trained checkpoints fail characteristically: outputs are often tens of times longer than targets and frequently hit the 128-token cap (p128 often close to 100%), much like the untuned base checkpoint (Table 6). Under exact inference, the same checkpoints are concise (R ≈ 1, p128 ≈ 0; Table 7). This shows that exact-attention fine-tuning does not prepare a model for KV eviction at inference. On json_kv, where the lenient SubEM metric does not penalize long outputs, "us" still clearly outperforms "sp" (48.0 vs. 13.0 for slr1k; 52.0 vs. 1.0 for h2o_norm1k), suggesting the benefit is not only length control, although both fail with h2o_orig.

Critical weaknesses

1. The evaluation does not experimentally disentangle improved long-context information use from recovered answer format and stopping behavior, and part of the QA comparison is metric-sensitive. The paper acknowledges qualitatively in Section 4, and quantifies in Tables 6/7 (Appendix), that under sparse inference SP-trained checkpoints produce over-long outputs (often tens of times the target length, with p128 often close to 100%), while the same checkpoints are concise under exact inference. However, it does not quantify how much of the gain comes from this recovered behavior versus better use of long-context information. In addition, for the four QA datasets, MatchFirstWordOrPhrase replaces Helmet's default SubEM and, per Section A.8.4, was adapted after observing "sp" output behavior. This does not by itself invalidate the metric, but under SubEM, "sp" outperforms "us" on 64k nq, trivia_qa, and hotpot_qa (Table 8). Because SubEM can be inflated by outputs that list multiple candidate answers, this reversal is not evidence that "sp" retrieves better; it does show, however, that the comparison depends on the metric. This dependence appears only in the appendix, while the main text states that "us outperforms sp across the board, with the possible exception of pop qa." Recovering stable answering under sparse inference is itself a practical improvement, and json_kv (48.0 vs. 13.0 for slr1k under SubEM) suggests that the gain is not only length control. Still, the main text should qualify the metric-dependent superiority claim and distinguish improvements in output behavior from improvements in long-context information use, e.g., via additional diagnostic evaluations that apply the same answer-extraction rules to both models, fixed independently of test outputs and reference answers, and that report correctness separately from stopping and format compliance.

2. The experiments do not isolate the contribution of cross-cell gradient propagation for the reported gains under the tested settings. The results support training under the inference-time cache policy, but they do not show that the cross-cell gradient propagation preserved by the proposed memory-efficient method is needed for these gains. The most direct ablation starts from the same initial checkpoint, with the same data, training budget, full input, and sparse forward policy, but detaches the KV-cache state at cell boundaries (stop-gradient across cells). Separately, as a cheaper practical alternative rather than a substitute for this ablation, briefly adapting the SP checkpoint under eviction would show whether a lower-cost fix exists. If such variants close much of the gap, the attribution of the gains, and the case for full cross-cell gradients, would need to be restated in terms of accuracy, training cost, and the conditions under which the full method pays off. This does not negate the systems contribution of enabling memory-bounded gradient computation, but the source of the improvement remains insufficiently attributed.

Questions for authors

Clarifications that could change my assessment:

  1. Evaluation robustness and metric selection. Could you report "us" vs. "sp" on the five many-shot ICL classification tasks and the four QA datasets using identical answer-extraction rules that are fixed in advance on development data and do not depend on reference answers (references are used only for scoring)? For example, use the first label in the output that belongs to the label set given by the in-context demonstrations for classification, and a fixed extraction rule for QA. Please report answer correctness separately from stopping and format compliance. This would help determine how much of the advantage remains when scoring is less sensitive to extraneous continuation. Results on representative classification and QA tasks would be sufficient for the rebuttal, with the full evaluation included in a revision. Also, was the decision in Section A.8.4 to adapt SubEM into MatchFirstWordOrPhrase based on development or test outputs?
  2. Contribution of cross-cell gradients. How does full "us" compare with a variant that detaches the KV-cache state at cell boundaries, keeping the full input, cache policy, and forward algorithm unchanged and modifying only gradient propagation (eviction decisions may then differ as the parameters diverge)? Please match initialization, data, and training tokens or update counts, and report actual compute costs. Please also clarify where the loss is applied. If supervision is limited to the final answer, detachment blocks gradient paths through the KV-cache state across the selected boundaries; comparing more than one truncation span would help characterize the accuracy–cost tradeoff.
  3. Comparability of the training pipelines. Were "sp" and "us" matched in training data, optimization settings, training tokens or update counts, decoding settings, and validation-based checkpoint selection? Please identify any relevant differences, including whether the same 8-bit KV-cache quantization was used when evaluating "sp" under sparse inference. With these controls clarified, how should the large gap on clinc150 between sparse-inference "us" and exact-inference SP (97.4 vs. 69.2) be interpreted, given that the latter already produces concise outputs (Table 7)?

Clarifications on the systems contribution:

  1. Gradient correctness. Have you compared your implementation against standard autograd on tractable cases (e.g., a smaller model or cache) that trigger eviction and span multiple cells, using identical replayed eviction decisions? Since the method reduces to standard training when N ≤ N_C, such cases are needed to validate the checkpointing and delta-reconstruction mechanism. Relative gradient errors with quantization disabled, and separately the error introduced by quantizing the checkpointed cache buffers (Section A.7.2), would be informative. Given that Section A.7.3 relies on tensor fingerprints for matching, how often do unmatched events occur, and what checks detect incorrect matches or reconstruction errors?
  2. Measured resource requirements. Under otherwise fixed configurations (cache length, chunk size, cell setting, and per-GPU batch size), could you report peak GPU memory, peak CPU memory, CPU–GPU transfer volume, and disk checkpoint storage (if used) at 64k and 128k, together with per-GPU training throughput (tokens/s) and the corresponding batch size? This would substantiate the stated single-A100-40GB requirement and clarify which costs remain bounded and which grow with sequence length.

Optional extensions (secondary to the questions above): broader validation across models, sizes, and cache policies beyond lastrec and H2O variants; uncertainty estimates on the smaller test sets; and direct latency benchmarks for the H2O kernel.

Reviewer_NwR2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper enables fine-tuning with an eviction-based KV cache through nested activation checkpointing, CPU offloading, replayed cache decisions, and delta-encoded cache states. It also implements efficient H2O variants. Experiments investigate whether training with the inference-time cache policy improves long-context performance relative to exact-attention training followed by sparse inference.

Critical strengths
  1. A meaningful systems contribution. Exploiting overwritten cache entries to avoid storing full cache states at every chunk directly addresses a difficult memory bottleneck. Combining this with two-dimensional checkpointing makes policy-matched long-context training practically interesting (Section 3.2).
  2. Broad evidence within the chosen setting. Multiple cache policies, chunk sizes, and tasks consistently show that matching training to sparse inference can restore useful output behavior. The appendix also transparently reports timings, failure examples, and an alternative metric.
Critical weaknesses
  1. Systems guarantees are insufficiently validated. The paper lacks peak GPU/host-memory scaling measurements and direct gradient checks against ordinary autograd. These matter because cache checkpoints are quantized and tensor matching uses fingerprints (Appendices A.7.2–A.7.3). The overwrite recurrence also does not by itself establish support for arbitrary compression policies.
  2. Accuracy gains are strongly metric-dependent. Appendix A.8.4 shows that exact-trained models often win under the original SubEM metric. The critique of SubEM is reasonable, but the discrepancy leaves better stopping/formatting entangled with better evidence retention. Task-specific tuning on one 4B backbone further limits generalization.
Questions for authors
  1. Implementation clarification: Can you provide gradient-error checks and GPU/CPU-memory scaling, with quantization enabled and disabled, and specify the cache-update classes supported by the recurrence?
  2. Evaluation clarification: How much advantage remains under answer extraction or constrained decoding that controls output-format failures? Does a jointly trained model retain the benefit on held-out tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.