- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method for fine-tuning language models with KV cache eviction active during training. It combines nested activation checkpointing, CPU offloading, and delta encoding of cache states to reduce GPU memory use. It also provides an optimized H2O implementation. Experiments on HELMET tasks compare this approach with full-attention fine-tuning followed by sparse-attention inference.
Critical strengths
-
The paper addresses a practical problem: fine-tuning a model under the same cache eviction conditions used at inference. The proposed training procedure makes this possible with limited GPU memory, which is useful for long-context adaptation.
-
The combination of nested checkpointing and delta encoding offers a way to avoid storing a full cache state for every chunk. The paper explains the implementation in detail and evaluates several cache policies at 64k and 128k context lengths.
Critical weaknesses
-
OOMB is closely related to this work, as it also combines chunk-wise training, activation recomputation, CPU offloading, and sparse attention. The paper discusses the differences, but does not provide a direct experimental comparison. Table 5 reports training time without corresponding peak GPU memory, CPU memory usage, or measurements across context lengths. This makes the practical benefits over existing memory-efficient training methods difficult to assess.
-
The main performance conclusion depends strongly on the evaluation metric. The main table replaces HELMET’s SubEM with MatchFirstWordOrPhrase, while Table 8 shows that the ranking reverses on several tasks under SubEM. For example, at 64k context with normalized H2O and a chunk size of 2048, the proposed method scores 50.0 on NQ and 55.3 on HotpotQA, compared with 58.8 and 68.0 for the SP baseline. The authors reasonably explain that irrelevant extra output can inflate SubEM scores. However, the current results do not clearly separate better answer accuracy from better answer formatting and stopping behavior. A comparison using a common answer extraction rule or controlled output format would help clarify how much the method improves the use of long-context information.
Questions for authors
- Can you provide a comparison with OOMB, or another closely related memory-efficient training method, under comparable settings? Training time, peak GPU memory, and CPU memory usage across context lengths would help clarify the practical benefits.
- Does the performance advantage remain with a common answer extraction rule or controlled output format? Reporting answer accuracy and stopping behavior separately would help explain the differences between Tables 1 and 8.