- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In long chain-of-thought, eviction fails because deleted tokens get needed again and the model loops until it runs out of budget. The evicted history is low-rank, so we can archive it instead of deleting it. The authors propose LEAP to score keys by expected future attention on cached post-RoPE keys and stores each evicted segment as rank-64/32 fp8 factors. Across four models and four benchmarks LARK reaches near-lossless accuracy with 5.0% of the dense device KV memory, with 2.0-3.3x iso-batch throughput.
Critical strengths
- Clear idea with large accuracy gains at tight budgets across 4 models × 4 benchmarks.
- The failure analysis is convincing that eviction baselines fail by never finishing, not by reasoning badly.
- The authors perform ablations on each component, rank, and k.
Critical weaknesses
NA
Questions for authors
Authors should report Confidence intervals for AIME and GPQA
Follow-up on Matched-Precision Results: Two Days Remaining
Dear Reviewer icav,
Thank you for engaging with our rebuttal and for your follow-up question on matched precision.
With only two days remaining in the discussion period, we would appreciate your feedback on our November 16 clarification. The FP8 rows compare LARK, BeaconKV, and SnapKV with all three using FP8 KV caches at the same device-memory budget; we corrected the table header to make this explicit. LARK achieves higher accuracy in all four reported FP8 benchmark–capacity comparisons.
Together with the added ShadowKV and xKV evaluations, these results support the practical value of retaining compressed history during long decoding beyond the memory savings from quantization alone.
We would be grateful to know whether this clarification addresses your remaining concern. We welcome further discussion so that the additional evidence and LARK’s contribution can be fully reflected in the final evaluation.
Wish you a pleasant day ahead.
Authors of LARK