- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a training-free page-level summary for sparse KV retrieval. Standard methods build each page's compact representation from keys only, use it to predict a page's attention mass for top-k ranking, and then run exact attention on the original tokens of the kept pages. The authors argue the selection objective should reflect the attention output, not just the logits, so they construct each page basis from keys and values via one tuning parameter that trades key fidelity against output sensitivity, computed cheaply as a small per-page eigenproblem. Values are used only when building the basis, not at decode, so the stored summary and its scoring cost match a key-only spectral method. Gains are reported on reasoning, long-context-understanding, and long-generation benchmarks.
Critical strengths
- The motivating observation is valid: preserving attention logits is genuinely not the same as preserving the attention output, and the worked example showing that the highest-variance key direction can be irrelevant to the output while a lower-variance one is critical is a clean way to make this precise.
- Results look strong as reported: OVAL achieves the highest avg@k among compressed methods in all long reasoning settings and the best LongBench v2, and to stay competitive on LongBench and InfiniteBench.
- Code is open-sourced via the anonymized repository, which supports reproducibility of the paper.
Critical weaknesses
- The gains look like oracle tuning, not a method property. No η is stable across datasets or models, yet every headline cell reports "η⋆ is the best evaluated mixing parameter for each setting," chosen on the test score itself. A per-setting sweep that keeps the winner is effectively parallel trials picking the best, and is not a fair comparison to baselines with no such knob.
- Latency is only reported on LongBench v2 and InfiniteBench, not on the long-reasoning benchmarks that carry the headline. Reasoning decodes far more tokens, so per-step scoring over more pages and the extra per-page summary build should cost more than the ~10% overhead shown. Report decode and build overhead on reasoning.
- By the paper's own numbers pass@k is below FreeKV on two of three models while avg@k is above. For reasoning, pass@k is often the metric that matters. Please explain the mechanism and state whether this is a real weakness or small-sample noise.
Questions for authors
I think Weakness 1 and Weakness 3 are linked: by sweeping over η (the five grid values, keeping the best per setting), you are running an implicit pass@5, which raises the average (avg@k). This is why I give you rejection. I'm not sure my understanding is correct, but if you can rebut this point, I would be willing to give the paper a higher score.