- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how evidence position affects the evaluation of KV-cache compression methods. It introduces evidence retention profiles and evidence-position priors to explain why aggregate rankings can change when methods have crossing position-resolved accuracy curves. The authors evaluate this framework using controlled retrieval tasks, experiments constructed from natural documents, and original benchmark examples. They also audit evidence positions in existing benchmarks and recommend depth-resolved reporting and a retention probe.
Critical strengths
Overall, the paper identifies a useful evaluation concern and investigates it with substantial experimental effort. Details are as follows:
1.Relevant observation about aggregate rankings. The distinction between a method’s position-resolved performance and its benchmark-averaged score is useful.
2.Careful controlled comparisons. The within-item depth comparisons, materialized matched cache budgets, and separate query-aware and query-agnostic protocols help isolate the factors under study. The free-generation checks also provide evidence that the observed reversals are not solely artifacts of teacher-forced scoring.
3.Transparent reporting of limitations. The paper reports unsuccessful transfer to benchmark rankings, failures of the retention model, and the concentration of reversals at boundary positions. These negative results make the scope of the findings more assessable.
Critical weaknesses
Overall, my main concern is that the paper establishes the possibility of position-dependent rankings more convincingly than it establishes their practical importance or the added value of the proposed framework.
-
Limited evidence of consequential ranking instability on existing benchmarks. Section 5 reports no certified reversals among the published methods on the 996 original RULER QA examples or the 593 located LongBench examples under their respective benchmark metrics. On the controlled natural-document experiment, query-aware compression certifies only two published pairs, both at the largest budget; the stronger query-agnostic result certifies seven pairs there. Moreover, reweighting the synthetic grid by measured task priors does not reliably improve prediction of the actual task rankings. I agree that the unpaired benchmark data have confounding and power limitations, so these findings do not disprove the phenomenon. However, they leave its practical impact on existing benchmark comparisons insufficiently established. The paper supports a caution about interpreting aggregate scores, but provides weaker evidence that this caution materially changes conclusions in realistic evaluations.
-
Much of the query-aware effect is concentrated in predictable window-boundary behavior. Appendix X.8 reports certified reversals in 26.1% of observations involving a window-template method, compared with 1.5% for content-versus-content pairs under query-aware compression. Most template certifications involve the final depth, and no certified content-versus-content reversal survives removing both endpoints. This is consistent with the proposed mechanism, but much of the strongest evidence reflects the expected fact that a recent-window policy retains tail evidence and discards interior evidence. The broader query-agnostic results are valuable, but the paper does not sufficiently establish how often the phenomenon changes a practically relevant choice between competitive methods. Consistently, the audited task priors rarely change the query-aware winner.
-
The formal framework adds limited explanatory power beyond its assumptions. Proposition 1 follows directly from expressing benchmark performance as a weighted average, while Proposition 2 obtains the correspondence between retention and accuracy rankings by assuming a shared monotone availability function and shared answering regimes. These statements organize the observation, but do not themselves establish that the assumptions describe real compressed models. Section 4.4 demonstrates a failure of the shared-floor assumption, and Appendix X.6 shows that the query-aware predictive advantage over depth-averaged retention is driven by window templates, with press-level intervals including zero in all four harder-task cells. The paper should more clearly separate mathematical bookkeeping, conditional predictions, and empirically established mechanisms. The current density of definitions and propositions makes the conceptual contribution appear broader than the demonstrated explanatory gain.
-
The practical advantage of the proposed diagnostic remains unclear. Depth-resolved reporting is sensible, but the retention probe’s additional value over a small direct evaluation is less convincing. Appendix X.11 reports that one labeled item evaluated across methods, budgets, and depths predicts the target accuracy surface better than the retention link, although that comparison is partly in-sample and requires more scoring. The probe also requires known evidence positions, while its accuracy link is fitted using scored data from other methods. Given the unsuccessful transfer to natural-task rankings, it remains unclear when the probe provides a sufficiently reliable and economical basis for a concrete evaluation or deployment decision.
Questions for authors
-
Practical significance: Given the absence of certified reversals on the original RULER and LongBench examples, what is the strongest supported conclusion about existing benchmark rankings? Using the current results, can the authors identify a practically meaningful method-selection conclusion that changes after accounting for evidence position, distinguishing actual benchmark measurements from reweighted synthetic results?
-
Scope beyond boundary effects: Could the authors consolidate the existing endpoint-removal and method-family analyses into a main-text summary, separately for the two protocols? In particular, what practically meaningful reversals remain between competitive content-driven methods, and how much probability mass do the audited benchmark distributions assign to the depths responsible for them?
-
Added value of the formalization: Which result provides explanatory or predictive content beyond the weighted-average identity and the known retention pattern of window policies? How should readers determine whether the shared-link assumptions are adequate in a new setting, especially when retained representations can carry information from evicted evidence?
-
Use case for the probe: Under what concrete conditions should a practitioner use the retention probe instead of directly evaluating a small labeled set? Could the authors clarify the required annotations, fitting data, computational costs, and reliability of the resulting decisions, taking the negative results in Appendices X.3 and X.11 into account?
Dear Authors, Thank you for the detailed response and the additional Lost-in-the-Middle experiments. I appreciate the clear characterization of the positional effect and the transparent reporting of both positive and negative results.
The new Lost-in-the-Middle results demonstrate significant winner switches between extreme first and last placements on paired real QA data, whereas the original unpaired RULER QA and LongBench rows certify zero reversals. However, I remain concerned about the query-aware setting, where significant winner switches no longer hold under soft placements (0/4 budgets). Specifically, I still have two concerns (given the limited remaining discussion time, no additional experiments are needed, and a brief discussion based on your existing findings is sufficient):
Practical effect size and possible mechanisms: Under query-aware compression, does the lack of significant switches under soft placements (and unpaired RULER/LongBench) reflect insufficient statistical power, or does it suggest that the practical impact on rankings becomes limited away from extreme endpoints? Could you also briefly discuss the possible mechanisms explaining why this occurs in the query-aware setting while query-agnostic compression still shows significant switches under soft placements?
Concrete guidance for practical evaluation: How should practitioners and benchmark designers apply your framework in practice? Specifically, should future evaluation sets adopt paired multi-slot layouts, and how should average accuracy be combined with worst-depth metrics ( Wm ) when making concrete selection decisions?
If you can clearly address these two points, I will be happy to re-evaluate my score.