- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether fixed evidence compression preserves comparisons between readers in retrieval-augmented generation. It distinguishes improvements in overall pipeline accuracy from the retention of performance differences among readers. Across five QA benchmarks, five compression families, and panels of 8–20 readers, the study compares raw evidence with stored compressed text that is shared identically by every reader. It measures reader-upgrade retention, excess ordering reversals relative to raw-evidence replays, and answer-level rescue and damage. The results show that compression can improve average accuracy while substantially shrinking reader performance gaps and making reader comparisons harder to resolve. The paper releases a row-level interaction dataset and the ragscale toolkit for auditing reader comparisons under raw and fixed compressed evidence.
Critical strengths
-
This paper draws a practically important distinction: end‑to‑end compressed‑pipeline accuracy versus preservation of reader performance gaps. The warning that compression can lift average accuracy yet diminish true reader upgrades offers valuable guidance for RAG evaluation.
-
Fixed‑artifact replay provides strong experimental control. By storing compressed evidence and verifying that every reader receives identical text, the design prevents variation in compressor outputs from contaminating reader comparisons.
-
The statistical analysis anticipates important pitfalls. The authors account for shared sampling noise in raw‑score–gain correlations, examine endpoint‑selection sensitivity, and compare compressed ordering reversals with raw‑evidence replays. They further distinguish split‑sensitivity ranges from population uncertainty and disclose numerical issues in the errors‑in‑variables correction.
Critical weaknesses
-
Treating performance‑gap attenuation as “evaluation distortion” lacks sufficient justification. Raw‑evidence performance is not an inherent ground truth; compression alters task conditions and may legitimately reduce model advantages. The manuscript should better distinguish misleading measurement artifacts from valid condition‑driven performance shifts.
-
Experiments do not fully disentangle compression from evidence‑selection confounds. HotpotQA and MuSiQue have mismatched document access between raw and compressed conditions. Existing matched‑access controls only apply to a tiny subset of readers, not the full panel.
-
Endpoint‑pair metrics (retention, excess flips) have limited representativeness for full‑panel behaviour. Gap shrinkage and sampling noise alone can create apparent reversals without true ranking changes. Analyses covering all reader pairs are needed to solidify global claims.
Questions for authors
-
In what evaluation scenarios should raw‑evidence reader performance be treated as the reference benchmark that compression ought to preserve? How do you operationally distinguish misleading measurement artifacts from legitimate, condition‑driven performance changes when evidence inputs are altered via compression?
-
In HotpotQA RECOMP setup, compressor selects top‑5 passages while raw input uses all 10 passages. How much of the observed drop in upgrade retention can be attributed purely to filtering out passages, versus subsequent token‑level compression / rewriting? Can you decompose variance from passage selection and compression using your existing data or supplementary control analyses
-
How many of the reported excess rank‑flip events occur among reader pairs that already have tiny performance gaps under raw evidence? When filtering to only pairs with large, statistically significant raw‑evidence performance differences, does excess‑flip magnitude shrink substantially?
-
Your matched‑access control only covers three readers on MuSiQue. From this small subset, what evidence do you have that the same confound magnitude would persist across the full 15‑reader panel? What risk remains that document‑access mismatch is inflating your measured attenuation effect?