acceptodds
Under review as a conference paper at ICLR 2027

Compression Is Not Evaluation-Neutral: Fixed RAG Compression Can Distort Reader Comparisons

Abstract

Retrieval-augmented generation (RAG) systems often compare readers after a compressor has changed their evidence. This mixes two questions: which complete pipeline works best, and how much of a reader upgrade survives compression. We show that fixed compression can raise average pipeline accuracy while hiding most of a reader upgrade. We hold questions, retrieved candidates, prompts, scoring, and compressed text fixed while comparing 8–20 readers across five question-answering benchmarks and five compression families. HotpotQA and MuSiQue are the main benchmarks, analyzed under a plan fixed in advance. In the 20-reader HotpotQA panel, the lowest- and highest-scoring readers under raw evidence are 31.8 percentage points (pp) apart before compression but only 7.8pp apart on the same stored output from a HotpotQA-trained RECOMP compressor. On both main benchmarks, lower raw-scoring readers gain more under exact match and token-overlap F1. Reader pairs also change order more often under compressed evidence than in raw replays across question halves. Row-level accounting explains how higher average accuracy and smaller upgrades coexist: compression rescues some wrong answers and damages some correct answers, with a different balance for each reader. We release ragscale to audit declared reader upgrades under raw and fixed compressed evidence. Reader evaluations should run this audit before attributing a compressed-only result to the reader.

Primary area:
datasets and benchmarks
Keywords:
retrieval-augmented generation, context compression, evaluation methodology, LLM evaluation, reader comparison, benchmark validity, question answering, measurement, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qNxMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether fixed evidence compression preserves comparisons between readers in retrieval-augmented generation. It distinguishes improvements in overall pipeline accuracy from the retention of performance differences among readers. Across five QA benchmarks, five compression families, and panels of 8–20 readers, the study compares raw evidence with stored compressed text that is shared identically by every reader. It measures reader-upgrade retention, excess ordering reversals relative to raw-evidence replays, and answer-level rescue and damage. The results show that compression can improve average accuracy while substantially shrinking reader performance gaps and making reader comparisons harder to resolve. The paper releases a row-level interaction dataset and the ragscale toolkit for auditing reader comparisons under raw and fixed compressed evidence.

Critical strengths
  1. This paper draws a practically important distinction: end‑to‑end compressed‑pipeline accuracy versus preservation of reader performance gaps. The warning that compression can lift average accuracy yet diminish true reader upgrades offers valuable guidance for RAG evaluation.

  2. Fixed‑artifact replay provides strong experimental control. By storing compressed evidence and verifying that every reader receives identical text, the design prevents variation in compressor outputs from contaminating reader comparisons.

  3. The statistical analysis anticipates important pitfalls. The authors account for shared sampling noise in raw‑score–gain correlations, examine endpoint‑selection sensitivity, and compare compressed ordering reversals with raw‑evidence replays. They further distinguish split‑sensitivity ranges from population uncertainty and disclose numerical issues in the errors‑in‑variables correction.

Critical weaknesses
  1. Treating performance‑gap attenuation as “evaluation distortion” lacks sufficient justification. Raw‑evidence performance is not an inherent ground truth; compression alters task conditions and may legitimately reduce model advantages. The manuscript should better distinguish misleading measurement artifacts from valid condition‑driven performance shifts.

  2. Experiments do not fully disentangle compression from evidence‑selection confounds. HotpotQA and MuSiQue have mismatched document access between raw and compressed conditions. Existing matched‑access controls only apply to a tiny subset of readers, not the full panel.

  3. Endpoint‑pair metrics (retention, excess flips) have limited representativeness for full‑panel behaviour. Gap shrinkage and sampling noise alone can create apparent reversals without true ranking changes. Analyses covering all reader pairs are needed to solidify global claims.

Questions for authors
  1. In what evaluation scenarios should raw‑evidence reader performance be treated as the reference benchmark that compression ought to preserve? How do you operationally distinguish misleading measurement artifacts from legitimate, condition‑driven performance changes when evidence inputs are altered via compression?

  2. In HotpotQA RECOMP setup, compressor selects top‑5 passages while raw input uses all 10 passages. How much of the observed drop in upgrade retention can be attributed purely to filtering out passages, versus subsequent token‑level compression / rewriting? Can you decompose variance from passage selection and compression using your existing data or supplementary control analyses

  3. How many of the reported excess rank‑flip events occur among reader pairs that already have tiny performance gaps under raw evidence? When filtering to only pairs with large, statistically significant raw‑evidence performance differences, does excess‑flip magnitude shrink substantially?

  4. Your matched‑access control only covers three readers on MuSiQue. From this small subset, what evidence do you have that the same confound magnitude would persist across the full 15‑reader panel? What risk remains that document‑access mismatch is inflating your measured attenuation effect?

Reviewer_k2kiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how context compression affects performance comparisons between different readers in RAG. Specifically, the authors compare multiple readers using both original and compressed documents across five QA benchmarks and several compression methods. They observe that compression can reduce the performance gaps between readers and sometimes change their rankings, even when the overall QA performance improves. The authors also release an evaluation toolkit called ragscale.

Critical strengths
  • The authors put some engineering effort on scaling up the experiments using multiple readers, although the findings themselves are not really novel.

  • There are some detailed experimental analysis, including changes in reader rankings and some answer transitions or rescue and damage, although the overall findings are not really surprising.

Critical weaknesses
  • The novelty is highly limited. The effects of the reader (or answer generator) of context compression have already been observed a lot in the prior work, such as EXIT and Refiner. While the authors also acknowledge this, simply adding more experiments with more readers and compression methods is not novel. There is no fundamentally novel methods or observations beyond the previously observed phenomena.

  • Furthermore, since LLMs are generally sensitive to changes in their input context, it is not particularly surprising that compression affects different readers differently.

  • The motivation is also not strong and the practical significance is weak. Basically, why is it important to evaluate how the performance gaps between different readers change before and after compression? Since compression changes the input evidence, it is natural that different readers would be affected differently.

  • Regarding the previous (fundamental) motivation concern, if a RAG system uses compressed evidence in practice, wouldn't the performance under compression be more relevant than preserving the performance differences observed with uncompressed evidence?

  • Moreover, the reduced performance gaps could actually be a positive outcome, as weaker (and potentially cheaper) readers may achieve comparable performance with compressed evidence.

Questions for authors

See above.

Reviewer_DJvSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that RAG readers should be evaluated with and without context compression. Weak readers benefit more from context compression, so the benefit of strong readers on noisy context is obscured when using compressed-only evals. They present results using HotpotQA and Musique.

Critical strengths

S1 - The paper accurately points out that weak readers can be effective when using advanced context management such as compression.

S2 - The protocol in the paper is straightforward: measure RAG system performance w/ and w/o compression.

S3 - The paper measures an extensive number of readers.

Critical weaknesses

W1 - It is not obvious why we should evaluate models in settings (i.e. uncompressed) that are not likely to occur in production. This is akin to evaluating reranker against a huge number of documents, when production is only reranking 50 documents. Additionally, it seems wasteful to spend extra compute generalizing readers to maintain such scenarios --- it seems more economical to use smallest / weakest model that can achieve the same quality in the likely setting.

W2 - The paper focuses on reader selection, despite experiments with multiple compression algorithms indicating that compression greatly influences pipeline performance. It is clear that these choices should not be made in isolation. It would be beneficial if the work considered the joint selection of readers and compression algorithms, and perhaps other pipeline decisions related to retrieval.

W3 - The paper does not propose a protocol that stresses readers while keeping context length controlled. For instance, shuffling tokens, swapping in distractor documents, swapping in adversarial tokens are all ways to test reader quality while maintaining the compressed setting.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.