acceptodds
Under review as a conference paper at ICLR 2027

CompEvidenceBench: Diagnosing Complementary Evidence in Context Compression

Abstract

Budgeted context compression is usually evaluated only by end-to-end answer accuracy. This aggregate score cannot reveal whether failure reflects unavailable evidence, a compressor that drops a jointly useful context, or a reader that does not use retained evidence. We introduce CompEvidenceBench, an auditable benchmark for reader-conditioned complementary-evidence selection failures. For each question, it fixes a budgeted candidate slate containing closed edit quartets, and uses an offline interaction score to audit whether a joint edit is more useful than its constituent edits under a fixed reader. Reference answers, candidate utilities, oracle choices, and interaction labels are never visible to the compressor. On HotpotQA-derived candidate slates at 512 and 1024 tokens, an answer-dependent candidate oracle is 13.18 and 13.04 F1 points above a frozen additive selector overall; on reader-conditioned complementary cases, the gaps are 41.13 and 33.62 points. Yet a three-seed direct-text selector with interaction supervision does not recover this headroom reliably, and a separate frozen pairwise-decision track also yields no stable gain. In contrast, an external generative compressor improves on the complementary subset under both the construction reader and a Mistral cross-reader, although its unknown training overlap prevents a leakage-free ranking claim. CompEvidenceBench therefore separates available opportunity, learned recovery, and external sensitivity: it is an auditable diagnostic testbed rather than evidence for a solved compression method. Our code and dataset are available at https://anonymous.4open.science/r/CompEvidenceBench-2434/README.md.

Primary area:
datasets and benchmarks
Keywords:
context compression, long-context evaluation, benchmark, complementary evidence, multi-hop question answering, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BHNfOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

CompEvidenceBench is an auditable benchmark for reader-conditioned complementary evidence in context compression. It constructs finite candidate slates containing closed edit quartets, keeps answer-derived interaction labels behind an audit boundary, and evaluates additive, learned, and external generative selectors under fixed readers and budgets.

Critical strengths

The benchmark contract is unusually explicit. The separation between compressor-visible inputs and offline answer-derived labels is carefully specified, and the oracle, matched-control, label-permutation, and cross-reader analyses are useful safeguards against over-interpreting a single F1 number. The paper is also candid that its learned selectors do not recover the available oracle headroom and that the external compressor may have training overlap.

Critical weaknesses

The current evidence is too narrow for the paper’s intended benchmark role. The prototype is derived from HotpotQA training questions, uses a single construction reader, and reports opened-development results rather than an untouched confirmation set. The “complementary” label is explicitly reader-conditioned and depends on a finite slate around one additive rollout; it therefore measures a particular operational failure mode, not a general semantic property of multi-hop evidence. Sensitivity to anchor selection, slate size, and candidate construction is not established.

The learned selectors do not show a stable improvement over the additive reference, so the paper’s main positive evidence comes from an answer-dependent oracle and an external generative compressor with unknown training overlap. Those results demonstrate that the slate contains repairable opportunity, but they do not validate the benchmark as a reliable basis for comparing future compressors. The paper would be stronger with a sealed, question-disjoint test set and at least one additional dataset or construction reader.

The related-work coverage is incomplete. The paper should also discuss the following directly relevant works:

  • COMI: Coarse-to-Fine Context Compression via Marginal Information Gain — ICLR 2026
  • GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment — ACL 2026
  • In-context Autoencoder for Context Compression in a Large Language Model — arXiv, 2023
  • Learning to Compress Prompts with Gist Tokens — NeurIPS 2023
  • Adapting Language Models to Compress Contexts — EMNLP 2023

These references would help distinguish the benchmark’s evidence-selection setting from prior latent, prompt-level, and context-reencoding compression methods.

Questions for authors

Can you evaluate the released contract on a sealed question-disjoint test set and an additional dataset or reader? How do complementary-case rates and oracle gaps change with the anchor rollout, candidate-slate size, and sentence-construction rules? What safeguards will prevent benchmark questions or generated candidate text from entering compressor pretraining in future evaluations?

Reviewer_3faUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CompEvidenceBench, an auditable benchmark for diagnosing complementary-evidence failures in context compression. Using fixed candidate slates and closed edit quartets, it shows substantial answer-dependent oracle headroom, while tested interaction-aware selectors fail to recover it reliably. An external generative compressor improves on complementary cases under two readers, although its unknown training overlap prevents a leakage-free ranking claim.

Critical strengths
  1. The benchmark clearly separates candidate opportunity, learned selection, and reader dependence.

  2. Closed edit quartets provide a concrete and auditable way to measure complementary evidence.

Critical weaknesses
  1. The main results are reported on opened or adaptive development data, without an independent held-out test set.

  2. Complementarity labels depend on the Qwen2.5-7B reader, its tokenizer, and its generation configuration, so they may not represent reader-independent semantic complementarity.

  3. Candidate slates are constructed around a single additive anchor with a limited edit space, which may constrain the measured opportunity.

  4. The negative result for the main learned selector is based on a specific architecture, objective, and only three random seeds, so it does not rule out stronger interaction-aware selectors.

  5. Cross-reader validation uses only one additional Mistral reader, and the replay may exceed its native token budget.

Questions for authors
  1. Can the authors validate the main conclusions on an independently held-out test set?

  2. How stable are the complementarity labels across different readers, tokenizers, and decoding configurations?

  3. Would expanding the candidate edit space or using a stronger set-level selector substantially reduce the observed oracle gap?

Reviewer_rYSCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an evaluation framework for context compression, focusing on cases where multiple pieces of evidence are complementary. The experiments use a particular setup where: (1) compression is primarily studied as extractive sentence selection on HotpotQA; (2) the compressor selects from a fixed set of candidate contexts rather than directly generating a new context; and (3) the candidate contexts are constructed by adding sentences from the original evidence pool to an existing compressed context. The results indicate that an external generative baseline (BRIEF-Pro) performs best, while the proposed interaction-aware trained selectors do not reliably improve over an additive baseline.

Critical strengths
  1. The paper identifies an interesting potential failure mode of context compression, where retaining only one of two related facts can make the retained information insufficient to answer a question. This is particularly relevant to extractive compression, where a fixed token budget may make it difficult to preserve all related evidence, although generative compression could potentially alleviate this issue.

  2. The paper presents a creative evaluation setup where compression is performed by selecting from a fixed set of extractive candidate contexts. This allows for controlled comparisons and explicit measurement of interactions between pieces of evidence.

  3. The experiments include both generative baselines and models trained directly for the proposed selection task.

Critical weaknesses
  1. The paper is very difficult to follow. The authors provide only one concrete example in the main text, and it does not show the actual anchor context or the resulting context with both complementary facts included. Additionally, it took me multiple reads to understand that the central evaluation concerns extractive sentence selection, where sentences are retained from the original context rather than generated. One of the clearest descriptions appears in Section 3.3: "A candidate is a retained-sentence set obtained from the anchor by atomic edits that add one pool sentence." However, this does not straightforwardly explain the setup. At minimum, the paper should make the extractive nature of the central evaluation explicit much earlier and provide complete worked examples.

  2. The evaluation setup is somewhat contrived. The core diagnostic is restricted to selecting among a small, synthetically constructed set of extractive candidate contexts. In contrast, generative compression is commonly used in modern agent systems such as Cursor, ChatGPT, and Claude Code. Additionally, HotpotQA is the only evaluated dataset, which does not sufficiently represent the range of scenarios where context compression is applied. It is unclear how well the findings generalize to realistic agent workloads or generative compression methods.

  3. There is already work demonstrating that generative context compression can be trained effectively using downstream task signals. For instance, KARL (https://arxiv.org/abs/2603.05218) and Cursor's "Training Composer for Longer Horizons" (https://cursor.com/blog/self-summarization) explore related approaches. The paper would benefit from more clearly positioning its proposed diagnostic relative to these methods and explaining whether its findings extend to learned generative compression.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.