- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CompEvidenceBench is an auditable benchmark for reader-conditioned complementary evidence in context compression. It constructs finite candidate slates containing closed edit quartets, keeps answer-derived interaction labels behind an audit boundary, and evaluates additive, learned, and external generative selectors under fixed readers and budgets.
Critical strengths
The benchmark contract is unusually explicit. The separation between compressor-visible inputs and offline answer-derived labels is carefully specified, and the oracle, matched-control, label-permutation, and cross-reader analyses are useful safeguards against over-interpreting a single F1 number. The paper is also candid that its learned selectors do not recover the available oracle headroom and that the external compressor may have training overlap.
Critical weaknesses
The current evidence is too narrow for the paper’s intended benchmark role. The prototype is derived from HotpotQA training questions, uses a single construction reader, and reports opened-development results rather than an untouched confirmation set. The “complementary” label is explicitly reader-conditioned and depends on a finite slate around one additive rollout; it therefore measures a particular operational failure mode, not a general semantic property of multi-hop evidence. Sensitivity to anchor selection, slate size, and candidate construction is not established.
The learned selectors do not show a stable improvement over the additive reference, so the paper’s main positive evidence comes from an answer-dependent oracle and an external generative compressor with unknown training overlap. Those results demonstrate that the slate contains repairable opportunity, but they do not validate the benchmark as a reliable basis for comparing future compressors. The paper would be stronger with a sealed, question-disjoint test set and at least one additional dataset or construction reader.
The related-work coverage is incomplete. The paper should also discuss the following directly relevant works:
- COMI: Coarse-to-Fine Context Compression via Marginal Information Gain — ICLR 2026
- GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment — ACL 2026
- In-context Autoencoder for Context Compression in a Large Language Model — arXiv, 2023
- Learning to Compress Prompts with Gist Tokens — NeurIPS 2023
- Adapting Language Models to Compress Contexts — EMNLP 2023
These references would help distinguish the benchmark’s evidence-selection setting from prior latent, prompt-level, and context-reencoding compression methods.
Questions for authors
Can you evaluate the released contract on a sealed question-disjoint test set and an additional dataset or reader? How do complementary-case rates and oracle gaps change with the anchor rollout, candidate-slate size, and sentence-construction rules? What safeguards will prevent benchmark questions or generated candidate text from entering compressor pretraining in future evaluations?