- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies evidence sufficiency estimation before answering. Its main contribution is identifying a paragraph-count shortcut in benchmarks constructed by deleting supporting passages, proposing a count-matched construction, and introducing MEMSAFE, which combines a cross-encoder with a set transformer. Experiments cover multi-hop QA, cross-construction transfer, clinical records, and downstream answer gating.
Critical strengths
- The evaluation artifact is clear and consequential: paragraph count alone achieves 0.979 AUROC, showing that deletion-based construction can substantially inflate performance.
- The comparisons and ablations are reasonably comprehensive, including matched mean pooling, a long-context encoder, and an LLM judge.
Critical weaknesses
- The constructed task remains distant from realistic applications. Missing-evidence examples primarily rely on annotated supporting passages, whose absence does not necessarily make a question unanswerable. Evaluation on memories produced by actual retrieval or compression systems is needed to establish broader validity.
- Generalization and downstream utility remain limited. AUROC drops to approximately 0.639 on SQuAD 2.0, and transferred risk thresholds fail. Although answer gating reduces errors, the 7B reader still has approximately 63% error at only 5% coverage, providing limited evidence of practical reliability.
Questions for authors
Can the authors validate the approach on memories produced by actual retrieval or compression, and audit cases where questions remain answerable despite missing annotated evidence? How much coverage does set-level interaction gain over mean pooling at matched test risk?