- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Eclipse, a framework for reducing the retrieved context of an existing RAG output while jointly assessing two requirements: output recovery and evidence completeness. Output recovery requires the model to reproduce its original answer from a reduced context, while evidence completeness requires retaining all benchmark-annotated supporting passages. The method starts with a protected set of top-ranked passages and greedily adds passages based on the likelihood of the original output until the answer is reproduced or a context-size limit is reached. Experiments show that protected forward search can improve complete matching coverage over retrieval-order addition.
Critical strengths
- Reproducing an original answer from a reduced context does not imply that all supporting evidence remains available. The paper formalizes these as separate evaluation requirements and demonstrates cases where output recovery succeeds despite missing annotated evidence. This distinction is relevant to evaluating the reliability of reduced contexts used for inspecting LLM outputs.
- The authors use separate calibration and test samples, account for multiple policy comparisons, and explicitly discuss the assumptions underlying their risk guarantees.
Critical weaknesses
The methodological and theoretical novelty is limited, as the core framework primarily combines existing techniques. The proposed protected forward search is essentially a greedy context-selection procedure guided by the likelihood of reproducing a reference output. Related ideas of output-preserving context selection and attribution have already been investigated in prior work, including SelfCite (https://arxiv.org/abs/2502.09604).
Theorem 1 applies standard Clopper–Pearson confidence bounds to two predefined risks and uses a union bound to obtain simultaneous control. This follows the established multi-risk calibration framework of Learn then Test (https://arxiv.org/abs/2110.01052).
Therefore, the main contribution appears to be combining output recovery and annotated evidence completeness as two requirements within an existing risk-control framework. While this is a reasonable formulation, the paper does not clearly establish what new technical capability or insight this combination provides beyond applying established multi-risk calibration to a context-selection procedure.
Questions for authors
The paper motivates reduced contexts as making model outputs easier to inspect, but does not directly measure human auditing accuracy, error detection, or inspection time. Is there any existing evidence that the reduced contexts improve the effectiveness of downstream auditing, instead of reducing the amount of text presented?