acceptodds
Withdrawn from ICLR 2027

Eclipse: Joint Risk Control for LLM Output Recovery and Evidence Completeness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting fewer retrieved passages can reduce the amount of text needed to inspect a language model's existing output. However, reproducing that output from fewer passages does not establish that the supporting evidence remains available. We introduce Eclipse, a framework for evaluating output recovery together with evidence completeness, defined as retaining all benchmark-annotated supporting passages. Its search keeps a small initial set of retrieved passages and adds passages chosen by the likelihood of the original output. It stops when the model reproduces that output or reaches a preset size limit. We then evaluate the fixed search procedure on separate calibration samples. These provide upper confidence bounds on recovery failure across queries and missing evidence among returned sets. Certification requires both bounds to meet preset limits under the stated independence and sampling assumptions. On held-out 2Wiki queries, search with Qwen and a supervised retriever satisfies both requirements for 94.1% of queries, exceeding retrieval-order addition by 9.2 percentage points. This configuration also meets the preset limits for both calibration bounds. Across our experiments, search mainly helps when the initial passages already contain the supporting evidence; recovering the output often leaves missing evidence unresolved. Jointly checking output recovery and evidence completeness therefore helps assess smaller passage sets, while neither establishes answer correctness.

Primary area:
interpretability and explainable AI
Keywords:
retrieval-augmented generation, large language models, context selection, evidence completeness, statistical risk control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5BM4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Eclipse, a framework for reducing the retrieved context of an existing RAG output while jointly assessing two requirements: output recovery and evidence completeness. Output recovery requires the model to reproduce its original answer from a reduced context, while evidence completeness requires retaining all benchmark-annotated supporting passages. The method starts with a protected set of top-ranked passages and greedily adds passages based on the likelihood of the original output until the answer is reproduced or a context-size limit is reached. Experiments show that protected forward search can improve complete matching coverage over retrieval-order addition.

Critical strengths
  1. Reproducing an original answer from a reduced context does not imply that all supporting evidence remains available. The paper formalizes these as separate evaluation requirements and demonstrates cases where output recovery succeeds despite missing annotated evidence. This distinction is relevant to evaluating the reliability of reduced contexts used for inspecting LLM outputs.
  2. The authors use separate calibration and test samples, account for multiple policy comparisons, and explicitly discuss the assumptions underlying their risk guarantees.
Critical weaknesses

The methodological and theoretical novelty is limited, as the core framework primarily combines existing techniques. The proposed protected forward search is essentially a greedy context-selection procedure guided by the likelihood of reproducing a reference output. Related ideas of output-preserving context selection and attribution have already been investigated in prior work, including SelfCite (https://arxiv.org/abs/2502.09604).

Theorem 1 applies standard Clopper–Pearson confidence bounds to two predefined risks and uses a union bound to obtain simultaneous control. This follows the established multi-risk calibration framework of Learn then Test (https://arxiv.org/abs/2110.01052).

Therefore, the main contribution appears to be combining output recovery and annotated evidence completeness as two requirements within an existing risk-control framework. While this is a reasonable formulation, the paper does not clearly establish what new technical capability or insight this combination provides beyond applying established multi-risk calibration to a context-selection procedure.

Questions for authors

The paper motivates reduced contexts as making model outputs easier to inspect, but does not directly measure human auditing accuracy, error detection, or inspection time. Is there any existing evidence that the reduced contexts improve the effectiveness of downstream auditing, instead of reducing the amount of text presented?

Reviewer_SXccOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Eclipse, a framework that selects a smaller set of passages to help users check an answer already generated by a language model. It uses greedy forward search to find contexts that reproduce the original output. Separate calibration samples assess output recovery failure and missing annotated evidence among returned contexts. Experiments on MuSiQue and 2Wiki compare several rankers and search rules. The results show that evidence retention depends strongly on whether the initial passages already contain the required support.

Critical strengths

S1. The framework evaluates two distinct types of failure. The paper measures output recovery failure across all queries and missing annotated evidence among returned contexts. This separates answer consistency from evidence completeness. It also explains that a model can reproduce an incorrect answer even when all annotated evidence is retained.

S2. The experiments examine important factors beyond final performance. The evaluation compares different rankers and search rules, reports computational costs, and includes additional studies of retrieval quality and document overlap. These analyses help explain when the search works and reveal its dependence on the initial evidence.

Critical weaknesses

W1. The example in Figure 1 needs a clearer explanation. Figure 1 shows the model answering "Lisbon" after the passage giving Mira's birthplace is removed. The paper does not explain how the model gets this answer without that passage. It could rely on knowledge from training or simply guess. Explaining this would help readers understand why matching the original answer does not ensure that the evidence is still present.

W2. The paper does not explain why the original answer must be reproduced. The paper aims to help users check an existing answer with less text. Keeping the evidence supports this goal, but the need to reproduce the original answer is less clear. If the original answer is wrong, a smaller context with complete evidence might lead the model to correct it. This would fail the recovery check despite being useful to the user. The paper does not show what requiring the same answer adds beyond keeping the evidence.

W3. The contribution beyond prior work is unclear. The paper does not discuss two closely related studies.

  • Feng et al. Pathologies of Neural Models Make Interpretations Difficult. EMNLP, 2018. This study showed that models can keep the same prediction even when reduced inputs lack enough information to support it.
  • Joren et al. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR, 2025. This study separates context sufficiency from answer correctness. It examines correct answers from contexts judged insufficient and discusses stored knowledge and guessing as possible explanations.

The paper does not clearly explain what new insight or method it adds beyond these studies. It should clarify what is gained by requiring both the same answer and all annotated evidence.

W4. The thresholds lack explanation and the handling of abstention is unclear. The paper treats 0.1 for recovery failure and 0.2 for evidence error as experimental limits. It does not explain why these error rates are acceptable for users checking answers or why at least 40 returned contexts are required in the evidence calibration sample. It also does not specify how users should proceed after abstention. The reported text reductions only cover queries where a context is returned and do not account for what users must read when search fails.

W5. The search method adds little beyond standard greedy search. The search starts from four ranked passages and scores candidate additions. Since forward search only adds passages, its initial set is already preserved. The protection rule adds no new constraint. Appendix E.1 also describes the procedure as standard greedy search.

Questions for authors

Q1. What benefit does requiring the original answer provide over retaining complete evidence alone? Please explain its value when the original answer is wrong and provide a concrete example or comparison showing how this requirement helps users inspect the answer.

Q2. What is the main new insight beyond prior work on input reduction and context sufficiency? Please distinguish the contribution of the task and certification protocol from the use of standard greedy forward search.

Q3. If users receive the full context after abstention, what is the average number of passage tokens shown across all queries? Please compare this with showing all candidate passages and exclude instructions, demonstrations, and chat formatting from both counts.

Reviewer_RrDFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study the task of sparsifying a retrieval set for the model, while retaining the original models' output as well as ensuring that the answer is supported by the set of passages given to the model.

Critical strengths

Retrieval augmented generation systtems are important, and ensuring that we understand how the input passages map to model outputs is generally an important problem.

Critical weaknesses

The paper is sufficiently badly written that I cannot actually parse the goal of the paper, or the rationale of how the high-level goals map to the actual experiments.

Why are we trying to sparify the retreival set of a LM on a benchmark?

Why do we do this incrementally via this 'add passages' process?

I apologize for the somewhat short review, and I can follow up after the rebuttal, but I genuinely think it's difficult for me to understand what the authors are trying to do (or whether it makes sense) from the writing.

The paper is obviously LM written, which is not inherently disqualifying, but written in a way that makes it basically impossible to follow the paper.

Questions for authors

See questions above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.