acceptodds
Under review as a conference paper at ICLR 2027

Evidential Entitlement in RAG: Beyond Correctness and Citation

Abstract

Retrieval-augmented generation is meant to make answers more trustworthy by grounding them in external evidence. Yet a correct answer, even with a plausible citation, does not show that the cited evidence actually backed that answer when it was generated. The same answer can follow from valid retrieval, stale evidence, an incomplete search, or retrieval failure. We call this missing property *evidential entitlement*. Entitlement depends on the relation between the current request, the evidence actually delivered, and the resulting claim. This creates a basic asymmetry: one matching source can support a positive claim, but a no-match claim also requires evidence that the relevant search was complete. We show that if these distinctions are absent from the visible output, no later evaluator can recover them, and we characterize the minimum additional information needed to preserve them. We test these predictions on official Python, Kubernetes, and Federal Register corpora across four model families. Among 11,520 pairs with different entitlement, 42.26% still produce the same correct answer. Source identity and version remove many ambiguities, but the remaining failures come from exactly the missing relations predicted by the theory: query binding and search completion. A held-out three-field interface preserves these distinctions without exposing the full execution trace.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Retrieval-augmented generation, evidential entitlement, provenance, verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E8mSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes and studies the idea of 'evidential entitlement', which is a bit more general than citation (which sources that were inputs to the LLM, not sources that were cited in the final output). The authors variously discuss ideas around citation and evidence, with some theory and targeted interventions, 'entitlement flipping'

Critical strengths

In general, the study of citations and evidence is interesting and useful.

Questions of coverage and scope of the retriever, and how that interacts with a lack of evidence, are an important problem in human-LM interfaces.

Critical weaknesses

I suspect most, or all, of this paper is AI-generated. Worse, I suspect that the ideation and execution of the paper are AI-generated. I've seen such papers enough to have a general sense of this - the paper includes spurious theory despite not really being a theory paper, detailed experiments that are based on niche ablation-type questions, and so on. AI generation may not be disqualifying in and of itself, but AI-generated papers that are unpleasant for reviewers to read are a disqualifying criterion.

I also don't really find the particular question they ask in this paper to be interesting. Evidence is, of course, interesting, but why focus on what evidence was given to the LM generator? Having read the paper, I've yet to understand why this is a key concern in generation reliability..

I also find the experiment design to be suspect. For a question that's tied to human downstream decisions, the eval designs are strange - no human eval or user studies, no downstream decision evals..

Questions for authors

What is the purpose of theorems 1, 3, 4. Can they not be entirely removed without affecting the main claims of the paper?

Do you have evidence that evidential entitlement is useful for downstream user decision making, users making calibrated decisions, or other more grounded outcomes?

1 comment
AuthorsOpenReview ↗

Evidence Delivery, Theoretical Results, and Downstream Decisions

Thank you for your feedback on the paper’s motivation, presentation, and evaluation.

W1. Concerns about AI involvement and readability

The submission form discloses AI assistance with writing, literature discovery, research ideation or execution, and mathematical proofs. Page 10 describes language and structure editing, literature discovery, analysis and figure code editing, and feedback on presenting experiments. As disclosed, the authors reviewed the assisted text and code, checked citations against their sources, and verified the numbers and figures against the original outputs. The authors take responsibility for the manuscript and its conclusions.

W2. Why focus on what evidence reaches the generator?

The delivery record tells users where to look next. In archived case 003, the assistant said it could not confirm whether the Kubernetes v1.30.0 API specification defines AppArmorProfile. The first delivered passage explicitly lists that definition, so the user can check the answer against the source. In the HotpotQA example, the answer and citation are correct, but the passage connecting the question to the answer was never delivered. The first case calls for reviewing or correcting the answer. The second calls for retrieving the missing evidence. We will introduce these examples before the definitions and experiments.

W3. Does the evaluation match the paper’s motivation?

The offline replay in Q2 evaluates whether the record leads an automated workflow to select the appropriate next action. Measuring the effect on users’ decisions requires observing how they use the record.

Q1. What would be lost by removing Theorems 1, 3, and 4?

Removing the theorems would leave the observed collision counts intact and remove three steps linking those observations to the design of our records.

  • Theorem 1 identifies what relationship to preserve. In the support-score model, changing which evidence is assigned to which claims can change total support even when the evidence and claims are the same. The theorem characterizes when that relationship can be omitted. In the HotpotQA example, the novel and screenwriter passages establish different steps.
  • Theorem 3 identifies what a no-match claim must prove. A record showing no match in the searched portion is consistent with both complete absence and a match in the unsearched portion. This motivates verifiable search coverage in a negative certificate.
  • Theorem 4 quantifies the cost of omitting information. If two cases require opposite decisions but show the verifier the same record, the verifier must incur at least 50% error on that equally weighted pair. The theorem extends this to the paired distribution. Fields that distinguish those executions can lower the bound.

Q2. What evidence is there that the record improves downstream decisions?

During a Kubernetes upgrade review, an engineer who sees “unable to confirm” must decide whether to close the check for AppArmorProfile or send it back for another search. In case 003, the first delivered passage lists the exact definition, so the engineer can close that specific check and move on to other compatibility questions. If the record shows a timeout, the check stays open and the engineer can search the requested version again. The same answer can lead to either action. We replayed four possible actions on 1,728 archived executions. The answer-only rule matched the predefined action in 666 cases, validated citations in 760, and the full execution record in 1,728. These results measure whether the system can direct the next step. The effect on time spent and repeated searches requires measurement in actual use.

Reviewer_ddy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to check whether a RAG system received supporting evidence before answering. A correct answer with a valid citation does not prove that the model saw the cited source. Similarly, “no match found” can hide a failed or unfinished search. The authors propose small, signed records showing what evidence reached the model and whether the search finished. They also develop a method for choosing which details to record. Tests across three corpora and four model families show that answer and citation checks miss many failures, while the selected records distinguish all tested pairs of valid and faulty runs in held-out domains.

Critical strengths
  • Clearly explains the difference between a correct answer, available evidence, evidence delivered to the model, and evidence that influenced the answer. The distinction between finding supporting evidence and completing a search with no matches is useful.
  • Experiments across several models, retrieval methods, and generation settings convincingly show that correct answers and valid citations can hide failures in the retrieval process.
Critical weaknesses
  • The certificate evaluation uses a narrow definition of support. Positive support checks whether a document ID known in advance was delivered, while negative certificates establish exact-key absence. The perfect results show that the records preserve these predefined conditions, but do not demonstrate verification of every statement in a generated answer.
  • The practical benefit and generality of field selection remain unclear. Held-out domains reuse the same failure types, and the three-field minimum depends on eight candidate fields with equal costs. The full eight-field record already separates all tested pairs. Tests with independently varying failures and measurements of savings over the full record would better establish practical value.
Questions for authors
  • How would you check support when the supporting document IDs are not known in advance, or when an answer adds unsupported statements?
  • Do the three selected fields still work when wrong versions, incomplete searches, and record tampering occur independently or together?
1 comment
AuthorsOpenReview ↗

Certificate Guarantees, Record Size, and Fault Tests

Thank you for your questions about what the certificates establish and when the selected fields suffice.

W1. What do the certificate results establish about support?

Each additional claim needs its own evidence. In archived case 020, the model correctly reads that a notice has a 30-day public comment period. It then says 30 days is standard practice for this type of Federal Register notice. The delivered text supports the specific notice and the commission’s work plan. We would mark the broader statement as having insufficient evidence. In 24 controlled cases containing an unsupported addition, the entity-presence baseline accepted all 24. Checking each statement preserves the supported answer and identifies the exact claim that needs evidence.

W2. What does automatic field selection add in practice?

On the original 11,520 held-out pairs, the selected three fields matched the eight-field record at zero observed collisions. The median signed receipt with the three-field retrieval-status configuration was about 16% smaller. On the expanded fault tests, the manual five-field record retained version and scope checks, matched the eight-field decisions, and used 9.4%–9.5% fewer receipt bytes. With a coverage proof included, the three-field saving for a complete negative certificate was 0.42%–1.66%. The applicable figure depends on whether the system sends a receipt alone or also needs to prove search coverage.

Q1. How can we check support without prespecified document IDs and handle unsupported additions?

On HotpotQA, the retriever receives the question and the candidate pool. With a fixed budget of four passages, using people or topics mentioned in the initial results to guide the second hop increased recall of both annotated evidence pages from 123/192 to 162/192. The checker identifies the source for each step among the delivered passages. We use the annotated pages for scoring. The claim-level check described in W1 handles additions.

We reused the signed-receipt mechanism. All 110 receipts issued for accepted complete-evidence cases verified with their original passages. All 110 failed verification when checked against the delivery with one hop missing because the evidence digest differed. If the verifier has a trusted digest of the current delivery, it can check which material the judgment covers.

Q2. Do the three fields suffice when version, scope, and integrity faults occur independently or together?

The original three-field configurations missed some faults in the expanded test. A complete search can support a no-match conclusion for the requested version and scope. A search of the wrong version or an incomplete scope calls for a new search. The authentication check rejects a receipt modified after signing.

We varied version, scope, and post-signing changes independently and in combination. Across 3,072 negative and 6,144 positive test executions, the five-field record with version and scope information and the full eight-field record made zero false accepts or false rejects. The signature check blocked every tested post-signing modification. A partial search that still finds the correct document shows exactly which decision the scope field changes.

Reviewer_19yqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces evidential entitlement, a verification target for retrieval-augmented generation (RAG) systems that distinguishes answer correctness and citation validity from whether the necessary supporting evidence was actually retrieved and delivered during the generation process. The authors formalize this problem through interface adequacy and trust-relevant collisions, showing that executions with different entitlement properties may be indistinguishable from their final outputs. They also establish an asymmetry between positive claims, which require supporting witnesses, and negative claims, which additionally require search completeness.

To address this problem, the authors propose a collision-guided framework that selects a minimal set of execution metadata fields to distinguish runs with different entitlement labels. These records are subsequently authenticated through signed receipts and coverage witnesses.

Experiments across Python, Kubernetes, and Federal Register corpora demonstrate that 42.26% of 11,520 opposite-entitlement execution pairs produce the same correct answer. The proposed three-field records eliminate observed collisions on held-out domains, while the authenticated verification prototype detects the tested integrity attacks.

Critical strengths
  1. The paper identifies an important distinction between answer correctness and execution-level evidential guarantees.

The distinction between evidence availability, evidential entitlement, and causal evidence use provides a useful perspective on RAG reliability. In particular, the paper demonstrates that a correct answer with a valid citation does not necessarily establish that the supporting evidence was available to the generator during the corresponding execution. The witness–coverage asymmetry is also practically meaningful: positive claims can be supported by individual witnesses, whereas negative claims require evidence about search completeness. These observations reveal limitations of conventional answer-level evaluation, especially for systems involving external retrieval and tool execution.

  1. The paper provides a systematic verification framework supported by extensive controlled experiments.

The collision-guided interface synthesis formulation offers a principled way to identify execution metadata required for verification without exposing complete execution traces. The evaluation covers multiple model families, retrieval methods, and versioned corpora. The empirical finding that 42.26% of opposite-entitlement pairs produce the same correct answer provides convincing evidence that answer correctness alone cannot establish entitlement. The integration of authenticated execution records and coverage witnesses further demonstrates the feasibility of the proposed verification approach under the stated trust assumptions.

Critical weaknesses
  1. Limited generalizability to realistic RAG scenarios.

The experiments primarily focus on exact document or entity matching, where evidential entitlement is determined by predefined retrieval conditions. However, realistic RAG tasks often require semantic reasoning over incomplete, conflicting, or multiple sources. It remains unclear whether the proposed framework can provide meaningful verification guarantees in these more challenging settings.

  1. Limited novelty and unclear practical benefits of the proposed verification framework.

The theoretical results largely build on established indistinguishability arguments and set-cover formulations. Moreover, the zero-collision results may be largely attributable to metadata fields that directly encode the predefined entitlement conditions. Since the held-out experiments use the same failure categories, they provide limited evidence of generalization to unseen failures. Comparisons with simpler manually designed verification records would help establish the added value of collision-guided interface synthesis.

Questions for authors

No.

1 comment
AuthorsOpenReview ↗

Multi-Hop Verification and Automatic Field Selection

Thank you for your feedback on generalizability and the practical value of field selection.

W1. Generalizability to realistic RAG scenarios

Our record lets a verifier check whether the material delivered to the model supports each step needed for the answer. We tested 192 pairs of two-hop questions on HotpotQA. One question asks which Disney film was scripted by the author of a particular novel. One passage identifies Linda Woolverton as the novel’s author, and another establishes that she wrote the screenplay for Beauty and the Beast. We removed the first passage while retaining the correct answer and the valid citation to the second. The citation still supports the screenplay claim, but the evidence linking Woolverton to the novel was no longer delivered.

The answer and citation were identical within each pair and could distinguish none of the 192 pairs. When Qwen2.5-7B directly judged the delivered evidence, it accepted the complete version and rejected the incomplete version in 59/192 pairs. Asking the same model to identify the source for each step increased this to 79/192. Checking the delivered material step by step can reveal gaps that a valid citation alone leaves hidden.

W2. Novelty, generalization to new faults, and comparison with manual records

Our contribution is to use indistinguishable execution pairs to design the record. A search of the requested version and a search of the wrong version can both return no match. If the verifier sees the same record in both cases, that pair tells us which information is missing. We use set cover to find the lowest-cost combination of candidate fields that separates the relevant pairs.

In the original experiment, claim_support records whether a prespecified document was delivered. Field selection asks which execution details are needed to distinguish cases that require different decisions. In HotpotQA, the support checker identifies which delivered passages support the two steps. The experiments show how to retain information for verification and how to obtain support judgments in a two-hop task.

We held out Python, Kubernetes, and the US Federal Register in turn, selecting fields on the other two domains and applying the selected record to the third. The three-field records matched the full eight-field record at zero observed collisions across 11,520 held-out pairs. The manual five- and six-field records also had zero collisions. Automatic selection establishes that three fields suffice and are minimal under equal costs for these evaluated pairs.

The new faults show how to apply the same design method when verification requirements change. A complete search and a partial search can both find the expected document, yet only one meets the requirement to search the full requested scope. The five-field baseline, which records version and scope, separated those cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.