acceptodds
Under review as a conference paper at ICLR 2027

Resource Attribution in RAG: Selection, Verification, and Utilization

Abstract

We develop a resource attribution framework for recovery after limited fact verification in retrieval-augmented generation. Grounded in statistical decision theory and information-channel comparisons, the framework separates ideal information advantages from rendering loss and a fixed generator's utilization gap. It distinguishes privileged selection of existing evidence from acquiring verification messages, and derives matched controls for attributing observed recovery. A 200-group study on HotpotQA and MuSiQue instantiates these controls across content availability, claim status, and answer selection. Same-text verification under competition raises reference-alias EM by 28.5 and 34.0 points for two Qwen generators, with a smaller, dataset-dependent interaction for Mistral. Retrieved-document controls reveal a different attribution: matches rise from 104/200 to 135/200 when an already-visible oracle-selected quotation is re-exposed as ordinary evidence, then to 137/200 when it is marked verified. The incremental EM effect is 1.0 point [], with secondary benefits in F1 and replacement matching. A frozen answer-blind selector reaches 98/200, failing to retain the oracle gain. These results demonstrate how the framework separates protocol-specific recovery effects and limits their extrapolation, while keeping reference recovery distinct from appropriate abstention. To facilitate further research in this area, we release our code anonymously at https://anonymous.4open.science/r/fsws62dwas212fdf345.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, resource attribution, evidence selection, fact verification, knowledge poisoning, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kcJGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes a resource attribution framework for understanding performance recovery under RAG settings following limited fact verification.
  • Drawing on stats decision theory and info-channel comparisons, the authors distinguish the effects of evidence selection, verification, and generator utilization.
  • The formulation separates ideal information advantages from rendering losses and generator excess risk, with matched experimental controls intended to isolate the contributions of different interventions.
  • Experiments on two-hop question groups drawn from HotpotQA and MuSiQue looks as content availability, competing claims, verification status etc.
  • The results suggest that substantial recovery improvements can arise from oracle-assisted evidence localization rather than verification itself.
  • In the retrieved-document setting, re-exposing an oracle-selected quotation improves exact match from 104/200 to 135/200, while adding its verification marker yields a smaller improvement to 137/200.
  • An executable answer-blind selector fails to reproduce the oracle-assisted gains. There might be limitations of attributing recovery improvements to verification alone.
Critical strengths
  1. Useful information-theoretic perspective on evidence attribution.
  • I find the problem setup and methodology to be an interesting and useful perspective on how retrieval-related improvements should be interpreted.
  • There are sufficient establishment on statistical decision theory to distinguish information advantages from rendering and generator-utilization effects.
  • I think this provides a more disciplined way of examination and observation, which I think should be a baseline for all technical papers going up for submission.
  1. Careful experimental controls and interpretation.
  • Experimentation set up is nice and I appreciate the authors' effort to isolate different intervention effects through matched comparisons.
  • Particularly the distinction between oracle-assisted localization and verification-status changes is useful for understanding the contributing factors.
  • The finding that re-exposing an already-visible quotation accounts for substantially more improvement than adding a verification marker is informative.
  • Also, the authors are also reasonably transparent about the limitations of their oracle-assisted setting, and the distinction between reference recovery and justified abstention is valuable.
Critical weaknesses
  1. Limited scientific novelty and significance.
  • While the resource attribution framework is a useful take on retrieval mechanisms, the underlying decomposition largely follows well-established principles of statistical decision theory and info channel comparisons. I fail to uncover anything significant or novel here apart from the validation that this is true.
  • It remains unclear what fundamentally new scientific insight emerges beyond carefully controlled selection and verification experiments.
  • Furthermore, the advantages introduced in the formulation are not empirically estimated -> limiting the extent to which the theoretical framework provides additional explanatory value.
  • I'm therefore not entirely convinced that the contribution represents a sufficiently substantial methodological advance. Unless, I'm missing something important here.
  1. Unclear operationalization and generalizability?
  • The framework provides a structured interpretation of recovery gains, but its applicability to real-world RAG systems remains uncertain.
  • The most substantial improvements arise from oracle-assisted evidence localization. But many times, we won't be getting such localisation due to platform settings, fragmented knowledge acquistion, partial observability issues or simply just dealing with noise in the system.
  • The executable answer-blind selector was shown to fail in reproducing the performance. Not sure why. I think the query construction may have biased selection toward the wrong evidence (it is possible that the selection query concatenates the original question with the verified first-hop text, and creating semantic similarity matches to the 1st-hop doc)
  • Additionally, verification is largely examined through controlled positive claim-status markers rather than realistic verification mechanisms. It remains unclear how the framework would translate into actionable evaluation or decision controls under heterogeneous retrieval conditions.
Questions for authors
  • On scientific novelty. As mentioned, beyond the established theoretical principles underpinning the formulation, what additional scientific insight does the proposed decomposition provide that otherwise could not be obtained through empirical evidence?

  • Operationalization. How do the authors see this being applied? How does this look like for real-world RAG pipelines and how one would apply a decision control plane with selection, verification and utilisation? Do we apply this for every single query retrieval?

  • Evals: On the related note to operationalization, given that HotpotQA and MuSiQue were primarily developed for multi-hop question answering, how representative are the 200 examples of contemporary RAG verification and recovery challenges? Would the attribution findings hold on more realistic retrieval benchmarks such as MultiHop-RAG (COLM 2024), CRAG (2024), or EnterpriseRAG-Bench (Sun et al., 2026), particularly under heterogeneous enterprise documents, fragmented or conflicting evidence? Can we demonstrate outcomes on more complex cross-document dependencies?

  • Generalizability: Given that the strongest recovery improvements depend on oracle-selected quotations, while the executable answer-blind selector does not preserve these gains, I wonder whether the main attribution conclusions remain valid under imperfect evidence selection? As mentioned, i think it's important to evaluate under heterogeneous document settings and varying verification outcomes to ascertain this.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kcJG

We thank Reviewer kcJG for the generous assessment of our statistical decision-theoretic perspective ("a more disciplined way of examination and observation, which I think should be a baseline for all technical papers"). We revised Sections 1, 4, 6.2, and 7 and address all four questions below.

1. Scientific Insight Beyond Empirical Evidence (W1, Q1)

While channel ordering builds on Blackwell (1953) and Cover & Thomas (2006) (Sec. 4 & App. A.3), our contribution is proving what interface-level attribution reveals about RAG selection and verification that empirical deltas (ΔRg\Delta R_g) cannot identify:

  1. Analytical Non-Identifiability (App. A.3): Two laws can share identical ΔRg=1/2\Delta R_g = 1/2 from D to D-U under opposite allocations: Family I has (AS=1/2, Hd−Hu=0)(A_S = 1/2,\ H_d - H_u = 0) (pure oracle localization) and Family II has (AS=0, Hd−Hu=1/2)(A_S = 0,\ H_d - H_u = 1/2) (pure utilization recovery on information already unambiguous in ZZ), proving empirical ablations alone are under-identified.
  2. Resolving the 27.5-Point Divergence (B1 -> C1 +28.5/+34.0 pp vs. D-U -> D-V +1.0 pp) (Prop. 2 & App. A.2): Prop. 2 (Eq. 3) proves that on a pre-isolated slot (D-U -> D-V), toggling UNVERIFIED <-> VERIFIED is an invertible rewrite (Xv=h(Xu),Xu=k(Xv)X_v = h(X_u), X_u = k(X_v)), so R∗(Xu)=R∗(Xv)R^*(X_u) = R^*(X_v) (zero channel information) and tests only utilization (Eu−EvE_u - E_v). In B1 -> C1 (App. A.2 & Eq. 4 in App. K), marking one of two symmetric competing claims (e0+e1≥ce_0 + e_1 \ge c) breaks epistemic symmetry (R∗(XC1)<R∗(XB1)R^*(X_{C1})\lt R^*(X_{B1})).
  3. In-Context Localization (Prop. 1, Eq. 2): Proves why copying a visible substring from ZZ (D -> D-U: +15.5+15.5 pp) can transmit target information AS>0A_S\gt 0 via index choice SoS_o alongside utilization gain Hd−HuH_d - H_u, and decomposes the blind-to-oracle gap into Rg(b)−Rg(u)=AS+(Hb−Hu)R_g(b)-R_g(u) = A_S + (H_b - H_u).

2. Operationalizing a Decision Control Plane in Real-World RAG (W2, Q2)

We do not run multi-arm evaluation or verification on every query at runtime (Section 7):

  • Offline Audit (Table 2): On a dev slice, D, D-B, D-U, D-V separate recall (9/2009/200 missing in D), selector localization (D-B vs. D-U: −18.5-18.5 pp), utilization (D vs. D-U: +15.5+15.5 pp), and conflict (B1 vs. C1: +20.0+20.0 to +34.0+34.0 pp).
  • Runtime 3-Tier Control Plane (Selective Routing per Query):
    1. Tier 1 (Always-on, ~7.8 ms/query GPU, App. D): Bridge-masked selection—mask resolved 1st-hop claims from the selector query and extract spans conditioned on the unresolved bridge entity.
    2. Tier 2 (Routing gate): Conflict check—when extracted candidate values agree (B0), verification adds +2.0+2.0 to +6.5+6.5 pp EM (Table 7) and review/gating hurts coverage (Table 32); answer directly.
    3. Tier 3 (ONLY under conflict B1): Under conflict (B1), unverified review (106→98106 \to 98, 102→53102 \to 53) and gating (SC-completed: 9.09.0\\%/7.5\\% coverage, 4/54/5 matches vs. 10.04/9.3910.04/9.39 random expectation, Table 12) fail due to epistemic symmetry (Eq. 4). Only on conflicted queries does the plane invoke an external verifier (B1 -> C1, +20.0+20.0 to +34.0+34.0 pp EM; cutting positional flips from 16/4016/40 to 4/404/40 and 1/401/40, Table 10).

3. Imperfect Selection & Confirming Your Query Hypothesis (W2, Q4)

Your hypothesis—that concatenating the question with verified 1st-hop text biased similarity toward the 1st-hop doc—is spot-on and confirmed in Sec. 6.2 & App. E (Tables 25–30):

  • Validating your diagnosis: Concatenating the verified 1st-hop claim FF causes D-B to select target_first in 159/200 (79.5%) groups (median 1st-hop overlap 0.800 vs. 0.333 on target_second, Table 28), where EM drops from D's 81/15981/159 (50.950.9\\%) to 77/15977/159 (48.448.4\\%) vs. D-U's 105/159105/159 (66.066.0\\%), causing 30/39 D-U -> D-B EM losses (76.9%)! On the 28/200 groups where D-B selects target_second, it achieves 15/28 (53.6%) EM, preserving 78.9% (15/19) of D-U's matches.
  • Imperfect retrieval: dense_query -> dense_bridge (Tables 7, 11, recall 121→192/200121 \to 192/200, poison 159→194/200159 \to 194/200) and Stage S3 (App. J.2) confirm localization (+17.0/+14.5 pp) and conflict disambiguation (B1 -> C1) remain the two distinct bottlenecks.

4. Representativeness & Extension to MultiHop-RAG, CRAG, EnterpriseRAG (Q3)

Two-hop chains form the minimal inductive unit of cross-document dependency (F1→F2F_1 \to F_2), evaluated across GraphRAG (LightRAG Stage S1, App. I–J), claim packs (Stage S3 & A0–C1), and multi-doc prefixes (D/D-B/D-U/D-V). In Section 7, we connect our decomposition to MultiHop-RAG (Tang & Yang, 2024), CRAG (Yang et al., 2024), and enterprise RAG: longer docs increase native loss HdH_d and query anchoring Hb−HuH_b - H_u (amplifying D -> D-U), while conflicting document versions map onto B1 -> C1.

Reviewer_kcJGOpenReview ↗

Thanks for the robust side expt to expand on some of the insights

  • Regarding1st-hop selection bias: Identifying and uncovering the failure is the first step. Proposing masking to improve selection sounds plausible - Why was this not experimentally verified?
  • I find the interpretation of the 78.9% oracle performance retention somewhat misleading - This was derived from only 28/200 cases where the automated selector already chose the correct 2nd hop document. Isn't this therefore a favourable subset? How do we disentangle this with the selector's overall effectiveness?
  • Additionally, of the 28 cases, only 15 achieved correct answers. This is same to the original system without any intervention. How do you account for any net improvement in the recovery process?
  • Regarding operationalising the Decision Control Plane: the 3-stage approach remains a proposal. I don't see any experimental validation that supports the soundness of it.
  • Tier-1 is a reasonable approach for dealing with 1st hop bias - but no empirical results demonstrates that it actually improves evidence selection or recovery.
  • Tier 2 assumes that the absence of conflicting claims - This sounds questionable, particularly in poisoned or unreliable KBs where multiple sources may agree on the same incorrect info. How reliable is this decision routing?
  • Tier 3 assumes there is accurate external verification - Can the verifier actually produce this outcome?

This will require end-to-end validation to substantiate. Otherwise, it remains a potential hypothesis at this juncture. One additional side comment, try to tackle the problems and highlight the intuition instead of dumping numbers - It is difficult to process and read without having to go back and forth the appendices and tallying the info.

Reviewer_EJ59OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the factors influencing answer recovery in RAG systems after knowledge poisoning. The authors propose a Resource Attribution Framework to distinguish the contributions of evidence selection, fact verification, and evidence utilization to answer recovery. Through experiments on HotpotQA and MuSiQue, the authors find that selecting and re-exposing correct evidence from the existing retrieved context can significantly improve answer accuracy, while adding verification markers provides relatively limited additional improvements.

Critical strengths

The paper focuses on answer recovery in RAG systems after knowledge poisoning and attempts to distinguish the contributions of evidence selection, verification, and utilization to performance improvements, providing a new perspective for understanding recovery mechanisms in RAG systems.

Critical weaknesses

1.What are the specific benefits of the theoretical framework proposed in Section 4? It seems that attributing RAG performance improvements could also be achieved through conventional ablation studies and controlled evidence selection.

2.The authors propose a resource attribution framework rather than an actual evidence selection algorithm. Beyond demonstrating the performance gap between Oracle selection and automatic selection, what additional practical value can this framework provide for real-world RAG systems?

3.The D → D-U comparison simultaneously introduces Oracle selection and evidence re-exposure, making it difficult to distinguish the individual effects of correct evidence selection, repeated presentation, and positional changes.

4.The writing throughout the paper is overly complicated and difficult to follow, making it challenging to clearly understand the authors' motivation.

Questions for authors

1.Could the main conclusions of the paper also be obtained through conventional ablation studies that control evidence selection, content, and verification status?

2.Can the authors further validate the practical value of the proposed framework in real-world RAG systems without Oracle access?

3.Can the authors include additional matched control experiments involving random evidence selection, different evidence positions, and repeated presentation to further distinguish the contributions of Oracle selection and evidence presentation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer EJ59

We thank Reviewer EJ59 for recognizing our perspective on decomposing RAG answer recovery after knowledge poisoning into evidence selection, verification, and utilization. We substantially revised Sections 1, 4, 6.2, and 7 to make the writing clear and self-contained and to address all questions.

1. Specific Benefits of the Section 4 Framework Beyond Conventional Ablations (W1, Q1)

While empirical ablations are essential, Section 4 (and Appendix A.1–A.3) provides three diagnostic insights and formal guarantees that conventional ablations cannot obtain on their own:

  1. Why B1 -> C1 (+28.5/+34.0 pp EM) and D-U -> D-V (+1.0 pp EM) differ by 27.5 points despite both "holding text fixed and adding a verification flag": Proposition 2 (Eq. 3) proves why:
    • In D-U -> D-V, the quotation is already isolated in a dedicated slot, so toggling UNVERIFIED <-> VERIFIED is an invertible rewrite (Xv=h(Xu),Xu=k(Xv)X_v = h(X_u), X_u = k(X_v)). Thus R∗(Xu)=R∗(Xv)R^*(X_u) = R^*(X_v) (zero channel information), testing only generator utilization (Eu−EvE_u - E_v).
    • Conversely, in B1 -> C1 (Appendix A.2 & Eq. 4 in Appendix K), two competing claims are epistemically symmetric (e0+e1≥ce_0 + e_1 \ge c). Marking the true claim breaks symmetry because stripping the marker loses the oracle's disambiguation (R∗(XC1)<R∗(XB1)R^*(X_{C1})\lt R^*(X_{B1})).
  2. Why D -> D-U (+15.5 pp) is NOT merely prompt formatting (Proposition 1, Eq. 2): Prop. 1 (Rg(d)−Rg(u)=AS+Hd−HuR_g(d)-R_g(u) = A_S + H_d - H_u) and Location-vs.-Verdict show that choosing which span to copy using OO can transmit up to 1 bit of target information (AS>0A_S\gt 0) alongside utilization reduction (Hd−HuH_d - H_u). Appendix A.3 proves two laws can share identical ΔRg=1/2\Delta R_g = 1/2 under (AS,Hd−Hu)=(1/2,0)(A_S, H_d-H_u) = (1/2, 0) vs. (0,1/2)(0, 1/2), so empirical ablations alone are under-identified.
  3. Decomposing blind-selector failure (Eq. 2): Rg(b)−Rg(u)=AS+(Hb−Hu)R_g(b)-R_g(u) = A_S + (H_b - H_u) shows D-B's −18.5-18.5 pp gap to D-U combines ASA_S with selector-induced utilization loss Hb−HuH_b - H_u.

2. Practical Value for Real-World RAG Systems Without Oracle Access (W2, Q2)

In Section 7, we detail how our framework delivers direct operational value without requiring an oracle at deployment:

  1. Preventing misallocated verification budgets: Standard evaluation (D -> D-V: +16.5+16.5 pp EM) leads practitioners to run costly verification across all retrieved docs, whereas our framework proves 94% of the EM recovery (+15.5+15.5 pp in D -> D-U) comes from localizing and re-exposing the already-retrieved 2nd-hop span.
  2. Diagnosing and fixing blind selector failures (Appendix E, Tables 25–30): Concatenating the verified 1st-hop claim into the query causes D-B to select the already-resolved 1st-hop doc (target_first) in 159/200 (79.5%) queries (median 1st-hop overlap 0.80 vs. 0.33 on the successor doc, Table 28), causing 30/39 match losses vs. D-U. On the 28/200 groups where D-B selects target_second, it achieves 15/28 (53.6%) EM, retaining 78.9% (15/19) of D-U's matches! Fix without an oracle: mask resolved 1st-hop claims from the selector query and extract spans conditioned on the unresolved bridge entity.
  3. A 3-Tier Runtime Control Plane: (i) Tier 1: bridge-masked span extraction; (ii) Tier 2: check if extracted candidate claims agree (B0, where verification adds +2.0+2.0 to +6.5+6.5 pp and review/gating hurts coverage, Table 32) or conflict (B1); (iii) Tier 3: trigger external verification only on conflicted candidates (B1 -> C1: +28.5/+34.0/+20.0 pp across Qwen3/Qwen3.8/Mistral), where unverified review (106→98106 \to 98, 102→53102 \to 53) and gating (9.09.0\\%/7.5\\% coverage, Table 12) fail.

3. Matched Controls for Repetition, Position, and Non-Oracle Selection (W3, Q3) & Writing (W4)

We rewrote Sections 1, 4, 6.2, and 7 for clarity and surface four matched controls from Appendices B & E:

  • 1. Repetition Only (Table 25): Across 172/200 groups where D-B repeats a visible non-successor span in the exact same terminal slot as D-U, EM is 80/172 (46.5%) vs. D's 86/172 (50.0%, -3.5 pp) and D-U's 116/172 (67.4%)—proving repetition gives zero benefit without localization.
  • 2. Blind Successor (Table 25): Across 28/200 groups where blind D-B selects target_second, EM is 15/28 (53.6%), retaining 78.9% (15/19) of D-U's matches.
  • 3. Position Swap (Table 10): Swapping claim order on 40 predetermined groups flips 16/40 (40%) in B1 (−10.0-10.0/−15.0-15.0 pp EM) vs. 4/40 & 1/40 in C1 (4×4\times–16×16\times reduction).
  • 4. Random Selection (Table 12): Under B1, analytical random retention (kM/NkM/N) expects 10.04 / 9.39 matches vs. SC-completed's 4 / 5 (9.09.0\\%/7.5\\% coverage).

Thank you to the authors for the detailed response, which addressed most of my concerns. However, I still have some doubts about the experimental attribution. In the D → D-U comparison, Oracle selection and evidence re-exposure are introduced together. The repetition-only control uses different evidence spans rather than keeping the content fixed, and the position-swap experiment is conducted in a different setting. Therefore, it is still unclear how much of the improvement comes from correct evidence selection versus repeated presentation. I have decided to keep my original score.

Thank you for clarifying this point. We agree that D → D-U does not tell us how much of the gain comes from selecting the relevant evidence and how much comes from showing it again. Our earlier statement that the blind control proves repetition alone has no benefit was too strong.

D-B and D-U use the same final slot, template, and UNVERIFIED label, but they can repeat different spans. The breakdown by document role (§6.2 and Table 25) shows what happened with this blind selector; it does not isolate the effect of repeating the oracle-selected quotation. The position-swap experiment was conducted in a different setting and does not resolve this question. As Appendix A.3 illustrates, the observed gain alone cannot separate these effects.

The result we can establish is narrower: exact match rose from 104/200 to 135/200 (+15.5 percentage points) when oracle selection and re-exposure were applied together, with the selected quotation labeled UNVERIFIED. In D-U → D-V, the quotation and its position are held fixed. Changing that label to VERIFIED increased exact match from 135/200 to 137/200 (+1.0 percentage point; 95% CI: −2.0 to +4.0).

Reviewer_vS4pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines why RAG answers improve after limited fact verification in a poisoned-evidence setting. It proposes a decision-theoretic resource attribution framework to distinguish evidence selection, verification messages, input presentation, and a generator’s use of these resources. Experiments on 200 question groups from HotpotQA and MuSiQue show that re-exposing an oracle-selected quotation improves EM from 104/200 to 135/200, while marking the same quotation as verified gives 137/200. A frozen answer-blind selector reaches only 98/200.

Critical strengths
  1. The D-U/D-V experiment keeps the selected quotation fixed and varies only its verification status. The small additional EM change is a useful counterpoint to attributing the entire recovery gain to verification.
  2. The paper distinguishes oracle localization, repeated presentation, and a fixed positive marker from an actual verifier with variable messages. These are important qualifications when interpreting RAG recovery results.
  3. The answer-blind selector and coverage-aware analyses help prevent overly strong conclusions from the oracle comparison. The paper also reports paired outcome changes and limitations rather than only aggregate EM.
Critical weaknesses
  1. The Introduction asks which dependent questions become answerable after one fact is verified, but the study mainly asks what causes an observed recovery gain. The connection between these questions is not established. More importantly, the paper never gives a convincing practical reason why this particular attribution problem needs to be solved, or identifies a concrete failure of existing evaluations that makes the problem urgent.
  2. The main findings seem obtainable through fairly standard matched ablations. The decomposition introduces ideal information advantages and utilization terms, but the central quantities are not estimated in the experiments. I would like to understand what this framework enables that a well-controlled experimental design would not, beyond a formal vocabulary for interpreting the comparisons.
  3. The opening moves abruptly from the states of successor evidence to sources of recovery gain, without explaining the link. The framework is then introduced before its purpose is clear. The three contributions also mix theoretical claims, unexplained result counts, and answer-selection analysis; terms such as privileged selection and variable verification messages appear before readers have the context needed to follow them. I had trouble identifying the paper’s central claim from the Introduction alone.
  4. The largest retrieval gain (D→D-U) combines oracle localization with re-exposing the quotation, so those two effects remain entangled. D-U→D-V tests a fixed positive label after oracle selection, not the benefit of a realistic verifier. The retrieval follow-ups use one generator and 200 groups; the answer-blind test covers one selector configuration. These limitations make it hard to judge how broadly the findings apply.
Questions for authors
  1. Can you point to a specific prior RAG verification/recovery evaluation whose conclusion would change under your framework? What would practitioners do differently as a result?
  2. What concrete insight or test does the risk decomposition provide beyond the matched comparisons themselves, given that the ideal information-advantage terms are not empirically estimated?
  3. Could you isolate oracle localization from merely repeating an already-visible span, or show whether an answer-blind selector can recover more of the oracle gain? This would materially affect the interpretation and usefulness of the main result.
2 comments
AuthorsOpenReview ↗

Response to Reviewer vS4p

We thank Reviewer vS4p for recognizing our D-U/D-V design (S1), our separation of localization, repetition, and verifier messages (S2), and our coverage-aware and answer-blind analyses (S3). We substantially revised Sections 1, 4, 6.2, and 7 to address every comment.

1. Rewritten Introduction, Central Claim, and Reversed Conclusions (W1, W3, Q1)

We rewrote Section 1 so the motivation, central claim, and intuition appear upfront before technical notation:

  • Multi-hop recovery attribution gap (W1, W3): When a poisoned RAG pipeline verifies a 1st-hop fact F1F_1 (a work's author), answering a dependent question (that author's birthplace) still requires an unverified successor fact F2F_2. Recovery gains can thus stem from (i) adding missing F2F_2 text (A -> B), (ii) localizing and re-exposing F2F_2 from noisy retrieved text (D -> D-U), (iii) marking F2F_2 verified (B -> C, D-U -> D-V), or (iv) altering coverage (Review/SC Gate).
  • Central claim (W3): End-to-end RAG recovery evaluations systematically conflate evidence localization and presentation with verification status, and conflate oracle-assisted span selection with deployable answer-blind selection.
  • Prior evaluations whose conclusions change (Q1, Sec. 7):
    1. Post-cleaning verification (RAGForensics [Zhang et al., 2025], HoG-GRAG [Noughabi et al., 2026], D -> D-V in Table 3): Comparing D (104/200104/200 EM) to D-V (137/200137/200 EM) credits +16.5 pp to verification. Inserting D-U (same span/slot, marked UNVERIFIED) reverses this: +15.5 pp (94% of EM gain, 104→135104 \to 135) comes from localizing and re-exposing a visible span, leaving +1.0 pp EM ([−2.0,+4.0][-2.0, +4.0]) for the marker (D-U -> D-V, +2.2+2.2 pp F1, poison 24→1624 \to 16).
    2. GraphRAG support injection (Stage S1, App. I–J, Fig. 6 & Tables 38–39): Comparing cleaning (12/2412/24 Qwen, 7/247/24 Mistral) to 1st-hop verification R1 (13/2413/24, 5/245/24) and full trusted docs R2 (19/2419/24, 10/2410/24) naively credits R2 to trust propagation, whereas R2 injects missing successor text (A -> B: +51.0+51.0 to +83.0+83.0 pp, Table 7).
    3. Oracle vs. blind compression (RECOMP [Xu et al., 2023]): Outcome-guided oracle selection (D-U: 135/200135/200) overstates deployable recovery when a blind selector (D-B: 98/20098/200) is anchored to 1st-hop terms (Table 25).
  • What practitioners do differently (Q1, Sec. 7): Deploy a 3-tier control plane: (1) when recall is high (191/200191/200 in D), use bridge-masked 2nd-hop span localization (D -> D-U); (2) route by whether extracted candidates agree (B0) or conflict (B1); and (3) trigger external verification selectively on claim conflicts (B1 -> C1: +20.0+20.0 to +34.0+34.0 pp EM across 3 models), where unverified review (106→98106 \to 98, 102→53102 \to 53) and sufficiency gating (9.09.0\\%/7.5\\% coverage, Tables 12, 32) fail.

2. Insights from Risk Decomposition Beyond Matched Ablations (W2, Q2)

In Section 4, we clarify why AS,AMA_S, A_M are structural population terms (Appendix A.3 proves two laws can share identical ΔRg=1/2\Delta R_g = 1/2 under (AS,Hd−Hu)=(1/2,0)(A_S, H_d-H_u) = (1/2,0) vs. (0,1/2)(0,1/2)) and provide two core insights beyond empirical ablations:

  1. Resolving the 27.5-point paradox (B1 -> C1 +28.5/+34.0 pp vs. D-U -> D-V +1.0 pp): Prop. 2 (Eq. 3) proves toggling a fixed marker on a pre-isolated slot (D-U -> D-V) is an invertible rewrite (R∗(Xu)=R∗(Xv)R^*(X_u)=R^*(X_v), zero channel information), testing only generator utilization (Eu−EvE_u - E_v). Conversely, marking one of two competing claims in B1 -> C1 (Appendix A.2 & Appendix K Eq. 4) breaks epistemic symmetry (R∗(XC1)<R∗(XB1)R^*(X_{C1})\lt R^*(X_{B1})).
  2. In-context selection vs. formatting (Prop. 1, Eq. 2): Rg(d)−Rg(u)=AS+Hd−HuR_g(d)-R_g(u)=A_S+H_d-H_u shows choosing which span to copy using OO can carry up to 1 bit of target information (AS>0A_S\gt 0) alongside utilization gain (Hd−HuH_d - H_u), while Rg(b)−Rg(u)=AS+(Hb−Hu)R_g(b)-R_g(u) = A_S + (H_b - H_u) separates ASA_S from blind-selector utilization loss Hb−HuH_b - H_u.

3. Isolating Localization from Repetition & Blind Recovery on Successor Docs (W4, Q3)

D-B and D-U share the exact same terminal slot, template, and UNVERIFIED tag, repeating a visible span from ZZ (Appendix E, Tables 25, 28, surfaced in Sec. 6.2):

  • Repetition without localization (172/200 groups: first 159, poison 9, other 4): Repeating a visible non-successor span yields 80/172 (46.5%) EM vs. D's 86/172 (50.0%, -3.5 pp; 3G/9L) and D-U's 116/172 (67.4%)—proving terminal repetition alone gives zero benefit.
  • Blind successor selection (28/200 groups: target_second): Achieves 15/28 (53.6%) EM (matching D), retaining 78.9% (15/19) of D-U's matches (Table 28: D-B picked target_first in 159/200 groups due to 0.80 vs. 0.33 median 1st-hop overlap, causing 30/39 losses).

Thank you for the response. The examples help explain why recovery gains should not be attributed entirely to verification and how this distinction could affect evaluation and practice.

However, I remain unconvinced that the theoretical framework provides enough additional insight beyond well-controlled ablations, particularly since some of its central quantities are not estimated in the experiments. This leaves my concern about the connection between the theory and the empirical findings unresolved. I therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.