- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper proposes a resource attribution framework for understanding performance recovery under RAG settings following limited fact verification.
- Drawing on stats decision theory and info-channel comparisons, the authors distinguish the effects of evidence selection, verification, and generator utilization.
- The formulation separates ideal information advantages from rendering losses and generator excess risk, with matched experimental controls intended to isolate the contributions of different interventions.
- Experiments on two-hop question groups drawn from HotpotQA and MuSiQue looks as content availability, competing claims, verification status etc.
- The results suggest that substantial recovery improvements can arise from oracle-assisted evidence localization rather than verification itself.
- In the retrieved-document setting, re-exposing an oracle-selected quotation improves exact match from 104/200 to 135/200, while adding its verification marker yields a smaller improvement to 137/200.
- An executable answer-blind selector fails to reproduce the oracle-assisted gains. There might be limitations of attributing recovery improvements to verification alone.
Critical strengths
- Useful information-theoretic perspective on evidence attribution.
- I find the problem setup and methodology to be an interesting and useful perspective on how retrieval-related improvements should be interpreted.
- There are sufficient establishment on statistical decision theory to distinguish information advantages from rendering and generator-utilization effects.
- I think this provides a more disciplined way of examination and observation, which I think should be a baseline for all technical papers going up for submission.
- Careful experimental controls and interpretation.
- Experimentation set up is nice and I appreciate the authors' effort to isolate different intervention effects through matched comparisons.
- Particularly the distinction between oracle-assisted localization and verification-status changes is useful for understanding the contributing factors.
- The finding that re-exposing an already-visible quotation accounts for substantially more improvement than adding a verification marker is informative.
- Also, the authors are also reasonably transparent about the limitations of their oracle-assisted setting, and the distinction between reference recovery and justified abstention is valuable.
Critical weaknesses
- Limited scientific novelty and significance.
- While the resource attribution framework is a useful take on retrieval mechanisms, the underlying decomposition largely follows well-established principles of statistical decision theory and info channel comparisons. I fail to uncover anything significant or novel here apart from the validation that this is true.
- It remains unclear what fundamentally new scientific insight emerges beyond carefully controlled selection and verification experiments.
- Furthermore, the advantages introduced in the formulation are not empirically estimated -> limiting the extent to which the theoretical framework provides additional explanatory value.
- I'm therefore not entirely convinced that the contribution represents a sufficiently substantial methodological advance. Unless, I'm missing something important here.
- Unclear operationalization and generalizability?
- The framework provides a structured interpretation of recovery gains, but its applicability to real-world RAG systems remains uncertain.
- The most substantial improvements arise from oracle-assisted evidence localization. But many times, we won't be getting such localisation due to platform settings, fragmented knowledge acquistion, partial observability issues or simply just dealing with noise in the system.
- The executable answer-blind selector was shown to fail in reproducing the performance. Not sure why. I think the query construction may have biased selection toward the wrong evidence (it is possible that the selection query concatenates the original question with the verified first-hop text, and creating semantic similarity matches to the 1st-hop doc)
- Additionally, verification is largely examined through controlled positive claim-status markers rather than realistic verification mechanisms. It remains unclear how the framework would translate into actionable evaluation or decision controls under heterogeneous retrieval conditions.
Questions for authors
-
On scientific novelty. As mentioned, beyond the established theoretical principles underpinning the formulation, what additional scientific insight does the proposed decomposition provide that otherwise could not be obtained through empirical evidence?
-
Operationalization. How do the authors see this being applied? How does this look like for real-world RAG pipelines and how one would apply a decision control plane with selection, verification and utilisation? Do we apply this for every single query retrieval?
-
Evals: On the related note to operationalization, given that HotpotQA and MuSiQue were primarily developed for multi-hop question answering, how representative are the 200 examples of contemporary RAG verification and recovery challenges? Would the attribution findings hold on more realistic retrieval benchmarks such as MultiHop-RAG (COLM 2024), CRAG (2024), or EnterpriseRAG-Bench (Sun et al., 2026), particularly under heterogeneous enterprise documents, fragmented or conflicting evidence? Can we demonstrate outcomes on more complex cross-document dependencies?
-
Generalizability: Given that the strongest recovery improvements depend on oracle-selected quotations, while the executable answer-blind selector does not preserve these gains, I wonder whether the main attribution conclusions remain valid under imperfect evidence selection? As mentioned, i think it's important to evaluate under heterogeneous document settings and varying verification outcomes to ascertain this.
2 comments
Thanks for the robust side expt to expand on some of the insights
- Regarding1st-hop selection bias: Identifying and uncovering the failure is the first step. Proposing masking to improve selection sounds plausible - Why was this not experimentally verified?
- I find the interpretation of the 78.9% oracle performance retention somewhat misleading - This was derived from only 28/200 cases where the automated selector already chose the correct 2nd hop document. Isn't this therefore a favourable subset? How do we disentangle this with the selector's overall effectiveness?
- Additionally, of the 28 cases, only 15 achieved correct answers. This is same to the original system without any intervention. How do you account for any net improvement in the recovery process?
- Regarding operationalising the Decision Control Plane: the 3-stage approach remains a proposal. I don't see any experimental validation that supports the soundness of it.
- Tier-1 is a reasonable approach for dealing with 1st hop bias - but no empirical results demonstrates that it actually improves evidence selection or recovery.
- Tier 2 assumes that the absence of conflicting claims - This sounds questionable, particularly in poisoned or unreliable KBs where multiple sources may agree on the same incorrect info. How reliable is this decision routing?
- Tier 3 assumes there is accurate external verification - Can the verifier actually produce this outcome?
This will require end-to-end validation to substantiate. Otherwise, it remains a potential hypothesis at this juncture. One additional side comment, try to tackle the problems and highlight the intuition instead of dumping numbers - It is difficult to process and read without having to go back and forth the appendices and tallying the info.
Response to Reviewer kcJG
We thank Reviewer
kcJGfor the generous assessment of our statistical decision-theoretic perspective ("a more disciplined way of examination and observation, which I think should be a baseline for all technical papers"). We revised Sections 1, 4, 6.2, and 7 and address all four questions below.1. Scientific Insight Beyond Empirical Evidence (W1, Q1)
While channel ordering builds on Blackwell (1953) and Cover & Thomas (2006) (Sec. 4 & App. A.3), our contribution is proving what interface-level attribution reveals about RAG selection and verification that empirical deltas (ΔRg) cannot identify:
App. A.3): Two laws can share identical ΔRg=1/2 fromDtoD-Uunder opposite allocations: Family I has (AS=1/2, Hd−Hu=0) (pure oracle localization) and Family II has (AS=0, Hd−Hu=1/2) (pure utilization recovery on information already unambiguous in Z), proving empirical ablations alone are under-identified.B1 -> C1+28.5/+34.0 pp vs.D-U -> D-V+1.0 pp) (Prop. 2 & App. A.2): Prop. 2 (Eq. 3) proves that on a pre-isolated slot (D-U -> D-V), togglingUNVERIFIED <-> VERIFIEDis an invertible rewrite (Xv=h(Xu),Xu=k(Xv)), so R∗(Xu)=R∗(Xv) (zero channel information) and tests only utilization (Eu−Ev). InB1 -> C1(App. A.2 & Eq. 4 in App. K), marking one of two symmetric competing claims (e0+e1≥c) breaks epistemic symmetry (R∗(XC1)<R∗(XB1)).Prop. 1, Eq. 2): Proves why copying a visible substring from Z (D -> D-U: +15.5 pp) can transmit target information AS>0 via index choice So alongside utilization gain Hd−Hu, and decomposes the blind-to-oracle gap into Rg(b)−Rg(u)=AS+(Hb−Hu).2. Operationalizing a Decision Control Plane in Real-World RAG (W2, Q2)
We do not run multi-arm evaluation or verification on every query at runtime (Section 7):
Table 2): On a dev slice,D,D-B,D-U,D-Vseparate recall (9/200 missing inD), selector localization (D-Bvs.D-U: −18.5 pp), utilization (Dvs.D-U: +15.5 pp), and conflict (B1vs.C1: +20.0 to +34.0 pp).App. D): Bridge-masked selection—mask resolved 1st-hop claims from the selector query and extract spans conditioned on the unresolved bridge entity.B0), verification adds +2.0 to +6.5 pp EM (Table 7) and review/gating hurts coverage (Table 32); answer directly.B1): Under conflict (B1), unverified review (106→98, 102→53) and gating (SC-completed: 9.0 coverage, 4/5 matches vs. 10.04/9.39 random expectation,Table 12) fail due to epistemic symmetry (Eq. 4). Only on conflicted queries does the plane invoke an external verifier (B1 -> C1, +20.0 to +34.0 pp EM; cutting positional flips from 16/40 to 4/40 and 1/40,Table 10).3. Imperfect Selection & Confirming Your Query Hypothesis (W2, Q4)
Your hypothesis—that concatenating the question with verified 1st-hop text biased similarity toward the 1st-hop doc—is spot-on and confirmed in Sec. 6.2 & App. E (
Tables 25–30):D-Bto selecttarget_firstin 159/200 (79.5%) groups (median 1st-hop overlap 0.800 vs. 0.333 ontarget_second,Table 28), where EM drops fromD's 81/159 (50.9) to 77/159 (48.4) vs.D-U's 105/159 (66.0), causing 30/39D-U -> D-BEM losses (76.9%)! On the 28/200 groups whereD-Bselectstarget_second, it achieves 15/28 (53.6%) EM, preserving 78.9% (15/19) ofD-U's matches.dense_query -> dense_bridge(Tables 7, 11, recall 121→192/200, poison 159→194/200) andStage S3(App. J.2) confirm localization (+17.0/+14.5pp) and conflict disambiguation (B1 -> C1) remain the two distinct bottlenecks.4. Representativeness & Extension to MultiHop-RAG, CRAG, EnterpriseRAG (Q3)
Two-hop chains form the minimal inductive unit of cross-document dependency (F1→F2), evaluated across GraphRAG (
LightRAGStage S1,App. I–J), claim packs (Stage S3&A0–C1), and multi-doc prefixes (D/D-B/D-U/D-V). In Section 7, we connect our decomposition to MultiHop-RAG (Tang & Yang, 2024), CRAG (Yang et al., 2024), and enterprise RAG: longer docs increase native loss Hd and query anchoring Hb−Hu (amplifyingD -> D-U), while conflicting document versions map ontoB1 -> C1.