- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that in sequential RAG, the value of a candidate document depends on the accumulated retrieval state: the same passage can be helpful, harmful, or neutral depending on what has already been retrieved. The authors conduct an audit showing that 33.9% of repeated query–candidate pairs change consequence category across retrieval states. Motivated by this, they propose SCM, which learns from offline counterfactual supervision produced by a frozen generator to predict the benefit and harm of adding each remaining candidate to the current state. The authors evaluate SCM against relevance-matched controls and adaptive-RAG baselines in a closed-pool setting with dataset-provided 10-document context pools.
Critical strengths
S1. The strictly matched state-agnostic ablation, practical state-aware relevance control, and strict objective-matched relevance control share identical architectures, training data, update budgets, verifiers, policy families, and calibration protocols. This disciplined matching isolates the effect of supervision semantics from confounding design differences.
S2. The paper offers a crisp conceptual distinction between static relevance R(q, c) and state-conditioned marginal consequence ∆V(q, S_k, c). Formulating next-evidence selection around predicted insertion consequences rather than isolated document relevance is a theoretically appealing departure from standard reranking.
S3. Results are accompanied by paired bootstrap confidence intervals and sign-flip p-values, lending the empirical claims a degree of statistical rigor that exceeds typical RAG papers. The reproducibility checklist and frozen configuration trail further support transparency.
Critical weaknesses
W1. The central claim that consequence modeling improves adaptive RAG collapses under strict matching. When compared against the strict objective-matched relevance control, SCM achieves EM .7240 versus relevance's .7200 (p = .358), none of these differences are statistically significant (Table 17). The only significant isolated effect is a marginal reduction in retrieval depth (≈0.009–0.088 fewer documents). For a method requiring expensive offline counterfactual labeling of 820k+ examples with dual generator calls per row, this marginal efficiency gain is empirically insufficient to justify the architectural and computational overhead.
W2. The evaluation is confined to two multi-hop QA datasets using a single generator, a fixed 10-document closed pool, and maximum depth of five. This closed-pool design deliberately removes corpus-coverage challenges that define real-world RAG. There is no validation on open-domain retrieval, diverse task types (e.g., summarization, fact verification), or alternative generators.
W3. SCM requires 820,134 rows of counterfactual supervision, each demanding two generator calls to produce ∆V labels. This is computationally extravagant and scales poorly with pool size, sequence depth, or question volume.
W4. The state-dependence phenomenon is substantially weaker than presented. At the observation level, 94.21% of stored and canonical labels agree, and only 0.14% of observations switch directly between helpful and harmful. This means the vast majority of "state dependence" involves minor shifts between a directional label and the stable category, not the consequential reversals implied by the paper's framing. The empirical motivation for SCM is therefore far weaker than the abstract and introduction suggest.
Questions for authors
see weakness