acceptodds
Under review as a conference paper at ICLR 2027

Beyond Relevance: State-Conditioned Consequence Modeling for Adaptive Retrieval-Augmented Generation

Abstract

The value of retrieved evidence is not fixed: the same candidate can have different consequences depending on what has already been retrieved. Yet sequential retrieval-augmented generation (RAG) commonly evaluates the next candidate without explicitly modeling this dependence on the accumulated retrieval state. In a canonical singleton-generation audit on HotpotQA and 2WikiMultiHopQA, 33.9% of repeated query–candidate groups change consequence category across retrieval states, confirming that this phenomenon persists after controlling for batch-composition effects. Motivated by this observation, we formulate next-evidence selection as state-conditioned consequence prediction and introduce the State-Consequence Model (SCM). SCM learns from offline counterfactual supervision produced by a frozen generator to predict the benefit and harm of adding each remaining candidate, while an independent answer verifier determines when retrieval should stop. A strictly matched state-agnostic ablation degrades both consequence prediction and held-out answer quality without reducing retrieval cost, directly showing that the accumulated state provides useful information for retrieval decisions. Across adaptive RAG evaluations, SCM achieves strong answer quality with efficient retrieval and remains competitive under stricter relevance-matched controls. Together, these results establish retrieval state as a central variable in predicting the consequence of the next evidence addition and motivate consequence-aware evidence selection for sequential RAG.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Large Language Model, Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uuW6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that in sequential RAG, the value of a candidate document depends on the accumulated retrieval state: the same passage can be helpful, harmful, or neutral depending on what has already been retrieved. The authors conduct an audit showing that 33.9% of repeated query–candidate pairs change consequence category across retrieval states. Motivated by this, they propose SCM, which learns from offline counterfactual supervision produced by a frozen generator to predict the benefit and harm of adding each remaining candidate to the current state. The authors evaluate SCM against relevance-matched controls and adaptive-RAG baselines in a closed-pool setting with dataset-provided 10-document context pools.

Critical strengths

S1. The strictly matched state-agnostic ablation, practical state-aware relevance control, and strict objective-matched relevance control share identical architectures, training data, update budgets, verifiers, policy families, and calibration protocols. This disciplined matching isolates the effect of supervision semantics from confounding design differences.

S2. The paper offers a crisp conceptual distinction between static relevance R(q, c) and state-conditioned marginal consequence ∆V(q, S_k, c). Formulating next-evidence selection around predicted insertion consequences rather than isolated document relevance is a theoretically appealing departure from standard reranking.

S3. Results are accompanied by paired bootstrap confidence intervals and sign-flip p-values, lending the empirical claims a degree of statistical rigor that exceeds typical RAG papers. The reproducibility checklist and frozen configuration trail further support transparency.

Critical weaknesses

W1. The central claim that consequence modeling improves adaptive RAG collapses under strict matching. When compared against the strict objective-matched relevance control, SCM achieves EM .7240 versus relevance's .7200 (p = .358), none of these differences are statistically significant (Table 17). The only significant isolated effect is a marginal reduction in retrieval depth (≈0.009–0.088 fewer documents). For a method requiring expensive offline counterfactual labeling of 820k+ examples with dual generator calls per row, this marginal efficiency gain is empirically insufficient to justify the architectural and computational overhead.

W2. The evaluation is confined to two multi-hop QA datasets using a single generator, a fixed 10-document closed pool, and maximum depth of five. This closed-pool design deliberately removes corpus-coverage challenges that define real-world RAG. There is no validation on open-domain retrieval, diverse task types (e.g., summarization, fact verification), or alternative generators.

W3. SCM requires 820,134 rows of counterfactual supervision, each demanding two generator calls to produce ∆V labels. This is computationally extravagant and scales poorly with pool size, sequence depth, or question volume.

W4. The state-dependence phenomenon is substantially weaker than presented. At the observation level, 94.21% of stored and canonical labels agree, and only 0.14% of observations switch directly between helpful and harmful. This means the vast majority of "state dependence" involves minor shifts between a directional label and the stable category, not the consequential reversals implied by the paper's framing. The empirical motivation for SCM is therefore far weaker than the abstract and introduction suggest.

Questions for authors

see weakness

Reviewer_HUxLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the usefulness of retrieved evidence changes as a RAG system accumulates context. The proposed State-Consequence Model (SCM) learns to predict the benefit and harm of adding a candidate passage, using counterfactual supervision from a frozen generator. An independent answer verifier is used alongside the predicted benefit to decide when retrieval should stop.

The authors evaluate the approach on HotpotQA and 2WikiMultiHopQA. They report a canonical audit showing that the same query–candidate pair can have different consequences across retrieval states, followed by state-conditioning ablations and comparisons with relevance-based and adaptive-RAG baselines.

Critical strengths

I found the canonical same-(q, c) audit to be the most convincing part of the paper. The authors recognize that the original batched FP16 labeling procedure could affect the measured consequence changes, and they repeat the analysis using singleton generation. The change rate falls to 33.9%, but the phenomenon remains substantial. The additional EM-only analysis is useful because it shows that the observation does not depend entirely on the chosen VALUE threshold.

The experimental evaluation also goes beyond comparing SCM with standard RAG baselines. The matched w/o-state ablation, in particular, reports a 1.55-point EM improvement and a 1.60-point F1 improvement at essentially unchanged retrieval depth. The use of question-disjoint splits, independent stopping calibration, and paired statistical tests makes these results more informative. I also appreciate the inclusion of a strict objective-matched relevance baseline, even though it makes the claimed methodological advantage less convincing.

Critical weaknesses

The paper motivates SCM as learning the consequences of evidence insertion rather than conventional relevance. However, when benefit-only consequence learning is compared with relevance learning under a matched ranking objective, neither EM nor F1 improves significantly. The main measurable advantage is a reduction of 0.088 retrieved documents per question. This is statistically significant, but I am not sure it is sufficient to justify the additional complexity of constructing counterfactual supervision.

Full SCM achieves the highest numerical answer quality, with 72.40% EM and 78.73% F1, compared with 72.00% EM and 78.39% F1 for strict matched relevance. Neither difference is statistically significant, and document usage is almost identical. These results do not establish that the two approaches are equivalent, but they leave the independent contribution of consequence supervision unresolved.

The harm component also seems less important to the deployed system than the method description might suggest. Removing harm supervision does not significantly reduce final answer quality, and the stopping rule uses benefit and verifier confidence without directly incorporating predicted harm. In contrast, removing the verifier reduces EM by 5.30 points. I would therefore distinguish more sharply between the demonstrated value of state-aware candidate scoring and answer verification, and the less clearly established value of the complete benefit–harm consequence formulation.

I have a separate reservation about the interpretation of the state-conditioning ablation. Appendix B.1 explains that the candidate models start from a shallow retrieved-state precursor already trained with state-conditioned consequence supervision. The w/o-state model then removes the accumulated state from its input during continuation training. Although the continuation settings are matched, the comparison may disadvantage the state-agnostic model because its initialization was learned with a different input structure. Training both variants independently from the original BGE-M3 initialization would be a cleaner test. I do not think this invalidates the reported result, but it limits how confidently I would attribute the entire improvement to explicit state conditioning.

There is also a simpler explanation for at least part of the measured state dependence that deserves attention. The accumulated state changes retrieval depth and context length, and the consequence target depends on the quality of the answer before insertion. An already correct answer, for example, cannot improve further under the defined VALUE function. The 33.9% change rate therefore establishes that consequences vary across observed states, but it does not tell us how much of that variation comes from interactions with the actual evidence content. A depth-controlled analysis would make this result more informative.

Finally, the evaluation remains limited to two multi-hop QA datasets, closed candidate pools, and one frozen generator. These are reasonable choices for controlled experiments, but they restrict what can be concluded about the usefulness of the approach in broader retrieval settings.

Questions for authors

The state-conditioning result is important to my assessment. Have the authors examined whether the advantage persists when both the state-aware and state-agnostic models are trained independently from the same original BGE-M3 initialization? Even a smaller-scale comparison would help address the possible precursor mismatch.

I would also be interested in a breakdown of the canonical consequence changes at fixed retrieval depth. In particular, how frequently does the same candidate change consequence when the states have the same depth and the pre-insertion answers have the same correctness status?

Regarding efficiency, Appendix D.4 reports online latency for Full SCM but not for the two models in the strict relevance comparison. Could the authors report comparable latency measurements for strict matched relevance and benefit-only consequence learning? The statistically significant reduction in document count is interesting, but its practical value is difficult to judge without these measurements and an estimate of the offline labeling cost.

Reviewer_f8nzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sequential retrieval-augmented generation, where documents are added one at a time. It argues that the value of a retrieved document is not fixed and depends on what has already been retrieved. To test this, the authors audit HotpotQA and 2WikiMultiHopQA and find that 33.9% of repeated query and candidate groups change consequence category when only the retrieval state changes. Based on this observation, they formulate next-evidence selection as predicting the marginal consequence of adding each candidate to the current state, and propose the State-Consequence Model, SCM. SCM is a cross-encoder trained on about 820k offline counterfactual labels produced by a frozen Qwen2.5-7B generator. It predicts the benefit and harm of each remaining candidate, and an independent verifier estimates whether the current answer is already correct. A calibrated rule combines the two signals to decide whether to add the best candidate or stop. The method is evaluated on a 2,000-question held-out set against non-adaptive baselines, adaptive RAG methods including Adaptive-RAG, FLARE, Stop-RAG and SeaKR, and matched relevance controls. A strictly matched ablation shows that removing the state from the scorer lowers EM by 1.55 points without changing retrieval depth. Overall, SCM achieves the best answer quality, using 2.37 documents on average.

Critical strengths
  1. The paper moves adaptive RAG away from static relevance and toward the marginal consequence of inserting the next document under the current state. It also tests this idea directly instead of only assuming it. The same query and candidate audit shows that 33.9% of repeated groups change consequence category when only the retrieval state changes. Building on this, SCM learns the benefit and harm of each candidate from offline counterfactual supervision, so no candidate-wise generation is needed at inference. It also separates candidate scoring from an independent verifier that judges whether the current answer is already correct. This separation matters: removing the verifier lowers EM by 5.3 points. Overall, SCM improves EM by 11.8 points over vanilla RAG while using 2.37 documents on average instead of 5.

  2. The experiments are carefully controlled. Training, development, calibration, and held-out sets are split by question, and all model and threshold choices are frozen before held-out evaluation. The w/o-state ablation and the strict objective-matched relevance control change only one factor while keeping data, objective, update budget, verifier, and calibration the same. All comparisons use paired bootstrap and sign-flip tests. The authors also report unfavorable results openly, for example that consequence supervision is not significantly better than strict relevance in answer quality and that its clearest benefit is efficiency. This makes the claims easy to trust and to check.

Critical weaknesses
  1. The method is only worthwhile if the large offline labeling cost of about 820k rows, each needing two generator calls, is paid once and then reused. However, training and testing both use HotpotQA and 2WikiMultiHopQA, with a single generator and closed 10-document pools. The labels depend on the specific generator by design, and the stopping thresholds are fitted on in-distribution data.

  2. The central claim that retrieval state is a central variable is only weakly supported. Removing the state from the scorer reduces EM by only 1.55 points, and the lower bound of the confidence interval is 0.55. Moreover, under the strict objective-matched control, consequence supervision gives no significant gain over relevance, with only 0.40 EM and p=.36. In addition, all figures in the paper are embedded as raster images rather than vector graphics

Questions for authors
  1. Can you report leave-one-dataset-out results, training on HotpotQA and testing on 2Wiki and the reverse, as well as results when the existing SCM is applied to a different generator without relabeling

  2. The paper reports that 33.9% of query and candidate groups change consequence category across states, yet removing the state costs only 1.55 EM. Why is the end-to-end gap so small, and how does this support the claim that state is a central variable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.