acceptodds
Under review as a conference paper at ICLR 2027

Learning to Compose Evidence with Counterfactual Con- tinuation Values

Abstract

Retrieval-augmented question answering selects passages that are consumed jointly, yet common selectors score passages independently or optimize hand-designed set proxies. We study counterfactual continuation-value learning for bounded evidence selection. Given a small candidate pool and a frozen reader, we evaluate every feasible passage subset, cache its an- swer utility, and derive action targets from the best utility reachable after adding each passage. A compact selector learns these targets by regression and composes a context sequentially, with an explicit stop action. On Hot- potQA, it improves over BGE Top-4 on two disjoint 64-question test sets across two training seeds. On a fresh 256-question sample, the budget-5 selector reaches 53.9 EM / 65.8 F1 versus 47.7 / 59.7 for BGE Top-4 and 50.8 / 62.7 for All-8. Without retraining, it transfers to two independent 2WikiMultiHopQA samples: 38.3 / 46.1 versus 31.6 / 40.3 on 256 ques- tions, and 34.9 / 41.8 versus 30.3 / 35.3 on a disjoint 768-question sample. On a matched 768-question 2Wiki pool, a GRPO refinement reaches 35.29 EM / 43.11 F1, exceeding E5 Top-5 by 2.21 / 4.59 points at similar token cost. Candidate pools retain annotated supports, so these results evaluate composition conditional on support availability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model Agents, Agent Memory, Memory Composition, Sequential Decision Making, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Gu2aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes counterfactual continuation-value supervision for sequential evidence composition in retrieval-augmented multi-hop QA. On HotpotQA and zero-shot 2WikiMultiHopQA, it outperforms BGE Top-4, full eight-passage context, and a MORSE-style proxy, showing learned evidence composition improves answer quality and efficiency.

Critical strengths

One critical strength of this paper is that it derives exact ADD/STOP targets from exhaustive frozen-reader evaluations over a small candidate pool. This captures higher-order evidence complementarity and avoids relying on handcrafted or proxy objectives, while requiring no reader calls during selection.

Critical weaknesses

One major issue is that the method is unscalable and evaluated under an unrealistic setup. Offline target construction requires 2^K reader evaluations per question (256 when K=8), which becomes intractable for larger candidate pools. Another issue is that the 2Wiki candidate pools are explicitly constructed to always contain annotated support passages, so the evaluation only tests composition given gold evidence already in the pool. This bypasses the real retrieval challenge in open-domain multi-hop QA. As a result, the reported gains over BGE Top-4 do not demonstrate end-to-end effectiveness in realistic settings where supports may not be retrieved.

Questions for authors

What is the scalability for the proposed method?

Reviewer_GGnNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a learning-based framework for evidence combination in Retrieval-Augmented Question Answering (RAG), addressing a limitation in prior methods that overlook the complementary effects of passages when ranking them independently. The authors introduce "Counterfactual Continuation Values" to estimate the maximum potential utility of an answer that could be achieved in subsequent stages after selecting a given candidate passage. By exhaustively evaluating subsets of evidence—with the reader model held "frozen"—the approach constructs a supervised "action-value" objective, which is then used to train a lightweight sequence selector; this selector incorporates explicit "ADD" and "STOP" operations.

Critical strengths
  1. Clear and Well-Motivated Problem Formulation.

  2. Interesting Counterfactual Value Learning Framework.

Critical weaknesses
  1. Limited evidence for the necessity of learning evidence composition. In the HotpotQA experiments, each candidate pool contains only two annotated supporting passages among eight candidates, while the proposed selector typically selects five passages. Therefore, the observed improvements may primarily result from filtering irrelevant passages rather than learning complex complementary relationships among evidence. Moreover, the paper lacks comparisons with simpler supervised passage selection methods trained directly on the available supporting-evidence annotations, as well as an oracle baseline using only gold supporting passages. These comparisons are important for establishing whether the proposed counterfactual continuation-value learning provides meaningful advantages over conventional evidence selection.

  2. Missing Comparison with Direct LLM-based Evidence Selection. The proposed method requires exhaustive offline evaluation of evidence subsets using Qwen3-8B to train an additional Qwen3-1.7B selector. However, the paper does not compare against directly prompting the same Qwen3-8B reader to select relevant and complementary passages before answering. Given the small candidate pool of only eight passages, such a straightforward approach could potentially achieve comparable performance without additional training or expensive offline supervision construction.

Questions for authors

See Weakness

Reviewer_BFKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates evidence set selection in RAG with a limited context budget. This work considers all possible subsets of passages from a small candidate pool, freezes the Reader model, and calculates their utility according to answer correctness, support recall, and a penalty for length. The highest possible utility that can be achieved after adding a passage is taken as the target continuation value. In addition, the Qwen3-1.7B Selector is trained to predict the ADD/STOP action values and perform evidence selection sequentially. Experimental results on HotpotQA and 2WikiMultiHopQA report improvements over Top-K retrieval and a MORSE-style subset-selection baseline.

Critical strengths
  1. Exhaustive subset evaluation gives us a straightforward method for creating supervision targets. Given a small number of possible candidates, the optimal continuation value of each action can be calculated based on the stored results of the Reader without estimating its value through sampling. The Reader is not invoked when selecting online evidence.
  2. The experiments cover same-budget comparisons and cross-dataset evaluation. As one can see in Table 6, the Supervised Composer gets 53.91 EM / 65.81 F1 on HotpotQA, whereas BGE Top-5 manages only 50.00 / 60.56. Table 5 also demonstrates gains over BGE Top-4 on two independent 2WikiMultiHopQA samples without retraining the Selector on 2Wiki.
Critical weaknesses
  1. The reasoning behind continuation-value supervision is not adequately justified. First of all, the key idea is that immediate utility cannot appropriately reward complementary evidence, but there is no empirical investigation of how frequently and how severely this issue appears in the considered tasks. Given that subset utilities have already been computed for the training questions, it should be straightforward to make the comparison between the exact immediate-greedy strategy and the optimal continuation policy, as well as to see when the latter is really required. In Appendix C, there is just one artificial example. Furthermore, the optimal continuation value does not directly reflect the importance of a particular passage itself, but rather it checks whether the maximal possible utility still can be achieved if the passage is chosen. When the budget is relatively high, even an irrelevant passage may have a high target since its presence does not prevent the optimal utility from being achieved. For instance, HotpotQA candidate pools have two annotated supporting passages while the budget-5 setting allows five passages.
  2. The principal ablation shows no clear benefit for continuation-value supervision. The performance of continuation value versus immediate utility is evaluated in Table 7, with all other training parameters kept constant across both sets of experiments. For HotpotQA, continuation value increases EM scores by 1.95 points while reducing F1 scores by 0.43 points. For 2Wiki, the benefits are less dramatic, with EM scores being increased by 1.04 points and F1 scores being increased by 1.08 points. All reported 95% confidence intervals contain 0, and the comparison is done using just one training seed. The presented numbers indicate no clear advantage of continuation-value targets over immediate-utility targets. Moreover, there is no analysis of the quality of Q-value prediction or of the gap between the learned and optimal policies.
  3. It is still not clear what the source of these improvements is. Equation (1) for utility contains gold support recall, and the candidate pools for both training and testing contain annotated supporting passages along with distractors. Thus, the Selector may mostly learn to detect the supporting passages rather than the complementarity of the evidence. There is no analysis where gold support recall is dropped, or a comparison with a Selector trained on supporting-passage labels using the same architecture. There is no breakdown of the experimental results based on evidence complementarity either.
  4. The separation from prior art and the baseline comparisons are not adequate. SetR [1] already looks into learned evidence set selection, and both SCARLet [2] and Influence Guided Context Selection [3] leverage downstream Generator feedback for supervision in evidence selection. The distinguishing feature in the current paper is the employment of state-conditioned optimal continuation value supervision, but the advantage over existing attribution-based supervision is not explored. Most of the baselines are conventional retrievers like BM25, BGE, and E5, and there is no comparison against closely related learned evidence selection approaches. Furthermore, Table 6 compares against an in-house MORSE-style proxy rather than the full MORSE approach [4].
  5. The computational complexity and scalability have not been properly analyzed. In the training process, there are (2^K) evaluations of the Reader for each training instance, where (K=8) in all experimental settings. During inference, the budget-5 Selector requires up to 35 action scores from Qwen3-1.7B. However, the reported token costs do not include the computational cost of the Selector. As such, the Reader input-token analysis does not measure the efficiency of the technique.
Questions for authors
  1. By using the cached subset utilities, what is the difference in performance between exact immediate-greedy selection and the exact continuation-value strategy? What is the rate of occurrence of substantial evidence complementarity or greedy-selection failures for the tested tasks?
  2. What would happen to performance if the gold support recall bonus term was set to zero by setting λe=0\lambda_e=0 in Equation (1)? How would the technique fare compared to a Selector model trained directly on supporting-passage labels with the same architecture?
  3. Is it possible to replicate the ablation on target choices in Table 7 using additional seeds during training? Why are the Supervised Composer scores in Tables 4/5 and Table 7 different? Are there any different checkpoints or evaluation procedures used?
  4. What is the end-to-end inference time if all Selector action scoring is considered? How would it compare with learned re-ranking or set selection baselines with the same passage budget?

[1] Lee et al. Shifting from Ranking to Set Selection for Retrieval Augmented Generation. ACL, 2025. [2] Xu et al. Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models. EMNLP, 2025. [3] Deng et al. Influence Guided Context Selection for Effective Retrieval-Augmented Generation. NeurIPS, 2025. [4] Ye et al. Multi-objective Evidence Construction in Retrieval-enhanced Question Answering: From Ranking to Subset Optimization. Complex & Intelligent Systems, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.