acceptodds
Under review as a conference paper at ICLR 2027

RIPPLE: Corpus-scale Retrieval Plan Optimisation via Iterative Failure-driven Supervision

Abstract

Retrieval-augmented generation (RAG) increasingly relies on iterative search when evidence cannot be recovered in a single retrieval pass. Existing iterative and RL-based search policies reformulate queries across search rounds, but fall short in certain aspects: the subqueries and documents considered per round bound how much of the corpus is ever brought into view, and although the planner adjusts its queries round to round, it does so mainly by reading its own accumulated search history within its context, with no trainable signal for how well a round resolved the information need, leaving little basis to learn which decisions closed the evidence gap. We introduce RIPPLE, which casts a wide initial retrieval net and compresses it with maximal marginal relevance (MMR) into a compact, diversity-optimized snapshot that anchors the planner's corpus view from the first round. Conditioned on this snapshot and prior search history, the planner issues subqueries; an LLM judge evaluates retrieved evidence and summarizes it as feedback for the next round. We train the planner with Group Relative Policy Optimization (GRPO), judging the answer generated from evidence gathered at each round to produce a reward at the level of individual rounds rather than only at the episode's end. On MSQA, TechQA, and ANTIQUE, RIPPLE improves LLM-based scores by up to 7% and CAR by up to 15% over strong baselines, including Search-R1, with ablations confirming both the diverse snapshot and search-round-level supervision are independently necessary. These gains generalize across planner backbones, an independent evaluator, multi-hop benchmarks (HotpotQA, 2WikiMultiHopQA), and zero-shot transfer, while also reducing search rounds needed.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Iterative Retrieval, Reinforcement Learning, LLM Planning, Query Reformulation, Agentic Retrieval, Corpus-Aware Planning, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5uGYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RIPPLE, a learned retrieval planner that combines a persistent, query-specific corpus snapshot with feedback from unsuccessful retrieval attempts. The snapshot is constructed by retrieving a broad candidate set and applying maximal marginal relevance to select diverse documents. The planner generates subqueries conditioned on this snapshot and previous failure summaries. It is trained using GRPO with answer-quality rewards at individual search states, alongside format, query-relevance, and diversity rewards.

The paper reports improvements over retrieval, planning, and RL-based baselines on ANTIQUE, TechQA, and MSQA. Additional experiments investigate component ablations, multi-hop QA, zero-shot transfer, different planner and generator backbones, independent evaluation, and inference latency.

Critical strengths
  1. The planner-state design addresses a useful practical problem. Combining a diversity-aware snapshot with summaries of unsuccessful retrievals gives the planner both an initial view of potentially complementary information and feedback about previously unproductive directions. This is a concrete approach to improving retrieval planning for underspecified and non-factoid questions.

  2. The reported empirical investigation is broad. Beyond the three primary benchmarks, the paper evaluates multi-hop tasks, zero-shot transfer, multiple planner and generator models, and an independent evaluator. The snapshot and feedback ablations also provide encouraging evidence that these components contribute to the reported gains. If the evaluation protocol is clarified, this breadth would strengthen the case that the approach is useful beyond a single dataset or model configuration.

Critical weaknesses
  1. The experiments do not clearly isolate the central round-level supervision claim. The terminal-only ablation in Table 2 is described as also removing the failure-conditioned training mode. It is therefore unclear whether the difference comes from reward granularity, training-state coverage, or the number of optimization updates. Furthermore, under the stated stopping rule, search terminates once at least one document is judged relevant. Consequently, any trajectory that continues to another round must have an empty previously accepted relevant-evidence pool. Under this literal formulation, later-state answer utility depends on the current round’s accepted evidence, rather than progressive accumulation of accepted partial evidence across rounds. This can support learning to retry after retrieval failures, but does not establish the stronger claim about credit assignment for progressively closing evidence gaps.

  2. The evaluation protocol leaves important questions about both numerical reliability and metric interpretation. Appendix A.3 defines LLM-Eval and CAR as means of binary judgments. With 200 ANTIQUE test questions, single-run scores must be multiples of 0.005, which cannot produce the reported 0.418 LLM-Eval or 0.762 CAR. Repeated runs or a different aggregation scheme could explain these values, but the repetition count, aggregation rule, and uncertainty are not reported. This limits confidence in the main comparisons and smaller ablation differences. Additionally, CAR only measures whether the generator abstains; it does not independently verify that the evidence supports the answer. Its improvement therefore cannot by itself substantiate the paper’s evidence-sufficiency claims.

Questions for authors

Clarification and validation requests:

  1. Please specify the exact evaluation denominators, number of training and evaluation repetitions, and aggregation rules behind Tables 1–3. How are the ANTIQUE scores of 0.418 and 0.762 obtained under the binary metric definitions? Per-run binary counts and uncertainty for the principal comparisons would help resolve this concern.

  2. Does the implemented training and inference procedure stop immediately when the cumulative relevant-document count reaches one? If so, how does the round-level objective supervise progressive accumulation of partial evidence? In the terminal-only ablation, do all decisions in failure-conditioned states still receive optimization updates? Please clarify whether state coverage and optimization budgets are matched.

  3. What evidence supports interpreting increased CAR as improved evidence sufficiency, beyond increased willingness to answer? Please also clarify how the 310-example TechQA split was used, how hyperparameters were selected, what retrieval corpus was available, and how unanswerable cases were handled.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer 5uGY (1/6)

We thank the reviewer for the close reading and for recognising the core of RIPPLE's design: the planner-state formulation that combines a diversity-aware corpus snapshot with summaries of unsuccessful retrievals, giving the planner both an initial view of complementary information and feedback on previously unproductive directions, together with the breadth of the empirical investigation across component ablations, multi-hop tasks, zero-shot transfer, multiple planner and generator backbones, and an independent evaluator.

Below we address the concerns raised: whether the experiments isolate the round-level supervision claim and how the stopping rule relates to it (W1, Q2); the evaluation protocol behind the reported metrics, covering the denominators, aggregation rule, and number of runs (W2, Q1); and the requested clarifications on the TechQA split, hyperparameter selection, the retrieval corpus, and unanswerable cases, together with an extensive analysis to establish that RIPPLE improves retrieved evidence sufficiency (Q3). Here we use the notation W to depict weakness and Q to depict Question.


W1 + Q2 (combined) — The terminal-only ablation isolates round-level supervision, and the stopping rule supports progressive credit assignment

"W1 - The experiments do not clearly isolate the central round-level supervision claim. The terminal-only ablation in Table 2 is described as also removing the failure-conditioned training mode. It is therefore unclear whether the difference comes from reward granularity, training-state coverage, or the number of optimization updates."

"Q2 - ... In the terminal-only ablation, do all decisions in failure-conditioned states still receive optimization updates? Please clarify whether state coverage and optimization budgets are matched."

The terminal-only variant retains the failure-conditioned training mode such that the failure summaries of previous rounds are used in subsequent rounds. The only change in terminal-only ablation is the reward granularity. Training-state coverage and the number of optimization updates are held fixed, so the comparison does isolate round-level supervision. We explain this elaborately in the following points -

  • Training-state coverage is identical: the failure-conditioned states and their failure summaries are kept. The terminal-only variant trains on the same states as RIPPLE. After a failed round, the next round still receives the earlier round's summary of non-relevant retrievals in its input state, so the planner can still read what failed to redirect its next search plan. Both the initial states X0X_0 and the later failure-conditioned states XtX_t (t > 0), which carry the prior subqueries and these failure summaries, remain in training, and no state family is dropped. Every decision in these failure-conditioned states receives an optimization update in the terminal-only variant as well.
  • Failure feedback enters through the state, not the reward, so changing the reward does not remove the failure summaries-conditioned training mode. The planner only ever sees failure summary through the state it reads at the start of each round, never through the reward. The reward does something separate: it sets how the search plan at each round's state are scored during training. Since these are two different parts of the method, switching to a terminal reward changes only the scoring and leaves the failure summaries in every input state untouched.
  • The only factor that differs in terminal-only variant is which reward scores each round's plan; the states, candidate plans, optimization updates, and per-round rollout budget are all matched. In the terminal-only variant, the planner runs through all its rounds as usual, and a single terminal reward is computed once at the end of the trajectory. That one reward is then used to score every round's plan and to provide the optimization update for the planner's output in that round, including the rounds that failed. In RIPPLE, each round's search plan is instead scored by its own round-level reward. This reward is computed using the answer generated from the evidence gathered till that round. The states visited, the candidate plans tried at each round, the number of optimization updates, and the per-state rollout budget are identical in both settings, so neither state coverage nor the optimization budget differs. The only remaining difference is reward granularity, which is exactly what the ablation is meant to isolate.

Section 5 (lines 411-417) has been updated to make explicit that the terminal-only variant retains all failure-conditioned training states along with failure summaries and changes only the reward granularity.

Answer to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer 5uGY (2/6)

"W1 - ...Furthermore, under the stated stopping rule, search terminates once at least one document is judged relevant. Consequently, any trajectory that continues to another round must have an empty previously accepted relevant-evidence pool. Under this literal formulation, later-state answer utility depends on the current round's accepted evidence, rather than progressive accumulation of accepted partial evidence across rounds. This can support learning to retry after retrieval failures, but does not establish the stronger claim about credit assignment for progressively closing evidence gaps."

"Q2 - Does the implemented training and inference procedure stop immediately when the cumulative relevant-document count reaches one? If so, how does the round-level objective supervise progressive accumulation of partial evidence?"

We describe the precise mechanism on how the relevance threshold based stopping works and discuss how failure summaries and round-level credit assignment operates on failed rounds to enable the planner to learn to progressively close evidence gaps. Finally, we also discuss an example from TechQA to illustrate this -

How relevance threshold-based search stopping rule works - During a search round, the relevance judge inspects the retrieved documents to determine if the retrieved evidence is relevant and sufficient to answer the question. If the judge finds at least one such relevant and sufficient document (hence the threshold of 1), then we stop the search early, do not execute any further search rounds (both during training as well as during inference) and generate the answer using the accepted evidence collected so far (as also shown in figure 2 in main paper). However, in scenarios where a document or more than one document are found to be relevant but collectively not sufficient enough to answer the question by the judge, it is deemed as a failed round and the relevant documents are added to the accepted relevant-evidence pool. The subsequent search round is then executed conditioned on the failure summary of previous rounds allowing for further cumulative accumulation of evidence. Hence, answer utility of the round would depend on accepted partial evidence accumulated so far. We have modified the paper pdf now to clarify the relevance threshold mechanism and now discuss how round level objective supervise accumulation of partial evidence -

Why failed rounds contribute towards effective training of planner: Post the completion of a failed round, the round-level reward determined based on answer generated using the evidence gathered till completion of that round can still carry a training signal and provides supervision to the planner to learn to close the evidence gap owing to the following reasons:

  • On a failed round, an intermediate answer is still generated and scored, so the round carries a real training signal. Even when relevance threshold is not being met in a failed round, the evidence gathered can be relevant and partially complete but insufficient. The evidence gathered till that round's completion is used to generate an answer and receives its round-level reward. Because the reward on this answer is graded rather than tied to the relevance threshold, it reflects exactly this partial usefulness: a plan whose evidence yields a better answer (even if still incomplete) scores higher than one whose evidence is unhelpful.
  • This per-round signal complements the failure summaries rather than restating them. The failure summaries sit in the state and tell the planner which search directions already failed giving planner the context to redirect search plan accordingly. The round-level reward then grades the new plan by the quality of the answer generated from the evidence it accumulated. Failure summaries supply the context for plan redirection; round level objective supplies the trainable signal reflecting the quality of the answer generated from the evidence the redirected plan gathered.
  • Because the planner is rewarded on the failed rounds as well as on the round that finally succeeds, it can tell which plans held the search back and which moved it forward. Each failed round's low answer-based reward pushes the planner away from the plan that produced it, while the succeeding round's higher reward pulls it toward the plan that retrieve answer-supporting evidence. Credit is therefore assigned to each round's plan individually. Based on the reward of the round that eventually succeeded against those of the failed rounds that preceded it, the planner implicitly learns what separated the productive plan from the unproductive ones.

Answer to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer 5uGY (3/6)

We now discuss an example to illustrate the relevance threshold working mechanism and how round level credit assignment supervises progressive accumulation of partial evidence:

An example RIPPLE trajectory from TechQA. A sample question is taken which is about reporting a WebSphere Portal 8.5 startup failure after a database transfer to Oracle, with errors EJPDB0085W, EJPDB0160E, and EJPFD0016E and the database character set shown as WE8MSWIN1252 (not UTF-8). The ground truth answer is that the database was created with an unsupported character set and must be dropped and recreated with a Unicode configuration; a successful transfer does not fix this. We discuss the evidence retrieved after each search round and corresponding relevance judge labels below -

  • Round 1 (fails the relevance threshold bar with all irrelevant documents). The planner's subqueries retrieve only documents about datasource and database-URL fixes; the relevance judge deems all the documents as irrelevant, so the accepted-evidence pool is empty. An intermediate answer is still generated from the round's documents and scored: the answer misreads the encoding warning as a connection-mapping issue and recommends retaining the database, which contradicts the ground-truth answer, so it gets a very low answer correctness reward. The failure summaries are written into the planner state.
  • Round 2 (fails the relevance threshold bar with a relevant document). Conditioned on those summaries, the planner redirects toward the encoding question and retrieves D1 (swg21383033), which establishes the right diagnostic branch, that a successful transfer is insufficient and the unsupported code set must be checked against UTF8 / AL32UTF8 / AL16UTF16, but does not contain the exact recovery procedure. The relevance judge labels this document as relevant but not sufficient, so the document gets added to the accepted evidence pool but relevance threshold is still not met; the answer generated using the accepted evidence pool gets slightly higher correctness this time for correctly identifying the unsupported-encoding branch and ruling out the URL-fix remedy. At the same time, further failure summaries are added regarding lacking recovery procedure.
  • Round 3 (clears the relevance threshold). The refined subqueries retrieve D2 (swg21317981), the gold document, which supplies the drop-and-recreate remedy and the Oracle character-set checks. The evidence now clears the relevance bar, and the final answer gives the complete, correct recovery which gets the highest correctness score reward for updating the learner accordingly.

The above case study illustrates the progressive improvement in the searched evidence across rounds and quality of the answer generated from each round's gathered evidence operating under the relevance threshold rule. The above case study has been added to Appendix A.8 and referred in main paper.


W2 + Q1 (combined) — The evaluation protocol accounts for the reported metrics: denominators, aggregation, and repetitions

"W2 - The evaluation protocol leaves important questions about both numerical reliability and metric interpretation. Appendix A.3 defines LLM-Eval and CAR as means of binary judgments. With 200 ANTIQUE test questions, single-run scores must be multiples of 0.005, which cannot produce the reported 0.418 LLM-Eval or 0.762 CAR. Repeated runs or a different aggregation scheme could explain these values, but the repetition count, aggregation rule, and uncertainty are not reported. This limits confidence in the main comparisons and smaller ablation differences."

"Q1 - Please specify the exact evaluation denominators, number of training and evaluation repetitions, and aggregation rules behind Tables 1–3. How are the ANTIQUE scores of 0.418 and 0.762 obtained under the binary metric definitions? Per-run binary counts and uncertainty for the principal comparisons would help resolve this concern."

Aggregation rule and denominators (how 0.418 and 0.762 arise).

Each of LLM-Eval and CAR is the mean of five evaluation runs. On ANTIQUE this is five runs over 200 questions, or 1000 binary judgments, so the reported values are multiples of 0.001 and 0.418 and 0.762 are exactly 418/1000 and 762/1000, which resolves the steps-of-0.005 concern. The same construction fixes the denominators on the other datasets: 5 x 310 = 1550 binary judgments on TechQA and 5 x 572 = 2860 on MSQA. This averaging rule and these denominators apply identically to every entry in Tables 1 to 3.

Answer to be continued ...

Reviewer_hHA3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RIPPLE trains a retrieval planner using a query-specific MMR snapshot and summaries of unsuccessful searches. GRPO compares candidate plans at each state using answer, format, relevance, and diversity rewards. The paper reports gains on ANTIQUE, TechQA, and MSQA, with additional transfer, backbone, and evaluator checks.

Critical strengths

The system has clear practical value. RIPPLE leads all twelve dataset–metric columns in Table 1, and Table 3 supports the usefulness of both diverse snapshots and failure summaries. The independent-evaluator and alternative-backbone results are also reassuring. Together, these results support a useful retrieval system rather than an improvement confined to one metric or evaluator. I would keep these positive findings separate from the unresolved explanation of why the training works.

Critical weaknesses

The main issue is the supervision ablation. Table 2's terminal-only variant also removes training on intermediate failure-conditioned states, so the comparison changes both reward timing and the states used for training. It therefore does not isolate round-level supervision. Also, under the stated stopping threshold of one relevant document, any reached later round has no previously accepted relevant evidence. Failure summaries can still help, but the cumulative-evidence interpretation needs to reflect this setting.

The numerical definitions also need clarification. With 200 ANTIQUE questions and binary judgments, a single-run mean moves in steps of 0.005; Table 1's 0.418 and 0.762 require another aggregation procedure. Repeated judgments or runs could explain this. Figure 6's roughly 0.64 reward at about 0.31 correctness also needs to be reconciled with Equation 5's components and logging conventions. These are reporting questions, not evidence that the system is ineffective.

Questions for authors

Could you compare the two supervision schemes on the same initial and failure-conditioned training states, holding context and update counts fixed and accounting for any additional rollout cost? Please clarify how the stopping rule limits the claimed cumulative-evidence mechanism.

Could you specify the metric averaging procedure and provide a small worked example that reconstructs the reported metrics and Figure 6 reward, including normalization and singleton-query diversity handling?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer hHA3 (1/6)

We thank the reviewer for the careful reading and for recognising the strength of the empirical results: that RIPPLE leads every one of the twelve columns in Table 1, with the Table 3 results, the independent evaluator, and the alternative-backbone experiments all corroborating these gains. Below we address the points raised: the terminal-only ablation and how the comparison isolates round-level supervision, together with the stopping rule (W1, Q1); and the numerical and metric clarifications, covering the denominators and aggregation behind the reported scores and the reconciliation of the Figure 6 reward with Equation 5, with a worked example (W2, Q2). Here we use the notation W to depict weakness and Q to depict Question.


W1 + Q1 (combined) — The terminal-only ablation isolates round-level supervision, and the stopping rule is consistent with the cumulative-evidence mechanism

W1 - "The main issue is the supervision ablation. Table 2's terminal-only variant also removes training on intermediate failure-conditioned states, so the comparison changes both reward timing and the states used for training. It therefore does not isolate round-level supervision."

This is also the first part of Question 1: "Could you compare the two supervision schemes on the same initial and failure-conditioned training states, holding context and update counts fixed and accounting for any additional rollout cost?"

The terminal-only variant retains the failure-conditioned training states and their failure summaries, and changes only the reward granularity. Training-state coverage and the number of optimization updates are held fixed, so the comparison does isolate round-level supervision. We make this concrete through the following points:

  • In the terminal-only variant, the previous rounds' failure summaries are not removed, and it trains on the same set of states as RIPPLE. The variant retains the failure summaries in its training states: after a failed round, the following round still receives the earlier round's summary of non-relevant retrievals as part of its input state, so the planner can still read what failed and redirect its search plan. Both the initial states X0X_0 and the later failure-conditioned states XtX_t (t > 0), which carry the prior subqueries and these summaries of non-relevant retrievals, remain part of its training, and no state family is dropped.
  • The terminal-only variant's change to the reward does not touch where failure feedback enters. Failure feedback reaches the planner only through the state it reads at the start of each round, and never through the reward. The reward controls something separate, namely how the candidate plans after each round are scored during training. Because these are two different parts of the method, switching the variant to a terminal reward changes only the scoring and leaves the failure summaries in every state untouched. The construction of these states is detailed in Appendix A.2.
  • In the terminal-only variant, the terminal reward is used to score each round's search plan. In the variant, the planner runs through all of its rounds exactly as usual, and a single terminal reward is computed once at the end of the trajectory. That one terminal reward is then used to score each round's search plan and to provide the optimization update for the planner's output in that round, including the rounds that failed. In RIPPLE, each round's search plan is instead scored by its own round-level reward. This reward is computed using the answer generated from the evidence gathered till that round. The states visited, the candidate plans tried at each round, the number of optimization updates, and the per-state rollout budget are identical in both RIPPLE and terminal-only variant so neither scheme is given additional rollouts. The ablation therefore isolates round-level supervision.

Section 5 (lines 411-417) has been updated to make explicit that the terminal-only variant retains all failure-conditioned training states and failure summaries and changes only the reward granularity.

Response to other questions to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer hHA3 (2/6)

W1 - "Also, under the stated stopping threshold of one relevant document, any reached later round has no previously accepted relevant evidence. Failure summaries can still help, but the cumulative-evidence interpretation needs to reflect this setting."

This is also the second part of Question 1: "Please clarify how the stopping rule limits the claimed cumulative-evidence mechanism."

We describe the precise mechanism on how the relevance threshold works and also discuss an example from TechQA to illustrate this. Further, we provide context on how failure summaries and round-level credit assignment operates on failed rounds to enable planner learn to gather cumulative-evidence.

How relevance threshold works - During a search round, the relevance judge inspects the retrieved documents to determine if the evidence retrieved so far is relevant and sufficient to answer the question. If the judge finds at least one such relevant and sufficient document then we stop the search early (hence the threshold of 1) and do not perform further search rounds and generate the answer using the evidence collected so far (as shown in figure 2 in the main paper). However, in scenarios where a document or more than one document are found to be relevant but collectively not sufficient enough to answer the question by the judge, it is deemed as a failed round and subsequent search round is still executed conditioned on the failure summary of previous rounds allowing for cumulative accumulation of evidence. Hence, any reached round can still have previously accepted relevant evidence such that this accepted evidence is not sufficient enough to answer the question, otherwise the reached search round would not have been started. We have modified the paper pdf to clarify the relevance threshold mechanism and now discuss an example to illustrate the relevance threshld working mechanism below:

An example RIPPLE trajectory from TechQA to illustrate relevance threshold working mechanism. A sample query is about reporting a WebSphere Portal 8.5 startup failure after a database transfer to Oracle, with errors EJPDB0085W, EJPDB0160E, and EJPFD0016E and the database character set shown as WE8MSWIN1252 (not UTF-8). The ground truth answer is that the database was created with an unsupported character set and must be dropped and recreated with a Unicode configuration; a successful transfer does not fix this. We discuss the evidence retrieved after each search round and corresponding relevance judge labels below -

  • Round 1 (fails the relevance threshold bar with all irrelevant documents). The planner's subqueries retrieve only documents about datasource and database-URL fixes; the relevance judge accepts none, so the accepted-evidence pool is empty. An intermediate answer is still generated from the round's documents and scored: it misreads the encoding warning as a connection-mapping issue and recommends retaining the database, which contradicts the ground-truth answer, so it gets a very low answer correctness reward. The failure summaries are written into the planner state.
  • Round 2 (fails the relevance threshold bar with a relevant document). Conditioned on those summaries, the planner redirects toward the encoding question and retrieves D1 (swg21383033), which establishes the right diagnostic branch, that a successful transfer is insufficient and the unsupported code set must be checked against UTF8 / AL32UTF8 / AL16UTF16, but does not contain the exact recovery procedure. The relevance judge labels this document as relevant but not sufficient, so the document gets added to the accepted evidence pool but relevance threshold is still not met; the answer generated using the accepted evidence pool gets slightly higher correctness this time for correctly identifying the unsupported-encoding branch and ruling out the URL-fix remedy. At the same time, further failure summaries are added regarding lacking recovery procedure.
  • Round 3 (clears the relevance threshold). The refined subqueries retrieve D2 (swg21317981), the gold document, which supplies the drop-and-recreate remedy and the Oracle character-set checks. The evidence now clears the relevance bar, and the final answer gives the complete, correct recovery which gets the highest correctness score reward for updating the learner accordingly.

The above case study illustrates the progressive improvement in the searched evidence across rounds and quality of the answer generated from each round's gathered evidence operating under the relevance threshold rule. The above case study has been added to Appendix A.8 and referred in main paper.

Answer to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer hHA3 (3/6)

Why failed rounds contribute towards effective training of planner: Post the completion of a failed round, the round-level reward determined based on answer generated using the evidence gathered till completion of that round can still carry a training signal and provides supervision to the planner to contribute toward closing the evidence gap owing to the following reasons:

  • After a failed round, an answer is still generated using its evidence and scored, so the round carries a real training signal. Even when relevance threshold is not being met in a failed round, the evidence gathered till that round can be relevant and partially complete. The evidence gathered till completion of the round is used to generate an answer and receives its round-level reward. Because the reward on this answer is graded rather than tied to the relevance threshold, it reflects exactly this partial usefulness: a plan whose evidence yields a better answer (even if incomplete) scores higher than some plan whose evidence is unhelpful.
  • This per-round signal complements the failure summaries rather than restating them. The failure summaries sit in the state and tell the planner which search directions already failed, giving it the context on how a different new plan should be proposed. The round-level reward then grades that new plan by the quality of the answer generated from the evidence it added. Failure summaries supply the context for redirection; round level objective supplies the trainable signal reflecting the quality of the answer generated from the evidence the redirected plan gathered.
  • Because the planner is rewarded on the failed rounds as well as on the round that finally succeeds, it can tell which plans held the search back and which moved it forward. Each failed round's low answer-based reward pushes the planner away from the plan that produced it, while the succeeding round's higher reward pulls it toward the plan that retrieved answer-supporting evidence. Credit is therefore assigned to each round's plan individually. Based on the reward of the round that eventually succeeded against those of the failed rounds that preceded it, the planner implicitly learns what separated the productive plan from the unproductive ones.

W2 + Q2 (combined) — The reported metrics and the Figure 6 reward follow a reproducible averaging procedure

We specify the averaging procedure and provide worked examples regarding each point below.

(i) Metric averaging and numerical reliability

"W2 - The numerical definitions also need clarification. With 200 ANTIQUE questions and binary judgments, a single-run mean moves in steps of 0.005; Table 1's 0.418 and 0.762 require another aggregation procedure. Repeated judgments or runs could explain this."

"Q2 - Could you specify the metric averaging procedure and provide a small worked example that reconstructs the reported metrics ..."

  • The reported value averages five evaluation runs, which sets the denominator and explains the step size. For each metric the evaluation is run five times and the reported number is the mean over those five evaluation runs. On ANTIQUE this is five runs over 200 questions, or 1000 binary judgments, so the reported values are multiples of 0.001, and 0.418 and 0.762 are exactly 418/1000 and 762/1000. The same construction explains the denominators on the other datasets: 5 x 310 = 1550 binary judgments on TechQA and 5 x 572 = 2860 on MSQA.
  • Averaging repeated judge runs is an established way to reduce the random error of a single run. A survey of LLM-as-a-judge methods (arXiv:2411.15594) notes that combining multiple evaluation results for the same content reduces the impact of accidental factors and random error in a single run, citing work that averages multiple scores of the same sample and work that reports the mean and standard deviation over repeated runs; G-Eval (Liu et al., 2023) likewise samples the judge several times and combines the scores. We follow this practice with five runs per metric for all the methods.
  • Averaging over five runs also stabilises the reported value against judge-run variability. Because each number is the mean of five independent evaluation runs, run-to-run fluctuations in the judge are averaged down rather than carried by a single-run score. The reported values therefore summarise repeated evaluation runs.

Answer to be continued ...

Reviewer_jcUDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RIPPLE, an iterative retrieval-augmented generation framework that aims to improve learned multi-round retrieval along two dimensions: the information available to the search policy and the granularity of its training signal. RIPPLE first performs a relatively broad retrieval and applies maximal marginal relevance (MMR) to construct a compact, diversity-aware corpus snapshot, which is retained throughout the search process. At later rounds, the planner additionally conditions on previously issued subqueries and summaries of retrieved documents judged to be non-relevant, allowing subsequent searches to account for unsuccessful retrieval directions. The planner is trained with GRPO using round-level utilities: multiple candidate search plans are sampled from the same planner state, executed independently through retrieval and evidence accumulation, and evaluated based primarily on the quality of an intermediate answer generated from the accumulated evidence, together with auxiliary relevance, diversity, and formatting rewards. The paper reports improvements over several iterative and RL-based RAG baselines on ANTIQUE, TechQA, and MSQA, and provides additional results on multi-hop QA, different planner/generator backbones, out-of-distribution transfer, efficiency, and several ablation and sensitivity studies.

Critical strengths
  1. The paper presents a coherent and practically relevant formulation of learned iterative retrieval, with consistent empirical improvements over strong baselines. Rather than optimizing only the final answer, RIPPLE explicitly connects the search policy to both a broader retrieval-derived context and the downstream utility of intermediate search decisions. The strongest aspect of the method is the combination of a persistent corpus snapshot and failure-conditioned state with same-state comparison of alternative candidate search plans. The reported gains are consistent across the three main benchmarks and extend to learned-search baselines such as Search-R1 and RAG-R1, suggesting that the proposed design provides value beyond a weak single-pass RAG baseline.

  2. The experimental evaluation is unusually broad for a multi-component RAG system. The paper includes ablations of the corpus snapshot, failure summaries, reward components, and supervision granularity, together with sensitivity studies for several search and training budgets. It additionally evaluates different planner backbones, an independent evaluator, multi-hop benchmarks, zero-shot transfer, end-to-end latency, corpus-size scaling, and training dynamics. This breadth is consequential because RIPPLE contains many interacting components, and the additional experiments provide evidence that the reported gains are not restricted to a single model configuration or benchmark.

Critical weaknesses
  1. The current experiments do not fully establish the paper’s central claim that RIPPLE learns a better evidence-gap-closing and failure-recovery search policy. The reported ablations show that the corpus snapshot, failure summaries, and round-level supervision improve downstream QA metrics, but they do not directly show that the planner actually redirects search more effectively after failures or closes missing evidence gaps more systematically than existing learned-search methods. As a result, the paper demonstrates that the proposed components are useful, but the connection between the observed gains and the specific search behavior emphasized in the motivation remains somewhat indirect.

  2. RIPPLE relies on many consequential hyperparameters whose default choices are only partially justified. The method depends on retrieval budgets, snapshot size, MMR trade-off, subquery budget, search depth, stopping threshold, number of GRPO rollouts, and several reward weights. Although the appendix analyzes some of these choices, several defaults are not the best-performing settings and appear to reflect empirical performance–cost trade-offs, while other important parameters receive little comparable analysis. This makes it difficult to assess how robust the method is to configuration choices and how readily the reported setup would transfer to different retrieval environments.

Questions for authors
  1. Can the authors quantify RIPPLE's claimed failure-recovery behavior more directly? For example, conditioned on a search round that fails to retrieve useful evidence, how often does the following round successfully retrieve new relevant evidence, and how does this compare with Search-R1 or the no-failure-summary ablation? A simple analysis from existing search trajectories would help establish whether the proposed feedback mechanism actually improves recovery after unsuccessful retrieval, rather than only improving final QA metrics.

  2. Please clarify the hyperparameter selection protocol used for the reported default configuration. Were the default settings selected using held-out validation data and fixed before test evaluation, and were the same settings used across all datasets? If so, what criterion was used to choose among configurations with different performance–cost trade-offs?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer jcUD (1/3)

We thank the reviewer for the careful reading and for recognising the core of RIPPLE's design: the persistent corpus snapshot, the failure-conditioned planning that reuses what a round missed, and failure-conditioned state with same-state comparison of alternative candidate search plans. Below we address the two concerns raised: direct evidence linking the gains to failure-driven redirection (W1, Q1), and the justification and transferability of the hyperparameter configuration (W2, Q2). Here we use the notation W to depict weakness and Q to depict Question.


W1 + Q1 (combined) — Direct evidence that RIPPLE learns failure-driven search redirection

W1: "The current experiments do not fully establish the paper's central claim that RIPPLE learns a better evidence-gap-closing and failure-recovery search policy. The reported ablations show that the corpus snapshot, failure summaries, and round-level supervision improve downstream QA metrics, but they do not directly show that the planner actually redirects search more effectively after failures or closes missing evidence gaps more systematically than existing learned-search methods. As a result, the paper demonstrates that the proposed components are useful, but the connection between the observed gains and the specific search behavior emphasized in the motivation remains somewhat indirect."

Q1: "Can the authors quantify RIPPLE's claimed failure-recovery behavior more directly? For example, conditioned on a search round that fails to retrieve useful evidence, how often does the following round successfully retrieve new relevant evidence, and how does this compare with Search-R1 or the no-failure-summary ablation? A simple analysis from existing search trajectories would help establish whether the proposed feedback mechanism actually improves recovery after unsuccessful retrieval, rather than only improving final QA metrics."

To establish that RIPPLE learns failure-driven redirection, we conduct multiple analysis based on planner and retrieval outputs across search rounds. We run this analysis on TechQA evaluation set which contains annotations of ground-truth document(s) needed to answer the given question, since these document-level labels let us check which round retrieves the correct evidence (we exclude unanswerable question samples in the dataset's evaluation set from this analysis). Using the existing evaluation trajectories (no retraining), we report three measurements, and in every case we compare RIPPLE against Search-R1 and against the no-failure-summary ablation. Following is a summary of the analysis metrics and results observed:

  • Analysis 1 (recovery rate after a miss): after a miss in the initial search round, RIPPLE recovers the ground-truth document in a later round for 40.4% of questions, against 34.1% without the failure summary and 30.1% for Search-R1.
  • Analysis 2 (subquery redirection towards relevant documents): the average cosine similarity between embedding of RIPPLE's subqueries and the ground-truth document rises with subsequent search rounds, from 0.621 to 0.748, while Search-R1 and the no-summary ablation stay near 0.60 across all search rounds.
  • Analysis 3 (cumulative retrieval accuracy after each round): the cumulative hit rate i.e. the fraction of questions whose at least one ground-truth document has been retrieved till the completion of a given round, increases after each round for RIPPLE (retrieval accuracy from round 1 to 5 is 0.194, 0.377, 0.477, 0.510, 0.519), reaching 0.519 by the final round against 0.465 for the no-summary ablation and 0.400 for Search-R1.

We discuss the three analysis in detail below.

Analysis 1: Recovery rate after an initial miss. For each question we consider its ground-truth answer document. We call the first search round a miss when it retrieves no ground truth document, and we call the question recovered when a later round retrieves at least one ground truth document. The recovery rate is the number of recovered questions divided by the number whose first round missed the document, so it measures how often the planner turns an initial miss into a correct retrieval in a later search round.

SystemRecovery rate
RIPPLE0.404
Search-R10.301
RIPPLE, no failure summary0.341

After an initial miss, RIPPLE recovers the ground-truth document in a later round for 40.4% of questions, against 34.1% without the failure summary and 30.1% for Search-R1. The drop from 40.4% to 34.1% when the failure summary is removed shows that failure-conditioned planning contributes directly to the recovery.

Discussion on detailed analysis to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer jcUD (2/3)

As a further sub-analysis, we decompose this recovery rate round by round. The aggregate rate above pools all later rounds together, so breaking it out by round shows how recovery is distributed across the search rounds. At each round the numerator is the questions that retrieve the ground-truth document for the first time in that round, and the denominator is the questions that are still missing it entering that round. Each entry therefore reads as: of the questions still missing the document going into this round, the fraction that retrieve it here.

SystemRound 2Round 3Round 4Round 5
RIPPLE0.2280.1610.0620.020
Search-R10.1500.1060.0590.021
RIPPLE, no failure summary0.1980.1240.0450.018

(These per-round values are conditional rates, each normalised by the questions still missing the document at the start of that round, so they are not additive and are not meant to sum to the aggregate, which normalises once by the initially-missed set.)

RIPPLE recovers the largest share of the still-missing questions at each of the first two post-miss rounds (R2 and R3) which have maximum scope to recover and find the relevant evidence, thereby having significantly better recovery rates for these rounds when compared to both Search-R1 and the no-failure-summary ablation.

Analysis 2: The planner progressively redirects the subqueries toward the correct evidence. We embed each subquery (in a search round) and the ground-truth document with the retriever's encoder, measure cosine similarity between each subsequery and ground-truth document and then average across all subqueries for the round. We report this similarity at each successive round of the search, averaged over the question samples still searching for the document at that round, so the table below reads round by round as how close the planner's queries have moved to the correct document by that round. If a failed round's summary redirects the search, the subsequent subqueries should move closer to the ground-truth document, so this similarity should rise as the search deepens.

SystemR1R2R3R4R5
RIPPLE0.6210.6840.7310.7440.748
Search-R10.5980.6060.5910.6070.599
RIPPLE, no failure summary0.6120.6190.6080.6240.617

RIPPLE's subqueries move steadily toward the ground-truth document as the search deepens, from 0.621 to 0.748, more than both Search-R1 and the no-failure-summary ablation. This links the planner's behaviour, moving the search toward the correct document, to the recovery measured in Analysis 1.

Analysis 3: Cumulative retrieval hit rate of the correct document across rounds. At each round we report the fraction of questions for which at least one ground-truth document has been retrieved in that round or any earlier round. This cumulative hit rate shows whether the search keeps adding correct evidence if needed as it goes deeper.

SystemR1R2R3R4R5
RIPPLE0.1940.3770.4770.5100.519
Search-R10.1420.2710.3480.3870.400
RIPPLE, no failure summary0.1870.3480.4290.4550.465

RIPPLE retrieves at least one correct document for a growing share of questions at every round and leads at every round, ending at 0.519 against 0.465 without the failure summary and 0.400 for Search-R1. The margin widens as the search deepens, which is exactly where failure-driven redirection accumulates.

Together these results show the gains come from the intended behavior: RIPPLE reads what failed, redirects the search, and recovers relevant evidence more reliably. We have updated the paper pdf and added these analysis and discussions in Appendix A.7 (and referred them appropriately in the main paper).

Response to other questions to be continued ...

AuthorsOpenReview ↗

Author Response to Reviewer jcUD (3/3)

W2 + Q2 — Hyperparameters are selected by cross-validation, corroborated by a sensitivity study. Configuration is fixed before any test evaluation which transfers well to new retrieval benchmarks

W2 - "Although the appendix analyzes some of these choices, several defaults are not the best-performing settings and appear to reflect empirical performance–cost trade-offs, while other important parameters receive little comparable analysis."

Q2 - "Were the default settings selected using held-out validation data and fixed before test evaluation, and were the same settings used across all datasets? If so, what criterion was used to choose among configurations with different performance–cost trade-offs?"

The hyperparameters were selected by 10-fold cross-validation on the training data of MSQA, and hyperparameter configuration was selected to balance cost-performance tradeoff. The selected configuration of hyperparameters obtained through this were used unchanged across evaluation on all datasets (we have added this detail to Section 4 under implementation details). For the parameters that carry a retrieval cost, balancing accuracy against cost served a deliberate second purpose beyond efficiency: it keeps RIPPLE within the same search budget as the learned-search baselines, so the comparison is fair and the reported gains reflect better planning under a matched search budget. The end-to-end efficiency analysis in Section 5 (Table 5) confirms this matched budget in practice: RIPPLE answers a query in 11.58s with 2.81 average search iterations, below Search-R1 at 12.08s and 3.80 iterations, even after the 0.73s snapshot-construction cost, so the accuracy gains come at comparable, in fact slightly lower, end-to-end cost.

Selected values for most parameters that control the search budget is corroborated through a dedicated study that validates our chosen set of hyperparameters. In each study the parameter is varied across a range of values and the resulting performance is reported. A few of these settings, chiefly the search depth (number of iterations T) and the retrieval and snapshot budgets (L0, M), trade additional answer quality against higher retrieval cost. For these, the sensitivity studies report the full trade-off, and validates that the chosen value balances quality against cost rather than the one that maximises quality at the highest cost. Furthermore, stopping threshold is not a hyperparameter but defined as a search-stopping criterion set to 1 to depict that search terminates once at least one relevant document which is sufficient to answer the question has been obtained.

The sensitivity studies for the hyperparameters that govern search budget and training are reported in the appendix: the initial retrieval budget L0 and snapshot size M in Appendix A.15, the MMR trade-off λ in Appendix A.11, the subquery budget K in Appendix A.16, the search depth T in Appendix A.10, and the number of GRPO candidates per state G in Appendix A.14. The reward weights (λ_ans, λ_fmt, λ_rel, λ_div) are analyzed through the reward-component ablations in Table 2, and the per-subquery retrieval budget L′ is held fixed across all datasets.

W2 - "This makes it difficult to assess how robust the method is to configuration choices and how readily the reported setup would transfer to different retrieval environments."

The reported hyperparameter configuration was selected by 10-fold cross-validation on the MSQA training data and fixed before any evaluation. It was then used unchanged on ANTIQUE and TechQA, and the paper additionally evaluates whether it transfers beyond the primary benchmarks (Section 5, Table 4). Using this same fixed configuration without any re-tuning, RIPPLE was trained and evaluated on two further multi-hop QA benchmarks with their dedicated corpora and question styles, HotpotQA and 2WikiMultiHopQA, where it reaches 0.498 / 0.813 and 0.478 / 0.798 LLM-Eval / CAR and outperforms both learned-search baselines, Search-R1 and RAG-R1. A balanced, cost-aware configuration that still transfers to these other benchmarks without any additional hyperparameter tuning shows that these settings transfer well across datasets.

The paper additionally reports another form of transfer: zero-shot out-of-distribution transfer (Appendix A.9). The planner is trained only on ANTIQUE and evaluated directly on HotpotQA with no fine-tuning on it, which measures whether the learned search policy itself carries over to a corpus it never saw during training. Even in this setting RIPPLE improves LLM-Eval / CAR to 0.417 / 0.756, a gain of +4.5 and +4.2 points over the strongest baseline, Search-R1.

Dear Reviewer jcUD,

Thank you again for your thoughtful review. We had posted a detailed response addressing your concerns and clarifications you requested. As the discussion window is closing soon, we wanted to request if you can please review the responses and share any further questions that we can clarify.

Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Author Comment Summarising Rebuttal

Summary of Rebuttal and Revisions (to the Area Chair and all Reviewers)

We thank everyone for their careful and constructive reviews. We have posted individual responses last week and uploaded a revised PDF. Below is a summary of the main concerns and how we addressed each one.

1. Direct evidence of failure-driven search behaviour (jcUD): The reviewer asked for evidence beyond final QA metrics that RIPPLE recovers from unsuccessful search rounds. Using the existing TechQA evaluation trajectories (no retraining), we added three document-level analyses, each comparing RIPPLE with Search-R1 and with a no-failure-summary ablation:

  • Recovery rate after a first-round miss: 40.4% for RIPPLE, versus 34.1% without failure summaries and 30.1% for Search-R1.
  • Similarity of subqueries to the ground-truth document across rounds: it rises from 0.621 to 0.748 for RIPPLE, while the two comparison systems stay near 0.60.
  • Cumulative hit rate of the ground-truth document: 0.519 for RIPPLE at the final round, versus 0.465 without failure summaries and 0.400 for Search-R1.

These are in Appendix A.7.

2. Isolating round-level supervision and the stopping rule (hHA3, 5uGY)

  • Terminal-only ablation. We clarified in Section 5 and Appendix A.8 that the terminal-only ablation does keep the failure-conditioned states and the failure summaries of previous rounds in subsequent rounds. The only change in this variant is the use of a terminal reward to score each round's plan. Hence it does isolate the role of round-level supervision when compared with RIPPLE.
  • Stopping rule. We clarified that a round whose documents are relevant but not sufficient counts as a failed round. Its relevant documents are kept in the accumulated evidence pool, which is used for judging sufficiency and generating the intermediate answer, and search continues. We also illustrated this with a TechQA example showing the round-by-round search plan and trajectory.

3. Metric definitions and reporting (hHA3, 5uGY)

  • Metric aggregation. We documented that LLM-Eval and CAR are means over five judge runs. This explains values such as 0.418 and 0.762 (418/1000 and 762/1000 positive judgments on ANTIQUE). We also added a worked example to illustrate this aggregation: five ANTIQUE questions judged over five runs, with per-question means, per-run means and the aggregated value. We additionally report the per-run LLM-Eval and CAR scores and the standard deviation across runs for RIPPLE on ANTIQUE.
  • Figure 6. We specified what each panel logs. Panel (a) is answer correctness averaged across rounds, and panel (b) is the group-mean reward at the final round. We added a worked example showing how each panel is computed and how the reward reconciles with Eq. 5. We also added detail that, for plans with a single subquery, the diversity contribution to the total reward is zero.

4. CAR and evidence sufficiency (5uGY) We clarified that CAR is calculated from whether the LLM judges that the question can be answered from the retrieved evidence, and hence indirectly indicates whether that evidence is sufficient. We then added analyses that directly measure how RIPPLE accumulates correct evidence and compared them against Search-R1. The recovery rate after a miss in the initial search round is substantially higher for RIPPLE (40.4% vs 30.1%). Its subqueries move progressively closer to the ground-truth document (measured through cosine similarity between subquery and ground-truth document) from search round 1 to round 5 (increases from 0.621 to 0.748 for RIPPLE while stays fairly flat around 0.60 for Search-R1). Its cumulative retrieval of the ground-truth document measured after each search round is also higher (reaching to 0.519 for RIPPLE vs 0.400 for Search-R1 after the final search round)

5. Hyperparameter selection and transfer (jcUD, 5uGY) We have now added to Section 4 that hyperparameters were selected by 10-fold cross-validation on the MSQA training data and fixed before evaluation. The same configuration was used unchanged on ANTIQUE and TechQA, and in the HotpotQA and 2WikiMultiHopQA experiments, with no re-tuning. The stopping threshold τ = 1 is a stopping criterion and not a tuned hyperparameter. For parameters that carry retrieval cost, we chose the accuracy–cost balance point rather than the highest-accuracy setting, which keeps RIPPLE's search budget comparable to that of the learned-search baselines (Table 5). The sensitivity studies in the appendix are reported as corroboration of this configuration.

6. Other clarifications (5uGY) We clarified how the 310-example TechQA split was obtained, which retrieval corpus is used for each dataset, and how unanswerable questions are handled (Appendix A.4).

We would be grateful if the reviewers could go through our individual responses. We encourage the reviewers to ask further questions, if needed, and we would be happy to answer them. Thank you for your time and effort.

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.