acceptodds
Under review as a conference paper at ICLR 2027

ReGain: Information-Gain Guided Rollback for Retrieval-Augmented Reasoning

Abstract

Large language models can learn to interleave reasoning with search engine calls through reinforcement learning, enabling multi-step retrieval-augmented generation. However, existing approaches either rely on sparse outcome-only rewards with no per-turn signal, or even when providing process-level supervision, retain misleading retrieval content in the trajectory without correction. We propose ReGain, a closed-loop search quality control framework that integrates two complementary components: (1) a per-turn quality signal based on the change in the model's answer confidence before and after each retrieval, which serves as both a rollback trigger and a step-level process reward; and (2) a state rollback mechanism that detects and undoes low-quality searches in real time during RL rollouts, restoring generation state and injecting explicit query reformulation feedback. Experiments on seven QA benchmarks show that ReGain achieves the highest average F1 among existing outcome-reward and process-reward RL methods, with particularly strong gains on multi-hop reasoning tasks

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Information Gain, Retrieval-Augmented Reasoning, State Rollback, Reinforcement Learning, Process Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i1cNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReGain, a framework for improving retrieval-augmented reasoning through information-gain-guided rollback. ReGain uses changes in answer confidence to evaluate each retrieval step and actively removes low-quality searches from the trajectory, prompting the model to reformulate its query. The framework combines rollback-aware SFT with GRPO training using both process-level information gain and outcome rewards. At inference time, a self-probe mechanism enables rollback without ground-truth answers. Experiments on seven QA benchmarks show improvements over existing RL-based search methods, particularly on multi-hop tasks.

Critical strengths
  1. The rollback mechanism provides a useful extension to existing process-reward methods. This addresses error propagation during multi-turn reasoning and helps to get better trajectory.
  2. ReGain improves average F1 from 48.8 to 50.6 over IGPO, with a larger improvement on multi-hop tasks. The evaluation across two model scales and the reported computational overhead also help assess the method's practical value.
Critical weaknesses
  1. The current ablations lack the setting of training without rollback but enabling it at inference. This makes it difficult to distinguish improvements from better learned policies versus inference-time intervention.
  2. Although Appendix C.1 shows that self-probe and GT-based rollback achieve similar overall F1 scores, this does not establish whether they make consistent step-level rollback decisions.
Questions for authors
  1. Could the authors provide a controlled ablation, particularly training without rollback but enabling self-probe rollback at inference?
  2. Please report the step-level agreement between GT-based and self-probe rollback decisions, with an analysis of cases where the two disagree.
  3. Could the authors provide actual rollback case studies and quantitative statistics on how often rollback corrects or worsens final answers? An analysis of whether query reformulation improves retrieval quality would also help verify that the model learns to use rollback feedback effectively.
Reviewer_EJqeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReGain, a method that uses information gain to judge the quality of each search step. When a search reduces the model's confidence in the correct answer, the method can roll back that search and ask the model to try again. The method also uses information gain as a process reward during RL training. The experiments show improvements over Search-R1 and other process reward methods on seven QA benchmarks.

Critical strengths
  • I think the idea of removing misleading search results from the context is interesting, although there're pretty many works out there doing this.

  • The method is easy to understand, and the paper includes several ablations.

  • The results look okay.

Critical weaknesses
  • 1. I have serious concerns about using information gain as a process reward.

The paper assumes that a good search should increase the model's confidence in the final answer. I do not think this is always true: in multi-hop QA, the agent often needs to find several pieces of information before it can answer the question. Some searches may only help with the next search, rather than directly helping with the final answer.

In fact, a useful search may even reduce the model's confidence if it finds evidence against the model's initial guess. I do not think this should be treated as a bad search. This is a general concern I have with process rewards for search agents. We should not assume that every useful intermediate step must make immediate progress toward the final answer. Some exploration may look unhelpful at first but become important later.

The problem is even more serious in this paper because the method (ReGain) does not just penalize these searches. It may also remove them from the context through rollback. This means the model can lose useful information before doing anything with it.

  • 2. I strongly disagree with the choice of QA evaluation metrics.

The paper follows the Search-R1 style of evaluation and relies on word-level F1. I think this is a very weak / questionable metric for QA.

A correct answer can be written in many different ways. For example, "USA" and "United States" mean the same thing, but word-level F1 may treat them as different answers. On the other hand, an incorrect answer may share many words with the reference answer and still receive a high F1 score.

I would strongly prefer evaluation based on semantic correctness such as an LLM judge that checks whether the answer is factually correct. Related work such as s3 (EMNLP 2025) has explored training search agents based on such metric.

I hope the above two concerns encourage the authors to rethink some common choices in search-agent research, especially the use of process rewards and lexical-overlap metrics, rather than simply following existing setups.)

  • 3. The improvements are limited and the baselines are too few.

Just look at the tables.

Questions for authors

I encourage the authors to fully re-design the method, do more feasible eval using better metrics, and include more comparisons with more baselines.

Reviewer_SZiWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReGain combines per-turn information-gain rewards with active rollback for search-augmented reasoning. An SFT cold start teaches the model to respond to rollback feedback. During RL, a large drop in ground-truth answer log-probability across a reasoning/search turn causes that turn to be removed and a query-reformulation note to be inserted. At inference, the trigger uses five greedily generated answer tokens as a label-free probe. The authors evaluate Qwen2.5-3B and 7B on seven QA datasets and report component ablations, a threshold sweep, an oracle-probe comparison, and runtime measurements.

Critical strengths
  1. The paper gives a concrete way to intervene when retrieval goes wrong. Section 3.4 describes state restoration, feedback listing failed queries, and bounded retries. Retrieved content and feedback are masked from the training loss. Removing a rejected turn changes the context available to later reasoning, which a negative process reward alone does not do. The multi-hop results are promising: ReGain raises average F1 over IGPO from 44.8 to 48.0 on 7B and from 36.1 to 40.7 on 3B.

  2. The inference ablations support the practical value of the procedure. For the same trained policy, enabling inference rollback raises average F1 from 49.4 to 50.6 on 7B and from 43.2 to 44.8 on 3B (Table 2). The label-free self-probe also comes close to the ground-truth oracle probe, at 50.6 versus 50.7 average F1 (Table 6). Appendix A.3 reports measured overhead of approximately 25.4% for RL training and 28.4% for inference. These comparisons show that the method can help at inference without access to answer labels, while making its computational cost explicit.

Critical weaknesses
  1. I cannot tell how much of the improvement over IGPO comes from active rollback. ReGain receives an additional SFT stage with 18,246 GPT-5.2-generated trajectories and four epochs of training (Section 4.2 and Table 4). Section 6.1 argues that SFT only teaches the format because the SFT checkpoint alone underperforms Search-R1. That result does not exclude a benefit from using this checkpoint to initialize RL. On 7B, ReGain without SFT scores 47.1, below IGPO's 48.8, while full ReGain scores 50.6 (Table 2). A process-reward baseline starting from the same SFT checkpoint, with the same normalization and RL budget, would help resolve this. Extra retries also matter: Appendix A.3 acknowledges additional retrieval calls and generation. The inference ablation measures the combined effect of deletion, feedback, and extra search opportunities. Without a matched search and generation budget, it does not establish the contribution of context deletion.

  2. Appendix C.1's argument that self-probing cannot make things worse misses a case: useful evidence can weaken an incorrect answer without changing the greedy prediction. Suppose the wrong answer's first-token probability falls from 0.9 to 0.6, while the correct alternative rises from 0.1 to 0.4. The wrong answer remains greedy after retrieval. If each of the remaining four probe tokens has conditional probability 0.99 in both contexts, the five-token average log-probabilities are about -0.0291 and -0.1102. Equation 5 then gives a relative change of -2.79, which triggers rollback at the default threshold of 1.0. The rule removes evidence that increased the correct answer's probability and could support later correction. This counterexample challenges the general argument; it does not measure the frequency or final-answer impact of such events in the experiments. Tables 2 and 6 are encouraging, but aggregate gains cannot rule out harmful individual rollbacks. Equation 3 also changes the reasoning/query as well as the retrieved content between its two contexts, so its signal does not isolate the effect of the retrieved documents.

Questions for authors
  1. How does ReGain compare with IGPO, or an equivalent process-reward baseline without rollback, initialized from the exact same SFT checkpoint and using the same reward normalization and RL budget? This comparison would affect my assessment of the claimed training contribution.

  2. Does the five-turn limit include rejected search attempts? Algorithm 1 uses a for-loop and continue, which leaves the same-position retry described in Section 3.4 unclear. Please explain the implemented counting rule. Under matched total retriever calls and generated tokens, how does deletion compare with retaining the rejected context while reformulating the query? This would help separate context removal from the benefit of extra retries.

  3. Please revisit the non-degradation claim and compare GT-based and self-probe trigger decisions on the same saved prefixes, grouped by whether the current answer is correct. How often does self-probing reject a turn that increases ground-truth answer likelihood? For those turns, how does retaining versus removing them affect final correctness? How sensitive are the decisions to the five-token probe length?

  4. What validation data selected checkpoints and the rollback threshold? Figure 3 is labeled as test-set F1, evaluation occurs every two steps, and Section 5.2 calls 50.6 a peak. Do the tables report final checkpoints, validation-selected checkpoints, or best test checkpoints? Please also report uncertainty across training seeds for the main comparison. These details affect how I interpret the 1.8-point average improvement over IGPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.