acceptodds
Under review as a conference paper at ICLR 2027

Consensus-Breaking Re-rollout: Recovering Missing Action Contrasts for Long-Horizon Agent Reinforcement Learning

Abstract

Reinforcement learning for long-horizon language agents relies on limited rollout budgets, yet group-relative objectives obtain no learning signal from groups in which every rollout fails. Existing exploration strategies decide where to spend extra samples based on reward diversity or policy uncertainty. This overlooks a structural signal inside failed groups. We identify this signal as failure-conditioned consensus, where failed rollouts repeatedly select the same action at a shared decision point. Such agreement does not show that the action is correct. It marks a decision whose alternatives the current budget has never tested. To exploit this signal, we propose Consensus-Breaking Re-rollout (CBR), a budget-aware exploration framework that turns failed consensus into missing action contrasts. CBR operates through two complementary mechanisms. First, it locates consensus anchors within all-failure groups and re-rolls from these anchors while excluding the consensus action, so that alternatives are sampled within the support of the current policy. Second, it merges the recovered outcomes into the original group for two-level group-relative credit assignment. CBR requires neither process supervision nor a learned critic. In a fixed-policy diagnostic with two additional requests per all-failure group, CBR recovers a successful trajectory in of groups, compared with for ordinary retries and for entropy-guided branching. Across ALFWorld and WebShop with 1.5B- and 7B-parameter agents, CBR improves its base learner, GiGPO, by – percentage points at a matched generation budget. It also remains competitive with recent credit-assignment methods. These results validate failed consensus as an effective signal for allocating exploration in long-horizon agent learning.

Primary area:
reinforcement learning
Keywords:
Large Language Model Agents, Reinforcement Learning, Long-Horizon Tasks, Exploration, Credit Assignment, Group-Relative Policy Optimization, Rollout Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W8UNOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes consensus-breaking re-rollout (CBR). It finds consensus anchors (shared states in failed rollouts, where the same action was chosen), to then replay those anchors and sample different actions. The paper reports gains over GiGPO on ALFWorld and WebShop.

Critical strengths

(1) The idea is simple, compelling and well-motivated: Identifying common state-action pairs that led to failure and selecting different actions seems to be a fitting approach to inform exploration.

(2) The posed question is highly relevant: Identifying how to spend extra rollout budget to maximize information gain is a relevant and timely question.

Critical weaknesses

(1) This submission does not report the experimental setup and the origin of their empirical results to a sufficient level of clarity. This submission further does not provide code. See questions for further specifics.

(2) The authors propose a method, that is limited to a very specific setting (anchors are comparable; there is an alternative action that is better than the selected action; terminal rewards are sufficiently meaningful) and targets a very specific failure case (repetition of a single, non-useful action in a given state). While mentioned in the paper, I feel that this aspect may be more limiting than the paper suggests, and that it is not sufficiently discussed (also, there is no dedicated Limitations section). In consequence, I am concerned that the proposed solution may lack the generality or wide applicability required to meaningfully advance the field.

Questions for authors

(1) Could the authors please provide the link to an anonymous GitHub repository that contains their source code to enable the full reproducibility of all results?

(2) Could the authors please clarify which of their reported results were generated vs. reported from other work? Please, also elaborate what "source-reported" means in this context.

(3) Could the authors please elaborate which variant of GiGPO is used, normalization with or without standard deviation, and why?

(4) The formulation of revisits seems to allow revisitation within the same trajectory. How often did that occur in practice? Would it make sense to limit counts to distinct trajectories, or treat revisitations within a single trajectory and across trajectories differently?

(5) Could the authors provide a real example from their experiments that showcases a single instance where a repeated action in a given state stalled progress, and how selection of an alternative method helped?

(6) Could you please elaborate upon the link of consistently choosing an action and the rollout failing? It is not obvious to me that a subsequent failure be attributed reliably to this precise action.

(7) Given the limitations of their method, could the authors please strengthen the argument that their method is sufficiently general and widely applicable?

Reviewer_DSHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CBR targets rollout groups in which every trajectory fails, leaving no reward contrast for terminal-reward learning. It finds decision points with repeated agreement on one action, resamples there while excluding that action, and adds the resulting trajectories to GiGPO training. Fixed-policy diagnostics test where to explore, followed by learning experiments. Across four ALFWorld/WebShop settings with 1.5B/7B models, CBR improves over GiGPO by 4.4–8.1 percentage points under matched generation-token budgets.

Critical strengths
  1. this work tests a concrete question well: after failure, where should additional exploration be spent? Table 2 compares selection rules on the same failed groups with the same number of extra requests. Rescue is 39.6% for CBR, 15.6% for ordinary retries, and 27.3% for entropy-guided branching. The advantage over entropy selection is supported by a confidence interval. This diagnostic is more informative about the mechanism than final training scores alone. 2. The cost controls strengthen the learning results. Generation budgets include extra rollouts and rejected proposals, with replay steps and GPU time reported separately. Gains occur in all four main settings, and selection and credit-assignment ablations help rule out explanations based only on more samples or extra loss weight.
Critical weaknesses
  1. The main text should qualify its on-policy description. Excluding the consensus action changes the sampling distribution from π_old to q_old; remaining within the original policy support does not remove this difference. Appendix B.3 is clearer about the limits of local reweighting, including approximating an action-class probability with one surface form. Removing this multiplier costs 2.8 points in Table 10, so the training correction deserves explicit treatment as part of the method. 2. the provenance of Table 3's baselines is unclear. Section 4.1 describes shared settings and a fixed budget, whereas Section 4.4 calls strong baselines source-reported.
Questions for authors
  1. Which Table 3 baselines were retrained under the 12.0B-token protocol, and which were taken from their original papers? Please provide the task splits, seeds, and key optimization settings. 2. How much does the single-surface-form approximation typically underestimate consensus-class probability? Does the behavior-log-probability variant in Table 10 use the same approximation? This would help explain the reweighting results and their sensitivity to action wording.
Reviewer_iqPCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CBR, which explores alternative actions at shared decision points where failed agent trajectories repeatedly agree. It improves learning under matched generation budgets on ALFWorld and WebShop.

Critical strengths
  1. The paper identifies failed action consensus as a useful signal for targeted exploration.
  2. Experiments show consistent gains across two environments and two model sizes under matched generation budgets.
  3. Diagnostic studies and ablations help distinguish the contributions of exploration and credit assignment.
Critical weaknesses
  1. CBR changes which tasks and actions are sampled, but its correction only addresses the action sampled at the intervention step. It remains unclear how this affects the overall policy update.
  2. Evaluation is concentrated on text-based environments, leaving applicability to more stochastic settings uncertain. Also, the state-based counting may not be applied to more complex environments where states are not discrete or countable.
  3. A single rollout repeating the same action can meet the paper’s consensus criterion. Additional experiments are needed to distinguish the benefit of finding agreement across rollouts from simply helping the agent escape loops.
  4. When CBR groups steps by observation, it may compare actions taken in different situations, making the step-level reward signal less reliable. The paper studies this risk, but its impact during normal training remains unclear.
Questions for authors
  1. How do exact-history matching and observation-based matching compare in final training performance?
  2. How much of CBR’s improvement comes from breaking repetitive loops, rather than finding shared action choices across different rollouts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.