- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Re:Zero, a framework for learning from all-fail rollout groups in agentic reinforcement learning. To address the lack of effective learning signals from unsuccessful trajectories, Re:Zero recomposes action fragments from different failed rollouts and verifies the resulting trajectories through environment replay. The verified trajectories are subsequently used for continuation-only supervised learning, while preserving the original reinforcement learning objective.
Experiments are conducted across ALFWorld, WebShop, ScienceWorld, and Visual Sokoban, using models ranging from 1.7B to 27B parameters. The results demonstrate that Re:Zero consistently improves GRPO and GiGPO without requiring additional policy decoding.
Critical strengths
-
Effective supervision recovery from failed trajectories. Re:Zero introduces a practical approach to exploiting complementary behaviors across all-fail rollout groups. By recomposing action fragments and verifying them through environment replay, the method constructs additional training supervision without requiring extra policy decoding. The continuation-only learning objective also provides a reasonable mechanism for avoiding incorrect credit assignment to unsuccessful prefixes.
-
Extensive experimental evaluation and ablation studies. The paper evaluates Re:Zero across multiple benchmarks, model scales, and reinforcement learning algorithms, demonstrating consistent performance improvements. The ablation studies on trajectory recomposition, verification, continuation learning, and search strategies provide meaningful evidence supporting the effectiveness of the proposed components.
Critical weaknesses
-
Ambiguity in verification criteria and implementation details. Although Re:Zero emphasizes learning from environment-verified successful trajectories, WebShop and ScienceWorld accept recomposed trajectories with scores above a threshold of 0.8, which does not necessarily correspond to full task completion. In WebShop, the original binary RL reward requires a task score of 1.0. Therefore, some accepted trajectories may not represent genuine successes under the original task objective. Additionally, the completion-queue construction described in Appendix C is insufficiently explained in the all-fail setting, particularly regarding how goal-completing actions can be identified from unsuccessful donor trajectories.
-
Limited generalization evidence and statistical validation. The experiments are conducted entirely in deterministic, resettable environments, leaving the applicability of Re:Zero to stochastic or more complex interactive environments insufficiently explored. Moreover, the paper does not report performance variations across multiple independent training seeds or provide confidence intervals. Some improvements are relatively modest, such as the increase from 42.2% to 44.5% on Visual Sokoban, making their statistical significance unclear. Although Re:Zero avoids additional policy decoding, it introduces substantial environment replay overhead, which may limit its efficiency in computationally expensive environments.
Questions for authors
-
Motivation for Joint RL and SFT Optimization. Figure 1 shows that all-fail rollout groups are particularly prevalent during the early stages of training, which may reflect limitations in sparse reward signals or the initial policy's capabilities. Why do the authors choose to address this issue by introducing an auxiliary SFT objective during RL, rather than exploring more informative reward designs or stronger SFT initialization?More importantly, since the verified recomposed trajectories are used exclusively for continuation-only SFT and do not directly contribute to the RL policy loss, the necessity of integrating this mechanism into the RL training process remains unclear. Could the authors compare the proposed joint optimization approach with an alternative that constructs verified trajectories offline, performs SFT, and subsequently conducts standard RL? Such a comparison would help clarify whether the improvements arise from the online integration with RL or simply from additional high-quality supervised data.
-
Verification Criteria. In WebShop and ScienceWorld, Re:Zero accepts recomposed trajectories with scores above 0.8. What proportion of these trajectories actually satisfy the original task success criteria? Could the authors provide additional results using strict task completion as the verification condition?
-
Completion Queue Construction. Appendix C describes the completion queue as containing fragments whose final actions satisfy task goals. However, these fragments originate from all-fail rollout groups. Could the authors clarify how such completion actions are identified and how the completion queue is constructed in practice?
-
Generalization and Computational Efficiency. How would Re:Zero extend to stochastic environments where exact trajectory replay is unavailable? Furthermore, although the paper provides matched environment-step comparisons, could the authors evaluate the method under matched total training time or computational budgets to better demonstrate its overall efficiency?
If the aforementioned Questions are all resolved, I will consider raising my rating.
5 comments
Response to Reviewer Kpgb (Part B)
Critical Weakness 1 & Question 3 (Completion Queue Construction, §3.3 & App. C, in teal)
We have formalized the queue construction rules in Section 3.3 and Appendix C: a donor rollout in an all-fail group never completed the full task, and completion-queue () membership never assumes donor success. Rather, with probability , search prioritizes using syntactic terminal or goal-delivery action patterns:
- WebShop (
click[buy now]): A donor rollout fails when it purchases an item with mismatched attributes, yet its final command is stillclick[buy now](79.3% of failed parent rollouts in our audit terminate withclick[buy now]). Donor fragments ending inclick[buy now]populate so the spliced candidate ends with a purchase attempt, which is then verified from scratch by environment replay (§3.4). - ALFWorld & ScienceWorld (Goal-Delivery Suffixes): Where no single checkout keyword exists, prioritizes donor suffixes whose final action applies a goal-delivery or manipulation verb to the target object/receptacle/substance named in the task instruction (e.g.,
put <target_obj> in/on <target_receptacle>in ALFWorld, or actions triggering a positive sub-goal progress increment in ScienceWorld), while remaining fragments enter the general queue ranked by goal-word overlap.
Question 1 (Motivation for Joint RL and SFT Optimization & Offline vs. Online Comparison)
- Why Auxiliary Continuation SFT Rather Than Direct RL Loss or Reward Shaping: Donor log-probabilities were computed under the donor's original history rather than the replayed history , so they do not represent valid behavior probabilities for on-policy PPO importance ratios, whereas the continuation loss cleanly supervises verified continuations conditioned on . Moreover, on ScienceWorld and Visual Sokoban—which already use native shaped rewards—Re:Zero still adds +7.29 pp and +2.34 pp over GiGPO by supplying verified goal-reaching demonstrations.
- Comparison with Offline Re:Zero SFT Standard RL (
Qwen3-1.7B, 3 Seeds): Following your proposed experimental design, we compared (i) Offline Re:Zero SFT Standard GiGPO (constructing verified trajectories offline from initial policy rollouts, performing SFT, and subsequently running 150 updates of standard GiGPO), (ii) Fixed Offline Verified Corpus + Interleaved GiGPO, and (iii) Online Re:Zero + GiGPO:
| Training Paradigm (Qwen3-1.7B, 3 Seeds) | ALFWorld Seen /128 (%) | WebShop Succ /128 (%) | SciWorld Succ /128 (%) |
|---|---|---|---|
| Base GiGPO (Standard RL Reference) | (110,113,113) | (79,82,80) | (23,26,25) |
| (i) Offline Re:Zero SFT Standard GiGPO | (114,117,116) | (83,86,84) | (28,31,28) |
| (ii) Fixed Verified Corpus + Interleaved GiGPO | (115,118,117) | (85,88,85) | (29,32,30) |
| (iii) Online Re:Zero + GiGPO (Joint Online, Ours) | (118,121,121) | (87,91,89) | (32,36,34) |
All three augmented conditions use matched collection/verification budgets, auxiliary sample caps (), and 150 RL updates. Online integration continuously refreshes verified continuations as the policy evolves during RL, outperforming Offline SFT RL by +3.39 to +3.91 pp and Fixed-Corpus Interleaving (which retains joint optimization but removes online refresh) by +2.34 to +2.86 pp.
Response to Reviewer Kpgb (Part C)
Critical Weakness 2 & Question 4 (Stochastic Regimes, Multi-Seed Statistics & Matched Wall-Clock Efficiency)
1. Extension to Stochastic (3-Pass Consensus) & Non-Replayable (Critic-Filtered) Regimes
We evaluated Re:Zero under 15% transition stochasticity (action-slip noise in ALFWorld; stochastic search/action noise in WebShop). Relaxing single-pass audit to 3-Pass Consensus Verification (re-executing candidates across 3 independent seeds and accepting those succeeding in executions) or using a Zero-Replay Surrogate (a replay-free baseline that filters candidate splices via an offline LLM rubric critic and trains on Donor context with 0 environment replay steps) both preserve clear gains over the 128 evaluation tasks:
| Environment & Regime (Qwen3-1.7B, 3 Seeds, /128) | Base GiGPO (%) | + Zero-Replay Surrogate (%) | + Consensus Re:Zero (3-Pass, %) |
|---|---|---|---|
| Stochastic ALFWorld (15% Transition Noise) | (100,103,102) | (105,108,107; +3.91 pp) | (108,111,110; +6.25 pp) |
| Stochastic WebShop (15% Search/Action Noise) | (70,73,73) | (75,78,77; +3.65 pp) | (78,81,79; +5.73 pp) |
2. Multi-Seed Statistics & Matched Wall-Clock Time Comparison (§4.2 & Appendix G, in purple)
Tables 9–10 in Appendix G report 3 independent training seeds across the fixed evaluation suites (the 128 Seen and 134 Unseen ALFWorld splits, and the 128 fixed held-out tasks on WebShop, ScienceWorld, and Visual Sokoban). Additionally, on WebShop we compare under an equal 21.6-hour wall-clock budget on identical hardware (where Base GiGPO completes 174 RL updates vs. 150 for Re:Zero):
| Benchmark / Comparison (3 Independent Seeds) | Base GRPO (%) | GRPO + Re:Zero (%) | Base GiGPO (%) | GiGPO + Re:Zero (%) |
|---|---|---|---|---|
| ALFWorld Fixed Seen (/128, 1.7B) | (93,99,96) | (104,108,106) | (110,113,113) | (118,121,121) |
| ALFWorld Full Unseen (/134, 1.7B) | (91,96,88) | (102,106,104) | (109,114,112) | (120,124,121) |
| WebShop Fixed (/128, 1.7B, 150 updates) | (71,75,73) | (84,87,84) | (79,82,80) | (87,91,89) |
| WebShop Matched Wall-Clock (21.6h, 1.7B) | -- | -- | (81,84,81; 174 updates) | (87,91,89; +5.47 pp) |
| ScienceWorld Fixed (/128, 1.7B) | (19,23,21) | (26,29,26) | (23,26,25) | (32,36,34) |
| Visual Sokoban Fixed (/128, 4B) | -- | -- | (53,55,54) | (56,59,56) |
Response to Reviewer Kpgb (Part A)
We thank Reviewer Kpgb for the constructive feedback. Below, we address every question across Parts A, B, and C. In the revised PDF, verification and multi-seed updates are highlighted in purple, queue specifications in teal, and scaling analyses in blue.
Critical Weakness 1 & Question 2 (Verification Criteria: Threshold 0.8 vs. Strict Completion 1.0)
1. Distinguishing Auxiliary Acceptance (0.8) from Native Task Success (§3.4 & App. C)
Across all benchmarks, reported test success rates require native task completion (score=1.0), while WebShop and ScienceWorld additionally report normalized task scores. On WebShop, the base RL reward remains binary (10 if score=1.0, else 0; ScienceWorld retains its native shaped RL return). The threshold τ(rescue)=0.8 governs only which replayed continuations enter the auxiliary continuation loss L(cont). In Section 3.4 and Appendix C, we strictly reserve "verified strict successes" for native 1.0 completions and refer to accepted episodes scoring in [0.8,1.0) as "environment-validated high-score continuations."
2. Proportion of Strict (1.0) vs. High-Score ([0.8, 1.0)) Trajectories & Concrete Case (§4.3 & App. C)
Auditing the 150-step
Qwen3-1.7Bruns on WebShop (883 eligible all-fail pools, 319 rescued pools [36.1%] yielding 345 accepted episodes) and ScienceWorld (H=30; 1,120 pools, 324 rescued pools [28.9%] yielding 348 accepted episodes) confirms that the majority of accepted episodes achieve native strict completion (score=1.0):click[leopard],click[9], andclick[buy now](score=0.8, reward =0). Splicing Parent B's option-selection and checkout fragment onto Parent A's prefix produced a native strict success (score=1.0) under replay.3. Full 150-Step Training: Strict Completion Only (1.0) vs. Threshold (0.8) (§4.3)
We trained
Qwen3-1.7Bfor 150 steps across 3 independent seeds on WebShop and ScienceWorld (128 held-out test tasks):