- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes RegPO, a policy optimization method for multi-turn LM agents that focuses on step-level credit assignment. The authors argue that Tree-GRPO’s intra-tree advantage effectively compares actions at a shared prefix by their mean return if that action were taken under the current policy. That comparison can rank an action with a better achievable continuation below a better alternative, and so lower the probability of selecting it. RegPO instead estimates, at selected branching prefixes, each sampled action’s value as the best continuation found in a sampled rollout subtree. For the LM experiments, this uses a pseudo-posterior estimator that extrapolates to unsampled continuations. Comparing that value with the prefix-wise maximum gives a proxy for the action’s regret-based advantage. After centering, this term is added to the Tree-GRPO advantage.
The paper motivates the approach with a regret decomposition and a learning-dynamics analysis in a constructed “Hard-Safe” tree MDP. Experiments on Hard-Safe and random tree MDPs examine how the advantage constructions affect learning. The method is then evaluated on four agentic benchmarks: RoboTwin 2.0 (Beat Block Hammer), WebShop, MiniWoB++, and ALFWorld. RegPO achieves the highest point estimate on every reported split, with the largest margins on ALFWorld.
Critical strengths
The core motivation is clear and well argued.
I like that the authors first show how the method behaves in two controlled toy settings (Hard-Safe and random trees) before evaluating it in the environments of interest. This makes the intended mechanism easier to understand. The toy experiments are also run carefully, with many seeds and instances.
The benchmarks are diverse, including a vision-language-action manipulation task alongside text-based agent environments.
Critical weaknesses
-
Baselines are substantially below published results. On ALFWorld with Qwen2.5-1.5B-Instruct, the GiGPO paper (Feng et al., 2025) reports about 72.8% success for GRPO and about 86% for GiGPO, with the same number of training iterations. The reproductions here reach 28.6% and 45.5%. The authors attribute the gap to a reduced episode horizon (30 vs. 50 steps) and response limit (256 vs. 512 tokens). That means the headline ALFWorld margins (+20.2 ID, +29.1 OOD) are measured in a regime where all methods are heavily handicapped, and it is unclear whether they would persist under standard settings. On WebShop, where the reproduction is closer to published numbers, RegPO (65.9) falls within the range of published GiGPO results (65.0–67.4) rather than clearly above it.
-
Statistical evidence is thin on half the benchmarks. RoboTwin and MiniWoB++ each use a single training seed. MiniWoB++ is evaluated on 64 episodes, so the 2.1-point margin over GiGPO corresponds to roughly 1.3 episodes. It is also smaller than the reported standard deviation, which reflects only evaluation repetitions, not training variance. These results do not support a claim of improvement without additional training seeds.
-
No ablations on the agentic benchmarks. RegPO combines several components: the pseudo-posterior estimator, a centering shift, and an additive regret term with w_reg = 1 on top of Tree-GRPO. None is ablated on the agentic tasks, so the gains cannot be attributed to regret-based feedback specifically. This matters particularly because the random-tree results show pure regret feedback underperforming Tree-GRPO at low branching factors, and the LM experiments use only two sampled actions per branching prefix. In that regime, the benefit may come from the pseudo-posterior extrapolation rather than from the regret signal itself. In addition, the regret term is only ever added to Tree-GRPO, even though GiGPO is the strongest baseline.
-
Clarity and presentation is a major weakness: The description of the method is very dense. I strongly suggest simplifying the main-text presentation and deferring more technical details to the appendix. The caption for Figure 1 is hard to understand and seems to be missing relevant context. The last paragraph of the introduction, which discusses results, is also hard to understand and missing context. For example, what is “the benchmark-specific reporting protocol,” and what do the stated margins mean (relative to which baseline, in what units)? The conclusion, limitations, and related work are all deferred to the appendix, so the main text ends abruptly after the experiments. This is highly atypical; at minimum, the conclusion and limitations should appear in the main body.
Note that while I answered that "I have significant concerns about the paper’s soundness that do not appear easy to address.", I was between this option and option 2. I think you can make this into a strong paper by running substantially more experiments (assuming results hold up) , as the method itself is interesting and makes sense. But doing this, and also improving the clarity of the paper, seems like a tall order.
Questions for authors
Can you report results on ALFWorld under the standard GiGPO configuration (50-step horizon, 512-token responses), so the comparison can be related to published baselines?
Can you provide additional training seeds for RoboTwin and MiniWoB++?
Can you ablate the components of RegPO on at least one or two agentic benchmarks? Specifically: sampled-max vs. pseudo-posterior estimation, with vs. without centering, and a sweep over w_reg.
Does the regret component also improve GRPO or GiGPO when added to them, rather than only Tree-GRPO?
8 comments
Response to Reviewer FK3U (1/3)
W1 + Q1. ALFWorld protocol and the WebShop comparison.
The extended-limit ALFWorld run reaches 84.3% ID / 83.5% OOD at 50 episode steps and 512 response tokens. The original baseline comparison uses 30-step/256-token limits; the extended RegPO run uses the combined settings reported below.
Completed extended-budget ALFWorld run. The final update-200 model reaches 84.3% ID and 83.5% OOD success at 50 episode steps and 512 response tokens (Appendix J.3). This completed run uses the bias-focused posterior tuple ; it changes the estimator settings as well as the interaction and training budgets. The result measures performance under this combined configuration. Extended-limit Tree-GRPO training and the default-parameter RegPO follow-up are ongoing, and we will incorporate their results as they become available. The existing baseline comparisons use the original 30-step/256-token protocol (Appendix I.2.2), so the completed extended-budget RegPO result alone does not establish the margin over baselines under the longer-limit recipe.
On WebShop, 65.9% lies within the published GiGPO range you cite; 62.0% is our in-stack reproduction. Appendix I.2.2 explains the common implementation, prior-work configurations where compatible, and adaptations to prompts, wrappers, parser and budgets. Appendix I.2.1 distinguishes RegPO's final checkpoints on WebShop from baseline selection on the reported splits. These are in-stack comparisons under the documented implementation and checkpoint rules (Appendices I.2.2 and I.2.1).
W2 + Q2. Training variance and additional seeds.
Additional independent training and evaluation are underway for MiniWoB++ GiGPO, Tree-GRPO and RegPO, with one extra seed per method. We will incorporate these results into the manuscript as they become available.
We treat MiniWoB++'s 2.1-percentage-point margin as marginal, given the reported evaluation variation and the single training seed. Our interpretation is limited to the observed point estimate; repeated evaluation of one checkpoint does not measure training-seed variation (Appendix I.2.1).
BBH additional-seed runs await capacity on the compatible VLA server. In our tested setup, the pretrained SFT policy worked on an Intel-based server but not the available AMD-based alternative, restricting replication to compatible hardware with limited capacity. No additional BBH training-seed result is reported here.
BBH's reported margins are larger: RegPO reaches 78.9% ID / 73.4% OOD, versus the strongest baseline GRPO's 64.1% / 67.2%, giving 14.8 / 6.2 percentage points (Table 1(a)). These remain point-estimate gaps from the reported training run; independent replications are needed to quantify training-seed variation.
Response to Reviewer FK3U (2/3)
W3. Which components account for the gains?
Appendix J reports estimator, posterior-parameter, regret-weight, base-removal and scaling contrasts. We also report the BBH centering-off result below.
- Posterior prediction on ALFWorld: replacing predictive values with observed sampled maxima, while retaining the base advantage and available-range centering, gives 37.1% ID / 40.3% OOD, compared with the RegPO point estimates in Table 1 of 65.7% / 62.4%. The reported comparison supports predictive estimation in the evaluated sparse-rollout setting (Appendix J.1.2); Q3 gives its interpretation.
- Posterior settings on Minesweeper: default and lower-concentration settings reach validation peaks of 60.94% and 59.38%, respectively. The latter preserves the posterior means for identical observations while reducing concentration. Each setting has one training run, providing sensitivity measurements for the two tested configurations (Appendix J.1.1 (Table 11)).
- Regret weight with the base fixed: Minesweeper now includes , with validation peaks 53.13%, 60.94%, 55.47%, 55.47%. Unit weight gives the highest observed peak among the tested retained-base settings (Appendix J.2.1 (Table 12)).
- Base removal on BBH: training with only the unit-weight regret component reaches 68.9% success. This provides agentic evidence that the feedback can be useful without the additive Tree-GRPO base (Appendix J.2.3). We report this as a success rate without assigning an ID/OOD-specific margin.
- Centering on BBH: the centering-off variant reaches 65.6% success, as reported in our response to Reviewer E9VM. Q3 discusses this measurement and its scope.
- Whole-base scaling on ALFWorld: the Tree-GRPO control reaches 35.0 ± 10.5% ID across three training seeds, with per-seed best-ID scores of 47.1%, 30.0%, and 27.9% (mean ± sample standard deviation of the reported percentages). All three checkpoints are selected by best ID performance within 150 updates at 30 actions/256 tokens. RegPO’s paper reference is 65.7 ± 1.5% ID, also across three best-ID-selected training seeds. The control uses three base trajectories/one branching prefix, versus four/two in the paper. Appendix J.2.2 (Table 13).
- Whole-base scaling on WebShop: the same doubled-base control improves faster during roughly the first 50 updates. From about update 55 through update 150, RegPO attains higher evaluation success; training success follows the same ordering after update 50, and the two training curves approach each other by update 150 (Appendix J.2.2, Figure 14).
The low-branching synthetic result in Section 4.1.2 (Figure 3) remains specific to its tested configuration; the new agentic evidence helps distinguish that result from the performance of individual components on the reported tasks. Q3 below gives the estimator, centering and weight interpretation, while Q4 addresses adding the term to GRPO or GiGPO.
W4. Presentation and reporting.
The main text now includes a short Tree-GRPO advantage formula and definitions at the end of Section 2, a method overview before Section 3.1, and a clarification of the raw → centered → combined advantage. Figure 1 now explains the current-policy mean versus best achievable continuation and identifies the schematic tree, omitted branches/levels and terminal-return labels. The introduction names the baseline set and reports point-estimate margins in percentage points of success rate. Table 1 now includes Minesweeper and Sokoban with explicit evaluation-repetition accounting. A main-text Conclusion and Limitations summary states the coverage, policy-support, predictive-model and stochastic-maximum restrictions and the absence of an LLM convergence guarantee (Section 5).
W5. Closing soundness note.
The two new benchmarks, estimator/weight/base-removal/scaling studies, reported centering-off result and extended-budget ALFWorld result expand the empirical evidence. The revised main text presents the method alongside these measurements, with the clearer advantage construction, Figure 1 caption, conclusion and limitations requested in W4.
Response to Reviewer FK3U (General Response)
Thank you for your detailed review and for recognizing the method's clear motivation, controlled studies and benchmark diversity. You identified substantially more experimental evidence and clearer presentation as the main requirements for making this a strong paper. We have organized the revision around these two priorities, expanding the evaluation from four to six agentic benchmarks, adding component ablations on agentic tasks, and simplifying the main-text explanation.
Please also see the updated manuscript and the revisions highlighted in color.
The main additions are:
Additional MiniWoB++ training-seed runs for GiGPO, Tree-GRPO and RegPO are ongoing, while BBH replications await capacity on the compatible VLA server. We will incorporate these results as they become available; W2/Q2 explains the current evidence and the limited interpretation of the MiniWoB++ margin.
Together, the expanded benchmark evaluation, agentic component studies and presentation revisions directly address the experimental and clarity priorities you outlined. We would be grateful if you would reassess the revised paper in light of these additions. The detailed replies below follow your original weakness/question order, with joint W1/Q1 and W2/Q2 answers.