- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PROVER, which adds segment-level credit on top of GRPO for LLM agents. For groups with one to three successes out of eight, an external LLM judge (gpt-5.4-mini with two trajectory-search tools) compares the shortest successful trajectory with the failed ones and picks a non-terminal segment of at most four turns. The environment is then restored to the states before and after the segment, and K = 8 continuations are sampled from each. If the success rate after the segment is higher, the difference is added to the GRPO advantage of the segment's tokens. On ALFWorld, WebShop and SearchQA with Qwen3.5-2B/4B (100 training steps), the average relative gain over GRPO is 9.91% and 7.12%.
Critical strengths
-
The split between proposing and verifying is clean. The judge only decides where to spend rollouts, and the size of the bonus comes from environment outcomes.
-
The controls are sensible: Budget-Matched GRPO, random segment selection, four judges (including a local Qwen3.5-9B), and costs in tokens, API spend and wall-clock time.
-
All prompts are given, and Appendix B is open about the bias of the positive-part estimator. The numbers in the tables are internally consistent.
Critical weaknesses
My main concerns are that the gains are not separated from training noise, and that the key claim (verifying beats trusting the judge) is not tested directly.
-
Gains vs. noise. Table 1 reports std over three evaluation runs, and no training seeds are mentioned, so each setting appears to be trained once. The paper’s own controls already differ from GRPO by over 3 points (Budget-Matched GRPO: −3.5 on 2B WebShop; Random selection: −3.3 on 4B WebShop), comparable to several reported gains. The intervention is also sparse: Table 4 implies about 0.7–3.2 valid proposals per step (0.6–2.5% of 128 trajectories), and only some are accepted. On 4B ALFWorld, fewer than one trajectory per step gets a bonus, yet the failure rate drops from 6.2% to 1.7% (about 6 of 134 games). The paper does not explain how such a sparse signal yields gains of this size.
-
Verification is not ablated. No run gives the judge’s segment a fixed bonus without boundary rollouts, and CriticSearch computes credit differently, so it is unclear whether verification matters. Keeping only positive estimates also gives a bonus when the true advantage is zero (up to about +0.1 in expectation with K = 8), so part of the gain may come from simply up-weighting successful trajectories. λ, K, the eligibility rule and the judge’s tools are not ablated.
-
Mismatch with the motivation. The introduction attributes the problem to “reinforcing mistakes in successful trajectories and penalizing useful progress in failed ones”, but PROVER changes neither: every token keeps its GRPO advantage, failed trajectories are untouched, and negative estimates are dropped. It only adds credit to a few turns of one successful trajectory.
-
Baselines, cost and reproducibility. Several reimplemented baselines fall well below GRPO (SPO-tree by 34 points on 2B ALFWorld, GiGPO by 7 on 4B WebShop), which weakens the comparison. No code is provided, and the evaluated checkpoint and evaluation decoding settings are not stated.
Questions for authors
-
Could you add (a) a run where the judge’s segment gets a fixed bonus (e.g., the mean accepted Δ) without boundary rollouts, and (b) a run that keeps negative estimates? If (a) matches PROVER, verification is not what drives the gain.
-
Per step, how many groups are eligible, how many proposals are valid and accepted, and what fraction of policy tokens gets a bonus? In Table 3, are invalid random intervals counted as rejections?
-
SPO-tree reaches 92.30 on 4B ALFWorld but only 56.96 on 2B. GiGPO is 4.8 points below GRPO on 4B WebShop, although the GiGPO paper reports gains over GRPO on WebShop. What explains these gaps, and how were these baselines tuned?
-
Section 3.4 says boundary continuations use the policy that generated the source group. Since Prime-RL can train asynchronously, how is this enforced? Can you provide code and weights for review?
5 comments
Response to Reviewer xLLu (Part 2/2)
W3: mismatch with the motivation.
You are right that ProVer does not reduce credit for mistakes in successful trajectories or change failed ones, and the introduction is not precise about this. That sentence was meant to describe the general limitation of uniform trajectory-level credit, of which ProVer addresses one part. Adding verified credit to a pivotal segment raises its weight relative to the rest of the same trajectory, so the remaining tokens, including any mistakes, are reinforced comparatively less. We will revise the introduction to make ProVer's scope explicit.
W4 / Q3: baselines.
We understand the concern about baselines that fall below GRPO. We adapt the existing codebases for the baselines to our training infrastructure and change only the experimental settings, so that all methods are compared in a controlled study. All baselines share GRPO's backbone, optimizer, prompts, environment servers and batch, as described in Appendix A.2, and method-specific hyperparameters follow published defaults with no tuning per environment or model scale.
The baseline configurations are given in Appendix A.3. On the two gaps you ask about: GiGPO's published WebShop gains were obtained in a different setting (backbone, training budget and reward), so they need not carry over to ours.
For SPO-tree on 2B ALFWorld, we believe the tree interacts poorly with a weaker policy on a long-horizon task: its leaves share prefixes, which reduces early exploration, and sibling-relative advantages give no learning signal when both children fail, which is more common at 2B than at 4B.
We also note that our main conclusions rest primarily on controls that share ProVer's exact pipeline: GRPO, Budget-Matched GRPO, random selection, and the four new ablations in the Component Ablations Comment.
Q4: fixed policy, code and evaluation settings.
Section 3.4 specifies that the policy that generated the source group is held fixed during verification. In Prime-RL this is enforced by the orchestrator, which allows zero off-policy steps: a training wave is dispatched only after every group has finished its judge call and boundary continuations, continuations use the same inference server and policy version as the source group, and groups from an older policy version are discarded.
For evaluation, every method uses the final checkpoint after the 100 optimizer steps stated in Section 4.3, with no checkpoint selection, and identical decoding: temperature 0.4 and at most 512 tokens per turn on ALFWorld and WebShop; temperature 1.0, top-p 1.0 and at most 256 tokens per turn on SearchQA.
Releasing the code and model weights requires institutional approval. We will release both with the camera-ready version.
Thank you again for the detailed feedback. We hope the new ablations, the additional training seeds and these clarifications address your concerns, and we would be glad to answer any further questions.
Dear Reviewer xLLu
Dear Reviewer xLLu,
Thank you again for your detailed review. We wanted to follow up on our two-part response. It includes additional training seeds for GRPO and ProVer (W1), the fixed-bonus and Signed Δ variants you proposed (W2 / Q1), and the per-step proposal counts (Q2), with results in the Component Ablations Comment and the Multiple Training Seed Comment. We also responded to the motivation (W3), the baselines (W4 / Q3), and the implementation and evaluation details (Q4).
We would be grateful to know whether these address your concerns, and we would be glad to clarify anything further during the discussion period.
Best regards, The authors
Response to Reviewer xLLu (Part 1/2)
Thank you for the detailed and careful review, which helped us tighten both the experiments and the claims. We answer each point below.
W1: gains vs. training noise.
To measure the training noise, we trained GRPO and ProVer with two additional training seeds on 4B SearchQA (Multiple Training Seed Comment). ProVer's improvement holds on every seed, and the gap is about seven times GRPO's training-seed standard deviation. The deviations you point out (Budget-Matched GRPO on 2B WebShop and random selection on 4B WebShop) are both below GRPO, while ProVer is above GRPO on WebShop at both scales and in all six settings of Table 1. We will extend the additional training seeds to the remaining settings in the final version.
W1: sparsity of the intervention.
You are right that the intervention is sparse, and the paper should explain more clearly why it can still matter. The per-step counts are given under Q2 below. The signal is large where it applies: an eligible successful trajectory has a GRPO advantage of 0.625–0.875, and the mean accepted bonus adds 0.33–0.56 on the segment's tokens. It is also targeted: bonuses go only to tasks the policy solves less than half the time, where extra learning signal matters most, and only to segments whose effect on the success rate was measured. The controls suggest that this targeting is what matters: the same rollout budget spent on additional GRPO trajectories (Budget-Matched GRPO, Appendix A.3) or on randomly selected segments does not reproduce the gains.
Q2: per-step counts and Table 3.
Per optimizer step (16 groups, 128 trajectories):
At 4B, 0.02% (ALFWorld), 0.40% (WebShop) and 0.18% (SearchQA) of policy tokens receive a bonus.
In Table 3, invalid random intervals are not counted as rejections. Random selection gets up to three attempts to sample a valid segment; if none is valid, the group is discarded and receives no additional credit (Appendix A.3). The acceptance rate is computed over verified proposals only, for random selection and the judges alike.
W2 / Q1: is verification what drives the gain?
We ran both variants, Fixed bonus and Signed Δ (Component Ablations Comment, Findings 1 and 4). As you noted, a fixed bonus that matched ProVer would mean verification is not what drives the gain. It comes close on WebShop, which shows that judge selection contributes a substantial part, but it recovers less than half of ProVer's gain on SearchQA and falls below GRPO on ALFWorld. Without verification, the effect of the same judge-selected credit varies by environment and can be negative. Keeping negative estimates (Signed Δ) gives no improvement over positive-only credit on any benchmark.
W2: does the positive-only rule simply up-weight successful trajectories?
ProVer does add credit to segments of successful trajectories, and the question it addresses is which segments. If up-weighting by itself produced the gain, the location of the credit would not matter. It does. Random selection uses the same verification and the same positive-only rule, and it is the weakest variant on all three benchmarks (Table 3). Entropy-based selection is less consistent than the judge, and a fixed bonus on unverified segments falls below GRPO on ALFWorld (Component Ablations Comment, Findings 1 and 2). The estimates themselves agree: across runs, positive estimates far outnumber negative ones, so the credited segments are mostly ones with a measured positive effect.