acceptodds
Under review as a conference paper at ICLR 2027

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Abstract

Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment of trajectory-level advantages to all policy tokens fails to distinguish consequential decisions from less relevant ones, obscuring which intermediate decisions contributed to success. We introduce ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning. Given a rollout group, an agentic judge contrasts successful and failed trajectories to propose a segment potentially responsible for their divergent outcomes. Rather than directly trusting the judge’s assessment, ProVer verifies the proposed segment by estimating its advantage from the difference in terminal success rates between current-policy continuations sampled before and after the segment. Positive estimates are then incorporated into the GRPO advantages of policy tokens within the proposed segment. By using model judgment only to select where to verify, ProVer grounds local credit in observed outcomes without exhaustively evaluating every intermediate state. Across ALFWorld, WebShop, and SearchQA, ProVer achieves the strongest average performance at both model scales, with relative improvements over GRPO of 9.91% and 7.12% for Qwen3.5-2B and Qwen3.5-4B, respectively. Further analyses demonstrate that informed segment selection improves policy training with modest additional generation overhead, even without a frontier-scale judge model, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, LLM Agent, Credit Assignment, Reinforcement Learning, Agentic Judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xLLuOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PROVER, which adds segment-level credit on top of GRPO for LLM agents. For groups with one to three successes out of eight, an external LLM judge (gpt-5.4-mini with two trajectory-search tools) compares the shortest successful trajectory with the failed ones and picks a non-terminal segment of at most four turns. The environment is then restored to the states before and after the segment, and K = 8 continuations are sampled from each. If the success rate after the segment is higher, the difference is added to the GRPO advantage of the segment's tokens. On ALFWorld, WebShop and SearchQA with Qwen3.5-2B/4B (100 training steps), the average relative gain over GRPO is 9.91% and 7.12%.

Critical strengths
  • The split between proposing and verifying is clean. The judge only decides where to spend rollouts, and the size of the bonus comes from environment outcomes.

  • The controls are sensible: Budget-Matched GRPO, random segment selection, four judges (including a local Qwen3.5-9B), and costs in tokens, API spend and wall-clock time.

  • All prompts are given, and Appendix B is open about the bias of the positive-part estimator. The numbers in the tables are internally consistent.

Critical weaknesses

My main concerns are that the gains are not separated from training noise, and that the key claim (verifying beats trusting the judge) is not tested directly.

  1. Gains vs. noise. Table 1 reports std over three evaluation runs, and no training seeds are mentioned, so each setting appears to be trained once. The paper’s own controls already differ from GRPO by over 3 points (Budget-Matched GRPO: −3.5 on 2B WebShop; Random selection: −3.3 on 4B WebShop), comparable to several reported gains. The intervention is also sparse: Table 4 implies about 0.7–3.2 valid proposals per step (0.6–2.5% of 128 trajectories), and only some are accepted. On 4B ALFWorld, fewer than one trajectory per step gets a bonus, yet the failure rate drops from 6.2% to 1.7% (about 6 of 134 games). The paper does not explain how such a sparse signal yields gains of this size.

  2. Verification is not ablated. No run gives the judge’s segment a fixed bonus without boundary rollouts, and CriticSearch computes credit differently, so it is unclear whether verification matters. Keeping only positive estimates also gives a bonus when the true advantage is zero (up to about +0.1 in expectation with K = 8), so part of the gain may come from simply up-weighting successful trajectories. λ, K, the eligibility rule and the judge’s tools are not ablated.

  3. Mismatch with the motivation. The introduction attributes the problem to “reinforcing mistakes in successful trajectories and penalizing useful progress in failed ones”, but PROVER changes neither: every token keeps its GRPO advantage, failed trajectories are untouched, and negative estimates are dropped. It only adds credit to a few turns of one successful trajectory.

  4. Baselines, cost and reproducibility. Several reimplemented baselines fall well below GRPO (SPO-tree by 34 points on 2B ALFWorld, GiGPO by 7 on 4B WebShop), which weakens the comparison. No code is provided, and the evaluated checkpoint and evaluation decoding settings are not stated.

Questions for authors
  1. Could you add (a) a run where the judge’s segment gets a fixed bonus (e.g., the mean accepted Δ) without boundary rollouts, and (b) a run that keeps negative estimates? If (a) matches PROVER, verification is not what drives the gain.

  2. Per step, how many groups are eligible, how many proposals are valid and accepted, and what fraction of policy tokens gets a bonus? In Table 3, are invalid random intervals counted as rejections?

  3. SPO-tree reaches 92.30 on 4B ALFWorld but only 56.96 on 2B. GiGPO is 4.8 points below GRPO on 4B WebShop, although the GiGPO paper reports gains over GRPO on WebShop. What explains these gaps, and how were these baselines tuned?

  4. Section 3.4 says boundary continuations use the policy that generated the source group. Since Prime-RL can train asynchronously, how is this enforced? Can you provide code and weights for review?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xLLu (Part 1/2)

Thank you for the detailed and careful review, which helped us tighten both the experiments and the claims. We answer each point below.

W1: gains vs. training noise.

To measure the training noise, we trained GRPO and ProVer with two additional training seeds on 4B SearchQA (Multiple Training Seed Comment). ProVer's improvement holds on every seed, and the gap is about seven times GRPO's training-seed standard deviation. The deviations you point out (Budget-Matched GRPO on 2B WebShop and random selection on 4B WebShop) are both below GRPO, while ProVer is above GRPO on WebShop at both scales and in all six settings of Table 1. We will extend the additional training seeds to the remaining settings in the final version.

W1: sparsity of the intervention.

You are right that the intervention is sparse, and the paper should explain more clearly why it can still matter. The per-step counts are given under Q2 below. The signal is large where it applies: an eligible successful trajectory has a GRPO advantage of 0.625–0.875, and the mean accepted bonus adds 0.33–0.56 on the segment's tokens. It is also targeted: bonuses go only to tasks the policy solves less than half the time, where extra learning signal matters most, and only to segments whose effect on the success rate was measured. The controls suggest that this targeting is what matters: the same rollout budget spent on additional GRPO trajectories (Budget-Matched GRPO, Appendix A.3) or on randomly selected segments does not reproduce the gains.

Q2: per-step counts and Table 3.

Per optimizer step (16 groups, 128 trajectories):

RunEligible groupsValid proposalsAccepted
2B WebShop2.322.281.33
2B SearchQA2.762.721.63
4B ALFWorld0.630.610.24
4B WebShop2.592.581.76
4B SearchQA1.561.450.90

At 4B, 0.02% (ALFWorld), 0.40% (WebShop) and 0.18% (SearchQA) of policy tokens receive a bonus.

In Table 3, invalid random intervals are not counted as rejections. Random selection gets up to three attempts to sample a valid segment; if none is valid, the group is discarded and receives no additional credit (Appendix A.3). The acceptance rate is computed over verified proposals only, for random selection and the judges alike.

W2 / Q1: is verification what drives the gain?

We ran both variants, Fixed bonus and Signed Δ (Component Ablations Comment, Findings 1 and 4). As you noted, a fixed bonus that matched ProVer would mean verification is not what drives the gain. It comes close on WebShop, which shows that judge selection contributes a substantial part, but it recovers less than half of ProVer's gain on SearchQA and falls below GRPO on ALFWorld. Without verification, the effect of the same judge-selected credit varies by environment and can be negative. Keeping negative estimates (Signed Δ) gives no improvement over positive-only credit on any benchmark.

W2: does the positive-only rule simply up-weight successful trajectories?

ProVer does add credit to segments of successful trajectories, and the question it addresses is which segments. If up-weighting by itself produced the gain, the location of the credit would not matter. It does. Random selection uses the same verification and the same positive-only rule, and it is the weakest variant on all three benchmarks (Table 3). Entropy-based selection is less consistent than the judge, and a fixed bonus on unverified segments falls below GRPO on ALFWorld (Component Ablations Comment, Findings 1 and 2). The estimates themselves agree: across runs, positive estimates far outnumber negative ones, so the credited segments are mostly ones with a measured positive effect.

AuthorsOpenReview ↗

Response to Reviewer xLLu (Part 2/2)

W3: mismatch with the motivation.

You are right that ProVer does not reduce credit for mistakes in successful trajectories or change failed ones, and the introduction is not precise about this. That sentence was meant to describe the general limitation of uniform trajectory-level credit, of which ProVer addresses one part. Adding verified credit to a pivotal segment raises its weight relative to the rest of the same trajectory, so the remaining tokens, including any mistakes, are reinforced comparatively less. We will revise the introduction to make ProVer's scope explicit.

W4 / Q3: baselines.

We understand the concern about baselines that fall below GRPO. We adapt the existing codebases for the baselines to our training infrastructure and change only the experimental settings, so that all methods are compared in a controlled study. All baselines share GRPO's backbone, optimizer, prompts, environment servers and batch, as described in Appendix A.2, and method-specific hyperparameters follow published defaults with no tuning per environment or model scale.

The baseline configurations are given in Appendix A.3. On the two gaps you ask about: GiGPO's published WebShop gains were obtained in a different setting (backbone, training budget and reward), so they need not carry over to ours.

For SPO-tree on 2B ALFWorld, we believe the tree interacts poorly with a weaker policy on a long-horizon task: its leaves share prefixes, which reduces early exploration, and sibling-relative advantages give no learning signal when both children fail, which is more common at 2B than at 4B.

We also note that our main conclusions rest primarily on controls that share ProVer's exact pipeline: GRPO, Budget-Matched GRPO, random selection, and the four new ablations in the Component Ablations Comment.

Q4: fixed policy, code and evaluation settings.

Section 3.4 specifies that the policy that generated the source group is held fixed during verification. In Prime-RL this is enforced by the orchestrator, which allows zero off-policy steps: a training wave is dispatched only after every group has finished its judge call and boundary continuations, continuations use the same inference server and policy version as the source group, and groups from an older policy version are discarded.

For evaluation, every method uses the final checkpoint after the 100 optimizer steps stated in Section 4.3, with no checkpoint selection, and identical decoding: temperature 0.4 and at most 512 tokens per turn on ALFWorld and WebShop; temperature 1.0, top-p 1.0 and at most 256 tokens per turn on SearchQA.

Releasing the code and model weights requires institutional approval. We will release both with the camera-ready version.

Thank you again for the detailed feedback. We hope the new ablations, the additional training seeds and these clarifications address your concerns, and we would be glad to answer any further questions.

AuthorsOpenReview ↗

Dear Reviewer xLLu

Dear Reviewer xLLu,

Thank you again for your detailed review. We wanted to follow up on our two-part response. It includes additional training seeds for GRPO and ProVer (W1), the fixed-bonus and Signed Δ variants you proposed (W2 / Q1), and the per-step proposal counts (Q2), with results in the Component Ablations Comment and the Multiple Training Seed Comment. We also responded to the motivation (W3), the baselines (W4 / Q3), and the implementation and evaluation details (Q4).

We would be grateful to know whether these address your concerns, and we would be glad to clarify anything further during the discussion period.

Best regards, The authors

Reviewer_MtsWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProVer, a framework for improving credit assignment in agentic reinforcement learning. An LLM judge compares successful and failed trajectories and selects a short segment that may explain their different outcomes. The method then samples continuations from the states before and after that segment, using the difference in success rates to assign extra credit. Across three environments and two model sizes, ProVer achieves the best average performance among the evaluated methods and consistently improves over GRPO.

Critical strengths

The separation between segment selection and credit estimation is a sensible design choice. Rather than asking the judge to provide numerical rewards directly, ProVer uses it to focus the rollout budget on potentially useful decisions. This makes the approach appealing when evaluating every intermediate state would be expensive.

The empirical results are encouraging. Improvements over GRPO appear across all six model–environment settings, and the comparisons include both additional-rollout controls and several methods for finer-grained credit assignment. The experiments with smaller judges also suggest that the approach does not depend on using the strongest available judge.

Critical weaknesses

With only eight continuations per boundary, the estimated advantage can be noisy. Since ProVer keeps positive estimates and discards negative ones, even a segment with no real benefit can receive extra credit. Appendix B explicitly acknowledges this bias, which is helpful, but the experiments leave its practical importance unclear. A comparison across different verification budgets would make the credit signal more convincing.

It is not yet clear how much the verification step contributes beyond reinforcing behavior selected by the judge. The random-selection baseline tests whether informed selection helps, but it does not answer this separate question. Giving the same judge-selected segments a matched fixed bonus would provide a useful control: if that performs similarly, the explanation for the gains would be different.

The uncertainty reporting leaves some doubt about training stability. Table 1 describes three evaluation runs without clearly stating whether each uses an independently trained policy. Repeated evaluation of one checkpoint would not capture variation from training. Table 3 also reports the judge comparisons without uncertainty estimates, making the smaller differences difficult to interpret.

Questions for authors

Have you compared rollout-based verification with a matched fixed bonus on the same judge-selected segments? Does increasing the number of continuations materially change the results?

Do the three evaluation runs correspond to independent training seeds, and how stable are the judge comparisons across training runs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MtsW

Thank you for the review. We answer each question below.

W2 / Q1: fixed bonus on the same judge-selected segments.

Thank you for suggesting this control. We ran the Fixed bonus ablation, and the results are in Finding 1 of the Component Ablations Comment. It comes close to ProVer on WebShop, which confirms that judge selection contributes, but it falls short on all three benchmarks. Reinforcing judge-selected behavior alone therefore does not fully explain the gains, and on ALFWorld it lowers performance. Verification is what makes that credit reliable.

W1 / Q1: noise and positive-part bias at K = 8.

This is a valid concern. On the noise, we agree that a larger K, such as K = 16, would give more accurate estimates: by the 1/K variance rate in Appendix B, it would halve the variance of each estimate. It would also double the rollouts spent on verification. Given our computational resources, we fixed K = 8, which matches the group size commonly used in the GRPO literature.

On the bias, we tested its practical importance by running the unbiased alternative. Signed Δ applies negative estimates as penalties instead of discarding them (Component Ablations Comment, Finding 4). It gives no improvement over the positive-only rule on any benchmark and falls close to GRPO on all three. In practice, then, discarding negative estimates is the better choice despite its bias. A plausible explanation is that the judge proposes segments it expects to have helped, so a negative estimate marks a proposal that failed verification, not a segment identified as harmful, and penalizing it gives the policy a signal the propose stage is not designed to provide.

Q2 / W3: training seeds.

The three runs in Table 1 are evaluation runs of one trained policy per setting, so they do not capture training variance. To measure it, we trained GRPO and ProVer with two additional training seeds on 4B SearchQA (Multiple Training Seed Comment). Across three training seeds, GRPO reaches 40.89 ± 0.88 and ProVer 47.08 ± 0.08. The gap of 6.2 points is about seven times GRPO's training-seed standard deviation, and the lowest ProVer seed exceeds the highest GRPO seed by more than 5 points, so the improvement is stable across training runs. We will extend the additional seeds to the remaining settings in the final version.

W3: uncertainty in Table 3.

We agree that small differences in Table 3 are hard to interpret without uncertainty estimates. The conclusion we draw from it is that every judge outperforms random selection on all three benchmarks, which holds in all twelve comparisons (the random procedure is described in Appendix A.3). Table 3 reports single evaluations except for the gpt-5.4-mini row, so we do not rank the judges by small differences, and we will report uncertainty for all rows in the final version.

We hope the fixed-bonus control, the additional training seeds and the discussion of the positive-only rule address your concerns. We would be glad to answer any further questions.

AuthorsOpenReview ↗

Dear Reviewer MtsW

Dear Reviewer MtsW,

Thank you again for your review. We wanted to follow up on our response, which includes the fixed-bonus control you suggested (W2 / Q1) and additional training seeds for GRPO and ProVer (W3 / Q2), reported in the Component Ablations Comment and the Multiple Training Seed Comment. We also addressed the noise and bias of the estimate at K = 8 (W1) and the uncertainty in Table 3.

We would be grateful to know whether these address your concerns, and we would be glad to clarify anything further during the discussion period.

Best regards, The authors

AuthorsOpenReview ↗

Dear Reviewer MtsW

Dear Reviewer MtsW,

As the discussion period closes today, we wanted to check in one last time. We would be grateful if you could let us know whether our response addressed your concerns, and we are happy to answer any remaining questions today.

Reviewer_ahXJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces ProVer, a GRPO-based method for improving performance on agentic tasks. ProVer uses an external judge model to select a segment from a successful rollout and samples additional continuations to estimate its advantage. Experiments on ALFWorld, WebShop, and SearchQA demonstrate its effectiveness compared with several baselines.

Critical strengths
  1. The experiments are generally solid, covering 2 model sizes, 3 benchmarks, and multiple baselines, along with comparisons across different judge models. The authors also address potential concerns about the additional computational cost of the judge model and the runtime overhead introduced by the extra rollout and verification steps.

  2. The paper is clearly written and easy to follow, with well-organized explanations and detailed reporting of the experimental setup.

Critical weaknesses
  1. Limited methodological novelty. ProVer builds on existing ideas from outcome-based segment-level credit assignment (e.g., SPO) and LLM-guided credit assignment (e.g., CriticSearch). While their combination is reasonable, the paper could better distinguish its contributions from prior work. Related methods using selective or entropy-guided branching, such as ARPO and AT²PO, also warrant discussion.

  2. Insufficient isolation of the contribution of MC verification. The comparisons with CriticSearch and random segment selection are informative, but they do not fully isolate the benefit of outcome-based verification. In particular, it remains unclear whether the improvements mainly arise from selecting promising segments or from estimating their advantages through additional MC rollouts.

  3. Limited flexibility of credit assignment. ProVer selects only one contiguous segment from a successful trajectory and assigns it a uniform positive advantage bonus. This limits its ability to capture more complex credit assignment patterns in long-horizon tasks, where multiple disjoint segments may contribute differently to task success, potentially with both positive and negative advantages. Compared with methods such as SPO, GiGPO, and CriticSearch, which support more fine-grained credit assignment across multiple steps or segments, ProVer sacrifices flexibility for computational efficiency. The paper does not sufficiently discuss or evaluate this trade-off, raising concerns about its generalizability to more complex agentic tasks.

Questions for authors

Clarification requests

How can the contribution of MC verification be distinguished from simply assigning additional positive credit to judge-selected segments? A lightweight fixed-bonus control could help clarify this.

How does the judge-guided segment selection compare with a simple entropy-based selection heuristic (e.g., selecting high-uncertainty turns, inspired by ARPO and AT²PO), while keeping the MC verification and credit assignment procedures unchanged? A discussion or lightweight controlled comparison would help clarify the added value of the external judge.

Suggestions for future work

Future work could explore more flexible credit assignment mechanisms that go beyond selecting a single segment and assigning only positive advantage bonuses. In more complex, long-horizon tasks, multiple disjoint segments may deserve different levels of credit, including negative advantages. Extending ProVer to support such cases, while maintaining computational efficiency, could improve its applicability to more challenging agentic settings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ahXJ

Thank you for the review. We ran the additional experiments you asked for, Entropy selection and Fixed bonus, and report them in the Component Ablations Comment.

W2 / Q1: selection or verification?

We ran this control as the Fixed bonus ablation (Component Ablations Comment, Finding 1): the same judge-selected segments receive a constant bonus, set to ProVer's mean accepted Δ, with no boundary rollouts. It falls short of ProVer on all three benchmarks. Selection alone therefore does not account for the gain. We agree that judge selection contributes a substantial part of the gain, as the WebShop result shows. However, the effect of unverified credit ranges from helpful to harmful depending on the environment. Verification removes that dependence, because credit is granted only when the measured success rate rises, and this is why ProVer improves over GRPO on every benchmark.

Q2: judge vs. entropy-based selection.

We implemented this comparison as you describe: each turn of the same successful trajectory is scored with AT²PO's turn entropy (mean negative log-probability of the tokens sampled in that turn), the highest-entropy valid turn becomes a one-turn segment, and verification and credit assignment are unchanged. Entropy-based selection is competitive on WebShop, where it matches ProVer (Component Ablations Comment, Finding 2). On SearchQA and ALFWorld it trails ProVer by 2.7 and 5.2 points and stays within 1.6 points of GRPO, so the judge gives more consistent gains across benchmarks. Entropy looks at one trajectory and marks where the policy was uncertain. The judge compares the successful trajectory with the failed ones and marks the decision that separates them. An uncertain turn is not necessarily a consequential one.

W1: novelty and related methods.

Sections 2.2 and 2.3 position ProVer against CriticSearch and SPO. We agree that ProVer builds on both lines of work. Its distinguishing feature is what model judgment and rollouts are each used for, and the experiments are designed to test that difference.

In CriticSearch, the critic's labels directly become the credit. In ProVer the judge never supplies credit: it only chooses where rollouts are spent, and outcomes decide whether credit is given. The fixed-bonus control isolates the consequence, since unverified judge credit falls below GRPO on ALFWorld (Component Ablations Comment, Finding 1).

In SPO, continuation-based estimates are computed at fixed boundaries along every trajectory, without regard to which decisions matter. ProVer evaluates the two boundaries of one segment chosen by cross-trajectory comparison. It outperforms SPO-tree in all six settings and SPO-chain in five of six, at lower wall-clock time (Tables 1 and 2).

ARPO and AT²PO train the policy on the rollouts they branch, so their entropy criterion decides where to explore. ProVer never trains on its continuations (Section 3.5); they only measure whether the selected segment helped. Our entropy selection ablation applies their criterion inside ProVer, and it is less consistent than the judge (Component Ablations Comment, Finding 2). To our knowledge, no prior method uses model judgment solely to allocate outcome verification. We will add ARPO and AT²PO to the related work.

W3: one positive segment per group.

We agree that crediting one positive segment is less flexible than assigning credit at many steps, and that the paper should discuss this trade-off more explicitly. However, ProVer has the highest average score at both model scales compared to SPO-chain, SPO-tree, GiGPO and CriticSearch. Our interpretation is that flexibility helps most when each credit value is accurate. Spreading a fixed budget over many locations makes each estimate noisier (SPO), ties coverage to repeated states (GiGPO), or relies on unverified critic judgments (CriticSearch). ProVer spends its budget on one segment where the estimate can be trusted, which we believe explains these results.

On negative credit, we tested it directly. Signed Δ applies negative estimates as penalties, and it lowers performance to near GRPO on all three benchmarks (Component Ablations Comment, Finding 4). ProVer also does not remove negative credit: failed trajectories keep their negative GRPO advantages, and the verified bonus is added on top of that signal.

On multiple segments and longer tasks, a different segment is credited in each eligible group at every update, so over training the policy receives verified credit for many different decisions, not for a single pattern. Verifying a segment costs two boundaries whatever the trajectory length, while evaluating every step or segment grows with the horizon, so the trade-off moves further in ProVer's favor as tasks get longer.

We hope the two new ablations and the clarified positioning against prior methods address your concerns. We would be glad to answer any further questions.

AuthorsOpenReview ↗

Dear Reviewer ahXJ

Dear Reviewer ahXJ,

Thank you again for your review. We wanted to follow up on our response, which includes the two experiments you suggested: the fixed-bonus control (W2 / Q1) and entropy-based selection (Q2), both reported in the Component Ablations Comment. We also clarified how ProVer relates to SPO, CriticSearch, ARPO and AT²PO (W1) and discussed the trade-off of crediting one positive segment (W3).

We would be grateful to know whether these address your concerns, and we would be glad to clarify anything further during the discussion period.

Best regards, The authors

Thank you for the additional experiments and clarifications. The new ablations have addressed my concerns about the separate contributions of judge-guided segment selection and outcome-based verification in ProVer. While I still consider the methodological novelty modest given the substantial related work in this area, the additional empirical evidence supports the proposed design. I have raised my Soundness score to 3 and my Rating to 3 (Weak acceptance).

AuthorsOpenReview ↗

Response from the authors

Thank you for taking the time to review our response and the new ablations, and for updating your scores. We are glad the additional results clarified the separate contributions of judge-guided selection and outcome-based verification. We appreciate your remaining point on novelty. In the revision, we will sharpen the positioning against SPO, CriticSearch, ARPO and AT²PO in the related work, and include the new ablations with their discussion in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Component Ablations Comment

We thank all reviewers for their suggestions. In response, we ran four additional component ablation experiments on ProVer on all three benchmarks. Results with additional training seeds are in the Multiple Training Seed Comment.

The ablation study uses Qwen3.5-2B with the training and evaluation protocol of the paper. Each variant changes one component of ProVer and keeps everything else fixed. The GRPO and ProVer rows are from Table 1.

  • Fixed bonus (Reviewers ahXJ, MtsW, xLLu): judge-selected segments receive a constant bonus without verification (no boundary rollouts). The bonus is set to ProVer's mean accepted Δ on each benchmark: 0.56 (WebShop), 0.40 (SearchQA), and 0.33 (ALFWorld).
  • Entropy selection (Reviewer ahXJ): the judge is replaced by the highest-entropy turn. Verification is unchanged.
  • Judge without tools (Reviewer xLLu): the judge selects segments without tool access. Verification is unchanged.
  • Signed Δ (Reviewer xLLu): negative verified estimates are applied as penalties instead of being discarded.

Results.

MethodWebShopSearchQAALFWorld
GRPO42.53 ± 0.3135.58 ± 0.3884.08 ± 3.02
ProVer47.67 ± 0.4239.83 ± 2.0890.80 ± 3.02
Fixed bonus46.60 ± 0.2037.32 ± 1.8978.61 ± 1.14
Entropy selection47.60 ± 0.6937.17 ± 0.8085.57 ± 3.02
Judge without tools46.93 ± 0.4237.58 ± 0.3889.80 ± 3.02
Signed Δ43.80 ± 0.3535.75 ± 0.0086.32 ± 3.83

Findings.

1. Verification makes judge-guided credit reliable (Fixed bonus vs. ProVer). The value of an unverified bonus depends on the environment: it can be nearly sufficient (WebShop), or worse than no bonus at all (ALFWorld). We attribute this to how often the judge's proposals truly improve the success rate. Verification removes this dependence by granting credit only to proposals that do, which is why ProVer improves over GRPO on every benchmark while the fixed bonus does not. We will revise the paper to describe the role of verification in these terms.

2. Judge selection is more consistent than an entropy heuristic (Entropy selection vs. ProVer). A cheap uncertainty heuristic can find useful segments in some environments (WebShop), but with verification held fixed its benefit does not carry across benchmarks, so it is not a reliable substitute for the judge. We attribute this to the judge's cross-trajectory contrast, which identifies decisions that separate successful from failed attempts, information that per-turn entropy does not carry.

3. Tool access contributes modestly (Judge without tools vs. ProVer). Most of the judge's benefit does not depend on tool access. The tools add at most a small gain, so the propose stage remains effective with a simpler, tool-free judge.

4. Negative credit harms performance (Signed Δ vs. ProVer). Penalizing segments whose estimate is negative is counterproductive, which supports discarding such estimates: the judge proposes segments it expects to have helped, so a negative estimate marks a proposal that failed verification, not a segment identified as harmful, and penalizing it gives the policy a signal the propose stage is not designed to provide. We discuss the bias of the positive-only rule in the responses to Reviewers MtsW and xLLu.

AuthorsOpenReview ↗

Multiple Training Seed Comment

We thank the reviewers for raising the question of training variance (Reviewers MtsW, xLLu). To measure it, we trained GRPO and ProVer with two additional training seeds on SearchQA with Qwen3.5-4B. Each cell is the mean ± std over three evaluation seeds of one trained policy. The last row is the mean ± std of the three per-seed means.

Training seedGRPOProVer
Seed 1 (Table 1)40.92 ± 0.5847.00 ± 0.90
Seed 241.75 ± 0.7547.08 ± 0.72
Seed 340.00 ± 0.6647.17 ± 0.63
Across training seeds40.89 ± 0.8847.08 ± 0.08

The improvement is consistent across training seeds. The gap of 6.2 points is about seven times GRPO's training-seed standard deviation, and the lowest ProVer seed exceeds the highest GRPO seed by more than 5 points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.