acceptodds
Under review as a conference paper at ICLR 2027

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Abstract

Reinforcement learning from verifiable rewards assigns a single scalar reward to an entire reasoning trajectory, even though different parts of the trajectory can contribute very differently to success. GRPO converts that reward into one group-normalized advantage shared by every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their noisy value estimates refine credit? ProbePO uses a reward-free uncertainty score to select a few partial solutions and estimates their expected terminal verifier reward from a capped set of continuation rollouts. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage, changing where credit falls while preserving each trajectory's mean token-level advantage; the sampled continuations inform credit without entering the policy update. Across DeepSeekMath and Qwen-family models spanning 1.5B, 4B, and 7B parameters, ProbePO shows consistent observed improvements over GRPO: it is higher in all 18 benchmark–setting cells and is the highest evaluated method in 16 of 18. On Qwen3-4B-Base, ProbePO is 2.84 Avg-6 points above GRPO and 2.07 points above TreePO. In an end-to-end DeepSeekMath-7B comparison, ProbePO produced higher observed scores across all six benchmarks while using 76% fewer H200-hours than the released VinePPO recipe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, LLM post-training, Credit Assignment, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KKZsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the credit assignment problem for LLM training. In the literature, GRPO computes a trajectory-level advantage and distributes it evenly to every token while VinePPO densely evaluates the expected return of all intermediate pivots by sampling multiple continuations for each pivot. When compared with them, this work proposes a reward-free selector to rank pivots for evaluation and for each selected pivot, it samples M continuations to probe its return. The overall number of sampled continuations is capped at a budget B. Finally, the method redistributes the probed return to GRPO advantages for training. The experiments demonstrates that the method achieves better performance and uses less training time.

Critical strengths
  1. The proposed budgeted probing and reward-free boundary selection provide a practical way to balance fine-grained credit assignment and computational cost.
  2. The authors conduct extentive experiments across multiple models and benchmarks to show the effectiveness of the proposed method.
Critical weaknesses
  1. The credit redistribution mechanism is relatively heuristic. Although the paper proves that the mean token-level advantage is preserved, this does not guarantee that the redistributed advantages can better approximate the true token-level advantages or lead to improved policy.
  2. The reward-free boundary selector uses entropy score for ranking. However, there is no theoretical guarantee that this metric can filter out the most critical boundaries for exploration.
Questions for authors
  1. Please provide the intuition about using the entropy score to rank boundaries. High token uncertainty does not necessarily imply a large impact on the final reward. Also, high token entropy may not indicate high action-fork variance, especially when multiple high-probability tokens lead to similar outcomes.

  2. What is the usage of s(b) in equation (4)? If wpos(u)w_{\mathrm{pos}}(u) is not always 1, will s(b) replace Hmax⁡(b)H_{\max}(b)?

  3. In Section 5.6, the authors show the offline forwardentropy diagnostic correlates positively with the action-fork-variance proxy. However, the supporting evidence seems relatively weak (data related to only one model with two datasets).

4 comments

We thank the reviewer KKZs for the positive feedback!

W1. Approximating token advantages and improving the policy

The credit redistribution mechanism is relatively heuristic. Although the paper proves that the mean token-level advantage is preserved, this does not guarantee that the redistributed advantages can better approximate the true token-level advantages or lead to improved policy.

ProbePO provides an efficient, regularized approximation to value-based token credit. Its measurements target true advantage increments; shrinkage limits finite-probe uncertainty; and redistribution preserves the measured relative credit while retaining GRPO's trajectory-level anchor.

  1. The measurement target is true token-advantage information. A token's true advantage is the change in expected final verifier reward after taking it. For a fixed policy π\pi, sts_t denotes the prefix after token yty_t, and Vπ(s)V^\pi(s) is its expected final verifier reward. Using the standard definition,

    =Q^\pi(s_{t-1},y_t)-V^\pi(s_{t-1}) =V^\pi(s_t)-V^\pi(s_{t-1}).$$ Fresh continuations from the same policy estimate each prefix value. Measuring every token position, including the initial prompt value, would recover the true advantages as the raw estimates become accurate (Section 2, Eq. (2)). With sparse pivots, an exact value difference equals the sum of the true token advantages in the intervening segment. ProbePO approximates this cumulative credit under a capped measurement budget.
  2. We acknowledge that the finite-budget placement and credit-refinement rule include heuristic design choices. GRPO likewise makes practical choices about group reward standardization, advantage construction and the clipped surrogate. The empirical comparisons with GRPO support the learning utility of ProbePO. Our ablation study also showed the effectiveness of the critical components.

W2 and Q1. Why use entropy when uncertainty can be irrelevant to reward?

The reward-free boundary selector uses entropy score for ranking. However, there is no theoretical guarantee that this metric can filter out the most critical boundaries for exploration.

“High token uncertainty does not necessarily imply a large impact on the final reward. Also, high token entropy may not indicate high action-fork variance, especially when multiple high-probability tokens lead to similar outcomes.”

Action-fork variance measures how much expected final reward differs across next-token choices. We agree that high entropy need not imply high action-fork variance. However, next-token entropy provides an upper bound on this variation, making it useful for filtering candidates under a limited budget.

Entropy is computed from the backbone logits and serves as an inexpensive screening signal. Our response to Q3 empirically evaluates whether this ranking identifies boundaries where next-token choices lead to larger differences in expected final reward.

  1. Entropy bounds the possible variation in expected rewards. At a prefix ss, let aa be the next token sampled from policy π\pi. Let Qπ(s,a)Q^\pi(s,a) be the expected final verifier reward after taking aa. Let H(a∣s)H(a\mid s) be the entropy of that next-token distribution, computed with natural logarithms. For rewards in [0,1][0,1],

    \le \frac12 H(a\mid s).$$ The left side is the action-fork variance: how much expected rewards differ across next-token choices. Low entropy therefore limits the variation available to resolve at that choice. This supports giving nearly deterministic choices lower probing priority. We add the complete proof to paper Appendix H.
  2. Entropy is practical under the fixed budget. Entropy uses backbone logits already computed, with no extra forward pass. In contrast, direct Monte Carlo estimates of reward information or action-fork variance require verifier-scored continuations at every candidate before selection. Entropy therefore preserves additional generation for funded value estimates.

Q2. The role of s(b)s(b) and position weights

“What is the usage of s(b) in equation (4)?

s(b)s(b) is the ranking score of candidate boundaries when selecting pivots. In experiments, wpos(u)=1w_{\mathrm{pos}}(u)=1, s(b)=Hmax⁡(b)s(b)=H_{\max}(b), i.e., we use the max of next several tokens’ entropy as the ranking score.

If wpos(u)w_{\mathrm{pos}}(u) is not always 1, will s(b) replace Hmax⁡(b)H_{\max}(b)?”

Good point! Yes. Denote u=tb/Tiu=t_b/T_i as the boundary’s relative token position within the response. We use s(b)=Hmax⁡(b)wpos(u)s(b)=H_{\max}(b)w_{\mathrm{pos}}(u). The position weight supplies an optional prior and can change the ranking.

For example, the tent weight wpos(u)=1−∣2u−1∣w_{\mathrm{pos}}(u)=1-|2u-1| peaks at the middle (u=0.5u=0.5) and decreases toward both ends.

Q3. Scope of the selector evidence

We agree that the original diagnostic had limited model and dataset coverage. We extended it to three model families (1.5B–7B—DeepSeekMath-7B-SFT, R1-Distill-Qwen-1.5B, and Qwen3-4B-Base) on three datasets (GSM8K, MATH-500, and AMC 2023). We used 32 questions per reported setting. Under the same one-boundary budget, we compared selecting the highest-entropy boundary with uniform random selection over the same candidate boundaries.

We measure capture as the selected boundary’s estimated action-fork variance divided by the sum over audited candidate boundaries on the same trajectory. The table reports the average capture gain of entropy selection over uniform random selection, in percentage points.

ModelGSM8KMATH-500AMC 2023
DeepSeekMath-7B-SFT+10.18+9.26+7.52
R1-Distill-Qwen-1.5B+15.37+4.33+5.16
Qwen3-4B-Base+6.64+5.28+4.77

The capture point estimates favor entropy selection in all nine reported settings. Spearman correlations are also positive throughout this matrix, ranging from 0.110 to 0.328. We will include the full protocol and per-setting results in Appendix F.

I thank the authors for the detailed response. I have no remaining concerns.

We are glad that our response has addressed your concerns, and we greatly appreciate your constructive feedback!

Reviewer_sUJfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProbePO, a critic-free credit assignment method based on GRPO. It identifies candidate boundaries using predefined rules, selects pivots by entropy, and samples additional continuations from them. Average verifier rewards estimate intermediate values, which are used to assign token-level credit between pivots. ProbePO achieves the highest average performance among the compared methods on mathematical reasoning benchmarks. Ablations support the effectiveness of the selector.

Critical strengths
  1. It refines GRPO’s coarse advantage assignment using intermediate value estimates without a separate critic.
  2. It separates auxiliary trajectories for value estimation from original trajectories for actor updates.
  3. It achieves the best average performance among the compared methods on mathematical reasoning benchmarks.
Critical weaknesses
  1. High entropy may reflect uncertainty between phrases that do not affect the final answer. Conversely, confidently incorrect reasoning steps may have low entropy and be overlooked by the selector.
  2. The method may still lack informative credit signals in all-fail groups, since additional rollouts from intermediate pivots may also fail.
Questions for authors
  1. How does the method learn useful actions discovered by probes but absent from the original trajectories, given that probe continuations are excluded from policy updates?
  2. Could shrinking intermediate values toward the group mean while leaving the terminal reward unchanged assign extra negative credit to the final segment, even when an earlier mistake has already caused the failure?
4 comments

We thank reviewer sUJf for recognizing the benefit of refining GRPO's credit assignment without a separate critic and for the detailed questions!

W1. Entropy, inconsequential phrasing, and confidently incorrect steps

High entropy may reflect uncertainty between phrases that do not affect the final answer. Conversely, confidently incorrect reasoning steps may have low entropy and be overlooked by the selector.

ProbePO seeks prefixes where alternative reasoning choices may lead to different verifier rewards. The motivation is reward variance, rather than judging whether an individual step is right or wrong. For example, one continuation may reach the correct answer while another fails. Probing can then reveal a useful difference in expected return. Entropy provides an inexpensive screen for such choices. The two cases raised by the reviewer have different implications:

  1. High entropy from equivalent phrases can produce a redundant pivot. We acknowledge this potential redundancy, which can consume part of the capped probe budget.

    We checked whether selected locations offered different reasoning or equivalent wording. Using a trained R1-Distill-Qwen-1.5B checkpoint, we screened 79 AMC questions, generating eight original responses per question. The 37 questions with both correct and incorrect responses were eligible. For each, we audited two boundaries selected by ProbePO. Each was paired with an unselected candidate boundary from the same original response at a similar relative position (matched controls), giving 37×2=7437\times2=74 selected–control pairs.

    At each boundary’s entropy-peak prefix, we separately forced each of the four most likely next tokens and generated one greedy continuation per choice—always choosing the highest-probability subsequent token. We then compared the next reasoning steps in these four continuations, grouping equivalent wording or notation together.

    What the next-step examples showedProbePO-selected locationsMatched controls
    Equivalent meaning (wording or notation)19/74 (25.7%)42/74 (56.8%)
    Different reasoning content55/74 (74.3%)32/74 (43.2%)

    74.3% of ProbePO-selected positions offered next-step alternatives with different reasoning content, compared with 43.2% of matched unselected positions. It shows that our selected locations more often offered different reasoning content instead of equivalent phrases.

  2. Overlooking a confidently wrong, consistently failing prefix is intended behavior. Suppose the model has committed to an incorrect intermediate value and keeps using it in subsequent reasoning. Continuations from that prefix will likely repeat the same failure. The reasoning is wrong and its continuation rewards have little variance. Repeated probing would mainly confirm the failure already observed in the original response. We therefore give such prefixes lower measurement priority, even when their common reward is zero. The limited budget is better spent where alternative reasoning choices may change the outcome.

  3. Entropy bounds the possible variation in expected rewards. At a prefix ss, let aa be the next token sampled from policy π\pi. Let Qπ(s,a)Q^\pi(s,a) be the expected final verifier reward after taking aa. Let H(a∣s)H(a\mid s) be the entropy of that next-token distribution, computed with natural logarithms. For rewards in [0,1][0,1],

    \le \frac12 H(a\mid s).$$ The left side is the action-fork variance: how much expected rewards differ across next-token choices. Low entropy therefore limits the variation available to resolve at that choice. This supports giving nearly deterministic choices lower probing priority. We add the complete proof to paper Appendix H.

W2. All-fail groups and failed probe continuations

The method may still lack informative credit signals in all-fail groups, since additional rollouts from intermediate pivots may also fail.

We agree that complete failure can leave ProbePO without additional credit. However, probes do not need to succeed at every pivot. Credit comes from comparing continuation outcomes at different prefixes.

  1. If all measured rewards are identical, no additional signal is available. For example, suppose every original response and every probe receives reward zero. All pivot values and endpoint values are then zero. Their differences vanish, so ProbePO uses the same reward coefficients as GRPO. This can occur when the current policy cannot solve a difficult problem. It is also a limitation of other Monte Carlo credit methods.

  2. Failure at one pivot can still reveal where progress was lost. Consider an all-fail backbone group and one response with two selected pivots. Three of eight probes from the earlier pivot succeed. All eight probes from the later pivot fail. The estimated return therefore falls between the two prefixes. ProbePO assigns a negative correction to the intervening segment, even though every probe from the later pivot failed. Thus, different outcomes across pivots can provide credit without requiring success at each pivot.

  3. We quantified the case of all-probe-fail using the trained Qwen3-4B checkpoint on 500 MATH-500 questions. With eight backbone responses per question, five pivots per group and sampled eight fresh continuations per pivot. Only 37 groups have all failed probes, accounted for 7.4% of all groups. Although the ratio is small, the reviewer provided a very good point for future direction: to adaptively decrease the probe budget on all failure groups while allocating more to the remaining.

Q1. Learning useful actions found only by probes

How does the method learn useful actions discovered by probes but absent from the original trajectories, given that probe continuations are excluded from policy updates?

ProbePO can learn these actions indirectly through later backbone rollouts. Successful probes give the shared prefix more favorable credit. This can create more opportunities to sample the successful suffix in a later training batch. Consider the reviewer's example:

TrajectoryOutcome
Original response: prefix A → suffix BFailure
Probe continuation: prefix A → suffix CSuccess

Revisiting A provides a route to learning C. Compared with probes that all fail, the successful returns raise A's estimated value. This shifts credit toward A and away from the failed suffix B. More favorable credit for A can make it more likely in later backbone rollouts. Each visit to A provides another opportunity to generate C. The policy has already demonstrated that it can generate C during probing. When a later backbone batch contains A → C, its successful reward directly reinforces C's tokens. Thus, C receives direct reward supervision when it becomes a backbone sample, rather than in the current probe-based update.

Q2. Shrinkage and negative credit on the final segment

Could shrinking intermediate values toward the group mean while leaving the terminal reward unchanged assign extra negative credit to the final segment, even when an earlier mistake has already caused the failure?

Shrinkage balances strong local credit attribution against uncertainty in sparse value estimates. Stronger shrinkage gives the group mean more weight, limiting the influence of sampling variability in the value estimates but potentially weakening credit localization.

  1. The final segment can be still penalized less than GRPO. Suppose the mistake occurs before the pivot, and all eight probes from that prefix fail. The original response also fails. For two equally long segments, a group mean of 0.5, and a GRPO coefficient of approximately −1, the coefficients are:

    MethodEarlier segmentFinal segment
    GRPO−1.00−1.00
    ProbePO without shrinkage−1.50−0.50
    ProbePO with default shrinkage−1.17−0.83

    The reviewer is correct that −0.83 is lower than −0.50: shrinkage adds about −0.33 relative to the unshrunk variant. However, −0.83 remains less negative than GRPO's −1.00. GRPO applies the same penalty to both segments, although the mistake occurred earlier. Unshrunk ProbePO localizes the penalty more strongly. Default shrinkage moderates that localization while still giving the earlier segment the stronger penalty.

  2. Shrinkage provides a continuous tuning knob for this tradeoff. With eight probes per pivot, shrinkage helps control finite-sample estimation variability. The pseudo-count n0n_0 in Eq. (5) controls how strongly each prefix-value estimate is pulled toward the group mean. We use n0=4n_0=4. Tuning n0n_0 balances variance control with retaining measured local credit contrasts.


[Update on Nov 17]:

As stated above, shrinkage can make the final segment advantage more negative than without shrinkage. However, that advantage can still remain less negative than GRPO’s uniform advantage. We conducted additional empirical analysis to quantify this comparison.

We replayed a trained Qwen3-4B checkpoint on MATH-500 and focused on groups containing both successful and failed responses. Among them, we examined the 85 failed responses, whose all eight probes from the last pivot failed.

This subset directly examines the mechanism raised by the reviewer: last-pivot raw value estimate is zero and the backbone reward (i.e., original response's terminal reward) is zero. Shrinkage pulls the last-pivot estimate toward the positive group mean while leaving the terminal reward at zero.

On identical pivots and probe outcomes, we recomputed the segment advantages with n0=4n_{0}=4 (shrinkage) and n0=0n_{0}=0 (no shrinkage).

Table 1. Final-segment advantage with default shrinkage, compared with the GRPO base (n=85)

Final-segment advantageResponsesPercentage
Less negative than the GRPO base7992.9%
More negative than the GRPO base67.1%
Total85100.0%

Table 2. Most-penalized segment with vs. without shrinkage

Penalty locationResponsesPercentage
The same earlier segment remained most penalized85100.0%
The most-penalized segment changed00.0%
Total85100.0%

With shrinkage, the final segment's advantage was less negative than the GRPO base advantage in 79/85 responses (92.9%). For each of the 85 responses, the same earlier segment received the strongest penalty both with and without shrinkage. The final segment was never the most penalized.

AuthorsOpenReview ↗

Follow Up and Kind Reminder

Dear Reviewer sUJf,

Thank you again for your time and valuable feedback. We hope our responses have fully addressed your concerns; if so, we would greatly appreciate it if you would consider updating your assessment. In any case, we are happy to continue the discussion until the end of the discussion period on Nov 18.

We truly appreciate your time and effort in reviewing our paper!

Authors

Reviewer_YXjeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProbePO refines GRPO's credit assignment with a fixed budget of extra continuations. It selects reasoning boundaries using token entropy, estimates their values, and pulls these estimates toward the group's average reward to reduce noise. Its centered value differences adjust credit within each response while preserving the mean advantage in the stated formula. The extra continuations measure values without entering the policy-update batch. Experiments cover three models and six math benchmarks, with further comparisons at equal training time and against dense probing.

Critical strengths
  • Figure 1 makes the method easy to follow. I like the idea of spending a limited rollout budget where intermediate feedback could be useful. Keeping probe continuations outside the policy-update batch makes their role clear. The correction is simple, and the paper states precisely what it preserves: the mean advantage. Appendix B also explains why preserving this mean does not preserve the policy gradient.

  • The reported gains cover three different starting models. The equal-time comparison is particularly useful: ProbePO improves Avg-6 by 1.95 points on the same eight-B200 allocation (Table 4). The VinePPO comparison also supports the practical benefit of sparse probing, with higher accuracy and substantially lower measured training cost (Table 3).

Critical weaknesses
  • I am unsure how the normalization behaves when all original responses receive the same reward. Eq. (1) then gives σg=ϵ\sigma_g=\epsilon, but the probe values can still differ, so the correction in Eqs. (6)–(7) need not vanish. Since the controlled experiments retain every group (lines 425–427), this seems relevant in practice. The appendix mentions coefficient clipping, but I didn't find its rule or threshold. Its order relative to centering also matters: clipping the final coefficients can change their token mean. Which advantage tensor was actually used in the loss?

  • The introduction to the formal argument could be more intuitive. Lines 98–100 do connect the filtration to the prefix, but immediately invoke the tower property and a Doob martingale. Could the authors explain how observing another token changes the expected final reward before introducing these terms?

Questions for authors
  1. What happens when the original rewards have zero variance but probe returns differ? Could you please provide ϵ\epsilon and the normalization/clipping rule? If advantages are clipped, at which stage is this done, and does the token-mean identity still hold for the coefficients used in the loss?

  2. How many independent training runs underlie Tables 1–2, and what is their variation? This would help assess the smaller reported gains.

4 comments

We thank reviewer YXje for the positive feedback!

W1 / Q1. Normalization, clipping, and the coefficients used in the loss

What happens when the original rewards have zero variance but probe returns differ? Could you please provide ϵ\epsilon and the normalization/clipping rule? If advantages are clipped, at which stage is this done, and does the token-mean identity still hold for the coefficients used in the loss?

  1. We first provide the rule and threshold used in normalization and base advantage clipping

    (1) normalization

    We use an explicit standard-deviation floor σfloor=0.15\sigma_{\mathrm{floor}}=0.15, so equal backbone rewards do not amplify probe contrasts by 1/ϵ1/\epsilon.

    \sigma_g &= \operatorname{std}_{\mathrm{pop}}(r_j)+\epsilon,\\\\ \epsilon &= 10^{-4},\\\\ \sigma_{\mathrm{eff}} &= \max(\sigma_g,\sigma_{\mathrm{floor}}). \end{aligned}$$ For binary rewards with $G=8$, mixed groups have population standard deviation at least $\sqrt7/8\approx0.331>0.15$. Thus the floor affects only homogeneous groups. **(2) clipping:** Clipping is applied **only** to the base advantage. $$A_{\mathrm{base},i}=\mathrm{clip}\big((r_i-\mu_g)/\sigma_{\mathrm{eff}},\pm c\big),$$ clipping the group-normalized trajectory base to $[-c,c]$ is a stabilization safeguard, consistent with existing well-known repositories of GRPO (e.g., Alibaba's ROLL and Nvidia's NeMo RL). This limits the influence of large trajectory-base weights. ProbePO’s centered correction is added afterward, with no subsequent clipping of the token coefficients. Let $\tilde A_{i,t}$ denote the probe correction, computed using the same $\sigma_{\mathrm{eff}}$ and centered over response tokens. Then $$\begin{aligned} A_{i,t} &= A_{\mathrm{base},i}+\alpha\tilde A_{i,t},\\\\ \sum_{t=1}^{T_i}\tilde A_{i,t} &= 0. \end{aligned}$$ Here $c=2$ and $\alpha=1$. We will add the thresholds and $\epsilon=10^{-4}$ to the appendix G.
  2. ProbePO can recover useful credit when original rewards have zero variance.

    When all original rewards agree, Abase,i=0A_{\mathrm{base},i}=0 and σeff=σfloor\sigma_{\mathrm{eff}}=\sigma_{\mathrm{floor}} (instead of ϵ\epsilon). Different probe returns can still supply a nonzero residual. Centering it over valid response tokens permits positive and negative local coefficients while keeping their response mean zero. So yes you are right that the correction need not vanish. ProbePO can supply reward-based learning signals in groups where GRPO supplies none.

    For example, every original response may fail, while some probes from an intermediate prefix succeed. These successes reveal an opportunity that the original suffix failed to realize. ProbePO can reinforce the segment reaching that prefix and penalize the failed continuation.

  3. Advantage tensor

    The loss uses Ai,tA_{i,t} directly. Only the base advantage is clipped, before the centered correction is added. Consequently, each trajectory's token-mean advantage equals its clipped base exactly.

W2. Intuition before the tower property and Doob martingale

Could the authors explain how observing another token changes the expected final reward before introducing these terms?

We thank the reviewer for this suggestion. We agree that the prefix-value intuition should come before the formal terminology.

Observing a token updates the reward forecast because it rules out incompatible continuations. We will introduce this intuition before the tower property and Doob martingale in Section 2.

  1. For a fixed policy, a prefix's value averages final verifier rewards over its possible completions. Before the next token is observed, this average includes all next-token choices and their continuations.

  2. After observing the token, we average only over completions consistent with the enlarged prefix. For example, two equally likely choices lead to expected rewards of 0.80.8 and 0.20.2. The preceding value is 0.50.5, so observing the token changes it by +0.3+0.3 or −0.3-0.3. Since appending it determines the next prefix, this change is its policy-relative action advantage:

    Aπ(st−1,yt)=Vπ(st)−Vπ(st−1).A^\pi(s_{t-1},y_t)=V^\pi(s_t)-V^\pi(s_{t-1}).

  3. Averaging the possible updated forecasts recovers the preceding value, 0.5 in this example. Thus the change averages to zero before observing the token. The tower property formalizes this consistency; the sequence of prefix values is a Doob martingale.

Q2. Independent training runs and their variation

How many independent training runs underlie Tables 1–2, and what is their variation? This would help assess the smaller reported gains.

  1. Each method–backbone configuration in Table 1 and each ablation arm in Table 2 has one training run using a single seed. Training-seed variation was not reported in these original tables.

  2. The same reporting limitation occurs in the compared work: the original GRPO, SPO, TreePO and VinePPO papers also omit training-seed variation. As pointed out by the SPO paper, this is due to limited resources and dozens of hours per RL run.

  3. We additionally conducted 3-seed experiments on Qwen3-4B-Base for both ProbePO and GRPO. We report mean ± sample standard deviation (%).

    BenchmarkGRPOProbePO
    MATH-50075.96 ± 0.1179.11 ± 0.46
    OlympiadBench38.37 ± 0.1341.17 ± 0.13
    CollegeMath31.43 ± 0.2332.33 ± 0.34
    AMC47.84 ± 0.6151.15 ± 0.60
    MinervaMath24.46 ± 0.4326.31 ± 0.18
    AIME24 (@32)12.26 ± 0.2217.43 ± 0.12
    Avg-638.39 ± 0.1641.25 ± 0.19

    ProbePO's mean Avg-6 is 2.86 percentage points higher in this three-run comparison. And such multi-seed variation is reasonably small.

Thanks for these clarifications. I have no remaining concern. It would be nice that you make sure to include those precisions in the final version of your paper.

Thank you so much for your careful review, reading our rebuttal and recommending the paper’s acceptance! I have updated the paper to include the precisions in Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.