- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the credit assignment problem for LLM training. In the literature, GRPO computes a trajectory-level advantage and distributes it evenly to every token while VinePPO densely evaluates the expected return of all intermediate pivots by sampling multiple continuations for each pivot. When compared with them, this work proposes a reward-free selector to rank pivots for evaluation and for each selected pivot, it samples M continuations to probe its return. The overall number of sampled continuations is capped at a budget B. Finally, the method redistributes the probed return to GRPO advantages for training. The experiments demonstrates that the method achieves better performance and uses less training time.
Critical strengths
- The proposed budgeted probing and reward-free boundary selection provide a practical way to balance fine-grained credit assignment and computational cost.
- The authors conduct extentive experiments across multiple models and benchmarks to show the effectiveness of the proposed method.
Critical weaknesses
- The credit redistribution mechanism is relatively heuristic. Although the paper proves that the mean token-level advantage is preserved, this does not guarantee that the redistributed advantages can better approximate the true token-level advantages or lead to improved policy.
- The reward-free boundary selector uses entropy score for ranking. However, there is no theoretical guarantee that this metric can filter out the most critical boundaries for exploration.
Questions for authors
-
Please provide the intuition about using the entropy score to rank boundaries. High token uncertainty does not necessarily imply a large impact on the final reward. Also, high token entropy may not indicate high action-fork variance, especially when multiple high-probability tokens lead to similar outcomes.
-
What is the usage of s(b) in equation (4)? If is not always 1, will s(b) replace ?
-
In Section 5.6, the authors show the offline forwardentropy diagnostic correlates positively with the action-fork-variance proxy. However, the supporting evidence seems relatively weak (data related to only one model with two datasets).
4 comments
Q2. The role of and position weights
“What is the usage of s(b) in equation (4)?
is the ranking score of candidate boundaries when selecting pivots. In experiments, , , i.e., we use the max of next several tokens’ entropy as the ranking score.
If is not always 1, will s(b) replace ?”
Good point! Yes. Denote as the boundary’s relative token position within the response. We use . The position weight supplies an optional prior and can change the ranking.
For example, the tent weight peaks at the middle () and decreases toward both ends.
Q3. Scope of the selector evidence
We agree that the original diagnostic had limited model and dataset coverage. We extended it to three model families (1.5B–7B—DeepSeekMath-7B-SFT, R1-Distill-Qwen-1.5B, and Qwen3-4B-Base) on three datasets (GSM8K, MATH-500, and AMC 2023). We used 32 questions per reported setting. Under the same one-boundary budget, we compared selecting the highest-entropy boundary with uniform random selection over the same candidate boundaries.
We measure capture as the selected boundary’s estimated action-fork variance divided by the sum over audited candidate boundaries on the same trajectory. The table reports the average capture gain of entropy selection over uniform random selection, in percentage points.
| Model | GSM8K | MATH-500 | AMC 2023 |
|---|---|---|---|
| DeepSeekMath-7B-SFT | +10.18 | +9.26 | +7.52 |
| R1-Distill-Qwen-1.5B | +15.37 | +4.33 | +5.16 |
| Qwen3-4B-Base | +6.64 | +5.28 | +4.77 |
The capture point estimates favor entropy selection in all nine reported settings. Spearman correlations are also positive throughout this matrix, ranging from 0.110 to 0.328. We will include the full protocol and per-setting results in Appendix F.
I thank the authors for the detailed response. I have no remaining concerns.
We are glad that our response has addressed your concerns, and we greatly appreciate your constructive feedback!
We thank the reviewer KKZs for the positive feedback!
W1. Approximating token advantages and improving the policy
ProbePO provides an efficient, regularized approximation to value-based token credit. Its measurements target true advantage increments; shrinkage limits finite-probe uncertainty; and redistribution preserves the measured relative credit while retaining GRPO's trajectory-level anchor.
The measurement target is true token-advantage information. A token's true advantage is the change in expected final verifier reward after taking it. For a fixed policy π, st denotes the prefix after token yt, and Vπ(s) is its expected final verifier reward. Using the standard definition,
=Q^\pi(s_{t-1},y_t)-V^\pi(s_{t-1}) =V^\pi(s_t)-V^\pi(s_{t-1}).$$ Fresh continuations from the same policy estimate each prefix value. Measuring every token position, including the initial prompt value, would recover the true advantages as the raw estimates become accurate (Section 2, Eq. (2)). With sparse pivots, an exact value difference equals the sum of the true token advantages in the intervening segment. ProbePO approximates this cumulative credit under a capped measurement budget.We acknowledge that the finite-budget placement and credit-refinement rule include heuristic design choices. GRPO likewise makes practical choices about group reward standardization, advantage construction and the clipped surrogate. The empirical comparisons with GRPO support the learning utility of ProbePO. Our ablation study also showed the effectiveness of the critical components.
W2 and Q1. Why use entropy when uncertainty can be irrelevant to reward?
Action-fork variance measures how much expected final reward differs across next-token choices. We agree that high entropy need not imply high action-fork variance. However, next-token entropy provides an upper bound on this variation, making it useful for filtering candidates under a limited budget.
Entropy is computed from the backbone logits and serves as an inexpensive screening signal. Our response to Q3 empirically evaluates whether this ranking identifies boundaries where next-token choices lead to larger differences in expected final reward.
Entropy bounds the possible variation in expected rewards. At a prefix s, let a be the next token sampled from policy π. Let Qπ(s,a) be the expected final verifier reward after taking a. Let H(a∣s) be the entropy of that next-token distribution, computed with natural logarithms. For rewards in [0,1],
\le \frac12 H(a\mid s).$$ The left side is the action-fork variance: how much expected rewards differ across next-token choices. Low entropy therefore limits the variation available to resolve at that choice. This supports giving nearly deterministic choices lower probing priority. We add the complete proof to paper Appendix H.Entropy is practical under the fixed budget. Entropy uses backbone logits already computed, with no extra forward pass. In contrast, direct Monte Carlo estimates of reward information or action-fork variance require verifier-scored continuations at every candidate before selection. Entropy therefore preserves additional generation for funded value estimates.