- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GRPO assigns the same advantage to all tokens within a response, without distinguishing the contributions of different reasoning steps. To address this limitation, the authors propose three components. ''Where'' assigns higher weights to earlier reasoning steps using cosine decay. ''What'' identifies reasoning segments with high token-level surprisal and increases their weights. ''When'' measures the disagreement between Where and What using Jensen–Shannon Divergence. It then adaptively adjusts the weights and selectively applies Beam–Monte Carlo Verification to further calibrate them based on the differences in final rewards across alternative reasoning branches. Finally, the resulting weights are normalized and incorporated into the original GRPO objective, without modifying its advantage estimation mechanism.
Critical strengths
Introducing fine-grained process-level learning signals into RLVR without the additional cost and complexity of training a Process Reward Model remains an important research challenge. This paper offers a potential approach by combining reasoning position, uncertainty, and selective verification to redistribute response-level advantages across reasoning steps, enabling more fine-grained credit assignment.
Critical weaknesses
-
The core ideas behind Where, What, and When have been extensively explored in recent RLVR and LLM reasoning research. For example, the use of surprisal for token weighting in What is closely related to existing methods based on entropy or surprisal, such as Beyond the 80/20 Rule (NeurIPS 2025), STARE. The position-based weighting in Where also shares similarities with previous work on step-level credit assignment, such as GRPO-λ and SSVPO. Similarly, selective verification and Monte Carlo rollouts, as used in When, are not new ideas. Overall, I feel that this paper mainly combines several existing techniques rather than introducing a fundamentally new approach to credit assignment. I would like the authors to provide a more thorough discussion of related work and clearly explain how WWW differs from previous methods. The authors should also clarify what the unique contributions of this particular combination are.
-
I also have some concerns about the motivation behind the three components. Where assumes that earlier reasoning steps are more important, but critical reasoning steps can also appear in the middle or near the end of a response. What uses surprisal to identify important segments, but high uncertainty does not necessarily imply a greater contribution to the final answer. For When, the authors use JSD disagreement between Where and What to adjust the weights and determine whether additional verification is needed. However, it is not clear why disagreement between these two heuristic signals should indicate the importance of a reasoning step. Furthermore, BMCV estimates criticality using the variance of final rewards across different branches, but outcome sensitivity does not necessarily reflect the correctness or actual contribution of the original reasoning step. I would like to see more direct empirical evidence supporting these design choices.
-
From Table 2, we can see that Where alone already provides a substantial performance improvement. At Step 500, Where achieves 74.45%, compared to 76.72% for Full WWW. More importantly, at Step 1000, the simpler Where + What combination achieves 76.04%, outperforming Full WWW at 74.60%. These results make me question whether When and BMCV provide consistent and necessary improvements. This is particularly concerning because Full WWW introduces 12.69% additional training overhead, compared to only 1.99% for the version without BMCV. I would like to see more targeted ablation studies, such as random selection, surprisal-only selection, and comparisons under different verification budgets. These experiments would help establish the actual benefits of disagreement-guided verification.
-
The paper mainly compares WWW against vanilla GRPO, but there are already many studies on fine-grained credit assignment and token-level advantage weighting. For example, GRPO-λ, STARE, GTPO, OPPO, Step-GRPO, and Beyond Uniform Credit are closely related to the problem studied in this paper. In addition, methods such as DAPO, Dr. GRPO, and GSPO have introduced improvements to the optimization efficiency, bias, or training stability of GRPO. Although not all of these methods directly address fine-grained credit assignment, they could serve as stronger RL baselines. I believe the authors should compare WWW with more relevant methods under matched backbones, training data, and computational budgets. Simply outperforming vanilla GRPO is not sufficient to establish the competitiveness of WWW against existing approaches.
-
The experiments are limited to two model backbones, Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct. The evaluation is also restricted to four mathematical reasoning benchmarks, namely GSM8K, MATH500, AIME24, and AIME25. I would like to see results on larger models and other RLVR tasks, such as code generation and logical reasoning. In addition, the paper does not report results across multiple random seeds or provide an analysis of performance variance. Given the small sizes of AIME24 and AIME25, correctly answering just a few additional questions can lead to noticeable accuracy improvements. It is therefore unclear how robust the reported gains are. Furthermore, the paper mainly reports the best observed accuracy for each benchmark during training, rather than results at a common checkpoint. I believe the authors should provide more complete training curves and comparisons at matched checkpoints to better demonstrate the stability and effectiveness of the proposed method.
Questions for authors
See weakness.
2 comments
Author Response to Reviewer RN5U
We sincerely thank the reviewer for the comprehensive assessment and for recognizing the importance of fine-grained credit assignment without an additional process-reward model. We clarify the interaction among WWW's components and provide further empirical context.
W1–W2. Novelty, motivation, and component interaction.
WWW builds upon established ideas in positional weighting, surprisal-based selection, and Monte Carlo verification. Its main methodological design is a conditional, coarse-to-fine advantage redistribution mechanism within GRPO. We retain the group-relative response advantage; Where provides an inexpensive structural prior, and What selects content-dependent candidate segments from rollout surprisal. Their response-level distributional disagreement controls both the content-based intervention strength and selective invocation of BMCV. BMCV evaluates local alternatives and uses branch-outcome sensitivity to calibrate additional weighting; response-wise normalization controls the average scale. This establishes an explicit interaction among the three sources of evidence.
The signals have complementary roles: position is a prior over relative emphasis, with a positive floor retaining weight for later steps; surprisal provides content evidence for candidate selection; JSD identifies conflicting allocation preferences; and BMCV measures variation in downstream branch returns. In a step-500 3B/GSM8K positional-control study, cosine decrease / linear decrease / reverse linear weighting = 75.51% / 74.07% / 28.05%. A separate four-setting lower-bound study finds 75.36% for 0.3, the highest of the evaluated values. We will expand the comparison with GRPO-λ, STARE, GTPO, Beyond the 80/20 Rule, and related work to explain the methodological distinctions more precisely.
W3. Component effectiveness and computation.
At step 500, Where-only / Where+What / WWW without BMCV / Full WWW = 74.45% / 73.84% / 74.78% / 76.72% on GSM8K. At step 1000, Where+What reaches 76.04%, versus 75.51% without BMCV and 74.60% for Full WWW. The variation across training stages motivates stage-aware, budgeted verification. Full WWW's measured overhead is 12.69%, compared with 1.99% without BMCV. We will compare JSD-guided, random, surprisal-only, and position-only selection under matched verification budgets, reporting both gains and generated-token costs.
W4. Competitive baselines.
Our matched performance results compare WWW with GRPO under the reported protocol; R3L and SORT are included as external references. We plan a stronger controlled suite with tuned GRPO, Dr.GRPO, DAPO, GRPO-λ, and STARE, using consistent backbones, training data, reward verification, checkpoint selection, and compute accounting. These comparisons will better position WWW relative to both optimized GRPO variants and fine-grained credit-assignment methods.
W5. Seeds, learning curves, and generalization.
We recovered three independently seeded WWW trajectories. Their per-run peak GSM8K accuracies are 76.72%, 75.21%, 75.89% at steps 500, 600, 1000, respectively, corresponding to 75.94% ± 0.76 percentage points (mean ± sample SD). We will show the trajectories with seed/run identifiers and report matched-checkpoint and validation-selected results in further evaluations. The Qwen2.5-7B-Instruct study provides a second model setting; we also plan longer-response and additional RLVR-task evaluations.
We appreciate the reviewer's suggestions and will use these targeted studies to strengthen the evidence for component interaction, reproducibility, and comparative performance.
I will maintain my original score, as I have not received a response from the authors.