acceptodds
Under review as a conference paper at ICLR 2027

Where, What, and When: Structured Intra-Response Advantage Redistribution for LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards has substantially improved large-language-model reasoning, yet Group Relative Policy Optimization (GRPO) typically applies the same response-level advantage to all tokens in a sampled response. We introduce Where–What–When (WWW), a structured intra-response advantage redistribution framework that retains GRPO's group-relative advantage estimator while differentiating credit across reasoning segments. Where supplies a structural position prior, What uses segment-level surprisal evidence to select candidate segments, and When uses their disagreement to adapt the intervention strength and selectively invoke Beam–Monte Carlo Verification (BMCV) for outcome-sensitive calibration. Response-wise mean normalization controls the average weight scale before the redistributed advantage is used in the clipped policy objective. On Llama-3.2-3B-Instruct, WWW improves the best observed accuracy over our matched GRPO baseline by 2.20–3.34 percentage points across GSM8K, MATH500, AIME24, and AIME25. On Qwen2.5-7B-Instruct, within the first 500 training steps, WWW improves the best observed accuracy over GRPO by 2.84–6.70 points across the same benchmarks. The lightweight WWW variant without BMCV adds only 1.99% end-to-end step-time overhead, while Full WWW adds 12.69%. We will release our code and model weights.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RN5UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

GRPO assigns the same advantage to all tokens within a response, without distinguishing the contributions of different reasoning steps. To address this limitation, the authors propose three components. ''Where'' assigns higher weights to earlier reasoning steps using cosine decay. ''What'' identifies reasoning segments with high token-level surprisal and increases their weights. ''When'' measures the disagreement between Where and What using Jensen–Shannon Divergence. It then adaptively adjusts the weights and selectively applies Beam–Monte Carlo Verification to further calibrate them based on the differences in final rewards across alternative reasoning branches. Finally, the resulting weights are normalized and incorporated into the original GRPO objective, without modifying its advantage estimation mechanism.

Critical strengths

Introducing fine-grained process-level learning signals into RLVR without the additional cost and complexity of training a Process Reward Model remains an important research challenge. This paper offers a potential approach by combining reasoning position, uncertainty, and selective verification to redistribute response-level advantages across reasoning steps, enabling more fine-grained credit assignment.

Critical weaknesses
  1. The core ideas behind Where, What, and When have been extensively explored in recent RLVR and LLM reasoning research. For example, the use of surprisal for token weighting in What is closely related to existing methods based on entropy or surprisal, such as Beyond the 80/20 Rule (NeurIPS 2025), STARE. The position-based weighting in Where also shares similarities with previous work on step-level credit assignment, such as GRPO-λ and SSVPO. Similarly, selective verification and Monte Carlo rollouts, as used in When, are not new ideas. Overall, I feel that this paper mainly combines several existing techniques rather than introducing a fundamentally new approach to credit assignment. I would like the authors to provide a more thorough discussion of related work and clearly explain how WWW differs from previous methods. The authors should also clarify what the unique contributions of this particular combination are.

  2. I also have some concerns about the motivation behind the three components. Where assumes that earlier reasoning steps are more important, but critical reasoning steps can also appear in the middle or near the end of a response. What uses surprisal to identify important segments, but high uncertainty does not necessarily imply a greater contribution to the final answer. For When, the authors use JSD disagreement between Where and What to adjust the weights and determine whether additional verification is needed. However, it is not clear why disagreement between these two heuristic signals should indicate the importance of a reasoning step. Furthermore, BMCV estimates criticality using the variance of final rewards across different branches, but outcome sensitivity does not necessarily reflect the correctness or actual contribution of the original reasoning step. I would like to see more direct empirical evidence supporting these design choices.

  3. From Table 2, we can see that Where alone already provides a substantial performance improvement. At Step 500, Where achieves 74.45%, compared to 76.72% for Full WWW. More importantly, at Step 1000, the simpler Where + What combination achieves 76.04%, outperforming Full WWW at 74.60%. These results make me question whether When and BMCV provide consistent and necessary improvements. This is particularly concerning because Full WWW introduces 12.69% additional training overhead, compared to only 1.99% for the version without BMCV. I would like to see more targeted ablation studies, such as random selection, surprisal-only selection, and comparisons under different verification budgets. These experiments would help establish the actual benefits of disagreement-guided verification.

  4. The paper mainly compares WWW against vanilla GRPO, but there are already many studies on fine-grained credit assignment and token-level advantage weighting. For example, GRPO-λ, STARE, GTPO, OPPO, Step-GRPO, and Beyond Uniform Credit are closely related to the problem studied in this paper. In addition, methods such as DAPO, Dr. GRPO, and GSPO have introduced improvements to the optimization efficiency, bias, or training stability of GRPO. Although not all of these methods directly address fine-grained credit assignment, they could serve as stronger RL baselines. I believe the authors should compare WWW with more relevant methods under matched backbones, training data, and computational budgets. Simply outperforming vanilla GRPO is not sufficient to establish the competitiveness of WWW against existing approaches.

  5. The experiments are limited to two model backbones, Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct. The evaluation is also restricted to four mathematical reasoning benchmarks, namely GSM8K, MATH500, AIME24, and AIME25. I would like to see results on larger models and other RLVR tasks, such as code generation and logical reasoning. In addition, the paper does not report results across multiple random seeds or provide an analysis of performance variance. Given the small sizes of AIME24 and AIME25, correctly answering just a few additional questions can lead to noticeable accuracy improvements. It is therefore unclear how robust the reported gains are. Furthermore, the paper mainly reports the best observed accuracy for each benchmark during training, rather than results at a common checkpoint. I believe the authors should provide more complete training curves and comparisons at matched checkpoints to better demonstrate the stability and effectiveness of the proposed method.

Questions for authors

See weakness.

2 comments

I will maintain my original score, as I have not received a response from the authors.

AuthorsOpenReview ↗

Author Response to Reviewer RN5U

We sincerely thank the reviewer for the comprehensive assessment and for recognizing the importance of fine-grained credit assignment without an additional process-reward model. We clarify the interaction among WWW's components and provide further empirical context.

W1–W2. Novelty, motivation, and component interaction.

WWW builds upon established ideas in positional weighting, surprisal-based selection, and Monte Carlo verification. Its main methodological design is a conditional, coarse-to-fine advantage redistribution mechanism within GRPO. We retain the group-relative response advantage; Where provides an inexpensive structural prior, and What selects content-dependent candidate segments from rollout surprisal. Their response-level distributional disagreement controls both the content-based intervention strength and selective invocation of BMCV. BMCV evaluates local alternatives and uses branch-outcome sensitivity to calibrate additional weighting; response-wise normalization controls the average scale. This establishes an explicit interaction among the three sources of evidence.

The signals have complementary roles: position is a prior over relative emphasis, with a positive floor retaining weight for later steps; surprisal provides content evidence for candidate selection; JSD identifies conflicting allocation preferences; and BMCV measures variation in downstream branch returns. In a step-500 3B/GSM8K positional-control study, cosine decrease / linear decrease / reverse linear weighting = 75.51% / 74.07% / 28.05%. A separate four-setting lower-bound study finds 75.36% for 0.3, the highest of the evaluated values. We will expand the comparison with GRPO-λ, STARE, GTPO, Beyond the 80/20 Rule, and related work to explain the methodological distinctions more precisely.

W3. Component effectiveness and computation.

At step 500, Where-only / Where+What / WWW without BMCV / Full WWW = 74.45% / 73.84% / 74.78% / 76.72% on GSM8K. At step 1000, Where+What reaches 76.04%, versus 75.51% without BMCV and 74.60% for Full WWW. The variation across training stages motivates stage-aware, budgeted verification. Full WWW's measured overhead is 12.69%, compared with 1.99% without BMCV. We will compare JSD-guided, random, surprisal-only, and position-only selection under matched verification budgets, reporting both gains and generated-token costs.

W4. Competitive baselines.

Our matched performance results compare WWW with GRPO under the reported protocol; R3L and SORT are included as external references. We plan a stronger controlled suite with tuned GRPO, Dr.GRPO, DAPO, GRPO-λ, and STARE, using consistent backbones, training data, reward verification, checkpoint selection, and compute accounting. These comparisons will better position WWW relative to both optimized GRPO variants and fine-grained credit-assignment methods.

W5. Seeds, learning curves, and generalization.

We recovered three independently seeded WWW trajectories. Their per-run peak GSM8K accuracies are 76.72%, 75.21%, 75.89% at steps 500, 600, 1000, respectively, corresponding to 75.94% ± 0.76 percentage points (mean ± sample SD). We will show the trajectories with seed/run identifiers and report matched-checkpoint and validation-selected results in further evaluations. The Qwen2.5-7B-Instruct study provides a second model setting; we also plan longer-response and additional RLVR-task evaluations.

We appreciate the reviewer's suggestions and will use these targeted studies to strengthen the evidence for component interaction, reproducibility, and comparative performance.

Reviewer_MpTVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies credit redistribution within a response under GRPO. It retains the original group-relative response advantage and combines a positional prior that favors earlier reasoning segments (Where), selection based on token surprisal (What), and a disagreement signal that adjusts intervention strength and selectively invokes Beam-Monte Carlo Verification (When). The resulting segment weights are normalized within each response and applied to the original advantage. Experiments on Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct report higher best-observed accuracy than the authors' GRPO baseline on mathematical reasoning benchmarks, together with component ablations and training overhead measurements.

Critical strengths
  1. A meaningful problem and an intuitive approach. Improving credit assignment without training an additional critic or process reward model is a useful direction. The proposed framework is understandable and compatible with GRPO, and its lightweight variant has low reported training overhead.

  2. Initial positive evidence and transparent ablations. The paper reports improvements over its matched GRPO baseline on two model families and includes component and reverse-position controls. It also acknowledges that the complete method does not consistently outperform simpler combinations at later checkpoints.

Critical weaknesses
  1. The necessity and distinct contribution of the heuristic design are not established. The intuition for emphasizing early segments does not explain the particular cosine decay or the lower weight bound of 0.3. Comparisons with alternative decay functions and sensitivity to these parameters are missing. The appendix states that several token-selection ratios were tested, but does not provide the corresponding numerical results; sensitivity to the global segment-selection ratio is also unclear.

More importantly, the paper does not directly establish that disagreement between position and surprisal identifies segments that deserve stronger intervention or additional verification. Equal-budget controls using random selection or surprisal alone would be needed to isolate this contribution. The reported ablations also leave the added complexity insufficiently justified: at step 1000, Where + What reaches 76.04% GSM8K accuracy, compared with 74.60% for Full WWW. The relationship to existing position-dependent and surprisal-based advantage reweighting methods, particularly GRPO-lambda and STARE, should be clarified and evaluated under a common protocol.

  1. The experimental scope and analysis are insufficient to support the claimed contribution. The GRPO baseline falls from 61.64% GSM8K accuracy at step 500 to 34.04% at step 1000, without an adequate diagnosis of this substantial deterioration. Consequently, it is difficult to determine whether WWW improves credit assignment or mainly mitigates a failure specific to the current training configuration. The Qwen experiments compare only GRPO and Full WWW, without component ablations or controlled comparisons with relevant methods.

The primary 3B experiments also cap training responses at 1024 tokens, leaving effectiveness on long reasoning trajectories untested. This is particularly relevant to the motivation of the paper: later corrections may challenge the fixed preference for early segments. The 384-token BMCV continuation budget may also be insufficient to reveal downstream outcome differences when branches require longer reasoning. Results on older instruct models and a limited set of conventional math benchmarks do not establish applicability to longer reasoning or tool-assisted mathematical problem solving. A controlled study across response lengths, with actual length and truncation statistics, would be more informative than the current evaluation alone.

Questions for authors
  1. Choice of weights and selection ratios. How were the positional lower bound of 0.3, the What boost coefficient of 0.3, and the disagreement threshold of 0.30 selected? How sensitive are the results to these choices and to the cosine decay? Please provide the results of the reported token-selection ratio comparison and examine the global segment-selection ratio.

  2. Baseline deterioration. What causes the large decline in GRPO performance during training? Can training reward, entropy, KL divergence, response length, and truncation statistics explain it? Do the gains persist against a stable, appropriately tuned GRPO baseline?

  3. Added value of disagreement and verification. Under the same verification budget, does the disagreement gate outperform random selection or selection based only on surprisal? What evidence establishes the independent value of When and BMCV beyond Where + What, including on Qwen and relative to relevant credit-assignment methods?

  4. Effectiveness on long reasoning trajectories. Does WWW remain effective as reasoning trajectories become substantially longer? In particular, does the early-position prior remain appropriate when decisive corrections occur late, and is the BMCV continuation budget sufficient to distinguish branch outcomes in this setting?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer MpTV

We sincerely thank the reviewer for the detailed technical assessment. We revisited our experimental records and recovered additional evidence on positional design, segment selection, and training dynamics. We address the four questions below.

Q1. Positional design and hyperparameters.

Two independent positional-control studies were conducted on Llama-3.2-3B-Instruct/GSM8K. At step 500, cosine decrease / linear decrease / reverse linear weighting obtain 75.51% / 74.07% / 28.05%. In a separate lower-bound sweep, values 0.2 / 0.3 / 0.5 / 0.6 produce 74.68% / 75.36% / 74.53% / 73.77%, respectively. These comparisons provide empirical support for cosine decay and the selected lower bound among the settings tested.

We distinguish eta_tok, the top-surprisal token fraction within each segment, from eta_seg, the global fraction of selected segments. Our Top-10/Top-20 experiments vary eta_seg: step-500 GSM8K accuracy is 73.84% / 73.77% and step-1000 accuracy is 74.91% / 76.72%. These are independent selection-ratio runs. The What boost (beta=0.3) sets additional segment emphasis; the JSD threshold (tau_D=0.30) governs expensive verification. We plan controlled sweeps of these parameters at matched verification budgets and a separate analysis of eta_tok.

Q2. GRPO deterioration and checkpoint evaluation.

The GRPO decline from 61.64% at step 500 to 34.04% at step 1000 in the original ablation run merits closer examination. Best-observed accuracy characterizes the performance attainable within a training budget, while fixed-checkpoint accuracy and full trajectories reveal optimization behavior and variability. We report these complementary views and identify the originating runs for each figure and table.

We recovered additional checkpoint-wise records and three independently seeded WWW runs. Their per-run peak GSM8K scores are 76.72%, 75.21%, 75.89% at steps 500, 600, 1000, respectively (75.94% mean, 0.76-point sample SD). We will present these alongside their trajectories. Our baseline audit will examine reward, entropy, approximate KL, response lengths, truncation, and optimization diagnostics, followed by a tuned GRPO comparison under the same protocol as WWW. This will help identify the cause of the late-stage decline and assess the contribution of credit redistribution.

Q3. Independent value of When/BMCV.

When conditionally coordinates structural and content evidence: JSD adjusts intervention strength and triggers BMCV; branch-outcome sensitivity then calibrates the added weighting. At step 500, Where+What / WWW without BMCV / Full WWW = 73.84% / 74.78% / 76.72%. At step 1000, the results are 76.04% / 75.51% / 74.60%. The changing ordering motivates investigating when verification is most useful. Full WWW adds 12.69% step-time overhead versus 1.99% without BMCV.

We plan equal-budget JSD/random/surprisal-only selection controls, verification-budget sweeps, and further Qwen component studies, reporting accuracy together with verification-token and time costs.

Q4. Longer reasoning trajectories.

The positive positional floor preserves learning weight for later reasoning segments. Our current 3B setup uses a 1024-token response limit and 384-token BMCV continuations. We plan length-stratified analysis, actual length/truncation statistics, longer response limits, and continuation-budget comparisons to characterize late corrections and verification in longer trajectories.

We appreciate the reviewer's technically actionable questions. They will guide our additional analysis of training stability, component interactions, and computational efficiency.

Reviewer_tagLOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

WWW redistributes GRPO advantages across reasoning segments using position, surprisal, and disagreement between these signals, with selective branch verification. Mathematical reasoning experiments evaluate the full framework and simpler variants on Llama-3.2-3B-Instruct and evaluate transfer on Qwen2.5-7B-Instruct.

Critical strengths

The method improves reported benchmark-wise peak accuracy over matched GRPO baselines on both model families.

Critical weaknesses
  1. The empirical advantage over existing methods is not established. The matched experiments compare WWW with GRPO and its own ablations. R3L and SORT appear only as external results under different protocols, so Table 1 cannot establish whether WWW improves on these related approaches.

  2. The disagreement-based verification rule lacks direct validation. Disagreement between position and surprisal does not by itself show where verification is most useful. The ablations test removing components, but do not compare disagreement-based selection with random or surprisal-only selection under the same verification budget.

Questions for authors
  1. How does WWW compare with related methods such as R3L or SORT under matched training and evaluation settings?

  2. With the same verification budget, does disagreement-based selection outperform random selection or selection based only on surprisal?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer tagI

We thank the reviewer for the focused assessment and for recognizing WWW's improvements over our matched GRPO baseline. We clarify the comparison protocol and the motivation for disagreement-guided verification below.

Q1. Comparison with R3L and SORT.

Table 1 includes R3L and SORT as external references under their respective training and evaluation protocols. Our controlled comparison is between GRPO and WWW under matched settings. On Llama-3.2-3B-Instruct, WWW improves the benchmark-wise best-observed accuracy by 2.66, 2.20, 3.34, and 3.34 percentage points on GSM8K, MATH500, AIME24, and AIME25. On Qwen2.5-7B-Instruct, the corresponding gains within the first 500 steps are 3.24, 2.84, 3.30, and 6.70 points. These results provide encouraging evidence across two model settings. The reported benchmark-wise peaks may occur at different checkpoints.

We will extend the evaluation with matched implementations of relevant credit-assignment approaches, including R3L and SORT where feasible, using consistent backbones, training data, reward verification, evaluation settings, and training budgets. We will report both checkpoint-matched accuracy and generation/verification costs.

Q2. Motivation and effectiveness of disagreement-guided verification.

The disagreement mechanism coordinates WWW's structural and content signals. Where provides a positional prior; What identifies candidate segments from surprisal. The Jensen–Shannon divergence between their normalized allocation distributions identifies responses in which the two signals favor different allocations. This guides the intervention strength and selective use of BMCV. BMCV evaluates alternative reasoning branches and uses their downstream outcome sensitivity to calibrate the additional weighting. The components therefore operate in a conditional, coarse-to-fine sequence.

At step 500, Full WWW achieves 76.72% GSM8K accuracy, versus 74.78% without BMCV and 73.84% for Where+What. At step 1000, Where+What obtains 76.04%, compared with 74.60% for Full WWW. This pattern suggests that the utility of verification varies with training progress, motivating more adaptive use of its budget.

To directly evaluate the selection rule, our next experiments compare JSD-guided, random, surprisal-only, and position-only selection with matched verification calls, branch/continuation counts, and generated-token budgets. We will examine downstream performance together with the relationship between selection scores and measured branch-outcome sensitivity. These controls will provide a direct assessment of the value of disagreement-guided selection.

We appreciate these concrete suggestions and will use them to strengthen the evaluation of the proposed mechanism and its competitiveness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.