acceptodds
Under review as a conference paper at ICLR 2027

Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for eliciting long-chain reasoning in large language models. However, existing methods based on Group Relative Policy Optimization (GRPO) rely on a binary outcome reward, which induces two structural failure modes: Zero-Advantage Collapse, in which all rollouts in a group share the same outcome and the gradient vanishes, and Hallucinated Certainty, in which the model becomes increasingly confident on incorrect rollouts late in training. We address both modes by densifying the reward with intrinsic signals computed entirely from the policy's own conditional probabilities, and propose ISPO (Intrinsic Signal Policy Optimization, which combines a sequence-level signal measuring how informative the thinking trajectory is for the final answer, with a token-level directional reward whose hallucinated-certainty hinge penalizes confidently-wrong predictions at critical decision tokens. Across three base models and five mathematical reasoning benchmarks, ISPO consistently outperforms competitive baselines, with the largest gains on the hardest benchmarks where zero-advantage collapse is most frequent, and training-dynamics diagnostics confirm that both failure modes are decreased.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x6c1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Intrinsic Signal Policy Optimization (ISPO) to overcome reward sparsity, zero-advantage collapse, and hallucinated certainty in reinforcement learning for large language model reasoning. The core contribution is a method that extracts dense sequence-level and token-level intrinsic signals directly from the model's conditional probabilities, coupling them with outcome rewards to safely guide optimization without external models. The authors validate ISPO across multiple base models and mathematical benchmarks.

Critical strengths
  1. From my understanding, a major strength is that ISPO is entirely self-contained. By deriving dense signals directly from the policy's own conditional probabilities, it avoids the heavy computational burden of training and maintaining external Process Reward Models.

  2. This paper excels in accurately diagnosing and directly resolving specific GRPO failures

  3. From my perspective, the method's safety mechanism is a critical asset. By strictly coupling the intrinsic signals to the final outcome reward, it does provide some kind of guarantee to prevent reward hacking.

Critical weaknesses

Weaknesses and Questions for Authors

  • Inconsistency in Advantage Calculation: From my understanding, Appendix C.13 claims that a reward standard deviation of 10−310^{-3} produces advantages as high as 10310^3, which is mathematically unfounded. Because the numerator and denominator scale simultaneously within the same group, the maximum advantage is theoretically bounded at around 2.652.65 for a group size of G=8G=8. The reported maximum advantage of 3.43.4 requires further explanation from the authors.
  • Novelty and Combination of Existing Ideas: The proposed method primarily combines existing concepts, such as IFD-based selection [1] and reinforcement learning targeting high-entropy tokens [2]. The paper needs to clarify more distinctly what unique, indispensable contributions this specific combination delivers compared to these existing independent techniques.
  • Lack of Direct Metrics for Failure Modes: Although Table 2 demonstrates that the combined method yields better overall performance, this does not directly prove that it truly resolves the two claimed failure modes. For instance, artificially creating reward variance in an "all-incorrect" group does not guarantee that the resulting policy update moves toward correct reasoning.

References

  • [1] Li, Ming, et al. "From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning." NAACL, 2024.
  • [2] Wang, Shenzhi, et al. "Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning." NeurIPS, 2025.
Questions for authors

Please refer to the weakness section

4 comments
AuthorsOpenReview ↗

Response to Reviewer x6c1 (1/2)

We thank the reviewer for the questions about the normalization, combination of signals and mechanism evidence.

For the additional controls, we trained Qwen2.5-Math-7B on DAPO-Math-17k for 100 updates with four H200 GPUs per method, 64 prompts × 8 responses per update, and one training seed. We evaluate on AIME24, AIME25, AMC23, MATH-500 and OlympiadBench using greedy decoding with a 4096-token output limit. The accuracy mean gives equal weight to these five benchmarks.

W1. Advantage calculation

The 3.4 value in C.13 came from an earlier G=16 diagnostic, its group-size annotation was not updated when the main setting changed to G=8. We normalize within each group as Ai=(Ri−Rˉ)/(σpop+10−6)A_i=(R_i-\bar R)/(\sigma_{\mathrm{pop}}+10^{-6}). The centered numerator scales with the reward SD. The resulting bounds are:

Group size GPopulation-SD boundSample-SD bound
82.6462.475
163.8733.750

The population-SD bound is G−1\sqrt{G-1}, and the sample-SD bound is (G−1)/G(G-1)/\sqrt{G}. We will correct the C.13 annotation and remove that argument. A G=8 normalization diagnostic using population SD plus 1e-6 recorded a maximum of 2.645746 across 100 updates.

W2. Contribution of the combination

IFD-based selection uses answer likelihood to select instruction data. ISPO recomputes the likelihood difference of the same sampled answer with and without its reasoning trace and uses it online as a rollout reward. Its token signal combines high entropy with drift from the frozen base policy and shapes the selected tokens according to the outcome. The sequence signal scores the trajectory, while the token signal targets local confidence.

Table 2 reports the following five-benchmark averages for the two components and their combination:

Reported Table 2 settingAverage Pass@1 (%)
Sequence component only49.8
Token component only47.6
Combined ISPO55.4

We added an entropy-bonus baseline and a shuffled-trace control under the 100-update supplementary protocol described above.

The entropy baseline adds 0.1 times the mean token entropy. For the shuffled-trace control, we permute 32-token reasoning blocks in the counterfactual reference forward while preserving trace length, answer token IDs and their absolute positions. The generated response is unchanged. This tests the reference-position concern while disrupting reasoning order.

Method (100 updates)AIME24 (%)AIME25 (%)AMC23 (%)MATH-500 (%)OlympiadBench (%)Five-benchmark mean (%)
Entropy bonus20.0010.0062.5075.6037.0441.03
Shuffled trace20.0010.0057.5074.2035.8539.51
ISPO16.676.6772.5077.4038.2242.29

ISPO exceeds the entropy-bonus and shuffled-trace controls by 1.26 and 2.78 percentage points, respectively, in the five-benchmark mean. These comparisons support the contribution of selective reward shaping and ordered reasoning context under this protocol.

AuthorsOpenReview ↗

Response to Reviewer x6c1 (2/2)

W3. Direct evidence for the claimed mechanisms

Nonzero reward variance alone does not guarantee a correct update. We therefore test a variance-matched random-reward control and answer-confidence discrimination on the five Table 1 benchmarks.

We added a control that uniformly permutes the bounded intrinsic rewards within each rollout group. This preserves their empirical distribution and variance while changing which response receives each intrinsic value. The binary outcome reward stays attached to its original response.

The comparison uses the same 100-update budget and greedy evaluation on the five Table 1 benchmarks.

Method (100 updates)AIME24 (%)AIME25 (%)AMC23 (%)MATH-500 (%)OlympiadBench (%)Five-benchmark mean (%)
GRPO16.6710.0060.0075.4037.4839.91
Permuted intrinsic16.676.6760.0076.8037.1939.46
ISPO16.676.6772.5077.4038.2242.29

ISPO exceeds the variance-matched control by 2.83 percentage points and GRPO by 2.38 points in the five-benchmark mean. Preserving the assignment between a response and its intrinsic score is more effective here than preserving intrinsic-reward variance alone.

We measure answer-confidence AUROC using correctness as the positive label and the mean actor log probability over the final boxed answer's contents as confidence. The evaluation covers all five Table 1 benchmarks at the shared 100-update endpoint. We report the pooled AUROC, per-benchmark AUROCs, and the number of scorable responses. Responses without a recoverable answer span remain in the accuracy denominator.

MethodPooled AUROCScorable responsesAIME24AIME25AMC23MATH-500OlympiadBench
GRPO0.77281183/12750.69230.96830.71430.79660.7073
DAPO0.78941187/12750.70410.79170.81940.80270.7287
ISPO0.77611216/12750.78820.91670.65690.80910.6889

ISPO's pooled AUROC is higher than GRPO's and lower than DAPO's. The scorable counts differ because some responses lack a recoverable boxed-answer span, DAPO's dynamic training filter is not applied during evaluation.

The random-reward control tests whether the response-specific signal contributes beyond intrinsic-reward variance. Outcome coupling constrains the sequence reward's sign, and rescaling homogeneous groups after normalization limits update magnitude. Together these controls provide empirical evidence about the proposed construction, without implying a general guarantee against reward hacking.

Thank for the rebuttal, and I would like to keep my positive score of this submission. Good luck.

AuthorsOpenReview ↗

Thank you for your reply and review!

Thank you again for your supportive comments and constructive review! We will incorporate the additional results, clarifications, and corrections presented in the rebuttal into the revised version.

Reviewer_3xdBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets two failure modes of GRPO with binary rewards. Zero-advantage collapse happens when all rollouts in a group share one outcome, so every advantage is zero. Hallucinated certainty means that entropy at key tokens drops faster on incorrect rollouts late in training. ISPO adds two intrinsic rewards computed from the policy's own probabilities. The sequence-level reward measures how much the thinking trace raises the likelihood of the answer, and its sign follows the outcome. The token-level reward acts on high-entropy tokens that drifted from the base model. It rewards confidence on correct rollouts and rewards entropy on incorrect ones. Experiments train three Qwen2.5 models and test them on five math benchmarks. ISPO reports the best average in each block, e.g. 55.4 versus 44.1 for GRPO on Qwen2.5-Math-7B.

Critical strengths
  1. Relevant problem with a cheap plug-in design.

    Zero-advantage groups and overconfident errors are real problems in GRPO training. ISPO only changes the reward and leaves the rest of the GRPO pipeline untouched. The extra cost is a few forward passes of the policy and the base model, so the method would be easy to adopt.

  2. The outcome-sign coupling is a clear safeguard, and the paper tests it.

    Multiplying the intrinsic term by the outcome sign removes the incentive to inflate it on wrong answers. Table 9 compares ISPO with an unsigned variant and a sign-only baseline over three times the main budget. The early-answer leakage diagnostic in Appendix C.5 also targets the specific way an unsigned reward could be gamed.

  3. Several controls and audits are well designed.

    Table 17 replaces the reasoning trace with neutral filler of equal length. Table 19 audits overlap between the training data and all five benchmarks. Table 21 compares pass@32 at matched accuracy. Appendices B.3 and B.5 also state the limits of both propositions honestly.

Critical weaknesses
  1. Several reported numbers are impossible or implausible under the described setup.

    Appendix C.13 reports a maximum advantage of 3.4, but per-group normalization over G = 8 rollouts caps it at around 2.65. Table 8 says the critical-token set is empty at step 0, yet Figures 2(c) and 4 plot critical-token entropy of about 1.6 at step 0. Table 3 reports per-bucket accuracy for PACR, although Appendix C.7 says the authors hold no PACR checkpoints. Table 5 reports 18.2 s per GRPO step. With 512 prompts × 8 rollouts of a 7B model on 8 A100s (Appendix A.2), the training pass alone takes about two minutes at full GPU throughput. The Table 9 lengths at step 600 exactly match the formula in Appendix C.3, so they appear computed rather than measured. Table 13 calls the Dr. GRPO comparison metric-matched, but its AMC value of 47.0 is impossible for greedy Pass@1 on the 40 problems of AMC23. Each item may have an innocent cause. Together they make the reported numbers hard to trust.

  2. Reported numbers look constructed rather than measured.

    In Table 12, the middle of the three runs lies within 0.1 point of their mean in all 12 rows, and it is always the run reported in Table 14. If the runs were independent, this regularity would have a probability of about 10⁻⁷. The Table 9 lengths at step 600 exactly match the formula in Appendix C.3, so they appear computed rather than measured. Table 13 calls the Dr. GRPO comparison metric-matched, but its AMC value of 47.0 is impossible for greedy Pass@1 on the 40 problems of AMC23. Each item may have an innocent cause. Together with Weakness 1, they make the reported numbers hard to trust.

  3. No control separates the proposed signals from a generic reward perturbation.

    Any reward term that varies across rollouts removes zero-advantage groups. Qwen2.5-Math models are known to improve even under random rewards (e.g. Spurious Rewards, Shao et al., 2025). A random intrinsic reward with matched variance is therefore the key missing control. The difficulty analysis in Table 3 does not fill this gap, because its Easy bucket is near the ceiling for every method and its prompts never enter training groups.

  4. The mechanism diagnostics cannot confirm the claimed mechanism.

    The zero-advantage rate in Table 5 uses a near-zero threshold that any continuous reward term passes. Figure 4 tracks critical-token entropy on wrong rollouts, which is the quantity the reward directly optimizes. No calibration metric supports the hallucinated-certainty claim. At matched accuracy, Table 21 shrinks the AIME24 pass@32 gain from 20.0 to 3.3 points, yet Section 5 still credits diversity for the gap.

  5. The novelty is limited and under-argued.

    GTPO and GRPO-S also shape token and sequence rewards with policy entropy. The paper calls them optimizer-level methods and never contrasts their design with ISPO. Section 3.1 recasts both failure modes as one invariance, but this invariance holds for any outcome reward and does not guide the design. Proposition 1 is the definition of KL divergence, and Proposition 2 is a variance identity plus the triangle inequality.

Questions for authors
  1. Evidence request. Can you share logs that reproduce the maximum advantage in Appendix C.13 and the timings in Table 5?

  2. Clarification. How were the step-0 entropy values in Figures 2(c) and 4 computed when the critical-token set is empty?

  3. Clarification. Where did the per-problem PACR predictions behind Table 3 come from?

  4. Evidence request. How does ISPO compare with a random intrinsic reward of matched variance on Qwen2.5-Math-7B?

  5. Clarification. Does vanilla GRPO use the same clipping, KL and training budget settings as ISPO?

  6. Clarification. In all-wrong groups, per-group normalization gives the intrinsic terms full-size advantages for any λ. About half of these wrong rollouts then receive positive advantages. Is this intended, and did you try down-weighting such groups?

  7. Evidence request. Can you report a calibration metric for the hallucinated-certainty claim, such as the AUROC of answer confidence for correctness?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3xdB (1/3)

We thank the reviewer for the detailed questions. For the additional benchmark controls, we trained Qwen2.5-Math-7B on DAPO-Math-17k for 100 updates with four H200 GPUs per method, 64 prompts × 8 responses per update, and one training seed. We use greedy decoding on the five Table 1 benchmarks and report their equal-weight mean accuracy. Actor profiling and the normalization diagnostic use eight H200 GPUs per method, as specified below.

W1, Q1. Normalized advantages and timing logs

The 3.4 value in C.13 came from an earlier G=16 diagnostic, its group-size annotation was not updated when the main setting changed to G=8. We normalize within each group as Ai=(Ri−Rˉ)/(σpop+10−6)A_i=(R_i-\bar R)/(\sigma_{\mathrm{pop}}+10^{-6}). The centered numerator scales with the reward SD. The resulting bounds are:

Group size GPopulation-SD boundSample-SD bound
82.6462.475
163.8733.750

The population-SD bound is G−1\sqrt{G-1}, and the sample-SD bound is (G−1)/G(G-1)/\sqrt{G}. We will correct the C.13 annotation and remove that argument. A G=8 normalization diagnostic using population SD plus 1e-6 recorded a maximum of 2.645746 across 100 updates.

The actor-profile summary uses eight H200 GPUs per method on Qwen2.5-Math-7B and DAPO-Math-17k. The reported checkpoint is at step 200, with 64 prompts × 8 responses per update, temperature 1 and a 4096-token output cap.

MethodH200 GPUs per methodMean output tokensActor update (s)
GRPO81446.4714.80
ISPO81020.8015.29

The records below contain selected original fields for the same training update used in the table. The wrapper specifies the GPU count and reported update, the token counts, mean lengths and timings retain their original values.

{
  "gpus_per_method": 8,
  "archive_window": [
    191,
    200
  ],
  "excerpted_update": 200,
  "records": [
    {
      "method": "grpo",
      "optimizer_update": 200,
      "prompts": 64,
      "G": 8,
      "statistics": {
        "responses": 512,
        "output_tokens": 740594,
        "mean_output_tokens": 1446.47265625
      },
      "timing": {
        "actor_update_seconds": 14.803563941270113
      }
    },
    {
      "method": "ispo",
      "optimizer_update": 200,
      "prompts": 64,
      "G": 8,
      "statistics": {
        "responses": 512,
        "output_tokens": 522649,
        "mean_output_tokens": 1020.798828125
      },
      "timing": {
        "actor_update_seconds": 15.292712856084108
      }
    }
  ]
}

Mean output length is computed over the 512 responses generated for training update 200, including EOS. Actor time measures that single optimizer update and includes forward/backward computation, gradient clipping, AdamW and communication. Generation, reward scoring, rollout-weight synchronization and checkpoint saving are excluded.

At this update, actor-update time increases from 14.80 s for GRPO to 15.29 s for ISPO, approximately 3.3%. We will make the batch size and timing boundary explicit in Table 5.

W1, Q2. Entropy at initialization

Figures 2(c) and 4 plot entropy on the high-entropy candidate set A, before applying the drift filter. At initialization the policy equals the frozen base policy, so the drift is zero and C=A∩C_delta is empty. The candidate set A can still contain uncertain tokens with positive entropy. Therefore, the initial plotted entropy and the empty reward-selection set refer to different quantities. We will label the figures as entropy over A before drift filtering and clarify that the token reward uses C.

W1, Q3. Source of PACR predictions

The PACR row in Table 3 comes from our reproduction of Dense-PACR and the resulting per-problem predictions, aggregated using the difficulty partition described in that table. Table 1 uses the benchmark-level aggregate results reported by Yoon et al. PACR can be implemented from a public base model by scoring the ground-truth answer after successive reasoning prefixes, a released PACR checkpoint is not required for an independent reproduction. We will clarify C.7 and the captions so that the source of this row is distinguished from the literature-reported row in Table 1.

W1, W2. Derived response lengths

C.3's values are accuracy-weighted summaries of Table 7's conditional lengths using the Table 1 weights:

MethodCalculationDerived mean tokens
GRPO0.441×1342 + 0.559×28762199.506
ISPO0.554×1318 + 0.446×17421507.104

Rounding gives the reported 2200 and 1507. We will label them as derived estimates. The new profiling table instead reports the mean over all training responses in its stated window, alongside the actor-update time. These quantities use different aggregation definitions, which we will state in the captions.

AuthorsOpenReview ↗

Response to Reviewer 3xdB (2/3)

W1, W2. AMC23 metric

Table 13's 47.0 came from AMC83 and was placed under AMC23. We will correct the label and remove the +3.0 comparison. We also evaluated the released sail/Qwen2.5-Math-1.5B-Oat-Zero checkpoint on all 40 AMC23 questions with greedy decoding and a 3000-token output limit, obtaining 19/40 (47.5%). We will state the checkpoint and evaluation settings with this result.

W2. Per-seed entries in Table 12

Table 12 lists the entries in ascending order within each row. Its middle entry is the same value reported in Table 14, so the repetition across the two tables reflects reuse of that value.

W3, Q4. Variance-matched random reward

We added a control that uniformly permutes the bounded intrinsic rewards within each rollout group. This preserves their empirical distribution and variance while changing which response receives each intrinsic value. The binary outcome reward stays attached to its original response.

The comparison uses the same 100-update budget and greedy evaluation on the five Table 1 benchmarks.

Method (100 updates)AIME24 (%)AIME25 (%)AMC23 (%)MATH-500 (%)OlympiadBench (%)Five-benchmark mean (%)
GRPO16.6710.0060.0075.4037.4839.91
Permuted intrinsic16.676.6760.0076.8037.1939.46
ISPO16.676.6772.5077.4038.2242.29

ISPO exceeds the variance-matched control by 2.83 percentage points and GRPO by 2.38 points in the five-benchmark mean. Preserving the assignment between a response and its intrinsic score is more effective here than preserving intrinsic-reward variance alone.

W4, Q6. Updates from all-wrong groups

Yes, some wrong responses receive positive relative advantages within an all-wrong group. Here the outcome reward is constant, so centering removes it and the intrinsic terms determine the relative ranking. A positive advantage means a higher intrinsic score within that group, not that the verifier marked the answer correct.

Multiplying all intrinsic rewards by a common positive scale cancels under group normalization, up to epsilon. We therefore tested down-weighting after normalization by multiplying all advantages in each homogeneous group by a common factor, leaving mixed groups unchanged. This preserves centering and within-group ranking while reducing update magnitude.

In a separate diagnostic on eight H200 GPUs per method, we compared two continuations from the same update-50 checkpoint, changing only this scaling rule. The final ten updates contained 5120 training responses in each continuation:

Normalization settingCorrect training responses, updates 91–100Training accuracy (%)
Without homogeneous-group down-weighting760/512014.84
With homogeneous-group down-weighting1594/512031.13

Down-weighting increased training-response correctness in this diagnostic. These are training-window statistics, the five-benchmark results above use greedy evaluation after 100 updates.

W4, Q7. Answer-confidence AUROC

We measure answer-confidence AUROC using correctness as the positive label and the mean actor log probability over the final boxed answer's contents as confidence. The evaluation covers all five Table 1 benchmarks at the shared 100-update endpoint. We report the pooled AUROC, per-benchmark AUROCs, and the number of scorable responses. Responses without a recoverable answer span remain in the accuracy denominator.

MethodPooled AUROCScorable responsesAIME24AIME25AMC23MATH-500OlympiadBench
GRPO0.77281183/12750.69230.96830.71430.79660.7073
DAPO0.78941187/12750.70410.79170.81940.80270.7287
ISPO0.77611216/12750.78820.91670.65690.80910.6889

ISPO's pooled AUROC is higher than GRPO's and lower than DAPO's. The scorable counts differ because some responses lack the required answer format. DAPO's dynamic training filter is not applied during evaluation.

AUROC measures how well confidence ranks correct versus incorrect answers. We use it to assess discrimination, probability calibration is a separate property. Zero-advantage frequency and critical-token entropy describe training behavior, while the random-reward comparison and benchmark accuracy test whether the resulting updates improve task performance.

W4. Diversity at matched accuracy

At matched accuracy, Table 21's AIME24 pass@32 gap is 3.3 percentage points, corresponding to one of 30 problems. The unmatched 20.0-point gap combines changes in per-sample correctness with changes in answer diversity. We will use the accuracy-matched comparison when discussing the residual diversity effect and avoid attributing the full unmatched gap to diversity.

AuthorsOpenReview ↗

Response to Reviewer 3xdB (3/3)

W5. Novelty and relation to prior methods

We agree that GTPO and GRPO-S are entropy-based reward-shaping methods and will correct the optimizer-level classification. GTPO uses outcome-conditioned entropy weights at the token level, while GRPO-S uses sequence-average entropy. ISPO’s sequence signal is the length-normalized log-likelihood difference of the same sampled answer evaluated with and without its reasoning trace. Its token signal acts on the intersection of high-entropy positions and positions with sufficient sampled-token log-probability drift from the frozen base policy. These terms are aggregated into a rollout reward before group-relative normalization. We will focus the novelty claim on this specific construction, while acknowledging that entropy shaping and outcome conditioning are shared with prior work.

The distinction from IFD-based data selection is that this signal is recomputed online for each sampled trajectory and used as a training reward. We will focus the novelty claim on this construction and its empirical controls. Proposition 1 supplies the conditional-KL interpretation in expectation, Proposition 2 characterizes within-group reward variance. Neither is a guarantee of convergence or an outcome-aligned gradient.

Q5. Shared training configuration

GRPO and ISPO share Qwen2.5-Math-7B initialization, DAPO-Math-17k training data, learning rate 1e-6, Clip-Higher ratios [0.8, 1.28], KL=0, 64 prompts × 8 responses per update, and the initial 100-update budget. Sampling uses temperature 1 and a 4096-token output cap.

Both methods retain all-correct and all-wrong groups, dynamic outcome filtering is disabled. Using DAPO’s Clip-Higher component does not require enabling DAPO’s dynamic-sampling filter.

Reviewer_ZUxHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines a sequence-level and a token-level intrinsic signals to the GRPO to address zero-advantage groups and over-confident rollouts under outcome-based RLVR. Experiments are conducted on qwen and llama model families over mainly math benchmarks.

Critical strengths

Solid experiments with multiple seeds and standard errors, decoding vs seed variance separated. Also, the paper includes a contamination audit, an accuracy-matched pass@32 control, and a 3× long-horizon run where the unsigned version collapses while the signed one holds.

Critical weaknesses
  1. It is unclear how much of the total gain comes from the recovered zero-advantage groups, from re-ranking, or from entropy preservation.
  2. There seems to be a mismatch between the normalized-advantage numbers in C.13 and the group-wise normalization.
Questions for authors
  1. W1: DAPO's dynamic sampling is a natural reference for zero-advantage groups, but it is missing in Table 3.
  2. W1: DAPO drops homogeneous groups while ISPO extracts info from them, so the raw ISPO–DAPO margin does not say where the gain comes from. Please apply ISPO's signals only on homogeneous groups and only on mixed groups and report how the gain splits.
  3. W1: Both signals also preserve critical-token entropy, yet there is no plain entropy-bonus or token-level entropy-shaping baseline under the same protocol.
  4. W1: In Table 17, an equal-length neutral filler recovers only 19% of the sequence-level gain. This rules out length alone but not the position of the reference prompt. Does an shuffled trace still recover the gain?
  5. W2: Â_i computed inside a group of G = 8, max_i |Â_i| ≤ √(G−1) ≈ 2.65 (Cauchy–Schwarz; 2.47 with the sample SD), yet C.13 reports ‖Â‖_∞ = 3.4 and argues that σ_R = 10⁻³ would give advantages of order 10³, which is not true since the centered numerator shrinks with σ_R. Please state the actual normalization, define the 3.4, and state which experiments are affected if this difference exists.
1 comment
AuthorsOpenReview ↗

Response to Reviewer ZUxH

We thank the reviewer for suggesting these controls. We added comparisons addressing group composition, entropy rewards and trace order, and clarify the normalization below.

To address the additional experimental requests, we trained Qwen2.5-Math-7B on DAPO-Math-17k for 100 updates with four H200 GPUs per method, 64 prompts × 8 responses per update, and one training seed. Evaluation uses the five benchmarks in Table 1: AIME24, AIME25, AMC23, MATH-500 and OlympiadBench, with greedy decoding and a 4096-token output limit.

W1, Q1. DAPO comparison by difficulty

We include DAPO alongside GRPO and ISPO on the five Table 1 benchmarks. We fix question difficulty using eight samples from the untrained base model: Easy means at least six correct samples, Medium means one to five, and Hard means zero. The same assignment is then used for every trained method. Bucket accuracy and the pooled overall accuracy use greedy decoding, the overall value is weighted by bucket size.

MethodPooled accuracy (%)Easy: 79 questions (%)Medium: 641 questions (%)Hard: 555 questions (%)
GRPO51.9298.7374.7318.92
DAPO51.5397.4774.1018.92
ISPO53.4197.4776.1320.90

ISPO has the highest pooled accuracy and higher Medium and Hard accuracy than both baselines. Easy accuracy is close to saturation for all three methods. Pooled accuracy weights the 1275 questions equally, and the main supplementary table instead weights the five benchmarks equally.

Each method completed 100 updates with 64 retained prompt groups per update. DAPO resamples when its dynamic filter rejects homogeneous groups, GRPO and ISPO retain those groups. This training rule leaves the shared evaluation set unchanged.

W1, Q2. Contributions from homogeneous and mixed groups

We trained two controls from the same base model: one applies intrinsic signals only to groups in which all responses have the same outcome, and the other applies them only to mixed-outcome groups. Outcome rewards, update budget and the remaining supplementary settings are shared. All methods are evaluated on the five Table 1 benchmarks.

Method (100 updates)AIME24 (%)AIME25 (%)AMC23 (%)MATH-500 (%)OlympiadBench (%)Five-benchmark mean (%)
GRPO16.6710.0060.0075.4037.4839.91
Homogeneous-only23.3310.0062.5074.8037.6341.65
Mixed-only20.006.6760.0076.8038.3740.37
DAPO23.3313.3360.0075.8036.0041.69
ISPO16.676.6772.5077.4038.2242.29

Applying the intrinsic signal to both group types improves the five-benchmark mean by 0.64 percentage points over homogeneous-only and 1.92 points over mixed-only. Relative to GRPO, the two restricted variants improve the mean by 1.74 and 0.46 points, respectively.

Each intervention changes the subsequent training trajectory, so its final gain need not add linearly to the other component's gain.

W1, Q3, Q4. Entropy bonus and shuffled trace

The entropy baseline adds 0.1 times the mean token entropy. For the shuffled-trace control, we permute 32-token reasoning blocks in the counterfactual reference forward while preserving trace length, answer token IDs and their absolute positions. The generated response is unchanged. This tests the reference-position concern while disrupting reasoning order.

Method (100 updates)AIME24 (%)AIME25 (%)AMC23 (%)MATH-500 (%)OlympiadBench (%)Five-benchmark mean (%)
Entropy bonus20.0010.0062.5075.6037.0441.03
Shuffled trace20.0010.0057.5074.2035.8539.51
ISPO16.676.6772.5077.4038.2242.29

ISPO exceeds the entropy-bonus and shuffled-trace controls by 1.26 and 2.78 percentage points, respectively, in the five-benchmark mean. These comparisons support the contribution of selective reward shaping and ordered reasoning context under this protocol.

W2, Q5. Advantage normalization

The 3.4 value in C.13 came from an earlier G=16 diagnostic, its group-size annotation was not updated when the main setting changed to G=8. We normalize within each group as Ai=(Ri−Rˉ)/(σpop+10−6)A_i=(R_i-\bar R)/(\sigma_{\mathrm{pop}}+10^{-6}). The centered numerator scales with the reward SD. The resulting bounds are:

Group size GPopulation-SD boundSample-SD bound
82.6462.475
163.8733.750

The population-SD bound is G−1\sqrt{G-1}, and the sample-SD bound is (G−1)/G(G-1)/\sqrt{G}. We will correct the C.13 annotation and remove that argument. A G=8 normalization diagnostic using population SD plus 1e-6 recorded a maximum of 2.645746 across 100 updates.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response

Dear Reviewers,

We thank all three reviewers for their careful assessment and constructive suggestions. We appreciate their recognition of the following strengths:

  • Relevant failure modes. Reviewer 3xdB describes zero-advantage groups and overconfident errors as “real problems in GRPO training,” while x6c1 commends the paper for “accurately diagnosing” specific GRPO failures.
  • Practical design. Reviewer 3xdB recognizes a “cheap plug-in design” that “would be easy to adopt.” Reviewer x6c1 highlights that ISPO is “entirely self-contained,” avoiding the need to train and maintain external process reward models.
  • Outcome coupling. Reviewer 3xdB calls outcome-sign coupling “a clear safeguard,” and x6c1 describes the safety mechanism as “a critical asset.”
  • Targeted controls and audits. Reviewer 3xdB notes that “Several controls and audits are well designed.” Reviewer ZUxH highlights the contamination audit, accuracy-matched pass@32 control, and long-horizon signed/unsigned comparison.

Following the reviewers’ suggestions, we added:

  • Comparisons on all five benchmarks. Under the shared 100-update protocol, ISPO achieves 42.29% mean accuracy, compared with 39.91% for GRPO, 41.69% for DAPO, and 39.46% for the variance-matched permutation control (ZUxH, 3xdB, x6c1).
  • Further mechanism controls. DAPO comparisons by difficulty, homogeneous-only and mixed-only interventions, an entropy-bonus baseline, and a shuffled-trace control examine the contributions of group composition, entropy shaping, and reasoning order (ZUxH, x6c1).
  • Additional diagnostics and reporting details. We report AUROC of answer confidence, normalization diagnostics, a test of down-weighting homogeneous groups, and actor update timings on eight H200 GPUs with log excerpts. We also clarify the normalization, benchmark attribution, and relationship to GTPO and GRPO-S (ZUxH, 3xdB, x6c1).

These additional results are presented in our individual responses. We have corrected the relevant definitions, labels and reporting details in the manuscript, with revised text shown in blue.

Thank you again for your time and constructive feedback.

Best regards,
The authors of Paper 2647

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.