- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Intrinsic Signal Policy Optimization (ISPO) to overcome reward sparsity, zero-advantage collapse, and hallucinated certainty in reinforcement learning for large language model reasoning. The core contribution is a method that extracts dense sequence-level and token-level intrinsic signals directly from the model's conditional probabilities, coupling them with outcome rewards to safely guide optimization without external models. The authors validate ISPO across multiple base models and mathematical benchmarks.
Critical strengths
-
From my understanding, a major strength is that ISPO is entirely self-contained. By deriving dense signals directly from the policy's own conditional probabilities, it avoids the heavy computational burden of training and maintaining external Process Reward Models.
-
This paper excels in accurately diagnosing and directly resolving specific GRPO failures
-
From my perspective, the method's safety mechanism is a critical asset. By strictly coupling the intrinsic signals to the final outcome reward, it does provide some kind of guarantee to prevent reward hacking.
Critical weaknesses
Weaknesses and Questions for Authors
- Inconsistency in Advantage Calculation: From my understanding, Appendix C.13 claims that a reward standard deviation of produces advantages as high as , which is mathematically unfounded. Because the numerator and denominator scale simultaneously within the same group, the maximum advantage is theoretically bounded at around for a group size of . The reported maximum advantage of requires further explanation from the authors.
- Novelty and Combination of Existing Ideas: The proposed method primarily combines existing concepts, such as IFD-based selection [1] and reinforcement learning targeting high-entropy tokens [2]. The paper needs to clarify more distinctly what unique, indispensable contributions this specific combination delivers compared to these existing independent techniques.
- Lack of Direct Metrics for Failure Modes: Although Table 2 demonstrates that the combined method yields better overall performance, this does not directly prove that it truly resolves the two claimed failure modes. For instance, artificially creating reward variance in an "all-incorrect" group does not guarantee that the resulting policy update moves toward correct reasoning.
References
- [1] Li, Ming, et al. "From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning." NAACL, 2024.
- [2] Wang, Shenzhi, et al. "Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning." NeurIPS, 2025.
Questions for authors
Please refer to the weakness section
4 comments
Response to Reviewer x6c1 (2/2)
W3. Direct evidence for the claimed mechanisms
Nonzero reward variance alone does not guarantee a correct update. We therefore test a variance-matched random-reward control and answer-confidence discrimination on the five Table 1 benchmarks.
We added a control that uniformly permutes the bounded intrinsic rewards within each rollout group. This preserves their empirical distribution and variance while changing which response receives each intrinsic value. The binary outcome reward stays attached to its original response.
The comparison uses the same 100-update budget and greedy evaluation on the five Table 1 benchmarks.
| Method (100 updates) | AIME24 (%) | AIME25 (%) | AMC23 (%) | MATH-500 (%) | OlympiadBench (%) | Five-benchmark mean (%) |
|---|---|---|---|---|---|---|
| GRPO | 16.67 | 10.00 | 60.00 | 75.40 | 37.48 | 39.91 |
| Permuted intrinsic | 16.67 | 6.67 | 60.00 | 76.80 | 37.19 | 39.46 |
| ISPO | 16.67 | 6.67 | 72.50 | 77.40 | 38.22 | 42.29 |
ISPO exceeds the variance-matched control by 2.83 percentage points and GRPO by 2.38 points in the five-benchmark mean. Preserving the assignment between a response and its intrinsic score is more effective here than preserving intrinsic-reward variance alone.
We measure answer-confidence AUROC using correctness as the positive label and the mean actor log probability over the final boxed answer's contents as confidence. The evaluation covers all five Table 1 benchmarks at the shared 100-update endpoint. We report the pooled AUROC, per-benchmark AUROCs, and the number of scorable responses. Responses without a recoverable answer span remain in the accuracy denominator.
| Method | Pooled AUROC | Scorable responses | AIME24 | AIME25 | AMC23 | MATH-500 | OlympiadBench |
|---|---|---|---|---|---|---|---|
| GRPO | 0.7728 | 1183/1275 | 0.6923 | 0.9683 | 0.7143 | 0.7966 | 0.7073 |
| DAPO | 0.7894 | 1187/1275 | 0.7041 | 0.7917 | 0.8194 | 0.8027 | 0.7287 |
| ISPO | 0.7761 | 1216/1275 | 0.7882 | 0.9167 | 0.6569 | 0.8091 | 0.6889 |
ISPO's pooled AUROC is higher than GRPO's and lower than DAPO's. The scorable counts differ because some responses lack a recoverable boxed-answer span, DAPO's dynamic training filter is not applied during evaluation.
The random-reward control tests whether the response-specific signal contributes beyond intrinsic-reward variance. Outcome coupling constrains the sequence reward's sign, and rescaling homogeneous groups after normalization limits update magnitude. Together these controls provide empirical evidence about the proposed construction, without implying a general guarantee against reward hacking.
Thank for the rebuttal, and I would like to keep my positive score of this submission. Good luck.
Thank you for your reply and review!
Thank you again for your supportive comments and constructive review! We will incorporate the additional results, clarifications, and corrections presented in the rebuttal into the revised version.
Response to Reviewer x6c1 (1/2)
We thank the reviewer for the questions about the normalization, combination of signals and mechanism evidence.
For the additional controls, we trained Qwen2.5-Math-7B on DAPO-Math-17k for 100 updates with four H200 GPUs per method, 64 prompts × 8 responses per update, and one training seed. We evaluate on AIME24, AIME25, AMC23, MATH-500 and OlympiadBench using greedy decoding with a 4096-token output limit. The accuracy mean gives equal weight to these five benchmarks.
The 3.4 value in C.13 came from an earlier G=16 diagnostic, its group-size annotation was not updated when the main setting changed to G=8. We normalize within each group as Ai=(Ri−Rˉ)/(σpop+10−6). The centered numerator scales with the reward SD. The resulting bounds are:
The population-SD bound is G−1, and the sample-SD bound is (G−1)/G. We will correct the C.13 annotation and remove that argument. A G=8 normalization diagnostic using population SD plus 1e-6 recorded a maximum of 2.645746 across 100 updates.
IFD-based selection uses answer likelihood to select instruction data. ISPO recomputes the likelihood difference of the same sampled answer with and without its reasoning trace and uses it online as a rollout reward. Its token signal combines high entropy with drift from the frozen base policy and shapes the selected tokens according to the outcome. The sequence signal scores the trajectory, while the token signal targets local confidence.
Table 2 reports the following five-benchmark averages for the two components and their combination:
We added an entropy-bonus baseline and a shuffled-trace control under the 100-update supplementary protocol described above.
The entropy baseline adds 0.1 times the mean token entropy. For the shuffled-trace control, we permute 32-token reasoning blocks in the counterfactual reference forward while preserving trace length, answer token IDs and their absolute positions. The generated response is unchanged. This tests the reference-position concern while disrupting reasoning order.
ISPO exceeds the entropy-bonus and shuffled-trace controls by 1.26 and 2.78 percentage points, respectively, in the five-benchmark mean. These comparisons support the contribution of selective reward shaping and ordered reasoning context under this protocol.