acceptodds
Under review as a conference paper at ICLR 2027

Semifactual Credit-Augmented Policy Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024–2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. Our code and data can be found in the Supplementary Material.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards (RLVR), Token-Level Credit Assignment, Group Relative Policy Optimization (GRPO), Semifactual Interventions, Large Language Models, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9k67OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on the influence of token pairs of irrelevant reasoning on the rollout process of reasoning. The paper uses four different semifactual perturbations and calculates a probability drift by scoring with a teacher-forced method after placing the same response token and its prefix under different prompts. The negative part of this drift is stopped from gradient training. The results are verified in two types of tasks: Mathematical reasoning and Out-of-distribution generalization.

Critical strengths

The method pipeline is complete. Starting from token-level semifactual sensitivity, it provides a complete process from diagnosis, fixed response detection to training correction. Moreover, the detection can reuse the already generated responses, and does not need to generate new trajectories for perturbations. In the experimental results, by randomly altering the correction positions, using perturbations that change the answers, and comparing negative/positive/bidirectional corrections, ablation studies were conducted directly to examine the signal position, intervention semantics, and correction direction. This further supports the claims of the paper.

Critical weaknesses
  1. There is no basis for equating task invariance and token-by-token probability invariance. Equivalent questions may reasonably induce different phrasings or solutions; fixing the prefix of the original answer and then observing the high drift, it may reflect harmful dependencies or normal language adaptation or context compatibility changes. The paper does not report how to distinguish between the two.
  2. The connection between unscaled drift to a unified negative correction is still heuristic. Probability increases and decreases are both included in the same drift, but both methods reduce the corresponding advantages; it is not an optimization of consistency loss. The bounded auxiliary gradient and finite-time optimization bound in the appendix only constrain the scope of the correction's influence, and do not prove that this correction reduces drift or improves returns.
  3. Training only covers Qwen backbones, and it cannot be confirmed whether the probability drift described in the paper is also widespread in other model structures. In addition, according to Appendix B.1, the perturbations generated using prompts do not have semantic supervision. The manuscript does not report the generation quality of the perturbations, etc., and cannot simply rely on the instructions to determine that the generation meets the requirements.
Questions for authors

See weeknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9k67 (1/2)

We sincerely thank you for recognizing the complete pipeline and the targeted ablations. Your comments helped us identify key areas to clarify and improve, and we have carefully addressed your concerns in detail below.

[W1] How is harmful dependence distinguished from normal language adaptation?

Thank you for raising this point. We agree that task invariance does not imply token-level probability invariance. SCAPO does not require token-by-token probability invariance. We use semifactual perturbations to probe sensitivity and empirically investigate whether this sensitivity can provide a useful token-level credit signal.

(1) Lexical adaptation. Our perturbations alter task-irrelevant surface features, with a typo potentially changing only a single character. On 128 held-out training problems (8 responses each, about 1.15M tokens), we identify response tokens matching words added or removed by each prompt edit. Under paraphrase, these tokens account for only 0.63% of tokens and 4.35% of drift. Thus, the observed sensitivity is not confined to direct lexical overlap with the edited wording.

(2) Empirical support. Shuffling the credit scores substantially reduces the gains, and using answer-changing perturbations further degrades performance to below the GRPO baseline. These controls support the usefulness of token-aligned credit under semifactual answer-preserving perturbations.

Conclusion: Our claim is that semifactual sensitivity provides a useful credit signal, rather than that task invariance requires token-level probability invariance. We have revised Section 3.4 to describe how relative semifactual sensitivity refines outcome-based credit (text in blue).

[W2] Why are probability increases and decreases treated in the same way?

Thank you for pointing this out. We agree that the link from unsigned drift to a negative correction is heuristic. SCAPO is a credit-shaping method and does not optimize a consistency loss.

(1) Hypothesis. We hypothesize that tokens whose probability is stable under semifactual perturbations are more closely tied to solving the problem. For an unstable token, an increase and a decrease both mean that its probability depends on features that leave the answer unchanged. We therefore measure sensitivity by the magnitude of the change and lower the credit of sensitive tokens.

(2) Observation and empirical support. On the same 1.15M-token probe, probability decreases and increases under semifactual perturbations are similarly frequent (52.3% and 45.6% of token–perturbation pairs, respectively).

Conclusion: The design follows the hypothesis above and is validated empirically. Our training experiments and ablations further demonstrate the effectiveness of this heuristic credit-shaping approach.

[W3, model scope] Does the drift pattern occur beyond Qwen?

Yes. We observe the same drift pattern on two Llama models, where dd-filtered decoding also improves accuracy. The experiments use 200 problems randomly sampled from the diagnostic data of Section 2.

(1) Token-level drift. For each additional model, we sample one response per original prompt (T=1T = 1, top-pp = 1) and teacher-force it under the original and the four edited prompts. Category values are relative to each model's mean drift, since tokenizers and responses differ across models. The Qwen3-4B-Base row is the original 1,000-question reference.

ModelMean driftReflection markersDiscourse connectivesOther wordsMath symbolsNumbers
Qwen3-4B-Base (original paper)0.0162.74×2.74\times2.32×2.32\times1.53×1.53\times0.47×0.47\times0.37×0.37\times
Qwen3-1.7B-Base0.0282.20×2.20\times1.75×1.75\times1.42×1.42\times0.53×0.53\times0.49×0.49\times
Llama-3.1-8B-Instruct0.0152.25×2.25\times2.15×2.15\times0.90×0.90\times1.24×1.24\times1.10×1.10\times
Llama-3.2-3B-Instruct0.0172.27×2.27\times2.11×2.11\times0.91×0.91\times1.11×1.11\times1.00×1.00\times

Reflection markers and discourse connectives are the two highest-drift categories in all four models. Numbers and math symbols are the least sensitive in Qwen and close to the mean in Llama.

(2) dd-filtered decoding. Accuracy is evaluated with Math-Verify.

ModelVanilla (%)dd-filter (%)Gain (pp)
Qwen3-1.7B-Base8.5017.50+9.00
Llama-3.1-8B-Instruct10.0014.00+4.00
Llama-3.2-3B-Instruct5.5013.50+8.00

Conclusion: High drift on reflection markers and connectives, and the benefit of filtering high-drift tokens, both extend beyond the Qwen family.

We have added a brief summary to Section 2 and the settings and detailed results to Appendix C.5 of the revised manuscript (text in blue).

AuthorsOpenReview ↗

Response to Reviewer 9k67 (2/2)

[W3, data quality] What is the quality of the generated perturbations?

Thank you for highlighting this. To assess the generation quality, we used Qwen3.8-27B as a judge model to evaluate all 68,968 perturbations (17,242 problems ×\times 4 types) and then invited three human experts to review a subset.

(1) LLM evaluation. The judge, which is from a different family than the generator, checks task preservation, answer preservation, and whether hints were added or removed.

Judge outcomeParaphraseTypo noiseScenario wrapIrrelevant contextAll
Valid99.27%98.82%99.68%99.54%99.33% (68,503)
Invalid or Uncertain0.73%1.18%0.32%0.46%0.67% (465)

(2) Human evaluation. Three human experts reviewed all 465 flagged prompts and 465 prompts randomly sampled from those the judge marked valid.

Judge decisionHuman: validHuman: invalidHuman agrees with the judge
Valid (random sample of 465)464199.78%
Invalid or Uncertain (all 465)19027559.14%

Weighting the human validity rates by the full judge-stratum sizes gives an estimated overall validity of 99.39%.

Conclusion: The generated perturbations largely preserve the task and the answer (>99%).

We have added a summary to Section 2 and the audit protocol and detailed results to Appendix B.1.1 of the revised manuscript (text in blue).

We sincerely thank the reviewer again for the constructive feedback. We hope the additional experiments and clarifications address your concerns.

Dear Reviewer 9k67,

As the discussion period draws to a close, we wanted to ask whether our responses have addressed your concerns. If anything remains unclear, we would be happy to clarify.

Your feedback is really important to us. Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_QNnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semifactual Credit-Augmented Policy Optimization (SCAPO), which augments GRPO with token-level credit derived from answer-preserving prompt perturbations. For responses sampled from the original prompt, the rollout policy evaluates the same response tokens under the original prompt and four perturbed prompts. Probability drift is converted into a group-relative stability score through two stages of standardization. Only negative stability scores are added to the original GRPO advantages, reducing credit for relatively unstable tokens. This augmentation is applied during the first 20% of training, followed by standard GRPO.

The paper motivates the method through token-sensitivity analysis and an online filtering experiment. Evaluation on Qwen3-4B-Base and Qwen3-1.7B-Base covers mathematical reasoning, perturbed mathematical problems, and GPQA-Diamond. Ablations examine the source and sign of the credit correction.

Critical strengths
  1. The method provides a practical token-level credit signal without additional inference-time computation. SCAPO evaluates fixed responses through teacher forcing rather than generating separate responses for each perturbed prompt. The correction integrates directly into GRPO, and evaluation uses ordinary generation. The reported probing and credit-construction cost accounts for 1.8% of measured training runtime in the 4B experiment, supporting the feasibility of the approach.

  2. The empirical evaluation is reasonably broad and includes meaningful controls. The paper evaluates two model scales against five RLVR baselines across multiple mathematical and out-of-distribution benchmarks. SCAPO improves over GRPO on all reported mathematical metrics, including AIME gains of 5.63 and 4.17 percentage points at 4B and 1.7B, respectively. The shuffled-credit and correction-sign ablations further support the usefulness of token alignment and negative-only corrections. The online filtering experiment also provides substantive evidence that the drift signal is actionable.

Critical weaknesses
  1. Important design choices are not sufficiently validated.

    The correction is applied only during the first 20% of training, but the paper does not compare this schedule with full-duration augmentation or alternative durations. Consequently, it is unclear whether disabling the correction later is necessary or whether the chosen duration is a robust setting.

    The two-stage z-score normalization in Equation (7) lacks an ablation. Even small drift differences can receive substantial weight after normalization. Comparing one-stage and two-stage normalization, with similar correction strength, would clarify whether both stages are needed.

  2. The reliability and interpretation of the gains remain incompletely established.

    No results across independent training seeds are reported, leaving the reliability of several small margins over competing methods uncertain.

    SCAPO achieves higher NoOp-AIME accuracy but shows a larger drop from original AIME than GRPO: 4.10 versus 1.52 percentage points at 4B, with the same pattern at 1.7B. Given the paper’s motivation of addressing spurious dependence, this pattern deserves discussion.

The method is interesting and the reported performance improvements are promising. However, the correction schedule and normalization lack targeted validation, while training variability and the robustness interpretation remain unresolved. These concerns concern central aspects of the method and the evidence supporting it.

Questions for authors
  1. Correction schedule: Why is augmentation restricted to the first 20% of training? How does this configuration compare with applying the correction throughout training? Is later deactivation necessary, and how sensitive are the results to the duration and correction strength?

  2. Normalization: What is the empirical contribution of each z-score operation in Equation (7)? Please compare the current scheme with averaging drift across perturbations and standardizing once, or removing the final standardization, while controlling correction magnitude.

  3. Training variability: Are results available across independent training seeds? Reporting variability for key benchmarks would help establish whether the improvements and ablation rankings are reliable.

  4. NoOp robustness: How do the authors interpret the larger original-to-NoOp accuracy decrease for SCAPO at both model scales?

4 comments
AuthorsOpenReview ↗

Response to Reviewer QNna (1/3)

We sincerely thank you for recognizing our token-level credit signal and the broad empirical evaluation. Your questions helped us clarify the schedule, normalization, and robustness claims.

[Q1 / W1] Why the first 20%? How sensitive is SCAPO to duration and strength?

Thank you for suggesting these controls. We conducted additional experiments on Qwen3-1.7B-Base to assess sensitivity to N0N_0 and λ0\lambda_0.

(1) Duration.

Credit Durationλ0\lambda_0AIME 24–26AMC 23–25ID Avg.OOD Avg.
0 (GRPO)–7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.2218.29±0.6318.29 \pm 0.6313.80±0.7213.80 \pm 0.72
First 10%0.0112.0839.1221.1917.03
First 20% (SCAPO)0.0111.67±0.6811.67 \pm 0.6841.01±0.53\boldsymbol{41.01 \pm 0.53}23.10±0.14\boldsymbol{23.10 \pm 0.14}17.21±0.12\boldsymbol{17.21 \pm 0.12}
First 30%0.017.7135.8819.2214.58
All steps0.011.3222.1511.7710.68
All steps0.0018.2633.2218.4614.41

(2) Strength, with credit active for the first 20%.

λ0\lambda_0AIME 24–26AMC 23–25ID Avg.OOD Avg.
0 (GRPO)7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.2218.29±0.6318.29 \pm 0.6313.80±0.7213.80 \pm 0.72
0.00310.6936.5621.2215.51
0.00611.4638.4822.4716.21
0.01 (SCAPO)11.67±0.68\boldsymbol{11.67 \pm 0.68}41.01±0.53\boldsymbol{41.01 \pm 0.53}23.10±0.14\boldsymbol{23.10 \pm 0.14}17.21±0.12\boldsymbol{17.21 \pm 0.12}
0.036.9433.2219.3414.74
0.056.7431.7918.5114.11

At λ0\lambda_0 = 0.01, credit over the first 10–20% of training performs best, while longer windows lose most of the gain. With the window fixed to 20%, accuracy increases on all four metrics as λ0\lambda_0 grows from 0.003 to 0.01, whereas larger values (0.03–0.05) lose most of the gain. This degradation under longer or stronger credit agrees with Appendix D, where the augmentation enters the bound through (2βHλ0+H2λ02) N0/T(2\beta H\lambda_0 + H^2\lambda_0^2)\,N_0/T, a term that grows with both λ0\lambda_0 and N0N_0, so a larger total correction permits a larger deviation from the GRPO update.

Conclusion: Moderate early credit is most effective, and a longer window appears to require a smaller λ0\lambda_0.

We have added a brief discussion to the revised manuscript (text in blue) in Section 4.3 and Appendix D, and both ablation tables and their settings to Appendix F.5.

AuthorsOpenReview ↗

Response to Reviewer QNna (2/3)

[Q2 / W1] What does each normalization contribute at matched correction magnitude?

Thank you for this question. The inner normalization balances perturbation types, while the outer normalization puts the semifactual score on a comparable scale to the outcome advantage, making λ0\lambda_0 easier to interpret.

(1) Distinct roles. With zscore⁡x\operatorname{zscore}_x denoting standardization over all valid response tokens in the group of prompt xx, Eq. (7) can be written as

v=1K∑k=1Kzscore⁡x(−d(k)),u=zscore⁡x(v).\boldsymbol v=\frac{1}{K}\sum_{k=1}^{K}\operatorname{zscore}_x\big(-\boldsymbol d^{(k)}\big),\qquad \boldsymbol u=\operatorname{zscore}_x(\boldsymbol v).

  • Inner: each perturbation type is standardized before averaging, so a type with larger raw drift variance does not dominate merely because of its scale.
  • Outer: the outer normalization places u\boldsymbol u on a standardized scale comparable to the GRPO outcome advantage within the same group, so λ0\lambda_0 controls the nominal relative strength of the correction.

(2) Two ablations with matched correction magnitude. We trained both alternatives:

  • No inner (average drift across perturbations, then standardize once): u=zscore⁡x(−1K∑k=1Kd(k))\boldsymbol{u}=\operatorname{zscore}_{x}\left(-\frac{1}{K}\sum_{k=1}^{K}\boldsymbol{d}^{(k)}\right).
  • No outer (remove the final standardization): u=v\boldsymbol{u}=\boldsymbol{v}.

To control correction magnitude, we rescale λ0\lambda_0 for each variant so that the root mean square of the applied negative correction λ0u−\lambda_0\boldsymbol u^- matches that of SCAPO on the calibration set.

The scale factor is measured once on 128 held-out training problems (8 responses each, about 1.15M tokens) and then fixed before training, 0.997 for no inner and 1.276 for no outer.

VariantAIME 24–26AMC 23–25ID Avg.OOD Avg.
GRPO7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.2218.29±0.6318.29 \pm 0.6313.80±0.7213.80 \pm 0.72
SCAPO11.67±0.68\boldsymbol{11.67 \pm 0.68}41.01±0.5341.01 \pm 0.5323.10±0.14\boldsymbol{23.10 \pm 0.14}17.21±0.1217.21 \pm 0.12
No inner10.0738.3922.2716.03
No outer11.4642.0323.0017.38

SCAPO and GRPO results are reported as three-seed mean ±\pm std (see [Q3 / W2] below), and each ablation is a single run. With correction magnitude matched, removing the outer normalization closely matches SCAPO, while removing the inner normalization lowers performance but still outperforms GRPO.

(3) Amplification of small drift differences. We agree that z-score normalization can amplify small absolute drift differences, since it measures relative sensitivity within each group. In practice, however, each group collects all valid response tokens from the prompt's rollouts (about 9,000 on average in our calibration data), and only tokens less stable than the group average (about 28%) receive a correction, so small differences among stable tokens are never used. A small group standard deviation means that nearly all token drift values are close to the same value, which is not typical in our calibration data.

Conclusion: Removing the inner normalization lowers all four metrics relative to SCAPO. Removing the outer normalization closely matches SCAPO after rescaling λ0\lambda_0. Its main role is to put the semifactual score on a comparable scale to the outcome advantage, making λ0\lambda_0 easier to interpret.

We have added a brief discussion to Section 4.3 and the ablation settings and detailed results to Appendix F.6 of the revised manuscript (text in blue).

AuthorsOpenReview ↗

Response to Reviewer QNna (3/3)

[Q3 / W2] Results across independent training seeds.

SCAPO consistently improves over GRPO across three seeds. We conducted additional experiments on Qwen3-1.7B-Base. The results for seed 1 are from the original paper, and seeds 2 and 3 are additional runs. The seeds control the training data order and rollout sampling. Evaluation follows the same protocol.

  • In-distribution accuracy (%).
MethodSeedAIME 24–26AMC 23–25HMMT 25–26BRUMO 25SMT 25Omni-MathMinervaOlympiadID Avg.
GRPO17.7132.971.9813.546.9620.4527.9431.3117.86
GRPO28.1333.023.178.336.9620.1731.6232.6418.01
GRPO37.5733.371.1917.927.7821.4530.5132.3419.02
GRPOmean ±\pm std7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.222.12±1.002.12 \pm 1.0013.26±4.8013.26 \pm 4.807.23±0.487.23 \pm 0.4820.69±0.6720.69 \pm 0.6730.02±1.8930.02 \pm 1.8932.10±0.7032.10 \pm 0.7018.29±0.6318.29 \pm 0.63
SCAPO111.8841.394.9619.1710.8524.7430.8840.8023.08
SCAPO210.9041.244.1720.6312.6224.6029.4140.2122.97
SCAPO312.2240.403.0818.3313.0925.1733.4640.2123.24
SCAPOmean ±\pm std11.67±0.68\boldsymbol{11.67 \pm 0.68}41.01±0.53\boldsymbol{41.01 \pm 0.53}4.07±0.95\boldsymbol{4.07 \pm 0.95}19.38±1.16\boldsymbol{19.38 \pm 1.16}12.19±1.18\boldsymbol{12.19 \pm 1.18}24.84±0.30\boldsymbol{24.84 \pm 0.30}31.25±2.05\boldsymbol{31.25 \pm 2.05}40.41±0.34\boldsymbol{40.41 \pm 0.34}23.10±0.14\boldsymbol{23.10 \pm 0.14}
  • Out-of-distribution accuracy (%).
MethodSeedNoOp-AIMEThinkBench-AIMEGPQA-DiamondOOD Avg.
GRPO16.117.5727.4213.70
GRPO26.748.0628.8914.56
GRPO37.858.4023.1513.13
GRPOmean ±\pm std6.90±0.886.90 \pm 0.888.01±0.428.01 \pm 0.4226.49±2.9826.49 \pm 2.9813.80±0.7213.80 \pm 0.72
SCAPO18.4012.0131.2517.22
SCAPO210.1411.3930.4717.33
SCAPO39.3811.6030.2817.08
SCAPOmean ±\pm std9.30±0.87\boldsymbol{9.30 \pm 0.87}11.67±0.32\boldsymbol{11.67 \pm 0.32}30.67±0.51\boldsymbol{30.67 \pm 0.51}17.21±0.12\boldsymbol{17.21 \pm 0.12}

Conclusion: SCAPO achieves higher mean accuracy on all 11 benchmarks across seeds, and every SCAPO seed exceeds every GRPO seed on AIME, AMC, and both averages.

Other baselines and the Table 2 ablations remain single runs, so their smaller differences are not established across seeds. We have added the three-seed mean ±\pm std results to Section 4.2 (Figure 5 of the revision) and the exact means, standard deviations, and individual seed results to Appendix F.3 of the revised manuscript (text in blue).

[Q4 / W2] Why is the clean-to-NoOp drop larger for SCAPO?

Thank you for this insightful observation. While SCAPO indeed exhibits a larger clean-to-NoOp drop on AIME, it nonetheless achieves higher NoOp accuracy.

AIME is relatively difficult. In the paper, clean accuracy is below 30% for both methods at both scales. We think the low clean accuracy limits the possible absolute drop. For example, a model that fails on both versions has a zero gap.

An easier benchmark: MATH500 and NoOp-MATH500. To examine this issue at higher clean accuracy, we additionally evaluate the same SCAPO and GRPO checkpoints on MATH500 and its NoOp variant:

ModelMethodMATH500NoOp-MATH500Drop (pp)
1.7BGRPO69.0057.8011.20
1.7BSCAPO74.2064.609.60
4BGRPO85.0069.4015.60
4BSCAPO86.8072.0014.80

SCAPO has both higher NoOp accuracy and a smaller gap at both scales, improving NoOp accuracy by 6.80 / 2.60 pp and reducing the gap by 1.60 / 0.80 pp at 1.7B / 4B.

Conclusion: We believe the gap is most informative when clean accuracy is sufficiently high. When clean accuracy is low, absolute accuracy on the perturbed problems may be the more informative measure.

We have added the evaluation settings and detailed results to Appendix F.4 of the revised manuscript (text in blue).

We sincerely thank the reviewer again for the constructive feedback. We hope the additional experiments and clarifications address your concerns.

Dear Reviewer QNna,

As the discussion period draws to a close, we wanted to ask whether our responses have addressed your concerns. If anything remains unclear, we would be happy to clarify.

Your feedback is really important to us. Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_NfjdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how RLVR-trained LLMs react to task-irrelevant prompt changes. The authors apply semifactual prompt edits that keep the math problem and answer fixed, measure token probability drift on fixed responses, and show that a decoding filter based on this signal can improve accuracy without weight updates. They then propose SCAPO, which adjusts GRPO advantages with a group-relative stability score: only the negative part is used, and only in an early training phase. On Qwen3-4B and 1.7B, SCAPO improves AIME 2024–2026 over GRPO and reports strong results on most math and reported OOD benchmarks among the compared methods. Ablations support aligning the credit signal with semifactual drift and using negative-only corrections.

Critical strengths
  1. The path from diagnosis to method to ablation is clear. Section 2 shows heterogeneous token drift and a useful decoding filter. Section 3 turns this into a simple rule: do not reward stability alone; only down-weight relatively unstable tokens. Table 2’s shuffle / counterfactual / sign ablations point in the expected direction. This matters for my assessment.
  2. The empirical coverage is broader than one metric. Table 1 covers many contest-math benchmarks at two scales, plus several OOD settings. Gains over GRPO are fairly consistent, and the paper reports that probing costs only a small fraction of training time. For a methods paper, this completeness helps.
Critical weaknesses
  1. The causal / “spurious dependence” language is stronger than the evidence. Drift measures how token probabilities move under surface rewrites of the prompt. Higher drift for some token types may just mean they are more context-sensitive, not that they are spurious. A helpful decoding filter also does not by itself show that lowering advantage on unstable tokens removes spurious dependence in training. Softer wording, or more direct evidence, would help.
  2. The main results lack uncertainty estimates, and the method is only shown in a limited setting. I did not find multi-seed variance for the main tables. AIME sets are not large, so “best on most metrics” from a single run is hard to trust. In addition, experiments stay at 1.7B/4B math RLVR, and λ is nonzero only for the first N0 steps. It is still unclear how much of the final gain comes from lasting semifactual credit versus better early trajectories followed by plain GRPO. If the authors add multi-seed results, soften the causal wording, and report N0/λ0 (and on/off) controls, I am willing to consider a higher score.
Questions for authors
  1. Please report GRPO and SCAPO on AIME 2024–2026 (and AMC) over at least 3 seeds, with mean ± std. Is the ranking stable?
  2. Did you check that semifactual prompts keep the answer? What fraction of perturbations were discarded?
  3. Please show sensitivity to N0 and λ0, and compare full-time credit / no credit / early-only credit. Where do most gains come from?
  4. If “spurious dependence” is rewritten as the milder “prompt-surface sensitivity,” do the main claims still hold?
  5. (Clarification) Is the compute assumption of d-filtered decoding comparable to the teacher-forced training probe? How much of the decoding gain might come from constrained sampling?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Nfjd (1/2)

We sincerely appreciate your insightful questions and constructive suggestions.

[Q1 / W2] Are the AIME and AMC gains stable across at least three training seeds?

Yes. SCAPO consistently improves over GRPO across three seeds. We conducted additional experiments on Qwen3-1.7B-Base. The results for seed 1 are from the original paper, and seeds 2 and 3 are additional runs. The seeds control the training data order and rollout sampling. Evaluation follows the same protocol.

  • In-distribution accuracy (%).
MethodSeedAIME 24–26AMC 23–25HMMT 25–26BRUMO 25SMT 25Omni-MathMinervaOlympiadID Avg.
GRPO17.7132.971.9813.546.9620.4527.9431.3117.86
GRPO28.1333.023.178.336.9620.1731.6232.6418.01
GRPO37.5733.371.1917.927.7821.4530.5132.3419.02
GRPOmean ±\pm std7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.222.12±1.002.12 \pm 1.0013.26±4.8013.26 \pm 4.807.23±0.487.23 \pm 0.4820.69±0.6720.69 \pm 0.6730.02±1.8930.02 \pm 1.8932.10±0.7032.10 \pm 0.7018.29±0.6318.29 \pm 0.63
SCAPO111.8841.394.9619.1710.8524.7430.8840.8023.08
SCAPO210.9041.244.1720.6312.6224.6029.4140.2122.97
SCAPO312.2240.403.0818.3313.0925.1733.4640.2123.24
SCAPOmean ±\pm std11.67±0.68\boldsymbol{11.67 \pm 0.68}41.01±0.53\boldsymbol{41.01 \pm 0.53}4.07±0.95\boldsymbol{4.07 \pm 0.95}19.38±1.16\boldsymbol{19.38 \pm 1.16}12.19±1.18\boldsymbol{12.19 \pm 1.18}24.84±0.30\boldsymbol{24.84 \pm 0.30}31.25±2.05\boldsymbol{31.25 \pm 2.05}40.41±0.34\boldsymbol{40.41 \pm 0.34}23.10±0.14\boldsymbol{23.10 \pm 0.14}
  • Out-of-distribution accuracy (%).
MethodSeedNoOp-AIMEThinkBench-AIMEGPQA-DiamondOOD Avg.
GRPO16.117.5727.4213.70
GRPO26.748.0628.8914.56
GRPO37.858.4023.1513.13
GRPOmean ±\pm std6.90±0.886.90 \pm 0.888.01±0.428.01 \pm 0.4226.49±2.9826.49 \pm 2.9813.80±0.7213.80 \pm 0.72
SCAPO18.4012.0131.2517.22
SCAPO210.1411.3930.4717.33
SCAPO39.3811.6030.2817.08
SCAPOmean ±\pm std9.30±0.87\boldsymbol{9.30 \pm 0.87}11.67±0.32\boldsymbol{11.67 \pm 0.32}30.67±0.51\boldsymbol{30.67 \pm 0.51}17.21±0.12\boldsymbol{17.21 \pm 0.12}

Conclusion: SCAPO achieves higher mean accuracy on all 11 benchmarks across seeds, and every SCAPO seed exceeds every GRPO seed on AIME, AMC, and both averages.

We have added the three-seed mean ±\pm std results to Section 4.2 (Figure 5 of the revision) and the exact means, standard deviations, and individual seed results to Appendix F.3 of the revised manuscript (text in blue).

[Q2] Are the perturbations answer-preserving?

Thank you for highlighting this. To assess whether the semifactual prompts preserve the answers, we used Qwen3.8-27B as a judge model to evaluate all 68,968 perturbations (17,242 problems ×\times 4 types) and then invited three human experts to review a subset.

(1) LLM evaluation. The judge, which is from a different family than the generator, checks task preservation, answer preservation, and whether hints were added or removed.

Judge outcomeParaphraseTypo noiseScenario wrapIrrelevant contextAll
Valid99.27%98.82%99.68%99.54%99.33% (68,503)
Invalid or Uncertain0.73%1.18%0.32%0.46%0.67% (465)

Only 0.67% of the perturbations (465) are flagged as invalid or uncertain.

(2) Human evaluation. Three human experts reviewed all 465 flagged prompts and 465 prompts randomly sampled from those the judge marked valid.

Judge decisionHuman: validHuman: invalidHuman agrees with the judge
Valid (random sample of 465)464199.78%
Invalid or Uncertain (all 465)19027559.14%

Weighting the human validity rates by the full judge-stratum sizes gives an estimated overall validity of 99.39%.

Conclusion: The semifactual prompts are largely answer-preserving (>99%).

We have added a summary to Section 2 and detailed results to Appendix B.1.1 of the revised manuscript (text in blue).

AuthorsOpenReview ↗

Response to Reviewer Nfjd (2/2)

[Q3 / W2] How sensitive are N0N_0 and λ0\lambda_0?

We conducted additional experiments on Qwen3-1.7B-Base to assess sensitivity to N0N_0 and λ0\lambda_0.

(1) Duration.

Credit Durationλ0\lambda_0AIME 24–26AMC 23–25ID Avg.OOD Avg.
0 (GRPO)–7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.2218.29±0.6318.29 \pm 0.6313.80±0.7213.80 \pm 0.72
First 10%0.0112.0839.1221.1917.03
First 20% (SCAPO)0.0111.67±0.6811.67 \pm 0.6841.01±0.53\boldsymbol{41.01 \pm 0.53}23.10±0.14\boldsymbol{23.10 \pm 0.14}17.21±0.12\boldsymbol{17.21 \pm 0.12}
First 30%0.017.7135.8819.2214.58
All steps0.011.3222.1511.7710.68
All steps0.0018.2633.2218.4614.41

(2) Strength, with credit active for the first 20%.

λ0\lambda_0AIME 24–26AMC 23–25ID Avg.OOD Avg.
0 (GRPO)7.80±0.297.80 \pm 0.2933.12±0.2233.12 \pm 0.2218.29±0.6318.29 \pm 0.6313.80±0.7213.80 \pm 0.72
0.00310.6936.5621.2215.51
0.00611.4638.4822.4716.21
0.01 (SCAPO)11.67±0.68\boldsymbol{11.67 \pm 0.68}41.01±0.53\boldsymbol{41.01 \pm 0.53}23.10±0.14\boldsymbol{23.10 \pm 0.14}17.21±0.12\boldsymbol{17.21 \pm 0.12}
0.036.9433.2219.3414.74
0.056.7431.7918.5114.11

At λ0\lambda_0 = 0.01, credit over the first 10–20% of training performs best, while longer windows lose most of the gain. With the window fixed to 20%, accuracy increases on all four metrics as λ0\lambda_0 grows from 0.003 to 0.01, whereas larger values (0.03–0.05) lose most of the gain. This degradation under longer or stronger credit agrees with Appendix D, where the augmentation enters the bound through (2βHλ0+H2λ02) N0/T(2\beta H\lambda_0 + H^2\lambda_0^2)\,N_0/T, a term that grows with both λ0\lambda_0 and N0N_0, so a larger total correction permits a larger deviation from the GRPO update.

Conclusion: Moderate early credit is most effective, and a longer window appears to require a smaller λ0\lambda_0.

We have added a brief discussion to the revised manuscript (text in blue) in Section 4.3 and Appendix D, and both ablation tables and their settings to Appendix F.5.

[Q4 / W1] Do the claims hold with the milder term “prompt-surface sensitivity”?

Yes. We appreciate this distinction and have adopted milder wording. Drift measures token-probability changes under answer-preserving prompt edits. We now mainly state our claims in terms of “semifactual sensitivity”, i.e., sensitivity to answer-preserving prompt edits, and we have softened the causal wording of our method. The method and all empirical claims are defined by this measured drift and do not rely on a causal interpretation. We have revised the manuscript accordingly (text in blue).

[Q5] How does dd-filter compute compare with training probes, and could its gain come from constrained sampling?

Compute. dd-filter evaluates the original prompt and K=4K=4 perturbed prompts at each decoding step, requiring roughly K+1=5K+1=5 times the forward-pass computation of ordinary decoding. During training, SCAPO only teacher-forces already sampled responses, without autoregressive generation, and only during the early credit phase. Probing accounts for 1.8% of the training runtime on Qwen3-4B-Base (Section 4.4).

Constrained sampling. We add a shuffle decoding control for Qwen3-4B-Base on the same 1,000-question panel as in Section 2. Shuffle-filter randomly reorders candidate tokens instead of ranking them by drift, while matching the probability mass removed by dd-filter.

Decoding methodVanilladd-filterShuffle-filter
Accuracy15.8%30.0%17.0%

Conclusion: Removing the same probability mass at random yields only +1.2 points, compared with +14.2 for dd-filter, so the gain comes mainly from filtering high-drift tokens rather than from constrained sampling itself.

We have added a brief comparison to Section 2 and the control definition and detailed results to Appendix C.4 of the revised manuscript (text in blue).

We sincerely thank the reviewer again for the constructive feedback. We hope the additional experiments and clarifications address your concerns.

Dear Reviewer Nfjd,

As the discussion period draws to a close, we wanted to ask whether our responses have addressed your concerns. If anything remains unclear, we would be happy to clarify.

Your feedback is really important to us. Thank you very much for your time and consideration.

Best regards,

The Authors

Comments on the paper

3 comments

Happy to confirm that the perturbation quality concern has been resolved. We want to address the remaining questions for you.

1. How to distinguish harmful dependence from normal language adaptation

We believe that the significant shifts in token probabilities caused by edits unrelated to the mathematical problem can provide a useful credit signal for RLVR. For example, in an arithmetic sequence problem, simply misspelling "smallest" as "smalelst" caused the probability of a plus sign in a+1a+1 to drop from 65.5% to 47.4%. This spelling error had nothing to do with the addition operation. We believe that this degree of sensitivity in the derivation steps goes far beyond the scope of "normal language adaptation". Training ablation experiments also validated the value of this signal for credit attribution: SCAPO’s AIME accuracy was 11.88%, but dropped to 8.33% when the same corrections are shuffled within each response (Table 2).

This difference reflects how the trained model handles irrelevant information. We use Qwen3.8-27B to determine whether answers of GRPO and SCAPO clearly identify and reject irrelevant information inserted into the question. We evaluated 1,440 answers for each model size on NoOp-AIME[1]. As shown in the table below, on the 1.7B model, the explicit rejection rate of SCAPO is 20.14%, which is more than that of GRPO (9.51%). The ratios of 4B are 34.10% and 29.93%, respectively. SCAPO identifies and explicitly excludes irrelevant information more frequently

Model SizeGRPOSCAPO
1.7B9.51%20.14%
4B29.93%34.10%

In addition, we respectfully disagree with the interpretation that "SCAPO is very likely a learned reflection-marker or redundancy regularizer". Reflection markers and discourse connectors exhibit the highest average drift, but they account for only 0.29% and 2.68% of all answer token positions, respectively (Appendix B.3, Table 5). We apply the SCAPO credit augmentation to 100 prompts from the train data, and we find that 93.6% of the total negative correction fell outside the two word categories. In addition, Figures 7 and 8 intuitively demonstrate that drift can broadly cross mathematical expressions and normal words.

2. About the duration and strength sweeps

At 1.7B, the average GRPO ID score is 18.29. When λ0=0.01\lambda_0=0.01, a 10% credit window can reach 21.19. When the credit window is 20%, the values for λ0\lambda_0 of 0.003, 0.006, and 0.01 reach 21.22, 22.47, and 23.10, respectively. These results support the initial hypothesis that this type of credit enhancement performed during the early stages of training can influence trajectory selection (Section 3.4).

We selected λ₀ = 0.01 from a coarse grid of {0.001, 0.01, 0.1} on AIME 2024 and set the 20% window empirically. On the AIME 2025 and 2026 datasets, SCAPO outperformed GRPO by 4.58 points and 6.67 points at 4B scale, respectively. Related work also used AIME for setting comparisons. For example, FIPO[2] compared clipping ranges and decay horizons on AIME 2024/2025, while SAPO[3] compared temperature settings on validation benchmarks including AIME 2025.

The distinction between decoding and training explains why their scheduling schemes may differ. During decoding, the filter selects the next token for each prefix under fixed model weights. During training, however, each update revises the shared weights and alters the rollout distribution that provides future gradients. Consequently, a signal can improve local token selection throughout decoding while providing the greatest training benefit early in optimization. The bound quantifies the possible deviation from GRPO. The scheduling scan provides empirical evidence of when this bias can improve accuracy.

3. Applicability

A complete RL training run requires a significant amount of computing resources. On Qwen3-1.7B-Base, we trained both SCAPO and GRPO with additional seeds per method, bringing the total to six full RL runs. Different seeds simultaneously change the order of training data and rollout sampling. The lowest AIME score of the three SCAPO trainings was higher than the highest scores of the GRPO trainings. For example, on AIME, the lowest SCAPO score is 10.90%, which exceeds the highest GRPO score 8.13%. Across eight ID benchmarks, SCAPO achieves 23.10 ± 0.14%, and GRPO is 18.29 ± 0.63%. This demonstrates that our implementation consistently reproduces SCAPO’s gains as data order and rollout sampling change. On two Llama models, dd-filtered decoding improved accuracy by 4.0 and 8.0 points, respectively.

For context, given the substantial cost of full RL training, our choice is consistent with prior work FIPO[2], SAPO[3], GSPO[4], and DAPO[5]. All four use Qwen models for their RL comparisons and report their headline gains without quantifying variability across independent training seeds.

References

[1] Mirzadeh et al. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. ICLR, 2025.

[2] Ma et al. FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization. COLM, 2026.

[3] Gao et al. Soft Adaptive Policy Optimization. 2025.

[4] Zheng et al. Group Sequence Policy Optimization. 2025.

[5] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2025.

AuthorsOpenReview ↗

General Response

Dear ACs, SACs, PCs, and Reviewers,

we sincerely thank you for your time and effort in overseeing the review of our paper. Reviewers Nfjd and 9k67 praised the complete path from diagnosis to method and ablation. Reviewer Nfjd noted that "the path from diagnosis to method to ablation is clear". Reviewers QNna and 9k67 highlighted SCAPO’s reuse of existing rollouts. Reviewer QNna described our method as "a practical token-level credit signal without additional inference-time computation."

As the discussion period closes, we summarize SCAPO’s main contributions, the new evidence added during the rebuttal, and our corresponding manuscript revisions.

1. Core Contributions

We find that suppressing tokens with high probability drift under answer-preserving prompt perturbations improves reasoning accuracy without updating model weights. Motivated by this finding, we introduce SCAPO, which uses semifactual drift signal to refine GRPO’s token-level credit assignment and shape trajectories during training. Across both model scales, SCAPO achieves the highest average accuracy among all evaluated methods on both in-distribution and out-of-distribution benchmarks.

2. Reviewer Concerns

Addressed by the Rebuttal

  • Experiments across training seeds (Nfjd, QNna): Across three independent training seeds on Qwen3-1.7B-Base, SCAPO achieves higher mean accuracy than GRPO on all 11 benchmarks. Every SCAPO run exceeds every GRPO run on both the in-distribution and out-of-distribution averages.
  • Quality of semifactual prompt perturbations (Nfjd, 9k67): An LLM audit of all 68,968 perturbations, paired with a check by human experts, yields an estimated validity rate of 99.39%. Reviewer 9k67 explicitly confirmed that this resolved the perturbation-quality concern.
  • Credit schedule and normalization (Nfjd, QNna): SCAPO outperforms GRPO under both 10% and 20% credit windows and across several correction strengths. At matched correction magnitudes, per-perturbation normalization improves all four aggregate metrics.
  • What the drift signal captures (Nfjd, 9k67): Drift-guided decoding reaches 30.0% accuracy, compared to 17.0% for a matched-mass shuffle control. Moreover, SCAPO’s negative credit corrections go well beyond reflection markers and discourse connectives, with other tokens accounting for 93.6% of their total magnitude.
  • Robustness and cross-family applicability (QNna, 9k67): Additional evaluations on NoOp-MATH500 show that SCAPO improves accuracy over GRPO by 6.8 and 2.6 percentage points at 1.7B and 4B, respectively, while exhibiting smaller clean-to-perturbed accuracy drops on this benchmark. Drift-guided decoding also improves accuracy on Llama models, gaining 4.0 points on Llama-3.1-8B-Instruct and 8.0 points on Llama-3.2-3B-Instruct.

Still Outstanding / Minor

  • Token-level interpretation (9k67): While the ablations show that semifactual drift is practically useful for credit assignment, distinguishing harmful dependence from language adaptation with complete precision at the level of individual tokens remains open.
  • Training generality (9k67): The six independent training runs establish reproducibility on Qwen3-1.7B-Base, and the Llama experiments show that the decoding signal transfers across model families. Given the substantial cost of full RL training, how well SCAPO training transfers to other model families remains open.

3. Manuscript Revisions

  • Section 2 and Appendices B.1.1 and C.4–C.5 now include the perturbation quality audit, the matched-mass shuffle control, and the Llama dd-filter results.
  • Section 4.2, Figure 5, and Appendix F.3-F.4 report the multi-seed results and the paired MATH500 and NoOp-MATH500 evaluation.
  • Section 4.3 and Appendices D and F.5–F.6 clarify the deviation bound and document the ablations on duration, strength, and normalization. Following Reviewer Nfjd’s suggestion, we softened causal wording throughout the manuscript to describe more precisely what token-level drift measures.

We believe these additions provide stronger empirical support for SCAPO, address the reviewers’ principal concerns, and clarify the scope of our conclusions. All revisions are highlighted in blue-colored text for ease of reference.

Thank you again for your careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.