- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes StepRS-GRPO,a method for reinforcement learning in diffusion language models that reweights terminal-reward advantages by adjusting a risk coefficient according to the denoising state. The method is evaluated on mathematical reasoning tasks in terms of accuracy,problem-solving coverage,and output diversity.The experiments compare the proposed schedule with constant-risk,state-weight-matched,and advantage-magnitude-matched controls,providing some empirical support for state-dependent risk scheduling.However,the precise definition of the denoising-state coordinate,the rationale for risk-parameter selection,and the robustness of the reported gains across independent training runs require further clarification.
Critical strengths
StepRS-GRPO addresses a relevant optimization question for diffusion language models:whether the terminal-reward learning signal can be adapted to the denoising process rather than applying a fixed risk coefficient throughout generation.The inclusion of constant-risk,state-weight-matched,and advantage-magnitude-matched controls is useful,as it helps assess whether the observed effects are specific to the proposed scheduling strategy rather than simply to reweighting or scale matching.
The evaluation considers accuracy,problem-solving coverage,and output diversity,providing a broader picture of performance than accuracy alone.These experiments offer preliminary empirical support for the proposed method.
Critical weaknesses
The main concern is the reproducibility of the denoising-state coordinate.The paper discusses global trajectory progress,within-block progress, and mask ratio,but does not clearly specify which definition is used for each model.For block-based diffusion models,whether the coordinate resets at the beginning of each block can change the assigned risk coefficients and affect the interpretation of the scheduling-direction comparisons.
A second concern is the calibration and selection of the risk coefficient.The paper does not clearly report the source and size of the calibration data,the number of samples per problem,or the relationship between the calibration data and the test benchmarks.In particular,the calibration value for SDAR-4B corresponds to a risk coefficient of approximately 1.08, whereas the final default schedule uses 1.5 as its upper endpoint.The rationale for this choice,and whether test results were involved in parameter selection,remain unclear.
Finally,the three repeated evaluations use fixed training checkpoints and vary only inference random seeds.Their error bars therefore reflect inference-sampling variability rather than variability across independent training runs.This distinction matters because the reported pass@1 improvements over constant-risk RS-GRPO with a risk coefficient of 1.5 are small on SDAR-4B:0.20 percentage points on AMC23,0.40 on AIME2024,and 0.50 on AIME2025.Without multi-seed training comparisons and suitable uncertainty estimates,the stability of the scheduling gains remains unestablished.
Questions for authors
1.For each evaluated model,what exact coordinate is used to determine the denoising state: global trajectory progress,within-block progress, or mask ratio?For block-based models,does the coordinate reset at the beginning of each block? Please provide an example generation trajectory with its per-step coordinates and resulting sequence of risk coefficients,including the scheduling-direction comparisons,so that the method can be reproduced.
2.Could the authors report the complete calibration procedure,including the source and size of the calibration data,samples per problem, its relationship to the evaluation benchmarks,the evaluation metric, and the search range?How was the final upper endpoint of 1.5 selected when the SDAR-4B calibration corresponds to approximately 1.08,and were any test benchmark results used in choosing the final setting?
3.Could the authors compare the proposed schedule with constant-risk RS-GRPO at a risk coefficient of 1.5 across multiple independent training seeds?Please report pass@1 and pass@k,with uncertainty estimates at larger sampling budgets,particularly on SDAR-4B for AMC23,AIME2024,and AIME2025.This would help distinguish inference-sampling variability from training-run variability and assess whether the reported gains are stable.
4 comments
Response to Reviewer tnhr (2/4)
W2.
Calibration and selection of the risk coefficient.
Response: Prior to RL training, the initial solve-rate proxy is measured on the base model using rollouts per prompt at temperature under the binary math verifier, giving :
| Backbone | Calibration Source | Samples per Problem | Initial Solve Rate | Calibrated Scale | Evaluated Upper-Endpoint Grid | Selected Default Schedule |
|---|---|---|---|---|---|---|
| Qwen3-0.6B-MDLM | GSM8K train subset (500 prompts) | 16 | 0.22 | 0.25 | ||
| SDAR-1.7B-Chat | MATH train L3–5 subset (500 prompts) | 16 | 0.51 | 0.71 | ||
| SDAR-4B-Chat | MATH train L3–5 subset (500 prompts) | 16 | 0.66 | 1.08 |
- Why was selected over and role of evaluation curves: As stated in Section 3.3 and Section 4.1 of our manuscript, serves as a capability-guided starting scale to anchor a coarse grid search over standard half-integer endpoints (), rather than an analytical optimum of the clipped policy gradient. On SDAR-4B, the directly calibrated schedule already outperforms across all 28 benchmark-budget pairs and is within points of on pass@ (Figure 6 / Figure 7). We explicitly disclose in Section 4.1 and Section 5 that final schedule and checkpoint selection across the coarse grid was guided by inspecting overall evaluation pass@ curves (sharing one common schedule and checkpoint across all benchmarks for each backbone, including for Constant RS-GRPO and all controls C1–C3).
Response to Reviewer tnhr (3/4)
W3.
Multi-seed training comparisons, suitable uncertainty estimates, and more evaluations.
Response: Due to limited time and compute during the rebuttal period, we could not run multiple independent RL training seeds on SDAR-4B. Instead, we highlight the uncertainty-bounded separation already established in our manuscript and report two additional evaluations on our existing SDAR-4B checkpoints: verified-correct reasoning-trace diversity and out-of-domain code generation.
-
Existing uncertainty estimates and consistency in the manuscript:
- Coverage and cross-backbone consistency: While SDAR-4B pass@ margins over Constant RS-GRPO () are modest (+0.2 on AMC23, +0.4 on AIME2024, +0.5 on AIME2025), three-seed error bars show wider separation at larger budgets: reaching +0.8, +1.1, and +2.2 at pass@, outperforming Constant RS-GRPO in 26 out of 28 comparisons, with the rest 2 tied (Section 4.3, Appendix D.3). The same ordering holds across two additional independent training runs on distinct backbones: SDAR-1.7B (28/28 vs. constant ) and Qwen3-0.6B-MDLM (all budgets on GSM8K/MATH500 and on AMC23 vs. constant ; Appendix D.4).
- Low-variance diversity gains: Across three inference seeds, answer-removed reasoning-trace diversity on SDAR-4B (Section 4.3, Figure 4) separates cleanly from GRPO (+2.3–6.6 in ; +1.3–3.5 in ), and the unique-answer ratio (Appendix D.5, Figure 12) shows tightly bounded gains on every model–benchmark pair (e.g., on SDAR-4B MATH500).
-
Verified-correct reasoning-trace diversity:
- Using the existing SDAR-4B evaluation rollouts ( samples/problem, ), we strip the final boxed answer and its containing sentence, filter strictly to verified-correct responses on problems with correct samples per seed, and compute (Self-BLEU-4 complement, Eq. 23) and (
all-MiniLM-L6-v2pairwise cosine distance, Eq. 25):
- Using the existing SDAR-4B evaluation rollouts ( samples/problem, ), we strip the final boxed answer and its containing sentence, filter strictly to verified-correct responses on problems with correct samples per seed, and compute (Self-BLEU-4 complement, Eq. 23) and (
| SDAR-4B Method, b=4 | MATH500 ( / ) | AMC23 ( / ) | AIME 2024 ( / ) | AIME 2025 ( / ) |
|---|---|---|---|---|
| Base | 34.2 / 6.5 | 40.8 / 8.1 | 46.6 / 9.9 | 49.6 / 10.6 |
| GRPO | 31.9 / 5.8 | 38.1 / 7.4 | 44.2 / 8.9 | 45.8 / 9.3 |
| Constant RS-GRPO () | 35.4 / 7.0 | 42.2 / 8.7 | 47.5 / 10.3 | 49.8 / 10.9 |
| StepRS-GRPO () | 38.6 / 8.2 | 46.8 / 10.0 | 49.4 / 10.9 | 50.8 / 12.2 |
- Out-of-domain code generation transfer:
- We evaluate the same math-trained SDAR-4B checkpoints zero-shot on HumanEval (164 problems) and MBPP (500 problems) using our standard SDAR-4B decoding protocol. Correctness is verified via standard unit-test execution, and unbiased pass@ (Eq. 10) is reported:
| SDAR-4B Method, b=4 | HumanEval (pass@1 / @16 / @64) | MBPP (pass@1 / @16 / @64) |
|---|---|---|
| Base | 49.4 / 74.8 / 81.7 | 54.6 / 76.9 / 82.4 |
| GRPO | 52.1 / 77.1 / 82.3 | 56.8 / 77.8 / 83.0 |
| Constant RS-GRPO () | 53.4 / 78.2 / 84.4 | 57.2 / 79.3 / 84.4 |
| StepRS-GRPO () | 54.3 / 79.5 / 85.6 | 57.4 / 80.2 / 85.2 |
StepRS-GRPO consistently surpasses both GRPO and Constant RS-GRPO () on verified-correct trace diversity and coding benchmarks with clear separation beyond sampling variability. We will incorporate these results in a future version.
Response to Reviewer tnhr (4/4)
Answers to Questions:
Q1.
For each evaluated model, what exact coordinate is used to determine the denoising state: global trajectory progress, within-block progress, or mask ratio? For block-based models, does the coordinate reset at the beginning of each block? Please provide an example generation trajectory with its per-step coordinates and resulting sequence of risk coefficients, including the scheduling-direction comparisons, so that the method can be reproduced.
Answer: Please refer to our response and tables under W1 above.
Q2.
Could the authors report the complete calibration procedure, including the source and size of the calibration data, samples per problem, its relationship to the evaluation benchmarks, the evaluation metric, and the search range? How was the final upper endpoint of 1.5 selected when the SDAR-4B calibration corresponds to approximately 1.08, and were any test benchmark results used in choosing the final setting?
Answer: Please see our response and calibration summary table under W2 above.
Q3.
Could the authors compare the proposed schedule with constant-risk RS-GRPO at a risk coefficient of 1.5 across multiple independent training seeds? Please report pass@1 and pass@k, with uncertainty estimates at larger sampling budgets, particularly on SDAR-4B for AMC23, AIME2024, and AIME2025.
Answer: Please see our response under W3 above.
Response to Reviewer tnhr (1/4)
Dear Reviewer
tnhr,We greatly appreciate your thoughtful assessment and constructive suggestions. Below, we offer detailed clarifications and supplementary evaluations in response to your comments. If these responses satisfy your concerns, we would be grateful if you would consider updating your evaluation.
Responses to Weaknesses:
W1.
Response: Every evaluated model uses the within-block local mask fraction: uqj=∣Iqj∣1∑p∈Iqj1x~qj,p=[MASK] where Iqj is the set of valid response positions in the active decoding block (excluding prompt, padding, and future blocks) and x~qj is the partially revealed text in state si,t. For all models, uqj resets to 1.0 at the start of each fully masked block and decreases as tokens in that block are unmasked.
StableDRL, a 4-token block with K∈4,3,2,1 remaining masks takes uqj=K/4∈1.0,0.75,0.50,0.25 (K=0 has no masked-token loss). For a 4-step single-token reveal trajectory under b(u)=0.75+0.75u versus the reversed control C2 (b(1−u) with mass-matching factor κq=∑jvqjw~qj∑jvqjwqj):Note that because u=0 is inactive, evaluating b(1−u) shifts the active coefficient set from 1.5,1.3125,1.125,0.9375 to 0.75,0.9375,1.125,1.3125 prior to mass normalization by κq.
TraceRL, static decoding reveals 2 tokens per step over 16 steps per block, visiting uqj∈32/32,30/32,…,2/32 and evaluating b(u)=0.25+0.25u∈0.5000,0.4844,…,0.2656. Even when trace shrinkage (s=8) groups 8 consecutive rounds into one forward pass, each target token retains its exact pre-reveal coordinate uqj and coefficient b(uqj).We will incorporate these exact coordinate specifications, worked trajectories, and ablation results in a future version.