- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper provides an empirical study evaluating and extending DAPO for mathematical reasoning on AIME benchmarks using Qwen2.5-Math-7B. The three key findings are:
- The standard deviation of DAPO's performance across different training seeds is approximately 0.99 percentage points, indicating that certain reported marginal improvements in the literature might not be reliable.
- Reducing the generation batch size from 256 to 128 produces nearly identical results, within the margin of error due to training-seed variance.
- The authors investigate several extensions of DAPO in order to improve token-level credit assignment. The authors propose new methods, called Pivotal Step Optimization (PSO) and Contrastive Step Reward (CSR), both of which rely on a shared contrastive signal . Their results are summarized in Table 3 and Appendix A.
Critical strengths
- The paper proposes novel methods for token-level credit assignment, called Pivotal Step Optimization and Contrastive Step Reward, both of which are built by introducing a per-token contrastive signal (defined in Appendix E), measuring the average difference in log probabilities between correct and wrong rollouts generated from the same input prompt .
- The authors demonstrate that DAPO's performance on AIME 2024 and 2025 remains statistically unaffected after reducing the generation batch size from 256 to 128, leading to measurable training efficiency gains.
Critical weaknesses
- The main algorithmic novelty does not appear to demonstrate any measurable benefit over DAPO. In fact, all variants of PSO measurably underperform the DAPO reference on pooled AIME accuracy (Table 3), with differences ranging from to the estimated SD of DAPO's accuracy across different training seeds ( percentage points). Therefore, it is unclear if PSO (and its variants) provides any measurable improvement over DAPO.
- While the second main contribution, concerning the reduction of the generation batch size from 256 to 128, demonstrates a measurable improvement in training efficiency, it is achieved via relatively straightforward hyperparameter tuning, rather than novel algorithmic ideas, limiting the potential novelty of the paper.
Questions for authors
- Are there any experimental settings where some variant of PSO demonstrates a statistically significant improvement over the DAPO and other credit-assignment techniques?
- Could the authors provide a more detailed comparison of the training efficiency of PSO and DAPO? Does PSO improve convergence speed?
- Does the generation batch reduction generalize to other, potentially harder mathematical reasoning datasets (e.g., BeyondAIME, HMMT Feb 2025), where the fraction of usable rollout groups may be significantly lower?
2 comments
Thank you for the clarifications regarding PSO+CSR and additional experiments. My concerns regarding the limited novelty and the limited scope of the contributions remain. Therefore, I maintain my score.
Reply to Reviewer uwKR
Thank you for the careful summary of the three findings.
Weakness 1, The paper does not propose PSO or CSR. It audits them. Over several months we built nine variants of DAPO, and none beat the baseline. The paper ask what measurement would have told us that earlier (three baseline seeds) and what survives that measurement (the rollout-budget result). Your first weakness, that the PSO variants lies one to three seed SDs below DAPO, is the same result we report in Table 3 and state as our conclusion in Section 5.4, where ten of the eleven variant runs fall below the baseline mean, by 1.45 points on average. We agree with it. Your reading shows that the framing was not clear enough, and the revision moves the negative result to the front of the abstract and introduction.
Weakness 2, the halving amounts to hyperparameter tuning. We agree that the change is the single configuration value, and Section 4.2 already says that equal accuracy “is not the finding”. The contribution lies instead in the measurement that makes the change possible. Read at face value, DAPO’s 75% discard statistic suggests that only 25% of generated groups are usable. Counted directly, 60 to 76% are, on three models. That gives a sizing rule (generation batch × mixed rate ≥ training batch) and a predicted breaking point at 96 prmopts, which we now confirm with three seeds (see the general comment).
Question 1. Does any PSO variant significantly outperform DAPO in some setting? No. We ran PSO at three amplification strengths, with four weighting schemes, and with and without the step controller, and in none of these settings is any variant better than DAPO on the pooled view. The only positive contrast is internal, between two of our own versions (rank-normalised v5 over hard-cutoff v1), and it is a single-seed difference that we do not claim.
Question 2. Training efficiency and convergence speed. PSO adds no rollouts and only reuses log-probabilities already computed, so its per-step cost equals DAPO's. It does not converge faster. Training-time AIME 2024 accuracy (32 samples per problem, mean over the validation points in each window):
(The early log of DAPO seed 1 was not kept, so seeds 2 and 3 are shown.) The one variant that looked faster on AIME 2024, FIPO-K, is matched by DAPO at the same steps on the pooled view, 24.33 against 24.33 (Appendix A.2).
Qeustion 3. Harder datasets. We did not train on a second dataset, and we say so in the limitations. One clarification: the fraction of usable groups depends on the training prompts and the policy being trained, so an evaluation set such as BeyondAIME or HMMT does not affect it. A harder training set would lower the rate, and the sizing rule then gives the batch needed; the rate comes from a counter that costs nothing. Our closest evidence is a weaker model on the same data. For this model the prompts start out hard. Qwen2.5-Math-1.5B begins at a 7% mixed rate, which is the situation a harder training set would create, and still reaches the same 66 to 73% plateau as the 7B model. In this low-rate phase the rule held in our two new runs at batch 128, where the first ten optimiser steps (mixed rate about 8%) needed 6.8 and 6.6 batches of 128 on average against the predicted 64/(128 × 0.08) ≈ 6. From step 32 on, one batch was enough at every step, and the accuracy of the 256 run lies between the two 128 seeds (see the general comment).