acceptodds
Under review as a conference paper at ICLR 2027

Half the Rollouts, No Measurable Cost: A Seeded Audit of Dynamic Sampling and Credit Assignment in DAPO

Abstract

Reinforcement learning for mathematical reasoning is routinely evaluated on AIME, where a single problem is worth over three points, and improvements of one to two points are commonly reported from a single training run. We ask whether such improvements are measurable, and answer with a pre-registered, seeded audit of DAPO on Qwen2.5-Math-7B. Three seeds of the unmodified baseline, read out at 128 samples per problem on AIME 2024 and 2025 pooled, span 1.96 points with a standard deviation of 0.99; across thirteen single-seed runs the rank correlation between the two AIME years is −0.03. Against that unit, we built nine variants: token-level credit-assignment methods that reweight advantages by pivotality, add contrastive step rewards, or importance-weight positions by future influence, and a prompt selector driven by a tracked difficulty estimate; none is individually distinguishable from the baseline under its three-seed prediction interval, and the modified-recipe runs sit on average about 1.5 points below it. A KL-targeted step-size controller, holding each variant at the baseline's late-phase KL, does not recover the largest harm, from high-amplification reweighting. The positive finding concerns the rollout budget: the 75% of prompts that DAPO's dynamic sampling discards are mostly surplus truncated once the training batch is filled; under a trained policy the mixed-group rate is 60–76%, far above the 25% implied by the discard statistic. Halving the generation batch trains the same 64 groups per step at half the rollouts and 24% less wall-clock, with a pooled accuracy difference of −0.18 points over three seeds each, inside the seed standard deviation on both sides.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM reasoning, GRPO, DAPO, credit assignment, dynamic sampling, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uwKROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper provides an empirical study evaluating and extending DAPO for mathematical reasoning on AIME benchmarks using Qwen2.5-Math-7B. The three key findings are:

  1. The standard deviation of DAPO's performance across different training seeds is approximately 0.99 percentage points, indicating that certain reported marginal improvements in the literature might not be reliable.
  2. Reducing the generation batch size from 256 to 128 produces nearly identical results, within the margin of error due to training-seed variance.
  3. The authors investigate several extensions of DAPO in order to improve token-level credit assignment. The authors propose new methods, called Pivotal Step Optimization (PSO) and Contrastive Step Reward (CSR), both of which rely on a shared contrastive signal ctc_t. Their results are summarized in Table 3 and Appendix A.
Critical strengths
  • The paper proposes novel methods for token-level credit assignment, called Pivotal Step Optimization and Contrastive Step Reward, both of which are built by introducing a per-token contrastive signal ct=l‾tC−l‾tWc_t = \overline{l}_t^{\mathcal{C}} - \overline{l}_t^{\mathcal{W}} (defined in Appendix E), measuring the average difference in log probabilities between correct and wrong rollouts generated from the same input prompt xx.
  • The authors demonstrate that DAPO's performance on AIME 2024 and 2025 remains statistically unaffected after reducing the generation batch size from 256 to 128, leading to measurable training efficiency gains.
Critical weaknesses
  • The main algorithmic novelty does not appear to demonstrate any measurable benefit over DAPO. In fact, all variants of PSO measurably underperform the DAPO reference on pooled AIME accuracy (Table 3), with differences ranging from 1×1\times to 3×3\times the estimated SD of DAPO's accuracy across different training seeds (0.990.99 percentage points). Therefore, it is unclear if PSO (and its variants) provides any measurable improvement over DAPO.
  • While the second main contribution, concerning the reduction of the generation batch size from 256 to 128, demonstrates a measurable improvement in training efficiency, it is achieved via relatively straightforward hyperparameter tuning, rather than novel algorithmic ideas, limiting the potential novelty of the paper.
Questions for authors
  1. Are there any experimental settings where some variant of PSO demonstrates a statistically significant improvement over the DAPO and other credit-assignment techniques?
  2. Could the authors provide a more detailed comparison of the training efficiency of PSO and DAPO? Does PSO improve convergence speed?
  3. Does the generation batch reduction generalize to other, potentially harder mathematical reasoning datasets (e.g., BeyondAIME, HMMT Feb 2025), where the fraction of usable rollout groups may be significantly lower?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer uwKR

Thank you for the careful summary of the three findings.

Weakness 1, The paper does not propose PSO or CSR. It audits them. Over several months we built nine variants of DAPO, and none beat the baseline. The paper ask what measurement would have told us that earlier (three baseline seeds) and what survives that measurement (the rollout-budget result). Your first weakness, that the PSO variants lies one to three seed SDs below DAPO, is the same result we report in Table 3 and state as our conclusion in Section 5.4, where ten of the eleven variant runs fall below the baseline mean, by 1.45 points on average. We agree with it. Your reading shows that the framing was not clear enough, and the revision moves the negative result to the front of the abstract and introduction.

Weakness 2, the halving amounts to hyperparameter tuning. We agree that the change is the single configuration value, and Section 4.2 already says that equal accuracy “is not the finding”. The contribution lies instead in the measurement that makes the change possible. Read at face value, DAPO’s 75% discard statistic suggests that only 25% of generated groups are usable. Counted directly, 60 to 76% are, on three models. That gives a sizing rule (generation batch × mixed rate ≥ training batch) and a predicted breaking point at 96 prmopts, which we now confirm with three seeds (see the general comment).

Question 1. Does any PSO variant significantly outperform DAPO in some setting? No. We ran PSO at three amplification strengths, with four weighting schemes, and with and without the step controller, and in none of these settings is any variant better than DAPO on the pooled view. The only positive contrast is internal, between two of our own versions (rank-normalised v5 over hard-cutoff v1), and it is a single-seed difference that we do not claim.

Question 2. Training efficiency and convergence speed. PSO adds no rollouts and only reuses log-probabilities already computed, so its per-step cost equals DAPO's. It does not converge faster. Training-time AIME 2024 accuracy (32 samples per problem, mean over the validation points in each window):

StepsDAPO seed 2DAPO seed 3PSO v5 A=1.2A=1.5A=2.0
10–10019.319.119.019.218.7
110–20030.130.828.227.828.6
210–30031.233.832.330.531.7

(The early log of DAPO seed 1 was not kept, so seeds 2 and 3 are shown.) The one variant that looked faster on AIME 2024, FIPO-K, is matched by DAPO at the same steps on the pooled view, 24.33 against 24.33 (Appendix A.2).

Qeustion 3. Harder datasets. We did not train on a second dataset, and we say so in the limitations. One clarification: the fraction of usable groups depends on the training prompts and the policy being trained, so an evaluation set such as BeyondAIME or HMMT does not affect it. A harder training set would lower the rate, and the sizing rule then gives the batch needed; the rate comes from a counter that costs nothing. Our closest evidence is a weaker model on the same data. For this model the prompts start out hard. Qwen2.5-Math-1.5B begins at a 7% mixed rate, which is the situation a harder training set would create, and still reaches the same 66 to 73% plateau as the 7B model. In this low-rate phase the rule held in our two new runs at batch 128, where the first ten optimiser steps (mixed rate about 8%) needed 6.8 and 6.6 batches of 128 on average against the predicted 64/(128 × 0.08) ≈ 6. From step 32 on, one batch was enough at every step, and the accuracy of the 256 run lies between the two 128 seeds (see the general comment).

Thank you for the clarifications regarding PSO+CSR and additional experiments. My concerns regarding the limited novelty and the limited scope of the contributions remain. Therefore, I maintain my score.

Reviewer_7d6AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper audits rollout efficiency and credit assignment in DAPO, primarily using Qwen2.5-Math-7B. It measures training variability with three baseline seeds and evaluates several credit-assignment and prompt-selection variants. The main finding is that many discarded rollout groups are usable groups truncated after the training batch is filled.

Critical strengths
  1. This paper identifies a concrete source of wasted rollout computation. Distinguishing unusable groups from surplus usable groups explains why the discard rate can be misleading.

  2. This paper carefully distinguishes training-seed variability from checkpoint and evaluation variability.

Critical weaknesses
  1. The main efficiency result has limited scope beyond correcting an oversized generation batch in the tested implementation. The savings come from reducing surplus generation while preserving the number of groups used for each update.

  2. The credit-assignment audit has limited statistical power to establish the effects of the variants or explain their causes. Most variants are the authors' own designs and are evaluated with a single training run. The baseline prediction interval of approximately +/-4.9 percentage points is too wide to exclude practically relevant gains or losses.

Questions for authors
  1. Can the accuracy and runtime benefits of halving the generation batch be validated directly on one of the additional models or a second training dataset?

  2. What evidence supports ruling out step size as the explanation, beyond the failure to observe recovery in the single-run KL-controlled comparisons?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 7d6A

Thank you for the precise reading. We agree with both weaknesses and answer the questions with what we have.

Weakness 1, limited scope. We agree. The saving corrects an over-provisioned default and keeps the trained batch unchanged; the limitations section says this in those terms. We think it is worth reporting because the discard statistic is easy to misread. It suggests that only 25% of groups are usable, which is how we read it ourselves when we designed a prompt selector around it, whereas the measured rate is 60 to 76%. The measured rate and the sizing rule are what we offer, with the breaking point now confirmed on three seeds at 96 (general comment).

Weakness 2, limited statistical power. We agree, and the audit says so. With only three baseline seeds, the interval around the baseline is wide enough that no single run of these methods falls outside it, and we therefore claim nothing about any individual variant’s effect or cause. The ±4.9 interval is wide because a three-seed SD is poorly determined, and we chose not to hide that. Since submission we have nine seeds of baseline-type recipes (three each at 256, 128 and 96). If the three budgets are assumed to share one variance, the pooled SD is 0.93 on 6 degrees of freedom and the single-run interval is ±2.6. The conclusion does not change. The ±2.6 interval excludes the same two runs as the paper’s sensitivity interval (PSO v1 and PSO A=1.5 under the controller), and no others. Since the assumption of a shared variance was not pre-registered, we treat this as a sensitivity check, not as the rule.

Question 1. Halving validated on the additional model. We run this when the reviews arrived: Qwen2.5-Math-1.5B at generation batch 128, two seeds, 400 generation rounds each, against the existing 1.5B run at 256. The table is in the general comment.

  • Warm-up: As the rule predicts, halving saves nothing while the mixed rate is low. Over the first ten steps (mixed rate about 8%), a step needed 6.8 and 6.6 batches of 128, close to the predicted 6, against 3.5 batches of 256. The first 20 steps therefore cost about the same in prompts at either size (12.4k and 11.9k at 128, 12.5k at 256).
  • After warm-up: the last step that needed a second batch was step 26 in seed 1 and step 31 in seed 2 (step 14 at 256). After that, 0% of steps needed one. The mixed rate was 66 to 77% (interdecile), with a lowest logged value of 60% after step 100, or 77 groups where 64 are needed. A step took 5.4 and 5.2 minutes, against 8.6 at 256.
  • Cost of the warm-up: in 400 rounds the 128 runs reached optimiser steps 319 and 324, against 371 at 256. Reaching step 310 took 42 to 43% fewer rollouts, not 50%.
  • Accuracy at matched steps (290, 300, 310; mean of AIME 2024, 2025 and 2026): the comparison detects no cost, but because it sets two runs against one, it cannot rule out a cost of about one point. The run at 256 reads 12.0 and lies between the two seeds at 128, which read 10.9 and 12.3 and differ from each other by 1.4 points. On AIME 2026 alone, the readings are 9.7 and 12.1 at 128 against 11.7 at 256. A second seed at 256 would take more than 50 hours and cannot be run within the discussion period, so we will add it to the final version.
  • We did not run second training dataset and cannot in this window.

Question 2. Evidence for ruling out step size. There is none beyond what you describe and the sentence in the introduction overstated it. The evidence is two controlled comparisons, each between a pair of single runs. At A=1.5, the controlled PSO run is 1.83 points below controlled DAPO, a gap only somewhat larger than the SD of about 1.4 for difference of two single runs. At A=2.0, the pre-registered contrast read −0.92, but the two branches it was meant to separate were only 2 points apart. Section 5.4 already says that this shows non-recovery "within the resolution two single runs allow". The revision changes the introduction and the contribution list to match: the controller tests step size as the explanation and finds no recovery; it does not exclude it. Excluding it would need seeded pairs of controlled runs, which we do not have.

Thank you for the additional experiments and candid clarifications. The 1.5B results strengthen the efficiency finding, and the revised step-size claim addresses the overstatement. However, the efficiency contribution remains limited to correcting an oversized generation batch, while the credit-assignment audit still lacks sufficient statistical power to establish reliable conclusions about the variants’ effects. I therefore maintain my score.

Reviewer_YVavOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the reliability and efficiency of DAPO for mathematical reasoning. The authors first show that different training runs can lead to noticeable differences in performance, making it difficult to tell whether a new method actually improves the results. They then test several credit assignment methods, but none shows a clear improvement over the DAPO baseline. The main finding is that many generated rollouts are discarded simply because the training batch is already full. By reducing the generation batch size by half, the authors cut the number of rollouts by 50% and training time per step by 24%, without a significant drop in accuracy. The paper also studies how changes in credit assignment affect the size of policy updates.

Critical strengths

None

Critical weaknesses
  1. Limited novelty. The main finding is largely based on a specific DAPO implementation and its settings. The official verl documentation already shows that generation and training batch sizes can differ, and that extra samples are discarded once the training batch is full. Therefore, the paper mainly measures the impact of an existing behavior rather than discovering a new mechanism.

  2. Limited model coverage. The experiments mainly use Qwen2.5 models, which are relatively old. It is unclear whether the same findings would hold for newer models or other model families.

  3. Writing and presentation. The paper could be better organized and more focused. Some sections read more like a research log than a conference paper, making it difficult to follow the main contributions and conclusions.

Questions for authors

Does CSR actually produce different advantages for different tokens within the same response? CSR seems to change only the sequence-level reward.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer YVav

Thank you. Your question identified a real error in the paper.

Question: Do CSR produce different advantages for different tokens of a response? No. You are correct. CSR add α⋅sgn(Ri)⋅c^t\alpha \cdot \mathrm{sgn}(R_i) \cdot \hat{c}_t to the reward at each position, but the group-relative advantage is computed from the summed reward, so what enters is

Ri′=Ri+α⋅sgn(Ri)⋅∑tc^tmi,t,A^i=Ri′−μ′σ′R'_i = R_i + \alpha \cdot \mathrm{sgn}(R_i) \cdot \sum_t \hat{c}_t m_{i,t}, \qquad \hat{A}_i = \frac{R'_i - \mu'}{\sigma'}

and A^i\hat{A}_i is broadcast to every token of response ii. CSR is therefore sequence-level reward shaping: it shifts each response’s scalar reward by an amount set only by the response’s outcome sign and length, and leaves the advantage identical across its tokens. The same hold for the CSR stage of the three PSO+CSR variants, whose token-level variation comes only from the PSO stage.

Changes in the revision:

  • Abstract and introduction: CSR is no longer listed among the token-level methods. The count becomes seven methods with a token-level component, one sequence-level reward-shaping method (CSR), and one prompt selector.
  • Appendix E and the taxonomy table: CSR is described as sequence-level shaping, with the formula above. The paragraph that explained its failure by per-position interference is removed, since no per-position effect reaches the gradient.
  • Consequence for the claims: the CSR runs are not evidence about dense token-level rewards, and the paper no longer presents them as such. Tables 1 to 3 and all counts are unchanged, because the CSR runs were stopped early and appear only in Appendix A.1.

Weakness 1, limited novelty. We agree that generation and training batch sizes are already separate settings, that dropping the surplus groups is documented behaviour of the framework, and that we did not discover this mechanism. Section 4.2 says equal accuracy "is not the finding". What the documentation does not give is the rate ==> the 75% discard statistic suggests 25% usable groups, and the direct count is 60 to 76% on three models. The rate gives a sizing rule (generation batch × mixed rate ≥ training batch), and the rule predicts where halving stops working. We registered that prediction before running at 96 prompts; across three seeds, a second batch was needed at 19–26% of steps, against 0% at 128. We are presenting this as a measured baselines for rollout-efficiency methods and not as a new method.

Weakness 2, model coverage. We agree with this. All three models are from the Qwen2.5 family, and the limitations say that a non-Qwen model was not tested. We cannot add a newer family within the discussion period. Within the discussion period, we instead repeated the halving on Qwen2.5-Math-1.5B with two seeds. Once the warm-up is over, a single batch of 128 fills every step, and the 256 run’s accuracy falls between the two 128 seeds (general comment).

Weakness 3, writing. We totally accept this one. Parts of Sections 3 to 5 narrate the order in which we learned things. In the revision the first paragraph states the audit and its negative result, each section opens with its claim, and the passages that recount earlier readings ("with two seeds they had looked like…", "we had pre-registered the opposite expectation…") move to the appendix.

Thanks for the response. The clarification on CSR is clear, and I appreciate the authors acknowledging the issue and revising the paper. However, I still have some concerns about the novelty, generalization, and overall writing and presentation of the paper.

I will keep my original rating. Thanks again for the effort!

AuthorsOpenReview ↗

Reply to Reviewer YVav

Thank you for reading our response and for CSR question which corrected a real error in the paper.

We understand that the three remaining concerns are judgments about the paper's scope and we will not repeat our response. For the record and to be helpful for others who will read this, one sentence on each:

  • Novelty. We claim no new mechanism. Instead, the paper adds the measured mixed rate (60 to 76%, against the 25% the discard statistic suggests) and the sizing rule it give, and the revision tests that rule twice, at the breaking point of 96 prompts with three seeds and in the warm-up of the 1.5B model, where the rule predicts that halving saves nothing.
  • Generlization. The results are for the Qwen2.5 family on one training dataset, and the limitations section says so.
  • Writing. The revision states the audit and its negative result in the abstract and the first paragraph, opens Sections 3 to 5 with their claims, and moves the readings that later data overturned to Apendix A.16. If particular passages still read as a log, we would be glad to know which ones, and we will revise them in the final version.

Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

Thank you for three reviewers. We think two points in the reviews are errors on our side. We correct both in the revised PDF.

1. CSR is not token-level (YVav is right). Out CSR code add a per-token term to the reward, but the group-relative advantage sums the reward over the response before it is broadcast. As a result, CSR only alters the scalar reward of each response, and every token in that response still receives one shared advantage. The abstract, introduction and Appendix E described it as token-level credit assignment. In the revision, we now describe it as sequence-level reward shaping. No number in Tables 1 to 3 changes, because the CSR runs were stopped early and appear only in Appendix A.1.

2. "Rules out step size" was too strong (7d6A, Question 2). The introduction said the step controller "rules out step size as the explanation". Section 5.4 supports only the weaker statement that holding the step did not recover loss back, as far as two single runs can resolve. The revision states only the weaker claim.

New results since submission.

  • Generation batch 96 and 3 seeds (There was one seed at 96 at the submission).
Generation batchSeedsPooled AIME 24+25 (mean ± SD)Steps needing a 2nd batch (after step 20)
256 (DAPO)325.31 ± 0.990%
128325.13 ± 0.370%
96324.74 ± 1.2226%, 21%, 19%

In the submission, generation batch 96 had a single seed, which read 23.39 pooled, the lowest of the three budgets. The two new seeds read 25.08 and 25.75, so that low reading was seed noise. The sizing rule predicted that one batch would stop being enough at 96 prompts but not at 128. In all three seeds this holds ==> after step 20 a second batch was needed at 19–26% of steps at 96, and at 0% at 128.

  • Halving on a second model: We trained Qwen2.5-Math-1.5B at generation batch 128 with two seeds, for the same 400 generation rounds as the existing 1.5B run at 256.

    Mechanism: When the mixed rate is low, a step needs about 64/(batch × rate) generation batches, so the number of prompts a step consumes does not depend on the batch size, and halving saves nothing. The first 20 optimiser steps bear this out: 97 and 93 batches of 128 (12.4k and 11.9k prompts) against 49 batches of 256 (12.5k prompts). From step 32 on, no step needed a second batch in either seed. The mixed rate settled at 66 to 77% (interdecile), and its lowest logged value after step 100 was 60%, which is 77 mixed groups where 64 are needed.

    Accuracy (mean@128, averaged over steps 290, 300 and 310):

Generation batchAIME 2024AIME 2025AIME 2026Mean of three
256, one seed13.710.511.712.0
128, seed 114.18.99.710.9
128, seed 214.410.312.112.3

The 256 run lies between the two 128 seeds, which differ from each other by 1.4 points. The comparison shows no cost. Because there is only one seed at 256, but, it cannot rule out a cost of about one point, and the revision states this. Reaching step 310 took 42–43% fewer rollouts at 128 rather than 50%, since the warm-up phase give no saving.

Scope of the paper. Reviewer uwKR understood the paper as proposing PSO and CSR. Our intent was an audit, with the negative result for our own methods as its conclusion; the submitted version said this only at the start of the introduction’s second paragraph. That a careful reader took it otherwise is a presentation failure, and the revision states the negative result in the abstract's first lines and in the first paragraph.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.