- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how teacher supervision is aggregated in on-policy distillation. DOpt-OPD equalizes total loss weight across problems and pools reasoning spans from multiple rollouts of each problem. It constructs low-dimensional LM-head gradient proxies from hidden states and sparse student–teacher corrections, then allocates supervision using a single ridge-leverage update from a uniform active-span design. The theoretical analysis establishes near-optimality under orthogonal direction groups for sufficiently small ridge values and provides a computable optimality-gap bound for general geometries. Experiments evaluate two main teacher–student scales on seven mathematical reasoning benchmarks, with component ablations, length-stratified analyses, repetition diagnostics, and a supplementary comparison using a 0.6B student.
Critical strengths
1.The paper is well organized, with a clear motivation and a fairly complete experimental analysis. The authors first identify two issues in OPD supervision allocation, then introduce the corresponding solutions and provide experiments to examine each part. The length-based analysis, component ablations, and redundancy diagnostics are well connected to the proposed method. This makes the overall contribution easier to understand and evaluate. 2.The problem is meaningful, and the reported results provide reasonably convincing evidence for the proposed allocation strategy. At both main teacher–student scales, DOpt-OPD achieves the highest seven-benchmark macro-averaged Avg@8 and Pass@8 among OPD, TIP, EOPD, and FiRe-OPD. Table 3 and its per-task expansion in Table 7 show additional gains over both problem normalization and uniform span allocation, supporting the contribution of the leverage-based weighting.
Critical weaknesses
The practical computational overhead and scalability are not sufficiently characterized. Appendix C.4 provides a complexity analysis and explains that existing hidden states and teacher signals are reused, so no additional teacher forward pass is required. However, the analysis excludes token-sketch construction, and the paper reports no direct measurements of wall-clock training time, peak GPU memory, or throughput. Training responses are capped at 7,168 tokens; although evaluation permits responses up to 31,744 tokens, this does not establish the computational cost of the allocator during longer-trajectory training. End-to-end overhead measurements, including gradient-proxy construction, would help assess whether the reported performance gains justify the additional training cost.
Questions for authors
1.Appendix F.1 already documents substantial textual repetition in actual DAPO-Math-17K rollouts. Beyond this text-based audit, how common is severe directional redundancy comparable to the imbalance modeled in Appendix E, and how representative are the nine repeated-suffix cases in Appendix F.3 of naturally occurring repetition? 2.What is the actual additional training cost of DOpt compared with standard OPD, including the construction of gradient proxies? 3.The method uses low-dimensional LM-head gradient proxies to estimate redundancy between reasoning spans. While the empirical results are encouraging, it would be useful to understand how closely these proxies reflect full-model gradient similarities. Have the authors considered a small-scale comparison between the proxy-based and full-gradient-based allocation weights?
3 comments
Part 2 (W/Q2): Actual training time and memory overhead
(1) Proxy construction and weight allocation
We measured proxy construction and weight allocation over the full DAPO-Math-17K dataset, using four rollouts per problem, an 8×8 proxy, and 32-token spans. The mean time is 1.445 ms per problem, corresponding to a linear estimate of 0.185 s for a step containing 128 problems.
(2) End-to-end training overhead
We compared OPD and DOpt-OPD at training response caps of 7,168, 16,384, and 31,744 tokens, using 128 problems per step, four rollouts per problem, and FP32. The end-to-end timings include proxy construction and weight allocation. Thus, the longer-response settings measure costs during training, rather than only allowing longer responses at evaluation.
| Training response cap | Precision | Method | Peak memory (MiB/GPU) | Mean response tokens | Mean step time (s) | Throughput (tokens/s/GPU) |
|---|---|---|---|---|---|---|
| 7,168 | FP32 | OPD | 70,211 | 2,436.72 | 222.70 | 1,481.02 |
| 7,168 | FP32 | DOpt-OPD | 72,371 | 2,782.73 | 252.99 | 1,465.94 |
| 16,384 | FP32 | OPD | 72,379 | 3,271.53 | 445.70 | 974.84 |
| 16,384 | FP32 | DOpt-OPD | 72,411 | 3,837.50 | 520.58 | 967.08 |
| 31,744 | FP32 | OPD | 80,999 | 3,658.42 | 658.74 | 740.10 |
| 31,744 | FP32 | DOpt-OPD | 81,103 | 4,210.84 | 751.62 | 731.16 |
Across these settings, DOpt-OPD increases step time by 13.6%–16.8%, alongside a 14.2%–17.3% increase in mean response length. Throughput decreases by approximately 0.8%–1.2%, while peak memory increases by at most 3.1%.
The similar increases in response length and step time, together with nearly unchanged throughput, suggest that the longer steps are primarily due to longer responses requiring more generation and training computation. Proxy construction and weight allocation are estimated to take only 0.185 s per step, which is negligible relative to the full training-step time.
Part 3 (Q3): Comparison between the proxy and full-model gradients
(1) Agreement between allocation-weight rankings
At each of checkpoints 50 and 139, we selected 20 problems. For each problem, we used 64 candidate spans of 32 tokens from four natural rollouts and eight distinct held-out spans to evaluate update effectiveness. We compared three representations using the same candidate spans and teacher signals: the 8×8 (64-D) proxy, exact output-layer gradients using the same correction signal without random compression, and full-model gradients computed by backpropagation. We normalized the directions and computed allocation weights using the same ridge parameter (0.001) and D-optimal allocation rule.
| Checkpoint | Spearman with full-model weights [95% interval] | Spearman with exact output-layer weights [95% interval] |
|---|---|---|
| step50 | 0.735 [0.622, 0.830] | 0.787 [0.691, 0.882] |
| step139 | 0.387 [0.326, 0.439] | 0.606 [0.579, 0.648] |
We use Spearman’s rank correlation coefficient to compare weight rankings. At both checkpoints, proxy-based weights are positively correlated with full-model weights and more closely aligned with exact output-layer weights. This indicates agreement in identifying which spans should receive greater weight.
(2) Actual one-step update effectiveness
We fixed the rollouts and teacher signals and started each comparison from identical model parameters. Ordinary SGD used the same learning rate for Proxy and Uniform. For equal-L2 updates, we adjusted the Proxy learning rate to match Uniform’s parameter-displacement L2 norm. We then measured objective reduction on the eight held-out spans, which were excluded from weight computation and the update. Each checkpoint includes 20 problems; values below are mean improvements over Uniform in units of .
| Checkpoint | Ordinary SGD: Proxy − Uniform [95% interval] | Equal-L2 update: Proxy − Uniform [95% interval] |
|---|---|---|
| step50 | +3.855 [+1.968, +7.503] | +1.768 [+0.272, +4.701] |
| step139 | +1.245 [+0.965, +1.751] | +0.571 [+0.206, +1.251] |
At both checkpoints, the mean improvements are positive under ordinary SGD and equal-L2 updates, with all 95% intervals above zero. The benefit remains after matching update magnitude, indicating that the proxy’s advantage on these problems is not solely due to taking a larger step.
Proxy dimension and training effectiveness. Increasing proxy dimension generally reduces random compression error, but the effect of DOpt-OPD depends on how these directions jointly determine the allocation weights, rather than on dimensionality alone. We evaluated models trained separately with 8×8 and 16×16 proxies under identical output caps on seven benchmarks. At caps of 7,168 and 16,384 tokens, 8×8 achieves 2.79 and 2.60 percentage points higher Pass@8 despite slightly longer responses and higher truncation rates. At 31,744 tokens, its Avg@8/Pass@8 advantage is 1.15/2.86 points. Increasing proxy dimension therefore does not guarantee better training outcomes; we selected 8×8 based on its allocation effectiveness and final evaluation performance.
Part 1 (Q1): Directional redundancy in natural rollouts and the representativeness of the repeated-suffix cases
Thank you for asking about directional redundancy in natural rollouts and the representativeness of the nine repeated-suffix cases. We evaluated the same 500 DAPO problems at checkpoints 0, 50, 100, and 139, generating four natural rollouts per problem. Problems were selected independently of answer correctness. We grouped similar proxy directions using complete-linkage clustering with an absolute cosine-similarity threshold of 0.75.
Comparison with Appendix E. We measured group-size imbalance using the coefficient of variation (CV), defined as the standard deviation of group sizes divided by their mean. The Moderate and Severe settings in Appendix E have CVs of approximately 0.569 and 0.827, respectively. The table reports the proportion of natural problems whose group-size CV reaches or exceeds each level.
Total variation distance (TV) measures the weight mass reallocated relative to Uniform:
TV=21∑j=1Nqj−N1,
where N is the number of spans and qj is the DOpt-OPD allocation weight. All quantities are computed per problem before aggregation.
At the 0.75 threshold, 28.6%–56.2% of problems reach the Moderate group-size imbalance level, and 11.0%–34.2% reach the Severe level. These proportions compare group-size imbalance under the stated clustering criterion; the natural direction groups need not be strictly orthogonal. Across the four checkpoints, DOpt-OPD reallocates 14.08%–16.13% of the total weight on average, showing that it actively changes the allocation on natural rollouts.
Representativeness of the nine cases. Appendix F.3 keeps the original prefix and the other three rollouts fixed while extending the repeated suffix of one rollout to approximately 25%, 50%, or 75% of that response. The nine cases are the common set for which all three repetition levels could be constructed. They test whether DOpt-OPD preserves supervision on the prefix as repetition increases.
These cases are controlled tests of a specific mechanism, rather than a random sample of natural rollouts. The 500-problem analysis above separately measures the prevalence of directional group-size imbalance in naturally generated responses.