acceptodds
Under review as a conference paper at ICLR 2027

Visitation Is Not Supervision: D-Optimality-Guided Allocation for On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student with dense teacher feedback along its own generated trajectories. Yet the way these token-level signals are aggregated implicitly decides how the supervision budget is distributed. Standard token averaging assigns more total weight to problems that generate longer responses, and repeated visits to similar reasoning states can continue to accumulate weight even when they induce redundant updates. In this paper, we propose a D-Optimality-guided framework that hierarchically allocates supervision across problems and reasoning states, termed DOpt-OPD. At the problem level, we normalize the supervision so that each problem receives equal total loss weight, removing the direct dependence of problem-level supervision on generated length. Within each problem, we pool reasoning spans across rollouts and redistribute the fixed supervision budget using ridge-leverage scores derived from a regularized D-optimal objective. This one-step allocation reduces redundant weighting of similar update directions under a fixed loss-weight budget. Our theoretical analysis shows that the one-step update is near-optimal for small ridge values under orthogonal direction groups, and further provides a computable upper bound for its optimality gap under general, non-orthogonal directions. Across two teacher–student scales and seven mathematical reasoning benchmarks, DOpt-OPD achieves the highest macro-averaged Avg@8 and Pass@8 among the evaluated baselines. These results highlight supervision allocation as a key design choice in on-policy distillation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-policy distillation, Supervision allocation, D-optimal design, LLM post-training, Mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_omupOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how teacher supervision is aggregated in on-policy distillation. DOpt-OPD equalizes total loss weight across problems and pools reasoning spans from multiple rollouts of each problem. It constructs low-dimensional LM-head gradient proxies from hidden states and sparse student–teacher corrections, then allocates supervision using a single ridge-leverage update from a uniform active-span design. The theoretical analysis establishes near-optimality under orthogonal direction groups for sufficiently small ridge values and provides a computable optimality-gap bound for general geometries. Experiments evaluate two main teacher–student scales on seven mathematical reasoning benchmarks, with component ablations, length-stratified analyses, repetition diagnostics, and a supplementary comparison using a 0.6B student.

Critical strengths

1.The paper is well organized, with a clear motivation and a fairly complete experimental analysis. The authors first identify two issues in OPD supervision allocation, then introduce the corresponding solutions and provide experiments to examine each part. The length-based analysis, component ablations, and redundancy diagnostics are well connected to the proposed method. This makes the overall contribution easier to understand and evaluate. 2.The problem is meaningful, and the reported results provide reasonably convincing evidence for the proposed allocation strategy. At both main teacher–student scales, DOpt-OPD achieves the highest seven-benchmark macro-averaged Avg@8 and Pass@8 among OPD, TIP, EOPD, and FiRe-OPD. Table 3 and its per-task expansion in Table 7 show additional gains over both problem normalization and uniform span allocation, supporting the contribution of the leverage-based weighting.

Critical weaknesses

The practical computational overhead and scalability are not sufficiently characterized. Appendix C.4 provides a complexity analysis and explains that existing hidden states and teacher signals are reused, so no additional teacher forward pass is required. However, the analysis excludes token-sketch construction, and the paper reports no direct measurements of wall-clock training time, peak GPU memory, or throughput. Training responses are capped at 7,168 tokens; although evaluation permits responses up to 31,744 tokens, this does not establish the computational cost of the allocator during longer-trajectory training. End-to-end overhead measurements, including gradient-proxy construction, would help assess whether the reported performance gains justify the additional training cost.

Questions for authors

1.Appendix F.1 already documents substantial textual repetition in actual DAPO-Math-17K rollouts. Beyond this text-based audit, how common is severe directional redundancy comparable to the imbalance modeled in Appendix E, and how representative are the nine repeated-suffix cases in Appendix F.3 of naturally occurring repetition? 2.What is the actual additional training cost of DOpt compared with standard OPD, including the construction of gradient proxies? 3.The method uses low-dimensional LM-head gradient proxies to estimate redundancy between reasoning spans. While the empirical results are encouraging, it would be useful to understand how closely these proxies reflect full-model gradient similarities. Have the authors considered a small-scale comparison between the proxy-based and full-gradient-based allocation weights?

3 comments
AuthorsOpenReview ↗

Part 1 (Q1): Directional redundancy in natural rollouts and the representativeness of the repeated-suffix cases

Thank you for asking about directional redundancy in natural rollouts and the representativeness of the nine repeated-suffix cases. We evaluated the same 500 DAPO problems at checkpoints 0, 50, 100, and 139, generating four natural rollouts per problem. Problems were selected independently of answer correctness. We grouped similar proxy directions using complete-linkage clustering with an absolute cosine-similarity threshold of 0.75.

Comparison with Appendix E. We measured group-size imbalance using the coefficient of variation (CV), defined as the standard deviation of group sizes divided by their mean. The Moderate and Severe settings in Appendix E have CVs of approximately 0.569 and 0.827, respectively. The table reports the proportion of natural problems whose group-size CV reaches or exceeds each level.

CheckpointModerate or aboveSevere or aboveMean TV(DOpt-OPD, Uniform)
step028.6%11.0%0.1423
step5046.8%34.2%0.1613
step10056.2%20.4%0.1408
step13953.8%23.4%0.1411

Total variation distance (TV) measures the weight mass reallocated relative to Uniform:

TV=12∑j=1N∣qj−1N∣,\mathrm{TV}=\frac{1}{2}\sum_{j=1}^{N}\left|q_j-\frac{1}{N}\right|,

where NN is the number of spans and qjq_j is the DOpt-OPD allocation weight. All quantities are computed per problem before aggregation.

At the 0.75 threshold, 28.6%–56.2% of problems reach the Moderate group-size imbalance level, and 11.0%–34.2% reach the Severe level. These proportions compare group-size imbalance under the stated clustering criterion; the natural direction groups need not be strictly orthogonal. Across the four checkpoints, DOpt-OPD reallocates 14.08%–16.13% of the total weight on average, showing that it actively changes the allocation on natural rollouts.

Representativeness of the nine cases. Appendix F.3 keeps the original prefix and the other three rollouts fixed while extending the repeated suffix of one rollout to approximately 25%, 50%, or 75% of that response. The nine cases are the common set for which all three repetition levels could be constructed. They test whether DOpt-OPD preserves supervision on the prefix as repetition increases.

These cases are controlled tests of a specific mechanism, rather than a random sample of natural rollouts. The 500-problem analysis above separately measures the prevalence of directional group-size imbalance in naturally generated responses.

AuthorsOpenReview ↗

Part 2 (W/Q2): Actual training time and memory overhead

(1) Proxy construction and weight allocation

We measured proxy construction and weight allocation over the full DAPO-Math-17K dataset, using four rollouts per problem, an 8×8 proxy, and 32-token spans. The mean time is 1.445 ms per problem, corresponding to a linear estimate of 0.185 s for a step containing 128 problems.

(2) End-to-end training overhead

We compared OPD and DOpt-OPD at training response caps of 7,168, 16,384, and 31,744 tokens, using 128 problems per step, four rollouts per problem, and FP32. The end-to-end timings include proxy construction and weight allocation. Thus, the longer-response settings measure costs during training, rather than only allowing longer responses at evaluation.

Training response capPrecisionMethodPeak memory (MiB/GPU)Mean response tokensMean step time (s)Throughput (tokens/s/GPU)
7,168FP32OPD70,2112,436.72222.701,481.02
7,168FP32DOpt-OPD72,3712,782.73252.991,465.94
16,384FP32OPD72,3793,271.53445.70974.84
16,384FP32DOpt-OPD72,4113,837.50520.58967.08
31,744FP32OPD80,9993,658.42658.74740.10
31,744FP32DOpt-OPD81,1034,210.84751.62731.16

Across these settings, DOpt-OPD increases step time by 13.6%–16.8%, alongside a 14.2%–17.3% increase in mean response length. Throughput decreases by approximately 0.8%–1.2%, while peak memory increases by at most 3.1%.

The similar increases in response length and step time, together with nearly unchanged throughput, suggest that the longer steps are primarily due to longer responses requiring more generation and training computation. Proxy construction and weight allocation are estimated to take only 0.185 s per step, which is negligible relative to the full training-step time.

AuthorsOpenReview ↗

Part 3 (Q3): Comparison between the proxy and full-model gradients

(1) Agreement between allocation-weight rankings

At each of checkpoints 50 and 139, we selected 20 problems. For each problem, we used 64 candidate spans of 32 tokens from four natural rollouts and eight distinct held-out spans to evaluate update effectiveness. We compared three representations using the same candidate spans and teacher signals: the 8×8 (64-D) proxy, exact output-layer gradients using the same correction signal without random compression, and full-model gradients computed by backpropagation. We normalized the directions and computed allocation weights using the same ridge parameter (0.001) and D-optimal allocation rule.

CheckpointSpearman with full-model weights [95% interval]Spearman with exact output-layer weights [95% interval]
step500.735 [0.622, 0.830]0.787 [0.691, 0.882]
step1390.387 [0.326, 0.439]0.606 [0.579, 0.648]

We use Spearman’s rank correlation coefficient to compare weight rankings. At both checkpoints, proxy-based weights are positively correlated with full-model weights and more closely aligned with exact output-layer weights. This indicates agreement in identifying which spans should receive greater weight.

(2) Actual one-step update effectiveness

We fixed the rollouts and teacher signals and started each comparison from identical model parameters. Ordinary SGD used the same learning rate for Proxy and Uniform. For equal-L2 updates, we adjusted the Proxy learning rate to match Uniform’s parameter-displacement L2 norm. We then measured objective reduction on the eight held-out spans, which were excluded from weight computation and the update. Each checkpoint includes 20 problems; values below are mean improvements over Uniform in units of 10−610^{-6}.

CheckpointOrdinary SGD: Proxy − Uniform [95% interval]Equal-L2 update: Proxy − Uniform [95% interval]
step50+3.855 [+1.968, +7.503]+1.768 [+0.272, +4.701]
step139+1.245 [+0.965, +1.751]+0.571 [+0.206, +1.251]

At both checkpoints, the mean improvements are positive under ordinary SGD and equal-L2 updates, with all 95% intervals above zero. The benefit remains after matching update magnitude, indicating that the proxy’s advantage on these problems is not solely due to taking a larger step.

Proxy dimension and training effectiveness. Increasing proxy dimension generally reduces random compression error, but the effect of DOpt-OPD depends on how these directions jointly determine the allocation weights, rather than on dimensionality alone. We evaluated models trained separately with 8×8 and 16×16 proxies under identical output caps on seven benchmarks. At caps of 7,168 and 16,384 tokens, 8×8 achieves 2.79 and 2.60 percentage points higher Pass@8 despite slightly longer responses and higher truncation rates. At 31,744 tokens, its Avg@8/Pass@8 advantage is 1.15/2.86 points. Increasing proxy dimension therefore does not guarantee better training outcomes; we selected 8×8 based on its allocation effectiveness and final evaluation performance.

Reviewer_MbCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies supervision allocation in on-policy distillation, arguing that token averaging favors problems with longer responses and can overcount redundant update directions within a problem. DOpt-OPD assigns equal total loss weight to each problem and redistributes it across reasoning spans using ridge-leverage scores computed from sketched LM-head gradient proxies. The authors provide experimental validations for the one-step allocation and report improved macro-averaged performance across several mathematical benchmarks.

Critical strengths
  • The paper explains why both problem length and directional redundancy matter, supported by rollout analysis and geometric observations. The proposed method is practical, as it makes use of existing forward-pass information, making it efficient.
  • The ablation studies show that both allocation strategies are useful, with additional results on smaller students and TIDE supporting the findings.
Critical weaknesses
  • The paper only reports best-of-N averaged scores, without results from multiple training runs or confidence intervals. The improvements over the main baselines are relatively small (around 0.54-0.55% on Avg@8), making it difficult to judge their reliability and efficacy
  • The theoretical analysis is based on a simplified objective, but the paper does not show how this theorem is connected to practical gradients or optimization behavior during training. The theorem needs to be properly reflected.
  • Evaluation is limited to Qwen3 models and mathematical reasoning, and runtime and memory are not reported.
Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Part 1 (W1): Evaluation stability and confidence intervals

Thank you for raising this concern. We evaluated seven benchmarks using three seeds, first computing the seven-benchmark macro-average for each seed and then reporting the mean and sample standard deviation (SD). Differences between DOpt and each baseline, together with 95% intervals, were computed using 5,000 paired bootstrap resamples at the seed level. These intervals describe evaluation variability rather than variance across repeated training runs.

MethodAvg@8 (mean ± SD)Pass@8 (mean ± SD)DOpt − baseline Δ Avg@8 [95% interval]DOpt − baseline Δ Pass@8 [95% interval]
OPD18.19 ± 0.1033.06 ± 0.38+2.77 [+2.67, +2.95]+5.66 [+5.38, +6.10]
PN20.31 ± 0.2635.93 ± 0.40+0.66 [+0.18, +0.91]+2.79 [+2.50, +2.94]
Uniform20.00 ± 0.2537.06 ± 0.50+0.97 [+0.53, +1.25]+1.66 [+1.38, +1.94]
EOPD20.53 ± 0.1036.75 ± 1.11+0.44 [+0.14, +0.61]+1.96 [+0.89, +3.03]
FiRe-OPD18.89 ± 0.1435.98 ± 0.77+2.07 [+1.97, +2.22]+2.73 [+2.14, +3.65]
DOpt-OPD20.97 ± 0.2038.71 ± 0.23——

DOpt ranks first in seven-benchmark macro-averaged Avg@8 and Pass@8 under each of the three seeds. Against PN and EOPD, the closest baselines in Avg@8, DOpt improves the mean by 0.66 and 0.44 percentage points, with corresponding intervals of [0.18, 0.91] and [0.14, 0.61]. The differences are positive under each seed.

Results under other training settings.

We also evaluated DOpt across the following model scales, method combinations, and domains (teacher → student):

  • Qwen models at three student scales: Qwen3-4B → Qwen3-0.6B, Qwen3-4B → Qwen3-1.7B, and Qwen3-8B → Qwen3-4B.
  • TIDE+DOpt: applying DOpt allocation within TIDE for Qwen3-4B → Qwen3-1.7B and Qwen3-8B → Qwen3-4B.
  • Llama mathematical reasoning: Llama-3.1-8B-Instruct → Llama-3.2-3B-Instruct, with macro-averaged Avg@8/Pass@8 gains of 1.25/4.72 points over OPD across AIME24, AIME25, and AMC23 (Section 4.4, Table 6).
  • Code generation: Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B, with macro-averaged gains of 3.23/4.06 points over OPD across HumanEval, MBPP, and LiveCodeBench (Section 4.4, Table 7).
AuthorsOpenReview ↗

Part 2 (W2): Comparison with full-model gradients and the one-step optimality gap

(1) Comparison between the proxy and full-model gradients

At each of checkpoints 50 and 139, we selected 20 problems. For each problem, we used 64 candidate spans of 32 tokens from four natural rollouts and eight distinct held-out spans. We compared three representations using the same spans and teacher signals: the 8×8 (64-D) proxy, exact output-layer gradients using the same correction signal without random compression, and full-model gradients computed by backpropagation. We normalized the directions and computed weights using the same ridge parameter (0.001) and D-optimal allocation rule.

CheckpointSpearman with full-model weights [95% interval]Spearman with exact output-layer weights [95% interval]
step500.735 [0.622, 0.830]0.787 [0.691, 0.882]
step1390.387 [0.326, 0.439]0.606 [0.579, 0.648]

We use Spearman’s rank correlation coefficient to compare weight rankings. At both checkpoints, proxy-based weights are positively correlated with full-model weights and more closely aligned with exact output-layer weights, indicating agreement in which spans receive higher priority.

To evaluate actual updates, we fixed the rollouts and teacher signals and started from identical model parameters. Ordinary SGD used the same learning rate for Proxy and Uniform. For equal-L2 updates, we adjusted the Proxy learning rate to match Uniform’s parameter-displacement L2 norm. We measured objective reduction on the eight held-out spans, excluded from weight computation and the update. Each checkpoint includes 20 problems; values below are mean improvements over Uniform in units of 10−610^{-6}.

CheckpointOrdinary SGD: Proxy − Uniform [95% interval]Equal-L2 update: Proxy − Uniform [95% interval]
step50+3.855 [+1.968, +7.503]+1.768 [+0.272, +4.701]
step139+1.245 [+0.965, +1.751]+0.571 [+0.206, +1.251]

The mean improvements are positive at both checkpoints under both update settings, with all 95% intervals above zero. The benefit remains after matching update magnitude, indicating that it is not solely due to taking a larger step.

Proxy dimension and training effectiveness. Increasing proxy dimension generally reduces random compression error, but the effect of DOpt-OPD depends on how these directions jointly determine the allocation weights, rather than on dimensionality alone. We evaluated separately trained 8×8 and 16×16 models under identical output caps on seven benchmarks. At caps of 7,168 and 16,384 tokens, 8×8 achieves 2.79 and 2.60 points higher Pass@8 despite slightly longer responses and higher truncation rates. At 31,744 tokens, its Avg@8/Pass@8 advantage is 1.15/2.86 points. Higher proxy dimensionality therefore does not guarantee better training outcomes.

(2) One-step DOpt-OPD versus a numerical reference solution

We used the same 500 DAPO problems at checkpoints 0, 50, 100, and 139, generating four natural rollouts per problem. On each problem’s candidate spans, we compared Uniform, one D-optimal weight update initialized from Uniform, and a numerical reference solution obtained with up to 5,000 iterations. The teacher, 8×8 proxy, and ridge parameter of 0.001 matched the training setup.

We measured the gap in the normalized proxy-design objective and used the reference solution’s Frank–Wolfe residual to obtain lower and upper bounds. All quantities were computed per problem and then averaged.

CheckpointPer-dimension gap boundsUniform D-efficiencyOne-step DOpt-OPD D-efficiency lower boundInformation-matrix cosine similarity
step0[0.082690, 0.082692]83.15%92.18%0.9740
step50[0.052224, 0.052228]83.99%94.98%0.9765
step100[0.098198, 0.098202]81.92%90.71%0.9535
step139[0.093998, 0.094002]82.24%91.09%0.9535

Let Φ(q)=log⁡det⁡M(q)\Phi(q)=\log\det M(q), with q1q_1 and qrq_r denoting the one-step and reference allocations, and d=64d=64.

  • Per-dimension gap: ΔΦ/d=[Φ∗−Φ(q1)]/d\Delta\Phi/d = [\Phi^* - \Phi(q_1)]/d, where Φ∗\Phi^* is the optimal objective value. The reported bounds are numerical bounds, not statistical confidence intervals.
  • D-efficiency: exp⁡(−ΔΦ/d)\exp(-\Delta\Phi/d). The one-step lower bound uses the gap upper bound; Uniform is evaluated relative to the numerical reference.
  • Information-matrix Frobenius cosine similarity: ⟨M(q1),M(qr)⟩F∥M(q1)∥F ∥M(qr)∥F\frac{\langle M(q_1),M(q_r)\rangle_F}{\lVert M(q_1)\rVert_F\,\lVert M(q_r)\rVert_F}.

Across the four checkpoints, the mean per-dimension gap is approximately 0.052–0.098, and the mean D-efficiency lower bound is 90.71%–94.98%. Relative to the same numerical reference, one-step allocation improves D-efficiency over Uniform by 8.79–10.99 percentage points. Thus, a single weight update improves the design objective over Uniform and achieves high D-efficiency on these training checkpoints.

AuthorsOpenReview ↗

Part 3 (W3): Additional model and domain evaluations, and computational overhead

(1) Mathematical reasoning with Llama

We used a Llama-3.1-8B-Instruct teacher and a Llama-3.2-3B-Instruct student, comparing Student, OPD, and DOpt-OPD on AIME24, AIME25, and AMC23. Each entry reports Avg@8 / Pass@8 (%).

MethodAIME24AIME25AMC23Macro-average
Student1.25 / 6.671.25 / 6.6710.94 / 42.504.48 / 18.61
OPD2.08 / 10.000.83 / 6.6714.06 / 45.005.66 / 20.56
DOpt-OPD2.92 / 13.332.50 / 10.0015.31 / 52.506.91 / 25.28

DOpt-OPD outperforms OPD on both metrics across all three benchmarks, improving macro-averaged Avg@8/Pass@8 by 1.25/4.72 percentage points. These results extend the evaluation beyond the Qwen model family and have been added to Section 4.4, Table 6.

(2) Code generation

We used a Qwen3-30B-A3B-Instruct-2507 teacher and a Qwen3-4B student, trained on the code subset of Eurus-2-RL-Data. We compared Student, OPD, and DOpt-OPD on HumanEval, MBPP, and LiveCodeBench, sampling eight responses per problem. Each entry reports Avg@8 / Pass@8 (%).

MethodHumanEvalMBPPLiveCodeBenchMacro-average
Student79.73 / 89.6374.50 / 84.6626.43 / 34.6060.22 / 69.63
OPD80.87 / 90.8578.94 / 89.6830.88 / 40.3863.56 / 73.64
DOpt-OPD82.09 / 92.0780.26 / 91.2738.03 / 49.7666.79 / 77.70

DOpt-OPD outperforms OPD on both metrics across all three code benchmarks, with macro-averaged gains of 3.23/4.06 points. The largest gains occur on LiveCodeBench, at 7.15/9.38 points. These results have been added to Section 4.4, Table 7.

(3) Training time and memory overhead

Proxy construction and weight allocation. We measured both operations over the full DAPO-Math-17K dataset, using four rollouts per problem, an 8×8 proxy, and 32-token spans. The mean time is 1.445 ms per problem, corresponding to a linear estimate of 0.185 s for a step containing 128 problems.

End-to-end training-step time and memory. We compared OPD and DOpt-OPD at training response caps of 7,168, 16,384, and 31,744 tokens, using 128 problems per step, four rollouts per problem, and FP32. The end-to-end timings include proxy construction and weight allocation.

Training response capPrecisionMethodPeak memory (MiB/GPU)Mean response tokensMean step time (s)Throughput (tokens/s/GPU)
7,168FP32OPD70,2112,436.72222.701,481.02
7,168FP32DOpt-OPD72,3712,782.73252.991,465.94
16,384FP32OPD72,3793,271.53445.70974.84
16,384FP32DOpt-OPD72,4113,837.50520.58967.08
31,744FP32OPD80,9993,658.42658.74740.10
31,744FP32DOpt-OPD81,1034,210.84751.62731.16

Across these settings, DOpt-OPD increases step time by 13.6%–16.8%, alongside a 14.2%–17.3% increase in mean response length. Throughput decreases by approximately 0.8%–1.2%, while peak memory increases by at most 3.1%.

The similar increases in response length and step time, together with nearly unchanged throughput, suggest that the longer steps are primarily due to longer responses requiring more generation and training computation. Proxy construction and weight allocation are estimated to take only 0.185 s per step, which is negligible relative to the full training-step time.

Reviewer_4wZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that standard token averaging in on-policy distillation implicitly allocates the supervision budget by generated length. An audit on DAPO-Math-17K rollouts shows the longest problem-length quartile absorbing 60.7% of the budget against 3.5% for the shortest. It also argues that repeated visits to similar reasoning states keep accumulating weight even when their update directions are already covered. The proposed fix, DOpt-OPD, works at two levels: problem-level normalisation giving each problem equal total loss weight 1/N, and within each problem, pooling rollouts into fixed-length spans, building sketched LM-head gradient proxies, and redistributing the fixed budget by ridge-leverage scores from one multiplicative step of a regularised D-optimal objective. The theory covers near-optimality under orthogonal direction groups and gives a computable KKT-residual certificate otherwise. Evaluation is on two teacher-student scales and seven mathematical reasoning benchmarks.

Critical strengths
  1. The supervision-budget framing is an under-examined angle on OPD, and the audit in Figure 1 is a crisp diagnostic that stands on its own whatever one thinks of the proposed remedy. Pointing out that a standard loss-aggregation convention silently ties supervision to response length is useful to anyone setting up these runs.

  2. Problem-level normalisation is essentially a one-line change and recovers most of the reported benefit, so the practical takeaway is easy for others to adopt and check.

Critical weaknesses
  1. No seeds and no error bars anywhere in Tables 1 to 4. AIME24 and AIME25 have 30 problems each and the metric is Avg@8, so a difference like 20.00 vs 19.17 on AIME24 is well under one problem. The whole contribution is a reweighting scheme with modest aggregate deltas (+2.17 and +2.70 Avg@8 over OPD), and single-run numbers are not enough to establish an ordering against four baselines.

  2. The simple half does most of the work. In Table 3, problem normalisation alone takes OPD from 18.28/32.74 to 20.07/36.33. The entire D-optimal apparatus, two theorems plus Rademacher/CountSketch proxies plus Cholesky solves plus three new hyperparameters, adds +0.91 Avg@8 on top of that, and +1.25 over uniform span allocation. Without variance estimates that increment is not clearly separable from run-to-run noise, and it is a lot of machinery for the margin.

  3. The evidence points away from the stated mechanism. If the sketched proxy really captured update geometry, a higher-fidelity proxy should help. Table 4 shows the opposite: 16×16 (256-D) underperforms 8×8 (64-D), 21.18 vs 22.71. Meanwhile the winning configuration also has the shortest outputs (8,246 tokens) and the lowest truncation rate (15.50%), and the largest Pass@8 gains sit in the longest-response group (+13.25). That looks more like a length or truncation control effect than directional-redundancy reduction, and the paper does not rule it out.

Questions for authors
  • Could you report three seeds on at least the Table 3 ablation? That is where the attribution between the two components rests.
  • What happens if the leverage weights are swapped for random span weights with matched entropy? Does the gain over uniform span allocation survive?
  • Theorem 1 notes that equal-size direction groups make uniform initialisation already optimal. How unequal are group sizes on real rollouts, i.e. how far from the no-op regime is the method actually operating?
4 comments
AuthorsOpenReview ↗

Part 1 (W1/W2/Q1): Table 3 ablations and evaluation stability

Thank you for highlighting the need to test whether the gains from the two components persist across seeds. We first evaluated all four Table 3 configurations across three seeds on seven benchmarks, then extended the comparison to EOPD and FiRe-OPD.

Table 3 ablations across three seeds. The table reports the mean and sample standard deviation (SD) of the seven-benchmark macro-average.

MethodAvg@8 (mean ± SD)Pass@8 (mean ± SD)
OPD18.19 ± 0.1033.06 ± 0.38
Problem normalization (PN)20.31 ± 0.2635.93 ± 0.40
Uniform span allocation20.00 ± 0.2537.06 ± 0.50
DOpt-OPD20.97 ± 0.2038.71 ± 0.23

PN improves Avg@8/Pass@8 over OPD by 2.12/2.87 percentage points. DOpt adds a further 0.66/2.79 points over PN and 0.97/1.66 points over Uniform. DOpt exceeds both PN and Uniform on both macro-averaged metrics under each of the three seeds. Thus, the incremental benefit of directional weighting persists across these evaluations.

We also computed paired seed-bootstrap intervals using 5,000 resamples, retaining equal weight for all seven benchmarks. These intervals summarize variation across the three evaluations.

ComparisonΔ Avg@8 [95% interval]Δ Pass@8 [95% interval]
DOpt − OPD+2.77 [+2.67, +2.95]+5.66 [+5.38, +6.10]
DOpt − PN+0.66 [+0.18, +0.91]+2.79 [+2.50, +2.94]
DOpt − Uniform+0.97 [+0.53, +1.25]+1.66 [+1.38, +1.94]

Additional comparisons with EOPD and FiRe-OPD. We evaluated both baselines using the same three seeds.

MethodAvg@8 (mean ± SD)Pass@8 (mean ± SD)
EOPD20.53 ± 0.1036.75 ± 1.11
FiRe-OPD18.89 ± 0.1435.98 ± 0.77
DOpt-OPD20.97 ± 0.2038.71 ± 0.23

The corresponding differences are reported below in percentage points, with intervals computed as above.

ComparisonΔ Avg@8 [95% interval]Δ Pass@8 [95% interval]
DOpt − EOPD+0.44 [+0.14, +0.61]+1.96 [+0.89, +3.03]
DOpt − FiRe-OPD+2.07 [+1.97, +2.22]+2.73 [+2.14, +3.65]

DOpt also exceeds both baselines in macro-averaged Avg@8 and Pass@8 under each seed.

Results across model scales, method combinations, and domains. We additionally evaluated the method in the following settings (teacher → student):

  • Qwen models at three student scales: Qwen3-4B → Qwen3-0.6B, Qwen3-4B → Qwen3-1.7B, and Qwen3-8B → Qwen3-4B.
  • TIDE+DOpt: applying DOpt allocation within TIDE for Qwen3-4B → Qwen3-1.7B and Qwen3-8B → Qwen3-4B.
  • Llama mathematical reasoning: Llama-3.1-8B-Instruct → Llama-3.2-3B-Instruct, with macro-averaged Avg@8/Pass@8 gains of 1.25/4.72 points over OPD across AIME24, AIME25, and AMC23 (Section 4.4, Table 6).
  • Code generation: Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B, with macro-averaged gains of 3.23/4.06 points over OPD across HumanEval, MBPP, and LiveCodeBench (Section 4.4, Table 7).
AuthorsOpenReview ↗

Part 2 (W2/Q2): Random span weights with matched entropy and uniform span allocation

Following your suggestion, we constructed a random-span-weight control with matched entropy by randomly permuting DOpt’s leverage weights among spans within each problem. This preserves the weight distribution and entropy exactly, while keeping training steps, span length, and supervision budgets unchanged. The table compares this control with the Uniform and DOpt results reported in the paper. Each entry is Avg@8 / Pass@8 (%), with equal weight given to the seven benchmarks in the macro-average.

MethodAIME24AIME25AMC23MinervaOlympiad-BenchCMIMC25BRUMO25Macro-average
Uniform span allocation10.42 / 23.339.58 / 23.3343.75 / 70.0020.40 / 38.6034.37 / 56.683.75 / 15.0015.83 / 33.3319.73 / 37.18
Random span weights with matched entropy10.83 / 23.338.75 / 20.0041.25 / 70.0020.40 / 37.8733.23 / 54.305.31 / 15.0017.08 / 30.0019.55 / 35.79
DOpt-OPD12.50 / 30.0010.00 / 23.3345.63 / 77.5021.05 / 38.6035.00 / 54.905.62 / 17.5017.08 / 30.0020.98 / 38.83

Random span weights with matched entropy fall 0.18/1.39 percentage points below Uniform in macro-averaged Avg@8/Pass@8. DOpt, in contrast, exceeds Uniform by 1.25/1.65 points and the randomized control by 1.43/3.04 points. The advantage over uniform allocation therefore does not survive randomization: the gains depend not only on the weight distribution and entropy, but also on which spans receive those weights.

We have added this experiment to Section 4.3, Table 4 of the revised manuscript.

AuthorsOpenReview ↗

Part 3 (W3): Proxy dimension, response length, and truncation controls

(1) Proxy fidelity and allocation effectiveness

Thank you for raising this point. The original manuscript did not sufficiently distinguish the accuracy of the proxy’s directional approximation from its downstream training benefit. Increasing the proxy dimension generally reduces random compression error, but DOpt’s effect depends on how these directions jointly determine the allocation weights, rather than on dimensionality alone.

DOpt computes leverage weights using the regularized information matrix of the entire candidate-span pool for each problem. Increasing the proxy dimension therefore affects both the gradient approximation and the resulting allocation. In the training records, the 8×8 and 16×16 configurations contain an average of 335.4 and 337.5 candidate spans per problem. Under the same ridge setting, their ridge effective dimensions—the sums of ridge leverage scores—are 63.1 and 198.8, respectively. Thus, similarly sized span pools produce substantially different effective dimensions.

We further compared the 64-dimensional proxy with exact output-layer gradients using the same correction signal. At each of checkpoints 50 and 139, we used 20 problems, with the same 64 candidate spans and eight distinct held-out spans per problem. Each table entry reports “64-D proxy / exact output-layer gradient.” Downweighting is measured relative to Uniform, and held-out objective reduction is measured after one SGD step.

CheckpointMean downweighting of repeated spansMean held-out objective reduction (×10⁻⁶)
step5015.72% / 4.39%35.08 / 33.33
step1392.46% / 0.33%17.06 / 16.33

In this experiment, the 64-D proxy assigns less weight to repeated spans and produces a larger mean reduction in the held-out objective. These results show that using exact gradients does not necessarily yield a more effective update. Together with the final evaluation results, they support our choice of 8×8 as the default configuration.

(2) Comparison under identical output caps

To examine whether the advantage of 8×8 comes from shorter responses or lower truncation rates, we evaluated models trained separately with 8×8 and 16×16 proxies under identical output caps. All metrics below are equally weighted averages over seven benchmarks. The 7,168- and 16,384-token evaluations use the same seed, 20261107; the 31,744-token comparison uses the paper’s 8×8 results and the supplementary 16×16 evaluation.

Output capTraining proxyAvg@8Pass@8Mean response tokensTruncation rate
7,1688×8 (64-D)19.7837.043,87623.41%
7,16816×16 (256-D)19.1034.253,82622.22%
16,3848×8 (64-D)19.7839.145,66717.71%
16,38416×16 (256-D)19.3836.545,51217.38%
31,7448×8 (64-D)20.9838.837,82814.83%
31,74416×16 (256-D)19.8335.978,03915.63%

At caps of 7,168 and 16,384 tokens, 8×8 achieves 2.79 and 2.60 percentage points higher Pass@8 despite slightly longer responses and higher truncation rates. At 31,744 tokens, its Avg@8/Pass@8 advantage is 1.15/2.86 points.

(3) Contributions to the gains on the longest-response group

The paper groups problems within each benchmark by OPD’s mean response length, using the same groups for all methods. On the longest 25% of problems, PN, Uniform, and DOpt improve Pass@8 over OPD by 12.23, 11.73, and 13.25 points, respectively. DOpt’s additional gains over PN and Uniform are therefore 1.02 and 1.52 points. On the middle 50%, its corresponding gains are 3.90 and 2.02 points.

Most of the improvement in the longest-response group comes from problem normalization, with directional weighting providing a further gain. This additional benefit also appears on medium-length problems. Together with the identical-output-cap comparisons, these results show that the advantage of 8×8 cannot be explained solely by shorter responses or a lower overall truncation rate.

AuthorsOpenReview ↗

Part 4 (Q3): Direction-group sizes and changes in allocation weights

We evaluated the same 500 DAPO problems at checkpoints 0, 50, 100, and 139, generating four natural rollouts per problem at each checkpoint. Problems were selected independently of answer correctness. We measured direction-group sizes and the change in allocation weights after one DOpt update from Uniform.

(1) How unequal are the direction-group sizes?

We grouped proxy directions using complete-linkage clustering with an absolute cosine-similarity threshold of 0.75. We measured group-size imbalance using the coefficient of variation (CV), defined as the standard deviation of group sizes divided by their mean.

(2) How much does DOpt change the allocation?

We measured the weight mass reallocated relative to Uniform using total variation distance (TV):

TV=12∑j=1N∣qj−1N∣,\mathrm{TV}=\frac{1}{2}\sum_{j=1}^{N}\left|q_j-\frac{1}{N}\right|,

where NN is the number of spans and qjq_j is the DOpt weight. TV = 0 means that the allocation is unchanged. We also report the proportion of problems with TV < 0.05, where less than 5% of the weight mass is reallocated. CV and TV are computed per problem and then averaged.

CheckpointMean group-size CVMean TV(DOpt, Uniform)Problems with TV < 0.05
step00.5640.14231.4%
step500.9150.16134.8%
step1000.6910.14080.2%
step1390.7140.14110.2%

The mean group-size CV ranges from 0.564 to 0.915, showing that direction groups in natural rollouts are not equally sized. DOpt reallocates 14.08%–16.13% of the total weight on average. At step139, the mean reallocated mass is 14.11%, with a 95% interval of [13.75%, 14.50%]; only one of the 500 problems has TV below 0.05. These results show that DOpt actively changes the allocation on most problems rather than remaining near the no-op regime.

Effect of the grouping threshold. At the stricter similarity thresholds of 0.90, 0.95, and 0.99, the mean group-size CV at step139 is 0.460, 0.329, and 0.142, respectively. The threshold affects the estimated degree of group-size imbalance, but unequal group sizes remain present across all three thresholds. The TV values above are computed directly from DOpt’s allocation weights and do not depend on the clustering threshold.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Updates for the AC and All Reviewers

We sincerely thank the reviewers for their careful reading and constructive feedback. We have taken the concerns raised seriously and added experiments and analyses to address them, with corresponding updates to the manuscript. For ease of reference, we summarize the main additions below and indicate the relevant manuscript locations and detailed responses.

1. Evaluation stability. We evaluated the Table 3 configurations—OPD, PN, Uniform, and DOpt—along with EOPD and FiRe-OPD across three evaluation seeds, reporting means, standard deviations, and paired seed-bootstrap intervals. DOpt ranks first in seven-benchmark macro-averaged Avg@8 and Pass@8 under each seed. See our response to Reviewer 4wZU, Part 1, and Reviewer MbCK, Part 1.

2. Matched-entropy random-weight control (Section 4.3, Table 4). We added a control that randomly permutes DOpt weights among spans within each problem, preserving the weight distribution, entropy, and supervision budget. Its macro-averaged Avg@8/Pass@8 falls below DOpt by 1.43/3.04 points, supporting the importance of assigning weights to appropriate spans. See our response to Reviewer 4wZU, Part 2.

3. Proxy dimension and truncation controls. We added evaluations of the 8×8 and 16×16 proxies under identical output caps. At caps of 7,168 and 16,384 tokens, the 8×8 proxy achieves 2.79 and 2.60 points higher Pass@8, respectively, despite slightly longer responses and higher truncation rates. See our response to Reviewer 4wZU, Part 3(2).

4. Redundancy in natural rollouts. We analyzed direction-group imbalance and allocation changes on 500 problems at four checkpoints. Natural rollouts exhibit directional imbalance, with DOpt reallocating 14.08%–16.13% of the total weight on average. We relate these measurements to Appendix E’s imbalance settings and clarify that Appendix F.3’s nine repeated-suffix cases are controlled mechanism tests. See our response to Reviewer 4wZU, Part 4, and Reviewer omup, Part 1.

5. Comparison with full-model gradients. We added comparisons of allocation weights and one-step updates using the proxy, full-model gradients, and exact output-layer gradients at checkpoints 50 and 139. Under both ordinary SGD and displacement-matched updates, the proxy yields greater mean held-out objective improvement than Uniform, with all corresponding 95% intervals above zero. See our response to Reviewer MbCK, Part 2(1), and Reviewer omup, Part 3.

6. Approximation to a numerical reference solution. We compared one-step DOpt with an iterative numerical reference solution on 500 problems across four checkpoints. One-step DOpt achieves mean D-efficiency lower bounds of 90.71%–94.98% and information-matrix cosine similarities of 0.9535–0.9765 with the reference solution. See our response to Reviewer MbCK, Part 2(2).

7. Evaluation beyond Qwen. We added mathematical reasoning experiments with Llama-3.1-8B-Instruct → Llama-3.2-3B-Instruct (Section 4.4, Table 6), GT-Llama-3.2-3B-Instruct-MATH → Llama-3.2-3B-Instruct, and Gemma-3-12B-it → Gemma-3-4B-it. DOpt outperforms OPD on both metrics across the reported benchmarks for each pair. With the stronger starting Gemma student, DOpt achieves four-benchmark macro-averaged Avg@8/Pass@8 of 37.29/55.89 versus OPD’s 35.94/49.18. See our response to Reviewer MbCK, Part 3(1), and our follow-up comment “Response to Reviewer MbCK,” items 1–3.

8. Code generation (Section 4.4, Table 7). We added Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B experiments on HumanEval, MBPP, and LiveCodeBench. DOpt outperforms OPD on all three benchmarks, improving macro-averaged Avg@8/Pass@8 by 3.23/4.06 points. See our response to Reviewer MbCK, Part 3(2).

9. Practical overhead. We added end-to-end measurements under three training response caps, including proxy construction and weight allocation. Throughput decreases by 0.8%–1.2%, while peak memory increases by at most 3.1%. Step time increases by 13.6%–16.8%, alongside 14.2%–17.3% longer responses. Proxy construction and weight allocation average 1.445 ms per problem. See our response to Reviewer omup, Part 2, and Reviewer MbCK, Part 3(3).

We are grateful for the reviewers’ time and thoughtful suggestions. Their feedback has helped us identify where further evidence and clearer explanations were needed, and has strengthened the evaluation of the method’s stability, mechanism, applicability, and practical cost. We hope these additions address the concerns raised and help the reviewers and Area Chair assess the revised work. We would be happy to clarify any remaining questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.