acceptodds
Under review as a conference paper at ICLR 2027

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

10 views

Abstract

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model. Despite its empirical success, the conditions under which OPD yields reliable improvement remain poorly understood. We identify two fundamental bottlenecks that limit effective OPD: insufficient student exploration of informative states and unreliable teacher supervision for student rollouts. Building on this insight, we propose Uni-OPD, a unified OPD framework centered on a dual-perspective optimization strategy that generalizes across LLMs and MLLMs. On the student side, we adopt offline and online data balancing strategies to promote exploration of informative states during training. On the teacher side, we show that reliable supervision hinges on whether aggregated token-level guidance remains order-consistent with the outcome reward. We therefore introduce an outcome-guided margin calibration mechanism to restore this consistency between correct and incorrect trajectories. We conduct extensive experiments on 16 benchmarks across 5 domains, spanning single-teacher, multi-teacher, strong-to-weak, and cross-modal distillation settings. The results verify the effectiveness and versatility of Uni-OPD, with an average gain of 2.1 points over vanilla OPD. Code is available at https://anonymous.4open.science/r/Uni-OPD/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Post-training, Large Language Models, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X8ZvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Uni-OPD is a unified on-policy distillation framework. The authors state that there are limitations in existing OPD (insufficient student exploration and unreliable teacher supervision). They mainly propose (1) offline difficulty-aware data balancing, (2) online correctness-aware data balancing, and (3) outcome-guided margin calibration.

Critical strengths
  • I believe improving the efficiency of student learning and the reliability of teacher supervision is an important direction.

  • The paper provides extensive experimental evaluations across single, multi-teacher, strong-to-weak, and cross-modal distillation setups, and the results are consistent.

Critical weaknesses
  • The claim of improved student exploration is not sufficiently supported. The author states insufficient student exploration as a bottleneck of OPD. However, the proposed method primarily modifies the distribution of training data (via upsampling) and their trajectory (balancing correct/incorrect) rather than actual exploration. In detail, (1) online correctness-aware balancing filters already-generated trajectories and does not inherently increase the diversity or coverage of student rollouts. For (2) offline difficulty-aware data balancing, the author did not specify how they upsample underrepresented difficulty (The author should specify; I saw A.1, but still, I could not find detailed information). The authors should provide direct evidence of improved exploration, such as reasoning-strategy diversity or the discovery rate of distinct successful trajectories.

  • Offline difficulty-aware balancing introduces additional computational overhead and relies on potentially outdated difficulty estimates. In Appendix A.1, the authors estimate prompt difficulty by generating eight responses for every training example over the entire dataset. Moreover, difficulty is estimated using the initial student policy, these estimates may become outdated as the student improves. Considering that OPD already generates multiple trajectories during training, it is unclear why difficulty cannot be estimated and updated using existing online rollouts. Furthermore, the difficulty distribution will change as the student learns (because the student is the labeler). This raises concern about whether balancing a fixed initial difficulty distribution remains beneficial throughout training.

  • Data balancing and margin calibration may discard or distort informative training signals. Both methods rely on final-answer correctness, which does not necessarily reflect the quality of intermediate reasoning. For example, an incorrect trajectory may contain valid reasoning with only a minor mistake near the end. However, (1) online correctness-aware balancing may discard such informative trajectories to maintain a 1:1 correct-to-incorrect ratio within each prompt. For example, when one correct and nine incorrect trajectories are generated, only two may be retained, whereas all ten are retained when every trajectory is incorrect. It is also unclear why balancing must be performed per prompt rather than across the entire batch, which could preserve more trajectories. (2) Outcome-guided margin calibration applies a uniform trajectory-level correction to token-level advantages based on outcome correctness (Section 3.4 and Algorithms 2,3), potentially suppressing valid intermediate reasoning in incorrect trajectories while reinforcing uninformative reasoning in correct ones.

Questions for authors
  1. Does Uni-OPD encourage student exploration? If so, can the authors provide direct evidence of improved student exploration? Although Appendix C.5 reports policy entropy and response length, these metrics do not necessarily reflect reasoning-strategy diversity or exploration coverage. Could the authors evaluate reasoning-strategy diversity of distinct successful trajectories [1]?

  2. Why is an additional offline rollout pass necessary for difficulty estimation? Could the authors instead use online rollouts to account for the student's evolving capabilities? Also, could the authors clarify how difficulty-based upsampling is performed?

  3. Why is correctness balancing performed per prompt rather than across the batch? Could batch-level balancing preserve more informative trajectories while achieving similar performance?

  4. How does outcome-guided margin calibration improve the reliability of teacher supervision? Uniform corrections may suppress valid intermediate reasoning in incorrect trajectories, while calibration is skipped when all trajectories are incorrect. Moreover, unlike vanilla OPD, Uni-OPD uses ground-truth correctness labels (Equation 6, Algorithm 3) to adjust token-level advantages. Could the authors compare Uni-OPD against vanilla OPD with an additional outcome reward (e.g., assigning positive rewards to correct trajectories and negative rewards to incorrect ones) to clarify whether the improvements arise from the proposed margin calibration or simply from incorporating additional correctness supervision?

  5. Can the authors provide longer training curves to demonstrate the convergence and stability of Uni-OPD? Figure 1 shows Uni-OPD training only up to approximately 50 steps, while RL continues for several hundred steps. Appendix C.5 also reports OPD and Uni-OPD training curves only up to 50 steps, making it unclear whether Uni-OPD has converged, continues improving, or degrades with additional training. Could the authors extend the training curves to longer horizons (e.g., 350 steps, like RL in Figure 1)? This will also clarify whether the faster convergence compared to RL comes from the proposed Uni-OPD or the OPD.

[1] Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning, EMNLP, 2026

7 comments
AuthorsOpenReview ↗

Response to Reviewer X8Zv (1/5)

Dear Reviewer X8Zv,

Thank you for your insightful feedback and for acknowledging that (i) improving the efficiency of student learning and the reliability of teacher supervision is an important direction, and (ii) our extensive experimental evaluation across diverse distillation settings yields consistent results. We address your questions and concerns below.

W1 & Q1: About student exploration.

R1: Thank you for pointing out the need to evaluate student exploration more directly and for recommending relevant work. We agree that policy entropy and response length may not directly measure the diversity of reasoning strategies. We clarify our method and provide additional evaluations below.

  • Data balancing: improving exposure to and use of informative problems and trajectories.

Data balancing does not create new trajectories within the current training step, but it can provide richer opportunities for exploration during training. Data balancing adjusts the sampling distribution of training problems and the use of existing trajectories. This helps the student gain better exposure to informative problems and trajectories and use them more effectively.

Specifically, offline difficulty-aware balancing preserves the full range of difficulty levels. It replicates examples from underrepresented levels until their counts are close to that of the dominant level. This increases their chances of being sampled during training. Online correctness-aware balancing balances correct and incorrect trajectories within already-generated mixed rollout groups. Both types of trajectories can then provide training signals for model updates.

  • Additional reasoning-strategy diversity evaluation following [1].

Following [1], we use Cov@N to measure the expected number of distinct reasoning strategies covered by N correct solutions to the same problem. We randomly select 150 problems from the public dataset after removing overlaps with our training data. We compare the initial Student, Teacher, OPD, and Uni-OPD.

All models use the same decoding settings and generate 32 responses per problem. Pass@1 is computed over all 150 problems. We evaluate strategy diversity on the 120 problems for which all four models produce at least 16 correct solutions. For each problem, we match the number of correct solutions across models. We then use GPT-5.5 with the publicly available clustering prompt from [1] to group solutions by their reasoning strategies. We then compute Cov@2, Cov@4, Cov@8, and Cov@16.

  • Uni-OPD shows higher accuracy and reasoning-strategy diversity than OPD.

As shown in Table C1, Uni-OPD improves Pass@1 from 91.5% for vanilla OPD to 92.0%. It also increases Cov@16 from 1.201 to 1.231. Cov@2, Cov@4, and Cov@8 also improve. These results show that Uni-OPD achieves higher accuracy and slightly higher reasoning-strategy coverage than vanilla OPD.

We also observe that the Cov@16 values of OPD and Uni-OPD are both close to the Teacher’s value of 1.205. However, they remain below the initial Student’s value of 1.512. This observation is consistent with the findings in [1]. That study also observes reduced reasoning-strategy diversity after RL post-training.

Table C1: Comparison of solution accuracy and reasoning-strategy diversity following [1].

MethodPass@1 (%)Cov@2Cov@4Cov@8Cov@16Avg. # Clusters
Student77.41.1411.2791.4031.5121.59
Teacher91.71.0631.1181.1651.2051.23
OPD91.51.0571.1111.1631.2011.22
Uni-OPD92.01.0591.1191.1781.2311.29
  • Revising “improved exploration” to “improved informative-state exposure.”

To align our description with the mechanisms and evidence above, we replace “improved exploration” with “improved informative-state exposure” in the revised manuscript. This wording emphasizes how data balancing improves exposure to and use of informative training problems and trajectories.

We also include these evaluations and analyses in Appendix E.4 and update the related descriptions in the introduction, method, and experimental analysis.

[1] Lee et al. Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning. EMNLP, 2026.

AuthorsOpenReview ↗

Response to Reviewer X8Zv (2/5)

W2 & Q2: Motivation and implementation details of offline difficulty-aware balancing.

R2: We thank the reviewer for this insightful review. We provide experiments with dynamic difficulty updates and analyze the computational cost.

(1) Motivation: preserving the full difficulty range and improving problem sampling.

Our goal is to provide practical training recipes for OPD from the student's perspective. As shown in Figure 3, removing either easy or hard problems reduces final performance. OPD provides token-level distillation supervision. A problem can therefore still provide useful training signals when all its responses are correct or all are incorrect.

This observation motivates us to preserve the full difficulty range and upsample underrepresented bins. Our intuition is that a more balanced sampling distribution helps each batch cover problems at different difficulty levels. This helps the student gain better exposure to informative problems and trajectories and use them more effectively.

(2) Implementation of difficulty-based upsampling.

We provide the implementation details in Appendix A.1 of the revised manuscript. The initial student generates eight responses per training problem. We check correctness using the verifier for the corresponding task. We then divide the problems into nine bins, from 0 to 8, based on the number of correct responses. We process the difficulty distribution of each data source separately.

For U-shaped distributions, we upsample bins with 1–7 correct responses. For mirrored-J-shaped distributions, we upsample bins with 1–8 correct responses. We replicate problems in these bins until their counts are close to that of the dominant bin. This produces an approximately balanced difficulty distribution. All original problems are retained. In the default setting, we construct the training pool once, before training. During training, the current student generates fresh responses for each sampled problem. The offline responses are used only for the initial difficulty estimation.

(3) Changes in student ability and dynamic difficulty updates.

We agree that problem difficulty depends on the student's ability and therefore changes during training. The initial estimates are used to construct a fixed problem-sampling distribution.

The reviewer's suggestion to update difficulty using online rollouts is a reasonable way to adapt to changes in student ability. However, existing online rollouts directly provide information only about problems that have already been sampled. Problems that have not yet been sampled lack corresponding difficulty observations. One-time offline estimation provides a difficulty reference for the entire training pool before problem sampling.

To address the concern about outdated estimates, we add a comparison with dynamic difficulty updates. We re-estimate the difficulty of the remaining training data with the current student every 10 steps. We then adjust the sampling distribution for subsequent training. As shown in Table C2, one-time offline balancing improves the math/code averages from 47.8/60.5 to 48.8/61.5. Dynamic updates further improve them to 49.1/62.3. These results show that initial difficulty balancing improves performance, while dynamic updates provide further gains.

Table C2: Comparison of data balancing with fixed and dynamically updated difficulty estimates using Qwen3-4B-RL as the teacher.

MethodAIME 2024AIME 2025HMMT 25 Feb.HMMT 25 Nov.Math Avg.HumanEval+MBPP+LCBCode Avg.
OPD63.355.634.837.547.885.269.826.660.5
OPD w/ offline data balancing63.956.735.239.548.887.770.026.761.5
OPD w/ dynamic offline data balancing63.358.334.939.749.188.072.326.762.3

(4) One-time offline cost and per-step training overhead.

Appendix C.6 separates the cost of one-time offline estimation from that of online operations during training. The offline stage generates eight responses per problem with the student and checks their correctness. It requires neither teacher scoring nor gradient updates. The estimates can be reused across runs with the same initial student.

For Qwen3-1.7B and Qwen3-4B, offline generation over 3,200 problems with eight responses each costs about 2.0 and 3.8 GPU-hours. Both models converge within 50 steps. These 50-step runs take 9.2/12.0 hours on 16 GPUs, or about 147/192 GPU-hours. At this processing scale, offline generation costs 1.4%/2.0% of training.

AuthorsOpenReview ↗

Response to Reviewer X8Zv (3/5)

Q3: Prompt-level vs. batch-level correctness balancing and informative trajectory retention.

R3: Thank you for this question. Our method also supports batch-level balancing. We further compare the number and composition of retained trajectories using the same rollouts. We explain the results below.

(1) Performance comparison.

Table C3 summarizes the results of the full batch-level (Global) and prompt-level (Group) configurations from Table E.7 in Appendix E.3. Both apply correctness balancing before margin calibration. Prompt-level achieves higher math/code averages (48.2/62.4) than batch-level (47.8/60.9), and both outperform OPD (no shift).

Table C3: Performance comparison of batch-level and prompt-level balancing. Results are from Table E.7 in Appendix E.3. Please refer to that table for the full configurations.

ConfigurationMath Avg.Code Avg.
OPD (no shift)44.659.0
Batch-level47.860.9
Prompt-level (ours)48.262.4

(2) Trajectory retention analysis.

We apply both balancing methods to the same rollout batches and directly compare the trajectories they retain. Each batch contains 64 prompts, with 16 trajectories per prompt. Table C4 reports the retention statistics at a batch-level correctness rate similar to that observed during training.

Here, a mixed group contains both correct and incorrect trajectories for the same prompt before filtering. Minority trajectories belong to the less frequent outcome class within originally mixed groups with unequal numbers of correct and incorrect trajectories. Rare correct solutions are correct trajectories from prompts with only 1–3 correct responses out of 16. The last column of Table C4 reports the percentage of originally mixed groups that retain both outcomes after filtering.

Batch-level balancing retains 811 trajectories per batch on average, versus 694 for prompt-level balancing. It retains 82.7% of within-prompt minority trajectories and 67.2% of rare correct solutions. After filtering, 95.7% of originally mixed groups retain both outcomes. Prompt-level balancing retains all minority trajectories, including all rare correct solutions, and preserves both outcomes in every originally mixed group.

Table C4: Trajectory retention under prompt-level and batch-level balancing on the same rollouts.

Balancing scopeAvg. trajectories retainedMinority trajectories retained (%)Rare correct solutions retained (%)Groups retaining both outcomes (%)
Prompt-level694100100100
Batch-level81182.767.295.7

These results highlight that prompt-level balancing focuses on the composition of training signals within each problem. For difficult problems, the few correct trajectories provide successful solution paths. For easier problems, the few incorrect trajectories reveal failures that the student may still make. These trajectories provide scarce contrasts between successful and unsuccessful attempts on the same problem.

Under the default 1:1 ratio, prompt-level balancing retains all minority trajectories in mixed groups and samples from the majority side. Thus, every originally mixed group retains at least one correct and one incorrect trajectory after filtering. Batch-level balancing only controls the correct-to-incorrect ratio across the entire batch. The minority side within a prompt group may be the majority side across the batch. These trajectories can therefore be removed. For example, correct trajectories may form the majority across a batch. In this case, the few correct solutions to difficult problems may also be randomly removed.

We also evaluate the reasoning-strategy diversity of models trained with the two balancing configurations. Under the same evaluation settings, prompt-level achieves a Cov@16 of 1.352, compared with 1.296 for batch-level. The prompt-level value is slightly higher.

We therefore adopt prompt-level as the default configuration. This choice considers the performance of the evaluated full configurations. It also considers the preservation of minority trajectories and correct–incorrect contrasts within each problem. Batch-level is a feasible alternative, but retaining more trajectories does not necessarily preserve more complete within-prompt contrasts.

We have also added these trajectory retention comparisons and analyses to Appendix A.2 of the revised manuscript.

Reviewer_hECnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Uni-OPD combines offline difficulty-aware prompt balancing, online correctness-aware trajectory balancing, and outcome-guided margin calibration of OPD rewards. The calibration uses outcome labels to adjust the separation between correct and incorrect trajectory groups. Experiments cover textual and multimodal models, multiple teachers, strong-to-weak transfer, and joint textual/multimodal training. Table 1 reports improvements over OPD and ExOPD.

Critical strengths

Addresses both training-data composition and teacher-feedback reliability through a practical recipe with straightforward components.

Evaluates five domains and 16 benchmarks, including multimodal tasks. Tables 5–6 provide component ablations and compare two calibration strategies.

Critical weaknesses
  1. A simpler outcome-reward baseline is needed. The experiments show that margin calibration helps, but do not show whether the adaptive margin is necessary. A simpler baseline would give a fixed bonus to correct answers and a fixed penalty to incorrect ones. Comparing these methods under the same data balancing and rollout budget would clarify the benefit of the proposed calibration rule.

  2. The single- versus multi-teacher trade-off needs clarification. In Table 2, Uni-OPD’s mathematical average is 63.9 with a single teacher and 61.3 with multiple teachers; logic and document averages are 51.7 vs. 52.2 and 83.7 vs. 83.9. These results establish improvements over multi-teacher OPD, but do not isolate the cause of the single-/multi-teacher math gap. A comparison with matched per-domain exposure would clarify whether joint training carries a residual performance cost.

Questions for authors

How does margin calibration compare with OPD plus a tuned, fixed-coefficient outcome reward under the same data balancing and rollout budget?

Does the math gap in Table 2 persist when single-teacher and multi-teacher training have matched math-data exposure and update budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hECn (1/2)

Dear Reviewer hECn,

We appreciate your helpful feedback and your positive assessment of (i) our practical recipe for addressing training-data composition and teacher-feedback reliability, (ii) the evaluation across five domains and 16 benchmarks, and (iii) the component ablations and comparison of calibration strategies. We respond to your concerns below.

W1 & Q1: OPD with a fixed-coefficient outcome reward.

R1: Thank you for this insightful review. We further conducted experiments with a simpler outcome-reward baseline (i.e., OPD plus a fixed-coefficient outcome reward), using the same offline/online data balancing and rollout budgets. We use a fixed coefficient of 0.2, selected after comparing several candidate values. Tables B1 and B2 report the results with Qwen3-4B-RL and Qwen3-30B-A3B-Instruct-2507 as teachers, respectively.

Specifically, we compare vanilla OPD, OPD with data balancing, OPD with data balancing and a fixed-coefficient outcome reward, and Uni-OPD. Across both teacher settings, data balancing improves the math and code averages over vanilla OPD. Adding the fixed outcome reward provides further gains, showing that outcome-based supervision is beneficial. Uni-OPD achieves the highest averages in both settings. Compared with the fixed outcome reward baseline, it improves math by 0.8 points and code by 1.4 points in Table B1. The corresponding gains in Table B2 are 0.5 points on math and 0.8 points on code. These results support the additional benefit of our margin calibration over a fixed-coefficient outcome reward.
We have also included the fixed outcome reward baseline in Table 5 of the revised manuscript to provide a more complete ablation study and clarify the additional benefit of adaptive margin calibration.

Table B1: Comparison of a fixed-coefficient outcome reward baseline and margin calibration (ours) on math reasoning and code generation with the Qwen3-4B-RL teacher and a Qwen3-4B student.

MethodAIME 2024AIME 2025HMMT 25 Feb.HMMT 25 Nov.Math Avg.HumanEval+MBPP+LCBCode Avg.
OPD60.955.233.438.347.086.370.923.460.2
OPD w/ data balancing63.054.733.438.147.386.471.625.761.2
Uni-OPD62.357.234.939.648.588.072.630.163.6
OPD w/ data balancing & fixed reward61.255.533.740.347.786.671.428.462.2

Table B2: Comparison of a fixed-coefficient outcome reward baseline and margin calibration (ours) on math reasoning and code generation with the Qwen3-30B-A3B-Instruct-2507 teacher and a Qwen3-4B student.

MethodAIME 2024AIME 2025HMMT 25 Feb.HMMT 25 Nov.Math Avg.HumanEval+MBPP+LCBCode Avg.
OPD56.546.428.533.441.282.972.421.659.0
OPD w/ data balancing54.948.129.135.842.082.870.425.759.6
Uni-OPD55.950.229.835.642.983.171.328.060.8
OPD w/ data balancing & fixed reward56.447.228.937.442.481.371.227.460.0
AuthorsOpenReview ↗

Response to Reviewer hECn (2/2)

W2 & Q2: Single- vs. multi-teacher trade-off under matched math exposure.

R2: Thank you for this detailed comment. We clarify the training budgets and provide additional comparisons to assess the single-/multi-teacher math gap.

In Table 2, single-teacher distillation uses 50 updates, whereas multi-teacher distillation uses 100 updates with an approximately 1:1:1 mixture of 14.8K math samples, 14.8K logic samples, and 14.6K document samples (training has already converged at these steps). The global batch size and number of rollouts per prompt are the same across the two settings. Consequently, the original multi-teacher run receives approximately 2/3 of the cumulative math exposure of the single-teacher run.

To examine whether the gap persists after matching cumulative math exposure, we first extend the 1:1:1 multi-teacher run to 150 updates. Its math average is 61.0, compared with 63.9 for single-teacher distillation. Thus, a 2.9-point gap remains after matching cumulative math exposure. Further training beyond convergence does not improve math performance in this setting.

Furthermore, we evaluate a math-focused 3:1:1 mixture for 80 updates, reaching approximately 100% of the single-teacher math exposure. This setting achieves 61.7 on math, narrowing the gap to 2.2 points, while logic and document performance decrease slightly relative to the original multi-teacher setting.

These results show that the math gap persists after matching cumulative math exposure. The math-focused setting partially narrows this gap, with small decreases in logic and document performance. These observations suggest a performance trade-off across domains when integrating multiple capabilities into a single model.

Table B3: Single- and multi-teacher Uni-OPD with different numbers of updates and domain mixtures.

SettingUpdatesRelative math exposure (%)Math Avg.Logic Avg.Document Avg.
Single-teacher, separate runs per domain (ours)5010063.951.783.7
Multi-teacher, 1:1:1 (ours)1006761.352.283.9
Multi-teacher, 1:1:115010061.051.583.6
Multi-teacher, 3:1:180≈10061.751.783.8

We are grateful for your thoughtful suggestions, which motivated these additional comparisons. The new results help clarify the benefit of margin calibration and the trade-off between single- and multi-teacher distillation.

Thank you for the thorough rebuttal and for clarifying my concerns. I maintain my positive score for this work.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer hECn,

We sincerely appreciate your response and continued support. Thank you again!

Reviewer_1Kj3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uni-OPD, a unified on-policy distillation framework that improves knowledge transfer across LLMs and multimodal large language models. The authors identify two major limitations of existing OPD methods: insufficient student exploration of informative trajectories and unreliable teacher supervision. To address the first limitation, they introduce offline difficulty-aware and online correctness-aware data balancing strategies to improve training diversity and the quality of student rollouts. To address the second, they propose outcome-guided margin calibration, which adjusts teacher-provided token-level rewards to better align with trajectory-level correctness.

Critical strengths
  1. The paper identifies two meaningful limitations of existing OPD approaches. The analysis of teacher supervision failures, including approximately 45% pairwise ordering violations, provides compelling empirical motivation.

  2. Uni-OPD combines data balancing and outcome-guided calibration without requiring major architectural changes. Its components can be integrated into existing OPD pipelines.

  3. Comprehensive experimental evaluation. The experiments cover 16 benchmarks across five domains, including multi-teacher, strong-to-weak, and cross-modal distillation, demonstrating broad applicability.

Critical weaknesses
  1. The proposed offline/online balancing strategies largely rely on established data sampling heuristics, while margin calibration resembles outcome-based reward shaping. Comparisons against simpler alternatives, such as OPD with directly incorporated outcome rewards, are missing.

  2. The main formulation emphasizes strict pairwise order consistency using a MinMax margin, whereas the experiments primarily employ mean-based calibration, which does not guarantee this property. This weakens the theoretical justification of the proposed reliability mechanism in Section 3.4 and Appendix B.5.

  3. The proposed calibration requires reliable binary correctness signals. Its applicability to open-ended generation, subjective preferences, and tasks without deterministic verifiers remains unclear.

  4. All experiments use models from the Qwen family, primarily with relatively small students. Furthermore, the cross-modal experiments combine textual and multimodal tasks using teachers derived from the same multimodal backbone, leaving broader cross-architecture transfer insufficiently validated.

  5. The reported computational overhead does not fully account for the one-time offline difficulty-estimation rollout process.

Questions for authors

Please refer to the weakness. Happy to adjust the initial evaluation if all concerns are resolved.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1Kj3 (1/5)

Dear Reviewer 1Kj3,

Thank you for your constructive feedback and for recognizing (i) the compelling empirical motivation behind our work, (ii) the ability to integrate Uni-OPD into existing OPD pipelines without major architectural changes, and (iii) our comprehensive experimental evaluation. We address your concerns below.

W1: Direct outcome-reward baselines.

R1: Thank you for your careful review and concise summary of our research motivation. Indeed, our core motivation is to identify and address key OPD bottlenecks from both the student and teacher perspectives. Through this, we seek to provide a practical and general training recipe. Offline/online balancing takes a heuristic or empirical approach to data sampling. It aims to maintain data diversity and a balanced difficulty spectrum. This helps the student gain better exposure to informative problems and trajectories and use them more effectively.

Margin calibration shares similarities with reward shaping in reinforcement learning. Its purpose is to calibrate unreliable teacher supervision and better align distillation signals with trajectory correctness.

We also thank you for pointing out the need to compare with simpler baselines. This helps further assess the additional benefit of our margin calibration over a fixed outcome reward. To this end, we have added an fixed outcome reward baseline. It uses exactly the same offline/online balancing, rollout count, and update budget as Uni-OPD. It assigns a fixed bonus to correct trajectories and a fixed penalty to incorrect ones.

Tables A1 and A2 present comparisons under multi-teacher and strong-to-weak distillation, respectively. Both comparisons cover math reasoning and code generation. Specifically, compared with the fixed outcome reward baseline, Uni-OPD improves the math and code averages by 0.8 and 1.4 points, respectively, in Table A1. The corresponding gains in Table A2 are 0.5 and 0.8 points. These results consistently show that our margin calibration provides additional performance gains over directly adding a fixed outcome reward.

We have also added these baseline results to Table 5 of the revised manuscript to provide more complete comparisons and ablation analyses.

Table A1: Comparison of a direct outcome reward baseline and margin calibration (ours) on math reasoning and code generation with the Qwen3-4B-RL teacher and a Qwen3-4B student.

MethodAIME 2024AIME 2025HMMT 25 Feb.HMMT 25 Nov.Math Avg.HumanEval+MBPP+LCBCode Avg.
OPD60.955.233.438.347.086.370.923.460.2
Uni-OPD (ours)62.357.234.939.648.588.072.630.163.6
OPD w/ fixed reward61.255.533.740.347.786.671.428.462.2

Table A2: Comparison of a direct outcome reward baseline and margin calibration (ours) on math reasoning and code generation with the Qwen3-30B-A3B-Instruct-2507 teacher and a Qwen3-4B student.

MethodAIME 2024AIME 2025HMMT 25 Feb.HMMT 25 Nov.Math Avg.HumanEval+MBPP+LCBCode Avg.
OPD56.546.428.533.441.282.972.421.659.0
Uni-OPD (ours)55.950.229.835.642.983.171.328.060.8
OPD w/ fixed reward56.447.228.937.442.481.371.227.460.0
AuthorsOpenReview ↗

Response to Reviewer 1Kj3 (2/5)

W2: Relationship between the strict MinMax formulation and the Mean-based experimental calibration.

R2: As discussed in Appendix B.5, MinMax serves as the worst-case diagnostic and, under MinMax shift, enforces strict pairwise order consistency. Mean setting instead directly calibrates the separation between the average returns of correct and incorrect trajectories. For every non-degenerate group, its post-calibration mean separation is at least δ\delta, with equality whenever a shift is triggered. Under either margin definition, whenever a shift is applied, every sampled positive–negative score gap increases by the same nonnegative amount and therefore no new pairwise inversion is introduced.

We provide three complementary pieces of evidence connecting the Mean-based experimental implementation to the strict pairwise reliability criterion:

  • Empirical pairwise repair. Across 1,107 non-degenerate training groups, Mean calibration reduces the pairwise inversion rate from 45.55% to 0.06%. Among the 1,075 groups containing at least one inversion before calibration, 99.0% are fully repaired afterward. (Appendix E.3)
  • Effectiveness of the strict variant. With group scope and Spread fixed, the strict MinMax variant achieves 48.1 on math and 61.4 on code, substantially outperforming vanilla OPD at 44.6/59.0. Mean achieves 48.2/62.4 under the same setting. This confirms that the strict MinMax formulation itself is empirically effective, while Mean does not sacrifice final performance. (Table E.7)
  • Practical training stability. In controlled comparisons differing only in the margin mode, Mean produces less variable corrections (shift s.d. 0.031 vs. 0.050), a substantially smaller entropy decline (0.0025 vs. 0.0176), and one-third the average rollout repetition (0.0014 vs. 0.0042). This explains why Mean is used as the default implementation after establishing its empirical connection to the pairwise criterion. (Figure E.1; Table E.8)

Together, these results establish a coherent connection between our formulation and implementation: MinMax provides the worst-case reliability criterion, while Mean offers a robust practical calibration that guarantees the target group-mean separation and empirically removes nearly all pairwise inversions observed during training. The corresponding analyses and visualizations are provided in Appendix E.3, and the main paper now explicitly states that Mean is our default calibration mode.

AuthorsOpenReview ↗

Response to Reviewer 1Kj3 (3/5)

W3: Open-ended tasks and subjective preferences.

R3 : Thank you for raising this concern about applicability. To examine our method's applicability to open-ended generation and subjective quality assessment, we extended margin calibration to continuous quality scores.

We use Skywork-Reward-Llama-3.1-8B-v0.2 to assign a continuous quality score to each complete generated trajectory. These scores guide calibration during training. The OPD teacher continues to provide token-level distillation supervision. We extend the original margin constraint between binary outcome groups to a slope constraint on OPD advantages with respect to quality ranks.

Specifically, for KK trajectories generated for the same prompt, we construct standardized quality ranks UiU_i from their scores. These ranks have zero mean and unit variance within the group. Let Aˉi\bar{A}_i denote the mean token-level advantage of trajectory ii. The slope of the linear regression of Aˉi\bar{A}_i on UiU_i is

β=1K∑i=1KAˉiUi.\beta = \frac{1}{K}\sum_{i=1}^{K}\bar{A}_i U_i.

This slope measures the linear trend in advantage as quality rank increases. We require the slope to be at least δ/2\delta/2, where δ\delta is the target margin parameter. When the current slope is below this target, we apply the same correction to all tokens within each trajectory:

A~i(t)=Ai(t)+[δ2−β]+Ui,\tilde{A}_i(t) = A_i(t) + \left[\frac{\delta}{2}-\beta\right]_+ U_i,

where [x]+=max⁡(x,0)[x]_+ = \max(x,0). Trajectories ranked above the group mean receive a positive correction. Those ranked below it receive a negative correction. The correction magnitude is determined by the gap between the current slope and the target. The calibrated slope satisfies

β~=max⁡(β,δ2).\tilde{\beta} = \max\left(\beta,\frac{\delta}{2}\right).

This procedure uses the relative quality ordering provided by continuous scores. It does not require converting the scores into binary rewards or using a deterministic correctness verifier.

We evaluate the following benchmark categories:

  • Open-ended dialogue and preference evaluation: WildBench evaluates response quality in open-ended dialogue. AlpacaEval2-LC and ArenaHard2 evaluate quality and preferences through comparisons between responses.
  • Open-ended generation with subjective evaluation: CWv3 evaluates creative writing quality. It involves subjective judgments that cannot be captured by a single correct answer.
  • Additional capability evaluation: IFBench evaluates instruction following under verifiable constraints. PopQA evaluates factual question answering. MMLU-Redux evaluates multidisciplinary knowledge and reasoning through multiple-choice questions.

As shown in Table A3, OPD improves the average score across the three open-ended dialogue and preference benchmarks (AvgChat) from 23.6 to 40.1 over the SFT model. It also increases the overall average across all seven benchmarks from 34.4 to 43.5. These results show that OPD effectively transfers teacher capabilities on these tasks. Uni-OPD further raises AvgChat to 41.7 and the overall average to 44.7. On CWv3, which evaluates creative writing with subjective quality judgments, Uni-OPD also improves OPD's score from 69.9 to 71.0.

These results show that our calibration mechanism can be extended to supervision based on continuous quality scores. It further improves OPD on open-ended generation and preference-related evaluations without binary rewards or deterministic correctness verifiers. We have also added these experiments and analyses to the main paper of the revised manuscript to further clarify the method's applicability to these tasks.

Table A3: Results on open-ended generation and general capability benchmarks. GPT-4o serves as the judge model during evaluation.

MethodWildBenchAlpacaEval2-LCArenaHard2AvgChatCWv3PopQAIFBenchMMLU-ReduxAvg
Qwen2.5-7B-Instruct-SFT19.334.117.323.666.821.721.460.434.4
Qwen2.5-7B-Instruct-RLMT (Teacher)46.648.528.641.270.322.121.171.744.1
OPD44.647.827.840.169.921.921.470.943.5
Uni-OPD47.048.929.141.771.022.123.171.544.7

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Chairs and All Reviewers

Dear Chairs and Reviewers,

We sincerely thank the reviewers for their careful reviews and constructive feedback. We appreciate their recognition of the important research direction (Reviewer X8Zv), the compelling empirical motivation and broad applicability (Reviewer 1Kj3), and our practical recipe with straightforward components (Reviewer hECn). The reviewers also highlight the comprehensive experimental evaluation (Reviewers 1Kj3, hECn, and X8Zv), component ablations and improvements over OPD and ExOPD (Reviewer hECn), and consistent results across diverse distillation settings (Reviewer X8Zv).

We have revised the manuscript in response to the reviewers’ comments and suggestions. The main changes are marked in purple. Below, we summarize the key additions and clarifications. References below point to the revised manuscript unless explicitly marked as “rebuttal.”

Regarding questions about experiments, we have addressed the concerns as follows:

  • Fixed outcome-reward baselines (Table 5) and further analyses of calibration reliability (Appendix E.3). (For all reviewers)
  • Open-ended evaluations with continuous quality scores (new Table 7), along with additional model settings (new Tables E.4–E.5) and a larger student (Table 3). (For Reviewer 1Kj3)
  • Reasoning-strategy diversity (new Appendix E.4), dynamic difficulty updates (new rebuttal Table C2), and prompt-level vs. batch-level balancing (Appendices A.2 and E.3). (For Reviewer X8Zv)
  • Single-/multi-teacher comparisons under matched math exposure (new rebuttal Table B3). (For Reviewer hECn)

Regarding questions about methodological details and scope, we have made the following clarifications:

  • The guarantees of Mean and MinMax calibration and the scope of trajectory-level calibration (Section 3.4 and Appendix B.5). (For Reviewers 1Kj3 and X8Zv)
  • The balancing implementation and the revised description of “improved informative-state exposure.” (Section 3.3 and Appendices A.1–A.2) (For Reviewer X8Zv)
  • The distinction between one-time offline costs and per-step overhead (Appendix C.6), and the shared-tokenizer scope of our experiments (Section 4). (For Reviewers 1Kj3 and X8Zv)

Detailed responses are provided under each review. We look forward to further feedback and discussion. Please feel free to let us know if further details or explanations would be helpful.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.