- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Uni-OPD is a unified on-policy distillation framework. The authors state that there are limitations in existing OPD (insufficient student exploration and unreliable teacher supervision). They mainly propose (1) offline difficulty-aware data balancing, (2) online correctness-aware data balancing, and (3) outcome-guided margin calibration.
Critical strengths
-
I believe improving the efficiency of student learning and the reliability of teacher supervision is an important direction.
-
The paper provides extensive experimental evaluations across single, multi-teacher, strong-to-weak, and cross-modal distillation setups, and the results are consistent.
Critical weaknesses
-
The claim of improved student exploration is not sufficiently supported. The author states insufficient student exploration as a bottleneck of OPD. However, the proposed method primarily modifies the distribution of training data (via upsampling) and their trajectory (balancing correct/incorrect) rather than actual exploration. In detail, (1) online correctness-aware balancing filters already-generated trajectories and does not inherently increase the diversity or coverage of student rollouts. For (2) offline difficulty-aware data balancing, the author did not specify how they upsample underrepresented difficulty (The author should specify; I saw A.1, but still, I could not find detailed information). The authors should provide direct evidence of improved exploration, such as reasoning-strategy diversity or the discovery rate of distinct successful trajectories.
-
Offline difficulty-aware balancing introduces additional computational overhead and relies on potentially outdated difficulty estimates. In Appendix A.1, the authors estimate prompt difficulty by generating eight responses for every training example over the entire dataset. Moreover, difficulty is estimated using the initial student policy, these estimates may become outdated as the student improves. Considering that OPD already generates multiple trajectories during training, it is unclear why difficulty cannot be estimated and updated using existing online rollouts. Furthermore, the difficulty distribution will change as the student learns (because the student is the labeler). This raises concern about whether balancing a fixed initial difficulty distribution remains beneficial throughout training.
-
Data balancing and margin calibration may discard or distort informative training signals. Both methods rely on final-answer correctness, which does not necessarily reflect the quality of intermediate reasoning. For example, an incorrect trajectory may contain valid reasoning with only a minor mistake near the end. However, (1) online correctness-aware balancing may discard such informative trajectories to maintain a 1:1 correct-to-incorrect ratio within each prompt. For example, when one correct and nine incorrect trajectories are generated, only two may be retained, whereas all ten are retained when every trajectory is incorrect. It is also unclear why balancing must be performed per prompt rather than across the entire batch, which could preserve more trajectories. (2) Outcome-guided margin calibration applies a uniform trajectory-level correction to token-level advantages based on outcome correctness (Section 3.4 and Algorithms 2,3), potentially suppressing valid intermediate reasoning in incorrect trajectories while reinforcing uninformative reasoning in correct ones.
Questions for authors
-
Does Uni-OPD encourage student exploration? If so, can the authors provide direct evidence of improved student exploration? Although Appendix C.5 reports policy entropy and response length, these metrics do not necessarily reflect reasoning-strategy diversity or exploration coverage. Could the authors evaluate reasoning-strategy diversity of distinct successful trajectories [1]?
-
Why is an additional offline rollout pass necessary for difficulty estimation? Could the authors instead use online rollouts to account for the student's evolving capabilities? Also, could the authors clarify how difficulty-based upsampling is performed?
-
Why is correctness balancing performed per prompt rather than across the batch? Could batch-level balancing preserve more informative trajectories while achieving similar performance?
-
How does outcome-guided margin calibration improve the reliability of teacher supervision? Uniform corrections may suppress valid intermediate reasoning in incorrect trajectories, while calibration is skipped when all trajectories are incorrect. Moreover, unlike vanilla OPD, Uni-OPD uses ground-truth correctness labels (Equation 6, Algorithm 3) to adjust token-level advantages. Could the authors compare Uni-OPD against vanilla OPD with an additional outcome reward (e.g., assigning positive rewards to correct trajectories and negative rewards to incorrect ones) to clarify whether the improvements arise from the proposed margin calibration or simply from incorporating additional correctness supervision?
-
Can the authors provide longer training curves to demonstrate the convergence and stability of Uni-OPD? Figure 1 shows Uni-OPD training only up to approximately 50 steps, while RL continues for several hundred steps. Appendix C.5 also reports OPD and Uni-OPD training curves only up to 50 steps, making it unclear whether Uni-OPD has converged, continues improving, or degrades with additional training. Could the authors extend the training curves to longer horizons (e.g., 350 steps, like RL in Figure 1)? This will also clarify whether the faster convergence compared to RL comes from the proposed Uni-OPD or the OPD.
[1] Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning, EMNLP, 2026
7 comments
Response to Reviewer X8Zv (2/5)
W2 & Q2: Motivation and implementation details of offline difficulty-aware balancing.
R2: We thank the reviewer for this insightful review. We provide experiments with dynamic difficulty updates and analyze the computational cost.
(1) Motivation: preserving the full difficulty range and improving problem sampling.
Our goal is to provide practical training recipes for OPD from the student's perspective. As shown in Figure 3, removing either easy or hard problems reduces final performance. OPD provides token-level distillation supervision. A problem can therefore still provide useful training signals when all its responses are correct or all are incorrect.
This observation motivates us to preserve the full difficulty range and upsample underrepresented bins. Our intuition is that a more balanced sampling distribution helps each batch cover problems at different difficulty levels. This helps the student gain better exposure to informative problems and trajectories and use them more effectively.
(2) Implementation of difficulty-based upsampling.
We provide the implementation details in Appendix A.1 of the revised manuscript. The initial student generates eight responses per training problem. We check correctness using the verifier for the corresponding task. We then divide the problems into nine bins, from 0 to 8, based on the number of correct responses. We process the difficulty distribution of each data source separately.
For U-shaped distributions, we upsample bins with 1–7 correct responses. For mirrored-J-shaped distributions, we upsample bins with 1–8 correct responses. We replicate problems in these bins until their counts are close to that of the dominant bin. This produces an approximately balanced difficulty distribution. All original problems are retained. In the default setting, we construct the training pool once, before training. During training, the current student generates fresh responses for each sampled problem. The offline responses are used only for the initial difficulty estimation.
(3) Changes in student ability and dynamic difficulty updates.
We agree that problem difficulty depends on the student's ability and therefore changes during training. The initial estimates are used to construct a fixed problem-sampling distribution.
The reviewer's suggestion to update difficulty using online rollouts is a reasonable way to adapt to changes in student ability. However, existing online rollouts directly provide information only about problems that have already been sampled. Problems that have not yet been sampled lack corresponding difficulty observations. One-time offline estimation provides a difficulty reference for the entire training pool before problem sampling.
To address the concern about outdated estimates, we add a comparison with dynamic difficulty updates. We re-estimate the difficulty of the remaining training data with the current student every 10 steps. We then adjust the sampling distribution for subsequent training. As shown in Table C2, one-time offline balancing improves the math/code averages from 47.8/60.5 to 48.8/61.5. Dynamic updates further improve them to 49.1/62.3. These results show that initial difficulty balancing improves performance, while dynamic updates provide further gains.
Table C2: Comparison of data balancing with fixed and dynamically updated difficulty estimates using Qwen3-4B-RL as the teacher.
| Method | AIME 2024 | AIME 2025 | HMMT 25 Feb. | HMMT 25 Nov. | Math Avg. | HumanEval+ | MBPP+ | LCB | Code Avg. |
|---|---|---|---|---|---|---|---|---|---|
| OPD | 63.3 | 55.6 | 34.8 | 37.5 | 47.8 | 85.2 | 69.8 | 26.6 | 60.5 |
| OPD w/ offline data balancing | 63.9 | 56.7 | 35.2 | 39.5 | 48.8 | 87.7 | 70.0 | 26.7 | 61.5 |
| OPD w/ dynamic offline data balancing | 63.3 | 58.3 | 34.9 | 39.7 | 49.1 | 88.0 | 72.3 | 26.7 | 62.3 |
(4) One-time offline cost and per-step training overhead.
Appendix C.6 separates the cost of one-time offline estimation from that of online operations during training. The offline stage generates eight responses per problem with the student and checks their correctness. It requires neither teacher scoring nor gradient updates. The estimates can be reused across runs with the same initial student.
For Qwen3-1.7B and Qwen3-4B, offline generation over 3,200 problems with eight responses each costs about 2.0 and 3.8 GPU-hours. Both models converge within 50 steps. These 50-step runs take 9.2/12.0 hours on 16 GPUs, or about 147/192 GPU-hours. At this processing scale, offline generation costs 1.4%/2.0% of training.
Response to Reviewer X8Zv (3/5)
Q3: Prompt-level vs. batch-level correctness balancing and informative trajectory retention.
R3: Thank you for this question. Our method also supports batch-level balancing. We further compare the number and composition of retained trajectories using the same rollouts. We explain the results below.
(1) Performance comparison.
Table C3 summarizes the results of the full batch-level (Global) and prompt-level (Group) configurations from Table E.7 in Appendix E.3. Both apply correctness balancing before margin calibration. Prompt-level achieves higher math/code averages (48.2/62.4) than batch-level (47.8/60.9), and both outperform OPD (no shift).
Table C3: Performance comparison of batch-level and prompt-level balancing. Results are from Table E.7 in Appendix E.3. Please refer to that table for the full configurations.
| Configuration | Math Avg. | Code Avg. |
|---|---|---|
| OPD (no shift) | 44.6 | 59.0 |
| Batch-level | 47.8 | 60.9 |
| Prompt-level (ours) | 48.2 | 62.4 |
(2) Trajectory retention analysis.
We apply both balancing methods to the same rollout batches and directly compare the trajectories they retain. Each batch contains 64 prompts, with 16 trajectories per prompt. Table C4 reports the retention statistics at a batch-level correctness rate similar to that observed during training.
Here, a mixed group contains both correct and incorrect trajectories for the same prompt before filtering. Minority trajectories belong to the less frequent outcome class within originally mixed groups with unequal numbers of correct and incorrect trajectories. Rare correct solutions are correct trajectories from prompts with only 1–3 correct responses out of 16. The last column of Table C4 reports the percentage of originally mixed groups that retain both outcomes after filtering.
Batch-level balancing retains 811 trajectories per batch on average, versus 694 for prompt-level balancing. It retains 82.7% of within-prompt minority trajectories and 67.2% of rare correct solutions. After filtering, 95.7% of originally mixed groups retain both outcomes. Prompt-level balancing retains all minority trajectories, including all rare correct solutions, and preserves both outcomes in every originally mixed group.
Table C4: Trajectory retention under prompt-level and batch-level balancing on the same rollouts.
| Balancing scope | Avg. trajectories retained | Minority trajectories retained (%) | Rare correct solutions retained (%) | Groups retaining both outcomes (%) |
|---|---|---|---|---|
| Prompt-level | 694 | 100 | 100 | 100 |
| Batch-level | 811 | 82.7 | 67.2 | 95.7 |
These results highlight that prompt-level balancing focuses on the composition of training signals within each problem. For difficult problems, the few correct trajectories provide successful solution paths. For easier problems, the few incorrect trajectories reveal failures that the student may still make. These trajectories provide scarce contrasts between successful and unsuccessful attempts on the same problem.
Under the default 1:1 ratio, prompt-level balancing retains all minority trajectories in mixed groups and samples from the majority side. Thus, every originally mixed group retains at least one correct and one incorrect trajectory after filtering. Batch-level balancing only controls the correct-to-incorrect ratio across the entire batch. The minority side within a prompt group may be the majority side across the batch. These trajectories can therefore be removed. For example, correct trajectories may form the majority across a batch. In this case, the few correct solutions to difficult problems may also be randomly removed.
We also evaluate the reasoning-strategy diversity of models trained with the two balancing configurations. Under the same evaluation settings, prompt-level achieves a Cov@16 of 1.352, compared with 1.296 for batch-level. The prompt-level value is slightly higher.
We therefore adopt prompt-level as the default configuration. This choice considers the performance of the evaluated full configurations. It also considers the preservation of minority trajectories and correct–incorrect contrasts within each problem. Batch-level is a feasible alternative, but retaining more trajectories does not necessarily preserve more complete within-prompt contrasts.
We have also added these trajectory retention comparisons and analyses to Appendix A.2 of the revised manuscript.
Response to Reviewer X8Zv (1/5)
Dear Reviewer
X8Zv,Thank you for your insightful feedback and for acknowledging that (i) improving the efficiency of student learning and the reliability of teacher supervision is an important direction, and (ii) our extensive experimental evaluation across diverse distillation settings yields consistent results. We address your questions and concerns below.
R1: Thank you for pointing out the need to evaluate student exploration more directly and for recommending relevant work. We agree that policy entropy and response length may not directly measure the diversity of reasoning strategies. We clarify our method and provide additional evaluations below.
Data balancing does not create new trajectories within the current training step, but it can provide richer opportunities for exploration during training. Data balancing adjusts the sampling distribution of training problems and the use of existing trajectories. This helps the student gain better exposure to informative problems and trajectories and use them more effectively.
Specifically, offline difficulty-aware balancing preserves the full range of difficulty levels. It replicates examples from underrepresented levels until their counts are close to that of the dominant level. This increases their chances of being sampled during training. Online correctness-aware balancing balances correct and incorrect trajectories within already-generated mixed rollout groups. Both types of trajectories can then provide training signals for model updates.
Following [1], we use Cov@N to measure the expected number of distinct reasoning strategies covered by N correct solutions to the same problem. We randomly select 150 problems from the public dataset after removing overlaps with our training data. We compare the initial Student, Teacher, OPD, and Uni-OPD.
All models use the same decoding settings and generate 32 responses per problem. Pass@1 is computed over all 150 problems. We evaluate strategy diversity on the 120 problems for which all four models produce at least 16 correct solutions. For each problem, we match the number of correct solutions across models. We then use GPT-5.5 with the publicly available clustering prompt from [1] to group solutions by their reasoning strategies. We then compute Cov@2, Cov@4, Cov@8, and Cov@16.
As shown in Table C1, Uni-OPD improves Pass@1 from 91.5% for vanilla OPD to 92.0%. It also increases Cov@16 from 1.201 to 1.231. Cov@2, Cov@4, and Cov@8 also improve. These results show that Uni-OPD achieves higher accuracy and slightly higher reasoning-strategy coverage than vanilla OPD.
We also observe that the Cov@16 values of OPD and Uni-OPD are both close to the Teacher’s value of 1.205. However, they remain below the initial Student’s value of 1.512. This observation is consistent with the findings in [1]. That study also observes reduced reasoning-strategy diversity after RL post-training.
Table C1: Comparison of solution accuracy and reasoning-strategy diversity following [1].
To align our description with the mechanisms and evidence above, we replace “improved exploration” with “improved informative-state exposure” in the revised manuscript. This wording emphasizes how data balancing improves exposure to and use of informative training problems and trajectories.
We also include these evaluations and analyses in Appendix E.4 and update the related descriptions in the introduction, method, and experimental analysis.
[1] Lee et al. Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning. EMNLP, 2026.