acceptodds
Under review as a conference paper at ICLR 2027

Outcome-Guided On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) provides denser token-level supervision and better computational efficiency than Reinforcement Learning with Verifiable Rewards (RLVR). However, this denser supervision may introduce substantial noise and training instability. Existing improvements often rely on high-variance per-token statistics and introduce extra hyperparameters and trade-offs. Based on the advantage formulation in RLVR, we analyze the OPSD objective from the same perspective, incorporating outcome correctness signals. We find that vanilla OPSD imposes insufficient penalties and excessive rewards on incorrect trajectories because it applies a fixed divergence objective regardless of outcome correctness. Furthermore, the reliability of teacher supervision is associated with both trajectory outcome and the cumulative average teacher entropy along the rollout. Based on these observations, we propose Outcome-Guided On-Policy Self-Distillation (OG-OPSD), which dynamically adapts both the divergence objective and distillation position according to binary outcome rewards and the cumulative average teacher entropy. Extensive experiments show that OG-OPSD consistently improves the performance of vanilla OPSD and multiple strong baselines in mathematical reasoning, multimodal reasoning, and out-of-distribution tasks across Qwen3 models at 1.7B, 4B, and 8B scales, as well as Qwen3-VL-2B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_do2ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes two changes to on-policy (self-)distillation. First, it argues that a forward KL objective is more informative as a reward signal on correct trajectories, and that reverse KL is better for incorrect ones. Second, it posits that on incorrect trajectories, the teacher signal is meaningless beyond a certain token number where the response has already strayed too far from a correct response. The two methods together lift the accuracy of Qwen3 1.7-8B models on AIME/HMMT by 2-3 points over vanilla OPSD. Ablation studies are given, and analysis shows that the method preserves accuracy on another domain somewhat better than vanilla OPSD.

Critical strengths

I see two strengths that merit acceptance:

  • The claim that teacher supervision on some tokens is meaningless is indeed true and in my opinion could be useful in informing RL design.
  • The experiments are well-conducted. I have a minor comment (see weaknesses and questions) but in general I think the hyperparameters and settings were chosen well.
Critical weaknesses

I have the following concerns:

  • My biggest concern is perhaps with the framing of the paper and I have two points here. First of all, I think the paper is not really proposing a new method but performing a careful study of a few factors like which direction of KL divergence is better for learning. Indeed, such an analysis is helpful but the paper should be clearer that it is an analysis of several aspects of OPD.
  • Second point from above: I don’t think on-policy self distillation is the right setting for these experiments. All of the proposed changes would apply and work well without the self-component. In fact, due to reasons like self-distillation penalizing thought tokens and suppressing exploration, I believe that the paper would be much more informative (and would perform better too) in a bigger-teacher distillation setting.
  • Minor: In my own experiments, I have seen that even with 24-28K token budgets a lot of the gains from OPSD simply come from responses that used to be truncated, and now the model learned to output responses that fit in the token budget. I wonder how much of the (already small) gains come from this. The authors can remedy this concern by answering the question in the next section.
  • Minor: Qwen models have gained infamy for being able to fit spurious supervision signals. It would be useful to have at least one non-Qwen model in the comparisons. TeX is supported
Questions for authors
  • As mentioned in the weaknesses, I am interested in truncation-related metrics. Specifically, suppose you take the 100 problems with the largest gains in avg@32 accuracy, for the sake of argument (can be a different number). What was the truncation@32 rate for these problems before and after training with different methods with your token budgets?
  • If time permits, do you have (preliminary) results with a non-Qwen model? Even a small model is fine for this part.
  • I thought Qwen3-4B and 8B already get very high accuracies on MATH-500 and also on DeepScaleR and OpenThoughts, which are more suitable for earlier generations of models. Why not use harder datasets for training?
4 comments

W1: My biggest concern is perhaps with the framing of the paper and I have two points here. First of all, I think the paper is not really proposing a new method but performing a careful study of a few factors like which direction of KL divergence is better for learning. Indeed, such an analysis is helpful but the paper should be clearer that it is an analysis of several aspects of OPD.

Thank you for this insightful comment. We agree that the analysis of key design choices in OPD is an important contribution of our work. To better reflect this perspective, we have added supplementary experiments under the OPD setting and revised the Abstract, Introduction, and Conclusion to clarify the framing of the paper. Specifically, we now explicitly emphasize our analysis of two key aspects of distillation, namely KL divergence selection and distillation position. We also reposition OG-OPSD as a practical instantiation of the insights gained from these analyses, rather than emphasizing methodological novelty alone. These revisions aim to better highlight the analytical contributions of our work while preserving its original structure.

W2: Second point from above: I don’t think on-policy self distillation is the right setting for these experiments. All of the proposed changes would apply and work well without the self-component. In fact, due to reasons like self-distillation penalizing thought tokens and suppressing exploration, I believe that the paper would be much more informative (and would perform better too) in a bigger-teacher distillation setting.

Thank you very much for your valuable suggestion. We have added experiments in a conventional OPD setting. We use DeepMath as the training dataset, Qwen3-4B-Non-Thinking as the student model, and Qwen3-4B-Non-Thinking-RL-Math as the teacher model. Training and evaluation are conducted using the verl framework.

MethodAIME24AIME25HMMT25(Feb.)HMMT25(Nov.)Avg.
Teacher58.054.632.538.946.0
Student21.521.910.08.015.4
OPD60.757.933.238.947.6
OPD w/ Trunc62.156.734.138.647.8
OPD w/ OG-KL62.158.233.439.848.3
OG-OPD(Ours)63.057.534.640.048.7

OPD w/ Trunc denotes the variant that applies a fixed RKL objective to all trajectories together with the truncation strategy, while OPD w/ OG-KL uses dynamic divergence selection without any truncation. These results provide some indication that our method can also yield benefits in the OPD setting. We will incorporate these results and the corresponding discussion into the experimental section of the paper.

W3: Minor: In my own experiments, I have seen that even with 24-28K token budgets a lot of the gains from OPSD simply come from responses that used to be truncated, and now the model learned to output responses that fit in the token budget. I wonder how much of the (already small) gains come from this. The authors can remedy this concern by answering the question in the next section.

Q1: As mentioned in the weaknesses, I am interested in truncation-related metrics. Specifically, suppose you take the 100 problems with the largest gains in avg@32 accuracy, for the sake of argument (can be a different number). What was the truncation@32 rate for these problems before and after training with different methods with your token budgets?

Regarding this concern, we have computed the format_rate metric during evaluation, which measures the proportion of model-generated responses from which the answer can be successfully extracted. Since we set max_token to 38k during inference, the model has sufficient room to carry out its reasoning. We compare the Base model, vanilla OPSD, and our OG-OPSD, and observe that the format_rate remains consistently high across all three methods.

MethodAIME24AIME25HMMT25
Base99.4%96.6%97.7%
OPSD98.6%97.5%98.8%
OG-OPSD99.4%96.9%98.3%

We do not observe any apparent correlation between the performance improvements achieved by our method and changes in format_rate. Therefore, under our evaluation setting, we believe that the observed performance differences are largely independent of the model learning to generate responses that fit within the token limit. We hope this helps address your concern.

W4: Minor: Qwen models have gained infamy for being able to fit spurious supervision signals. It would be useful to have at least one non-Qwen model in the comparisons. TeX is supported

Q2: If time permits, do you have (preliminary) results with a non-Qwen model? Even a small model is fine for this part.

We agree that evaluating a non-Qwen model would strengthen our empirical analysis. We are currently conducting additional experiments in this direction. However, due to our limited computational resources, we plan to incorporate the corresponding results into the paper as they become available.

Q3: I thought Qwen3-4B and 8B already get very high accuracies on MATH-500 and also on DeepScaleR and OpenThoughts, which are more suitable for earlier generations of models. Why not use harder datasets for training?

We use OpenThoughts as our training dataset in accordance with the established training paradigm adopted by existing OPSD methods, such as OPSD and PW-OPSD. By contrast, MATH-500 and DeepScaleR are used exclusively in our observational analysis of teacher continuation accuracy, providing diversity across datasets. Neither dataset is used for training. In our newly added OPD experiments, we use mathematical samples from DeepMath with difficulty scores of 6 or higher for training.

Reviewer_do2ROpenReview ↗

Thank you for the rewriting and results

Thank you to the authors for the rewriting and results. I still have some doubts regarding how exactly these insights may be leveraged in a practical setting (where mixing reverse/forward KL etc. may be a bit hacky). Nonetheless, I think the rebuttal has done a great job in directly addressing my two main concerns, i.e. writing (framing) and my most pressing concerns about truncation. I would strongly urge the authors to also include Qwen results in the next version as promised.

Based on these points, I will raise my score to "3: weak accept."

Dear Reviewer do2R,

Thank you for your response and for confirming that your main concerns have been addressed. We appreciate your time and consideration.

Best,

Authors

Reviewer_6choOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OG-OPSD, which adapts the distillation strategy according to whether the final outcome of a trajectory is correct. Specifically, full-length forward KL (FKL) distillation is applied to correct trajectories, whereas reverse KL (RKL) distillation is applied only within a truncated prefix for incorrect trajectories. The truncation point for incorrect trajectories is determined by the last intersection between the cumulative teacher-entropy curves of correct and incorrect trajectories. Experiments across Qwen3 models of different scales and multimodal tasks show that the proposed method generally outperforms several OPSD baselines.

Critical strengths
  1. The method is simple and intuitive. It leverages existing verifier outcomes to jointly determine the KL-divergence direction and the distillation range, without requiring the training of an additional reward model.
  2. The experiments cover multiple model scales and tasks, and the ablation studies provide preliminary evidence supporting the effectiveness of both KL routing and the truncation of incorrect trajectories.
Critical weaknesses

Using the last intersection of the cumulative teacher-entropy curves to determine the truncation point is intuitively motivated. However, entropy does not directly indicate the correctness or reliability of the supervision, and the set of trajectories included in the statistics may vary across positions. It would therefore be helpful to provide further clarification or evidence regarding whether this criterion can consistently identify the boundary of reliable teacher supervision.

Questions for authors
  1. The choice of KL-divergence direction is determined by the final outcome of a trajectory. However, identical or similar prefixes may appear in both correct and incorrect trajectories due to differences in subsequent sampling, and may therefore receive FKL or RKL supervision, respectively. Could the authors clarify why the eventual outcome of a trajectory is an appropriate signal for determining the divergence type at an earlier prefix? In particular, could this retrospective routing mechanism introduce inconsistent local supervision for similar prefixes?
  2. Teacher entropy primarily reflects uncertainty in the output distribution. Could the authors provide further explanation or empirical justification for why the last intersection between the cumulative entropy curves of correct and incorrect trajectories serves as an appropriate cutoff for teacher reliability?
4 comments

We sincerely thank you for your questions and suggestions, and for recognizing our design of using existing verifier outcomes to determine the KL-divergence direction and the distillation range, which is the main contribution of our work. We address your concerns regarding outcome-guided KL routing and entropy-based truncation separately below.

W1: Entropy does not directly indicate the correctness or reliability of the supervision, and the set of trajectories included in the statistics may vary across positions. It would therefore be helpful to provide further clarification or evidence regarding whether this criterion can consistently identify the boundary of reliable teacher supervision.

Q2: Could the authors provide further explanation or empirical justification for why the last intersection between the cumulative entropy curves of correct and incorrect trajectories serves as an appropriate cutoff for teacher reliability?

Our goal is to identify a global, fixed prefix cutoff at the population level, without additional hyperparameter tuning, to reduce potentially noisy teacher supervision in the later portions of incorrect trajectories throughout training. We do not aim to precisely identify the point at which teacher guidance becomes unreliable for each individual failed trajectory.

We calculate the average accuracy of teacher continuations before and after the truncation point NtN_t determined by this approach:

ModelTCA Before NtN_tTCA After NtN_t
Qwen3-1.7B0.680.60
Qwen3-4B0.630.57
Qwen3-8B0.600.55

These results provide preliminary evidence that the intersection of the cumulative average teacher entropy curves serves as a population-level indicator of changes in teacher-guidance reliability. Specifically, teacher continuation accuracy is lower after the truncation point than before it. Through this simple approach, we provide a fixed-prefix strategy for selecting the truncation position, eliminating the need for hyperparameter search.

Based on our experimental observations, we hypothesize that, regardless of whether the sampled trajectory is correct or incorrect, the prefix corresponds to the initial stage of reasoning, during which the cumulative average teacher entropy remains statistically consistent. In later stages, however, statistically distinguishable patterns in cumulative average teacher entropy begin to emerge across reasoning trajectories leading to different correctness outcomes.

As you pointed out, teacher entropy primarily reflects uncertainty in the output distribution. The higher cumulative teacher entropy observed on incorrect trajectories beyond the intersection point suggests that, at the population level, the teacher is more uncertain when providing guidance on the later portions of incorrect trajectories than on correct trajectories. This pattern is also accompanied by a decline in teacher continuation accuracy. Based on this observation, we propose this simple prefix truncation strategy and integrate it with our outcome-guided approach to handling prefix unreliability. We further validate the effectiveness of this differentiated treatment through ablation studies (Figure 2 of the paper).

We repeatedly sampled different subsets of examples using the Qwen3-4B model to examine the cumulative average teacher entropy curves. The position of the final intersection point across these curves was 102.00±5.29102.00 \pm 5.29. Furthermore, the parameter sensitivity experiments (Figure 3 of the paper) also demonstrate that the model achieves good performance in the vicinity of the selected truncation position. Taken together, we think this provides a simple and practical way to determine the fixed truncation hyperparameter before training.

Q1: The choice of KL-divergence direction is determined by the final outcome of a trajectory. However, identical or similar prefixes may appear in both correct and incorrect trajectories due to differences in subsequent sampling, and may therefore receive FKL or RKL supervision, respectively. Could the authors clarify why the eventual outcome of a trajectory is an appropriate signal for determining the divergence type at an earlier prefix? In particular, could this retrospective routing mechanism introduce inconsistent local supervision for similar prefixes?

Thank you for raising this insightful question. In short, outcome-guided routing does not introduce contradictory local supervision, as FKL and RKL yield consistent update directions for the same token. The trajectory outcome serves as a Monte Carlo signal for selecting the form of teacher supervision, rather than as a deterministic correctness label for the prefix.

Optimization perspective. For a fixed prefix, let r(v)=pT(v)/pS(v)r(v)=p_T(v)/p_S(v). The implicit token-level advantages induced by FKL and RKL are:

A_{\mathrm{RKL}}(v)=\log r(v).$$ Since $\operatorname{sign}(r-1)=\operatorname{sign}(\log r)$, both divergences induce the same update direction for each token, differing primarily in the magnitude of their corresponding gradient weights. Thus, switching between FKL and RKL does not introduce opposing update signals. **Statistical perspective.** For a fixed prefix $s$, let $q(s)=P(R(y)=1\mid s)$ denote the probability of obtaining a correct outcome from a sampled continuation. The expected outcome-conditioned advantage is: $$\mathbb{E}[A(v)\mid s] =q(s)A_{\mathrm{FKL}}(v) +(1-q(s))A_{\mathrm{RKL}}(v).$$ Since both advantages share the same sign, their weighted expectation preserves the update direction. Thus, different sampling outcomes only change the relative contributions of the two weighting schemes, rather than reversing the expected update direction. Our method therefore uses trajectory outcomes to adaptively select the form of dense teacher supervision, without assigning contradictory correctness labels to the same prefix. We do not assume that trajectory-level outcomes precisely identify individual token correctness. Instead, they provide a coarse-grained but verifiable supervision signal. A prefix may naturally lead to either correct or incorrect continuations under stochastic sampling. Its final outcome is therefore a Monte Carlo signal from one sampled continuation, not a deterministic correctness classification of the prefix.

Dear Reviewer 6cho,

Thank you once again for your valuable comments on our work. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please let us know. If you are satisfied with our responses we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We appreciate your time and consideration.

Best,

Authors

Dear Reviewer 6cho,

We sincerely thank you for your careful evaluation and constructive feedback. As the discussion period is drawing to a close (only 3 hours left), we would greatly appreciate any further feedback on whether our responses and additional clarifications have adequately addressed your concerns. We sincerely hope they have helped clarify the points raised and look forward to hearing your thoughts. Thank you once again for your time and consideration!

Best,

Authors of Submission 31207

Reviewer_hVRWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OG-OPSD, which uses outcome correctness to control both the distillation objective and the positions receiving supervision. Correct student trajectories receive forward-KL supervision throughout, while incorrect trajectories receive reverse-KL supervision only within a prefix. A shared prefix cutoff is estimated before training from the last crossing of outcome-conditioned cumulative teacher-entropy curves. Experiments cover mathematical reasoning with three Qwen3 model sizes, multimodal reasoning, and coding retention after mathematical post-training.

Critical strengths
  1. The proposed intervention is simple and testable. Rather than introducing another learned weighting module, it uses correctness to choose the divergence and restrict supervision on failed trajectories. Figure 2 tests fixed divergences, reversed divergence assignment, truncation of all trajectories, and removal of truncation. These comparisons directly address the main design choices and make the empirical case more convincing.

  2. The improvements are consistent across the three mathematical benchmarks and model sizes. The coding-retention results are also useful: OG-OPSD reduces the degradation seen with vanilla OPSD, although it does not eliminate it. Together with the multimodal experiments, these results suggest that the method is useful beyond one model–dataset configuration.

Critical weaknesses
  1. The entropy-based cutoff is less well justified than the divergence-selection rule. A crossing between two population-average entropy curves does not necessarily identify where teacher guidance becomes unreliable on individual failed trajectories. The cutoff is calculated once before training, even though student-generated prefixes change during training. The sensitivity experiment is helpful, but it does not establish whether the entropy estimate is better than a simple fixed prefix budget. The no-crossing rule also sets the cutoff to zero, potentially removing all supervision on incorrect trajectories.

  2. The gains over the strongest baseline are modest, and their uncertainty is not reported. The average improvements over PW-OPSD are approximately 1.1, 1.8, and 1.0 points at the three model sizes. Each mathematical benchmark contains only 30 problems; sampling 12 responses per problem reduces decoding noise but does not provide evidence of stability across training runs. This matters because some individual differences are small.

Questions for authors
  1. How many training examples are used to estimate the entropy curves, and what cutoffs are obtained for each model and the multimodal setting? Does the crossing remain stable across sampled diagnostic sets or later training checkpoints?

  2. How does the entropy-derived cutoff compare with a fixed cutoff that retains a similar number of supervised tokens? What happens in practice when the curves do not cross?

  3. Can you report uncertainty across independent training runs, or at least paired problem-level uncertainty for the main comparisons with PW-OPSD?

5 comments

We sincerely thank you for your questions and suggestions, and for recognizing our use of trajectory outcomes to determine the KL-divergence direction and restrict the scope of supervision on incorrect trajectories. This design constitutes the main contribution of our work. Below, we address your concerns regarding the selection of the truncation position and the uncertainty of our results separately.

W1: A crossing between two population-average entropy curves does not necessarily identify where teacher guidance becomes unreliable on individual failed trajectories. The cutoff is calculated once before training, even though student-generated prefixes change during training. The sensitivity experiment is helpful, but it does not establish whether the entropy estimate is better than a simple fixed prefix budget. The no-crossing rule also sets the cutoff to zero, potentially removing all supervision on incorrect trajectories.

Q2: How does the entropy-derived cutoff compare with a fixed cutoff that retains a similar number of supervised tokens? What happens in practice when the curves do not cross?

Our goal is to identify a global, fixed prefix cutoff at the population level, without additional hyperparameter tuning, to reduce potentially noisy teacher supervision in the later portions of incorrect trajectories throughout training. We do not aim to precisely identify the point at which teacher guidance becomes unreliable for each individual failed trajectory.

We measure the average accuracy of teacher continuations before and after the resulting truncation point NtN_t:

ModelTCA Before$N_t$TCA After$N_t$
Qwen3-1.7B0.680.60
Qwen3-4B0.630.57
Qwen3-8B0.600.55

These results provide preliminary evidence that the intersection of the cumulative average teacher entropy curves serves as a population-level indicator of changes in teacher-guidance reliability. Specifically, teacher continuation accuracy is lower after the truncation point than before it. Through this simple approach, we provide a fixed-prefix strategy for selecting the truncation position, eliminating the need for hyperparameter search. We also acknowledge that, from a training perspective, this is fundamentally equivalent to simply fixing the prefix length. Importantly, our approach determines the fixed prefix length through a data-driven statistical criterion, rather than manually specifying it as a hyperparameter.

Furthermore, in our actual experiments, we did not observe any cases in which the cumulative average teacher entropy curves lacked an intersection across different models and samples. Based on our experimental observations, we hypothesize that, regardless of whether the sampled trajectory is correct or incorrect, the prefix corresponds to the initial stage of reasoning, during which the cumulative average teacher entropy remains statistically consistent. In later stages, however, the cumulative average teacher entropy begins to exhibit statistical differences across reasoning trajectories leading to different correctness outcomes. This phenomenon is correlated with a decline in teacher continuation accuracy. Based on this observation, we propose this simple prefix truncation strategy and integrate it with our outcome-guided approach to handling prefix unreliability. We further validate the effectiveness of this differentiated treatment through ablation studies (Figure 2 of the paper).

W2: The gains over the strongest baseline are modest, and their uncertainty is not reported...

Q3: Can you report uncertainty across independent training runs, or at least paired problem-level uncertainty for the main comparisons with PW-OPSD?

We independently trained and evaluated the models using three different random seeds, reporting the mean ±\pm std deviation, and compared OG-OPSD with PW-OPSD. The results are presented below:

MethodAIME24AIME25HMMT25
PW-OPSD76.20±0.5876.20 \pm 0.5867.78±1.2767.78 \pm 1.2743.33±1.2143.33 \pm 1.21
OG-OPSD77.67±0.2977.67 \pm 0.2969.57±0.5769.57 \pm 0.5746.23±0.9146.23 \pm 0.91

These results provide evidence that the performance improvements achieved by our method are reasonably stable across independent training runs. We hope this addresses your concern.

Q1: How many training examples are used to estimate the entropy curves, and what cutoffs are obtained for each model and the multimodal setting? Does the crossing remain stable across sampled diagnostic sets or later training checkpoints?

Before training, we use the student model to generate samples from the training set and repeatedly draw subsets of 500 samples to estimate the entropy curves. The resulting truncation thresholds are as follows:

ModelQwen3-1.7BQwen3-4BQwen3-8BQwen3-VL-2B
NtN_t80100108100

For Qwen3-4B, we repeatedly sample different subsets of examples to examine the cumulative average teacher entropy curves. The position of the last intersection point across these curves is 102.00±5.29102.00 \pm 5.29. Furthermore, the parameter sensitivity experiments (Figure 3 of the paper) also demonstrate that the model achieves good performance in the vicinity of the selected truncation position. Taken together, we think this provides a simple and practical way to determine the fixed truncation hyperparameter before training.

Dear Reviewer hVRW,

Thank you once again for your valuable comments on our work. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please let us know. If you are satisfied with our responses we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We appreciate your time and consideration.

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear AC,

We thank all reviewers for their valuable and constructive feedback. We would like to briefly summarize the key points of our rebuttal.

Reviewers acknowledged several strengths of our work:

  • Simple and Intuitive Design: (hVRW, 6cho). Reviewers recognized the simplicity of our approach and the value of outcome-guided KL selection and selective truncation.
  • Empirical Effectiveness: (hVRW, do2R). Reviewers acknowledged the consistent improvements across different model scales and tasks, as well as our comprehensive ablation studies.

We have carefully addressed the reviewers' concerns through point-by-point responses and additional experiments:

  • Extension to OPD (do2R, W1/W2): Following the reviewer's suggestion, we extended our approach to conventional OPD with an external teacher, demonstrating its effectiveness beyond self-distillation.
  • Robustness (hVRW, W2/Q3): We conducted three independent training runs for OG-OPSD and PW-OPSD, reporting mean ± std across three mathematical benchmarks.
  • Further Analysis (hVRW, 6cho, do2R): We provided additional evidence and explanations regarding entropy-based truncation, outcome-guided KL routing, and generation truncation.

We sincerely appreciate the reviewers' feedback and hope that our additional experiments and clarifications address their concerns and assist your assessment. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.