acceptodds
Under review as a conference paper at ICLR 2027

When to Follow the Teacher: Advantage-Based On-Policy Distillation

14 views

Abstract

Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student’s expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, On-Policy Distillation, Reinforcement Learning, Outcome Supervision, LLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7wb8OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors study on-policy distillation (OPD) setting. They argue that always following the teacher might not always be helpful for student performance. They therefore propose a strategy to essentially scale the student logits on teacher trajectories by the student advantage, so the responses which are helpful for the student should receive more learning signal than others. The method is then Advantage-based on-policy distillation with an additional importance-sampling correction (since the trajectories are coming from the student).

The empirical results suggest that overall their method seesm to be better than OPD and GRPO in most of the scenarios. (while sometimes it underperforms)

Critical strengths

Simple and well motivated method -- rescale the logits based on student advantages, therefore emphasizing the responses which are more helpful for the student.

Overall, empirical performance suggests that the method is better than baselines.

Critical weaknesses

I do not fully understand why the method sometimes is worsse than OPD or than GRPO. Could the authors explain why it happens?

When I look at figure 2, the X-axis is not the same for all the methods. Why is it the case? What happens to the method if we run it for longer? Will it collapse? Figure 2 middle suggests that the performance of blue method starts to go down. Can you show the same figure 2 but for 1000 iterations?

Why is it the case that GRPO doesnt learn much on AIME and OPD doesnt learn much on TACO?

The authors use different sampling settings for their method and for and for OPD / OPSD. Are these parameters only for training-time trajectories or is it the same for evaluation? Would it be possible to report performance under matched hyperparameters settings for both, the method and the baselines? The fact that the parameters are different makes it a bit hard to judge where the advantage comes from.

Also, you use different G for GRPO and OPD/AOPD. Would it be possible to report results under matched settings? Does your method improves / worsens with increased G?

Questions for authors

See above.

3 comments

We thank the reviewer for the positive and constructive feedback to our paper. Below we address their questions:

I do not fully understand why the method sometimes is worse than OPD or than GRPO. Could the authors explain why it happens?

As figure 2 (left/right) and figure 3 show, AOPD achieves higher evaluation rewards than OPD and GRPO in most of our experiments. However, maximizing the reward does not always transfer to the best benchmark results since benchmarks evaluate different capabilities. For instance, the TACO dataset consists of python codes only, but Multipl-E evaluates the models on other languages with.

Nevertheless, the few exceptions are relatively small. For example, OPD achieves 33.3 versus 32.5 on MultiPL-E and 16.0 versus 15.8 on HMMT, while AOPD achieves better results on AMC and HumanEval.

X-axis is not the same for all the methods. Why is it the case?

The difference in the x-axis is between experimental settings (coding and math), not between methods within the same comparison. For external-teacher distillation, we train all methods for 450 iterations on math and 350 iterations on coding. For self-distillation, we use 400 iterations for all methods. These are arbitrary fixed training budgets chosen for each setting, and all methods within each comparison use the same number of iterations.

What happens to the method if we run it for longer?

In order to assure the reviewer that our algorithm is stable for longer trainings, we ran our algorithm for 750 steps. This is shown in this figure: https://ibb.co/5WWyjwCD. The evaluation reward remains stable and it does not collapse.

Regarding the drop in the last iteration on AIME, this is because of oscillations during training rather than a sustained decline. The performance on AIME remains high over the iterations as the attached figure above shows. Note that other methods such as OPD oscillate as well.

Why is it the case that GRPO doesnt learn much on AIME and OPD doesnt learn much on TACO?

GRPO optimizes the training reward using only student rollouts and without external teacher signals. On harder problems and harder benchmarks such as AIME, this limits the benefit of GRPO since the student cannot discover the solutions based on its own exploration. In contrast, OPD provides dense supervision from a stronger teacher regardless of the reward. This can help transfer capabilities beyond what the student can discover.

OPD imitates the teacher regardless of whether its actions improve the student's reward. This can explain why OPD makes little progress on TACO, while it still improves on other benchmarks. For example, OPD outperforms GRPO on HumanEval+ despite achieving lower TACO evaluation reward. AOPD combines these benefits by using teacher supervision while weighting it according to its usefulness to the student.

Are these parameters only for training-time trajectories or is it the same for evaluation?

We thank the reviewer for pointing this out. These parameters are only at training time. We evaluate all the models at temp = 0.6, top_p = 0.95 and top_k = 20, similar to [1, 3]. We added this to our explanation at the end of the hyperparameters paragraph in the updated file.

We also ran OPD under the same training setting as ours ( temp = 0.8, top_p = 1). As the following figure shows they achieve similar performance: https://ibb.co/TDjNzgKS.

Would it be possible to report results under matched settings? Does your method improves / worsens with increased G?

We report the effect of increasing group size in Appendix A, where increasing G from 4 to 8 provides only negligible improvements and does not consistently improve benchmark performance. Similar observations for OPD have been reported in previous work [4]. Therefore, we use G=4 for AOPD and OPD in our main experiments to reduce computational cost, while GRPO uses G=8 following its standard setting. All other training settings are kept consistent within each comparison.

[1] Zhao et al, “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models”.

[2] Hubotter et al, “Reinforcement Learning via Self-Distillation”.

[3] Jin et al, “Entropy-Aware On-Policy Distillation of Language Models”.

[4] Hou et al, “Uni-opd: Unifying on-policy distillation with a dual-perspective recipe.”

Reviewer_7wb8OpenReview ↗

Thank you for the rebuttal

Thank you for the rebuttal.

You have addressed my concerns. I'm maintaining my score, but increasing confidence.

We thank the reviewer for their positive assessment of our paper. We are glad that our additional experiments and clarifications addressed their concerns. We will incorporate these points in the revised version.

Reviewer_k3DuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Advantage-Based On-Policy Distillation (AOPD), which aims to address a limitation of standard on-policy distillation: teacher-preferred actions may not always be beneficial when the remaining trajectory is generated by the student. Instead of uniformly matching the teacher, AOPD weights teacher actions by their advantage under the current student policy. The main technical contribution is a two-term estimator that recovers the proposed advantage-weighted objective using student rollouts and terminal rewards, without additional counterfactual rollouts or a learned value function. The paper also provides a theoretical analysis showing reward improvement under exact KL-proximal updates in policy space, and evaluates the method on math, coding, and medical question-answering tasks under both external-teacher and self-distillation settings.

Critical strengths
  1. A clear and practically relevant motivation. I like the focus on whether following a teacher actually helps the current student. In OPD, a teacher can suggest a step that it could successfully continue from, while the student may not be able to. Framing this in terms of the student's own advantage makes the intended supervision more precise than simply matching teacher preferences.
  2. An appealing estimator with a clearly scoped theoretical analysis. The two-term estimator is a useful way to obtain student-advantage-weighted supervision from ordinary student rollouts and terminal rewards, without extra counterfactual continuations or a learned critic. I also appreciate that the paper analyzes exact KL-proximal updates rather than claiming a guarantee for SGD. The results across external- teacher and self-distillation settings are encouraging, even though the comparisons could be stronger.
Critical weaknesses
  1. The contribution of the advantage-centering term is not well validated. I think the two-term estimator in Eq. (3.6) is one of the more interesting parts of this paper. In particular, the second term is important for recovering the student advantage rather than simply using teacher-weighted returns. However, I couldn't find an ablation that directly tests its contribution. Appendix A mainly studies the group size and importanceweight clipping. I'm not sure how much of the improvement actually comes from this correction term, as opposed to using rewardweighted teacher supervision in general. For example, what if we simply remove the second term while keeping the same importance weights and rewards? I think this would be a useful ablation, especially since the two-term construction is one of the main technical contributions.
  2. The comparisons with closely related distillation methods are insufficient. The paper compares against OPD, GRPO, and KDRL, which is a reasonable starting point. However, I think RLAD(2026) [1] is an important missing baseline. RLAD also considers the conflict between teacher guidance and reward optimization, and uses an advantage-aware, clipped objective to incorporate teacher information. I'd like to see how AOPD compares with this method under similar settings. I'm also a bit confused by the discussion around Eq. (5.4), where RLAD is cited as an example of combining RL with FKL. As far as I understand, its main TRRD objective is not simply an additive RL + FKL loss. The authors should clarify this distinction, since the counterexample in Lemma 5.4 does not directly apply to TRRD. Another concern is the self-distillation setting. Table 2 compares AOPD with GRPO and OPSD (2026) [2] separately, but not with a simple combination of the two. Since AOPD uses both rewards and privileged teacher supervision, I think such a baseline would help determine whether the gains come from the proposed objective or just from combining these two training signals.
  3. I'm not fully convinced about the novelty of the overall formulation. The motivation is interesting, but several ingredients are closely related to existing ideas. For example, advantageweighted policy learning has been studied in AWR (2019) [3], and LOLS (2015) [4] also considers improving upon a reference policy by evaluating actions w.r.t. the learner. In fact, if we set T = S in Eq. (3.5), the objective reduces to an advantage-weighted policy-gradient surrogate at the onpolicy expansion point. Also, the reward-improvement argument in Sec. 5 seems to extend to general onnegative action weights, rather than relying on a specific property of the teacher. To me, the main novelty is therefore more likely to be the particular teacher-weighted objective and its two-term estimator, rather than the general idea of using student advantage to decide which actions to reinforce. I'd like the authors to make this distinction clearer and explain how their formulation differs from these earlier approaches.

[1] Reinforcement-aware Knowledge Distillation for LLM Reasoning. ICML 2026.

[2] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

[3] Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv preprint, 2019.

[4] Learning to Search Better than Your Teacher. ICML 2015.

Questions for authors
  1. Regarding the novelty of AOPD: Could the authors discuss the relationship with AWR (2019) and LOLS (2015) in more detail? In particular, what is the key difference between AOPD and existing advantage-weighted policy-learning objectives? I'd also be interested in the authors' interpretation of the T = S special case.
  2. Regarding the two-term estimator: Could the authors provide an ablation without the second term in Eq. (3.8), while keeping the other settings unchanged? I think this would help show whether the advantage-centering correction is important in practice, beyond the theoretical derivation.
  3. Regarding the baselines: Could the authors compare AOPD with RLAD (2026), ideally using the same student, teacher, and comparable training budgets? Also, in the self-distillation setting, how does AOPD compare with a simple GRPO + OPSD baseline using the same privileged teacher? I think these results would help me better assess the empirical contribution of the proposed method.
4 comments

We thank the reviewer for their thoughtful comments on our paper. Below we have addressed all of their concerns and the conducted the requested experiments:

The contribution of the advantage-centering term is not well validated. What if we simply remove the second term while keeping the same importance weights and rewards?

We ran the ablation that the reviewer suggested and found that keeping only the first term hurts the performance of the student, as shown in the following figure: https://ibb.co/GQcwLKXw.

Note that only keeping the first term means applying OPD (with importance sampling rather than full distribution) on only correct trajectories if the rewards are not centered. Recent work [1, 2] has directly compared correct-only versus incorrect-only training, and shows that incorrect-only is more effective. The intuition behind this is that the model needs more guidance at the wrong trajectories. The experiment above showcases the same thing: with centered rewards in our algorithm, the first term reinforces teacher actions on correct trajectories and suppresses those on the incorrect ones.

Why the second term avoids this: The second term complements the first term. After failure, it encourages alternative teacher actions over the current action. After success, it discourages alternative teacher actions since the current one works already. In expectation, the two terms compare the value of each teacher action with the student’s own expected reward.

I'd like to see how AOPD compares with RLAD under similar settings.

We thank the reviewer for mentioning this method. We ran RLAD using the setting mentioned in the [3] on both coding and math. We have added these comparisons in table 1 and Figure 6 in the updated file. RLAD has similar performance as AOPD on AIME, but lower on AMC and OpenThoughts. Moreover, on coding and TACO, AOPD achieves significantly better reward and outperform RLAD on HumanEval+ by 6 points. We think the reason that RLAD cannot learn more in coding is that OPD underperforms in this setting as well, and RLAD cannot resolve the teacher’s harmful signals. The following figure shows the comparison plots: https://ibb.co/bjntLJDF.

RLAD is cited as an example of combining RL with FKL. TRRD objective is not simply an additive RL + FKL loss. The authors should clarify this distinction, since the counterexample in Lemma 5.4 does not directly apply to TRRD.

Our reference to RLAD was about its interpretation of TRRD as reward optimization with regularization toward both the teacher and the previous student (Section 4.1, end of page 4). In the updated file (lines 475-477), we clarified that Equation (5.4) analyzes the additive RL + FKL, and that we investigate RLAD separately.

We now have shown in Appendix B.3 that the actual RLAD objective with clipping cannot avoid the counterexample either. The student reward decreases even though the teacher is stronger. This is while AOPD increases the reward with guarantee.

Table 2 compares AOPD with GRPO and OPSD (2026) [2] separately, but not with a simple combination of the two.

We appreciate the suggestion to compare with the hybrid version. We implemented this via Lhybrid=LGRPO+λLOPSDL_\text{hybrid} = L_\text{GRPO} + \lambda L_\text{OPSD}, selecting λ\lambda to balance the gradient magnitudes of both terms.

For Qwen3-8B, we run this with λ=0.5\lambda = 0.5. The following figure shows that, compared to GRPO and OPSD, this indeed results in improvements across both the evaluation reward and AIME accuracy, but it still remains significantly below AOPD: https://ibb.co/Q3zQPzqV.

Then, we repeated this for Qwen3-1.7B, with λ=0.5\lambda = 0.5 and λ=1.0\lambda = 1.0, but it does not lead to significant gains: https://ibb.co/jkPkBwfX. Similar to the RLAD coding setting, we attribute this to the fact that OPSD does not learn effectively in this setting either (Table 2), and the hybrid version fails to extract the useful parts of the teacher signal.

We have updated Table 2 in the updated file and added “GRPO + OPSD”.

several ingredients are closely related to existing ideas. For example, advantageweighted policy learning has been studied in AWR and LOLS also considers improving upon a reference policy by evaluating actions.

We agree that the existing literature already uses advantages for weighting (AWR) or evaluating actions under the learner (LOLS). Advantage function is one of the most fundamental concepts in RL and is used across many papers.

Our contribution is specific to distillation, where teacher actions are weighted by their advantage under the current student. Part of the contribution is choosing a formulation that admits an unbiased estimator from the existing student rollouts. For instance, exponential weighting, as in AWR, would not admit this same estimator and we have to run the intervention rollouts.

AuthorsOpenReview ↗

Rebuttal (part 2)

if we set T = S, the objective reduces to an advantage-weighted policy-gradient surrogate. Also, the reward-improvement argument does not rely on a specific property of the teacher.

We agree that setting (T=S) recovers the standard policy-gradient. This is a special case of our formulation. However, our contribution concerns distillation from a different teacher: the theory establishes the contrast with ordinary OPD under the same proximal framework, where imitating even a stronger teacher can reduce student reward.

It is correct that we remain agnostic to the properties of the teacher. But view this as the strength of our paper since the focus must be placed on the student. The relevant criterion is whether a teacher action helps the current student. For example, if the signals of the teacher are bad everywhere, the student should not follow it at all. The guarantee therefore does not require teacher guidance to be beneficial at every decision.

[1] Lin et al, “ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation”.

[2] Zheng et al, “SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting”.

[3] Zhang et al, “Reinforcement-aware Knowledge Distillation for LLM Reasoning”

We thank the reviewer for their thoughtful comments. To address these points, we have:

  • Conducted the ablation on AOPD’s terms and explained their importance.
  • Compared AOPD with RLAD and GRPO + OPSD. We updated the tables regarding these experiments.
  • We extended the counterexample to RLAD’s objective, and explained the connection of our advantage-based objective with prior work.


Please let us know if any concerns remain unaddressed before the deadline. If these updates resolve your concerns, we appreciate your reconsideration of the score.

Thank you for the response. My concerns have been adequately addressed, and I'm willing to raise my rating to 3.

Reviewer_1sGzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AOPD, which weights distillation by teacher actions' advantages under the current student. Student trajectories and terminal rewards estimate the objective without additional counterfactual continuations. Experiments cover math, coding, and medical question answering with external teachers and self-distillation.

Critical strengths
  1. Distinguishing teacher preference from actual student benefit is well motivated.
  2. The estimator has technical merit and uses existing student rollouts.
  3. Multiple domains and both teacher settings provide relatively broad empirical coverage.
Critical weaknesses
  1. The closest comparison is missing. Section 1 cites RLAD/TRRD), which studies advantage-aware selective distillation, but the baseline list and Table 1 omit it. Comparisons with standard OPD and KDRL are insufficient to establish the additional contribution. A mathematical distinction and direct evaluation are needed.
  2. I have concerns about attributing gains to the objective. Section 4, “Hyperparameters”, uses temperature 0.8 without pruning for AOPD, versus 1.1, top-p 0.95, and top-k 20 for OPD/OPSD. A common-sampling control is needed. Equation (3.6)'s denominator must also be clarified: matching the actual temperature-adjusted behavior probability is necessary for the stated unbiasedness.
  3. Small gains lack uncertainty reporting. Table 1 gives AIME24 scores of 35.0 versus 32.6, a 2.4-point gap, without error estimates; Figure 2 does not define its shaded bands. AOPD also scores 32.5 on MultiPL-E versus OPD's 33.3. Please specify independent training seeds, evaluation-sampling counts, and the scope of the improvements.
Questions for authors
  1. How does AOPD compare with RLAD/TRRD and OPD under common sampling settings?
  2. How do importance weights account for temperature?
  3. Are the main gains stable across independent training seeds?
4 comments

We appreciate the reviewer’s constructive comments on our paper. Below we address all of their points:

Comparison to RLAD

We thank the reviewer for pointing out this missing comparison. To address this, we ran the RLAD experiments for Qwen3-1.7B on both math and coding. We did this by following every detail mentioned in their paper, and evaluating under the same setting as AOPD (we explain below). On math, RLAD achieves similar performance as AOPD on AIME but lower on OpenThoughts and AMC. On coding, however, AOPD obtains significantly higher evaluation reward and better HumanEval performance. We have updated Table 1 regarding these new results and included the training plots for RLAD in Figure 6 in the updated file. This can also be seen in the following: https://ibb.co/bjntLJDF.

RLAD studies advantage-aware selective distillation. A mathematical distinction is needed.

We understand that RLAD also uses the outcome rewards for selective distillation and more effective utilization of the teacher’s signal. However, the advantage function plays an action-wise role in our case as opposed to trajectory-wise. I.e., RLAD/TRRD weights a teacher-anchored policy ratio using a group-relative trajectory advantage, shared across the steps of each sampled response. This cannot distinguish what particular teacher actions within a trajectory are beneficial for the student and what is harmful. In contrast, AOPD weights teacher actions by their student advantage, AS(h,a)=QS(h,a)−VS(h)A^S(h,a)=Q^S(h,a)-V^S(h). This measures whether taking a particular action and then continuing with the student improves expected reward. Our estimator targets this action-conditioned objective using student rollouts. We will add this comparison to the related work.

Moreover, as we have shown in the updated version (lines 485-488 in the updated file), RLAD’s clipped objective does not resolve the counter-example and the objective can reduce expected student reward even when the teacher is stronger overall.

Hyperparameters”, uses temperature 0.8 without pruning for AOPD, versus 1.1, top-p 0.95, and top-k 20 for OPD/OPSD. A common-sampling control is needed.

We evaluated all of the models with T = 0.6, top-p = 0.95 and top-k = 20. We did not mention this in the previous version, but have now added this to the end of the hyperparameters paragraph. The mentioned hyperparameters above are used for training. Furthermore, we also tested the OPD experiment with the same hyperparameters as AOPD. As the following figure shows, we did not notice any difference: https://ibb.co/TDjNzgKS.

Equation (3.6)'s denominator must also be clarified: matching the actual temperature-adjusted behavior probability is necessary for the stated unbiasedness.

In Equation (3.6), S denotes the same student distribution (with no gradient) used to generate the rollouts, including the sampling temperature. I.e., to compute the loss, we divide both of the student logits and the teacher logits by the same temperature :

student_logits = student_logits / temperature
teacher_logits = teacher_logits / temperature

Small gains lack uncertainty reporting. Are the main gains stable across independent training seeds?

Figure 2 and Table 1 results are averaged over three independent training seeds (except for the new RLAD was done only once). Then, the shades show the standard deviation across these seeds at each iteration. We have added this to the updated file.

The results for math in Table1 are measured by average@8 (at temperature = 0.6, as we explained above). For AIME24, this gives 240 answers across its 30 problems. We report average@64 for HumanEval (the rollouts are significantly shorter) and average pass@1 (temperature=0) for Multipl-E with 1116 samples.

We agree that the 2.4-point AIME24 improvement should be presented alongside its uncertainty. We have shown this in Figure 2 already where the accuracies for AOPD and OPD overlap. Nevertheless, other benchmarks have more samples. For instance, std~2% for AMC and the gains are more meaningful. For Multipl-E, std~1% and OPD is slightly better than AOPD. We have these details in the updated version.

We wanted to thank you again for your feedback. To address your points, we have:

  • Conducted the requested RLAD experiments and clarified the mathematical distinction between the methods.

  • Clarified that all methods use identical evaluation settings and provided additional OPD results under matched training settings.

  • Clarified that our original experiments use three independent training seeds and that the reported math results are average@8.

Please let us know if any concerns remain unresolved so that we can address them before the deadline. If these updates resolve your concerns, we highly appreciate your reconsideration of the score.

Thank you for your detailed reply. Most of my concerns have been addressed, therefore, I raise my score to 3.

We sincerely thank the reviewer for increasing their score. We are glad that our updates and responses addressed their concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals

We thank the reviewers for their constructive feedback. We updated the paper and conducted the additional experiments to address the concerns. More specifically:

Comparison to RLAD: We implemented RLAD as a suggested baseline by reviewers 1sGz and k3Du, and compared it with AOPD on both math and coding. We showed that our method achieves better results on math and substantially better on coding. We also clarified the mathematical distinction between the methods, and clarified that the two state counter-example still holds for RLAD as well.

Hyperparameter choices: We clarified to reviewers 1sGz and 7wb8 that all methods use the same evaluation settings, and that the differences are only for training. We also ran OPD with the same training generation parameters as AOPD and found that its performance remains similar to the original setting.

Uncertainty and evaluation: We clarified that our main results are averaged over three independent training seeds. Math benchmarks use average@8 and HumanEval uses average@64.

GRPO + OPSD comparison: We implemented the requested hybrid baseline by reviewer k3Du for both Qwen3-1.7B and Qwen3-8B. While the hybrid improves over both GRPO and OPSD in some settings, AOPD still outperforms the rest across math benchmarks.

Ablation of the second term: We conducted the requested ablation by reviewer k3Du. We removed the second term from AOPD, and showed that keeping only the first term hurts performance. We also explained why both terms are necessary to distinguish useful teacher actions from harmful ones. We also explained how this aligns with studies in previous work.

We hope these points address the reviewers’ concerns. We would highly appreciate their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.