acceptodds
Under review as a conference paper at ICLR 2027

Selective Feedback Attenuation for Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation consolidates specialized capabilities into a single student by leveraging teacher feedback on student-generated responses. However, strictly matching teacher preferences can penalize valid alternative solutions, creating a fundamental misalignment between task correctness and imitation signals. To address this, we propose Selective Attenuation for Multi-Teacher On-Policy Distillation (SA-MOPD). SA-MOPD uses verifier outcomes to attenuate negative feedback on successful student responses, while preserving positive feedback and standard supervision elsewhere. The rule modifies feedback coefficients within the existing distillation update. Our reverse-KL analysis identifies a mechanism by which disagreement among successful solutions can reduce student success despite a stronger teacher. A local analysis further characterizes when feedback selection improves task performance at a fixed attenuation budget. Our experiments demonstrate the value of selective supervision retention for capability integration: SA-MOPD raises recovery from Open-MOPD's 83.4% to 96.4% across mathematical reasoning, code generation, and instruction following. These gains extend to agentic tasks, where the single student surpasses the routed teacher ensemble in aggregate performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-teacher distillation, On-policy distillation, Verification-guided learning, Capability integration, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7iWwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies capability integration in multi-teacher on-policy distillation (MOPD). In MOPD, student rollouts are routed to domain-specialist teachers, which provide token-level signed feedback. The authors point out that this feedback is not conditioned on task outcomes: even when the student has produced a verified-correct response, the teacher may still assign negative feedback, and this negative feedback may reflect an alternative valid solution or a teacher error rather than a genuine student mistake. Based on this observation, the paper proposes SA-MOPD: when a verifier confirms student success (gi=vici=1g_i = v_i c_i = 1), negative feedback is multiplied by a retention factor λu\lambda_u that decreases linearly from 1 to 0; positive feedback and full supervision on other responses remain unchanged. The paper provides two propositions: Proposition 1 shows that ideal reverse-KL distillation can reduce student success even with a stronger teacher when within-outcome matching dominates; Proposition 2 shows that, at equal attenuation mass, selecting where to release feedback based on verified outcomes yields a first-order task benefit over uniform attenuation. Experiments cover base capabilities (SmolLM3-3B on math, code, and instruction following) and agentic tasks (Qwen-3.6-27B on BFCL v4, TAU2, Claw-Eval, and SkillsBench), and include Global matched / Domain matched controls that match the amount of attenuated feedback mass.

Critical strengths
  1. The motivation is clear and practically relevant: teacher disagreement is not the same as student error, especially on tasks with multiple valid solutions. Figure 1 and Figure 3 convey this mechanism intuitively.
  2. The method is simple and low-cost: it only modifies the negative-feedback coefficients in the existing learner, without requiring an additional reward objective, teacher rollouts, or a critic.
  3. The theoretical analysis is targeted. Proposition 1 uses the KL chain rule to separate outcome probabilities from within-outcome matching, and gives conditions under which teacher matching can oppose task success; Proposition 2 separates “where to attenuate” from “how much to attenuate,” directly motivating the Global matched / Domain matched experimental design.
  4. The control experiments are well designed. Comparing SA-MOPD with Global matched and Domain matched at the same attenuation mass is more convincing than comparing only with MOPD. The single-update analysis in Table 4 and Figure 4 is also relatively clean.
  5. The experiments cover both base and agentic settings and report variance across three training seeds.
Critical weaknesses
  1. The method itself is very close to a heuristic rule.
    The core change in SA-MOPD is essentially one line: multiply negative feedback by λu\lambda_u when verification succeeds. This is simple and effective, but it also means the contribution is more “a reasonable engineering rule plus a local analysis of why it might work” than a new learning objective or training framework. The conclusion of Proposition 2—selective attenuation is better than uniform attenuation when the gate aligns with release benefit—is almost tautological at an intuitive level; its value lies mainly in formalizing the experimental design rather than providing a non-trivial algorithmic insight.

  2. The gap between the theoretical analysis and actual training is not sufficiently discussed.
    Propositions 1 and 2 are local, first-order analyses under fixed rollouts and detached coefficients. In actual training, λu\lambda_u follows a schedule over training steps, feedback is refreshed, and the rollout distribution changes with the policy. The paper does not explain how far these local conclusions extend to the full training process. Proposition 2 depends on the sign of ed,i=hd⊤BUi/Mie_{d,i} = h_d^\top B U_i / M_i, but this quantity is not explicitly estimated during training; instead, “verified success” is used as a proxy. When this proxy holds and when it fails is not discussed enough.

  3. The scale of experiments and statistical evidence are limited.
    The base experiments use only one student model (SmolLM3-3B), and the agent experiments use only one student model (Qwen-3.6-27B). Table 1 reports only means over three seeds, with variance deferred to Table 6 in the appendix; Table 2 does not report seed variance. Some gaps are small: in Table 3, SA-MOPD and RG-OPD† differ by only 0.54 points in Total (32.11 vs. 31.57), and in Table 2, SA-MOPD and Route Teacher differ by only 0.51 points in Total. Given that the agent setting trains a single model, more evidence is needed to establish that these gaps are stable. In addition, the “Recovery > 100%” (109.7%) claim should be stated more cautiously: it shows that the consolidated student exceeds routed specialists in aggregate, but the student still trails them on individual benchmarks (TAU2 84.36 vs. 84.96; Claw-Eval 73.72 vs. 74.03). The main text could be more restrained in emphasizing “exceeding the routed reference.”

  4. The scope of verifier dependence and gate semantics needs clearer boundaries.
    SA-MOPD’s effect depends directly on whether the verifier can reliably judge “student success.” Math and code have relatively clear verifiers, IF uses rule-based checkers, and agent tasks use task-level success. However, the paper does not systematically discuss verifier noise, partial success, or cases where a multi-turn task succeeds overall but contains intermediate errors. Appendix B.2 states that the gate uses the full interaction outcome, but partial progress sets ci=0c_i = 0, meaning many “near-success” responses still receive full negative feedback. The rationale for this design choice needs more discussion. In addition, the teachers in the agent experiments are themselves RL specialists; whether the verifier is coupled with the teacher training signal is not discussed.

Questions for authors
  1. Propositions 1 and 2 are local first-order analyses. Can the authors provide more direct evidence that these local conclusions still hold during full training, rather than only in single-update or intermediate-checkpoint settings?
  2. Under what conditions does “verified success” fail as a proxy for ed,i>0e_{d,i} > 0? For example, when successful responses contain many low-quality tokens or when the verifier itself is noisy, is SA-MOPD still better than Domain matched?
  3. The agent setting trains only one student model, and the gap to Route Teacher in Table 2 is only 0.51 points. Can the authors add multiple training seeds or at least report variance to show that this result is not a single-run fluctuation?
  4. How was the linear schedule of λu\lambda_u from 1 to 0 chosen in the base experiments? Did the authors try other schedules (e.g., fixed λu\lambda_u, or stepwise decay over training), and how sensitive are the results?
  5. How does the verifier handle partial success and intermediate errors in multi-turn agent tasks? If the gate opens only on final task success, does it miss “eventually successful but flawed” responses and thereby limit the method’s applicability?
2 comments

We sincerely thank the reviewer for the valuable feedback.

W1. The method itself is very close to a heuristic rule.

R. The theory explains when teacher imitation can harm task success and why selecting feedback can help. Proposition 1 gives an explicit condition under which matching a stronger teacher reduces student success because preferences among valid solutions differ.

The two-route example in Appendix A.3 illustrates the distinction. The student succeeds on two equally likely reasoning routes with probabilities 80% and 20%. Taking a teacher that succeeds with 60% probability on each route makes it stronger overall, but weaker on the student's better route. Unconditional attenuation weakens useful correction on the student's weaker route as well. Correctness-based gating more often relaxes imitation on the stronger route while retaining correction on the weaker one. In this construction, a single update yields unconditional attenuation < MOPD < correctness-gated attenuation in task success.

Proposition 2 then quantifies the local value of selection at equal attenuation mass, separating where feedback is removed from how much is removed. This motivates the Global-matched and Domain-matched controls; on the TACO code-generation diagnostic set, SA achieves 2.93 percentage points higher accuracy than Domain matched after a single update. The analysis thus supplies a concrete failure mechanism and an experimentally testable principle for selecting feedback.

W2. The gap between the theoretical analysis and actual training is not sufficiently discussed.

R. We connect the local analysis to training through an actual-Adam audit and end-to-end evaluation (Q1), and identify how verifier quality affects feedback selection (Q2).

W3. The scale of experiments and statistical evidence are limited.

R. The base-capability results report three independent SA-MOPD training runs. Due to limited computational resources, the Agent experiment uses a single training run.

W4. The scope of verifier dependence and gate semantics needs clearer boundaries.

R. Response Q2 addresses verifier errors and successful-but-flawed responses; Response Q5 states how the current gate treats partial progress and intermediate errors. The gate uses task-level success; its relationship to the specialists’ RL reward specification is unspecified in the reported configuration.

Q1. Can you provide direct evidence that the local first-order conclusions hold during full training, beyond single-update or intermediate-checkpoint settings?

R. Our analysis describes how selective attenuation affects task performance at the student's current state. In actual training, that state changes as the student learns and generates new responses. The analysis therefore explains the effect of an individual update, while training experiments assess how the method performs as the policy and feedback evolve.

We check the connection to actual optimization in Appendix G.5. For SA versus Domain matched, the local approximation and measured Adam updates produce closely aligned task projections: correlations are 0.9936–0.99999, with 96.04%–100% sign agreement on prompts with nonzero task-gradient estimates. This supports the optimizer approximation at the audited state.

The full-training results complement this check. Across three independent base-capability training runs, SA-MOPD achieves a mean Total score of 32.11%, exceeding MOPD by 0.87 pp, with improvements in all three domains. The Adam audit validates the local optimizer approximation at the audited state, while the full-training results establish the method's aggregate performance. Isolating selection's contribution throughout training calls for a full-training attenuation-matched comparison.

Q2. When does verified success fail as a proxy for positive release benefit? With low-quality tokens or verifier noise, is SA-MOPD still better than Domain matched?

R. Verified success can fail as a proxy when the negative feedback on a successful response still provides useful correction—for example, correcting flawed intermediate reasoning despite a correct final answer. A false-positive verifier judgment can similarly activate attenuation on a response that needs correction. A false negative instead leaves full baseline supervision active on that response.

Proposition 2 identifies the governing condition: selection is beneficial when the gate favors feedback with greater release benefit. Low-quality content or verifier errors can weaken or reverse this alignment. Our reported advantage over Domain matched is measured with the original task verifiers; performance under these perturbations is an empirical question.

Q4. How was the linear 1→0 schedule chosen, and how sensitive is performance to alternative schedules?

R. The decreasing schedule retains full teacher supervision initially, then gradually relaxes imitation constraints on verified successes while preserving supervision on failures. We compare three schedules for retaining negative feedback on verified successes: decreasing retention from all to none (1→0), increasing it from none to all (0→1), or retaining half throughout training (constant 0.5). All runs share the same initialization and training budget. Our main 1→0 schedule achieves the best Math, Code and Total scores, while 0→1 performs best on IF.

The MOPD row reports absolute scores (%); Δ denotes the change from MOPD in percentage points (pp).

MethodMathIFCodeTotal
MOPD (%)22.4249.5821.7331.24
SA, constant 0.5 (Δ)+0.70-0.67-0.02+0.01
SA, 0→1 (Δ)+0.73+0.67-0.05+0.45
SA, 1→0 (Δ)+1.75+0.11+0.75+0.87

Q5. How does the verifier handle partial success and intermediate errors in multi-turn Agent tasks? What happens to eventually successful but flawed responses?

R. The current Agent gate evaluates the final task outcome. Partial progress that falls short of the success criterion leaves attenuation inactive, preserving the full corrective signal. An unavailable judgment also leaves the gate inactive. If an interaction succeeds despite a recoverable intermediate error, the final success judgment activates response-level attenuation. Positive feedback remains unchanged.

Tool-call outcomes, intermediate verification and LLM judges could support finer-grained step assessments. Our current focus is using task-level success to guide feedback selection during specialist integration.

Reviewer_4spjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Selective Attenuation for Multi-Teacher On-Policy Distillation (SA-MOPD). The method modifies an existing top-K distillation learner by attenuating negative teacher-feedback coefficients on student responses that pass task verification, while preserving positive coefficients and standard supervision on other responses. The paper provides an idealized reverse-KL analysis of why distillation from a stronger teacher can temporarily reduce student success, together with a local criterion for beneficial selection at matched attenuation mass. Experiments cover mathematical reasoning, code generation, instruction following, and agentic tasks. The reported improvements over Open-MOPD are 0.87 percentage points in the base-capability aggregate and 1.27 points in the agentic aggregate.

Critical strengths
  1. The intervention is simple and clearly defined.

The distinction between retaining positive feedback and selectively attenuating negative feedback makes the proposed modification easy to understand. It can be incorporated into the existing learner without additional teacher rollouts or a separate reward-optimization objective.

  1. The matched-attenuation controls are well motivated.

The global- and domain-matched comparisons distinguish the amount of feedback removed from where it is removed. Sharing the initial parameters, optimizer state, source responses, and teacher scores makes the single-update study informative. The Code result against the domain-matched control provides evidence based on actual held-out task outcomes.

  1. The paper provides useful analysis and evaluates more than one application setting.

The reverse-KL example illustrates a meaningful distinction between matching a stronger teacher and monotonically improving task success. The experiments cover both base-capability integration and a larger agentic model, and include a comparison with a multi-teacher adaptation of whole-response gating.

Critical weaknesses
  1. The relationship between verified success and beneficial feedback attenuation remains insufficiently established.

Equation (2) attenuates all negative coefficients on the top-K support along a successful response, including coefficients for candidates that were not generated. A successful sampled response does not establish the validity of those alternative candidates, and a correct final answer does not guarantee that all intermediate reasoning is correct or useful. Consequently, the observation that all 377 verified-successful responses contain negative feedback does not identify how much of that feedback is harmful.

Proposition 2 gives a conditional local characterization: selection helps when the gate is positively correlated with the task benefit of releasing feedback. The correctness-based construction in Appendix A.3 and the controlled Code result provide supporting evidence, but do not establish the reliability of this proxy across realistic training states. The optimizer audit in Appendix G.5 supports the preconditioner approximation rather than the quality of the selection rule. A representative feedback audit or a more direct assessment of release benefit would strengthen the central mechanism claim.

  1. The controlled evidence has limited scope, and the additional value of correctness over length-related selection remains unresolved.

The clearest matched-attenuation advantage is the Code result after one update at a single intermediate checkpoint using one fixed source dataset: +2.93 points over Domain matched, with a Bonferroni-adjusted 95% interval of [0.10, 5.86]. This is useful evidence, but the prompt-bootstrap interval is conditional on that intervention and does not establish robustness across checkpoints, source data, or independent runs. The end-to-end comparison with Open-MOPD does not separately identify the contribution of correctness-based selection relative to matched attenuation throughout training.

Appendix F also shows that response shortness predicts verified success strongly in Math and moderately in Code, with AUCs of 0.911 and 0.797. The length analyses characterize this association, but no length-based or length-matched update control is evaluated. The contribution of correctness information beyond this association therefore remains unclear.

  1. The robustness of the end-to-end gains and the dependence on attenuation settings are not sufficiently documented.

The base-capability improvement over Open-MOPD is 0.87 points overall, with substantial domain variation: +1.75 in Math, +0.75 in Code, and +0.11 in instruction following, including a 0.34-point decrease on IFBench. Appendix C.1 reports three training seeds for SA-MOPD, but does not provide paired training-seed comparisons with Open-MOPD or uncertainty for the resulting method differences. The agentic results likewise lack reported replication or uncertainty supporting the 0.51-point aggregate advantage over the routed teachers. The Recovery metric is correctly defined, but is a rescaling of the same aggregate scores and does not provide independent evidence of robustness.

Reproduction is also limited by the missing values of u_0 and H in the attenuation schedule and the incomplete agentic retention configuration. Since base training eventually removes all negative coefficients on verified successes, limited comparisons with fixed retention or a nonzero terminal value would help establish sensitivity. The incremental verification and training costs should also be reported, including whether verifier outcomes are reused from the existing pipeline.

  1. The contribution of the multi-teacher setting needs clearer positioning.

The attenuation rule operates on feedback from each response's routed teacher, and Proposition 1 identifies a failure mode that already arises with one teacher. Table 1 includes single-domain OPD baselines, but does not evaluate the proposed attenuation rule within those settings. Appendix G.3 defines cross-domain task influence without reporting how the intervention changes it. The current evidence therefore establishes applicability to multi-teacher integration more clearly than a distinct insight into integration-specific interference. Clarifying the contribution as a general OPD feedback intervention, or providing a targeted comparison that identifies its role in cross-domain integration, would strengthen the paper's positioning.

Questions for authors
  1. Does success-guided selection retain an advantage over matched attenuation across additional checkpoints or source datasets, particularly when source domain and response length are controlled? A targeted comparison with a domain- and length-matched gate, or a direct assessment using the influence diagnostic in Appendix G.3, would help determine the additional selection value of correctness.

  2. Can you provide paired per-seed results for SA-MOPD and Open-MOPD, together with uncertainty for their domain-level and aggregate differences? Please also clarify the repetition and uncertainty estimates underlying the agentic results.

  3. What are the exact u_0, H, total training steps, and agentic attenuation settings? What evidence is available regarding sensitivity to fixed retention or a nonzero terminal value, and what is the actual incremental verification and training overhead?

  4. Is the intended contribution primarily a general outcome-guided correction to OPD, or a mechanism specifically addressing multi-teacher integration? If the latter, are there single-teacher attenuation comparisons or measured cross-domain effects that support this distinction?

2 comments

We sincerely thank the reviewer for the valuable feedback.

W1. The relationship between verified success and beneficial feedback attenuation remains insufficiently established.

R. Verified success provides a task-level basis for deciding where teacher imitation can be relaxed. A student may already solve a task correctly while receiving negative feedback because the teacher prefers another valid solution. SA uses this outcome signal to select responses for attenuation, and evaluates the selection by its net effect on task performance.

Proposition 2 makes this criterion explicit: at equal attenuation mass, the local advantage depends on whether the gate favors feedback whose removal is more beneficial. On 128 TACO diagnostic prompts, our controlled single-update experiment shows a 2.93 percentage-point advantage over Domain matched, with both attenuation mass and its allocation across teachers controlled.

We also examine repeated updates on a constructed program task with two valid, equal-length solutions and teachers with different solution preferences. It uses 16 training and 32 evaluation tasks across two domains. The eight conditions combine two initialization seeds, two student solution preferences and two assignments of opposing teacher preferences to the domains. Over 20 updates, the uniform control matches the attenuation that success gating would apply under its own current policy. The table reports time-weighted success over training, averaged across these conditions.

MethodProcess-average success (%)
Matched uniform attenuation87.520
SA88.429

SA improves process-average success by 0.908 percentage points, with gains in all eight conditions. Together, the experiments support the value of success-guided feedback selection in both a Code intervention and repeated updates with response length held fixed.

W2 & Q1. Does success-guided selection retain an advantage over matched attenuation across additional checkpoints or source datasets, controlling domain and response length? Can the influence diagnostic directly assess selection value?

R. All branches share the same checkpoint, source batch and update protocol. Domain × length matched removes the same negative-feedback mass as SA within each domain–length bin, distributing attenuation uniformly within that bin. Short-response-first matches SA's removed mass within each domain but allocates attenuation to the shortest responses first.

The MOPD row reports absolute scores (%); the remaining rows report changes from MOPD in percentage points (pp).

MethodMathIFCodeTotal
MOPD (%)37.93943.94524.36535.417
Domain × length matched (Δ)-0.5860.000+0.684+0.033
Short-response-first (Δ)-0.342+0.195+0.830+0.228
SA-MOPD (Δ)+0.488+0.195+0.635+0.439

In this single-update comparison, SA improves Total over MOPD by 0.439 pp, exceeding short-response-first by approximately 0.21 pp and Domain × length matched by 0.41 pp.

W3 & Q2. Can you provide paired per-seed SA-MOPD and Open-MOPD results, with uncertainty in domain and aggregate differences, and clarify repetition and uncertainty for Agent results?

R. The base-capability results include three independent SA-MOPD training runs, with the main table reporting their mean scores. The benchmark standard deviations are:

BenchmarkStandard deviation (pp)
AIME240.5512
AIME250.6315
IFEval0.8057
IFBench1.3878
LCB v50.4149
LCB v60.2969

SA-MOPD's mean Total score across the three training runs is 32.11%, exceeding the MOPD reference by 0.87 pp. The benchmark standard deviations range from 0.297 to 1.388 pp, quantifying the variation across independent training runs.

W4 & Q3. What are the exact u₀, H, total training steps and Agent attenuation settings? How sensitive is the method to fixed or nonzero terminal retention, and what verification/training overhead is added?

R. We use u₀=1, H=159 and 160 total training steps, with retention decreasing linearly from 1 to 0: λᵤ = 1 − clip((u − 1)/159, 0, 1). The Agent experiment uses the same attenuation schedule, gated by task success over the full interaction.

At a matched training budget, we compare this schedule with fixed retention 0.5 and increasing retention 0→1. Their Total improvements over MOPD are +0.87, +0.01 and +0.45 pp, respectively. The 1→0 schedule performs best overall; 0→1 gives the highest IF score. The domain-level results appear in the schedule tables for Reviewers 6Uu3 and 7iWw.

SA adds a verification process whose cost depends on the task and verifier. We observed approximately 18% longer training time than MOPD.

W5 & Q4. Is the contribution a general outcome-guided OPD correction or a mechanism specific to multi-teacher integration? Are there single-teacher attenuation comparisons or measured cross-domain effects?

R. Our primary focus is integrating multiple specialist capabilities into one shared student. Teacher routing determines which expert provides supervision; SA addresses how the student should use that feedback during integration. It uses verified student success to selectively relax negative imitation feedback while retaining positive feedback and full supervision on failures. Our contribution combines an analysis of the tension between teacher matching and student task success with controlled feedback-selection experiments and evaluations of multi-teacher capability integration.

Reviewer_6Uu3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SA-MOPD, which selectively attenuates negative distillation feedback on verified successful student responses while retaining positive feedback and full supervision on failed responses. Its theoretical analysis shows how teacher matching can reduce task success and characterizes local conditions under which selective attenuation helps. Experiments on basic capability integration and agentic tasks improve aggregate scores over Open-MOPD by 0.87 and 1.27 percentage points, respectively.

Critical strengths
  1. The method is simple and demonstrates practical gains in two capability integration settings. SA-MOPD directly modifies the feedback coefficients of an existing learner, improving aggregate scores over Open-MOPD by 0.87 percentage points on basic capability benchmarks and 1.27 percentage points on agentic tasks. These results support its practical utility.
Critical weaknesses
  1. The incremental contribution over closely related methods is not sufficiently established. OPDVR and SG-OPD also use verifier outcomes to regulate distillation supervision, but neither is included in the experimental comparisons. In particular, OPDVR’s successful-trajectory branch follows the same positive-only filtering principle as SA-MOPD at λ\lambda=0. The comparison with RG-OPD is useful, but does not resolve this overlap. Comparisons that separately control the attenuation schedule, treatment of failed responses, and candidate support are needed to identify what SA-MOPD adds and when its design is preferable.

  2. The connection between the theoretical analysis and the chosen gate remains insufficiently established. The theory establishes local benefits when the gate favors feedback whose attenuation improves task success, but the algorithm selects responses based on final verification. It does not establish that verified success reliably predicts the benefit of removing negative feedback. The single-checkpoint experiments provide local support, but do not establish this relationship throughout training. Consequently, the paper offers limited additional insight into the sources of its improvements and the conditions under which they hold.

Questions for authors
  1. The authors provide single-step evidence consistent with preserving existing successful behavior. Is this preservation effect the primary source of SA-MOPD’s gains over full training, or is there evidence for additional mechanisms? Suggested validation: Tracking the probability of the student’s existing correct solutions and task success rates throughout training, compared with Open-MOPD, could help establish whether preservation explains the final gains.

  2. What motivates decreasing λ\lambda from 1 to 0? Figure 2a illustrates early degradation followed by recovery under teacher matching, but does not directly establish this schedule direction. Have you compared 1 to 0, 0 to 1, and constant-λ\lambda schedules under comparable training budgets to separate the benefit of selective attenuation from that of its scheduling?

2 comments

We sincerely thank the reviewer for the valuable feedback.

W1. The incremental contribution over closely related methods is not sufficiently established.

R. Our focus is how a shared student should selectively use expert feedback when integrating multiple specialist capabilities. Domain routing determines which expert supervises a response, but all experts update the same student. Their preferences among valid solutions need not coincide with preserving the student's ability to solve tasks. SA uses the student's verified success as a common criterion for deciding where imitation constraints can be relaxed during integration.

Among contemporaneous works, OPDVR aligns feedback signs with response correctness. SG-OPD regulates token-level supervision through teacher–verifier sign agreement and supplements early training with verified teacher trajectories. These methods share the use of outcome signals. SA's contribution centers on the distinction between absorbing expert capabilities and reproducing expert preferences, developed through task-level analysis and specialist-integration experiments.

Our analysis establishes when matching a stronger teacher can reduce task success, then separates the value of selecting feedback from the amount of supervision removed. This motivates SA's learning policy: gradually relax negative feedback on verified successes while retaining positive feedback and full supervision on failures. Its zero-retention success filter overlaps with OPDVR's, while failure supervision and scheduling differ. SA-MOPD operates on student trajectories without additional teacher rollouts.

With unified experimental settings, SA-MOPD improves Total over MOPD by 0.87 pp, compared with 0.13 pp for OPDVR† and 0.33 pp for RG-OPD†:

MethodMathIFCodeTotal
MOPD (%)22.4249.5821.7331.24
OPDVR† (Δ)+0.68+0.24-0.53+0.13
RG-OPD† (Δ)+1.48-0.87+0.37+0.33
SA-MOPD (Δ)+1.75+0.11+0.75+0.87

† denotes our multi-teacher adaptations, since original methods are designed for single-teacher on-policy distillation.

W2. The connection between the theoretical analysis and the chosen gate remains insufficiently established.

R. Proposition 1 shows that, even with a stronger teacher, differences in preferences among correct solutions can make imitation reduce student task success. This motivates using verified student outcomes: a successful response can still receive negative feedback reflecting the teacher's solution preferences. Proposition 2 then provides a testable criterion—at equal attenuation mass, does the gate allocate attenuation to feedback whose release yields greater task benefit? Our matched-attenuation experiments test this selection value directly.

All branches receive one update on the same 1,024-response batch, then are evaluated on 128 TACO diagnostic prompts disjoint from that batch, with eight responses per prompt.

MethodCode accuracy (%)
MOPD29.004
Domain matched28.125
SA-MOPD31.055

On the Code diagnostic task, SA exceeds Domain matched by 2.93 pp, controlling both total attenuation and its allocation across teachers. In a separate constructed, equal-length program task, SA improves process-average success over matched uniform attenuation by 0.908 pp, averaged across eight mixed-teacher conditions. Together, these single-update and multi-update results support verified success as a useful signal for allocating attenuation.

Q1. Is preservation the primary source of the full-training gains, or do other mechanisms contribute? How do existing correct-solution probabilities and task success evolve during training?

R. SA is designed to protect task competence while preserving the learning signal. It relaxes negative imitation feedback on verified successes while retaining positive feedback and full supervision on failed responses.

To examine this effect, we track performance on 384 prompts from the training pool, with 128 each from Math, IF and Code. Using the common starting model, we group prompts by how many of two initial responses are correct (two, one or none), then compare the final success rates of MOPD and SA with fixed retention 0.5:

Initial responsesMOPD (%)SA, constant 0.5 (%)
Both correct84.63185.451
One of two correct50.00051.613
Neither correct8.8758.875

Relative to MOPD, SA improves endpoint success by 0.820 pp on prompts with two initially correct responses and 1.613 pp on prompts with one initially correct response; the initially unsuccessful group has the same endpoint score. The gains are concentrated in groups with initial successful responses, supporting preservation of task competence as a contributing mechanism.

Q2. Why decrease retention from 1 to 0? How does this compare with reverse and constant schedules under comparable training budgets?

R. The schedule is designed to shift emphasis from acquiring teacher capabilities to allowing the student greater flexibility on verified successes. Starting at retention 1 preserves the full MOPD supervision while the student learns from the specialists. Gradually lowering retention then relaxes pressure to reproduce teacher preferences on responses that already satisfy the task, allowing the student to retain valid alternative solutions during integration. At retention 0, only negative feedback on verified-successful responses is removed. Positive feedback and the full supervision on failed or unverifiable responses remain active, so capability learning continues.

We compare decreasing retention (1→0), increasing retention (0→1), and constant retention (0.5, keeping half) at the same training budget. The main 1→0 schedule achieves the highest Math, Code and Total scores, while reverse retention achieves the highest IF score. The MOPD row reports absolute scores (%); the remaining rows report Δ in pp.

MethodMathIFCodeTotal
MOPD (%)22.4249.5821.7331.24
SA, constant 0.5 (Δ)+0.70-0.67-0.02+0.01
SA, 0→1 (Δ)+0.73+0.67-0.05+0.45
SA, 1→0 (Δ)+1.75+0.11+0.75+0.87

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.