- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies capability integration in multi-teacher on-policy distillation (MOPD). In MOPD, student rollouts are routed to domain-specialist teachers, which provide token-level signed feedback. The authors point out that this feedback is not conditioned on task outcomes: even when the student has produced a verified-correct response, the teacher may still assign negative feedback, and this negative feedback may reflect an alternative valid solution or a teacher error rather than a genuine student mistake. Based on this observation, the paper proposes SA-MOPD: when a verifier confirms student success (), negative feedback is multiplied by a retention factor that decreases linearly from 1 to 0; positive feedback and full supervision on other responses remain unchanged. The paper provides two propositions: Proposition 1 shows that ideal reverse-KL distillation can reduce student success even with a stronger teacher when within-outcome matching dominates; Proposition 2 shows that, at equal attenuation mass, selecting where to release feedback based on verified outcomes yields a first-order task benefit over uniform attenuation. Experiments cover base capabilities (SmolLM3-3B on math, code, and instruction following) and agentic tasks (Qwen-3.6-27B on BFCL v4, TAU2, Claw-Eval, and SkillsBench), and include Global matched / Domain matched controls that match the amount of attenuated feedback mass.
Critical strengths
- The motivation is clear and practically relevant: teacher disagreement is not the same as student error, especially on tasks with multiple valid solutions. Figure 1 and Figure 3 convey this mechanism intuitively.
- The method is simple and low-cost: it only modifies the negative-feedback coefficients in the existing learner, without requiring an additional reward objective, teacher rollouts, or a critic.
- The theoretical analysis is targeted. Proposition 1 uses the KL chain rule to separate outcome probabilities from within-outcome matching, and gives conditions under which teacher matching can oppose task success; Proposition 2 separates “where to attenuate” from “how much to attenuate,” directly motivating the Global matched / Domain matched experimental design.
- The control experiments are well designed. Comparing SA-MOPD with Global matched and Domain matched at the same attenuation mass is more convincing than comparing only with MOPD. The single-update analysis in Table 4 and Figure 4 is also relatively clean.
- The experiments cover both base and agentic settings and report variance across three training seeds.
Critical weaknesses
-
The method itself is very close to a heuristic rule.
The core change in SA-MOPD is essentially one line: multiply negative feedback by when verification succeeds. This is simple and effective, but it also means the contribution is more “a reasonable engineering rule plus a local analysis of why it might work” than a new learning objective or training framework. The conclusion of Proposition 2—selective attenuation is better than uniform attenuation when the gate aligns with release benefit—is almost tautological at an intuitive level; its value lies mainly in formalizing the experimental design rather than providing a non-trivial algorithmic insight. -
The gap between the theoretical analysis and actual training is not sufficiently discussed.
Propositions 1 and 2 are local, first-order analyses under fixed rollouts and detached coefficients. In actual training, follows a schedule over training steps, feedback is refreshed, and the rollout distribution changes with the policy. The paper does not explain how far these local conclusions extend to the full training process. Proposition 2 depends on the sign of , but this quantity is not explicitly estimated during training; instead, “verified success” is used as a proxy. When this proxy holds and when it fails is not discussed enough. -
The scale of experiments and statistical evidence are limited.
The base experiments use only one student model (SmolLM3-3B), and the agent experiments use only one student model (Qwen-3.6-27B). Table 1 reports only means over three seeds, with variance deferred to Table 6 in the appendix; Table 2 does not report seed variance. Some gaps are small: in Table 3, SA-MOPD and RG-OPD† differ by only 0.54 points in Total (32.11 vs. 31.57), and in Table 2, SA-MOPD and Route Teacher differ by only 0.51 points in Total. Given that the agent setting trains a single model, more evidence is needed to establish that these gaps are stable. In addition, the “Recovery > 100%” (109.7%) claim should be stated more cautiously: it shows that the consolidated student exceeds routed specialists in aggregate, but the student still trails them on individual benchmarks (TAU2 84.36 vs. 84.96; Claw-Eval 73.72 vs. 74.03). The main text could be more restrained in emphasizing “exceeding the routed reference.” -
The scope of verifier dependence and gate semantics needs clearer boundaries.
SA-MOPD’s effect depends directly on whether the verifier can reliably judge “student success.” Math and code have relatively clear verifiers, IF uses rule-based checkers, and agent tasks use task-level success. However, the paper does not systematically discuss verifier noise, partial success, or cases where a multi-turn task succeeds overall but contains intermediate errors. Appendix B.2 states that the gate uses the full interaction outcome, but partial progress sets , meaning many “near-success” responses still receive full negative feedback. The rationale for this design choice needs more discussion. In addition, the teachers in the agent experiments are themselves RL specialists; whether the verifier is coupled with the teacher training signal is not discussed.
Questions for authors
- Propositions 1 and 2 are local first-order analyses. Can the authors provide more direct evidence that these local conclusions still hold during full training, rather than only in single-update or intermediate-checkpoint settings?
- Under what conditions does “verified success” fail as a proxy for ? For example, when successful responses contain many low-quality tokens or when the verifier itself is noisy, is SA-MOPD still better than Domain matched?
- The agent setting trains only one student model, and the gap to Route Teacher in Table 2 is only 0.51 points. Can the authors add multiple training seeds or at least report variance to show that this result is not a single-run fluctuation?
- How was the linear schedule of from 1 to 0 chosen in the base experiments? Did the authors try other schedules (e.g., fixed , or stepwise decay over training), and how sensitive are the results?
- How does the verifier handle partial success and intermediate errors in multi-turn agent tasks? If the gate opens only on final task success, does it miss “eventually successful but flawed” responses and thereby limit the method’s applicability?
2 comments
Q4. How was the linear 1→0 schedule chosen, and how sensitive is performance to alternative schedules?
R. The decreasing schedule retains full teacher supervision initially, then gradually relaxes imitation constraints on verified successes while preserving supervision on failures. We compare three schedules for retaining negative feedback on verified successes: decreasing retention from all to none (1→0), increasing it from none to all (0→1), or retaining half throughout training (constant 0.5). All runs share the same initialization and training budget. Our main 1→0 schedule achieves the best Math, Code and Total scores, while 0→1 performs best on IF.
The MOPD row reports absolute scores (%); Δ denotes the change from MOPD in percentage points (pp).
| Method | Math | IF | Code | Total |
|---|---|---|---|---|
| MOPD (%) | 22.42 | 49.58 | 21.73 | 31.24 |
| SA, constant 0.5 (Δ) | +0.70 | -0.67 | -0.02 | +0.01 |
| SA, 0→1 (Δ) | +0.73 | +0.67 | -0.05 | +0.45 |
| SA, 1→0 (Δ) | +1.75 | +0.11 | +0.75 | +0.87 |
Q5. How does the verifier handle partial success and intermediate errors in multi-turn Agent tasks? What happens to eventually successful but flawed responses?
R. The current Agent gate evaluates the final task outcome. Partial progress that falls short of the success criterion leaves attenuation inactive, preserving the full corrective signal. An unavailable judgment also leaves the gate inactive. If an interaction succeeds despite a recoverable intermediate error, the final success judgment activates response-level attenuation. Positive feedback remains unchanged.
Tool-call outcomes, intermediate verification and LLM judges could support finer-grained step assessments. Our current focus is using task-level success to guide feedback selection during specialist integration.
We sincerely thank the reviewer for the valuable feedback.
W1. The method itself is very close to a heuristic rule.
R. The theory explains when teacher imitation can harm task success and why selecting feedback can help. Proposition 1 gives an explicit condition under which matching a stronger teacher reduces student success because preferences among valid solutions differ.
The two-route example in Appendix A.3 illustrates the distinction. The student succeeds on two equally likely reasoning routes with probabilities 80% and 20%. Taking a teacher that succeeds with 60% probability on each route makes it stronger overall, but weaker on the student's better route. Unconditional attenuation weakens useful correction on the student's weaker route as well. Correctness-based gating more often relaxes imitation on the stronger route while retaining correction on the weaker one. In this construction, a single update yields unconditional attenuation < MOPD < correctness-gated attenuation in task success.
Proposition 2 then quantifies the local value of selection at equal attenuation mass, separating where feedback is removed from how much is removed. This motivates the Global-matched and Domain-matched controls; on the TACO code-generation diagnostic set, SA achieves 2.93 percentage points higher accuracy than Domain matched after a single update. The analysis thus supplies a concrete failure mechanism and an experimentally testable principle for selecting feedback.
W2. The gap between the theoretical analysis and actual training is not sufficiently discussed.
R. We connect the local analysis to training through an actual-Adam audit and end-to-end evaluation (Q1), and identify how verifier quality affects feedback selection (Q2).
W3. The scale of experiments and statistical evidence are limited.
R. The base-capability results report three independent SA-MOPD training runs. Due to limited computational resources, the Agent experiment uses a single training run.
W4. The scope of verifier dependence and gate semantics needs clearer boundaries.
R. Response Q2 addresses verifier errors and successful-but-flawed responses; Response Q5 states how the current gate treats partial progress and intermediate errors. The gate uses task-level success; its relationship to the specialists’ RL reward specification is unspecified in the reported configuration.
Q1. Can you provide direct evidence that the local first-order conclusions hold during full training, beyond single-update or intermediate-checkpoint settings?
R. Our analysis describes how selective attenuation affects task performance at the student's current state. In actual training, that state changes as the student learns and generates new responses. The analysis therefore explains the effect of an individual update, while training experiments assess how the method performs as the policy and feedback evolve.
We check the connection to actual optimization in Appendix G.5. For SA versus Domain matched, the local approximation and measured Adam updates produce closely aligned task projections: correlations are 0.9936–0.99999, with 96.04%–100% sign agreement on prompts with nonzero task-gradient estimates. This supports the optimizer approximation at the audited state.
The full-training results complement this check. Across three independent base-capability training runs, SA-MOPD achieves a mean Total score of 32.11%, exceeding MOPD by 0.87 pp, with improvements in all three domains. The Adam audit validates the local optimizer approximation at the audited state, while the full-training results establish the method's aggregate performance. Isolating selection's contribution throughout training calls for a full-training attenuation-matched comparison.
Q2. When does verified success fail as a proxy for positive release benefit? With low-quality tokens or verifier noise, is SA-MOPD still better than Domain matched?
R. Verified success can fail as a proxy when the negative feedback on a successful response still provides useful correction—for example, correcting flawed intermediate reasoning despite a correct final answer. A false-positive verifier judgment can similarly activate attenuation on a response that needs correction. A false negative instead leaves full baseline supervision active on that response.
Proposition 2 identifies the governing condition: selection is beneficial when the gate favors feedback with greater release benefit. Low-quality content or verifier errors can weaken or reverse this alignment. Our reported advantage over Domain matched is measured with the original task verifiers; performance under these perturbations is an empirical question.