acceptodds
Under review as a conference paper at ICLR 2027

When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation

12 views

Abstract

Teacher top-K logits can preserve nearly all teacher probability mass while omitting corrections needed for a student's behavioral decisions. We study this gap in routed, two-teacher tool-use distillation: a tool teacher reinforces an entry coordinate that the response teacher usually omits even when the student overestimates it. In Qwen3.5-9B, response-teacher top-32 retains 99.99% of its mass yet contains the tool-entry token on only 0.4% of 500 response prompts. A frozen full-vocabulary audit shows that restoring this coordinate with its exact teacher logit recovers full-vocabulary descent on that coordinate. Forced-entry replay and matched training connect the omission to behavior: first-position restoration relocates calls to later tokens, whereas all-position restoration reduces full-generation over-calling from (14.2 +/- 2.1)% to (3.7 +/- 0.5)% while lowering required-call recall by 12.4 points. Teacher-probability-matched and output-logit-gradient-rerouted non-tool controls produce smaller shifts and do not reproduce exact restoration. We then compare support construction, loss shaping, and validation-tuned decoding bias. The comparison separates changes in the call threshold from modest gains in discrimination and exposes the access and capability costs of reducing calls. Llama-3.1-8B reproduces the directional support asymmetry under native JSON. Together, these results identify decision-critical support omission as a causal contributor in the primary Qwen setting and show why teacher-mass coverage alone cannot certify behavioral fidelity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, sparse teacher logits, tool use, multi-teacher learning, behavioral calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dXcvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a token-coverage problem in teacher-top-KK on-policy distillation. The authors show that a teacher’s top-KK set can retain almost all of the teacher’s probability mass while still omitting high-probability tokens under the student distribution. This issue is particularly evident for the special <tool_call> token: although the response teacher has near-complete probability-mass coverage, it may fail to include the student-relevant call-entry coordinate and therefore cannot provide an effective corrective signal. In a routed multi-teacher setting, the tool teacher reinforces the call entry, whereas the response teacher often fails to provide the corresponding negative correction, creating an asymmetric supervision signal. Based on this diagnosis, the paper evaluates three interventions for correcting the finding.

Critical strengths

The paper conducts a comprehensive set of analytical experiments, revealing the biases introduced by top-KK-support distillation and evaluating the effects of several intervention mechanisms.

Critical weaknesses
  1. The diagnosis has limited persuasive power and conceptual novelty. The diagnosis is primarily based on a single <tool_call> token in the Qwen3.5 tool-use setting. Although the paper later reproduces the coverage pattern in Llama’s JSON setting, the analysis remains largely confined to tool-calling scenarios, which limits its generality and persuasiveness. Also, the findings do not appear sufficiently novel or insightful to provide the community with a substantially new understanding.

  2. The proposed methods are only weakly connected to the diagnosis, offer limited novelty, and yield modest improvements. The student-teacher support-union strategy is relatively straightforward, while the connection between Soft Clamp and the diagnosed probability-mass omission is not intuitive. In terms of performance, Soft Clamp provides only a limited improvement in overall decision accuracy and does not show a clear advantage over vanilla GKD or the support-union strategy.

Questions for authors

Please see the weakness section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dXcv

W1. Scope, persuasive power, and conceptual novelty

The empirical scope is tool use, with the complete matched chain in Qwen and directional behavioral/support replication in Llama. Llama's native JSON results rule out dependence on Qwen's specific XML entry marker, but do not establish generalization to tasks without an identifiable behavioral entry.

We do not base novelty on the general observations that top-K can bias distillation, that tail information matters, or that teacher–student support compatibility matters. These have precedents in Sparse Logit Sampling, Don't Ignore the Tail, and Teachability-Aware OPD (SLS, TAD, TA-OPD).

The specific empirical contribution is the connection from an omitted correction to generation dynamics under the implemented objective. The response route usually removes direct supervision on a student-overestimated entry coordinate, while the tool route reinforces it. Crucially, restoring the first position changes local entry but mostly relocates calls: initial calls on should-respond examples fall from 13.42% to 3.07%, while delayed calls rise from 0.78% to 10.42%. Restoring all response positions yields a similar initial boundary but reduces delayed calls to 0.65% and full-generation over-calling to 3.73% (Table 15).

This temporal behavior is not implied by the zero-gradient observation alone. The matched non-tool controls further narrow the explanation: over-calling is 13.25% with the teacher-probability-matched placebo and 11.65% with gradient rerouting, versus 3.73% with exact all-position restoration (Table 16). The former does not match student probability or gradient strength; the latter matches the frozen forward loss and selected output-logit-gradient magnitude, not full parameter gradients, and has collateral degradation.

We offer this as a bounded behavioral attribution and a caution about local repair and retained-mass audits. It does not establish a universal cause of tool overuse or a general law of multi-teacher capability conflict.

W2. Connection of interventions to the diagnosis, method novelty, and modest gains

The interventions change different parts of the system. Support construction can restore omitted coordinates. Soft Clamp reshapes retained training signals and cannot restore those coordinates. Entry bias adjusts the deployed boundary without retraining. Their comparison asks how much of the observed behavior can be changed at each layer, with different access requirements and capability costs; it does not establish three interchangeable repairs of the same omission.

Teacher/student union is a familiar construction. TA-OPD already uses union support to measure local disagreement for position selection; our experiment uses it as the actual within-position distillation support. The set operation is not an algorithmic novelty claim. Its role here is to test a student-aware construction that does not hard-code the entry token and to measure the resulting behavioral point.

The Soft Clamp accuracy gain is also small: 88.6±0.3% for vanilla versus 88.7±0.7% for Soft Clamp. Its over-calling reduction, 14.2% to 9.0%, accompanies lower call recall, 91.5% to 86.5%. It retains the original teacher top-K interface, without student-selected teacher probes or a union prepass. Union has a stronger single-turn point, while Soft Clamp preserves more required calls and higher dialogue exact success in the multi-turn diagnostic (Table 6); both remain below vanilla on dialogue exact. These differences support a comparison of operating points and access requirements, rather than overall superiority for Soft Clamp.

Accordingly, the mechanism evidence rests on the frozen audit, replay, exact temporal-scope restoration, and matched non-tool controls. Loss shaping and decoding bias are comparators for the scope of behavioral calibration. Their limited gains do not establish a new general-purpose optimization method, and the current evidence does not establish a universally preferable deployment strategy.

Reviewer_5PG4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate tool over-calling in routed on-policy distillation with separate tool and response teachers. Its central observation is that the response teacher's top-32 distribution retains almost all teacher probability mass while usually excluding the token that enters tool mode. Renormalizing both models on this restricted support removes the direct gradient on that omitted student logit, even when the student assigns it substantial probability. The paper combines frozen gradient audits, forced-entry replay, and matched training interventions to study the behavioral effect. Restoring the coordinate only at the first position mainly delays erroneous calls and restoring it throughout the response reduces over-calling but also loses required calls.

Critical strengths
  • S1: It seems to me that the first-position versus all-position comparison is the strongest result. First-position restoration reduces initial calls on should-respond examples from 13.42% to 3.07%, but delayed calls rise from 0.78% to 10.42%, as shown in Tab. 15.

  • S2: The paper also reports the costs of the interventions rather than presenting reduced calling as an unconditional improvement. Required-call recall, AUC, and dialogue success are reported alongside over-calling. I have checked selected aggregate results against the supplementary files; they agree with the reported values. These details make the empirical analysis substantially more informative than a comparison of call rates alone.

Critical weaknesses
  • W1: I think the missing training controls limit how much of the drift can be attributed to this truncation rule. Equation 1 discards the student's total probability outside the teacher support by renormalizing it away. A useful comparison would retain the omitted mass as one extra category, using teacher mass measured at the training temperature. This would clarify the practical benefit of the extra teacher probes required by support union.

  • W2: Also, there is also no matched full-vocabulary training run. Restoring one response-side coordinate establishes that changing its supervision changes behavior, but it does not measure the net effect of removing truncation on both routes.

  • W3: It appears to me that the practical recommendation is less settled than the local mechanism. Support union improves mean APIGen decision accuracy and some single-turn transfer results, but required-call coverage drops and multi-turn dialogue exact success falls from 7.04% to 4.12%. The AUC gain, from 0.9692 to 0.9760, is encouraging but small.

Questions for authors

Please find the comments in the weakness section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5PG4

Thank you for recognizing the temporal-scope finding and checking the supplementary results.

W1 — Preserving omitted probability mass. Adding an OTHER category would retain the teacher's and student's total tail probabilities and can restore gradients on omitted student coordinates without individual teacher probes. The probabilities must use the training temperature, 0.9; our raw retained-mass audit cannot substitute for this calculation.

We did not evaluate this alternative, so we cannot establish that union's additional probes are necessary or preferable. Our restoration controls address coordinate identity and temporal scope under Eq. 1; they do not replace the proposed mass-preserving comparison.

W2 — Full-vocabulary training. We have not run matched full-vocabulary OPD. It would change supervision on both routes, other coordinates, and subsequent rollouts. Response-entry restoration therefore cannot predict its net outcome. Our established claim is narrower: restoring the omitted response-side correction causally changes calling behavior under the implemented objective, with first-position repair relocating calls and all-position repair changing complete generations.

W3 — Practical value and capability costs. We agree that union is not an unconditional improvement. Its three-seed mean calls/turn falls from 1.515 to 1.118, but required-call coverage falls from 92.90% to 81.22% and dialogue exact from 7.04% to 4.12% (Table 6). The AUC increase from 0.9692 to 0.9760 indicates modest separation change, not broad capability improvement. The paired intervals for APIGen accuracy and BFCL overall gains include zero (Table 17).

Union also increases observed steady-state step time by 16.0% at the same epoch budget. The practical implication is to audit student-relevant corrections and assess calling restraint alongside correctness, required-call retention, and access costs. Appropriate operating points depend on error costs and system constraints; the present evidence does not settle the best correction. Potential latency and per-call costs motivate this question, but we did not measure deployment savings.

Reviewer_xWH7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a failure mode of top-K logit truncation in multi-teacher on-policy distillation for tool use. The authors show that top-K can retain almost all teacher probability mass while still missing a token that is important for correcting the student’s behavior. In the main setting, the response teacher often omits the tool-entry token from its top-K support, while the tool teacher consistently reinforces it. The paper analyzes this effect through full-vocabulary audits, forced-entry replay, restoration experiments, and several controls, and shows that restoring the omitted coordinate can shift the model’s tool-call behavior. It also compares support-level, loss-level, and decoding-level interventions and studies the trade-off between reducing unnecessary tool calls and preserving required-call recall. Similar support asymmetry is also observed in a Llama-based setting.

Critical strengths
  1. The paper highlights a clear failure mode of top-K distillation: retaining nearly all of the teacher’s probability mass does not necessarily preserve the supervision needed to correct the student’s behavior.
  2. The main claim is supported by experiments from multiple perspectives, including full-vocabulary audits, replay analysis, restoration experiments, and several control studies.
Critical weaknesses
  1. The main concern is that the paper studies a fairly narrow setting. Most of the causal evidence comes from routed multi-teacher tool-use distillation, and the analysis mainly focuses on one tool-entry token. It is not clear how often the same issue would appear in more general distillation settings where there is no single token that directly controls a behavior switch.
  2. It is also unclear how harmful the identified failure mode actually is. Restoring the omitted tool-entry coordinate substantially reduces over-calling, but also lowers required-call recall, and some multi-turn metrics become worse. Overall, the results look more like a shift in the call/respond operating point than a clear improvement in model behavior, which makes the significance of correcting this omission less clear.
Questions for authors
  1. The Llama experiments show that the support asymmetry is not specific to Qwen, but the full causal analysis is still only carried out in the Qwen setting. Is there additional evidence that the same causal mechanism holds across other model families or task settings?
  2. The paper does not train a full-vocabulary OPD baseline and instead relies on restoring selected omitted coordinates. How different would the behavior be under actual full-vocabulary distillation, and could other omitted coordinates change the overall training dynamics?
  3. Is the same failure mode observed in single-teacher OPD, or does it mainly arise from the asymmetric supervision introduced by routed multi-teacher distillation?
  4. The main experiments use teacher top-32 logits. Does the behavioral drift persist when using larger values of K, such as 64, 128, or 256, rather than only measuring whether the tool-entry coordinate is included in the support?
1 comment
AuthorsOpenReview ↗

Response to Reviewer xWH7

Thank you for recognizing the failure mode and the evidence from complementary analyses.

W1/Q1 — Scope and cross-family evidence. Tool use provides an observable boundary for testing how omitted supervision affects a generation branch. Our complete causal chain is established in Qwen. Llama provides matched three-seed training interventions, not only coverage audits: union reduces over-calling from 28.80±0.80% to 11.08±1.16%, with call recall changing from 97.27±0.33% to 91.23±0.94% (Table 25). Bilateral audits reproduce the directional support asymmetry under native JSON (Table 26). This extends the evidence beyond Qwen's XML marker, but does not establish a second complete causal chain or generalization to non-tool tasks.

W2 — Significance and capability costs. We agree that fewer calls do not alone imply a better model. The distinctive finding is temporal: first-position restoration reduces initial erroneous calls from 13.42% to 3.07%, yet delayed calls rise from 0.78% to 10.42%. All-position restoration has a similar first-token boundary but reduces delayed calls to 0.65% (Table 15). Thus, successful local repair can leave full-generation over-calling largely unchanged.

All-position restoration is diagnostic, not an unconditional improvement: call recall falls from 91.45% to 79.07%, and dialogue exact from 7.04% to 3.38%. Additional serial tool steps can increase waiting time and per-call cost, motivating study of this boundary. However, we did not measure latency, timeout rates, or monetary savings. The existing cost-sensitive analysis (Appendix C.2) illustrates conditional operating-point preferences, without establishing deployment benefits or offsetting the correctness loss.

Q2 — Full-vocabulary OPD. We have not run matched full-vocabulary training. Frozen audits show that full vocabulary changes tool-route reinforcement as well as response-side correction (Table 12). Other coordinates and evolving rollouts could alter the net outcome. Our evidence establishes that restoring the omitted response-entry supervision changes learned calling behavior under Eq. 1; it does not predict full-vocabulary OPD's final behavior.

Q3 — Single-teacher OPD. The omitted-logit zero-gradient property of Eq. 1 does not require multiple teachers. However, we have no matched single-teacher tool-use study and cannot establish whether comparable drift occurs or whether it would be weaker.

Q4 — Larger K. On 500 frozen Qwen response prompts, entry coverage is 0.4%, 4.6%, 22.6%, and 52.2% at K=32, 64, 128, and 256. These are support audits, not larger-K training results. They show that omission can remain at larger K on the audited contexts, but do not establish the behavior of retrained students.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.