acceptodds
Withdrawn from ICLR 2027

CoKD: Generating Student-Reachable Targets through Teacher–Student Collaborative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge distillation (KD) is a standard approach for compressing large language models, yet its effectiveness critically depends on the quality of the targets used to supervise the student. Teacher-generated targets can be informative but often lie outside what a small student can reliably learn, whereas student-only rollouts remain reachable but may encode suboptimal decisions. We propose Collaborative Knowledge Distillation (CoKD), which generates distillation targets through teacher–student collaborative decoding. At each decoding step, CoKD samples from a tempered distribution that combines teacher preference with student support, favoring tokens that are endorsed by both teacher and student. A task-aware sequence-level quality filter further retains only collaborative responses that provide useful training targets. Across arithmetic reasoning, instruction following, and dialogue summarization benchmarks with Qwen2.5 and Gemma teacher–student pairs, CoKD consistently improves over competitive KD baselines and remains stable across varying teacher scales. These results indicate that effective distillation requires targets that are not only teacher-preferred but also student-reachable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge distillation, Large language models, Model compression, Distillation target design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cTExOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposed a knowledge distillation (KD) algorithm. The work starts from the sequence-level KD which is supervised fine-tuning on teacher-generated trajectories. To mitigate the issue that the teacher's labels may lie in the space far beyond the reach of the student, they propose a mixture policy which is the convex combination of teacher and student distribution, and use this mixture policy to generate trajectories on which the student is trained. To ensure the quality of these trajectories, further quality filtration designs are applied. Through experiments, they show the effectiveness of their proposed algorithm.

Critical strengths

None

Critical weaknesses
  1. The main idea is just using the mixture policy of teacher and student as the target in sequence-level KD, which is nothing special. Using the mixture distribution is not new in knowledge distillation, for example, people usually use JS divergence as the objective, where the target is naturally mixture policy. The difference here is just that the prefix is also generated with this mixture policy. From a principled perspective, I didn't see the benefits of conditioning on the prefix generated by the mixture policy, so I would regard this work as a totally empirical work with an intuitive algorithm. It needs extensive experiments to support the claims. However, I didn't see such experiments, instead, the work uses outdated models, outdated and extremely simple benchmarks. The comparisons are unclear either, for example, logins-based KD is not explained and the simple on-policy KD baseline is missing (default GKD uses a combined objective). Only instruct models are used, no reasoning models or base models.
Questions for authors

No

Reviewer_1scYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoKD, which generates collaborative distillation targets using a weighted product of teacher and student next-token probabilities and restricts candidates with a student-probability threshold. Generated sequences are further retained by a task-aware quality filter based on reference answers, after which the student is trained with cross-entropy. Experiments cover Dolly, GSM8K, and SAMSum using Qwen2.5 and Gemma teacher-student pairs and several teacher scales. CoKD exceeds the strongest reported baseline in five of six main model-task settings.

Critical strengths
  1. CoKD is clearly motivated and simple to implement. It improves over several distillation baselines across two model families and most task settings, with useful analyses of hyperparameters and teacher scale.
  2. The paper targets a practical tension between target quality and student reachability, and the student threshold and collaborative distribution provide an intuitive approach.
Critical weaknesses
  1. Neither the novelty boundary nor the central mechanism is sufficiently isolated. Equation (1) is closely related to existing geometric-interpolation and Product-of-Experts targets, the student threshold is confidence filtering, and the subsequent CE objective is standard sequence-level distillation. The paper does not provide controlled comparisons with a simple arithmetic mixture, a TAID-style interpolated soft target, or a Veto-style Product-of-Experts soft target, nor does it establish that high current student probability implies greater learnability.
  2. The gains are strongly coupled to conventional reference-based rejection sampling. The quality filter contributes 1.21, 3.01, and 3.03 points, while zero teacher weight reaches about 44.3 on SAMSum compared with 43.08 for the main configuration. Although Table 8 applies the filter to several baselines, accepted sample counts, training-token budgets, and total compute are not matched, and all main results lack variance or significance tests. The evidence therefore does not separate the contributions of Product-of-Experts collaborative decoding, quality filtering, and student self-training. CoKD also underperforms GKD on Gemma GSM8K, making the abstract's “consistently improves” claim too strong.
Questions for authors
  1. Under identical filtering rules, accepted sample counts, training-token budgets, and seeds, please compare CoKD against zero-teacher student self-training, the arithmetic mixture αpT+(1−α)pS\alpha p_T+(1-\alpha)p_S, a TAID-style geometric-interpolation soft target, and a Veto-style Product-of-Experts soft target. Why does zero teacher weight substantially exceed the main configuration on SAMSum, and how do the current results establish that the gain comes from the collaborative distribution rather than student self-sampling and reference filtering?
  2. Why does SAMSum use ROUGE-L for filtering but METEOR for final evaluation, and how does this metric mismatch affect target selection? When δ=1\delta=1 and both the student and collaborative outputs are perfect, the second condition in Equation (4) still accepts the collaborative output. How does this match the statement that only strictly better targets are accepted?
Reviewer_bGubOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Collaborative Knowledge Distillation (CoKD), a sequence-level distillation framework designed for setups with a large teacher-student capacity gap. Instead of relying solely on teacher-generated targets (which are often too complex for small students) or student-generated targets (which are often noisy), CoKD blends teacher and student log-probabilities during autoregressive decoding. The authors bound this process with a reachability constraint that filters out tokens with low student probability. Afterward, a task-aware quality filter compares the collaborative rollout to a student-only baseline using a reference-grounded metric, keeping only the targets that actually improve on the student's baseline. The student is then trained on these filtered targets using standard Cross-Entropy loss. Evaluations on GSM8K, Dolly, and SAMSum show solid gains over several KD baselines and good stability under large capacity gaps of up to 28x.

Critical strengths

Target Construction: The token-level collaborative decoding mechanism is an intuitive way to balance teacher preference against student capacity. Using a strict student-reachability constraint ensures the model only trains on trajectories it can actually learn.

Handling Capacity Gaps: Standard KD methods tend to break down when the teacher is much larger than the student, since the supervision moves entirely outside the student's capacity. CoKD remains stable and improves student performance even with a 28x capacity gap, validating the core design.

Ablation Studies: The ablations effectively isolate the key components. Showing that a simple Cross-Entropy loss on these filtered targets outperforms complex KL-divergence objectives (like Forward/Reverse KL or JSD) is a practical and useful finding for the field.

Critical weaknesses

Computational Overhead: The method requires simultaneous autoregressive generation from both the teacher and student, plus an extra student rollout for the quality filter. This creates a massive bottleneck in training memory and wall-clock time. The paper does not provide a detailed efficiency comparison against baselines like GKD or standard offline methods, making the cost-benefit tradeoff difficult to evaluate.

Missing Baselines: The paper evaluates against on-policy methods like SKD and GKD, but skips a few important comparisons. MiniLLM is excluded without explanation, despite being a standard Reverse-KL baseline for on-policy distillation. Furthermore, for the GSM8K experiments, the authors do not compare against Distilling Step-by-Step, which is the standard approach for transferring reasoning rationales.

Evaluation Scope: The experiments only cover GSM8K, Dolly, and SAMSum. Bypassing broad capability benchmarks like MMLU or HumanEval leaves an open question: does enforcing a strict reachability constraint cause catastrophic forgetting of general knowledge or degrade coding capabilities?

Questions for authors

Can you provide a detailed quantitative comparison of the computational cost (FLOPs, wall-clock time, peak memory) for CoKD versus the on-policy baselines like GKD and SKD? Is the accuracy gain sufficient to justify the concurrent generation overhead?

Why was MiniLLM excluded from the empirical baselines, given that it targets similar on-policy generation dynamics and capacity gap issues?

Since you evaluate on GSM8K, how does CoKD compare to Distilling Step-by-Step, which is specifically engineered to distill reasoning rationales?

How does the CoKD-distilled student perform on broad capability benchmarks like MMLU? Does the strict reachability constraint lead to catastrophic forgetting of general knowledge?

The sequence-level quality filter currently relies on a reference-grounded utility metric like Exact Match or ROUGE. How do you envision adapting CoKD for open-ended generation or alignment tasks where gold references are subjective or unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.