- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Student-Ascendant Knowledge Distillation (SaKD), a method to improve machine translation using smaller language models. Traditional knowledge distillation helps a student model learn from a larger teacher model, but it is difficult for the student to perform better than the teacher. To solve this problem, the authors propose three stages: Imitation, Reflection, and Transcendence. In the Imitation stage, the student learns from teacher-generated translations. In the Reflection stage, it learns to identify better translations. In the Transcendence stage, reinforcement learning helps it generate higher-quality translations. The authors test their method using Qwen3-8B and Qwen3-4B models for English, Chinese, and German translation tasks. The results show that SaKD performs better than traditional knowledge distillation methods and even outperforms larger teacher models on the evaluated benchmarks. However, the method depends on automatic translation quality scores, which may not always match human judgment. Overall, this paper shows that a smaller model can improve beyond simply copying its teachers by learning through different stages.
Critical strengths
- Introduces a coherent, human-learning-inspired, three-stage knowledge distillation framework that goes beyond conventional single-stage distillation.
- Addresses a timely problem: distilling strong machine translation capabilities from large LLMs into efficient students without being constrained by teacher ceilings.
- The high-level framework and motivation are easy to follow.
- Evaluations across four directions and two student sizes; consistent gains over KD baselines and strong teachers.
Critical weaknesses
- The “student surpasses teacher” claim may conflate method vs. training budget: teachers are not fine-tuned with the same massive pseudo-parallel data or RL pipeline, making the ceiling comparison potentially unfair.
- Missing baselines where teachers themselves are fine-tuned on the same pseudo data or undergo CPO/GRPO.
- Analysis of the sensitivity to the teacher set is somewhat limited.
- Human evaluation details (e.g., inter-annotator agreement, bootstrap/randomization procedure for significance) are not fully reported.
- More discussion relating to known limitations of LLM-as-judge and QE metrics (as highlighted by recent MQM/QE works) would contextualize risks of metric-induced biases.
Questions for authors
- How were teacher outputs generated: deterministic decoding or sampled? Were prompts and decoding parameters harmonized across teachers?
- Could you provide a baseline where the strongest teacher is fine-tuned on the same filtered pseudo-parallel data (and optionally subjected to CPO/GRPO with the same QE reward)?
- Could you please share compute cost (teacher generation token count, SFT/CPO/GRPO GPU hours) and any efficiency ablations (e.g., smaller groups for GRPO, fewer rollouts) to inform practical adoption?
4 comments
Response to Reviewer r3rz: Part 2/2
Quesiton 1: Teacher Output Generation and Decoding Consistency
To ensure reproducibility, we set do_sample=False for all three teacher models during translation generation. Additionally, all teachers use the same translation prompt, as provided in Appendix H (Figure 7), ensuring consistent generation settings across different models.
We have clarified these implementation details in the revised manuscript.
Quesiton 3: Computational Cost and Training Efficiency
To clarify the computational requirements of SaKD, we provide a breakdown of the training costs. Teacher knowledge generation produces approximately 0.62B output tokens. All student training experiments are conducted on 8 NVIDIA A800 GPUs, with approximate training times summarized below.
Computational Cost:
| Stage | Qwen3-8B (Hours) | Qwen3-4B (Hours) |
|---|---|---|
| Imitation (SFT) | ~28 | ~17 |
| Reflection (CPO) | ~7 | ~4 |
| Transcendence (GRPO) | ~24 | ~14 |
| Total | ~59 | ~35 |
GRPO Efficiency Analysis: We further investigate the trade-off between translation quality and computational cost by varying the GRPO group size on Qwen3-8B. All experiments are conducted on 8 NVIDIA A800 GPUs.
| Group Size | En-Zh BLEU | COMET | En-De BLEU | COMET | Training Hours |
|---|---|---|---|---|---|
| 4 | 33.81 | 88.31 | 31.27 | 87.11 | ~12 |
| 8 | 34.03 | 88.84 | 31.89 | 87.71 | ~18 |
| 16 (Ours) | 34.45 | 89.21 | 32.23 | 88.09 | ~24 |
The results show that larger group sizes generally improve translation quality at the expense of additional computation. Notably, reducing the group size from 16 to 8 decreases training time by approximately 25%, with relatively modest performance reductions of 0.42/0.37 BLEU/COMET on En-Zh and 0.34/0.38 BLEU/COMET on En-De. Further reducing the group size to 4 saves approximately 50% of training time, but results in larger performance degradation.
We have incorporated these computational details and efficiency analysis into the revised manuscript.
References
[1] Zouhar et al. (2024). Pitfalls and Outlooks in Using COMET. WMT 2024.
[2] Kim (2025). RUBRIC-MQM: Span-Level LLM-as-judge in Machine Translation For High-End Models. ACL 2025 Industry Track.
Thank you for providing detailed responses. While the study demonstrates the effective use of SLMs for machine translation, I remain somewhat concerned about its novelty and overall contributions. Nevertheless, I recognize the practical value of the work and remain inclined to support its acceptance. Therefore, I would like to maintain my original assessment.
Appreciation for Recognition of The Practical Value of Our Work!
We sincerely appreciate your recognition of the practical value of our work.
We would like to emphasize that SaKD aims to contribute a generalizable and efficient knowledge distillation framework while exploring an important direction beyond conventional distillation: enabling compact student models to surpass their substantially larger teachers. We provide methodological insights and empirical evidence to support this objective.
Although our current evaluation is limited to machine translation, we sincerely hope that this work can inspire further research on knowledge distillation across broader tasks and domains.
Once again, we are especially grateful for your recognition of the practical value of our work and your willingness to support its acceptance. Your encouragement means a great deal to us and further strengthens our motivation to pursue this research direction, make meaningful contributions to the research community!
Response to Reviewer r3rz: Part 1/2
We greatly appreciate the reviewer's thoughtful and constructive feedback, which has provided valuable insights for strengthening our work. We have carefully considered all the comments and provide our point-by-point responses below:
Weakness 1& 2 & Question 2: Fairness of Teacher-Surpassing Comparisons
In conventional knowledge distillation research, student models are typically trained using knowledge transferred from larger, fixed teacher models, and their performance is evaluated against these original teachers. The two recent baselines considered in our study, GRACE and Beyond the Mode, also follow this teacher–student evaluation paradigm.
Similarly, SaKD focuses on knowledge distillation, investigating whether a compact student can surpass its substantially larger knowledge providers through multi-teacher knowledge transfer and progressive student-side optimization. Therefore, we adopt the same evaluation setting, comparing the trained student with the original, fixed teacher models.
Weakness 3: Teacher-Set Sensitivity Analysis
To investigate the sensitivity of SaKD to teacher composition, we conducted additional experiments with three two-teacher combinations on Qwen3-8B and Qwen3-4B, covering En-Zh and En-De translation tasks:
The three-teacher configuration consistently achieves the best performance across both student scales and translation directions, outperforming the strongest two-teacher combination. Moreover, different teacher combinations exhibit varying strengths: Qwen3 + Llama performs best on En-Zh, whereas Llama + DeepSeek achieves better results on En-De. These findings suggest that SaKD benefits from complementary knowledge across teachers and remains effective under different teacher compositions.
We have incorporated this teacher-set sensitivity analysis into the revised manuscript.
Weakness 4: Human Evaluation and Statistical Significance
We thank the reviewer for this valuable suggestion. Our human evaluation involves two professional translators with over five years of experience, who independently evaluate 50 documents per translation direction (2,045 sentences in total) using Direct Assessment (DA) scores ranging from 0 to 100.
Inter-annotator agreement. We calculate the intraclass correlation coefficient (ICC(2,1)) to assess agreement between the two annotators. The ICC values are 0.89, 0.91, 0.90, and 0.93 for En-Zh, Zh-En, En-De, and De-En, respectively, indicating high inter-annotator agreement.
Statistical significance. We employ a document-level paired bootstrap test with 10,000 resampling iterations to assess the significance of DA improvements. Documents are used as the resampling unit to account for within-document dependencies, with a significance threshold of p < 0.01.
We have incorporated these evaluation details into Section 4.3 and Appendix F of the revised manuscript to improve transparency and reproducibility.
Weakness 5: Potential Biases in QE Metrics and LLM-as-a-Judge
We acknowledge that learned QE metrics and LLM-based judges may introduce biases, including sensitivity to training distributions, evaluation criteria, and scoring preferences (Zouhar et al., 2024; Kim, 2025).
To mitigate these risks, we have separated the training reward from the final evaluation metrics. Specifically, XCOMET-QE is used for knowledge filtering, preference construction, and GRPO rewards, while final performance is evaluated using reference-based COMET and BLEU. We further complement automatic metrics with GPT-4o-based evaluation and professional human assessment, including DA scores and document-level coherence and logical errors.
The consistent improvements across these evaluation approaches provide supporting evidence that SaKD's gains are not solely attributable to optimizing XCOMET-QE. We have expanded the discussion of these limitations and related MQM/QE studies in the revised manuscript.