acceptodds
Under review as a conference paper at ICLR 2027

Beyond Teachers: Student-Ascendant Knowledge Distillation for LLM-based Machine Translation

Abstract

Knowledge distillation (KD) can effectively compress a large machine translation (MT) model into a smaller variant while preserving performance, supporting deployment in constrained environments, especially under the context of large language model (LLM)-based MT. However, the smaller student model is intrinsically bounded by the performance ceiling of the teacher model in recent studies of KD. Yet for humans, students routinely outstrip their teachers through progressive, deliberate learning. In this paper, we rethink the process of KD and propose SaKD, tudent-scentant nowledge istillation, to break the performance ceiling of teacher model in KD for LLM-based MT via mimicking the human learning behaviors. SaKD first distills knowledge from multiple teachers, and then introduces a human-like three-stage learning process, , and —to guide the student model toward surpassing its teachers. Experiments on extensive MT benchmarks show the effectiveness of SaKD, significantly outperforming existing KD methods. And importantly, the student model with SaKD transcends the performance bound of the teacher model, achieving a complete performance surpass.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge Distillation, Machine Translation, Preference Optimization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r3rzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Student-Ascendant Knowledge Distillation (SaKD), a method to improve machine translation using smaller language models. Traditional knowledge distillation helps a student model learn from a larger teacher model, but it is difficult for the student to perform better than the teacher. To solve this problem, the authors propose three stages: Imitation, Reflection, and Transcendence. In the Imitation stage, the student learns from teacher-generated translations. In the Reflection stage, it learns to identify better translations. In the Transcendence stage, reinforcement learning helps it generate higher-quality translations. The authors test their method using Qwen3-8B and Qwen3-4B models for English, Chinese, and German translation tasks. The results show that SaKD performs better than traditional knowledge distillation methods and even outperforms larger teacher models on the evaluated benchmarks. However, the method depends on automatic translation quality scores, which may not always match human judgment. Overall, this paper shows that a smaller model can improve beyond simply copying its teachers by learning through different stages.

Critical strengths
  • Introduces a coherent, human-learning-inspired, three-stage knowledge distillation framework that goes beyond conventional single-stage distillation.
  • Addresses a timely problem: distilling strong machine translation capabilities from large LLMs into efficient students without being constrained by teacher ceilings.
  • The high-level framework and motivation are easy to follow.
  • Evaluations across four directions and two student sizes; consistent gains over KD baselines and strong teachers.
Critical weaknesses
  • The “student surpasses teacher” claim may conflate method vs. training budget: teachers are not fine-tuned with the same massive pseudo-parallel data or RL pipeline, making the ceiling comparison potentially unfair.
  • Missing baselines where teachers themselves are fine-tuned on the same pseudo data or undergo CPO/GRPO.
  • Analysis of the sensitivity to the teacher set is somewhat limited.
  • Human evaluation details (e.g., inter-annotator agreement, bootstrap/randomization procedure for significance) are not fully reported.
  • More discussion relating to known limitations of LLM-as-judge and QE metrics (as highlighted by recent MQM/QE works) would contextualize risks of metric-induced biases.
Questions for authors
  • How were teacher outputs generated: deterministic decoding or sampled? Were prompts and decoding parameters harmonized across teachers?
  • Could you provide a baseline where the strongest teacher is fine-tuned on the same filtered pseudo-parallel data (and optionally subjected to CPO/GRPO with the same QE reward)?
  • Could you please share compute cost (teacher generation token count, SFT/CPO/GRPO GPU hours) and any efficiency ablations (e.g., smaller groups for GRPO, fewer rollouts) to inform practical adoption?
4 comments
AuthorsOpenReview ↗

Response to Reviewer r3rz: Part 1/2

We greatly appreciate the reviewer's thoughtful and constructive feedback, which has provided valuable insights for strengthening our work. We have carefully considered all the comments and provide our point-by-point responses below:


Weakness 1& 2 & Question 2: Fairness of Teacher-Surpassing Comparisons

In conventional knowledge distillation research, student models are typically trained using knowledge transferred from larger, fixed teacher models, and their performance is evaluated against these original teachers. The two recent baselines considered in our study, GRACE and Beyond the Mode, also follow this teacher–student evaluation paradigm.

Similarly, SaKD focuses on knowledge distillation, investigating whether a compact student can surpass its substantially larger knowledge providers through multi-teacher knowledge transfer and progressive student-side optimization. Therefore, we adopt the same evaluation setting, comparing the trained student with the original, fixed teacher models.


Weakness 3: Teacher-Set Sensitivity Analysis

To investigate the sensitivity of SaKD to teacher composition, we conducted additional experiments with three two-teacher combinations on Qwen3-8B and Qwen3-4B, covering En-Zh and En-De translation tasks:

Teacher Set8B En-Zh8B En-De4B En-Zh4B En-De
Qwen3 + Llama33.92 / 88.7431.81 / 87.6332.18 / 86.8931.06 / 86.78
Qwen3 + DeepSeek33.48 / 88.2931.36 / 87.1231.69 / 86.3530.62 / 86.19
Llama + DeepSeek33.71 / 88.5131.92 / 87.7631.94 / 86.6431.18 / 86.92
SaKD (Three Teachers)34.45 / 89.2132.23 / 88.0932.56 / 87.2231.45 / 87.21

The three-teacher configuration consistently achieves the best performance across both student scales and translation directions, outperforming the strongest two-teacher combination. Moreover, different teacher combinations exhibit varying strengths: Qwen3 + Llama performs best on En-Zh, whereas Llama + DeepSeek achieves better results on En-De. These findings suggest that SaKD benefits from complementary knowledge across teachers and remains effective under different teacher compositions.

We have incorporated this teacher-set sensitivity analysis into the revised manuscript.


Weakness 4: Human Evaluation and Statistical Significance

We thank the reviewer for this valuable suggestion. Our human evaluation involves two professional translators with over five years of experience, who independently evaluate 50 documents per translation direction (2,045 sentences in total) using Direct Assessment (DA) scores ranging from 0 to 100.

Inter-annotator agreement. We calculate the intraclass correlation coefficient (ICC(2,1)) to assess agreement between the two annotators. The ICC values are 0.89, 0.91, 0.90, and 0.93 for En-Zh, Zh-En, En-De, and De-En, respectively, indicating high inter-annotator agreement.

Statistical significance. We employ a document-level paired bootstrap test with 10,000 resampling iterations to assess the significance of DA improvements. Documents are used as the resampling unit to account for within-document dependencies, with a significance threshold of p < 0.01.

We have incorporated these evaluation details into Section 4.3 and Appendix F of the revised manuscript to improve transparency and reproducibility.


Weakness 5: Potential Biases in QE Metrics and LLM-as-a-Judge

We acknowledge that learned QE metrics and LLM-based judges may introduce biases, including sensitivity to training distributions, evaluation criteria, and scoring preferences (Zouhar et al., 2024; Kim, 2025).

To mitigate these risks, we have separated the training reward from the final evaluation metrics. Specifically, XCOMET-QE is used for knowledge filtering, preference construction, and GRPO rewards, while final performance is evaluated using reference-based COMET and BLEU. We further complement automatic metrics with GPT-4o-based evaluation and professional human assessment, including DA scores and document-level coherence and logical errors.

The consistent improvements across these evaluation approaches provide supporting evidence that SaKD's gains are not solely attributable to optimizing XCOMET-QE. We have expanded the discussion of these limitations and related MQM/QE studies in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer r3rz: Part 2/2

Quesiton 1: Teacher Output Generation and Decoding Consistency

To ensure reproducibility, we set do_sample=False for all three teacher models during translation generation. Additionally, all teachers use the same translation prompt, as provided in Appendix H (Figure 7), ensuring consistent generation settings across different models.

We have clarified these implementation details in the revised manuscript.


Quesiton 3: Computational Cost and Training Efficiency

To clarify the computational requirements of SaKD, we provide a breakdown of the training costs. Teacher knowledge generation produces approximately 0.62B output tokens. All student training experiments are conducted on 8 NVIDIA A800 GPUs, with approximate training times summarized below.

Computational Cost:

StageQwen3-8B (Hours)Qwen3-4B (Hours)
Imitation (SFT)~28~17
Reflection (CPO)~7~4
Transcendence (GRPO)~24~14
Total~59~35

GRPO Efficiency Analysis: We further investigate the trade-off between translation quality and computational cost by varying the GRPO group size on Qwen3-8B. All experiments are conducted on 8 NVIDIA A800 GPUs.

Group SizeEn-Zh BLEUCOMETEn-De BLEUCOMETTraining Hours
433.8188.3131.2787.11~12
834.0388.8431.8987.71~18
16 (Ours)34.4589.2132.2388.09~24

The results show that larger group sizes generally improve translation quality at the expense of additional computation. Notably, reducing the group size from 16 to 8 decreases training time by approximately 25%, with relatively modest performance reductions of 0.42/0.37 BLEU/COMET on En-Zh and 0.34/0.38 BLEU/COMET on En-De. Further reducing the group size to 4 saves approximately 50% of training time, but results in larger performance degradation.

We have incorporated these computational details and efficiency analysis into the revised manuscript.


References

[1] Zouhar et al. (2024). Pitfalls and Outlooks in Using COMET. WMT 2024.

[2] Kim (2025). RUBRIC-MQM: Span-Level LLM-as-judge in Machine Translation For High-End Models. ACL 2025 Industry Track.

Thank you for providing detailed responses. While the study demonstrates the effective use of SLMs for machine translation, I remain somewhat concerned about its novelty and overall contributions. Nevertheless, I recognize the practical value of the work and remain inclined to support its acceptance. Therefore, I would like to maintain my original assessment.

AuthorsOpenReview ↗

Appreciation for Recognition of The Practical Value of Our Work!

We sincerely appreciate your recognition of the practical value of our work.

We would like to emphasize that SaKD aims to contribute a generalizable and efficient knowledge distillation framework while exploring an important direction beyond conventional distillation: enabling compact student models to surpass their substantially larger teachers. We provide methodological insights and empirical evidence to support this objective.

Although our current evaluation is limited to machine translation, we sincerely hope that this work can inspire further research on knowledge distillation across broader tasks and domains.

Once again, we are especially grateful for your recognition of the practical value of our work and your willingness to support its acceptance. Your encouragement means a great deal to us and further strengthens our motivation to pursue this research direction, make meaningful contributions to the research community!

Reviewer_MQdSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SaKD, a multi-teacher distillation method which is designed to help smaller language model to surpass their teachers in machine translation. It consists of three stages, including limitation via SFT, Reflection with CPO, and Transcendence via GRPO by using translations generated by the student. Experiment spans four translation directions and two student model scales.

Critical strengths
  1. The problem is well-motivated. The paper asks whether smaller language models can learn from larger translation models and improve on their performance. This is a practical goal for deployment under resource constraints.
  2. The proposed framework is coherent. Imitation, Reflection, and Transcendence have clear roles, and the motivation of each stage is good.
  3. The evaluation supports the main empirical claim. The experiment cover four translation directions and two model sizes, with proper design in metrics, evaluation, and component ablations.
Critical weaknesses
  1. The role of the teacher baseline needs clarification. In Eqs. (6) and (7), the best-teacher score is constant across candidates for a given source and therefore cancels out during centering. It is unclear how this term changes the optimization compared with using translation-quality scores directly.
  2. The current work needs recent baselines for comparison to proof the effectiveness of the claims.
  3. The scope of the task-agnostic claim could be clearer. The framework is presented as task-agnostic but its implementation relies on XCOMET-QE for filtering, preference construction, and reward computation. It would be helpful if specifying which parts carry over to other tasks and which require task-specific quality estimation.
Questions for authors

The integration of multi-teacher distillation with machine translation make sense and the motivation is straightforward. Could the authors clarify which aspects of SaKD constitute general principles and which depend on translation-specific supervision and quality estimation? In particular, does “task-agnostic” refer to the abstract three-stage learning structure? Since XCOMET-QE supports several key stages, clarifying this distinction would help establish the scope of the claimed contributions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MQdS

We sincerely appreciate the reviewer for the thorough evaluation of our work and the constructive comments and suggestions. Below, we address each concern and question point by point:


Weakness 1: Clarification of the role of the teacher baseline

When preparing the manuscript, we simplified the reward formulation to make the underlying teacher-surpassing motivation more intuitive to readers. However, we did not rigorously examine the mathematical implications of this simplification under GRPO normalization, inadvertently overlooking the cancellation effect.

In our actual implementation, we apply a Softplus-based nonlinear transformation to the teacher-surpass reward:

Rj=r(Si,T^ij)+log⁡(1+exp⁡(r(Si,T^ij)−max⁡(Ci))).R_j = r(S_i,\hat{T}_{ij}) + \log\left(1+\exp\left(r(S_i,\hat{T}_{ij})-\max(C_i)\right)\right).

To further validate its effectiveness, we conducted an additional ablation by removing the proposed teacher-relative transcendence reward, i.e., Rj=r(Si,T^ij)R_j = r(S_i,\hat{T}_{ij}), while keeping all other training settings unchanged:

Setting8B En-Zh BLEUCOMET8B En-De BLEUCOMET4B En-Zh BLEUCOMET4B En-De BLEUCOMET
SaKD (Multi-T)34.4589.2132.2388.0932.5687.2231.4587.21
w/o Transcendence Reward33.1187.6630.7886.8131.4486.4130.1186.12

The proposed reward consistently outperforms standard quality-reward GRPO across both student scales and translation directions. These results demonstrate the effectiveness of the proposed nonlinear teacher-relative reward beyond the contribution of GRPO optimization itself.

We have corrected Eq. (7) and provide a more detailed explanation of the actual reward computation and its role in the GRPO objective.


Weakness 2: Comparison with Recent Knowledge Distillation Methods

To alleviate this concern, we additionally reproduce two recent knowledge distillation methods: 1) GRACE: which improves knowledge distillation through gradient-based teacher selection; 2) Beyond the Mode:, which improves sequence-level distillation by exploiting diverse teacher-generated translations.

They represent two complementary directions in recent knowledge distillation research: teacher selection and knowledge diversity, both closely related to SaKD's multi-teacher knowledge utilization. We adapt their core algorithms to our LLM-based MT setting using the same student backbones, translation directions, and evaluation:

ModelEn-ZhZh-EnEn-DeDe-EnAverage
Qwen3-8B
GRACE32.72 / 87.0632.15 / 86.6830.42 / 85.8231.52 / 87.9331.70 / 86.87
Beyond the Mode32.93 / 87.5432.07 / 86.8430.65 / 86.7331.78 / 88.1431.86 / 87.31
SaKD (Multi-T)34.45 / 89.2133.56 / 87.9932.23 / 88.0933.45 / 89.5133.42 / 88.70
Qwen3-4B
GRACE31.35 / 85.8430.83 / 85.7529.16 / 84.5530.46 / 86.6130.45 / 85.69
Beyond the Mode31.56 / 86.1831.04 / 86.1229.61 / 84.8830.54 / 87.0630.69 / 86.06
SaKD (Multi-T)32.56 / 87.2231.78 / 87.1131.45 / 87.2132.12 / 88.7731.98 / 87.58

Results are reported as BLEU / COMET.

Compared with the stronger baseline in average performance, Beyond the Mode, SaKD achieves improvements of +1.56 BLEU and +1.39 COMET on Qwen3-8B, and +1.29 BLEU and +1.52 COMET on Qwen3-4B. These findings further validate the effectiveness of SaKD compared with recent teacher-selection and knowledge-diversity-based distillation approaches.

We have incorporated these additional comparisons and the corresponding implementation details into the revised manuscript.


Weakness 3: Clarification of Task-Agnostic Generalizability

We clarify that task-agnostic refers to SaKD's overall learning framework, including multi-teacher knowledge generation and the three-stage learning process (SFT, CPO, and GRPO), rather than its specific quality evaluator.

While XCOMET-QE is used for knowledge filtering, preference construction, and reward computation in MT, it can be replaced with appropriate task-specific evaluators for other tasks, such as correctness verifiers for mathematical reasoning or execution-based evaluators for code generation.

We have clarified this distinction and refined the task-agnostic claim in the revised manuscript.


References

[1] Panigrahi, A., Liu, B., Malladi, S., Kakade, S., and Goel, S. (2026). In Good GRACES: Principled Teacher Selection for Knowledge Distillation. ICLR 2026.

[2] Galiano-Jiménez, A., Pérez-Ortiz, J. A., Sánchez-Martínez, F., and Sánchez-Cartagena, V. M. (2025). Beyond the Mode: Sequence-Level Distillation of Multilingual Translation Models for Low-Resource Language Pairs. Findings of NAACL 2025, 6676–6691.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Its End

Dear Reviewer MQdS,

Thank you for your thoughtful review and constructive feedback. We have carefully addressed all your concerns and questions in our official responses and incorporated the corresponding revisions into the manuscript.

As the discussion period is approaching its end, we would sincerely appreciate it if you could take a moment to review our responses and let us know whether they have adequately addressed your concerns. We would be happy to provide any further clarification if needed.

Thank you again for your valuable time and consideration.

Sincerely, The Authors

Reviewer_1UddOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SaKD combines translations from three large teacher models with three stages of student training: supervised fine-tuning, contrastive preference optimization using XCOMET-QE rankings, and GRPO using a quality reward offset by the best teacher's score. Experiments cover Qwen3-8B and Qwen3-4B students on English-Chinese and English-German translation in both directions. The reported 8B results exceed the individual teachers on automatic metrics and human direct assessment; ablations examine the contribution of each training stage.

Critical strengths
  1. The 8B results agree across automatic metrics and professional human evaluation. Table 1 reports average BLEU/COMET scores of 33.42/88.70, compared with 32.23/87.48 for the strongest teacher on average. Table 2 also reports higher human direct-assessment scores than the best teacher in each direction. This agreement supports the reported improvement in translation quality. [pp. 5 to 7, Tables 1 to 2]

  2. The paper compares full single- and multi-teacher variants, a best-teacher-output SFT baseline, and variants without CPO or GRPO. These comparisons consistently favor the complete pipeline, supporting the training recipe across more than one teacher-student comparison. [pp. 6 to 7, §4.1 and Table 3]

Critical weaknesses
  1. For a fixed source document, write qj=r(Si,T^ij)q_j=r(S_i,\hat T_i^j) and bi=max⁡(Ci)b_i=\max(C_i). Equations (6) to (7) imply

    =\frac{q_j-\bar q}{\operatorname{std}(q)}.$$ For the same sampled group, subtracting the best teacher's score therefore produces exactly the advantages obtained from the unshifted quality reward. The proposed offset has no effect under this normalization. The earlier stages remain teacher-dependent, and quality-reward GRPO can still improve translations. However, the offset does not supply the claimed teacher-surpassing signal. Table 3 establishes the benefit of the GRPO stage, not of this offset, leaving the paper's central mechanistic contribution substantially weakened. [p. 5, §3.4, Eqs. (5) to (7)]
  2. For 4B SaKD (Multi-T), the four BLEU values average to 31.9775; Table 1 reports 32.48. The corrected mean falls below the strongest teacher's 32.23, changing that particular teacher-surpassing comparison. The 4B KD (Single-T) and SaKD (Single-T) means are also 28.8925 and 31.06, rather than the reported 29.02 and 31.31. These discrepancies with the stated averaging convention require reconciliation. They do not overturn the 8B result. [p. 5, Table 1]

Questions for authors
  1. Does the implemented GRPO reward include any teacher-dependent transformation or group construction beyond the constant subtraction in Eq. (7)? Please specify the actual reward and normalization, and clarify whether Eq. (5)'s denominator is the rollout policy or the fixed CPO checkpoint. These details would determine whether the cancellation applies to the implemented objective.
  2. Which entries in the three affected 4B rows are correct: the direction-level scores or the reported averages? Please reconcile them and identify which smaller-student teacher-surpassing comparisons remain supported.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Udd

We really appreciate your precious time in reviewing our paper and providing helpful comments. We have carefully studied all the comments and tried our best to address every one of them. Below, we address each concern and question point by point:


Weakness1 & Question 1: Effectiveness and Implementation of the Teacher-Relative Reward

We sincerely appreciate the reviewer for identifying this important issue and raising these implementation questions.

(1) Clarification of the actual reward implementation and normalization in Eqs. (6)–(7):

When preparing the manuscript, we simplified the reward formulation to make the underlying teacher-surpassing motivation more intuitive to readers. However, we did not rigorously examine the mathematical implications of this simplification under GRPO normalization, leading to the cancellation issue identified by the reviewer.

In our actual implementation, we employ a Softplus-based transformation to teacher-surpassing reward:

Rj=r(Si,T^ij)+log⁡(1+exp⁡(r(Si,T^ij)−max⁡(Ci))).R_j = r(S_i,\hat{T}_{ij}) + \log\left(1+\exp\left(r(S_i,\hat{T}_{ij})-\max(C_i)\right)\right).

The corresponding advantage is computed using group-wise normalization:

Aj=Rj−mean⁡(R)std⁡(R).A_j = \frac{R_j-\operatorname{mean}(\mathbf{R})}{\operatorname{std}(\mathbf{R})}.

To further validate the effectiveness of teacher-surpassing reward, we conducted an additional ablation by replacing the proposed teacher-relative reward with the standard translation-quality reward (XComet-QE), i.e., Rj=r(Si,T^ij)R_j = r(S_i,\hat{T}_{ij}) , while keeping all other training settings unchanged.

Setting8B En-Zh BLEUCOMET8B En-De BLEUCOMET4B En-Zh BLEUCOMET4B En-De BLEUCOMET
SaKD (Multi-T)34.4589.2132.2388.0932.5687.2231.4587.21
w/o Teacher-relative Reward33.1187.6630.7886.8131.4486.4130.1186.12

The proposed reward consistently outperforms standard quality-reward GRPO across both student scales and translation directions. These results demonstrate the effectiveness of the nonlinear teacher-relative reward beyond the contribution of GRPO optimization itself.

(2) Clarification of the GRPO objective in Eq. (5):

We apologize for the confusion caused by the unclear formulation of Eq. (5). In our implementation, we initialize the rollout policy to the CPO checkpoint, and subsequent rollouts use the previous GRPO checkpoint as the old policy πold\pi_{\mathrm{old}}. We rewrite Eq. (5) to mitigate confusion:

Lgrpo=−1c∑j=1cπgrpo(T^ij∣Si)πold(T^ij∣Si)Aj+βDKL(πcpo∥πgrpo)\mathcal{L}_{\mathrm{grpo}} = -\frac{1}{c}\sum_{j=1}^{c}\frac{\pi_{\mathrm{grpo}}(\hat{T}_{ij}\mid S_i)}{\pi_{\mathrm{old}}(\hat{T}_{ij}\mid S_i)}A_j + \beta D_{\mathrm{KL}}(\pi_{\mathrm{cpo}}\Vert\pi_{\mathrm{grpo}})

Here, πold\pi_{\mathrm{old}} is the dynamically updated rollout policy, whereas πcpo\pi_{\mathrm{cpo}} remains fixed as the reference policy for KL regularization.

We sincerely appreciate the reviewer's careful examination again and have corrected Eqs. (5)–(7) and clarified the corresponding implementation details in the revised version.


Weakness 2 & Question 2: Which Entries in the Three affected 4B Rows are Correct

We thank the reviewer for this careful observation. The direction-level scores are correct, and the inconsistencies arise solely from manually calculated averages that were inadvertently not updated during collaborative revisions with other co-authors. We have thoroughly rechecked all experimental results and corrected the affected averages in the revised manuscript.

Specifically, the corrected average BLEU scores for 4B KD (Single-T), SaKD (Single-T), and SaKD (Multi-T) are 28.89, 31.06, and 31.98, respectively. Although the 4B SaKD (Multi-T) student shows weaker teacher-surpassing performance than its 8B counterpart and does not outperform the strongest teacher in average BLEU (31.98 vs. 32.23), it still exceeds the average BLEU performance of the three teachers (31.98 vs. 31.63). Moreover, it slightly surpasses the strongest teacher in average COMET (87.58 vs. 87.48) and achieves superior results in specific directions, such as En→De BLEU (31.45 vs. 30.89) and De→En COMET (88.77 vs. 88.56).


We would be happy to address any further questions or concerns the reviewer may have.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Its End

Dear Reviewer 1Udd,

Thank you for your thoughtful review and constructive feedback. We have carefully addressed all your concerns and questions in our official responses and incorporated the corresponding revisions into the manuscript.

As the discussion period is approaching its end, we would sincerely appreciate it if you could take a moment to review our responses and let us know whether they have adequately addressed your concerns. We would be happy to provide any further clarification if needed.

Thank you again for your valuable time and consideration.

Sincerely,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revision

We sincerely appreciate all reviewers for their constructive feedback. In response, we have substantially strengthened the manuscript through methodological clarifications, additional experiments, and improved evaluation and reproducibility analyses. All revisions are highlighted in blue in the revised manuscript. The major concerns and corresponding revision are summarized below:

  1. Reward Formulation and GRPO Implementation (Reviewer 1Udd, MQdS). We updated Eqs. (5)–(7) to clarify teacher-relative reward and GRPO policy updates (Section 3.4). We further conducted reward ablations on Qwen3-8B/4B to validate its effectiveness beyond standard GRPO (Section 4.4, Table 3).

  2. Numerical Corrections and Teacher-Surpassing Claims (Reviewer 1Udd, r3rz). We updated the average BLEU scores (Table 1) and refined the corresponding performance analysis, particularly for Qwen3-4B (Section 4.2). We also clarified that teacher-surpassing refers to outperforming the original fixed teachers rather than equally fine-tuned teachers.

  3. Additional Baseline Comparisons (Reviewer MQdS). We reproduced two recent knowledge distillation methods, GRACE (ICLR 2026) and Beyond the Mode (NAACL 2025), on Qwen3-8B/4B across four translation directions (Sections 4.1–4.2, Table 1), with implementation details provided in Appendix D.

  4. Teacher-Set Sensitivity (Reviewer r3rz). We evaluated three two-teacher combinations against the full three-teacher configuration on Qwen3-8B/4B, demonstrating the benefits of complementary teacher knowledge (Appendix E, Table 7).

  5. Evaluation Reliability and Generalizability (Reviewer MQdS, r3rz). We clarified the generalizability of the three-stage framework and the need for task-specific quality evaluators (Section 1). We also reported inter-annotator agreement and document-level bootstrap significance testing (Section 4.3, Table 2, Appendix F) and expanded the discussion of potential biases in QE metrics and LLM-based evaluation (Section 4.1, Appendix G).

  6. Reproducibility and Computational Efficiency (Reviewer r3rz). We clarified teacher decoding settings and shared translation prompts (Section 4.1, Appendix H), reported teacher-generation volume and stage-wise training costs on eight NVIDIA A800 GPUs (Appendix C, Table 5), and added GRPO group-size experiments to examine the trade-off between translation quality and training efficiency (Appendix C, Table 6).

We believe these revisions directly address the reviewers' major concerns and substantially improve the rigor, clarity, and reproducibility of our work. We would be happy to continue the discussion if there are any further questions or comments.

AuthorsOpenReview ↗

Summary of Discussion

We sincerely thank the Area Chair and all reviewers for their time and efforts throughout the review process. We particularly appreciate Reviewer r3rz's recognition of the practical value of our work and continued support for its acceptance. Although the other two reviewers have not participated in the discussion, we have carefully addressed their concerns through detailed point-by-point responses, additional experiments, and manuscript revisions. We believe these efforts have substantially addressed their major concerns.

We would like to reiterate three key contributions of our work:

  • A novel human-inspired distillation framework: Inspired by how human students learn from, reflect on, and eventually surpass their teachers, SaKD introduces a progressive knowledge distillation framework that moves beyond conventional knowledge imitation toward student capability advancement.

  • Teacher-surpassing capability through progressive optimization: By integrating imitation, reflection, and transcendence with a teacher-surpassing-aware reward mechanism, SaKD explicitly encourages compact student models to develop capabilities beyond those of their substantially larger teachers.

  • Comprehensive empirical validation: Extensive experiments across multiple translation directions, student scales, and competitive baselines demonstrate SaKD's effectiveness. Notably, our 8B student surpasses the strongest 70B–80B teacher in average BLEU and COMET, supported by extensive ablations and additional analyses.

Although our current evaluation focuses on machine translation, we believe the proposed SaKD offers a promising perspective that may inspire broader advances in knowledge distillation research.

Finally, we sincerely thank the Area Chair and reviewers again for their valuable time and consideration. We respectfully hope the final assessment will carefully consider our clarifications, additional evidence, and the potential contributions of SaKD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.