- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CONTRA-KD, an auxiliary feature-level objective for LLM distillation. A time- and layer-conditioned rectified-flow network learns corrections from projected student representations to corresponding teacher representations. These corrections generate detached targets with progressively increasing step sizes, providing curriculum-based supervision alongside existing distillation objectives. The method also supports different tokenizers through character-boundary span alignment. Experiments across several teacher–student pairs report improvements in average ROUGE-L and pairwise response preferences, supplemented by component ablations, a small human evaluation, and training-cost measurements.
Critical strengths
- A useful flow-matching perspective on feature distillation. The paper models the correction between aligned student and teacher representations through a learned velocity field. This perspective leads to a concrete training mechanism: generating intermediate feature targets with progressively increasing step sizes. The combination provides a flexible way to guide students toward teacher representations and is compatible with existing distillation objectives.
- Consistent gains supported by relevant ablations. CONTRA-KD improves average performance when combined with several output-level distillation methods and the feature-based FDD baseline across multiple teacher–student pairs. Comparisons against direct interpolation and variants without the curriculum further support the usefulness of its learned guidance and progressive supervision.
Critical weaknesses
- The ablations do not fully isolate the contribution of continuous-time flow matching. Student targets use only the velocity prediction at , while Table 4 compares the learned field against direct interpolation without an auxiliary predictor. A capacity-matched network trained directly to predict teacher–student feature residuals, using the same curriculum and adaptation schedule, is missing. Thus, the evidence establishes the benefit of learned correction but leaves the specific benefit of continuous-time training unclear.
- The evaluation focuses on five instruction-generation benchmarks, primarily using ROUGE-L and pairwise preferences. Without dedicated evaluations of mathematical reasoning or code generation, it remains unclear whether the improvements extend to capabilities with objectively verifiable correctness. This limits the evidence for broader capability transfer.
Questions for authors
- How does uniform-time flow matching compare with training the same correction network only at , holding network capacity, curriculum, and adaptation schedule fixed? This comparison would help determine the contribution of continuous-time training to the reported gains.
- Do the gains persist on mathematical reasoning or code generation benchmarks with objective correctness metrics? A representative controlled comparison using the existing models would help assess transfer beyond the current instruction-generation evaluation.
3 comments
Thanks for the thoughtful response and the additional experiments. The new ablations in your response and Appendix F.5 (comparing against and the MLP regressor) have addressed my first concern regarding the contribution of continuous-time flow matching. Regarding the second point, your explanation about the base models' limited capabilities on math and code benchmarks is understandable. For the final version, it would be great to include these observations and note the lack of evaluation on objectively verifiable tasks as a limitation in the paper. My score remains unchanged.
Thank you very much for your feedback. We will include these observations and clarify this limitation in the final version of the paper. If there is anything that you need clarified, we would be happy to elaborate!
Thank you for your thoughtful review and constructive feedback. We address the concerns as follow:
W1 & Q1: Benefit of continuous-time training
To isolate the necessity of training across continuous flow time t∼U(0,1), we evaluated a variant where the velocity field vθ is trained strictly at t=0 (i.e., parameterizing vθ(hS,t=0,j)≈hT−hS) with matched capacity, curriculum and adaption schedule. The details are in our revised Appendix F.5. Below are the empirical results averaged across 5 random seeds on the GPT-2 (1.5B→0.1B) setup across both DistiLLM and FDD base distillation objectives:
Table 1: Comparison of Flow Time Sampling (t∼U(0,1) (Ours) vs. Static t=0) on GPT-2 (1.5B→0.1B)
As shown in Table 1, restricting velocity training to t=0 causes a consistent degradation across all benchmarks. This confirms that sampling continuous flow time t∼U(0,1) provides essential flow-matching regularization, enabling the velocity field to generalize as student representations shift during optimization. We hypothesize that supervision along the interpolation path provides a more robust correction field as student representations evolve during optimization.
W2 & Q2: Extension of current models to math and code generation benchmark
We further evaluated the existing pretrained, supervised fine-tuned (SFT), and distilled GPT-2 and OpenLLaMA2 models on HumanEval and MBPP for code generation, and GSM8K and MATH for mathematical reasoning. The pretrained base models already exhibited limited capabilities on these benchmarks, particularly GPT-2, whose pretrained 0.1B and 1.5B variants achieved only approximately 0.4% and 1.7% accuracy on the math benchmarks, respectively, while performing near floor on code generation. The SFT and distilled models similarly showed limited performance, with no consistent improvements over their pretrained counterparts.
These results suggest that instruction-following distillation alone is insufficient to establish mathematical reasoning and coding capabilities when the underlying base models have limited proficiency in these domains. Since our current training primarily targets general instruction following rather than domain-specific reasoning or code generation, we leave the extension of CONTRA-KD to stronger base models and specialized math/code distillation settings for future work.