acceptodds
Under review as a conference paper at ICLR 2027

CONTRA-KD: Continuous Trajectory Alignment for Knowledge Distillation

Abstract

While Knowledge Distillation (KD) enables efficient compression of LLMs, current methods focus on matching discrete layer outputs, which fails to capture the continuous, dynamic nature of feature evolution within these models. We propose CONTRA-KD, a new paradigm for KD that reframes the process through continuous-time dynamics and the ordinary differential equations (ODE)-based nature of LLMs. We first learn a student-to-teacher correction flow in feature space using Rectified Flow, which provides a geometrically simple and stable transport path between intermediate representations. We then introduce a curriculum-based distillation strategy that guides the student to progressively follow this flow, targeting dynamic intermediate targets throughout the training process. This design enables the student to align not only with the teacher’s representations but also with its underlying feature dynamics, while remaining orthogonal and complementary to standard output-level distillation objectives. Extensive experiments across multiple teacher–student pairs and instruction-following benchmarks demonstrate that CONTRA-KD consistently improves distillation performance and training stability over strong baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_tza2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CONTRA-KD, an auxiliary feature-level objective for LLM distillation. A time- and layer-conditioned rectified-flow network learns corrections from projected student representations to corresponding teacher representations. These corrections generate detached targets with progressively increasing step sizes, providing curriculum-based supervision alongside existing distillation objectives. The method also supports different tokenizers through character-boundary span alignment. Experiments across several teacher–student pairs report improvements in average ROUGE-L and pairwise response preferences, supplemented by component ablations, a small human evaluation, and training-cost measurements.

Critical strengths
  1. A useful flow-matching perspective on feature distillation. The paper models the correction between aligned student and teacher representations through a learned velocity field. This perspective leads to a concrete training mechanism: generating intermediate feature targets with progressively increasing step sizes. The combination provides a flexible way to guide students toward teacher representations and is compatible with existing distillation objectives.
  2. Consistent gains supported by relevant ablations. CONTRA-KD improves average performance when combined with several output-level distillation methods and the feature-based FDD baseline across multiple teacher–student pairs. Comparisons against direct interpolation and variants without the curriculum further support the usefulness of its learned guidance and progressive supervision.
Critical weaknesses
  1. The ablations do not fully isolate the contribution of continuous-time flow matching. Student targets use only the velocity prediction at t=0t=0, while Table 4 compares the learned field against direct interpolation without an auxiliary predictor. A capacity-matched network trained directly to predict teacher–student feature residuals, using the same curriculum and adaptation schedule, is missing. Thus, the evidence establishes the benefit of learned correction but leaves the specific benefit of continuous-time training unclear.
  2. The evaluation focuses on five instruction-generation benchmarks, primarily using ROUGE-L and pairwise preferences. Without dedicated evaluations of mathematical reasoning or code generation, it remains unclear whether the improvements extend to capabilities with objectively verifiable correctness. This limits the evidence for broader capability transfer.
Questions for authors
  1. How does uniform-time flow matching compare with training the same correction network only at t=0t=0, holding network capacity, curriculum, and adaptation schedule fixed? This comparison would help determine the contribution of continuous-time training to the reported gains.
  2. Do the gains persist on mathematical reasoning or code generation benchmarks with objective correctness metrics? A representative controlled comparison using the existing models would help assess transfer beyond the current instruction-generation evaluation.
3 comments

Thank you for your thoughtful review and constructive feedback. We address the concerns as follow:

W1 & Q1: Benefit of continuous-time training

To isolate the necessity of training across continuous flow time t∼U(0,1)t \sim U(0,1), we evaluated a variant where the velocity field vθv_\theta is trained strictly at t=0t=0 (i.e., parameterizing vθ(hS,t=0,j)≈hT−hSv_\theta(h_S, t=0, j) \approx h_T - h_S) with matched capacity, curriculum and adaption schedule. The details are in our revised Appendix F.5. Below are the empirical results averaged across 5 random seeds on the GPT-2 (1.5B→0.1B1.5\text{B} \to 0.1\text{B}) setup across both DistiLLM and FDD base distillation objectives:

Table 1: Comparison of Flow Time Sampling (t∼U(0,1)t \sim U(0,1) (Ours) vs. Static t=0t=0) on GPT-2 (1.5B→0.1B1.5B \to 0.1B)

MethodDollySelf-InstVicunaS-NIAvg.
DistiLLM + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)26.22 ±\pm 0.3112.99 ±\pm 0.5317.41 ±\pm 0.3025.07 ±\pm 0.2120.42
DistiLLM + CONTRA, t=0t=025.31 ±\pm 0.3411.97 ±\pm 0.3616.82 ±\pm 0.4223.21 ±\pm 0.0819.33
FDD + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)25.12 ±\pm 0.4311.46 ±\pm 0.3816.05 ±\pm 0.2920.21 ±\pm 0.2218.21
FDD + CONTRA, t=0t=024.88 ±\pm 0.2010.83 ±\pm 0.5215.56 ±\pm 0.4119.12 ±\pm 0.1617.60

As shown in Table 1, restricting velocity training to t=0t=0 causes a consistent degradation across all benchmarks. This confirms that sampling continuous flow time t∼U(0,1)t \sim U(0,1) provides essential flow-matching regularization, enabling the velocity field to generalize as student representations shift during optimization. We hypothesize that supervision along the interpolation path provides a more robust correction field as student representations evolve during optimization.

W2 & Q2: Extension of current models to math and code generation benchmark

We further evaluated the existing pretrained, supervised fine-tuned (SFT), and distilled GPT-2 and OpenLLaMA2 models on HumanEval and MBPP for code generation, and GSM8K and MATH for mathematical reasoning. The pretrained base models already exhibited limited capabilities on these benchmarks, particularly GPT-2, whose pretrained 0.1B and 1.5B variants achieved only approximately 0.4% and 1.7% accuracy on the math benchmarks, respectively, while performing near floor on code generation. The SFT and distilled models similarly showed limited performance, with no consistent improvements over their pretrained counterparts.

These results suggest that instruction-following distillation alone is insufficient to establish mathematical reasoning and coding capabilities when the underlying base models have limited proficiency in these domains. Since our current training primarily targets general instruction following rather than domain-specific reasoning or code generation, we leave the extension of CONTRA-KD to stronger base models and specialized math/code distillation settings for future work.

Thanks for the thoughtful response and the additional experiments. The new ablations in your response and Appendix F.5 (comparing t∼U(0,1)t \sim U(0, 1) against t=0t=0 and the MLP regressor) have addressed my first concern regarding the contribution of continuous-time flow matching. Regarding the second point, your explanation about the base models' limited capabilities on math and code benchmarks is understandable. For the final version, it would be great to include these observations and note the lack of evaluation on objectively verifiable tasks as a limitation in the paper. My score remains unchanged.

Thank you very much for your feedback. We will include these observations and clarify this limitation in the final version of the paper. If there is anything that you need clarified, we would be happy to elaborate!

Reviewer_VbVCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses LLM knowledge distillation, arguing that existing feature-level methods match hidden states only at a few layer checkpoints and leave the layer-to-layer transformation of the teacher unconstrained. It proposes CONTRA-KD, which learns a rectified-flow velocity field that maps each student hidden state to its paired teacher hidden state and trains the student toward a one-step target along this field, with a step size that grows as a curriculum. The method achieves ROUGE-L gains of about 0.3–1.2 points when added to DistiLLM, DistiLLM-2, FDD and CSD on GPT-2, OpenLLaMA2 and LLaMA2 pairs.

Critical strengths

The method is an easy-to-adopt add-on that consistently raises average ROUGE-L across four baselines and three model pairs, although the gains are small.

Critical weaknesses
  • The method does not address the problem it motivates, and the theory does not close this gap. The introduction argues that prior feature KD leaves the transformation between neighboring layers unconstrained. However, the learned flow runs from a student state to its paired teacher state at the same layer pair (Eq. 6–8), and no term compares the student's layer-to-layer change with the teacher's, so CONTRA-KD is a smoothed form of layer-wise state matching. Theorem 3.1 does not separate it from prior work. If matching states bounds the dynamics difference, prior feature KD already transfers the teacher's dynamics. If it does not, the same applies to CONTRA-KD, which also matches states at a few layer pairs.

  • Key ablations are missing. The velocity field is never compared with a plain MLP that predicts the teacher state from the student state, and the curriculum is never compared with simply ramping up the loss weight. Without these, it is unclear whether the flow or the curriculum is what helps.

Questions for authors
  • Can you show that CONTRA-KD brings the student's layer-to-layer changes closer to the teacher's than plain layer-wise state matching or FDD does?
  • How does CONTRA-KD compare with (a) a velocity field trained only at t=0 and (b) a plain MLP regressor from h_S to h_T, both with the same curriculum?
  • How does the curriculum compare with a fixed k=1 plus a loss-weight ramp-up?
5 comments

We thank the reviewer for the insightful comments. We update the paper revision and address each concerns as follows:

W1 & Q1: Connection between CONTRA-KD and feature dynamics

We agree that the implication from trajectory alignment to dynamics alignment is not exclusive to CONTRA-KD. Theorem 3.1 serves as a general theoretical foundation motivating trajectory-based distillation, rather than a result intended to distinguish our method from all prior feature KD approaches. However, this does not make CONTRA-KD merely a smoothed form of layer-wise state matching. Our method addresses the lack of explicit modeling of teacher-directed feature correction through a learned continuous flow. Although this flow is distinct from the models' depth-wise dynamics, we show both theoretically and empirically that flow-guided supervision can improve alignment of their discrete feature transitions.

(1) Connection to depth-wise dynamics. Although our flow transports student features toward their corresponding teacher representations, its correction targets are directly related to depth-wise dynamics. Specifically, letting cj=tj−sjc_j=t_j-s_j denote the target correction at aligned layer jj, we have the exact identity:

(tj+1−tj)−(sj+1−sj)=cj+1−cj.(t_{j+1}-t_j)-(s_{j+1}-s_j)=c_{j+1}-c_j.

Thus, the variation of correction targets across depth encodes the discrepancy between student and teacher feature transitions. Furthermore, we establish a new objective-level result in revised Appendix B.2, showing that the discrepancy between normalized layer-to-layer changes is bounded by the actual CONTRA loss, curriculum discrepancy, and learned flow prediction errors. This provides a direct mathematical connection between our training objective and implicit discrete dynamics alignment, without explicitly matching finite differences.

(2) Beyond smoothed feature matching. Our method distinction lies in the optimization mechanism. Conventional feature KD directly supervises fixed representation targets, whereas CONTRA-KD learns a shared, state-conditioned continuous velocity field through Rectified Flow, constructs intermediate targets from its predicted corrections. Moreover, this distinction is supported by our ablations. Replacing the learned velocity field with direct interpolation under the same curriculum results in inferior performance, demonstrating that progressive interpolation alone cannot explain the effectiveness of CONTRA-KD. Similarly, applying curriculum learning directly to FDD does not achieve the same performance as CONTRA-KD, further highlighting the contribution of the learned correction field beyond curriculum scheduling alone.

(3) Direct dynamics validation. To further verify this connection, we measure student–teacher transition MSE across five aligned layer intervals in Appendix F.1. The new results in revised Figure 5 (Appendix F.1) show that FDD + CONTRA generally reduces transition discrepancies compared with FDD alone, even though FDD already explicitly supervises feature dynamics. This provides empirical evidence that our learned correction flow offers complementary guidance for feature evolution.

Overall, while Theorem 3.1 establishes the general motivation, we further provide objective-level analysis to explain how CONTRA-KD relates to discrete dynamics alignment, and the ablation studies demonstrate that its learned-flow mechanism provides benefits beyond feature matching. We have clarified this distinction and the scope of our theoretical claims in the revised manuscript.

W2 & Q2: Comparison with velocity field trained at t=0t=0 and plain MLP that predicts teacher states

a) To isolate the necessity of training across continuous flow time t∼U(0,1)t \sim U(0,1), we evaluated a variant where the velocity field vθv_\theta is trained strictly at t=0t=0 (i.e., parameterizing vθ(hS,t=0,j)≈hT−hSv_\theta(h_S, t=0, j) \approx h_T - h_S). The details are in our revised Appendix F.5. Below are the empirical results averaged across 5 random seeds on the GPT-2 (1.5B→0.1B1.5\text{B} \to 0.1\text{B}) setup across both DistiLLM and FDD base distillation objectives:

Table 1: Comparison of Flow Time Sampling (t∼U(0,1)t \sim U(0,1) (Ours) vs. Static t=0t=0) on GPT-2 (1.5B→0.1B1.5B \to 0.1B)

MethodDollySelf-InstVicunaS-NIAvg.
DistiLLM + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)26.22 ±\pm 0.3112.99 ±\pm 0.5317.41 ±\pm 0.3025.07 ±\pm 0.2120.42
DistiLLM + CONTRA, t=0t=025.31 ±\pm 0.3411.97 ±\pm 0.3616.82 ±\pm 0.4223.21 ±\pm 0.0819.33
FDD + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)25.12 ±\pm 0.4311.46 ±\pm 0.3816.05 ±\pm 0.2920.21 ±\pm 0.2218.21
FDD + CONTRA, t=0t=024.88 ±\pm 0.2010.83 ±\pm 0.5215.56 ±\pm 0.4119.12 ±\pm 0.1617.60

As shown in Table 1, restricting velocity training to t=0t=0 causes a consistent degradation across all benchmarks. These results demonstrate the benefit of training across continuous flow time rather than only at the student endpoint. We hypothesize that supervision along the interpolation path provides a more robust correction field as student representations evolve during optimization.

b) To test whether the flow formulation is necessary, we compare it with a plain MLP teacher-state regressor using a residual parameterization, h^T=hS+gϕ(hS)\hat{h}_T = h_S + g_\phi(h_S). The regressor is trained to predict the corresponding teacher representation, equivalently minimizing the correction prediction error. We then construct the same curriculum target hS+k(h^T−hS)h_S + k(\hat{h}_T - h_S), retaining the curriculum schedule, adaptation frequency, and loss weights.

Table 2: Comparison of the flow-matching velocity field (Ours) vs. a plain MLP regressor on GPT-2 (1.5B→0.1B1.5B \to 0.1B). Avg. is over the four datasets shown.

MethodDollySelf-InstVicunaS-NIAvg.
DistiLLM + CONTRA (Ours)26.22 ±\pm 0.3112.99 ±\pm 0.5317.41 ±\pm 0.3025.07 ±\pm 0.2120.42
DistiLLM + MLP regressor25.08 ±\pm 0.2912.69 ±\pm 0.3516.45 ±\pm 0.4923.98 ±\pm 0.2119.55
FDD + CONTRA (Ours)25.12 ±\pm 0.4311.46 ±\pm 0.3816.05 ±\pm 0.2920.21 ±\pm 0.2218.21
FDD + MLP regressor24.17 ±\pm 0.4210.71 ±\pm 0.3316.20 ±\pm 0.2920.07 ±\pm 0.4417.79

Our method achieves higher average performance under both base distillation objectives, therefore supports the importance of learning corrections across the interpolation path, rather than relying solely on endpoint regression. These results show that the benefits of CONTRA-KD extend beyond simply introducing a learned residual predictor, highlighting the contribution of continuous-time flow matching in providing more effective correction guidance throughout distillation.

Q3: The curriculum compare with a fixed k=1 plus a loss-weight ramp-up

To isolate the importance of target distance pacing (k∈(0,1]k \in (0, 1]) versus loss-weight scaling (λt\lambda_t), we evaluated a baseline variant where the target step size is fixed to full velocity (k=1k=1) in every epoch, while the loss weight for LCONTRAL_{\text{CONTRA}} is multiplied by epoch+1total-epochs\frac{\text{epoch} + 1}{\text{total-epochs}} (ramping linearly from 1total-epochs\frac{1}{\text{total-epochs}} to 1.01.0). All other loss weights and velocity-field training procedures remain identical. Below are the empirical results averaged over 5 random seeds on the GPT-2 (1.5B→0.1B1.5\text{B} \to 0.1\text{B}) benchmark setup across both DistiLLM and FDD distillation objectives:

Table 3: Comparison of Target Curriculum (k∈(0,1]k \in (0, 1]) vs. Loss-Weight Ramp-Up (k=1k=1 Fixed, λt\lambda_t) on GPT-2 (1.5B→0.1B1.5\text{B} \to 0.1\text{B}) Benchmark

MethodDollySelf-InstVicunaS-NIAvg.
DistiLLM + CONTRA (Ours)26.22 ±\pm 0.3112.99 ±\pm 0.5317.41 ±\pm 0.3025.07 ±\pm 0.2120.42
DistiLLM + CONTRA (loss-weight ramp)25.41 ±\pm 0.3011.49 ±\pm 0.1516.61 ±\pm 0.2523.12 ±\pm 0.2319.16
FDD + CONTRA (Ours)25.12 ±\pm 0.4311.46 ±\pm 0.3816.05 ±\pm 0.2920.21 ±\pm 0.2218.21
FDD + CONTRA (loss-weight ramp)24.19 ±\pm 0.2011.15 ±\pm 0.3814.73 ±\pm 0.2521.63 ±\pm 0.2517.93

Replacing the progressive target curriculum with loss-weight ramp-up yields inferior average performance under both base objectives. Although the two strategies induce the same feature-space gradient direction for a fixed student representation and correction velocity, they differ in how supervision strength is modulated. Specifically, the target curriculum produces a gradient proportional to k∥h∥∥h+kv∥\frac{k}{\|h\|\|h+kv\|}, whereas loss-weight ramp-up produces a gradient proportional to λt∥h∥∥h+v∥\frac{\lambda_t}{\|h\|\|h+v\|}. Consequently, curriculum scheduling induces representation-dependent gradient scaling, rather than uniformly rescaling the full-step loss. This provides a concrete optimization-level distinction that is consistent with the observed performance differences. We have updated this results in revised Appendix F.8.

Reviewer_SfgmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CONTRA-KD for LLM knowledge distillation in style models. The authors claim that existing methods fail to capture the continuous and dynamic nature of evolving features in this type of models. A small velocity network, conditioned on time and layer index, is trained with the Rectified Flow objective to carry the projected student hidden state toward the teacher hidden state at four uniformly selected layer pairs. During distillation, the student is pulled toward a stop-gradient target obtained by taking one Euler step of size k from its own hidden state. The cosine distance serves as the loss, and k grows during training so that targets start near the student’s own state and move toward the teacher’s. Experiments on several models show that the proposed method improves consistently.

Critical strengths

• The paper addresses a timely question of KD in LLM style models. The field is new and evolving. • The paper proposes a novel idea of of importing flow matching into feature-level distillation. Generating intermediate soft targets from a learned map, rather than regressing directly onto teacher states. • The paper structure is sound and easy to follow. • Results show consistent improvements over baseline methods.

Critical weaknesses

• The abstract states that the representations remain complementary and orthogonal. However, the gains shown are mainly based on final scores. The paper could benefit from a deeper discussion on these claims. • The discussion on Student being a coarse discretization of feature dynamics is pretty shallow. No explicit test are performed. • Minor: Grammatical errors around line 204-205 and 122-123.

Questions for authors
  1. How does the velocity field in Eq. 9 relate to the depth-wise fields vT and vS in Theorem 3.1? Do CONTRA-trained students show smaller layer-to-layer increment mismatch than baselines?
  2. Can you include more detail on the total training cost, including velocity-field pre-training and per-epoch re-adaptation. Do baselines also receive the same warm-up?
  3. Is there a way to formalize more the concept of student being a coarse discretization of feature dynamics. Is there a way to measure whether student layer output is closer, in a similarity measure such as CKA or after the learned projection, or if the teacher layers are merged or skipped, does the student also approximate that.
  4. Can you show evidence that uniformly selected layers correspond (for example by CKA), and that a non-uniform pairing performs differently?
3 comments

We thank the reviewer for their constructive feedbacks. We address each concerns as follows:

W1 & Q1: Connection to feature dynamics

We clarify that the velocity field vθv_\theta in Eq. 9 and the depth-wise fields vT,vSv_T,v_S in Theorem 3.1 describe two distinct processes. The former models continuous student-to-teacher feature correction at an aligned layer, whereas the latter describes representation evolution through network depth. Our method establishes an implicit connection between them through feature transitions.

Specifically, letting cj=tj−sjc_j=t_j-s_j denote the correction at aligned layer jj, we have

(tj+1−tj)−(sj+1−sj)=cj+1−cj.(t_{j+1}-t_j)-(s_{j+1}-s_j)=c_{j+1}-c_j.

Thus, variations in the correction targets across depth encode the corresponding student–teacher transition discrepancies. Furthermore, Appendix B.2 provides an objective-level connection, bounding normalized feature-transition discrepancies by the CONTRA losses, curriculum discrepancy, and learned correction prediction errors.

We additionally measure layer-to-layer transition MSE across five aligned intervals on four datasets as reported in Figure 5 of Appendix F.1. Incorporating CONTRA-KD into FDD reduces the measured discrepancies across the evaluated intervals, even though FDD already explicitly supervises feature differences. This provides empirical support for the contribution of flow-guided correction to discrete feature dynamics alignment.

Regarding orthogonality, our intended claim concerns objective-level complementarity, rather than mathematical orthogonality between optimization gradients. CONTRA-KD supervises intermediate feature corrections and can be incorporated into existing output-level distillation objectives. Its improvements across multiple baselines support this complementary role.

Q2: Details of velocity-field pretraining, warmups and per epoch-readaption

Following the experimental protocols of FDD and DistiLLM, all baselines and CONTRA-KD variants receive the same three-epoch student warm-up, ensuring a fair comparison. Velocity-field training requires approximately 8 minutes per epoch for GPT-2 and 25 minutes per epoch for OpenLLaMA2. With three epochs of initial velocity-field pretraining, the one-time overhead is approximately 24-25 minutes for GPT-2 and 75 minutes for OpenLLaMA2. The per-epoch training costs and memory usage of the baselines and their CONTRA-KD-augmented variants are reported in Appendix F.3 (Table 11). It is also worth noting that this increase in time and memory cost is only during the training process, where we use the velocity field to support training; during inference, only the model is loaded similar to when trained with other methods, thus CONTRA-KD does not introduce additional overhead here.

W2 & Q3: Discussion of Coarse-Discretization View of the Student

Our coarse-discretization perspective follows the interpretation of residual networks as Euler discretizations of ODEs, and also adopted by FDD. However, we agree that this view needs more formalization and provide further evidence detailed in Appendix F.7:

(1) Formalization. We model a student block as a coarse step corresponding to several consecutive teacher blocks. Under the continuous-depth assumptions, we derive the following bound on the discrepancy between the student block update uSju_S^j and the teacher's accumulated interval update UTjU_T^j:

\leq \Delta_S\left(M|e_j|+|\delta_{\tau_j}|\right) +\frac{K}{2}\Delta_S^2,$$ where $e_j$ and $\delta_{\tau_j}$ denote state and velocity discrepancies, respectively. This provides a conditional characterization of coarse approximation and predicts that student block updates should more closely resemble the teacher's full-interval updates than individual teacher-block updates. **(2) Evaluation.** We evaluate all 12 GPT-2 student blocks against corresponding intervals of four teacher blocks, using 256 evaluation prompts and three independent training seeds. Linear CKA confirms that student updates align more strongly with full teacher-interval updates than with last-block-only controls (approximately 0.40 vs. 0.37). Adding CONTRA improves endpoint-state CKA (0.5269 → 0.5351), full-update CKA (0.4017 → 0.4062), and ridge-projected update cosine similarity (0.2191 → 0.2263) over FDD. We additionally test whether student blocks can replace teacher-block sequences. Although matched student blocks achieve lower normalized MSE than neighboring blocks, neither method outperforms the skip-all or keep-last teacher controls, and FDD + CONTRA does not improve functional replacement over FDD. We therefore clarify that coarse discretization describes correspondence in accumulated feature evolution, not exact functional interchangeability. ## W3: Minor grammatical issues We thank the reviewer for pointing out these grammatical errors. We have corrected the wording and reviewed the surrounding text. ## Q4: Ablation on Layer Pairing Strategies To evaluate the effectiveness of uniform layer alignment, we conduct additional experiments comparing three pairing strategies: **(i) uniform pairing**, which matches layers at evenly spaced relative depths; **(ii) CKA-based pairing**, which maximizes representation similarity between teacher and initialized student layers under an order-preserving constraint; and **(iii) fixed non-uniform pairing**, which applies a predefined nonlinear depth schedule (exponent 1.5), selecting teacher layers more densely at earlier depths without using CKA. The CKA-based strategy uses Dolly training samples for calibration and separate samples for validation. All configurations retain the same distillation objectives and curriculum, with velocity fields and projectors trained separately for each pairing. **Table 1: Ablation of layer pairing strategies (GPT-2 1.5B → 0.1B).** ROUGE-L scores (%) are averaged over five evaluation seeds ± standard deviation. | Method | Dolly | Vicuna | Self-Inst | S-NI | Avg. | |---|---:|---:|---:|---:|---:| | DistiLLM + CONTRA (Uniform) | **26.22 ± 0.31** | 17.41 ± 0.30 | **12.99 ± 0.53** | **25.07 ± 0.21** | **20.42** | | DistiLLM + CONTRA (CKA-based) | 25.54 ± 0.22 | **17.43 ± 0.36** | 12.75 ± 0.71 | 23.99 ± 0.16 | 19.93 | | DistiLLM + CONTRA (Non-uniform) | 25.17 ± 0.51 | 16.53 ± 0.33 | 12.45 ± 0.38 | 23.88 ± 0.23 | 19.51 | || | FDD + CONTRA (Uniform) | **25.12 ± 0.43** | **16.05 ± 0.29** | 11.46 ± 0.38 | 20.21 ± 0.22 | **18.21** | | FDD + CONTRA (CKA-based) | 24.86 ± 0.36 | 15.19 ± 0.24 | **11.57 ± 0.56** | 20.58 ± 0.24 | 18.05 | | FDD + CONTRA (Non-uniform) | 24.43 ± 0.24 | 15.89 ± 0.32 | 11.12 ± 0.27 | **20.60 ± 0.23** | 18.01 | As shown above, **uniform pairing achieves the highest average ROUGE-L under both distillation objectives**, outperforming CKA-based pairing under DistiLLM/FDD. Although alternative strategies occasionally perform better on individual datasets, neither improves the overall average. We partly attribute this advantage to the progressive hierarchical structure of Transformers, where uniform pairing preserves relative depth correspondence between teacher and student representations. The detailed analysis is provided in revised **Appendix F.6**.

Dear Reviewer Sfgm,

With the discussion deadline approaching, we would be grateful if you could review our responses and revisions. We would be happy to address any remaining questions or concerns before the discussion period ends. Thank you for your time and consideration.

Reviewer_UC8eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CONTRA-KD, which learns a rectified-flow correction between student and teacher hidden representations and uses progressively larger correction steps as curriculum targets alongside standard knowledge distillation losses.

Critical strengths

The method improves average ROUGE-L across several distillation baselines and model pairs, including cross-tokenizer transfer and a 13B-to-7B setting. Comparisons with direct interpolation and curriculum variants, supplemented by human evaluation and computational-cost measurements, provide useful empirical support.

Critical weaknesses

Theorem 3.1 does not establish that trajectory alignment alone guarantees dynamics alignment, because its bound includes a derivative-error term that the proposed loss does not control. The correction flow between models at selected layers is not clearly connected to the teacher’s evolution through depth, while straight conditional interpolation does not guarantee straight learned ODE trajectories or exact one-step integration. Experimental robustness remains uncertain because Appendix C.3 describes response-sampling seeds without clearly establishing variation across independent training runs, and the small human evaluation reports only slight annotator agreement.

Questions for authors

Can the authors establish the missing connection between the training objective and dynamics alignment, and quantify the error of the one-step approximation? Can they compare against an equally expressive residual predictor trained only at flow time zero and report results across independent training seeds?

3 comments

We thank the reviewer for their valuable comments. We address the concerns as follows:

W1 & Q1: Connection to feature dynamics, one-step approximation, and experimental robustness

We clarify the theoretical scope of our approach and provide additional analysis and controlled comparisons addressing the concerns.

(1) Connection between the training objective and depth-wise dynamics. We agree that Theorem 3.1 establishes a conditional relationship between trajectory and velocity discrepancies, rather than guaranteeing continuous-depth velocity alignment from state matching alone. However, CONTRA-KD is designed to implicitly encourage feature dynamics alignment through a learned correction flow, rather than explicitly supervise the teacher's instantaneous depth-wise velocity field. Specifically, letting cj=tj−sjc_j=t_j-s_j denote the student-to-teacher correction at aligned layer jj, we have the exact identity

(tj+1−tj)−(sj+1−sj)=cj+1−cj.(t_{j+1}-t_j)-(s_{j+1}-s_j)=c_{j+1}-c_j.

Thus, variations in correction targets across aligned depths encode student–teacher feature-transition discrepancies. Furthermore, we revise our manuscript and add Appendix B.2 to directly connect our implemented objective to normalized discrete feature dynamics, bounding transition discrepancies by the CONTRA losses, curriculum discrepancy, and learned correction prediction errors Together, these results establish a mathematical connection between CONTRA-KD and feature evolution at supervised depths without requiring an explicit transition-matching loss.

(2) Straight conditional interpolation and one-step approximation error. We clarify that our objective does not require exact integration of the learned ODE. The curriculum target is defined as a single Euler step at t=0t=0, h~Sj=sg[hSj+k,vθ(hSj,0,j)]\tilde h_S^j=\mathrm{sg}[h_S^j+k,v_\theta(h_S^j,0,j)]. It is the target the student is distilled toward, rather than an approximation obtained by integrating the learned ODE. Thus, numerical integration error does not enter our training objective, regardless of whether the learned trajectories are straight. In the ideal regression setting, assuming a continuous extension of the optimal field to t=0t=0, we have vθ(h,0,j)=E[hTj−hSj∣hSj=h]v_\theta(h,0,j)=\mathbb{E}[h_T^j-h_S^j\mid h_S^j=h], so the target corresponds to a partial step toward the conditional mean teacher representation.

Moreover, the straight conditional path provides a constant regression target and a simple least-squares objective. To validate this design, we conduct several controlled ablations: (i) Direct interpolation (Section 5.2): replacing the learned velocity field with direct interpolation reduces average performance. (ii) Endpoint-only training and direct MLP regression (Appendix F.5): restricting velocity training to t=0t=0 reduces average performance of both DistiLLM and FDD, while direct MLP regression achieves 19.55 and 17.79, respectively. (iii) Alternative interpolation paths (Appendix F.2): curved VP-ODE and VE-ODE interpolants perform worse than Rectified Flow.

(3) Multiple training seeds. Due to short rebuttal time window, we conduct additional runs on the FDD + CONTRA setup as a robustness test using 3 different training seeds and report the results in Table 1.

Table 1: Training with different seeds on GPT-2 (1.5B→0.1B1.5B \to 0.1B) with FDD + CONTRA

Test setSeed #1Seed #2Seed #3Mean ± SDReportedFDD
Dolly25.315224.696024.985524.9989 ± 0.309925.1224.27
Self-Instruct12.199911.708411.529911.8127 ± 0.347011.4611.23
Vicuna16.120215.874616.566316.1870 ± 0.350616.0514.69
SInst20.753921.243820.242020.7466 ± 0.501020.2119.65

Across the 4 test sets, the new runs averaged to be close or even slightly better than the initially reported FDD + CONTRA results, and are obviously still better than the FDD baseline. The reported results are well within the range of standard deviation, with the exception of the S-NI set where the reported value is just below the range. While we only used 3 seeds, the results confirmed CONTRA-KD's robustness even under different training seeds that control non-deterministic aspects of the training process.

(4) Human evaluation and inter-annotator agreement. While the human evaluation exhibits slight inter-annotator agreement, we attribute this to the subjective nature of open-ended instruction-following evaluation, where annotators may prioritize different aspects of response quality. Nevertheless, the pooled judgments show a favorable trend toward CONTRA-KD. To provide further insight into these judgments, we include representative qualitative examples in Appendix E.2, covering cases of clear annotator preference as well as disagreement. These examples illustrate how differences in response quality can lead to varying human assessments. We emphasize that human evaluation serves as supplementary evidence, while our main conclusions are supported by quantitative benchmark results and controlled ablation studies.

(5) Equally expressive residual predictor trained only at flow time zero. To isolate the necessity of training across continuous flow time t∼U(0,1)t \sim U(0,1), we evaluated a variant where the velocity field vθv_\theta is trained strictly at t=0t=0 (i.e., parameterizing vθ(hS,t=0,j)≈hT−hSv_\theta(h_S, t=0, j) \approx h_T - h_S) with matched capacity, curriculum and adaption schedule.

Below are the empirical results averaged across 5 random seeds on the GPT-2 (1.5B→0.1B1.5\text{B} \to 0.1\text{B}) setup across both DistiLLM and FDD base distillation objectives:

Table 2: Comparison of Flow Time Sampling (t∼U(0,1)t \sim U(0,1) (Ours) vs. Static t=0t=0) on GPT-2 (1.5B→0.1B1.5B \to 0.1B)

MethodDollySelf-InstVicunaS-NIAvg.
DistiLLM + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)26.22 ±\pm 0.3112.99 ±\pm 0.5317.41 ±\pm 0.3025.07 ±\pm 0.2120.42
DistiLLM + CONTRA, t=0t=025.31 ±\pm 0.3411.97 ±\pm 0.3616.82 ±\pm 0.4223.21 ±\pm 0.0819.33
FDD + CONTRA, t∼U(0,1)t \sim U(0,1) (Ours)25.12 ±\pm 0.4311.46 ±\pm 0.3816.05 ±\pm 0.2920.21 ±\pm 0.2218.21
FDD + CONTRA, t=0t=024.88 ±\pm 0.2010.83 ±\pm 0.5215.56 ±\pm 0.4119.12 ±\pm 0.1617.60

As shown in Table 2, restricting velocity training to t=0t=0 causes a consistent degradation across all benchmarks. This confirms that sampling continuous flow time t∼U(0,1)t \sim U(0,1) provides essential flow-matching regularization, enabling the velocity field to generalize as student representations shift during optimization.

Dear Reviewer UC8e,

With the discussion deadline approaching, we would be grateful if you could review our responses and revisions. We would be happy to address any remaining questions or concerns before the discussion period ends. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals

We thank all the reviewers for their valuable time and constructive feedbacks. We believe that the comments and questions have helped to clarify and strengthen the manuscript. A summary of our responses is as follows:

Our paper strengths. CONTRA-KD brings flow matching into feature-level knowledge distillation: a velocity field learns the correction from the student's hidden state to the teacher's, and a single Euler step along it, with a growing step size, gives the curriculum target. The reviewers described this as a novel and useful perspective (Sfgm, tza2) and an easy-to-adopt add-on to existing objectives (VbVC). They noted consistent gains across four baselines, three model pairs, a cross-tokenizer setting (UC8e, Sfgm, VbVC, tza2), backed by ablations, human evaluation and cost measurements.

Our main revisions and additional experiments.

  • Strengthened the theory: building on Theorem 3.1, which relates trajectory and velocity discrepancies, we added a result connecting the CONTRA objective to layer-to-layer feature changes at aligned layers (Appendix B.2).
  • Added direct measurements of student–teacher layer transitions (Appendix F.1) and a formalization and test of the coarse-discretization view, using CKA and teacher-block replacement (Appendix F.7).
  • Ran the requested ablations: a velocity field trained only at t=0, a plain MLP regressor, a fixed k=1 target with a loss-weight ramp-up, and alternative layer pairings (Appendices F.5, F.6, F.8).
  • Reported additional results in three independent training seeds for FDD and FDD+CONTRA pairs.
  • Clarified the velocity-field pre-training cost, added qualitative examples to the human evaluation, and fixed the grammatical errors.

We also justified some common questions as follows:

How does CONTRA connect velocity matching to the alignment of student-teacher feature dynamics across layers? (UC8e, Sfgm, VbVC).

  • Theoretical clarification: We clarify that Theorem 3.1 establishes a sufficient condition linking trajectory and velocity discrepancies, providing theoretical motivation for aligning feature dynamics through velocity matching.

  • Connection to feature dynamics (Appendix B.2): We derive an exact identity showing that the discrepancy between student and teacher layer transitions equals the change in their feature corrections across consecutive aligned layers. Based on this, we bound the normalized transition discrepancy in terms of the CONTRA losses, curriculum discrepancy, and velocity prediction errors.

  • Empirical validation (Appendix F.1): FDD + CONTRA consistently achieves lower student–teacher transition MSE than FDD across all five evaluated intervals on four datasets, supporting improved alignment of depth-wise feature dynamics.

Ablation for the source of the gains (UC8e, VbVC, tza2).

We conduct controlled ablations to distinguish the contributions of the learned velocity field, the curriculum, and the regression objective.

  • Learned flow vs. direct interpolation (Section 5.2): Replacing the learned velocity field with direct interpolation reduces average performance, showing that the benefit is not simply due to moving student features toward teacher features.

  • Velocity learning (Appendix F.5): Restricting velocity training to t=0 reduces average performance for both DistiLLM and FDD. A direct MLP regressor also underperforms the full method, indicating the benefit of learning corrections along the interpolation path rather than only at its endpoint.

  • Curriculum vs. loss ramping (Appendix F.6): Keeping the target fixed at k=1 while gradually increasing its loss weight performs worse than the proposed curriculum. This supports progressively adapting the target itself, rather than merely adjusting its optimization weight.

Beside common questions, we address other questions in individual replies:

  • Robustness across training seeds (UC8e): We repeat FDD and FDD + CONTRA training on GPT-2 using three independent random seeds. The resulting FDD + CONTRA averages remain close to our reported results and consistently outperform FDD across all four benchmarks, supporting the robustness of the reported gains.
  • Training cost (Sfgm): every baseline and CONTRA variant receives the same three-epoch warm-up.
  • Layer pairing (Sfgm): uniform pairing has the highest average ROUGE-L under both objectives, ahead of CKA-based and fixed non-uniform pairings.
  • Math and code (tza2): The pretrained GPT-2 and OpenLLaMA2 models already have very limited mathematical reasoning and coding capabilities, performing near the floor on evaluated datasets. Their SFT and distilled variants similarly achieve low scores. Our training targets general instruction following, so we leave specialized math and code distillation to future work.

We thank the reviewers again for the careful consideration of our work. Please let us know if any further clarification is need and we would be glad to discuss further!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.