acceptodds
Under review as a conference paper at ICLR 2027

When Are Foundation Models Good Distillation Teachers? A Controlled Study of Teacher Selection in ECG

Abstract

Large pretrained models are increasingly used as knowledge-distillation teachers, yet selecting a teacher is often reduced to choosing the largest or strongest available pretrained model. This assumes that model-to-task transferability also predicts teacher-to-student transfer, an assumption that need not hold when supervision must pass through a compact student. We study this distinction in a controlled setting using multi-label ECG classification. Holding the student architecture, distillation objective, optimization protocol, and evaluation fixed, we compare two public foundation models with task-specific supervised teachers; on PTB-XL, our audit spans six distinct teacher models, and we repeat the central foundation-teacher comparison on PhysioNet/CinC 2021. The two foundation teachers exhibit sharply different transfer behavior: ECG-FM provides no robust student improvement on either dataset, whereas ECGFounder improves PTB-XL AUROC, mAP, and [email protected] and outperforms ECG-FM on those metrics on PhysioNet. We test calibration, downstream fine-tuning, teacher scale, and architecture family, but none individually accounts for the transfer gap. Across the tested teacher pool, downstream target-task fit separates ineffective KD candidates more clearly than foundation-model status or parameter scale, although it does not perfectly rank the strongest teachers. These results motivate treating distillation-teacher selection as a task-conditioned transfer problem distinct from selecting a model to fine-tune.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, foundation models, teacher selection, model selection, model compression, transfer learning, ECG, time-series classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5No5OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether a foundation model's strong predictive performance and large parameter count necessarily make it a good teacher for knowledge distillation into smaller models. The authors fix a 1.22M-parameter ResNet student and its training pipeline, then compare ECG-FM, ECGFounder, and supervised teachers on PTB-XL and PhysioNet/CinC 2021. They further examine the differences through temperature calibration, teacher fine-tuning, and comparisons across model sizes and architectures.

Critical strengths
  • The research question is practically relevant. Examining whether a teacher can actually improve a small student is more meaningful for knowledge distillation than simply comparing the foundation models' own performance.
  • The main experiments are reasonably well controlled on the student side. The student model, loss function, and training procedure are kept fixed, and a strong baseline without distillation is included. This helps reduce some potential confounding factors.
  • The evaluation covers a fairly broad range of aspects.
Critical weaknesses
  • The authors reconstruct the publicly available ECG-FM model layer by layer and report that its convolutional feature extractor contains seven blocks. However, the original ECG-FM paper describes four convolutional blocks. The authors should verify this discrepancy.

  • On page 7, Section 4.3 and Table 3, the authors use the cross-architecture experiments to argue that architecture family alone cannot explain the observed performance gap. However, the current teacher pool still shows a clear pattern: ECG-FM and the additional ECGTransformer are both Transformers, with relatively weak standalone task performance and poor distillation results. In contrast, ECGFounder and the supervised ResNet are both CNNs, with stronger standalone performance and better distillation results. Moreover, the student in the main distillation experiments is always a ResNet. Adding another poorly performing Transformer teacher does not rule out the possibility that Transformer teachers are simply less compatible with the current student architecture or training setup. In this teacher pool, architecture family and the so-called "task suitability" remain confounded.

  • The 4.8M-parameter Transformer is also not strictly capacity-matched to the 3.2M- and 7.8M-parameter ResNets. The additional students presented in the appendix mainly provide learning curves without distillation, which cannot establish whether the teacher rankings would change with a different student architecture.

  • A more convincing experiment would compare CNN and Transformer teachers with similar standalone task performance and repeat the main teacher comparisons using at least one other student architecture. With the current evidence, the conclusion should be more limited: these comparisons have not yet identified the source of the performance gap.

  • The paper also lacks an architecture diagram, which makes the overall setup and methodology harder to understand.

Questions for authors

see weakness.

1 comment

Thank you for identifying the discrepancy in the ECG-FM description. Checking the forward computation changed both our implementation and our interpretation.

  1. Four versus seven convolutional blocks. You were right about the block count. The released checkpoint and our implementation both have four blocks. The submitted text incorrectly said seven. We also found two implementation errors. We corrected the first convolutional block to use one GroupNorm group per channel and moved the stack-level LayerNorm before the Transformer layers. Against the upstream encoder and released head, the maximum absolute logit difference was 9.54 × 10⁻⁶ across 8,166 validation inputs. We also checked gradients, refit the ECG-FM teachers from the released checkpoint, and repeated the affected student distillation runs. On PTB-XL, both students had lower ensemble AUROC with the corrected default teacher than with labels alone. Retuning reduced the ResNet deficit without demonstrating a gain over labels alone (Table 1, p. 3, Figure 2, p. 5, and Appendix C, pp. 9–10).

  2. Capacity matching and architecture confounding. We agree that the original 4.8M Transformer was only within the ResNet size range. We now describe it that way. We also trained a 7.800M-parameter CNN teacher and a 7.804M-parameter Transformer teacher on the same inputs and schedule. Their best checkpoints differed in standalone quality. An exploratory pair met the validation matching criteria, but its test AUROC gap of 0.00750 exceeded the 0.005 tolerance. Section 4.3 and Table 3 now state that teacher architecture and standalone quality remain confounded (pp. 4–5). Table 7 gives the matching results (p. 11).

  3. Distillation into another student. We repeated the teacher comparisons with a compact Transformer and a matched labels-only control. On grouped PhysioNet, the supervised teacher improved Transformer-student ensemble AUROC by 0.01030, compared with 0.00001 for the ResNet student. The adjusted interval for the difference in gains was [0.00468, 0.01731]. This supports a difference between these two students. They differ in capacity as well as architecture, so the experiment leaves the cause open (Table 2, p. 4).

  4. Architecture diagram. Figure 1 now shows the ECG input, candidate teacher architectures, both compact students, and the two training losses. It makes explicit which predictions enter distillation and which model is used at inference (p. 2).

Reviewer_fnGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies whether ECG foundation models naturally serve as strong knowledge distillation teachers. Using a fixed 1.22M ResNet student and logit-based KD, it compares ECG-FM, ECGFounder and supervised teachers. ECG-FM yields no consistent gains. ECGFounder improves multiple metrics on PTB-XL and surpasses ECG-FM on PhysioNet. The authors suggest teacher selection should prioritize downstream task alignment instead of foundation model identity or size.

Critical strengths
  1. The experiments are practically useful. They clearly show foundation teachers produce very different distillation results. Foundation model status or parameter scale alone cannot guarantee a good distillation teacher.
  2. The paper is well written with transparent reporting. It flags exploratory experiments and discusses limitations including preprocessing mismatch and data overlap. This helps readers evaluate the evidence properly.
Critical weaknesses
  1. The “student-conditioned” claim is unvalidated. All main KD tests use only one ResNet student. The ranking of the two core foundation teachers remains unchanged across the two tasks. Evidence for task-conditioned teacher selection is weak. Native fit is merely a descriptive correlation within the PTB-XL teacher pool without held-out or prospective validation.
  2. Architecture confounding cannot be excluded. Top performing teachers are mostly high-fit CNNs while the two low-fit Transformer teachers underperform. Architecture and native fit are entangled. Additional experiments are needed such as high-fit Transformer teachers distilling to ResNet or shared teachers paired with CNN and Transformer students.
  3. Novelty is limited. SoTeacher, GRACE and prior works have covered student and task-dependent teacher quality. This paper adds new ECG-specific empirical results but introduces no new selection metrics or mechanisms.
Questions for authors
  1. Does teacher ranking change across CNN and Transformer students?
  2. Can a high-fit Transformer teacher rule out architecture mismatch effects?
  3. Can native fit prospectively predict KD performance on held-out teachers or PhysioNet? If GRACE is adapted for multi-label logit KD, how does its predictive power compare with native fit?
2 comments

Thank you for distinguishing three questions that our original single-student comparison left open.

  1. Does teacher ranking change across students or tasks? The two foundation teachers retain the same observed ensemble AUROC ordering on both tasks. The full teacher pool shows a change. For the Transformer student, ECGFounder had the highest observed ensemble AUROC on PTB-XL, while the supervised ResNet teacher led on grouped PhysioNet. On PhysioNet, ECGFounder led for the ResNet ensemble. The supervised teacher’s AUROC gain over labels alone was 0.01030 for the Transformer and 0.00001 for the ResNet. The direct difference was 0.01029, with a multiplicity-adjusted interval of [0.00468, 0.01731]. We measured this with matched initializations within each student and seed. All three individual ResNet seeds favored the supervised teacher over ECGFounder, so we qualify the ensemble ranking explicitly (Tables 1–2, pp. 3–4).

  2. Can a high-performing Transformer teacher rule out architecture mismatch? We attempted that control with CNN and Transformer teachers matched within 0.05% in parameter count. The best Transformer reached validation AUROC 0.90749 against 0.93341 for the CNN. With those checkpoints, the CNN produced higher student AUROC in both students. An exploratory pair met the validation matching criteria and reversed that ordering in both students, but its teacher test AUROC gap exceeded the matching tolerance. We did not obtain the high-quality match needed to separate architecture from teacher quality (Section 4.3, pp. 4–5, and Appendix D, Table 7, p. 11).

  3. Does native fit predict distillation prospectively? We froze the validation-AUROC ranking of four new PTB-XL configurations from familiar families before training any student pilot. It selected the Transformer ensemble’s best teacher and missed the ResNet student’s best teacher. It beat selection by total parameter count for both ensembles, but ranked candidates identically to trainable parameter count. On grouped PhysioNet, selecting ECGFounder by validation AUROC missed the supervised teacher’s stronger Transformer-student result (Section 5, Table 4, pp. 5–6).

Prior work and GRACE. SoTeacher and GRACE already address student-dependent teacher quality. Our contribution is the controlled ECG evidence above. We did not evaluate a multi-label adaptation of GRACE’s student-gradient score, so its predictive accuracy relative to native AUROC remains an open comparison (Sections 1 and 6, pp. 1 and 6).

Reviewer_fnGHOpenReview ↗

reply to author

I appreciate the authors’ response and their answers to my questions. However, I still think my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it unchanged.

Reviewer_2gKwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that larger or stronger pretrained models are not necessarily better teachers for knowledge distillation to smaller ECG classification models. It presents a controlled empirical audit on the PTB-XL dataset to evaluate teacher model selection for knowledge distillation, comparing foundation models against other candidates while holding the student architecture and training pipeline fixed.

Critical strengths

The paper is having a rigorously controlled experimental design and systematic evaluation, which establish a clear, practical guideline for screening teacher models in knowledge distillation based on target-task performance.

There is an interesting result that task-specific performance predicts teacher effectiveness better than model size/status, although it cannot perfectly identify the best teacher.

Critical weaknesses

While the paper compares public foundation models against task-specific teachers under strictly controlled settings, I couldn't find a clear articulation of the underlying problem this experimental design addresses.

The paper relies entirely on existing models and techniques, so I am not sure if I underestood the novelty of the paper.

The empirical evaluation of the proposed method is insufficient to me. Restricting the experiments to just the PTB-XL dataset limits the generalizability of the claims. To demonstrate generalizability, it would be great to validate the approach across diverse datasets and cross-domain tasks.

The current visualizations are not interpretable to me. They are more like graphical repetitions of the experimental results. Would be great if the authors revise these qualitative results to explicitly support the theoretical analysis.

This is minor, but the abstract contains a few abbreviations. Please provide their full forms in their first occurrence.

Questions for authors

Can you please explain the motivation behind your problem setting.

Can you clarify the novelty of the approach compared to the existing works.

Can you include in your results at least one more dataset / cross-domain task?

Please see weaknesses.

2 comments

Thank you for asking us to make the practical decision behind the study clearer. We revised the introduction, figures, and abstract around that question.

Motivation. Suppose a team has already chosen the compact ECG classifier it can deploy. Is it worth training that classifier with a particular teacher? The relevant baseline is the same student trained on diagnostic labels alone. Holding the student and its training protocol fixed lets us measure what each teacher adds. We now open with this decision (pp. 1–2).

Novelty. Our contribution is an empirical test of how well a teacher’s own target-task performance predicts what its student will learn. We chose a teacher before observing student outcomes and then distilled every candidate into both students. On PTB-XL, choosing by teacher validation AUROC beat choosing the largest model for both student ensembles, yet missed the best teacher for the ResNet student. This gives a concrete example of both the value and the limit of screening by standalone performance (Section 5, Table 4, pp. 5–6).

Datasets and generalizability. We could have made the second task more visible in the submission. Tables 1 and 2 now show the same comparison on PTB-XL (five diagnostic labels) and PhysioNet/CinC 2021 (30 labels). During revision we found duplicate waveforms across the original PhysioNet splits, grouped the recordings, and refit all three teachers and their student conditions, including the labels-only controls. On that partition, the teacher with the lowest standalone validation AUROC produced the highest Transformer-student test AUROC. Both tasks concern ECG, so generalization to other domains remains open (pp. 2–4).

Figures and terminology. Figure 1 shows where labels and teacher predictions enter training (p. 2). Figure 2 asks whether tuning can recover the deficit observed with default ECG-FM. Zero marks the labels-only baseline, and the intervals show that tuning reduces the deficit without establishing a gain over that baseline (p. 5). The abstract now spells out electrocardiogram and area under the receiver operating characteristic curve at first use (p. 1).

Reviewer_2gKwOpenReview ↗

Re Rebuttal

Many thanks to the authors for their rebuttal. After reading the other reviews and answers, I keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.