- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a foundation model's strong predictive performance and large parameter count necessarily make it a good teacher for knowledge distillation into smaller models. The authors fix a 1.22M-parameter ResNet student and its training pipeline, then compare ECG-FM, ECGFounder, and supervised teachers on PTB-XL and PhysioNet/CinC 2021. They further examine the differences through temperature calibration, teacher fine-tuning, and comparisons across model sizes and architectures.
Critical strengths
- The research question is practically relevant. Examining whether a teacher can actually improve a small student is more meaningful for knowledge distillation than simply comparing the foundation models' own performance.
- The main experiments are reasonably well controlled on the student side. The student model, loss function, and training procedure are kept fixed, and a strong baseline without distillation is included. This helps reduce some potential confounding factors.
- The evaluation covers a fairly broad range of aspects.
Critical weaknesses
-
The authors reconstruct the publicly available ECG-FM model layer by layer and report that its convolutional feature extractor contains seven blocks. However, the original ECG-FM paper describes four convolutional blocks. The authors should verify this discrepancy.
-
On page 7, Section 4.3 and Table 3, the authors use the cross-architecture experiments to argue that architecture family alone cannot explain the observed performance gap. However, the current teacher pool still shows a clear pattern: ECG-FM and the additional ECGTransformer are both Transformers, with relatively weak standalone task performance and poor distillation results. In contrast, ECGFounder and the supervised ResNet are both CNNs, with stronger standalone performance and better distillation results. Moreover, the student in the main distillation experiments is always a ResNet. Adding another poorly performing Transformer teacher does not rule out the possibility that Transformer teachers are simply less compatible with the current student architecture or training setup. In this teacher pool, architecture family and the so-called "task suitability" remain confounded.
-
The 4.8M-parameter Transformer is also not strictly capacity-matched to the 3.2M- and 7.8M-parameter ResNets. The additional students presented in the appendix mainly provide learning curves without distillation, which cannot establish whether the teacher rankings would change with a different student architecture.
-
A more convincing experiment would compare CNN and Transformer teachers with similar standalone task performance and repeat the main teacher comparisons using at least one other student architecture. With the current evidence, the conclusion should be more limited: these comparisons have not yet identified the source of the performance gap.
-
The paper also lacks an architecture diagram, which makes the overall setup and methodology harder to understand.
Questions for authors
see weakness.
Thank you for identifying the discrepancy in the ECG-FM description. Checking the forward computation changed both our implementation and our interpretation.
Four versus seven convolutional blocks. You were right about the block count. The released checkpoint and our implementation both have four blocks. The submitted text incorrectly said seven. We also found two implementation errors. We corrected the first convolutional block to use one GroupNorm group per channel and moved the stack-level LayerNorm before the Transformer layers. Against the upstream encoder and released head, the maximum absolute logit difference was 9.54 × 10⁻⁶ across 8,166 validation inputs. We also checked gradients, refit the ECG-FM teachers from the released checkpoint, and repeated the affected student distillation runs. On PTB-XL, both students had lower ensemble AUROC with the corrected default teacher than with labels alone. Retuning reduced the ResNet deficit without demonstrating a gain over labels alone (Table 1, p. 3, Figure 2, p. 5, and Appendix C, pp. 9–10).
Capacity matching and architecture confounding. We agree that the original 4.8M Transformer was only within the ResNet size range. We now describe it that way. We also trained a 7.800M-parameter CNN teacher and a 7.804M-parameter Transformer teacher on the same inputs and schedule. Their best checkpoints differed in standalone quality. An exploratory pair met the validation matching criteria, but its test AUROC gap of 0.00750 exceeded the 0.005 tolerance. Section 4.3 and Table 3 now state that teacher architecture and standalone quality remain confounded (pp. 4–5). Table 7 gives the matching results (p. 11).
Distillation into another student. We repeated the teacher comparisons with a compact Transformer and a matched labels-only control. On grouped PhysioNet, the supervised teacher improved Transformer-student ensemble AUROC by 0.01030, compared with 0.00001 for the ResNet student. The adjusted interval for the difference in gains was [0.00468, 0.01731]. This supports a difference between these two students. They differ in capacity as well as architecture, so the experiment leaves the cause open (Table 2, p. 4).
Architecture diagram. Figure 1 now shows the ECG input, candidate teacher architectures, both compact students, and the two training losses. It makes explicit which predictions enter distillation and which model is used at inference (p. 2).