- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes ECGDuo, a 0.137M-parameter depthwise-separable CNN for 12-lead ECG distilled from two frozen teachers: an ECG foundation model (ECGFounder by default, ECG-FM in the ablations) and a general time-series foundation model (Sundial). Teacher embeddings of 3-s windows from MIMIC-IV-ECG and CODE-15 are cached offline, and the student is trained with a symmetric InfoNCE loss through one linear adapter per teacher. Two information-theoretic results motivate the second teacher. Evaluation is a linear probe on five classification datasets (CPSC2018, CSN, G12EC, PTB-XL, PTB) with subject-independent five-fold cross-validation, against supervised end-to-end models and frozen foundation models, plus ablations with and without Sundial, label efficiency, latency and CKA analyses. ECGDuo averages 83.73 against 84.00 for ECGFounder.
Critical strengths
- The main strength, to me, is the practical result: a 0.137M-parameter frozen encoder with a linear head that stays within 0.3 points of ECGFounder on average over five datasets and is about 20 to 55 times faster at inference (fig 3). This sounds useful to practitioners.
- The evaluation is careful: subject-independent folds shared by all methods, labels of the distillation corpora never used, single- and dual-teacher students matched on windows, capacity and update budget, and code released.
Critical weaknesses
- Novelty is limited for ICLR, and the scope, only 12-lead ECG, is too narrow (i) In my opinion the methodological novelty is limited for ICLR. Distilling several heterogeneous foundation models into one student through teacher-specific adapters is the not new, which is also the setting the paper uses to set itself apart from multi-teacher distillation: different teachers looking at the same input. The loss is the CLIP-style symmetric InfoNCE of contrastive representation distillation. Multi-teacher distillation of compact ECG models already exists. What is new is choosing a general time-series model as the second teacher. The "single-proxy transfer bottleneck" is, to me, a new name for distilling from one teacher. (ii) The theory does not support the claims. Theorem 3.1 is the chain rule plus "conditioning cannot increase entropy", and neither the complementarity term nor the residual term is ever estimated. its headline case, a student that beats its ECG teacher, does not happen: ECGDuo is below ECGFounder on four of five datasets. Proposition 3.2 is the InfoNCE bound applied to each branch; with B = 1024, 2 log B is about 13.9 nats, so the bound only says something if I(ZE; ZT) is below that, which is never checked for two embeddings of the same window… Am I missing something? (iii) The scope is too narrow for ICLR… only 12-lead ECG, 3-s windows and single-label classification on five curated datasets. The idea is generic, a domain foundation model plus a general time-series model distilled into a compact student, so if that is the claim it has to be shown on other physiological signals where both kinds of teacher exist (EEG, PPG, EMG, IMU), and on wearable single-lead ECG, which is where a 0.137M encoder would potentially matter(?). To me, clinical 12-lead ECG is not where a 30M-parameter model is a deployment problem; Fig. 3 gives ECGFounder 20 to 65 ms per batch of 128. As it stands this is an ECG compression study, and I feel it fits better applied venues.
- Nothing shows that the time-series teacher is what helps… (i) The gain from adding Sundial is about one Avg point over the ECGFounder-only student from a single distillation seed. The stds in Table 2 are over downstream folds and the CIs in fig 6 over head seeds, so none of them capture distillation variance, which can easily be of that size. And nothing isolates the general time-series knowledge: there is no second ECG teacher (ECG-FM plus ECGFounder), no random or weak second teacher, no Sundial-only student, and no other time-series model. Without these, the gain could come from any second target. This needs to be done thoroughly imo. (ii) some key baselines are missing; the same DSCNN trained from scratch on each task, and the same DSCNN pretrained with ECG self-supervision on the same MIMIC-IV-ECG and CODE-15 windows without any teacher (e.g., CLOCS, which is cited, or ST-MEM, ICLR 2024). Setting A compares a frozen encoder that saw about 670k unlabeled windows and two foundation teachers with supervised models trained only on the downstream folds, several of which look under-trained … so the gains say little. (iii) The protocol makes the numbers hard to compare with anything published: 3-s windows carrying record-level labels are scored one by one, the label sets are curated single-label subsets (PTB-XL without HYP and only patients with one shared superclass), and signals are not resampled across 400, 500 and 1000 Hz, so the same kernels cover 2.5 times more time on CODE-15 than on PTB. Maybe I am missing something, could you please clarify how the 3-s windows are fed to ECGFounder and ECG-FM, which were pretrained on longer recordings? Also, the abstract's "0.1% of Sundial's parameters" is not the right reference. The one that matters is ECGFounder (0.45%), which ECGDuo trails on four of five datasets, and the PTB win (91.26 vs 87.33 +- 5.23) is on 200 subjects. (iv) For an ECG paper, the ECG in Fig. 1 is not an ECG. The traces labelled "ECG Signals"; they look like generic time series (probably ai generated?). The same figure also has "GruopNorm" and "Deepthwise", and calls the linear probe "fine-tuning". To me this goes with a wider issue: there is no analysis of what cardiac information the time-series teacher adds (rhythm, intervals, morphology?), and the CKA increase toward Sundial in Fig. 7 is by construction, since the student is trained to match it.
Questions for authors
My main concerns are the two weaknesses above.I copy-apste them here. I’d be open to raise the score if the scope is generalize and all my concerns are tackled, but novelty is a key limitation in any case…
- Novelty is limited for ICLR, and the scope, only 12-lead ECG, is too narrow (i) In my opinion the methodological novelty is limited for ICLR. Distilling several heterogeneous foundation models into one student through teacher-specific adapters is the not new, which is also the setting the paper uses to set itself apart from multi-teacher distillation: different teachers looking at the same input. The loss is the CLIP-style symmetric InfoNCE of contrastive representation distillation. Multi-teacher distillation of compact ECG models already exists. What is new is choosing a general time-series model as the second teacher. The "single-proxy transfer bottleneck" is, to me, a new name for distilling from one teacher. (ii) The theory does not support the claims. Theorem 3.1 is the chain rule plus "conditioning cannot increase entropy", and neither the complementarity term nor the residual term is ever estimated. its headline case, a student that beats its ECG teacher, does not happen: ECGDuo is below ECGFounder on four of five datasets. Proposition 3.2 is the InfoNCE bound applied to each branch; with B = 1024, 2 log B is about 13.9 nats, so the bound only says something if I(ZE; ZT) is below that, which is never checked for two embeddings of the same window… Am I missing something? (iii) The scope is too narrow for ICLR… only 12-lead ECG, 3-s windows and single-label classification on five curated datasets. The idea is generic, a domain foundation model plus a general time-series model distilled into a compact student, so if that is the claim it has to be shown on other physiological signals where both kinds of teacher exist (EEG, PPG, EMG, IMU), and on wearable single-lead ECG, which is where a 0.137M encoder would potentially matter(?). To me, clinical 12-lead ECG is not where a 30M-parameter model is a deployment problem; Fig. 3 gives ECGFounder 20 to 65 ms per batch of 128. As it stands this is an ECG compression study, and I feel it fits better applied venues.
- Nothing shows that the time-series teacher is what helps… (i) The gain from adding Sundial is about one Avg point over the ECGFounder-only student from a single distillation seed. The stds in Table 2 are over downstream folds and the CIs in fig 6 over head seeds, so none of them capture distillation variance, which can easily be of that size. And nothing isolates the general time-series knowledge: there is no second ECG teacher (ECG-FM plus ECGFounder), no random or weak second teacher, no Sundial-only student, and no other time-series model. Without these, the gain could come from any second target. This needs to be done thoroughly imo. (ii) some key baselines are missing; the same DSCNN trained from scratch on each task, and the same DSCNN pretrained with ECG self-supervision on the same MIMIC-IV-ECG and CODE-15 windows without any teacher (e.g., CLOCS, which is cited, or ST-MEM, ICLR 2024). Setting A compares a frozen encoder that saw about 670k unlabeled windows and two foundation teachers with supervised models trained only on the downstream folds, several of which look under-trained … so the gains say little. (iii) The protocol makes the numbers hard to compare with anything published: 3-s windows carrying record-level labels are scored one by one, the label sets are curated single-label subsets (PTB-XL without HYP and only patients with one shared superclass), and signals are not resampled across 400, 500 and 1000 Hz, so the same kernels cover 2.5 times more time on CODE-15 than on PTB. Maybe I am missing something, could you please clarify how the 3-s windows are fed to ECGFounder and ECG-FM, which were pretrained on longer recordings? Also, the abstract's "0.1% of Sundial's parameters" is not the right reference. The one that matters is ECGFounder (0.45%), which ECGDuo trails on four of five datasets, and the PTB win (91.26 vs 87.33 +- 5.23) is on 200 subjects. (iv) For an ECG paper, the ECG in Fig. 1 is not an ECG. The traces labelled "ECG Signals"; they look like generic time series (probably ai generated?). The same figure also has "GruopNorm" and "Deepthwise", and calls the linear probe "fine-tuning". To me this goes with a wider issue: there is no analysis of what cardiac information the time-series teacher adds (rhythm, intervals, morphology?), and the CKA increase toward Sundial in Fig. 7 is by construction, since the student is trained to match it.
6 comments
Response to Reviewer ATkf (2/2): controls, baselines, protocol and cardiac content
2(i) Distillation variance and second-teacher controls
Every second-teacher student in Table R1 (Table 4) now has the same five independent distillation seeds (2026–2030), with all requested controls.
Variance. The seed standard deviation (SD) is 0.17–0.45 Avg (mean of six metrics over five datasets). ECGDuo exceeds the ECGFounder-only student by 1.81 and the ECGFounder + ECG-FM student by 1.12, in 5/5 paired seeds (paired t p < .001).
Any second target? A duplicated teacher adds nothing (−0.01) and a random-network target hurts (−0.85). Pretrained targets help less: ECG-FM +0.69 (5/5), permuted Sundial +0.42 (4/5); none reaches ECGDuo, and Sundial alone gives the lowest student (80.56).
Another time-series foundation model (TSFM). Over five seeds, the ECGFounder + MOMENT student reaches 83.83 ± 0.13 Avg, +0.31 over ECGFounder-only (App. E.1, E.3). Adding a TSFM also helps other electrocardiogram (ECG) teachers, and Sundial helps most (App. E.5):
| ECG teacher | ECG-only student | + Sundial | + MOMENT |
|---|---|---|---|
| HuBERT-ECG | 76.88 | 80.13 | 78.34 |
| MERL | 80.01 | 81.14 | 80.47 |
2(ii) Same-encoder baselines
Same-encoder comparison. Both requested baselines use the same depthwise-separable convolutional neural network (DSCNN), one with contrastive multi-segment coding (CMSC) self-supervision and one trained end-to-end, and ECGDuo leads both (Table 2, App. E.4):
| Same DSCNN encoder | CPSC2018 | G12EC | PTB-XL | Avg |
|---|---|---|---|---|
| CMSC self-supervision, same 671k windows, linear probe | 67.88 | 66.77 | 76.70 | 76.15 |
| Trained end-to-end per task | 78.49 | 76.82 | 82.47 | 83.37 |
| ECGDuo, frozen + linear head | 80.40 | 81.57 | 82.94 | 85.56 |
The same encoder trained from scratch is the strongest supervised baseline, 2.19 Avg behind ECGDuo (0.47 on PTB-XL to 4.75 on G12EC). CMSC pairs two windows per subject, as in CLOCS; ST-MEM uses a transformer (App. E.4).
2(iii) Protocol
- Teacher inputs. ECGFounder and ECG-FM receive exactly the student's 3-s, 12-lead window at the native rate, z-scored per lead with whole-record statistics, without teacher-specific filtering or resampling. Sundial encodes each lead in 512-sample chunks, mean-pooled over tokens, chunks and leads (§3.2, App. D.1).
- Sampling rates. Signals are not resampled, so a fixed kernel spans different durations at 400, 500 and 1000 Hz; each benchmark is evaluated at its single rate with identical windows for every encoder; lowering the input to 250 or 125 Hz costs ECGDuo only 0.07 and 0.48 Avg (Table H.1).
- Window scoring. Subject-level scores now average window probabilities per label unit; ECGDuo reaches 86.82 vs 85.34 for the ECGFounder + ECG-FM student (5/5 seeds; App. D.2, E.2).
- Curated labels. We add the official multi-label PTB-XL benchmark, including hypertrophy (HYP), with record-level scoring and folds 1–8 / 9 / 10. ECGDuo's macro area under the receiver operating characteristic curve (AUROC) is above the ECGFounder + ECG-FM student on 6/6 tasks, significantly on superdiagnostic, form and rhythm, most on form (+0.045) and rhythm (+0.029) (§4.7, Table F.3).
- PTB. We agree that 200 subjects cannot carry a claim on their own; the revision makes no claim from PTB alone.
2(iv) Fig. 1 and cardiac content
Fig. 1. The original traces were not generated by artificial intelligence: they were ECG curves shown after moderate low-pass filtering and downsampling for a cleaner display. Fig. 1 now draws recorded 3-s ECG windows directly; this link plots these windows as recorded and after low-pass filtering and downsampling. "GroupNorm" and "Depthwise" are corrected; the downstream step is named "linear probe", and the label-scarcity figure (now Fig. 5) x-axis reads "Fraction of labeled training subjects".
Centered kernel alignment (CKA). We agree that the increase toward Sundial is expected by construction; it is now descriptive only (§4.5).
Which cardiac information improves. Ridge probes on frozen features, ECGDuo minus the ECGFounder-only student, 5 seeds (Table 7; RR: interval between consecutive R peaks):
| ΔR² | PTB-XL | CPSC2018 |
|---|---|---|
| RMSSD (root mean square of successive RR differences) | +0.067 | +0.070 |
| RR coefficient of variation | +0.054 | +0.069 |
| RR min/max ratio | +0.060 | +0.074 |
| Heart rate | +0.020 | +0.029 |
All four gains hold in 5/5 seeds (paired t p < .01), with a multilayer-perceptron (MLP) probe as well (App. G.1).
Thank you again for the time and care you gave this review. Whatever your final assessment, the paper is substantially better because of it, and we would be grateful if you would reassess the value and contribution of this application study in light of these results.
Response to the authors’ rebuttal
Dear authors,
Before turning to the content: I would prefer a more professional tone in this exchange. Remarks on how my review was first read, on its layout, or on your deliberations about withdrawing the paper are not appropriate in a rebuttal and have no bearing; in the opinion of this reviewer.
On the revision. Thank you for the substantial additional work. Several of my soundness concerns are addressed by the five-seed distillations, the second-teacher controls, the same-encoder baselines, the clarified teacher inputs and the corrected figg 1. The theory is also now appropriately framed as motivation. I still have two concerns
(i) The headline model changed during the rebuttal. ECGDuo was re-tuned. I did not find a search space, budget or validation split, and I could not find whether the ECGFounder-only and ECGFounder + ECG-FM students were tuned the same way. Please clarify if possible.
(ii) The official ptbxl benchmark does not reproduce Table 2. Table 2 places ECGDuo above ECGFounder, yet on the official benchmark ECGDuo trails the frozen ECGFounder probe on 5/6 tasks (Table 6). Its all-statement macro AUROC is also below that of the smaller ECGFounder-only student . It is on par with the Wavelet+nn feature baselin that Strodthoff (2021) report on the same folds, where xresnet1D101 and Inception1D reach 0.925. In Table 2 these architectures trail the DSCNN trained from scratch. I remain concerned that the Setting A baselines may be under-trained and that the curated protocol drives much of the reported advantage. Please let me know what part of my reasoning is not correct, or how you argument for that.
All in all, scope remains one of my main concerns. You state that ECGDuo "introduces no new component", that it "is an ECG application study", and that you "intend it to offer practical insight for ECG applications rather than a new algorithm" (or method). The finding the revision supports is therefore specific to ECG and to one teacher. ICLR is a machine learning venue. Even in its application areas, I expect claims that generalize beyond a single signal type and matter to the broader machine learning research community. I believe the work is not bad, but I strongly consider this paper out of scope for ICLR. It fits way better the scope of MLHC, IEEE JBHI and other technical-translational venues, where it would reach its natural audience.
Response to Reviewer ATkf (1/2): a personal note; novelty, theory, scope and deployment
Dear Reviewer ATkf,
We would like to begin with something more personal than a point-by-point reply.
When your review first arrived, we read it poorly. The lowest score, together with a dense layout in which the points ran into one another, made it feel dismissive to us, and with overall scores of 2/2/1, we came close to withdrawing the paper.
We then read it again, slowly, and our view changed completely. Your review is specific, fair and plainly written in good faith: nearly every point names a real weakness of the submission together with the experiment that would test it. We are sorry for our first reaction and grateful for the time you gave the paper! Your review is the main reason we did not withdraw: even if only to answer it properly, the rebuttal was worth doing, and much of the 16-to-25-page revision is built around your list.
We ran every control you listed and follow your numbering; Tables R1 and R2 are in the general response.
1(i) Novelty
We agree that, technically, ECGDuo introduces no new component: per-teacher adapters, same-input teachers and symmetric information noise-contrastive estimation (InfoNCE) losses are established, and the revision credits UNIC, Formont et al. and compact multi-teacher electrocardiogram (ECG) distillation (§§1–2). We believe the revision contributes something of a different kind:
We hope this evidence helps you reassess the value and contributions of our work. As the primary area we chose, applications, indicates, we intend it to offer practical insight for ECG applications rather than a new algorithm.
1(ii) Theory
You are not missing anything: Theorem 3.1 is a motivating sufficient condition and Proposition 3.2 a bound (§3, App. D.3).
1(iii) Scope and deployment
ECGDuo is an ECG application study; for the deployment you name, the revision adds single-lead input, per-window CPU cost, 8-bit size and noisy recordings.
Single lead. Students see only lead I; teacher targets use all 12 leads (App. E.7; ±: seed standard deviation). ECGDuo and ECGDuo-S exceed their same-size lead-I ECGFounder-only students by 1.29 and 2.03 Avg, respectively (Table E.2):
Cost per window. On one CPU core, as in bedside monitors and embedded devices, ECGDuo is about 15× faster than ECGFounder (Table 3, replacing the original Fig. 3):
Size reference. The 0.137M you cite counted the 512-d encoder, now ECGDuo-S (134,912), plus a five-class head; ECGDuo uses a 2048-d projector. ECGDuo-S nearly matches ECGDuo (85.21 vs 85.33 Avg), and mixed 8-bit weights shrink it to 0.20 MiB while retaining 98.9% of its 32-bit Avg (App. E.6).
Noise. ECGDuo-R, distilled with artifact augmentation, keeps 99.7% of ECGDuo's clean Avg (85.07 vs 85.33). At 0 dB it drops less than both frozen ECGFMs in every column of Table 8 (electromyographic (EMG): 5.17 vs 14.52 for ECGFounder); recorded MIT-BIH Noise Stress Test Database (NSTDB) noise never enters training or selection (App. H).