acceptodds
Withdrawn from ICLR 2027

ECGDuo: Complementary Distillation from ECG and Time-Series Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation models provide strong representations for electrocardiogram (ECG) analysis, but full ECG foundation models remain expensive for clinical settings that require compact, fast inference and efficient adaptation. A common remedy is to distill a large ECG teacher into a small encoder. ECG-only distillation, however, suffers from a *single-proxy transfer bottleneck*: the transferability of the student is bounded by the representational coverage of the ECG teacher proxy, which fails to capture temporal knowledge that transfers across time-series domains. We therefore propose **ECGDuo**, a compact ECG encoder distilled from two complementary teachers: *(i)* an ECG-specific teacher that transfers cardiac knowledge, and *(ii)* a general time-series teacher that injects domain-agnostic temporal knowledge. ECGDuo uses symmetric contrastive distillation to transfer both forms of knowledge into a lightweight raw-ECG encoder. Each teacher first encodes the same raw ECG signal into either an ECG-specific representation or a domain-agnostic temporal representation. Branch-specific adapters then map the shared student representation into the corresponding teacher spaces, where matched student–teacher pairs are aligned. Across five downstream ECG datasets, ECGDuo consistently outperforms baselines trained from scratch and remains competitive with full-scale foundation models, using only a **0.1% parameter budget**. All code is available in the [***Anonymous GitHub Repository***](https://anonymous.4open.science/r/ECGDuo-8848).

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ECG, Foundation model distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ATkfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes ECGDuo, a 0.137M-parameter depthwise-separable CNN for 12-lead ECG distilled from two frozen teachers: an ECG foundation model (ECGFounder by default, ECG-FM in the ablations) and a general time-series foundation model (Sundial). Teacher embeddings of 3-s windows from MIMIC-IV-ECG and CODE-15 are cached offline, and the student is trained with a symmetric InfoNCE loss through one linear adapter per teacher. Two information-theoretic results motivate the second teacher. Evaluation is a linear probe on five classification datasets (CPSC2018, CSN, G12EC, PTB-XL, PTB) with subject-independent five-fold cross-validation, against supervised end-to-end models and frozen foundation models, plus ablations with and without Sundial, label efficiency, latency and CKA analyses. ECGDuo averages 83.73 against 84.00 for ECGFounder.

Critical strengths
  1. The main strength, to me, is the practical result: a 0.137M-parameter frozen encoder with a linear head that stays within 0.3 points of ECGFounder on average over five datasets and is about 20 to 55 times faster at inference (fig 3). This sounds useful to practitioners.
  2. The evaluation is careful: subject-independent folds shared by all methods, labels of the distillation corpora never used, single- and dual-teacher students matched on windows, capacity and update budget, and code released.
Critical weaknesses
  1. Novelty is limited for ICLR, and the scope, only 12-lead ECG, is too narrow (i) In my opinion the methodological novelty is limited for ICLR. Distilling several heterogeneous foundation models into one student through teacher-specific adapters is the not new, which is also the setting the paper uses to set itself apart from multi-teacher distillation: different teachers looking at the same input. The loss is the CLIP-style symmetric InfoNCE of contrastive representation distillation. Multi-teacher distillation of compact ECG models already exists. What is new is choosing a general time-series model as the second teacher. The "single-proxy transfer bottleneck" is, to me, a new name for distilling from one teacher. (ii) The theory does not support the claims. Theorem 3.1 is the chain rule plus "conditioning cannot increase entropy", and neither the complementarity term nor the residual term is ever estimated. its headline case, a student that beats its ECG teacher, does not happen: ECGDuo is below ECGFounder on four of five datasets. Proposition 3.2 is the InfoNCE bound applied to each branch; with B = 1024, 2 log B is about 13.9 nats, so the bound only says something if I(ZE; ZT) is below that, which is never checked for two embeddings of the same window… Am I missing something? (iii) The scope is too narrow for ICLR… only 12-lead ECG, 3-s windows and single-label classification on five curated datasets. The idea is generic, a domain foundation model plus a general time-series model distilled into a compact student, so if that is the claim it has to be shown on other physiological signals where both kinds of teacher exist (EEG, PPG, EMG, IMU), and on wearable single-lead ECG, which is where a 0.137M encoder would potentially matter(?). To me, clinical 12-lead ECG is not where a 30M-parameter model is a deployment problem; Fig. 3 gives ECGFounder 20 to 65 ms per batch of 128. As it stands this is an ECG compression study, and I feel it fits better applied venues.
  2. Nothing shows that the time-series teacher is what helps… (i) The gain from adding Sundial is about one Avg point over the ECGFounder-only student from a single distillation seed. The stds in Table 2 are over downstream folds and the CIs in fig 6 over head seeds, so none of them capture distillation variance, which can easily be of that size. And nothing isolates the general time-series knowledge: there is no second ECG teacher (ECG-FM plus ECGFounder), no random or weak second teacher, no Sundial-only student, and no other time-series model. Without these, the gain could come from any second target. This needs to be done thoroughly imo. (ii) some key baselines are missing; the same DSCNN trained from scratch on each task, and the same DSCNN pretrained with ECG self-supervision on the same MIMIC-IV-ECG and CODE-15 windows without any teacher (e.g., CLOCS, which is cited, or ST-MEM, ICLR 2024). Setting A compares a frozen encoder that saw about 670k unlabeled windows and two foundation teachers with supervised models trained only on the downstream folds, several of which look under-trained … so the gains say little. (iii) The protocol makes the numbers hard to compare with anything published: 3-s windows carrying record-level labels are scored one by one, the label sets are curated single-label subsets (PTB-XL without HYP and only patients with one shared superclass), and signals are not resampled across 400, 500 and 1000 Hz, so the same kernels cover 2.5 times more time on CODE-15 than on PTB. Maybe I am missing something, could you please clarify how the 3-s windows are fed to ECGFounder and ECG-FM, which were pretrained on longer recordings? Also, the abstract's "0.1% of Sundial's parameters" is not the right reference. The one that matters is ECGFounder (0.45%), which ECGDuo trails on four of five datasets, and the PTB win (91.26 vs 87.33 +- 5.23) is on 200 subjects. (iv) For an ECG paper, the ECG in Fig. 1 is not an ECG. The traces labelled "ECG Signals"; they look like generic time series (probably ai generated?). The same figure also has "GruopNorm" and "Deepthwise", and calls the linear probe "fine-tuning". To me this goes with a wider issue: there is no analysis of what cardiac information the time-series teacher adds (rhythm, intervals, morphology?), and the CKA increase toward Sundial in Fig. 7 is by construction, since the student is trained to match it.
Questions for authors

My main concerns are the two weaknesses above.I copy-apste them here. I’d be open to raise the score if the scope is generalize and all my concerns are tackled, but novelty is a key limitation in any case…

  1. Novelty is limited for ICLR, and the scope, only 12-lead ECG, is too narrow (i) In my opinion the methodological novelty is limited for ICLR. Distilling several heterogeneous foundation models into one student through teacher-specific adapters is the not new, which is also the setting the paper uses to set itself apart from multi-teacher distillation: different teachers looking at the same input. The loss is the CLIP-style symmetric InfoNCE of contrastive representation distillation. Multi-teacher distillation of compact ECG models already exists. What is new is choosing a general time-series model as the second teacher. The "single-proxy transfer bottleneck" is, to me, a new name for distilling from one teacher. (ii) The theory does not support the claims. Theorem 3.1 is the chain rule plus "conditioning cannot increase entropy", and neither the complementarity term nor the residual term is ever estimated. its headline case, a student that beats its ECG teacher, does not happen: ECGDuo is below ECGFounder on four of five datasets. Proposition 3.2 is the InfoNCE bound applied to each branch; with B = 1024, 2 log B is about 13.9 nats, so the bound only says something if I(ZE; ZT) is below that, which is never checked for two embeddings of the same window… Am I missing something? (iii) The scope is too narrow for ICLR… only 12-lead ECG, 3-s windows and single-label classification on five curated datasets. The idea is generic, a domain foundation model plus a general time-series model distilled into a compact student, so if that is the claim it has to be shown on other physiological signals where both kinds of teacher exist (EEG, PPG, EMG, IMU), and on wearable single-lead ECG, which is where a 0.137M encoder would potentially matter(?). To me, clinical 12-lead ECG is not where a 30M-parameter model is a deployment problem; Fig. 3 gives ECGFounder 20 to 65 ms per batch of 128. As it stands this is an ECG compression study, and I feel it fits better applied venues.
  2. Nothing shows that the time-series teacher is what helps… (i) The gain from adding Sundial is about one Avg point over the ECGFounder-only student from a single distillation seed. The stds in Table 2 are over downstream folds and the CIs in fig 6 over head seeds, so none of them capture distillation variance, which can easily be of that size. And nothing isolates the general time-series knowledge: there is no second ECG teacher (ECG-FM plus ECGFounder), no random or weak second teacher, no Sundial-only student, and no other time-series model. Without these, the gain could come from any second target. This needs to be done thoroughly imo. (ii) some key baselines are missing; the same DSCNN trained from scratch on each task, and the same DSCNN pretrained with ECG self-supervision on the same MIMIC-IV-ECG and CODE-15 windows without any teacher (e.g., CLOCS, which is cited, or ST-MEM, ICLR 2024). Setting A compares a frozen encoder that saw about 670k unlabeled windows and two foundation teachers with supervised models trained only on the downstream folds, several of which look under-trained … so the gains say little. (iii) The protocol makes the numbers hard to compare with anything published: 3-s windows carrying record-level labels are scored one by one, the label sets are curated single-label subsets (PTB-XL without HYP and only patients with one shared superclass), and signals are not resampled across 400, 500 and 1000 Hz, so the same kernels cover 2.5 times more time on CODE-15 than on PTB. Maybe I am missing something, could you please clarify how the 3-s windows are fed to ECGFounder and ECG-FM, which were pretrained on longer recordings? Also, the abstract's "0.1% of Sundial's parameters" is not the right reference. The one that matters is ECGFounder (0.45%), which ECGDuo trails on four of five datasets, and the PTB win (91.26 vs 87.33 +- 5.23) is on 200 subjects. (iv) For an ECG paper, the ECG in Fig. 1 is not an ECG. The traces labelled "ECG Signals"; they look like generic time series (probably ai generated?). The same figure also has "GruopNorm" and "Deepthwise", and calls the linear probe "fine-tuning". To me this goes with a wider issue: there is no analysis of what cardiac information the time-series teacher adds (rhythm, intervals, morphology?), and the CKA increase toward Sundial in Fig. 7 is by construction, since the student is trained to match it.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ATkf (1/2): a personal note; novelty, theory, scope and deployment

Dear Reviewer ATkf,

We would like to begin with something more personal than a point-by-point reply.

When your review first arrived, we read it poorly. The lowest score, together with a dense layout in which the points ran into one another, made it feel dismissive to us, and with overall scores of 2/2/1, we came close to withdrawing the paper.

We then read it again, slowly, and our view changed completely. Your review is specific, fair and plainly written in good faith: nearly every point names a real weakness of the submission together with the experiment that would test it. We are sorry for our first reaction and grateful for the time you gave the paper! Your review is the main reason we did not withdraw: even if only to answer it properly, the rebuttal was worth doing, and much of the 16-to-25-page revision is built around your list.

We ran every control you listed and follow your numbering; Tables R1 and R2 are in the general response.

1(i) Novelty

We agree that, technically, ECGDuo introduces no new component: per-teacher adapters, same-input teachers and symmetric information noise-contrastive estimation (InfoNCE) losses are established, and the revision credits UNIC, Formont et al. and compact multi-teacher electrocardiogram (ECG) distillation (§§1–2). We believe the revision contributes something of a different kind:

  • An application contribution. A compact raw-ECG encoder supervised by an ECG foundation model (ECGFM) plus a second teacher. To our knowledge, no prior work jointly distills an ECGFM and a general-purpose time-series foundation model (TSFM) into one such encoder.
  • Instructive evidence. Sundial alone gives the weakest student (80.56 vs 83.52 for ECGFounder-only, Table R1), yet next to ECGFounder it adds more label information than a second ECGFM (+0.84 vs +0.48 Avg, mean of six metrics, on frozen embeddings, Table R2).
  • An extremely compact, broadly deployable, low-cost encoder. With 1.1% of ECGFounder's parameters, ECGDuo encodes a window in 1.45 ms on one central processing unit (CPU) core, 15× faster than ECGFounder (22.32 ms); ECGDuo-Small (ECGDuo-S) fits in 0.20 MiB (mebibytes) at 8 bits and also works from lead I alone (1(iii)).

We hope this evidence helps you reassess the value and contributions of our work. As the primary area we chose, applications, indicates, we intend it to offer practical insight for ECG applications rather than a new algorithm.

1(ii) Theory

You are not missing anything: Theorem 3.1 is a motivating sufficient condition and Proposition 3.2 a bound (§3, App. D.3).

  • Proposition 3.2 is informative only if I(ZE;ZT)<2ln⁡B−LE−LT=6.30I(Z_E;Z_T)<2\ln B-L_E-L_T=6.30 nats. Our held-out InfoNCE estimate, 6.15 nats, is a lower bound capped at ln⁡B=6.93\ln B=6.93; the paper therefore uses the bound to explain the objective and measures the gains directly.
  • Theorem 3.1 gives a sufficient condition in optimal log-loss (§3.1). Empirically, the HuBERT-ECG + Sundial student exceeds the frozen HuBERT-ECG probe in Avg (80.13 vs 77.27; Table E.3).
  • The claims rest on frozen-teacher probes (Table R2), controlled second-teacher students (Table R1) and physiology probes (2(iv)).

1(iii) Scope and deployment

ECGDuo is an ECG application study; for the deployment you name, the revision adds single-lead input, per-window CPU cost, 8-bit size and noisy recordings.

Single lead. Students see only lead I; teacher targets use all 12 leads (App. E.7; ±: seed standard deviation). ECGDuo and ECGDuo-S exceed their same-size lead-I ECGFounder-only students by 1.29 and 2.03 Avg, respectively (Table E.2):

Lead-I studentCPSC2018G12ECPTB-XLAvg
ECGFounder-only73.1175.5065.9476.65 ± 0.26
ECGDuo74.4477.9766.1577.94 ± 0.13

Cost per window. On one CPU core, as in bedside monitors and embedded devices, ECGDuo is about 15× faster than ECGFounder (Table 3, replacing the original Fig. 3):

EncoderParametersSize (MiB)1 CPU core, ms
ECG-FM90,883,072346.69173.05
ECGFounder30,819,014117.5722.32
ECGDuo336,1281.281.45
ECGDuo-S134,9120.511.50

Size reference. The 0.137M you cite counted the 512-d encoder, now ECGDuo-S (134,912), plus a five-class head; ECGDuo uses a 2048-d projector. ECGDuo-S nearly matches ECGDuo (85.21 vs 85.33 Avg), and mixed 8-bit weights shrink it to 0.20 MiB while retaining 98.9% of its 32-bit Avg (App. E.6).

Noise. ECGDuo-R, distilled with artifact augmentation, keeps 99.7% of ECGDuo's clean Avg (85.07 vs 85.33). At 0 dB it drops less than both frozen ECGFMs in every column of Table 8 (electromyographic (EMG): 5.17 vs 14.52 for ECGFounder); recorded MIT-BIH Noise Stress Test Database (NSTDB) noise never enters training or selection (App. H).

AuthorsOpenReview ↗

Response to Reviewer ATkf (2/2): controls, baselines, protocol and cardiac content

2(i) Distillation variance and second-teacher controls

Every second-teacher student in Table R1 (Table 4) now has the same five independent distillation seeds (2026–2030), with all requested controls.

Variance. The seed standard deviation (SD) is 0.17–0.45 Avg (mean of six metrics over five datasets). ECGDuo exceeds the ECGFounder-only student by 1.81 and the ECGFounder + ECG-FM student by 1.12, in 5/5 paired seeds (paired t p < .001).

Any second target? A duplicated teacher adds nothing (−0.01) and a random-network target hurts (−0.85). Pretrained targets help less: ECG-FM +0.69 (5/5), permuted Sundial +0.42 (4/5); none reaches ECGDuo, and Sundial alone gives the lowest student (80.56).

Another time-series foundation model (TSFM). Over five seeds, the ECGFounder + MOMENT student reaches 83.83 ± 0.13 Avg, +0.31 over ECGFounder-only (App. E.1, E.3). Adding a TSFM also helps other electrocardiogram (ECG) teachers, and Sundial helps most (App. E.5):

ECG teacherECG-only student+ Sundial+ MOMENT
HuBERT-ECG76.8880.1378.34
MERL80.0181.1480.47

2(ii) Same-encoder baselines

Same-encoder comparison. Both requested baselines use the same depthwise-separable convolutional neural network (DSCNN), one with contrastive multi-segment coding (CMSC) self-supervision and one trained end-to-end, and ECGDuo leads both (Table 2, App. E.4):

Same DSCNN encoderCPSC2018G12ECPTB-XLAvg
CMSC self-supervision, same 671k windows, linear probe67.8866.7776.7076.15
Trained end-to-end per task78.4976.8282.4783.37
ECGDuo, frozen + linear head80.4081.5782.9485.56

The same encoder trained from scratch is the strongest supervised baseline, 2.19 Avg behind ECGDuo (0.47 on PTB-XL to 4.75 on G12EC). CMSC pairs two windows per subject, as in CLOCS; ST-MEM uses a transformer (App. E.4).

2(iii) Protocol

  • Teacher inputs. ECGFounder and ECG-FM receive exactly the student's 3-s, 12-lead window at the native rate, z-scored per lead with whole-record statistics, without teacher-specific filtering or resampling. Sundial encodes each lead in 512-sample chunks, mean-pooled over tokens, chunks and leads (§3.2, App. D.1).
  • Sampling rates. Signals are not resampled, so a fixed kernel spans different durations at 400, 500 and 1000 Hz; each benchmark is evaluated at its single rate with identical windows for every encoder; lowering the input to 250 or 125 Hz costs ECGDuo only 0.07 and 0.48 Avg (Table H.1).
  • Window scoring. Subject-level scores now average window probabilities per label unit; ECGDuo reaches 86.82 vs 85.34 for the ECGFounder + ECG-FM student (5/5 seeds; App. D.2, E.2).
  • Curated labels. We add the official multi-label PTB-XL benchmark, including hypertrophy (HYP), with record-level scoring and folds 1–8 / 9 / 10. ECGDuo's macro area under the receiver operating characteristic curve (AUROC) is above the ECGFounder + ECG-FM student on 6/6 tasks, significantly on superdiagnostic, form and rhythm, most on form (+0.045) and rhythm (+0.029) (§4.7, Table F.3).
  • PTB. We agree that 200 subjects cannot carry a claim on their own; the revision makes no claim from PTB alone.

2(iv) Fig. 1 and cardiac content

Fig. 1. The original traces were not generated by artificial intelligence: they were ECG curves shown after moderate low-pass filtering and downsampling for a cleaner display. Fig. 1 now draws recorded 3-s ECG windows directly; this link plots these windows as recorded and after low-pass filtering and downsampling. "GroupNorm" and "Depthwise" are corrected; the downstream step is named "linear probe", and the label-scarcity figure (now Fig. 5) x-axis reads "Fraction of labeled training subjects".

Centered kernel alignment (CKA). We agree that the increase toward Sundial is expected by construction; it is now descriptive only (§4.5).

Which cardiac information improves. Ridge probes on frozen features, ECGDuo minus the ECGFounder-only student, 5 seeds (Table 7; RR: interval between consecutive R peaks):

ΔR²PTB-XLCPSC2018
RMSSD (root mean square of successive RR differences)+0.067+0.070
RR coefficient of variation+0.054+0.069
RR min/max ratio+0.060+0.074
Heart rate+0.020+0.029

All four gains hold in 5/5 seeds (paired t p < .01), with a multilayer-perceptron (MLP) probe as well (App. G.1).

Thank you again for the time and care you gave this review. Whatever your final assessment, the paper is substantially better because of it, and we would be grateful if you would reassess the value and contribution of this application study in light of these results.

Reviewer_ATkfOpenReview ↗

Response to the authors’ rebuttal

Dear authors,

Before turning to the content: I would prefer a more professional tone in this exchange. Remarks on how my review was first read, on its layout, or on your deliberations about withdrawing the paper are not appropriate in a rebuttal and have no bearing; in the opinion of this reviewer.

On the revision. Thank you for the substantial additional work. Several of my soundness concerns are addressed by the five-seed distillations, the second-teacher controls, the same-encoder baselines, the clarified teacher inputs and the corrected figg 1. The theory is also now appropriately framed as motivation. I still have two concerns

(i) The headline model changed during the rebuttal. ECGDuo was re-tuned. I did not find a search space, budget or validation split, and I could not find whether the ECGFounder-only and ECGFounder + ECG-FM students were tuned the same way. Please clarify if possible.

(ii) The official ptbxl benchmark does not reproduce Table 2. Table 2 places ECGDuo above ECGFounder, yet on the official benchmark ECGDuo trails the frozen ECGFounder probe on 5/6 tasks (Table 6). Its all-statement macro AUROC is also below that of the smaller ECGFounder-only student . It is on par with the Wavelet+nn feature baselin that Strodthoff (2021) report on the same folds, where xresnet1D101 and Inception1D reach 0.925. In Table 2 these architectures trail the DSCNN trained from scratch. I remain concerned that the Setting A baselines may be under-trained and that the curated protocol drives much of the reported advantage. Please let me know what part of my reasoning is not correct, or how you argument for that.

All in all, scope remains one of my main concerns. You state that ECGDuo "introduces no new component", that it "is an ECG application study", and that you "intend it to offer practical insight for ECG applications rather than a new algorithm" (or method). The finding the revision supports is therefore specific to ECG and to one teacher. ICLR is a machine learning venue. Even in its application areas, I expect claims that generalize beyond a single signal type and matter to the broader machine learning research community. I believe the work is not bad, but I strongly consider this paper out of scope for ICLR. It fits way better the scope of MLHC, IEEE JBHI and other technical-translational venues, where it would reach its natural audience.

Reviewer_hnkCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ECGDuo distills an ECG foundation model and a general time-series foundation model into a compact ECG encoder using teacher-specific adapters and symmetric contrastive losses. The student is evaluated through frozen-encoder linear probing on five datasets, with analyses of teacher contributions, label efficiency, representation alignment, and inference cost.

Critical strengths
  1. The student model with much less parameter achieves an average score of 83.73 across five datasets, close to ECGFounder, while removing both teachers and adapters at inference. This provides a useful accuracy–size trade-off.

  2. Single-teacher and dual-teacher comparisons control the student architecture, training data, optimizer, and update budget. Improvements across two ECG teachers and five datasets, together with subject-independent evaluation and label-scarcity experiments, support the practical benefit of additional TSFM supervision.

Critical weaknesses
  1. The ablations establish that adding Sundial helps, but do not isolate cross-domain temporal complementarity. A Sundial-only student and an ECGFounder+ECG-FM student are missing. These controls would distinguish the proposed heterogeneous pairing from generic multi-teacher benefits.

  2. The theoretical conditions are not connected sufficiently to the trained model. Neither task-relevant complementarity nor residual compression loss is measured, and the contrastive objective does not guarantee their required relationship. CKA changes measure representation similarity rather than preservation of task-relevant information.

  3. The evaluation unit is unclear. Table 3 specifies record-, patient-, or subject-level targets, whereas Appendix B.2 describes evaluation using three-second windows without an aggregation rule. This affects interpretation and sample weighting.

  4. Across Tables 4–5, the reported Avg standard deviations match the arithmetic mean of the six metric standard deviations to rounding. This generally differs from the across-fold standard deviation of the composite score. The calculation should be clarified and verified using fold-level results.

Questions for authors
  1. How does ECGDuo compare with capacity-matched Sundial-only and two-ECG-teacher students under the same training budget?
  2. How are window predictions aggregated into the stated evaluation units, and how are the Avg standard deviations computed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hnkC (1/2): teacher controls and theory

Dear Reviewer hnkC,

Thank you for a careful and constructive review. You caught two ambiguities: the evaluation unit and the standard deviation (SD) of Avg, our mean of six metrics. Fixing them made the paper more precise, and the subject-level scores they led to strengthened the main result: ECGDuo's margins grow. Your request for Sundial-only students and students with two electrocardiogram (ECG) teachers became the core of the new second-teacher study (Table 4). This post answers W1–W2; the next answers W3–W4. Tables R1 and R2 are in the general response.

W1 / Q1. Sundial-only and two-ECG-teacher students

Both are added. They use ECGDuo's encoder (336,128 parameters) and the same 671k unlabeled windows, and ECGDuo exceeds both (Table R1, Table 4; seeds 2026–2030; k/n: seeds with Δ > 0):

StudentAvgSubject AvgΔ Avg vs ECGFounder-only (k/n)
Sundial-only80.56 ± 0.4582.32−2.96 (0/5)
ECGFounder-only83.52 ± 0.2984.92—
ECGFounder + ECG-FM84.21 ± 0.2885.34+0.69 (5/5)
ECGDuo85.33 ± 0.2486.82+1.81 (5/5)

Against two ECG teachers. ECGDuo exceeds the ECGFounder + ECG-FM student by 1.12 Avg, and by 1.48 at the subject level, in 5/5 paired seeds (paired t p < .001 and .003).

Against the time-series foundation model (TSFM) alone. The Sundial-only student is the weakest configuration in Table R1, 2.96 below ECGFounder-only (0/5 seeds), so the TSFM helps next to an ECG teacher, not in place of it.

At a smaller size. ECGDuo-Small (ECGDuo-S; 134,912 parameters) reaches 85.21 ± 0.32 Avg, +1.84 over its same-size ECGFounder-only student (5/5, p < .001; Tables E.1–E.2).

Generic multi-teacher effects. A duplicated or untrained second target does not raise Avg (−0.01, −0.85). Permuted Sundial targets give +0.42 (4/5, paired t p = .070), well below ECGDuo (83.94 vs 85.33).

Before distillation, on frozen embeddings, Sundial adds +0.84 Avg to ECGFounder and ECG-FM adds +0.48; after principal component analysis (PCA) to 1,024 dimensions, +0.18 versus −0.60 (Table R2).

W2. Theory, residual terms and centered kernel alignment (CKA)

Theorem 3.1 is an elementary sufficient condition: the revision states that it motivates the design and measures the resulting gains directly (§3.1).

  • Task-relevant complementarity, measured on frozen teachers. A linear probe is the measurable counterpart of ΔT∣E\Delta_{T|E} (App. A.1). ECGFounder ⊕ Sundial lowers held-out log-loss by 0.017 nats while ECGFounder ⊕ ECG-FM leaves it unchanged (Table R2). On official PTB-XL, it raises macro area under the receiver operating characteristic curve (AUROC) over ECGFounder on 6/6 tasks, e.g., +0.048 on all statements, paired-bootstrap 95% confidence interval excluding 0 (Table 6).
  • Proposition 3.2. It is informative only if I(ZE;ZT)<2ln⁡B−LE−LT=6.30I(Z_E;Z_T)<2\ln B-L_E-L_T=6.30 nats, with B=1024B=1024, LE=2.84L_E=2.84 and LT=4.72L_T=4.72. Our held-out information noise-contrastive estimation (InfoNCE) estimate, 6.15 nats, is a lower bound capped at ln⁡B=6.93\ln B=6.93, so the bound explains the objective, while the gains are measured directly (Tables R1, R2; App. D.3).
  • An empirical student–teacher advantage. In the teacher study of App. E.5, the HuBERT-ECG + Sundial student reaches 80.13 Avg, above the frozen HuBERT-ECG probe (77.27) and the HuBERT-ECG-only student (76.88; App. D.3).
  • CKA. We agree that CKA measures similarity, not task information; it is now descriptive only (§4.5). Task information in the trained encoder is measured by probes instead: ECGDuo predicts RMSSD (root mean square of successive differences between RR intervals, the intervals between consecutive R peaks) better than the ECGFounder-only student, ΔR² +0.067 on PTB-XL and +0.070 on CPSC2018, in 5/5 seeds (Table 7).

(Continued in the next post: evaluation unit and Avg SD.)

AuthorsOpenReview ↗

Response to Reviewer hnkC (2/2): evaluation unit and Avg standard deviation

W3 / Q2. Evaluation unit and aggregation

Scoring was per 3-s window, with no aggregation; the unit column of the original Table 3, now Table B.1, gave the label unit. Each window inherits the label and fold of its unit, so units with more windows carry more weight (App. D.2).

Label units. Record for CPSC2018, CSN and G12EC; patient for PTB-XL; subject for PTB. All recordings of a subject fall in one fold.

New subject-level scores. Window softmax probabilities are averaged within each label unit before the metrics are computed, so every unit counts once; head training and selection are unchanged. Avg is the mean of six metrics; ± is the SD over seeds.

Student (seeds 2026–2030)CPSC2018G12ECPTB-XLAvg, windowAvg, subject
ECGFounder-only80.8479.3781.9183.52 ± 0.2984.92 ± 0.33
ECGFounder + ECG-FM80.9880.7581.0884.21 ± 0.2885.34 ± 0.40
ECGDuo82.8982.5182.7485.33 ± 0.2486.82 ± 0.38

Per-dataset columns are at the subject level; all five datasets are in App. E.1. The ranking is unchanged and ECGDuo's margins grow: +1.90 over ECGFounder-only and +1.48 over ECGFounder + ECG-FM (5/5 seeds each, paired t p = .003), while the ECG-FM student's own gain shrinks to +0.42 (App. E.2).

Record-level official benchmark. On official PTB-XL, each record is embedded as the mean of its window embeddings and scored once (folds 1–8 / 9 / 10, multi-label, App. F.2). Macro area under the receiver operating characteristic curve (AUROC), mean over seeds 2026–2030; * paired t p < 0.05 (Table F.3):

PTB-XL taskECGDuoΔ vs ECGFounder-onlyΔ vs ECGFounder + ECG-FM
All statements0.848+0.013+0.011
Diagnostic0.867+0.011+0.010
Subdiagnostic0.890+0.006+0.004
Superdiagnostic0.903+0.008*+0.012*
Form0.811+0.031+0.045*
Rhythm0.907+0.009+0.029*

ECGDuo is above both students on every task. All-statement area under the precision–recall curve (AUPRC) rises by 0.022 over both students in 5/5 seeds (Table F.3).

W4 / Q2. Avg standard deviation

You are right: the original Avg SD was the mean of the six per-metric SDs, which generally differs from the SD of the composite.

Composite SD. For fold ff, afa_f is the mean of accuracy, precision, recall, F1, AUROC and AUPRC; the composite SD is the SD of the five afa_f.

Computed from ECGDuo's five folds, the composite SD is smaller on every dataset (App. C):

SpreadCPSC2018CSNG12ECPTB-XLPTB
Mean of metric SDs1.430.471.090.654.06
SD of fold-level Avg1.390.351.060.453.86

What each table shows. Table 2 keeps the original spread convention (App. D.2). Table C.1 (original Tables 4–5) follows the same convention metric by metric. The new multi-seed student tables report the SD over distillation seeds (0.17–0.45 Avg for the configurations in Table 4), and Table 4 comparisons are paired tests on seeds. App. D.2 distinguishes these definitions.

Thank you again for pointing us to these fixes; the paper is clearer and its evidence stronger because of them. We hope the clarified protocol and the new controls address your concerns, and we are happy to answer any further questions.

Reviewer_mnjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ECGDuo, a lightweight ECG encoder distilled from an ECG foundation model and a general time-series foundation model. Two teacher-specific adapters align the shared student representation with the respective teachers using symmetric contrastive distillation. Experiments on five ECG datasets demonstrate competitive performance with substantially fewer parameters.

Critical strengths

The motivation for compressing large ECG foundation models into lightweight encoders is practically relevant.

The proposed model achieves competitive performance with only 0.137M parameters, demonstrating promising deployment efficiency.

The evaluation includes five datasets, subject-independent cross-validation, teacher ablations, and representation analyses.

Critical weaknesses

W1. Limited methodological novelty. The framework primarily combines established techniques, including multi-teacher distillation, independent projection heads, and contrastive representation alignment. Related methods such as UNIC (ECCV 2024) and Formont et al. (NeurIPS 2025) have already investigated complementary representation learning through multiple teachers. The main distinction appears to be the combination of ECG and general time-series foundation models rather than a fundamentally new distillation mechanism.

W2. Insufficient evidence for the central motivation. The paper assumes that a general time-series foundation model contains task-relevant temporal information absent from ECG-specific representations. However, this assumption is not directly verified. The conditional mutual information used to define complementarity is never estimated, and the experiments do not establish whether Sundial provides additional predictive information beyond the ECG teacher.

W3. Incomplete ablation studies. Improvements over ECG-only distillation demonstrate the benefit of additional supervision, but not necessarily complementary cross-domain knowledge. Important controls, including TSFM-only distillation, two ECG-specific teachers, and duplicated-teacher supervision, are missing. Without these comparisons, the improvements may reflect generic multi-teacher regularization rather than the claimed complementarity.

W4. Limited theoretical and experimental validation. The theoretical results rely on general information-theoretic identities and assumptions that are not verified experimentally. Furthermore, CKA improvements and better feature clustering do not establish that complementary diagnostic information is retained. The main comparisons also lack compact pretrained student baselines and repeated independent distillation runs.

Questions for authors

Novelty: What is the fundamental methodological difference between ECGDuo and existing multi-teacher representation distillation approaches, particularly UNIC and Formont et al.?

Complementarity: Can the authors demonstrate that combining the frozen ECG and TSFM teacher representations improves downstream prediction over the ECG teacher alone?

Ablations: Would distillation from two ECG-specific teachers or duplicated ECG-teacher supervision yield comparable improvements?

Mechanism: How do the theoretical complementarity conditions relate to the actual optimization objective? Can the authors demonstrate which diagnostic information is uniquely transferred from Sundial?

Evaluation: Can the authors include comparable compact pretrained baselines, report variability across independent distillation runs, and clarify potential pretraining-data overlap with downstream benchmarks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mnjP (1/2): novelty, complementarity and ablations

Dear Reviewer mnjP,

Thank you for a careful and generous review. Your questions were precise in the most useful way: each one named the experiment that would answer it, and we ran all of them. Your complementarity question (W2), whether frozen ECGFounder and Sundial together predict better than ECGFounder alone, is now a main-text analysis (Table 5), and its answer is a clear yes. This post answers W1–W3; the next answers W4. Tables R1 and R2 are in the general response.

W1 / Q-Novelty. Difference from UNIC and Formont et al.

The difference lies in the teacher pair and the question tested, not in the distillation mechanism. As an application study (our primary area), the contribution is an insight for electrocardiogram (ECG) practice: whether a general time-series foundation model (TSFM) or a second ECG foundation model (ECGFM) makes a compact ECG encoder stronger. Per-teacher adapters, same-input teachers and symmetric information noise-contrastive estimation (InfoNCE) losses are established; the revision credits them and removes the term "single-proxy transfer bottleneck" (abstract, §1, §5).

TeachersStudentQuestion tested
UNICseveral vision modelsimage encodermulti-task vision transfer
Formont et al.diverse pretrained embedderstask-agnostic encodergeneric transfer
ECG multi-teacher self-distillation (Wang et al., 2025)deeper branches of an ECG modellightweight arrhythmia classifierarrhythmia recognition
ECGDuoECGFM + general TSFMcompact raw-ECG encoder, linear probewhich second teacher helps an ECGFM

To our knowledge, no prior work jointly distills an ECG-specialist foundation model and a general-purpose time-series foundation model into one compact raw-ECG encoder. The pairing is not obvious: the Sundial-only student is the weakest in Table R1 (80.56 vs 83.52 for the ECGFounder-only student), yet next to ECGFounder it adds more label information than a second ECGFM does (Table R2). Adding Sundial also raised the Avg (mean of six metrics over the five datasets) of students of other ECG teachers, by 3.25 for HuBERT-ECG and 1.13 for MERL (App. E.5).

W2 / Q-Complementarity. Do frozen ECGFounder and Sundial together predict better than ECGFounder alone?

Yes. A linear probe on frozen ECGFounder ⊕ Sundial raises Avg on 5/5 datasets, by 0.84 on average, and lowers held-out log-loss by 0.017 nats; ECG-FM adds 0.48 and leaves log-loss unchanged. After principal component analysis (PCA) to 1,024 dimensions, the Sundial gain stays positive at +0.18 and the ECG-FM gain turns negative at −0.60 (Table R2). This probe gain is the measurable linear-probe counterpart of the conditional mutual information (App. A.1).

Official PTB-XL. The benchmark gives the same picture (folds 1–8 / 9 / 10, macro area under the receiver operating characteristic curve (AUROC)):

PTB-XL taskECGFounder ⊕ SundialΔ vs ECGFounderΔ vs ECGFounder ⊕ ECG-FM
All statements0.919+0.048*+0.045*
Diagnostic0.919+0.039*+0.047*
Subdiagnostic0.922+0.041*+0.029*
Superdiagnostic0.918+0.009*0.000
Form0.868+0.047*+0.038*
Rhythm0.956+0.040*+0.024

* Paired-bootstrap 95% confidence interval (CI) over test records excludes 0. Both concatenations are validation-selected (Table F.2).

W3 / Q-Ablations. Two ECG teachers, duplicated teacher, TSFM only

Two ECG teachers help, but less than ECGDuo does; a duplicated teacher does not help (Table R1, Table 4).

  • ECGFounder + ECG-FM: 84.21, +0.69 over the ECGFounder-only student (5/5 seeds). ECGDuo, 85.33, exceeds it by 1.12 in 5/5 paired seeds (paired t p < .001).
  • Duplicated ECGFounder: 83.51, −0.01 (2/5). An untrained network target gives 82.67, −0.85 (0/5).
  • Sundial-only: 80.56, −2.96 (0/5).
  • Permuted Sundial targets: 83.94, +0.42 (4/5, paired t p = .070), well below ECGDuo's 85.33.

So the largest gain comes from pairing Sundial with the ECG teacher; neither a second branch on the same teacher nor Sundial alone comes close.

(Continued in the next post: theory, mechanism and evaluation.)

AuthorsOpenReview ↗

Response to Reviewer mnjP (2/2): theory, mechanism and evaluation

W4 / Q-Mechanism. How the theory relates to the objective

The theory motivates the objective, and the revision now states its role precisely (§3.1, App. D.3).

  • Theorem 3.1 is a sufficient condition that follows from the chain rule: if the task-relevant complementarity of the time-series foundation model (TSFM) exceeds the information lost in compression, the student's optimal log-loss falls below the electrocardiogram (ECG) teacher's. Table R2 measures the teacher-side predictive gain through frozen linear probes; the student gains in Table R1 then show the benefit that reaches the compact encoder.
  • Proposition 3.2 links the two optimized branch losses to joint retention. It is informative only if I(ZE;ZT)<2ln⁡B−LE−LT=6.30I(Z_E;Z_T)<2\ln B-L_E-L_T=6.30 nats, with B=1024B=1024, LE=2.84L_E=2.84 and LT=4.72L_T=4.72. Our held-out information noise-contrastive estimation (InfoNCE) estimate, 6.15 nats, is a lower bound capped at ln⁡B=6.93\ln B=6.93, so it cannot settle the condition.

W4 / Q-Mechanism. Which diagnostic information is uniquely transferred from Sundial?

Diagnostic probes. The revision measures what ECGDuo represents better than the ECGFounder-only student, rather than attributing information to Sundial alone (Table 7, App. G.1; RR interval: interval between consecutive R peaks):

Ridge probe on frozen features, ΔR²PTB-XLCPSC2018
RMSSD (root mean square of successive RR differences)+0.067+0.070
RR coefficient of variation+0.054+0.069
RR min/max ratio+0.060+0.074
Heart rate+0.020+0.029

All four targets improve in 5/5 seeds (paired t p < .01). RMSSD error falls by 3.4 / 6.5 ms (Table G.1), and a multilayer-perceptron (MLP) probe agrees in 5/5 seeds (p ≤ .004).

Per class, all 20 classes improve, most for long QT (+9.48 F1) and sinus arrhythmia (+5.74; App. G.2). On official PTB-XL, ECGDuo's largest area under the receiver operating characteristic curve (AUROC) gains over the ECGFounder + ECG-FM student are on form (+0.045) and rhythm (+0.029) statements (Table F.3). We agree that centered kernel alignment (CKA) and cluster quality measure geometry, not diagnostic content; the revision presents CKA as descriptive only (§4.5).

W4 / Q-Evaluation. Compact baselines, run variability, data overlap

  • Compact pretrained baselines. The same depthwise-separable convolutional neural network (DSCNN) pretrained with contrastive multi-segment coding (CMSC), as in CLOCS, on the same 671k unlabeled windows reaches 76.15 Avg (mean of six metrics over five datasets); trained end-to-end per task, it reaches 83.37 (Table 2, App. E.4). The distilled students of Table R1 are compact pretrained baselines as well, and ECGDuo leads all of them.
  • Independent distillation runs. The second-teacher students in Table R1 share five distillation seeds (2026–2030); their seed standard deviation (SD) is 0.17–0.45 Avg, and their comparisons are paired on seeds (App. E.1–E.3).
  • Pretraining-data overlap. ECGDuo's distillation corpora, MIMIC-IV-ECG and CODE-15 (labels unused), share no source with the five benchmarks; its teachers are ECGFounder, pretrained on the Harvard–Emory ECG Database, and Sundial, a general-purpose TSFM (App. D.1).

Thank you again for the questions that made the paper sharper. We hope these answers resolve your concerns, and we would be glad to clarify anything further.

Reviewer_mnjPOpenReview ↗

Response

Thank you to the authors for the detailed response. While the additional experiments address some of my soundness concerns, the discrepancy between the curated evaluation and the official benchmark remains a critical issue.

Dear Reviewer mnjP,

Thank you for your reply. We are glad that the additional experiments resolved some of your concerns, and we especially appreciate that you read the full discussion, including our recent exchange with Reviewer ATkf. This shows real care, and we are grateful for your time and effort.

(i) Curated tasks vs. the official PTB-XL benchmark

The two evaluations ask different questions. The curated tasks (Table 2) test five single-label diagnoses across five datasets. The official ptbxl benchmark (Table 6) tests 71 multi-label statements per record. On the official benchmark, the main findings hold:

  • The TSFM teacher helps at every width. The TSFM-supervised student beats its ECGFM-only counterpart on all 6 tasks in macro AUPRC, at both 2048-d and 512-d (Table F.1).
  • A compact student can match the teacher. ECGDuo-Small beats frozen ECGFounder on all statements (0.890 vs. 0.871 macro AUROC) with 0.44% of its parameters.
  • ECGDuo is competitive end-to-end. Fine-tuned on the official ptbxl protocol, it reaches 0.924 macro AUROC on all statements. That matches the published XResNet1D and Inception1D (0.925). More details are in our most recent exchange with Reviewer ATkf.

To align every claim with the evidence behind it, we will report the official benchmark separately for each width and limit the comparison with frozen ECGFounder to the curated tasks. The central finding, that a general TSFM adds complementary supervision to an ECG foundation model, holds under both evaluations.

Thank you again for your careful engagement!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: thank you, and what the revision adds

We thank all three reviewers for the specific experiments you requested. The revision grows from 16 to 25 pages: five new appendix sections (D–H) report the teacher-target controls, repeated distillations over five seeds, same-encoder baselines, the official PTB-XL benchmark, lead-I and corrupted-input experiments, and measured deployment cost. Revisiting the training objective in light of mnjP's and hnkC's questions on how the complementarity conditions relate to it, we found that the submission had not adapted the hyperparameters of the full ECGDuo framework, such as the Sundial weight wTw_T, fixed to equal weighting. We now select them on validation data only (App. D.5); Table 2 and Fig. 2 report the resulting ECGDuo, whose 2048-d projector gives 336,128 parameters. Changes are color-marked by the reviewer.

Paired with an electrocardiogram (ECG) foundation model, Sundial gives the strongest compact student we trained, and on frozen embeddings it adds more to ECGFounder than a second ECG foundation model does.

Table R1. Second-teacher controls (Table 4; mean over seeds 2026–2030; Avg: mean of six metrics over the five datasets)

StudentCPSC2018G12ECPTB-XLAvgSubject AvgΔ Avg vs ECGFounder-only (k/n, p)
Sundial-only71.9677.5274.6480.56 ± 0.4582.32−2.96 (0/5, < .001)
ECGFounder + random network76.9376.5280.8082.67 ± 0.2484.11−0.85 (0/5, .008)
ECGFounder-only78.2078.1981.8783.52 ± 0.2984.92—
ECGFounder + duplicated ECGFounder78.1378.0681.8483.51 ± 0.1785.00−0.01 (2/5, .95)
ECGFounder + permuted Sundial78.3579.0581.8883.94 ± 0.1885.26+0.42 (4/5, .070)
ECGFounder + ECG-FM78.8079.5881.0384.21 ± 0.2885.34+0.69 (5/5, .021)
ECGDuo (ECGFounder + Sundial)80.0781.3382.6285.33 ± 0.2486.82+1.81 (5/5, < .001)

±: standard deviation (SD) over the five distillation seeds; k/n: seeds with Δ > 0; p: two-sided paired t test on seeds. All five datasets: App. E.1.

ECGDuo leads every configuration. +1.12 Avg over the ECGFounder + ECG-FM student and +1.81 over ECGFounder-only, in 5/5 paired seeds each (p < .001); +1.48 and +1.90 at the subject level (App. E.2). Even its lowest seed is above the best seed of every other configuration in Table R1 (at most 84.62).

An extra branch is not enough. A duplicated target (−0.01) and an untrained target (−0.85) do not raise Avg. Pretrained second targets help, but less than Sundial does in ECGDuo: ECG-FM +0.69 (5/5), permuted Sundial +0.42 (4/5).

Table R2. Frozen teacher embeddings, before any distillation (Table 5, App. F.1)

Linear probe on concatenated frozen embeddings, change from frozen ECGFounder; principal component analysis (PCA) rows reduce them to 1,024 dimensions on training windows.

Frozen embeddingCPSC2018G12ECPTB-XLAvgDatasets upΔ log-loss (nats)
ECGFounder ⊕ Sundial+1.13+0.39+0.23+0.845/5−0.017
ECGFounder ⊕ ECG-FM+0.55+0.18+0.22+0.485/5|Δ| < 0.001
ECGFounder ⊕ Sundial, PCA+0.88+0.01+0.22+0.184/5—
ECGFounder ⊕ ECG-FM, PCA+0.65−0.53+0.05−0.603/5—

Sundial adds more label information to ECGFounder than ECG-FM does, and at matched dimensions the Sundial gain stays positive (+0.18) while the ECG-FM gain turns negative (−0.60). On official PTB-XL, ECGFounder ⊕ Sundial significantly raises macro area under the receiver operating characteristic curve (AUROC) over ECGFounder on 6/6 tasks and over ECGFounder ⊕ ECG-FM probe on 4/6 (paired-bootstrap 95% confidence intervals; Table 6).

Shared concerns

ConcernResolutionWhere
Novelty (mnjP, ATkf)Prior multi-teacher work credited; contribution framed as an application insight: which second teacher makes a compact ECG encoder stronger§§1–2
Theory (all)Theorem 3.1 as a motivating sufficient condition, Prop. 3.2 as a bound§3, App. D.3
Evaluation (hnkC, ATkf)Window scoring stated; subject-level scores and fold-level SDs addedApp. C, D.2, E.2
Curated labels (ATkf)Official PTB-XL: ECGDuo above the ECGFounder + ECG-FM student on 6/6 tasks§4.7, App. F.2
Cardiac content (all)Probes of variability in intervals between successive R peaks (RR intervals) improve in all 5 seeds; centered kernel alignment (CKA) descriptive only§4.5/4.7, App. G.1
Compactness (ATkf)Compact encoder: ECGDuo-Small (ECGDuo-S) has 134,912 parameters (0.44% of ECGFounder), 85.21 Avg (seeds 2026–2030), 1.50 ms per window on one processor core§4.4, Tables 3, E.1

Again, thank you all for your insightful reviews, which strengthened the paper.

Reviewer_ATkfOpenReview ↗

Final Response

Dear authors,

Thank for following up. The value, strengths, limitations, and caveats, are clear to me. For now, following the discussions with all reviewers and myself, I keep my general score while updating the soundness metric. Thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.