ECGDuo: Complementary Distillation from ECG and Time-Series Foundation Models

Foundation models provide strong representations for electrocardiogram (ECG) analysis, but full ECG foundation models remain expensive for clinical settings that require compact, fast inference and efficient adaptation. A common remedy is to distill a large ECG teacher into a small encoder. ECG-only distillation, however, suffers from a *single-proxy transfer bottleneck*: the transferability of the student is bounded by the representational coverage of the ECG teacher proxy, which fails to capture temporal knowledge that transfers across time-series domains. We therefore propose **ECGDuo**, a compact ECG encoder distilled from two complementary teachers: *(i)* an ECG-specific teacher that transfers cardiac knowledge, and *(ii)* a general time-series teacher that injects domain-agnostic temporal knowledge. ECGDuo uses symmetric contrastive distillation to transfer both forms of knowledge into a lightweight raw-ECG encoder. Each teacher first encodes the same raw ECG signal into either an ECG-specific representation or a domain-agnostic temporal representation. Branch-specific adapters then map the shared student representation into the corresponding teacher spaces, where matched student–teacher pairs are aligned. Across five downstream ECG datasets, ECGDuo consistently outperforms baselines trained from scratch and remains competitive with full-scale foundation models, using only a **0.1% parameter budget**. All code is available in the [***Anonymous GitHub Repository***](https://anonymous.4open.science/r/ECGDuo-8848).