- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents CARP, a framework for text-conditioned ECG generation that combines a pretrained world model with a rectified-flow generator and an auxiliary clinical measurement loss. The proposed objective uses soft probabilities for ECG wave components to estimate their boundaries and amplitudes, enabling the computation of differentiable clinical measurements. The method also includes a lead-consistency loss to preserve the identities of individual ECG leads. Experiments on MIMIC-IV-ECG, PTB-XL, and HEEDB show improvements in clinical measurement errors, report alignment, and the utility of synthetic ECGs for training diagnostic classifiers.
Critical strengths
S1. The paper focuses on an important limitation that existing ECG generation methods cannot faithfully reproduce the morphological characteristics.
S2. The differentiable clinical measurement loss is rational and sound.
S3. The evaluation covers multiple aspects of generation quality.
Critical weaknesses
W1. The clinical measurement loss may not adequately enforce sample-specific alignment between clinical reports and generated ECGs. The proposed loss constructs soft histograms of clinical measurements and compares their cumulative distributions between real and generated ECGs. However, matching measurement distributions does not necessarily guarantee that each generated ECG exhibits the measurements appropriate for its own conditioning report. A model could potentially achieve good aggregate measurement statistics while producing individual ECGs with clinically inconsistent features.
W2. The method relies on a pretrained ECG segmentation model whose errors may affect the proposed supervision. The clinical measurement loss depends on differentiable wave delineation using SemiSegECG. Consequently, errors in estimating P-wave, QRS-complex, and T-wave boundaries can propagate into the computed measurements and the gradients used to train CARP. Such errors may be particularly consequential for abnormal ECGs, where wave morphology can deviate substantially from common patterns.
W3. The generated ECGs lack the direct assessment of the clinical validity. The paper reports substantial improvements in downstream classification performance, including for rare conditions and external datasets. These are encouraging results, but direct assessment such as cardiologist inspection would further strengthen the claims.
Questions for authors
Could the authors clarify whether the VAE used in CARP’s generative pipeline is trained from scratch or adopted from DiffuSETS? In Section 2.1, the authors state that they train a VAE to encode and decode ECG segments. However, Appendix C.3 states that all ECGs are encoded using the VAE released by DiffuSETS, and Table 7 further examines the choice of VAE encoder.
6 comments
W2
W2: The method relies on a pretrained ECG segmentation model whose errors may affect the proposed supervision.
Thank you for this pertinent concern. Empirically, we find that SemiSegECG’s segmentation is not an issue for learning, since CARP produces more accurate morphology for normal and abnormal ECGs for both RQ1 (train on synthetic, test on real), and RQ2 (train on real, test on synthetic), including, as shown in RQ1, on rare-conditions which have highly abnormal ECG morphology.
In addition, we evaluate SemiSegECG’s performance on abnormal ECGs, which is a SOTA model for ECG segmentation across morphologies. Since our existing datasets lack annotated segmentation labels, we use the LUDB dataset [1], which has 200 ECGs with cardiologist-annotated segmentation. We compare the precision and recall of detecting P, QRS and T waves within a 100ms region. We find that SemiSegECG and ECGDeli are generally well-performing tools, with SemiSegECG performing better in a number of instances.
| Wave / label family | SemiSegECG recall | SemiSegECG precision | ECGDeli recall | ECGDeli precision |
|---|---|---|---|---|
| P peak | ||||
| Normal / sinus rhythm | 0.99 | 0.99 | 0.98 | 0.98 |
| Sinus bradycardia | 0.99 | 1.00 | 0.98 | 0.99 |
| Bundle-branch / IV block | 0.99 | 0.94 | 0.98 | 0.78 |
| AV / fascicular / axis | 0.99 | 1.00 | 0.98 | 0.82 |
| Atrial / ventricular ectopy | 0.97 | 0.97 | 0.98 | 0.89 |
| Hypertrophy / enlargement | 0.99 | 0.97 | 0.98 | 0.87 |
| Myocardial infarction | 1.00 | 0.99 | 1.00 | 0.81 |
| Ischemia / ST-T | 0.98 | 0.99 | 0.98 | 0.88 |
| Misc. rare | 1.00 | 1.00 | 1.00 | 0.38 |
| All | 0.99 | 0.98 | 0.98 | 0.87 |
| QRS peak | ||||
| Normal / sinus rhythm | 1.00 | 1.00 | 0.99 | 1.00 |
| Sinus bradycardia | 1.00 | 1.00 | 1.00 | 1.00 |
| AF / flutter | 1.00 | 1.00 | 0.97 | 1.00 |
| Bundle-branch / IV block | 1.00 | 1.00 | 1.00 | 1.00 |
| AV / fascicular / axis | 1.00 | 1.00 | 0.99 | 1.00 |
| Atrial / ventricular ectopy | 1.00 | 1.00 | 0.93 | 1.00 |
| Hypertrophy / enlargement | 1.00 | 1.00 | 0.99 | 1.00 |
| Myocardial infarction | 1.00 | 1.00 | 1.00 | 1.00 |
| Ischemia / ST-T | 1.00 | 1.00 | 0.99 | 1.00 |
| Misc. rare | 1.00 | 1.00 | 0.96 | 1.00 |
| All | 1.00 | 1.00 | 0.99 | 1.00 |
| T peak | ||||
| Normal / sinus rhythm | 0.97 | 0.97 | 0.96 | 0.97 |
| Sinus bradycardia | 0.98 | 0.98 | 0.94 | 0.94 |
| AF / flutter | 0.91 | 0.96 | 0.84 | 0.86 |
| Bundle-branch / IV block | 0.96 | 0.97 | 0.95 | 0.96 |
| AV / fascicular / axis | 0.95 | 0.97 | 0.92 | 0.93 |
| Atrial / ventricular ectopy | 0.97 | 0.97 | 0.89 | 0.97 |
| Hypertrophy / enlargement | 0.96 | 0.97 | 0.93 | 0.95 |
| Myocardial infarction | 0.96 | 0.96 | 0.91 | 0.91 |
| Ischemia / ST-T | 0.95 | 0.96 | 0.93 | 0.93 |
| Misc. rare | 0.92 | 0.97 | 0.81 | 0.85 |
| All | 0.96 | 0.97 | 0.95 | 0.96 |
[1] Yusipov, I. et al (2021). Lobachevsky University Electrocardiography Database. https://www.physionet.org/content/ludb/1.0.1/
Q1
Q1: Could the authors clarify whether the VAE used in CARP’s generative pipeline is trained from scratch or adopted from DiffuSETS?
We clarify that CARP's generative VAE is trained from scratch and is different from the DiffuSETS VAE used in our evaluation. CARP's VAE (7M parameters) encodes each ECG into a sequence of latent tokens. In contrast, the DiffuSETS VAE (30M parameters) encodes an entire ECG into a single latent. This architectural difference is motivated by the original Cosmos 3's tokenization scheme, which represents inputs such as videos as sequences of latent tokens.
For downstream classification evaluation, we used the DiffuSETS VAE to ensure fair comparability with prior work. This evaluation choice is independent of CARP's generative architecture, since the DiffuSETS VAE encodes the synthetic ECGs after CARP has produced them. We additionally ablate the choice of classifier input representation in Table 7, comparing DiffuSETS VAE embeddings against raw ECG waveforms, and find that our conclusions remain consistent.
W1 & W3
We thank the reviewer for bringing up this potential possibility. From our evaluation, we see that when CARP is trained with the clinical measurement loss, it performs better along measurement error, real-to-synthetic (TRTS) classification, and FID (Table 2). Stronger results along classification and sample diversity (FID) provide evidence that CARP's objectives result in more clinically consistent features, as opposed to only improving on test measurements.
We also conducted a short cardiologist study with two board-certified cardiologists at a large academic hospital. Each cardiologist was shown generated ECGs and asked to rank CARP, DiffuSETS and SE-Diff. To prevent selection bias, we randomly sampled ECGs using the same training seed as in Appendix Table 3. The reviewer can inspect the images here. For each report, the cardiologists saw the three generated ECGs anonymized and in shuffled order, and ranked them from most to least clinically correct. We showed n = 13 ECGs across labels in RQ1 [1]. Ties were allowed. Neither cardiologist is an author or aware of this paper’s submission.
At the time of writing, one cardiologist has completed the study, with one pending. We provide the preliminary results below for cardiologist #1, and will update them once cardiologist #2 has completed the study.
These preliminary results show that cardiologists agree with the quantitative evaluation of CARP, showing its broad outperformance (mean rank 1.27 v/s 2.35 for DiffuSETS and 2.38 for SE-Diff).
[1] The labels were: atrial fibrillation, left bundle branch block, right bundle branch block, left axis deviation with sinus bradycardia, prolonged PR interval (first-degree AV block), sinus tachycardia with frequent PVCs, LVH with secondary repolarization abnormality, old inferior infarct, anterolateral ischemia, WPW pattern with sinus tachycardia, right ventricular hypertrophy, extensive ST elevation suggesting pericarditis, and dextrocardia.