- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In ECG2Act, the authors propose a benchmark based on MIMIC-IV-ECG to assess the value of ECG data to inform clinical care decisions therefore extending beyond usual diagnostic prediction settings. The authors use an auxiliary ECG feature loss and report and slight improvement in predictive accuracy through it.
Critical strengths
- The idea of extending clinical analyses beyond diagnostic/abnormality recognition to care processes is meaningful (but care processes will also depend on clinical judgement, local guidelines etc).
Critical weaknesses
Main
- Main concern I: The study represents a purely synthetic use-case; care decisions won't be made based on the ECG alone but will incorporate clinical context.
- Main concern II: "Together, these results support extending ECG analysis beyond abnormality recognition to jointly assess both diagnostic and care-process concerns." The study raises a question and could develop into an interesting direction (if extended by clinical context), but does not seem aligned with the topics of ICLR, i.e., provides little insights that extend beyond the use-case itself.
- training encoders from scratch instead of using pretrained encoders severely limits the applicability of the study in the age of FMs.
- "The diagnostic–care gap persists across concerns and ages." The gap between cardiac diagnostic conditions and care decisions is a result of the setup of the study: The ECG is a very informative modality for cardiac diagnostics, the restriction to ECG only leaves out relevant clinical information that would have been informative for care decisions.
- The authors claim that the auxiliary losses lead to physiological grounding. There is no direct way for jointly predicted to influence the main prediction task (apart from a reorganization of the joint feature space). There is no evidence for the added value of auxiliary tasks beyond macro AUROC performance increases.
- The discussion section is minimal and does not discuss a single limitation of the work.
- No source code is provided, which severly limits the reproducibility of benchmarking studies like this.
Minor
- Introduction: "Rates, intervals, and recognizable waveform patterns inform ECG interpretation (Kligfield et al., 2007; Mason et al., 2007), yet concern labels alone do not explicitly supervise these properties." but we do not want to connect them explicitly, rather learn it (and also features beyond these)
- Presentation of ECG FMs is very selective and does not align with the state of the field. Consider rather citing reviews or recent benchmarking studies instead of selecting specific models.
- Action query is very close to attention probes in V-JEPA (https://arxiv.org/abs/2404.08471) and should also be referenced as such.
- There is no evidence for the added value of auxiliary tasks beyond macro AUROC performance increases, which is not
Summary
- The study is a potentially interesting direction for a biomedical journal (when extending towards the inclusion of clinical context).
- Basing the benchmark on ECG only is from my perspective results in an ill-defined prediction task.
- The study provides little insights that extend beyond the presented use-case and seems poorly aligned with ICLR.
Questions for authors
- Why should the ECG be relevant as sole observable to inform care process decisions?
- While this is a potentially study for a biomedical journal, what is its insight for ICLR/machine learning community?
- What is the added value of MEETI instead of MIMIC-IV-ECG itself if only samples are retained?