- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EchoTab, a representation learning method for Echocardiography that leverages key clinical measurements. EchoTab consists of several components: an echo encoder that transforms echocardiography clips into echo tokens, learnable measurement queries, a tabular decoder that connects the measurement queries with their corresponding relevant views through a cross-attention mechanism, and a prediction head. Experiments on MIMIC-IV-Echo v1.0 and an internal dataset show that EchoTab outperforms the baselines.
Critical strengths
- The idea of using Echo-related measurements as intermediate representations is very well motivated, as it reflects the clinical workflow where diagnosis is conducted by considering multiple information, including measurements.
- EchoTab enables diagnosis-specific selection of relevant measurements, which could be helpful in clinical settings where Echocardiography view scans are incomplete.
- Experiments on both public and private data demonstrate EchoTab’s superiority over the baselines, and the ablation study shows the contribution of each of EchoTab's components and design considerations.
- The cross-attention map analysis shows that EchoTab is able to mimic cardiologists on the measurement-view relevancies to some extent, which demonstrates that measurement-grounded representations align with well-established clinical rules.
- The paper is well written and organized, with some caveats in atuhors’ claims (see weaknesses).
Critical weaknesses
- Abstract > "Each measurement token selectively attends to the views relevant ..." > made me wonder the first time reading the paper. Consider adding something like "through cross-attention".
- In the introduction, the authors say "A natural alternative is to use measurements directly as the target modality for contrastive learning" right after mentioning EchoPrime. However, I believe this claim is not well supported, as the authors utilize EchoPrime as EchoTab's view/clip encoder. Also there was no experiment comparison between the contrastive learning as in EchoPrime and EchoTab's measurement-grounded representation learning. I strongly suggest revising and lowering this claim, as I believe the experiments are inadequate to support such a claim.
- The authors say "EchoTAB consistently outperforms echo foundation models". However this could be that because they were not pre-trained on the MIMIC-IV-Echo v1.0 data distribution, while EchoTab is trained on that distribution. Thus, I believe the authors should revise and lower down the claim. Also, it is worth comparing the Echo foundation models with EchoTab trained on MIMIC and tested on the private set, or vice versa.
- No statistical analysis of the experimental results is provided in the main text.
Questions for authors
- What is the reasoning behind sampling randomly clip during training while deterministically selecting the first clip during inference, as mentioned in Sec. 2.1?
- Did the authors try to compare using Eq. 5 with simple mean-std normalization of measurements?
- Table 2: Did the authors run any experiment where the Echo encoder is randomly initialized and trained end-to-end using the proposed measurement-grounded representation learning?
- How does EchoTab’s performance on measurement prediction compare to PanEcho?
10 comments
(W2+W3) Comparison of EchoTab and echo foundation models
We agree that the phrase "A natural alternative..." could be misinterpreted as an unsupported claim that tabular-based learning is strictly superior to EchoPrime's text-based contrastive learning. Before introducing our own approach, we simply intended to explain how EchoPrime could be adapted to use measurements instead of text reports as supervision during contrastive learning, following established image-tabular contrastive methods.
We also agree that the phrasing "EchoTab consistently outperforms echo foundation models" could be misinterpreted as claiming our pre-training method is strictly superior to the ones used to train the foundation models without accounting for domain shift. As indicated in Table 1, the foundation models were included only as reference, not baselines, as they were not re-trained on our datasets. Because we initialized EchoTab and the other baselines with one such foundation model (EchoPrime), we included them to demonstrate that through echo-tabular pre-training, EchoTab and the other baselines improve upon the original EchoPrime weights.
To prevent any misunderstanding, we have softened these claims and have made the following changes in the manuscript:
-
Introduction (Section 1): We revised "A natural alternative is to use measurements directly as the target modality for contrastive learning" to: "Following image-tabular contrastive methods (Hager et al., 2023), EchoPrime's echo-text alignment could be adapted to echo-tabular alignment by directly using the tabular measurements instead of text reports as the target modality."
-
Table 1 Caption: We revised "EchoTab consistently outperforms echo foundation models, as well as study- and view-level contrastive and MLP baselines…" to "EchoTab consistently outperforms study- and view-level contrastive and MLP baselines…"
-
Diagnosis Classification (Section 3.3): We revised "EchoTab consistently improves diagnosis classification across both datasets and achieves the highest average AUC among echo foundation models, contrastive baselines, and the MLP baseline." to "EchoTab achieves the highest average AUC across diagnoses among the contrastive and MLP baselines on both datasets."
To rule out the possibility that EchoTab’s performance gains over echo foundation models are only due to dataset-specific training on the target distributions instead of the actual measurement-grounded learning itself, we further trained (FT) the echo foundation models EchoPrime and PanEcho on our datasets according to their original pre-training objectives. Since EchoPrime’s echo-text contrastive objective requires text reports for training, we generated synthetic text reports from the measurement data without any diagnostic labels. This guarantees that EchoPrime (FT) receives the same measurement-only supervision as EchoTab. We then evaluated these models in both in-dataset and cross-dataset settings on diagnosis classification. We have added this experiment to App B.3 (now App B.2).
The reported metric is average AUC across diagnoses in each dataset. A→B denotes that the model was trained on measurement data from dataset A and evaluated on dataset B.
| Model | Pre-training Objective | In-Dataset: MIMIC | In-Dataset: Internal | Cross-Dataset: Internal→MIMIC | Cross-Dataset: MIMIC→Internal |
|---|---|---|---|---|---|
| EchoPrime | – | 0.659 | 0.730 | – | – |
| PanEcho | – | 0.702 | 0.702 | – | – |
| EchoPrime (FT) | Echo-text contrastive | 0.689 | 0.707 | 0.652 | 0.729 |
| PanEcho (FT) | Tabular reconstruction | 0.713 | 0.734 | 0.663 | 0.729 |
| EchoTab (Ours) | Tabular reconstruction | 0.748 | 0.778 | 0.703 | 0.766 |
As shown above, further training EchoPrime with its original echo-text contrastive objective yields mixed results. EchoPrime (FT) improves on MIMIC (0.659→0.689) but degrades on the internal dataset (0.730→0.707). Meanwhile, PanEcho (FT) improves on both MIMIC (0.702→0.713) and on the internal dataset (0.702→0.734), consistently benefiting from training on the target data distribution.
Meanwhile, in both in-dataset and cross-dataset evaluations, EchoTab outperforms EchoPrime (FT) and PanEcho (FT). This confirms that EchoTab's performance gains come from our measurement-grounded representation learning framework, rather than mere exposure to the target data distribution.
(W4) Statistical analysis of the experimental results in Table 1
We added a paired bootstrap analysis to compare average AUCs between EchoTab and all methods in Table 1. Specifically, we resample the test set with replacement 1,000 times, resulting in 1,000 sets of test studies that are shared across methods and diagnoses. For each method, we compute the improvement in average AUC of EchoTab over that method and report its 95% percentile CI and two-sided bootstrap p-value, with p-values corrected using the Holm-Bonferroni procedure. Baselines significantly outperformed by EchoTab are marked with * in Table 1, with CIs and p-values reported in the Appendix B.4 (previously Appendix B.4, now Appendix B.3).
On MIMIC, the improvement of EchoTab in average AUC is statistically significant against all compared methods. On the internal dataset, it is statistically significant against all methods except the MLP baseline (for which the 95% confidence interval narrowly includes zero).
(W1) Clarification of "cross-attention" in the abstract
We have now added "cross-attention" to this sentence in the Abstract to "Each measurement token selectively attends to the views relevant to its estimation through cross-attention, and is trained through tabular reconstruction to predict the corresponding measurement value."