- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
"Actionwave" introduces a benchmark for clinical prediction problems involving waveform data covering three critical care scenarios, operating room, emergency department and intensive care unit. The authors investigate different prediction models varying tokenizer and encoder and task-related hyperparameters such as observation length, forecasting horizon etc. The authors identify specific models that perform best on specific tasks and datasets.
Critical strengths
- Broad coverage of three critical care scenarios with scenario-specific clinically relevant prediction tasks.
- Focus on inclusion of waveform data is very desirable, since most benchmarks and prediction models focus on EHR only.
Critical weaknesses
Major
- Clinical relevance: Waveform only does not really reflect the clinical reality of clinical decision making and therefore represents a totally synthetic application scenario.
- Organization: The paper contains roughly 100 pages of appendix, however no clear guide through it. The main text does not stand by itself but crucially relies on the appendix, which is too extensive to read it in full. And even then, essential technical details are not provided in the appendix. For example, I was unable to identify which EHR features were used where. The entire paper is written in a staccatto style that is hard to parse. As one example, consider the beginning of Section 4. Tokenizer and encoder are never introduced formally and used models are only referred to via architecture without hyperparameter specification. Similarly, the beginning of Section 6, is very hard to parse and misses essentially all details on the investigated approaches, which is impossible to parse without excessive use of the appendix. Similarly, I was not able to find details of the transfer to MIMIC IV, which does not possess continuous waveform data as the other considered datasets, but just short 12-lead ECG snapshots. Also, the fusion strategies are not discussed in depth.
- MIMIC-III ECG data: MIMIC-III ECG data is subject to serious limitations due to a unrecoverable loss of precision while saving the waveforms (see Physionet's readme page). This typically invalidates the use of MIMIC-III ECG data for deep learning approaches. As far as I could see, such issues are discussed are neither discussed in the manuscript nor in the appendix.
- Reproducibility: In spite of the excessive appendix, the study is clearly not reproducible given the information provided in the main text and the appendix. No code is provided, which is a central weakness for a paper presenting a new benchmark.
- Observation length etc: It is ill-defined to compare across models with different observation length, forecasting horizon etc as it is done in Section 5. Changing for example the observation length modifies the difficulty of the prediction task and performance level with different observation length are not directly comparable, which from my perspective put serious doubts on Section 5.
- Generalizable conclusions: I have a hard time drawing any generalizable conclusions from this study since every decision is very dataset and task-specific. This might actually be correct and reflect the clinical reality, however, leaves the practioner approaching a different task on a different (acute care) dataset without any guidance. Rather than focussing only on the top-performing solution, there would also be a lot of value in identifying setups that show maybe not the best but decent and robust performance across various tasks and/or datasets. Without such an analysis, the reader leaves the paper without any insights.
- Limitations: Limitations of the proposed approach are not discussed.
Minor
- Handling of missing values: Handling of missing values in EHR data remains unclear. This is particularly important since EHR data is typically MNAR, which cannot be resolved by imputation choices.
- No pretrained models: The benchmark builds entirely on models trained from scratch, completely ignoring pretrained foundation models that are in large number for example for ECG and PPG data.
- Model complexity: The differences in model complexity for different encoder/tokenizer choices is not mentioned nor assessed/discussed.
Summary
- The clinical relevance of the waveform-only remains doubtful. The paper is hard to parse with a main text that does not stand by itself. The reproducibility is severely limited by the lack of source code and missing details on datasets, models etc. I have a hard time drawing generalizable conclusions from the results.
Questions for authors
- Why does MC-MED not include PPG waveforms (according to text description), even though it is shown for MC-MED in the abstract figure?
- What are generalizable insights that can be drawn from the study, maybe in the form of robust model configurations that perform well across a broad range of tasks?
I haven’t seen a response from the authors to the questions I raised in my review. My concerns remain, so I’ll keep my original rating.