acceptodds
Under review as a conference paper at ICLR 2027

Actionwave: A Systematic Benchmark for Waveform-Centric Clinical Prediction Across Care Scenarios

Abstract

Continuous bedside waveforms record rapid physiological changes that can inform clinical decisions. Existing benchmarks examine selected prediction conditions, but it remains unclear how model advantages depend on signal duration, warning time, and available inputs across care settings. We introduce Actionwave, a benchmark for waveform-centric clinical prediction across intensive-care, operating-room, and emergency-department settings. Actionwave uses a common decision-time prediction interface and groups clinically grounded tasks by what is inferred and when. It varies conditions relevant to clinical use, including the warning required before an intervention. We compare diverse architectures and pretrained biosignal encoders under specified model-size and compute budgets. The protocol requires patient-level splits, strict information cutoffs, and comparisons that isolate individual modeling factors. Across 20 tasks and 1.4M task-labelled windows from source datasets, the experimental grids account for over 87,000 model fits. These comparisons identify task-specific choices and show why matched cohorts, unimodal controls, and explicit timing are necessary when interpreting waveform model performance.

Primary area:
datasets and benchmarks
Keywords:
Time-series Foundation Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tbmaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

"Actionwave" introduces a benchmark for clinical prediction problems involving waveform data covering three critical care scenarios, operating room, emergency department and intensive care unit. The authors investigate different prediction models varying tokenizer and encoder and task-related hyperparameters such as observation length, forecasting horizon etc. The authors identify specific models that perform best on specific tasks and datasets.

Critical strengths
  • Broad coverage of three critical care scenarios with scenario-specific clinically relevant prediction tasks.
  • Focus on inclusion of waveform data is very desirable, since most benchmarks and prediction models focus on EHR only.
Critical weaknesses

Major

  • Clinical relevance: Waveform only does not really reflect the clinical reality of clinical decision making and therefore represents a totally synthetic application scenario.
  • Organization: The paper contains roughly 100 pages of appendix, however no clear guide through it. The main text does not stand by itself but crucially relies on the appendix, which is too extensive to read it in full. And even then, essential technical details are not provided in the appendix. For example, I was unable to identify which EHR features were used where. The entire paper is written in a staccatto style that is hard to parse. As one example, consider the beginning of Section 4. Tokenizer and encoder are never introduced formally and used models are only referred to via architecture without hyperparameter specification. Similarly, the beginning of Section 6, is very hard to parse and misses essentially all details on the investigated approaches, which is impossible to parse without excessive use of the appendix. Similarly, I was not able to find details of the transfer to MIMIC IV, which does not possess continuous waveform data as the other considered datasets, but just short 12-lead ECG snapshots. Also, the fusion strategies are not discussed in depth.
  • MIMIC-III ECG data: MIMIC-III ECG data is subject to serious limitations due to a unrecoverable loss of precision while saving the waveforms (see Physionet's readme page). This typically invalidates the use of MIMIC-III ECG data for deep learning approaches. As far as I could see, such issues are discussed are neither discussed in the manuscript nor in the appendix.
  • Reproducibility: In spite of the excessive appendix, the study is clearly not reproducible given the information provided in the main text and the appendix. No code is provided, which is a central weakness for a paper presenting a new benchmark.
  • Observation length etc: It is ill-defined to compare across models with different observation length, forecasting horizon etc as it is done in Section 5. Changing for example the observation length modifies the difficulty of the prediction task and performance level with different observation length are not directly comparable, which from my perspective put serious doubts on Section 5.
  • Generalizable conclusions: I have a hard time drawing any generalizable conclusions from this study since every decision is very dataset and task-specific. This might actually be correct and reflect the clinical reality, however, leaves the practioner approaching a different task on a different (acute care) dataset without any guidance. Rather than focussing only on the top-performing solution, there would also be a lot of value in identifying setups that show maybe not the best but decent and robust performance across various tasks and/or datasets. Without such an analysis, the reader leaves the paper without any insights.
  • Limitations: Limitations of the proposed approach are not discussed.

Minor

  • Handling of missing values: Handling of missing values in EHR data remains unclear. This is particularly important since EHR data is typically MNAR, which cannot be resolved by imputation choices.
  • No pretrained models: The benchmark builds entirely on models trained from scratch, completely ignoring pretrained foundation models that are in large number for example for ECG and PPG data.
  • Model complexity: The differences in model complexity for different encoder/tokenizer choices is not mentioned nor assessed/discussed.

Summary

  • The clinical relevance of the waveform-only remains doubtful. The paper is hard to parse with a main text that does not stand by itself. The reproducibility is severely limited by the lack of source code and missing details on datasets, models etc. I have a hard time drawing generalizable conclusions from the results.
Questions for authors
  • Why does MC-MED not include PPG waveforms (according to text description), even though it is shown for MC-MED in the abstract figure?
  • What are generalizable insights that can be drawn from the study, maybe in the form of robust model configurations that perform well across a broad range of tasks?
Reviewer_mmskOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Actionwave introduces a clinical waveform benchmark covering 20 tasks and approximately 1.4 million task-labelled windows across MIMIC-III, VitalDB, and MC-MED. A shared decision-time interface specifies observation windows, information cutoffs, warning gaps, outcome intervals, and permitted inputs. The experiments compare representations, observation histories, warning gaps, sensor availability, missing-input handling, training-data budgets, and waveform–EHR fusion. The concurrent BIS probe is analyzed separately from forecasting. Additional studies examine MIMIC-IV adaptation and MoE+SSL. The main contribution is an evaluation framework and a broad empirical study of task-dependent modeling choices.

Critical strengths

1. The temporal interface addresses meaningful evaluation conditions. Section 3 explicitly separates observation history, prediction anchors, warning gaps, and outcome intervals, with patient-level partitioning and information cutoffs. The authors distinguish concurrent BIS estimation from forecasting and acknowledge that warning-gap changes can alter eligibility and outcome support. This provides a useful framework for studying clinically relevant prediction conditions.

2. Unimodal controls and qualified reporting improve interpretability. The fusion studies retain waveform-only and EHR-only references. Appendix F.3 additionally compares fusion with a head-wise stronger-unimodal envelope. Its completed test analysis reports inconclusive AUROC and AUPRC gains over that envelope, rather than presenting all fusion improvements as definitive. Together with the reporting of near-chance endpoints and unfavorable method comparisons, this helps identify the limits of waveform information.

Critical weaknesses

1. The contribution to the ICLR community is not yet sufficiently convincing. A benchmark can be valuable without introducing a new algorithm. My concern is whether this particular combination of tasks and comparisons establishes sufficiently clear new knowledge or a sufficiently validated reusable resource. Table 1 emphasizes coverage of experimental axes, while many headline conclusions concern task-dependent rankings, variable benefits from longer histories, and the importance of unimodal controls. These observations are useful, but the manuscript does not yet clearly distinguish its central advance from the cumulative value of performing more comparisons.

The different care settings also involve different datasets, endpoints, sensor availability, and protocols. This does not invalidate within-task comparisons, but it limits what can be inferred from differences across settings about broadly transferable modeling principles. The MoE+SSL example on p. 9 reports gains, yet lacks sufficient methodological detail and component controls to establish a clear additional modeling insight. My weak rejection recommendation therefore concerns the demonstrated contribution, not the clinical subject matter or the absence of a new architecture.

2. Reconstruction and comparison gaps weaken the benchmark’s central claims. From the supplied manuscript alone, several consequential task rules remain underspecified. Examples include the components of the MIMIC-III major rescue event, numerical criteria for rapid-fluid administration and MAP improvement, and MC-MED decompensation thresholds and durations (Appendices B.3, B.5, and B.7). Task-specific anchor schedules and forecast-gap event handling are not fully enumerated. Model and tokenizer specifications, including patch lengths, strides, and token counts, are also insufficiently detailed for independent reconstruction. These are reporting limitations, not evidence of an implementation error.

The evidence supporting some recommendations is also weaker than their specificity suggests. Table 10 reports MC-MED macro AUROC of 0.568 for ResNet1D versus approximately 0.567 for Transformer and DLinear. Five-seed SDs do not establish patient-level uncertainty in these differences. Some VitalDB analyses do provide whole-subject bootstrap intervals, so this concern applies to the principal comparisons lacking corresponding paired analyses, rather than to the entire paper. Appendix A.2 also discloses that some main AUROC and companion predictive or cost metrics come from different campaigns or configurations. These differences are not numerical contradictions, but such metrics cannot automatically be combined into one model’s trade-off profile. The benchmark’s guidance would be stronger with exact task/configuration specifications and matched patient-level analyses of its principal recommendations.

Questions for authors
  1. What is the central new finding or evaluation capability beyond broader task coverage and additional comparisons? Which result most clearly establishes the benchmark’s contribution relative to the complementary resources in Table 1?

  2. Can the authors identify or provide an inspectable, versioned specification of the primary tasks and model/tokenizer configurations, including the missing operational rules, and show how it reconstructs the Table 6 cohort counts?

  3. Which principal model-selection recommendations remain supported by patient-level paired intervals, and which should instead be presented as unresolved shortlists? Can the supporting AUROC, AUPRC, calibration, and cost results be mapped to the same configurations?

  4. For the main Figure 4 warning-gap results, can the authors provide gap-specific patient and positive-patient counts, window counts, prevalence, and exclusion reasons? How sensitive are the longer-gap rebounds to changing eligibility, and which conclusions remain supported on commonly ascertainable anchors?

1 comment

I haven’t seen a response from the authors to the questions I raised in my review. My concerns remain, so I’ll keep my original rating.

Reviewer_kjysOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Actionwave is a benchmark for clinical waveform prediction across intensive care, operating rooms, and emergency departments. It includes 20 tasks and about 1.4 million task-labelled windows. The experiments study representation and encoder choices, observation length and warning time, and information availability.

The main contribution is a shared task framework and systematic evaluation. The results show that the benefits of modeling choices often depend on the task and prediction conditions.

Critical strengths
  1. Useful evaluation framework. Actionwave systematically varies history length, warning time, and inputs across three care settings, helping assess when model advantages hold.

  2. Practical empirical findings. Task-level comparisons show when longer histories, complex representations, and fusion help or fail, providing useful guidance for model selection.

Critical weaknesses
  1. The scope of controlled comparisons needs clarification. Tokenizer–encoder adaptation details are not sufficiently clear in the main text, limiting interpretation of the reported rankings. Although Table 1 defines “–” as an unreported comparison, its aggregate checkmarks may suggest broader coverage than demonstrated within individual settings. Clearer interfaces and coverage descriptions would help establish the benchmark’s contribution.

  2. Some practical conclusions need stronger support. Warning-gap experiments legitimately change outcome windows, but changes in cohort eligibility also need to be quantified before interpreting performance trends. Fusion and performance-preservation claims require similar care: MIMIC-III fusion remains below EHR-only performance, while small subset gains need uncertainty estimates. These issues limit confidence in the recommendations unless supporting analyses are provided or clearly referenced.

Questions for authors
  1. Tokenizer–encoder separation and protocol scope: How is this separation defined across architectures, given that CNN input layers already perform local feature extraction and FITS includes frequency processing internally? Please clarify token shapes and meanings, which native components are retained or replaced, and how budgets are controlled. More broadly, does the protocol constrain only available information and prediction targets, or also representations and training objectives? For Mamba/RNNs, resetting states removes persistent history, while retaining them may exceed the allowed window. Are event/censoring times available for survival methods, and can irregular-time methods use original timestamps without mandatory resampling or patch tokenization?

  2. Warning-time comparisons: How do eligible anchors, positive-event counts, and exclusions change across warning gaps? Do existing analyses show whether key trends persist on common eligible anchors, while allowing labels to change with the outcome window?

  3. Practical recommendations: What uncertainty estimates support performance-preservation claims and small subset gains? How should the MIMIC-III fusion recommendation be interpreted when EHR-only performs better overall? Do data-budget comparisons account for selection costs and sampling variability?

  4. Coverage and contribution: Which task–dataset combinations support each Actionwave checkmark in Table 1? Which additions require new task or cohort construction, and which are additional experiments on existing resources?

Reviewer_4AZ2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Actionwave, a benchmark for clinical prediction from bedside waveforms in three care settings: the ICU (MIMIC-III), the operating room (VitalDB), and the emergency department (MC-MED). Its 20 tasks share one prediction framework with explicit observation windows, warning times, and information cutoffs. The authors compare architectures, tokenizers, observation lengths, sensor configurations, missing-data methods, and waveform–EHR fusion approaches. The benchmark contains about 1.4 million task-labelled windows, and the experiments amount to more than 87,000 model fits, counting hyperparameter searches and repeated training. The authors conclude that the best modeling choice depends on the clinical task, the available information, and the prediction conditions.

Critical strengths

S1. The benchmark spans 20 clinically relevant prediction tasks across ICU, operating-room, and emergency-department settings. The shared prediction framework, patient-level splits, explicit information cutoffs, and detailed task definitions support reproducible evaluation. I also appreciate the controlled comparisons and the EHR-only baselines, which show when waveform information adds predictive value. For example, the best MIMIC-III fusion model (0.622 macro AUROC) stays below the EHR-only baseline (0.632). The benchmark could be a useful resource for future work on waveform models.

Critical weaknesses

W1. The benchmark gives limited statistical evidence for the model rankings behind its recommendations. Architecture and tokenizer comparisons report seed variability, but no paired test-set uncertainty for the differences between models. On MC-MED, the same nominal Transformer–Conv1D configuration gives different results in two separately selected experiments, and the gap is comparable to the differences between architectures (Q1). Several tasks also stay near chance even with their best reported configurations. Reliable model comparisons are a central contribution of a benchmark, so it is unclear which task-specific recommendations rest on meaningful performance differences.

W2. The warning-time and efficacy–efficiency–safety comparisons do not fully support the conclusions drawn from them. The warning-time experiments keep patient splits fixed but let eligible episodes and outcome prevalence change across gaps. This makes it hard to read the non-monotonic curves as effects of warning time alone (Q3). In addition, several AUPRC, calibration, and cost results come from earlier experimental campaigns than the corresponding AUROC results (Appendix A.2). As a result, some reported trade-offs do not compare the same fitted configurations (Q4). These issues weaken the benchmark's claim of controlled comparisons across prediction conditions.

Questions for authors

Q1. On MC-MED, the architecture panel uses Conv1D as the common tokenizer (Appendix B.7). Its Transformer entry and the tokenizer panel's Conv1D entry are therefore the same nominal configuration, selected separately. The two differ by 0.007 in macro AUROC (0.567 vs. 0.574) and by 0.021 on hyperkalemia (0.555 vs. 0.576; Table 10). By comparison, eight of the nine MC-MED encoders lie within 0.008 of each other (0.560–0.568). The test sets are also small, with only 54, 70, and 90 positive cases for anemia, lactate, and hyperkalemia, respectively (Table 6). Could the authors report paired patient-bootstrap intervals for the architecture and tokenizer comparisons? Which recommendations still hold once this uncertainty is taken into account?

Q2. Several MC-MED configurations score below 0.5. SeFT, for example, reaches 0.469 on decompensation and 0.428 on hyperkalemia, while Conv1D reaches 0.647 on decompensation with the same data. How do the authors distinguish an unsuitable tokenizer from a configuration that did not train well under the parameter budget or the implementation adaptations? How much weight can model recommendations carry for tasks such as lactate non-clearance, where AUROC ranges from 0.453 to 0.509 across configurations?

Q3. Could the authors evaluate the warning-time comparisons on anchors that are eligible at every gap, and report test-set outcome prevalence for each gap? Do the non-monotonic patterns in Figure 4 persist under this analysis, in particular the VitalDB coupled-response increase from 0.603 at 45 minutes to 0.750 at 75 minutes? Appendix F.2 reports a separate fixed-recipe test in which macro AUROC declines from 0.659 to 0.558. What differences in model configuration or evaluation protocol explain the two trends, and how should they affect the warning-time recommendations?

Q4. Appendix A.2 states that some MC-MED AUPRC and reliability results come from an earlier experimental campaign rather than the updated AUROC configurations, and that the VitalDB cost measurements come from earlier grids. For which reported configurations were AUROC, AUPRC, calibration, and cost measured on the same fitted models? Which performance–efficiency–safety conclusions rest on these matched measurements?

Q5. The abstract mentions comparisons with pretrained biosignal encoders, but the main representation experiments use adapted architectures trained under a common parameter budget. Which pretrained biosignal encoders are evaluated, and where are their results reported?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.