- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a pipeline for predicting the rank of protocols in emergency medical services (EMS). The task can be summarized as a classification problem given the (part) of the actual medical records.
The proposed method first divides the record to six feature stages, where at each stage, new information (e.g., organ) is added to predict the protocol label. And six independent MLP models are trained to predict the label. Meanwhile, individual feature stage prediction is trained separately. These predictions are then pooled with a learned network.
The proposed method is evaluated on a well-sized dataset (NEMSIS) with reproducible code. Experiments compare a range of baseline methods including single NN predictors, seed ensembles, pooled predictors, alternative architectures, gradient-boosted baselines, and local LLMs. Results show increasing accuracy as evidence accumulates, with marginal gains from the progressive pooling module over seed ensembling.
Critical strengths
The paper studies a practically problem, with a dedicated formulation on the EMS task by introducing a progressively visible data input for enhancing the robustness of the prediction.
I also found the paper has an extensive experimental study with a proper set of baselines and ablations, which supports the acclaimed numerical gain.
Critical weaknesses
-
The overall novelty and contribution is somehow overstated. The authors listed several contributions with naming. However, the actual contribution is limited. For example, the PCFL (progressive clinical feature learning) is presented as a core contribution. But it appears to be a definition of manual separation of information into stages. The underlying task itself is essentially a tabular classification problem, while the formulation of sequence modeling is mainly a manual specification. It is then unclear that whether the claimed temporal/sequence-modeling is necessary for presenting the work. Overall, the paper does not justify the modeling strategy ieself for the underlying problem. STEP module is built from standard components including ensembling, logarithmic pooling, and input weighting. The paper itself explicitly describes STEP as assembling established parts. In terms of performance, the ablations show that a static log-linear pool already recovers most of the improvement, while the adaptive gating mechanism contributes only a small additional gain.
-
The readability of the paper is limited due to the presentation of the main illustrative figures and the organization of technical details. The figures do not provide a sufficiently clear high-level view of the proposed framework, which makes the overall method difficult to follow on a first read. Implementation and training details are scattered throughout the main text starting from Section 3.
Questions for authors
The main question affecting my assessment is whether the proposed framework is justified by the problem itself in the first place: Could the authors clarify what intrinsic difficulty of this tabular (structured) classification task necessitates the proposed stage-wise modeling and posterior pooling with multiple MLP models, and whether the resulting gains are sufficient to justify this added complexity over other existing learning-based tabular approaches, especially given the marginal performance gain?
Another question may be related to the application side. The current evaluation is based on retrospective features and offline-trained model. Could the authors discuss whether this formulation would translate into real-time EMS decision support: when the model would make predictions during the event, e.g., an encounter? And how to validate it when an expert is on-site?
Thank you for raising this central question. We agree that the contribution must be justified by the prediction setting and controlled evidence. The revision clarifies why progressive evaluation is beneficial, when posterior pooling helps, and whether its benefits warrant additional complexity.
Weakness 1: The application motivates progressive prediction; it does not require a particular sequence architecture. EMTs, especially new recruits, operate in high risk time-sensitive situations which affect the patient outcome based on their decision for selecting the right protocol. EMS decision making is refined through time: they constantly make decisions on the protocol selection starting from dispatch, to reviewing protocol descriptions in the ambulance to the scene as confirmed by our EMS domain expert. Therefore a temporal recommendation system, PCFL, helps them to decide quicker with higher reliability. Although our current progressive prediction model operates without EMT input such as clinical guidance or incident-specific inquiries, we plan to integrate EMTs in future iterations. This will allow us to test our hypothesis that progressive prediction reduces EMT cognitive load, improves protocol accuracy, and ultimately leads to better patient outcomes. PCFL therefore defines an architecture-independent evaluation protocol: rank the same 82 protocols on matched encounters while restricting each prediction to the evidence available at that stage. Its contribution is this formulation and benchmark. Because NEMSIS does not timestamp every observation, we explicitly describe the six stages as workflow-informed feature revelation, supported by timestamp-restricted vital-sign and alternative-order analyses (Sec. 5.2; Tables 24–25).
Question 1: The task’s difficulty also arises from NEMSIS itself. Cognitive workload, time pressure, and administrative documentation incentives contribute to general protocol labels (rather than specific encounter protocols), mislabeled and unlabeled encounters, and a noisy, long-tailed distribution. The majority class is a general protocol label which in itself captures the distributions of more specific protocols among the 82 classes. Previous studies used at most 46 handpicked classes because of this classification difficulty. Traditional and modern machine learning methods consequently produce low performance (<20% top-1 accuracy). Stagewise modeling reflects progressive information capture and decision making, enabling more informed decisions before arrival; its workflow benefits remain a hypothesis for a future user study. It also exploits our observation that earlier stages correctly predict some encounters that later stages misclassify (App. F, Fig. 16). Our empirical comparisons clarify the rationale for the framework’s components. First, LSTM, Transformer, and cascaded baselines do not significantly outperform the per-stage MLP ensemble, trailing it by 0.1–1.7 top-1 percentage points at Stages 5–6. This supports simple tabular predictors within the progressive formulation (Table 6). Tuned XGBoost improves over the single MLP but remains 1.22 top-1 points below STEP at Stage 6. Pooling XGBoost predictions adds 0.87/0.68 top-1/top-3 points, showing that the benefit extends beyond MLPs (Table 22).
Second, transition analysis identifies the difficulty pooling targets: 2.7–5.8% of development encounters switch from correct to incorrect predictions as features are added. Earlier-stage or block-specific predictors correctly rank some cases missed by the current-stage model (App. F), supplying complementary estimates rather than additional patient information. Third, equal-weight pooling underperforms the three-seed ensemble, whereas learned log-linear pooling improves accuracy, supporting selective weighting of heterogeneous evidence sources (Table 20). These findings motivate retaining a joint predictor for cross-feature interactions and supplementing it with learned pooling of restricted-view predictions. STEP yields Bonferroni-significant Stage 6 gains of 0.96/1.27 top-1/top-3 points over the three-seed ensemble. However, static pooling captures most of this gain, and a twelve-seed joint ensemble achieves comparable accuracy with fewer networks. STEP’s remaining advantage over that larger ensemble is lower temperature-scaled NLL (−0.036 nats; 95% CI [−0.047, −0.025]; Table 21). Thus, the evidence supports progressive evaluation and a measurable pooling benefit while identifying simpler alternatives when top-k accuracy alone is the priority.
Weakness 2: Regarding readability, we acknowledge the presentation concerns and have substantially revised references to specific figures and tables.
Question 2: Regarding the final concern, we plan to deploy the model as real-time EMS decision support through smartphones or smart glasses, generating predictions as information arrives through speech recognition. Expert validation with EMTs will be explored in a future user study.