- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Sensor-Language-Action (SLA) modeling, which takes physiological signals and patient context, predicts the clinician’s action, and generates a text explanation.
The authors build a benchmark from six datasets across three settings: clinical (MC-MED, MIMIC-III), operating room (MOVER, VitalDB) and CGM (MetaboNet, PEDAP). MIMIC-IV is held out for external evaluation. Actions are organized into three levels: necessity, category and fine-grained label. Caption targets come from a rule-based template pipeline with four facets: observational, relational, global and action evidence.
The model, OpenSLA, is a Qwen-3.5-2B backbone with LoRA. It adds a hierarchical sensor compressor (HCSM), structured action heads and a gated caption decoder, and is trained jointly on action and caption losses. Experiments cover action prediction, caption-based state estimation, evidence grounding (BERTScore), zero-shot prediction of an unseen drug, future-glucose probing and MIMIC-IV transfer.
Critical strengths
- Benefit from caption supervision. Removing caption training drops VitalDB necessity AUROC from 74.9 to 54.1 under the same setup (Table 15). This shows that language supervision helps sensor-action modeling.
- Breadth of the experiments. The benchmark spans three settings, six cohorts and a unified action hierarchy. If the benchmark and its curation process is released, it would be very useful for the community.
Critical weaknesses
- Claims exceed the results. Section 5 (p. 6) states: "As shown in Table 2, Table 3, and Table 4, OpenSLA consistently outperforms all baselines across all action granularities and datasets." Several cells contradict this:
- In Table 2 and Table 3, GPT-5.6-Luna beats OpenSLA on fine-label BAcc for MIMIC-III and MOVER.
- In Table 4, ConvTrans beats OpenSLA on PEDAP category.
- Caption-based evaluations are biased toward the trained models. Captions are deterministic templates computed from the input window.
- Table 5 (MAE) and Table 6 (BERTScore) therefore measure how well a model imitates this specific extractor and its template phrasing.
- OpenSLA and the adapted baselines are trained on these templates, but the zero-shot LLMs are not.
- No variance reporting. There are no seeds, number of runs, standard deviations or confidence intervals reported in the experiment results.
- The MIMIC-IV results do not strongly support OpenSLA’s advantage in transfer (p. 9)
- MIMIC-IV comes from the same hospital as MIMIC-III and overlaps it in time, and de-duplication isn’t discussed.
- Necessity: comparing Table 2 (MIMIC-III) with Table 7, no method degrades much (OpenSLA −1.0/−0.3, OpenTSLM-SP −1.9), and SensorLM improves (+4.8).
- Category: OpenSLA itself degrades sharply, with AUROC 88.3 → 74.6 (B) and 88.1 → 74.9 (H), and BAcc 73.1 → 56.4 and 74.5 → 58.5.
- Main text is hard to follow.
- Distant references: Section 4 (p. 4) relies on Fig. 7 (p. 9) and Table 3 (p. 6) to motivate the design. Fig. 7 also doesn’t state its dataset.
- Under-specified method: the main text describes and only verbally. Their exact forms and the differences between OpenSLA-B and OpenSLA-H (e.g., B omits HCSM and the sensor-evidence decoder branch) appear only in Fig. 3 and Appendix C.1.
Questions for authors
- How were GPT-5.6-Luna and DeepSeek-V3.2 served for MIMIC and MC-MED data? Can the authors refer us to the code? Was this compliant with the PhysioNet DUA? (See: https://physionet.org/news/post/llm-responsible-use/)
- All OpenSLA variants receive both sensor signals and context text, and no ablation removes either input. Could the authors report sensor-only and context-only variants of OpenSLA? This would show whether the gains come from combining the modalities or mainly from the LLM reading the patient record.
7 comments
Response to Reviewer DhYj (2/4)
W2: Caption-based evaluations are biased toward models trained on the templates.
We would like to clarify our caption-based evaluation and provide additional analyses of template effects below.
First, the ground-truth physiological values are computed directly from raw sensor recordings, independently of caption templates. As defined in Appendix D.2, we use well-established algorithms [1, 2] to derive these values. Moreover, we extract numerical values from model-generated captions and evaluate them against signal-derived ground truth using MAE. Thus, Table 5 evaluates physiological state accuracy regardless of the template wording. This holds equally for the zero-shot LLMs, since only the stated numbers are scored.
Second, our action-explanation evaluation measures semantic fidelity by computing BERTScore [6] between generated and ground-truth action-evidence captions. BERTScore is a well-established metric that is widely adopted in recent time-series–language studies [3-5]. To further clarify our evaluation, we separately compute BERTScore for action-relevant content and shared template text, alongside the full-caption results:
| Method | MC-MED Full | MC-MED Template | MC-MED Action | MIMIC-III Full | MIMIC-III Template | MIMIC-III Action |
|---|---|---|---|---|---|---|
| OpenTSLM | 79.5 | 72.01 | 79.27 | 80.3 | 73.35 | 79.97 |
| SensorLM | 77.2 | 71.79 | 78.97 | 78.8 | 73.62 | 82.66 |
| OpenSLA-B | 81.5 | 71.66 | 80.44 | 78.6 | 72.98 | 80.47 |
| OpenSLA-H | 82.0 | 72.18 | 81.25 | 80.9 | 73.97 | 82.35 |
As shown above, action-relevant scores remain comparable to full-caption scores, while template scores are consistently lower and nearly identical across models. This indicates that template wording does not drive the evaluation differences.
Furthermore, we separately swap the template wording and action-related terms in reference captions, repeating each swap five times per sample. The resulting changes in BERTScore (Δ ± SD) are shown below:
| Method | MC-MED Template | MC-MED Content | MIMIC-III Template | MIMIC-III Content |
|---|---|---|---|---|
| GPT-5.6-Luna | −0.08 ± 0.56 | +1.45 ± 1.37 | −0.02 ± 0.61 | +1.09 ± 1.54 |
| DeepSeek-V3.2 | −0.03 ± 0.45 | +1.69 ± 1.31 | −0.02 ± 0.46 | +1.49 ± 1.22 |
| OpenTSLM | −0.17 ± 1.67 | −4.22 ± 2.26 | −0.09 ± 1.32 | −2.82 ± 2.67 |
| SensorLM | +0.03 ± 1.92 | −3.96 ± 2.22 | −0.03 ± 1.44 | −2.41 ± 2.56 |
| OpenSLA-B | −0.16 ± 1.72 | −4.95 ± 2.45 | −0.10 ± 1.47 | −3.65 ± 2.48 |
| OpenSLA-H | −0.16 ± 1.74 | −5.36 ± 2.47 | −0.14 ± 1.56 | −4.29 ± 2.58 |
As confirmed, swapping template wording has minimal impact, whereas swapping action evidence substantially lowers the scores of the models trained on our captions. This further confirms that action evidence drives the evaluation of these models. In contrast, the zero-shot LLMs' scores rise slightly, since their captions are generic clinical descriptions with little patient-specific evidence. In Table 5, the zero-shot LLMs also make larger errors on the stated values (e.g., ECG-II rate MAE 26.5-28.4 vs. 15.9-20.1 for OpenSLA). The lower BERTScore of the zero-shot LLMs thus reflects less specific and less accurate evidence.
Finally, we would like to clarify how our captions differ from "deterministic templates". While the values are deterministic, since they are computed from the input window, the wording is not. Our pipeline integrates multi-faceted semantics spanning channel-level observations, cross-channel relationships, global patient context, and action-relevant evidence. Furthermore, we randomly sample from 162 diverse templates (Appendix F.2) to ensure linguistic diversity.
We have clarified the caption-based evaluation in the revised paper (Section 5, L423–427; Appendix D.2) and added the template and action-evidence analyses (Tables 26–27, Appendix E.7, L1401–1441).
Response to Reviewer DhYj (3/4)
W3: The results lack seeds, standard deviations or confidence intervals.
Thank you for raising this point. We agree that reporting uncertainty is essential for judging the differences between methods, and we have added it to the revised manuscript as described below.
First, Tables 2-4 now report 95% confidence intervals for all main action results. The intervals are computed by bootstrap over test patients with 2,000 resamples (Appendix D.1). Bootstrap intervals are a standard way to report uncertainty in clinical prediction models [7]. Moreover, data sampling is often a larger source of variance than random seeds [8]. OpenSLA's advantage holds within these intervals. For example, on MIMIC-III necessity, OpenSLA-B reaches 75.4 ± 1.4 BAcc and the strongest baseline, OpenTSLM, 71.2 ± 1.7, with no overlap.
Second, our conclusions already hold across independent training runs. OpenSLA is trained separately in each of the three domains, and each run is the best method in most cells of its own table. A gain from a lucky seed would be unlikely to repeat across three independently trained domains.
Taken together, our results are reliable under both test-set variation and training randomness. We have added these intervals to the revised paper (Tables 2–4; Appendix D.1, L1096–1099).
W4: MIMIC-IV may overlap with MIMIC-III, and the transfer results look weak.
Thank you for raising these two points. We clarify the de-duplication and the category drop below.
First, all potentially shared patients were removed. Since MIMIC-III (2001-2012) and MIMIC-IV (2008-2019) overlap only in 2008-2012, our MIMIC-IV evaluation set excludes every patient with any admission that could fall in 2012 or earlier. This matches PhysioNet's own CareVue subset of MIMIC-III, which removes the patients shared with MIMIC-IV [9]. The two databases are therefore distinct patient cohorts, and prior studies use one for development and the other for external validation [10, 11].
Second, the drop is not specific to OpenSLA. Category BAcc also falls for OpenTSLM (from 66.0 to 54.3) and SensorLM (from 55.3 to 50.0), so the drop reflects the shift between the two cohorts. OpenSLA remains the strongest method on MIMIC-IV in every column (Table 7). On necessity, OpenSLA's lead over the best baseline (OpenTSLM) grows from 4.2 points on MIMIC-III to 5.1 points on MIMIC-IV. SensorLM gains 4.8 points but stays 6.7 points below OpenSLA.
Finally, MIMIC-IV is a valid external cohort, and these results support OpenSLA's advantage in transfer. We have added the de-duplication details (Appendix D.4, L1168–1176) and revised Section 6 (Table 7, L499–506).
W5: The main text relies on distant references and under-specifies the method.
Thank you for these helpful suggestions. We agree that the design should be easy to follow from the main text, and we describe the corresponding revisions below.
First, for the distant references, the comparison of alignment strategies (LLaVA-style, Flamingo-style and LoRA) is now in Section 4 (Fig. 4, on VitalDB), next to the design choice it motivates. Section 4 also states the key results of Table 3.
Second, we agree that adding mathematical formulations helps the paper presentation. Due to the page limit, the exact forms of and are given in Appendix C.1 (Eqs. 1-2), as follows:
We set all weights to 1. CGM has no fine-grained action labels, so is 0 for CGM.
Finally, OpenSLA-B and OpenSLA-H share the same backbone, action heads and training objectives. OpenSLA-H additionally uses the Hierarchical Memory (HCSM in the original submission) and the sensor-evidence branch in the multimodal fusion decoder (Appendix C.1). In the clinical domain, the Hierarchical Memory reduces the sensor tokens by 14.49 times and speeds up inference by 3.4 times (Table 29, Appendix E.9).
We have revised Section 4 (L208–269) and Appendix C.1 (L955–1005), redrawn Fig. 3, and moved the caption-facet ablation into Section 6 (Table 8, L514–524). Fig. 8 now summarizes the caption and decoder ablations in the main text, and Table 18 (Appendix E.2, L1234–1239) gives the full results.
Response to Reviewer DhYj (1/4)
Thank you for your thoughtful feedback! We appreciate your recognition of the benefit of caption supervision and the breadth of our benchmark. We address each of your concerns below and have revised the manuscript accordingly. We hope these additional results further substantiate our work's contributions and merit a favorable reassessment.
Thank you for pointing out these cells. We agree that the original sentence in Section 5 was too strong. Below, we identify the cells where a baseline leads and compare OpenSLA with these baselines.
First, a baseline leads in only four of the 28 cells of Tables 2–4. OpenSLA is the best method in the other 24 cells. To compare across action levels, we average the BAcc of necessity, category and fine-label prediction on MIMIC-III and MOVER. These averages are computed directly from Tables 2 and 3. As shown below, OpenSLA significantly exceeds GPT-5.6-Luna on both datasets (p < 0.001):
GPT-5.6-Luna’s advantage is limited to fine-label BAcc, whereas OpenSLA leads by 10.5-35.6 BAcc points on necessity and category on these two datasets. OpenSLA therefore leads on average on both datasets.
Second, ConvTrans's advantage is specific to PEDAP category. OpenSLA-B is significantly better than ConvTrans when all metrics of Table 4 are averaged over both CGM cohorts (67.8 vs. 65.5, p < 0.001).
Together, the baselines lead only on the fine-label BAcc and PEDAP category, and OpenSLA leads on average in both cases. We have therefore revised the sentence in Section 5 (L367–368) to “OpenSLA consistently achieves strong performance across domains and action granularities, with particularly large gains for action necessity and category prediction”.