acceptodds
Under review as a conference paper at ICLR 2027

Sensor-Language-Action Models

Abstract

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multi-modal sensor observations, natural language, and actions within a shared model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark spanning more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns individual context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over state-of-the-arts. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Healthcare AI, Sensor signal, Sensor-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DhYjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sensor-Language-Action (SLA) modeling, which takes physiological signals and patient context, predicts the clinician’s action, and generates a text explanation.

The authors build a benchmark from six datasets across three settings: clinical (MC-MED, MIMIC-III), operating room (MOVER, VitalDB) and CGM (MetaboNet, PEDAP). MIMIC-IV is held out for external evaluation. Actions are organized into three levels: necessity, category and fine-grained label. Caption targets come from a rule-based template pipeline with four facets: observational, relational, global and action evidence.

The model, OpenSLA, is a Qwen-3.5-2B backbone with LoRA. It adds a hierarchical sensor compressor (HCSM), structured action heads and a gated caption decoder, and is trained jointly on action and caption losses. Experiments cover action prediction, caption-based state estimation, evidence grounding (BERTScore), zero-shot prediction of an unseen drug, future-glucose probing and MIMIC-IV transfer.

Critical strengths
  1. Benefit from caption supervision. Removing caption training drops VitalDB necessity AUROC from 74.9 to 54.1 under the same setup (Table 15). This shows that language supervision helps sensor-action modeling.
  2. Breadth of the experiments. The benchmark spans three settings, six cohorts and a unified action hierarchy. If the benchmark and its curation process is released, it would be very useful for the community.
Critical weaknesses
  1. Claims exceed the results. Section 5 (p. 6) states: "As shown in Table 2, Table 3, and Table 4, OpenSLA consistently outperforms all baselines across all action granularities and datasets." Several cells contradict this:
  • In Table 2 and Table 3, GPT-5.6-Luna beats OpenSLA on fine-label BAcc for MIMIC-III and MOVER.
  • In Table 4, ConvTrans beats OpenSLA on PEDAP category.
  1. Caption-based evaluations are biased toward the trained models. Captions are deterministic templates computed from the input window.
  • Table 5 (MAE) and Table 6 (BERTScore) therefore measure how well a model imitates this specific extractor and its template phrasing.
  • OpenSLA and the adapted baselines are trained on these templates, but the zero-shot LLMs are not.
  1. No variance reporting. There are no seeds, number of runs, standard deviations or confidence intervals reported in the experiment results.
  2. The MIMIC-IV results do not strongly support OpenSLA’s advantage in transfer (p. 9)
  • MIMIC-IV comes from the same hospital as MIMIC-III and overlaps it in time, and de-duplication isn’t discussed.
  • Necessity: comparing Table 2 (MIMIC-III) with Table 7, no method degrades much (OpenSLA −1.0/−0.3, OpenTSLM-SP −1.9), and SensorLM improves (+4.8).
  • Category: OpenSLA itself degrades sharply, with AUROC 88.3 → 74.6 (B) and 88.1 → 74.9 (H), and BAcc 73.1 → 56.4 and 74.5 → 58.5.
  1. Main text is hard to follow.
  • Distant references: Section 4 (p. 4) relies on Fig. 7 (p. 9) and Table 3 (p. 6) to motivate the design. Fig. 7 also doesn’t state its dataset.
  • Under-specified method: the main text describes LactL_{\text{act}} and LtextL_{\text{text}} only verbally. Their exact forms and the differences between OpenSLA-B and OpenSLA-H (e.g., B omits HCSM and the sensor-evidence decoder branch) appear only in Fig. 3 and Appendix C.1.
Questions for authors
  1. How were GPT-5.6-Luna and DeepSeek-V3.2 served for MIMIC and MC-MED data? Can the authors refer us to the code? Was this compliant with the PhysioNet DUA? (See: https://physionet.org/news/post/llm-responsible-use/)
  2. All OpenSLA variants receive both sensor signals and context text, and no ablation removes either input. Could the authors report sensor-only and context-only variants of OpenSLA? This would show whether the gains come from combining the modalities or mainly from the LLM reading the patient record.
7 comments
AuthorsOpenReview ↗

Response to Reviewer DhYj (1/4)

Thank you for your thoughtful feedback! We appreciate your recognition of the benefit of caption supervision and the breadth of our benchmark. We address each of your concerns below and have revised the manuscript accordingly. We hope these additional results further substantiate our work's contributions and merit a favorable reassessment.

 

W1: Some cells in Tables 2–4 contradict the claim in Section 5.

Thank you for pointing out these cells. We agree that the original sentence in Section 5 was too strong. Below, we identify the cells where a baseline leads and compare OpenSLA with these baselines.

First, a baseline leads in only four of the 28 cells of Tables 2–4. OpenSLA is the best method in the other 24 cells. To compare across action levels, we average the BAcc of necessity, category and fine-label prediction on MIMIC-III and MOVER. These averages are computed directly from Tables 2 and 3. As shown below, OpenSLA significantly exceeds GPT-5.6-Luna on both datasets (p < 0.001):

DatasetOpenSLA-BOpenSLA-HGPT-5.6-Luna
MIMIC-III74.174.866.1
MOVER69.972.755.5

GPT-5.6-Luna’s advantage is limited to fine-label BAcc, whereas OpenSLA leads by 10.5-35.6 BAcc points on necessity and category on these two datasets. OpenSLA therefore leads on average on both datasets.

Second, ConvTrans's advantage is specific to PEDAP category. OpenSLA-B is significantly better than ConvTrans when all metrics of Table 4 are averaged over both CGM cohorts (67.8 vs. 65.5, p < 0.001).

Together, the baselines lead only on the fine-label BAcc and PEDAP category, and OpenSLA leads on average in both cases. We have therefore revised the sentence in Section 5 (L367–368) to “OpenSLA consistently achieves strong performance across domains and action granularities, with particularly large gains for action necessity and category prediction”.

AuthorsOpenReview ↗

Response to Reviewer DhYj (2/4)

W2: Caption-based evaluations are biased toward models trained on the templates.

We would like to clarify our caption-based evaluation and provide additional analyses of template effects below.

First, the ground-truth physiological values are computed directly from raw sensor recordings, independently of caption templates. As defined in Appendix D.2, we use well-established algorithms [1, 2] to derive these values. Moreover, we extract numerical values from model-generated captions and evaluate them against signal-derived ground truth using MAE. Thus, Table 5 evaluates physiological state accuracy regardless of the template wording. This holds equally for the zero-shot LLMs, since only the stated numbers are scored.

Second, our action-explanation evaluation measures semantic fidelity by computing BERTScore [6] between generated and ground-truth action-evidence captions. BERTScore is a well-established metric that is widely adopted in recent time-series–language studies [3-5]. To further clarify our evaluation, we separately compute BERTScore for action-relevant content and shared template text, alongside the full-caption results:

MethodMC-MED FullMC-MED TemplateMC-MED ActionMIMIC-III FullMIMIC-III TemplateMIMIC-III Action
OpenTSLM79.572.0179.2780.373.3579.97
SensorLM77.271.7978.9778.873.6282.66
OpenSLA-B81.571.6680.4478.672.9880.47
OpenSLA-H82.072.1881.2580.973.9782.35

As shown above, action-relevant scores remain comparable to full-caption scores, while template scores are consistently lower and nearly identical across models. This indicates that template wording does not drive the evaluation differences.

Furthermore, we separately swap the template wording and action-related terms in reference captions, repeating each swap five times per sample. The resulting changes in BERTScore (Δ ± SD) are shown below:

MethodMC-MED TemplateMC-MED ContentMIMIC-III TemplateMIMIC-III Content
GPT-5.6-Luna−0.08 ± 0.56+1.45 ± 1.37−0.02 ± 0.61+1.09 ± 1.54
DeepSeek-V3.2−0.03 ± 0.45+1.69 ± 1.31−0.02 ± 0.46+1.49 ± 1.22
OpenTSLM−0.17 ± 1.67−4.22 ± 2.26−0.09 ± 1.32−2.82 ± 2.67
SensorLM+0.03 ± 1.92−3.96 ± 2.22−0.03 ± 1.44−2.41 ± 2.56
OpenSLA-B−0.16 ± 1.72−4.95 ± 2.45−0.10 ± 1.47−3.65 ± 2.48
OpenSLA-H−0.16 ± 1.74−5.36 ± 2.47−0.14 ± 1.56−4.29 ± 2.58

As confirmed, swapping template wording has minimal impact, whereas swapping action evidence substantially lowers the scores of the models trained on our captions. This further confirms that action evidence drives the evaluation of these models. In contrast, the zero-shot LLMs' scores rise slightly, since their captions are generic clinical descriptions with little patient-specific evidence. In Table 5, the zero-shot LLMs also make larger errors on the stated values (e.g., ECG-II rate MAE 26.5-28.4 vs. 15.9-20.1 for OpenSLA). The lower BERTScore of the zero-shot LLMs thus reflects less specific and less accurate evidence.

Finally, we would like to clarify how our captions differ from "deterministic templates". While the values are deterministic, since they are computed from the input window, the wording is not. Our pipeline integrates multi-faceted semantics spanning channel-level observations, cross-channel relationships, global patient context, and action-relevant evidence. Furthermore, we randomly sample from 162 diverse templates (Appendix F.2) to ensure linguistic diversity.

We have clarified the caption-based evaluation in the revised paper (Section 5, L423–427; Appendix D.2) and added the template and action-evidence analyses (Tables 26–27, Appendix E.7, L1401–1441).

AuthorsOpenReview ↗

Response to Reviewer DhYj (3/4)

W3: The results lack seeds, standard deviations or confidence intervals.

Thank you for raising this point. We agree that reporting uncertainty is essential for judging the differences between methods, and we have added it to the revised manuscript as described below.

First, Tables 2-4 now report 95% confidence intervals for all main action results. The intervals are computed by bootstrap over test patients with 2,000 resamples (Appendix D.1). Bootstrap intervals are a standard way to report uncertainty in clinical prediction models [7]. Moreover, data sampling is often a larger source of variance than random seeds [8]. OpenSLA's advantage holds within these intervals. For example, on MIMIC-III necessity, OpenSLA-B reaches 75.4 ± 1.4 BAcc and the strongest baseline, OpenTSLM, 71.2 ± 1.7, with no overlap.

Second, our conclusions already hold across independent training runs. OpenSLA is trained separately in each of the three domains, and each run is the best method in most cells of its own table. A gain from a lucky seed would be unlikely to repeat across three independently trained domains.

Taken together, our results are reliable under both test-set variation and training randomness. We have added these intervals to the revised paper (Tables 2–4; Appendix D.1, L1096–1099).

 

W4: MIMIC-IV may overlap with MIMIC-III, and the transfer results look weak.

Thank you for raising these two points. We clarify the de-duplication and the category drop below.

First, all potentially shared patients were removed. Since MIMIC-III (2001-2012) and MIMIC-IV (2008-2019) overlap only in 2008-2012, our MIMIC-IV evaluation set excludes every patient with any admission that could fall in 2012 or earlier. This matches PhysioNet's own CareVue subset of MIMIC-III, which removes the patients shared with MIMIC-IV [9]. The two databases are therefore distinct patient cohorts, and prior studies use one for development and the other for external validation [10, 11].

Second, the drop is not specific to OpenSLA. Category BAcc also falls for OpenTSLM (from 66.0 to 54.3) and SensorLM (from 55.3 to 50.0), so the drop reflects the shift between the two cohorts. OpenSLA remains the strongest method on MIMIC-IV in every column (Table 7). On necessity, OpenSLA's lead over the best baseline (OpenTSLM) grows from 4.2 points on MIMIC-III to 5.1 points on MIMIC-IV. SensorLM gains 4.8 points but stays 6.7 points below OpenSLA.

Finally, MIMIC-IV is a valid external cohort, and these results support OpenSLA's advantage in transfer. We have added the de-duplication details (Appendix D.4, L1168–1176) and revised Section 6 (Table 7, L499–506).

 

W5: The main text relies on distant references and under-specifies the method.

Thank you for these helpful suggestions. We agree that the design should be easy to follow from the main text, and we describe the corresponding revisions below.

First, for the distant references, the comparison of alignment strategies (LLaVA-style, Flamingo-style and LoRA) is now in Section 4 (Fig. 4, on VitalDB), next to the design choice it motivates. Section 4 also states the key results of Table 3.

Second, we agree that adding mathematical formulations helps the paper presentation. Due to the page limit, the exact forms of Lact\mathcal{L}_{\text{act}} and Ltext\mathcal{L}_{\text{text}} are given in Appendix C.1 (Eqs. 1-2), as follows:

L=Lact+λtextLtext,Lact=λnecLnec+λcatLcat+λlabelLlabel.\mathcal{L}=\mathcal{L}_{\text{act}}+\lambda_{\text{text}}\mathcal{L}_{\text{text}},\qquad \mathcal{L}_{\text{act}}=\lambda_{\text{nec}}\mathcal{L}_{\text{nec}}+\lambda_{\text{cat}}\mathcal{L}_{\text{cat}}+\lambda_{\text{label}}\mathcal{L}_{\text{label}}.

We set all weights to 1. CGM has no fine-grained action labels, so λlabel\lambda_{\text{label}} is 0 for CGM.

Finally, OpenSLA-B and OpenSLA-H share the same backbone, action heads and training objectives. OpenSLA-H additionally uses the Hierarchical Memory (HCSM in the original submission) and the sensor-evidence branch fsf_s in the multimodal fusion decoder (Appendix C.1). In the clinical domain, the Hierarchical Memory reduces the sensor tokens by 14.49 times and speeds up inference by 3.4 times (Table 29, Appendix E.9).

We have revised Section 4 (L208–269) and Appendix C.1 (L955–1005), redrawn Fig. 3, and moved the caption-facet ablation into Section 6 (Table 8, L514–524). Fig. 8 now summarizes the caption and decoder ablations in the main text, and Table 18 (Appendix E.2, L1234–1239) gives the full results.

Reviewer_Gs2JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpenSLA connects physiological sensor signals to clinical, operating-room, and insulin actions through language. The benchmark unifies six development datasets plus MIMIC-IV with a three-level action hierarchy. Rule-based captions describe observations, cross-signal relations, patient context, and action evidence. The Qwen-3.5-2B model combines a frozen signal encoder, HCSM compression, action heads, and caption supervision.

Critical strengths
  1. Combining clinical, OR and CGM datasets with a common three-level action hierarchy is a real data effort and could be useful to others if released.

  2. Caption construction is described in detail, including the templates, and the authors are careful to say captions describe associations, not causes or treatment recommendations.

  3. The ablations in the appendix (Tables 12 to 15) show that caption supervision improves action AUROC over action-only training, for example VitalDB necessity 54.1 to 74.9.

  4. Baselines cover several families (supervised, time-series foundation models, sensor-language models, API LLMs).

Critical weaknesses
  1. No test of whether the sensors matter. The text context includes patient background, presentation, and prior actions. Prior actions and the chief complaint can predict the next action strongly on their own. The sensor-only baselines (ConvTrans, Chronos-2, Moirai) do not get text, so they are not a fair comparison. Please add (a) OpenSLA with text only (no sensor tokens) and (b) a strong tabular baseline such as gradient boosting on structured context features. Without these, it is not clear that the model is "grounded in sensor evidence."

  2. Action explanation is also template matching. BERTScore with DistilBERT against template text strongly favors models trained on the same templates. This is why the adapted OpenTSLM and SensorLM come close to OpenSLA, and zero-shot LLMs do not. Also, the action captions are built using the recorded action label to select evidence, so the "explanation" is the pipeline's own rule, not a clinical rationale. A small clinician review of generated explanations would be much more convincing.

  3. The "world knowledge emerges" section is overclaimed. Monotonic insulin dose along an interpolation path, using nearest-neighbor retrieval, is weak evidence of anything beyond the embedding being sorted by dose, which it was trained to predict. For future glucose, the differences between models are small (mean MAE 24.0 vs 26.5 vs 26.9), and there is no simple baseline like "use the last glucose value" or a linear extrapolation of the recent slope, which are known to be strong for 2-hour horizons. Please add those baselines and soften the wording

Questions for authors
  1. What does OpenSLA get with text only, and with sensors only?

  2. Did you remove patients who appear in both MIMIC-III and MIMIC-IV?

  3. What are the results on the unseen action test if the negatives are other antibiotics?

  4. How does the future glucose readout compare with a last-value or linear-trend baseline?

  5. How were the API calls on MIMIC data done under the PhysioNet agreement?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Gs2J (1/4)

Thank you for your thoughtful feedback! We appreciate your recognition of the value of sensor-language-action modeling and the breadth of our evaluation. We address each of your concerns below and have revised the manuscript accordingly. We hope these additional results further confirm our work's contributions and merit a favorable increase of the score.

 

W1 & Q1: It is unclear whether the sensor signals matter, and how OpenSLA performs with text only or with sensors only.

Thank you for raising this question. We agree that demonstrating the value of sensor signals beyond textual context is essential, and we provide additional experimental evidence and clarifications below.

First, we would like to clarify that we have established the critical role of sensor signals in SLA modeling through the following two aspects:

  1. Independent predictive value. Sensor-only baselines already achieve non-trivial action prediction performance. For example, they obtain 70.9 AUROC on VitalDB category prediction (Tables 2–4), demonstrating the independent predictive value of sensor signals. These baselines assess the sensor-only modeling paradigm rather than serving as context-matched comparisons.
  2. Counterfactual sensor substitution. Fig. 9 demonstrates that changing sensor inputs produces corresponding, physiologically consistent changes in generated captions, even with fixed textual context.

Furthermore, following the reviewer’s suggestion, we introduce a context-only ablation with the same Qwen-3.5-2B backbone, as well as a sensor-only ablation, to quantify the complementary contribution of sensor signals. OpenSLA significantly outperforms its context-only and sensor-only counterparts on action prediction as shown below:

MethodNecessityCategory
Context-only72.379.6
Sensor-only57.169.8
OpenSLA79.183.1

Notably, OpenSLA achieves a 6.8-point AUROC gain over its context-only counterpart on necessity prediction, which directly confirms the additional predictive value of sensor signals beyond textual context.

We also mask OpenSLA's sensor inputs at test time. As shown below, masking sensors leads to a 9.9-point AUROC drop on necessity prediction, further demonstrating OpenSLA's reliance on sensor information.

MethodNecessityCategory
OpenSLA (w/o sensors)69.280.7
OpenSLA79.183.1

Finally, per the reviewer’s request, we further introduce a gradient-boosting baseline on structured clinical context. As shown below, OpenSLA substantially outperforms this baseline on action prediction tasks.

MethodNecessityCategory
Gradient boosting72.0061.15
OpenSLA80.8086.00

Together, these results demonstrate the additional predictive value of sensor signals in SLA modeling beyond textual and structured clinical context. We have added the classical ML baseline comparisons (Table 16, Appendix E.1, L1184–1225) and the sensor ablation and masking experiments (Tables 24–25, Appendix E.5, L1345–1376) to the revised paper, alongside clarifications of the controlled sensor-substitution analysis (Fig. 9, Appendix E.6, L1393–1400).

AuthorsOpenReview ↗

Response to Reviewer Gs2J (2/4)

W2: Action explanation is evaluated as template matching, the explanation follows the pipeline's own rule, and a clinician review is missing.

Thank you for your comments. We would like to provide additional results and clarifications below.

First, our action-explanation evaluation measures semantic fidelity by computing BERTScore [4] between generated and ground-truth action-evidence captions, which is widely adopted in recent time-series–language studies [1-3]. To further clarify our evaluation, we separately compute BERTScore for action-relevant content and shared template text, alongside the full-caption results:

MethodMC-MED FullMC-MED TemplateMC-MED ActionMIMIC-III FullMIMIC-III TemplateMIMIC-III Action
OpenTSLM79.572.0179.2780.373.3579.97
SensorLM77.271.7978.9778.873.6282.66
OpenSLA-B81.571.6680.4478.672.9880.47
OpenSLA-H82.072.1881.2580.973.9782.35

As shown above, action-relevant scores remain comparable to full-caption scores, while template scores are consistently lower and nearly identical across models. This indicates that template wording does not drive the evaluation differences.

Furthermore, we separately swap the template wording and action-related terms in reference captions, repeating each swap five times per sample. The resulting changes in BERTScore (Δ ± SD) are shown below:

MethodMC-MED TemplateMC-MED ContentMIMIC-III TemplateMIMIC-III Content
OpenTSLM−0.17 ± 1.67−4.22 ± 2.26−0.09 ± 1.32−2.82 ± 2.67
SensorLM+0.03 ± 1.92−3.96 ± 2.22−0.03 ± 1.44−2.41 ± 2.56
OpenSLA-B−0.16 ± 1.72−4.95 ± 2.45−0.10 ± 1.47−3.65 ± 2.48
OpenSLA-H−0.16 ± 1.74−5.36 ± 2.47−0.14 ± 1.56−4.29 ± 2.58

Swapping template wording causes minimal score changes with lower variance, whereas swapping action-relevant terms leads to substantial drops and greater variability. These results confirm that the evaluation is primarily driven by action evidence rather than template wording.

Second, the action evidence selected in our captions is consistent with clinical guidelines and therefore encodes clinical knowledge. Moreover, our pipeline goes beyond fixed templates, integrating channel-level observations, cross-channel relationships, patient context, and action-relevant evidence. We employ 162 randomly sampled templates to ensure linguistic diversity (Appendix F.2). Thus, our supervision captures structured physiological information and medically grounded evidence associations.

Third, our study focuses on sensor-grounded action explanations, evaluating whether generated explanations capture physiological evidence relevant to predicted actions (Appendix B.4). A clinician review would extend this toward the more demanding task of clinical reasoning and decision support, representing a broader direction for future validation. We have noted the need for clinical validation in Section 7 (L528–533).

We have clarified the caption-based evaluation in the revised paper (Section 5, L423–427; Appendix D.2) and added the template and action-evidence analyses (Tables 26–27, Appendix E.7, L1401–1441). Section 4.2 (L269) also describes how the fusion decoder conditions each explanation on the predicted actions and the sensor evidence.

AuthorsOpenReview ↗

Response to Reviewer Gs2J (3/4)

W3 & Q4: The “world knowledge emerges” section is overclaimed, and the future-glucose readout lacks last-value and linear-trend baselines.

Thank you for the feedback. In fact, these capabilities emerge from our SLA modeling, rather than the pretrained backbone alone. We provide additional results and clarifications below.

First, we demonstrate that these capabilities arise from SLA modeling rather than the pretrained backbone alone. Specifically, we compare OpenSLA with Qwen-3.5-2B on unseen-action detection and future-state readout. We provide Qwen with sensor observations converted into numerical text and apply the same probing protocol for future-state readout. As shown below, OpenSLA substantially outperforms the pretrained backbone on both tasks, demonstrating that these capabilities are enabled by SLA modeling, instead of simply inheriting from the backbone model.

Unseen Imipenem Detection

MethodF1BAcc
OpenTSLM17.454.8
Qwen-3.5-2B25.057.1
DeepSeek-V4.1-Flash50.066.7
OpenSLA73.772.6

Future Glucose Readout (MAE):

MethodMean↓Min↓Max↓
Qwen-3.5-2B31.728.936.6
OpenSLA24.022.328.0

Second, SLA exhibits capabilities beyond its training objectives, which supports our characterization of “emergent world knowledge”. In particular, future-state readout reveals predictive physiological structure, while zero-shot action generalization demonstrates transferable action semantics. These findings align with prior studies on emergent world representations [5] and action generalization [6]. To characterize these findings more precisely, we have retitled Section 6 “Generalization and Predictive Structure in SLA Representations”. To further validate these capabilities, we extend our analyses in two directions with additional actions, domains, and baselines:

(1) Unseen Action Generalization. We extend zero-shot action prediction to an additional unseen OR intervention, intravenous metoprolol. We evaluate it against seen BP-lowering drugs and vasodilators as negatives. OpenSLA outperforms the LLM baselines in both settings as shown below.

MethodBP-lowering F1BP-lowering BAccVasodilators F1Vasodilators BAcc
Qwen-3.5-2B51.657.953.961.7
DeepSeek-V4.1-Flash48.552.450.255.5
OpenSLA67.664.681.581.4

(2) Future-State Readout. We extend future-state readout to the OR domain, where OpenSLA outperforms the baselines across all three physiological targets (MAE), as shown below.

MethodETCO2BISMAP
Persistence3.577.9718.08
Chronos-23.848.8017.57
SensorLM2.725.8612.85
OpenSLA2.675.7512.79

We additionally compared last-value and linear-trend baselines for future glucose readout. OpenSLA still achieves the lowest errors for mean, minimum, and maximum glucose among all methods.

MethodMeanMinMaxSlope
Last value27.031.534.10.43
Linear trend42.933.649.10.73
OpenTSLM26.524.230.60.39
SensorLM26.924.630.50.39
Chronos-229.729.332.80.48
GluFormer33.230.038.70.41
Qwen-3.5-2B31.728.936.60.41
OpenSLA24.022.328.00.39

The ablations in Fig. 8 and Table 18 further show that action prediction drops without caption supervision. We have expanded the unseen-action and future-state evaluations (Tables 19–23, Appendices E.3–E.4, L1282–1341), clarified the evaluation setup (Appendix D.3, L1111–1120), and refined the claims in Section 6 (Figs. 6–7, L463–485) of the revised paper.

 

Q2: Were patients who appear in both MIMIC-III and MIMIC-IV removed?

Thank you for your question. We have removed all potentially shared patients.

First, MIMIC-III (2001–2012) and MIMIC-IV (2008–2019) can overlap only in 2008–2012. To eliminate this overlap, we excluded all MIMIC-IV patients with any admission in or before 2012.

Second, this matches PhysioNet's own CareVue subset of MIMIC-III, which excludes the patients shared with MIMIC-IV [7]. The two datasets thus form distinct patient cohorts, consistent with prior studies using them for development and external validation [8, 9].

We have clarified the patient de-duplication procedure in Appendix D.4 (L1168–1176) of the revised paper to make this process clearer.

Reviewer_BNt5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sensor-Language-Action modeling, which uses language to link physiological sensor signals, patient context, and clinical actions. The authors build a benchmark from several public datasets, and carefully generate rule-based captions as language supervision. Then, they train OpenSLA to jointly predict multi-level actions and produce captions. Experiments compare OpenSLA with supervised, time-series foundation, sensor-language, and zero-shot LLM baselines on action prediction, state understanding, and action explanation, and also include zero-shot, transfer, and ablation analyses.

Critical strengths
  1. The idea of moving sensor-language models from describing signals to predicting clinical actions is well motivated, and is a promising topic
  2. The caption pipeline is careful and well documented in the appendix. As a result, the language supervision could be easy to check and reproduce.
  3. The benchmark is broad and diverse. This could be a useful testbed for future work.
Critical weaknesses
  1. This paper does not demonstrate how much the sensor signals actually help. There is no text-only baseline with the same backbone, and no OpenSLA run with the sensors masked. Since the text context includes history and prior actions, the large gains in the clinical and OR settings may mostly be attributed to the text.
  2. The main comparisons are not sufficiently fair. The sensor-only baselines never see the text, the SensorLM-style baseline has a randomly initialized text side, and OpenTSLM uses a smaller 1B backbone. Moreover, OpenSLA uses classification heads with tuned thresholds, while the generative baselines only output discrete answers and get no AUROC.
  3. The state understanding and explanation results are hard to interpret. The reference values and evidence texts come from the same rule-based templates that OpenSLA is trained to produce, so Tables 5 and 6 mostly measure how well a model copies the template format.
Questions for authors
  1. Could you report a text-only baseline with the same Qwen-3.5-2B backbone, and OpenSLA with the sensor inputs masked at test time? This would show how much of the gain comes from the sensors rather than the text context.
  2. The comparisons with prior sensor-language models in Tables 2 to 4 are not well matched. OpenTSLM uses a smaller 1B backbone, and the SensorLM-style baseline has a randomly initialized text side. Could you evaluate at least one of them under comparable conditions, for example with a backbone of similar size? This would show whether OpenSLA's advantage over prior methods still holds in a fair comparison.
  3. Sec. 6 attributes abilities such as zero-shot Imipenem detection and future-state readout to SLA modeling. However, it is unclear whether these abilities come from SLA training or already exist in the pretrained Qwen backbone. In addition, some of these claims rely on very limited evidence, for example a single unseen label and a single interpolation path. So terms such as "world knowledge emerges" and "world-modeling capability" seem too strong unless the authors could provide more evidence.
7 comments
AuthorsOpenReview ↗

Response to Reviewer BNt5 (1/4)

Thank you for your thoughtful feedback! We appreciate your recognition of the value of sensor-language-action modeling and the breadth of our evaluation. We address each of your concerns below and have revised the manuscript accordingly. We hope these additional results further substantiate our work's contributions and merit a favorable reassessment.

 

W1 & Q1: This paper does not demonstrate how much the sensor signals actually help.

Thank you for raising this important question. We agree that demonstrating the value of sensor signals is essential, and we provide additional experimental evidence and clarifications below.

First, we have established the critical role of sensor signals in SLA modeling through two aspects:

  1. Independent predictive value. Sensor-only baselines already achieve non-trivial action prediction performance, e.g., 70.9 AUROC on VitalDB category prediction (Table 3).
  2. Counterfactual sensor substitution (Fig. 9) shows that changing sensor inputs, with the textual context fixed, produces physiologically consistent changes in generated captions.

Furthermore, following the reviewer’s suggestion, we introduce a text context-only ablation with the same Qwen-3.5-2B backbone, as well as a sensor-only ablation, to quantify the complementary contribution of sensor signals. OpenSLA significantly outperforms its context-only and sensor-only counterparts on action prediction as shown below:

MethodNecessityCategory
Context-only72.379.6
Sensor-only57.169.8
OpenSLA79.183.1

Notably, OpenSLA achieves a 6.8-point AUROC gain over its context-only counterpart on necessity prediction, which directly confirms the additional predictive value of sensor signals beyond textual context.

We also mask OpenSLA's sensor inputs at test time. As shown below, masking sensors leads to a 9.9-point AUROC drop on necessity prediction, further demonstrating OpenSLA's reliance on sensor information.

MethodNecessityCategory
OpenSLA (w/o sensors)69.280.7
OpenSLA79.183.1

Finally, on the clinical datasets, we compare OpenSLA with gradient boosting on the structured context alone. As shown below, OpenSLA substantially outperforms this baseline.

MethodNecessityCategory
Gradient boosting72.0061.15
OpenSLA80.8086.00

Together, these results demonstrate the predictive value of sensor signals beyond textual and structured clinical context. We have added the classical ML baselines (Table 16), the sensor ablation and masking experiments (Tables 24–25, Appendix E.5, L1345–1376), and clarifications of the sensor-substitution analysis (Fig. 9, Appendix E.6, L1393–1399).

 

W2 & Q2: Replace the official baseline implementations with the same backbone as ours.

Thank you for this comment. We would like to clarify our baseline comparisons and provide additional results as follows.

First, our comparisons deliberately preserve each method's official modeling design to evaluate distinct modeling paradigms. Imposing an identical architecture would remove their defining differences. Specifically, we used the best-performing 1B Llama backbone from the original OpenTSLM paper [4] (Table 2) and SensorLM's CoCa-based sensor-language pretraining.

Furthermore, following the reviewer’s suggestion, we retrain OpenTSLM with the same Qwen-3.5-2B backbone as OpenSLA and replace SensorLM's randomly initialized text encoder. As shown below, OpenSLA maintains a substantial BAcc advantage over both variants:

MethodNecessityCategory
OpenTSLM67.1061.00
SensorLM63.3056.60
OpenSLA72.7571.60

Second, our classification heads do not use hand-tuned thresholds. Instead, thresholds are set exclusively on the validation set by a fixed rule that maximizes balanced accuracy for each binary target (Appendix D.1), a long-established practice in classification [1] widely adopted in recent works [2, 3]. Since generative baselines produce discrete predictions by their native design, we preserve each method's native decision mechanism and evaluate all methods with the same BAcc metric.

To further address this concern, we consider two other threshold rules: F1 maximization on the validation set and a fixed threshold of 0.5. As shown below, the threshold rule does not affect OpenSLA's ranking against LLM API baselines.

MethodNecessityCategory
GPT-5.6-Luna57.6963.57
DeepSeek-V3.255.8659.18
OpenSLA (max-F1)72.7669.55
OpenSLA (fixed)72.7667.62
OpenSLA (max bacc)72.7771.56

We have added the baseline variants (Table 16, L1184–1225) and the threshold-rule comparison (Table 28, L1443–1459), and clarified the threshold selection protocol (Appendix D.1, L1090–1093).

AuthorsOpenReview ↗

Response to Reviewer BNt5 (2/4)

W3: Tables 5 and 6 may mostly measure how well a model copies the templates.

We would like to clarify our caption-based evaluation and provide additional analyses of template effects below.

First, the ground-truth physiological values are computed directly from raw sensor recordings, independently of caption templates. As defined in Appendix D.2, we use well-established algorithms [5, 6] to derive these values. Moreover, we extract numerical values from model-generated captions and evaluate them against signal-derived ground truth using MAE. Thus, Table 5 evaluates physiological state accuracy regardless of the template wording.

Second, our action-explanation evaluation measures semantic fidelity by computing BERTScore [10] between generated and ground-truth action-evidence captions, which is widely adopted in recent time-series–language studies [7-9]. To further clarify our evaluation, we separately compute BERTScore for action-relevant content and shared template text, alongside the full-caption results:

MethodMC-MED FullMC-MED TemplateMC-MED ActionMIMIC-III FullMIMIC-III TemplateMIMIC-III Action
OpenTSLM79.572.0179.2780.373.3579.97
SensorLM77.271.7978.9778.873.6282.66
OpenSLA-B81.571.6680.4478.672.9880.47
OpenSLA-H82.072.1881.2580.973.9782.35

As shown above, action-relevant scores remain comparable to full-caption scores, while template scores are consistently lower and nearly identical across models. This indicates that template wording does not drive the evaluation differences.

Furthermore, we separately swap the template wording and action-related terms in reference captions, repeating each swap five times per sample. The resulting changes in BERTScore (Δ ± SD) are shown below:

MethodMC-MED TemplateMC-MED ContentMIMIC-III TemplateMIMIC-III Content
OpenTSLM−0.17 ± 1.67−4.22 ± 2.26−0.09 ± 1.32−2.82 ± 2.67
SensorLM+0.03 ± 1.92−3.96 ± 2.22−0.03 ± 1.44−2.41 ± 2.56
OpenSLA-B−0.16 ± 1.72−4.95 ± 2.45−0.10 ± 1.47−3.65 ± 2.48
OpenSLA-H−0.16 ± 1.74−5.36 ± 2.47−0.14 ± 1.56−4.29 ± 2.58

Swapping template wording causes minimal score changes with lower variance, whereas swapping action-relevant terms leads to substantial drops and greater variability. These results confirm that the evaluation is primarily driven by action evidence rather than template wording.

Finally, we respectfully disagree with the characterization of our captioning pipeline as merely “rule-based templates”. Our pipeline goes far beyond template-based generation, integrating multi-faceted semantics spanning channel-level observations, cross-channel relationships, global patient context, and action-relevant evidence. Furthermore, we randomly sample from 162 diverse templates (Appendix F.2) to ensure linguistic diversity. Thus, our supervision captures structured physiological information rather than fixed textual patterns.

We have clarified the caption-based evaluation in the revised paper (Section 5, L423–427; Appendix D.2) and added the template and action-evidence analyses (Tables 26–27, Appendix E.7, L1401–1441).

AuthorsOpenReview ↗

Response to Reviewer BNt5 (3/4)

Q3: The abilities in Sec. 6 may come from the pretrained backbone, and the wording seems too strong.

Thank you for the feedback. In fact, these capabilities emerge from our SLA modeling, rather than the pretrained backbone alone. We provide additional results and clarifications below.

First, pretrained semantic knowledge does not naturally translate into sensor-grounded action understanding. Although Qwen may have encountered action-related concepts during pretraining, it cannot directly process sensor signals or infer actions from them. Prior work [8, 11] has demonstrated the importance of fine-tuning pretrained LLMs for time-series understanding. Our SLA training establishes the mapping between physiological observations and action semantics.

To examine the contribution of the pretrained backbone, we convert sensor observations into numerical text and evaluate Qwen-3.5-2B on unseen-action detection and future-state readout, using the same probing protocol for the latter. As shown below, OpenSLA outperforms Qwen-only on both tasks, confirming that these capabilities are not inherited from the pretrained backbone.

Unseen Imipenem Detection

MethodF1BAcc
OpenTSLM17.454.8
Qwen-3.5-2B25.057.1
DeepSeek-V4.1-Flash50.066.7
OpenSLA73.772.6

Future Glucose Readout (MAE):

MethodMean↓Min↓Max↓
Qwen-3.5-2B31.728.936.6
OpenSLA24.022.328.0

Second, SLA exhibits capabilities beyond its training objectives, which supports our characterization of “emergent world knowledge”. In particular, future-state readout reveals predictive physiological structure, while zero-shot action generalization demonstrates transferable action semantics. These findings align with prior studies on emergent world representations [12] and action generalization [13]. To characterize these findings more precisely, we have retitled Section 6 “Generalization and Predictive Structure in SLA Representations”. To further validate these capabilities, we extend our analyses in two directions with additional actions, domains, and baselines:

(1) Unseen Action Generalization. We extend zero-shot action prediction to an additional unseen OR intervention, intravenous metoprolol. We evaluate it against seen BP-lowering drugs and vasodilators as negatives. OpenSLA outperforms the LLM baselines in both settings as shown below.

MethodBP-lowering F1BP-lowering BAccVasodilators F1Vasodilators BAcc
Qwen-3.5-2B51.657.953.961.7
DeepSeek-V4.1-Flash48.552.450.255.5
OpenSLA67.664.681.581.4

(2) Future-State Readout. We extend future-state readout to the OR domain, where OpenSLA outperforms the baselines across all three physiological targets (MAE), as shown below.

MethodETCO2BISMAP
Persistence3.577.9718.08
Chronos-23.848.8017.57
SensorLM2.725.8612.85
OpenSLA2.675.7512.79

We additionally compared last-value and linear-trend baselines for future glucose readout. OpenSLA still achieves the lowest errors for mean, minimum, and maximum glucose among all methods.

MethodMeanMinMaxSlope
Last value27.031.534.10.43
Linear trend42.933.649.10.73
OpenTSLM26.524.230.60.39
SensorLM26.924.630.50.39
Chronos-229.729.332.80.48
GluFormer33.230.038.70.41
Qwen-3.5-2B31.728.936.60.41
OpenSLA24.022.328.00.39

The ablations in Fig. 8 and Table 18 further show that action prediction drops without caption supervision. We have expanded the unseen-action and future-state evaluations (Tables 19–23, Appendices E.3–E.4, L1282–1341), clarified the evaluation setup (Appendix D.3, L1111–1120), and refined the claims in Section 6 (Figs. 6–7, L463–485) of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all the reviewers for acknowledging the contributions of our work and providing insightful comments and suggestions! We are glad to see that the reviewers found that:

  1. Our idea of sensor-language-action modeling is novel, well-motivated, and promising (Reviewer BNt5).
  2. The unified benchmark is broad, diverse, and valuable for future research (Reviewers BNt5, Gs2J, DhYj).
  3. The language supervision pipeline is carefully designed, well-documented, and reproducible (Reviewers BNt5, Gs2J).
  4. Language supervision substantially improves sensor-action modeling, as supported by the ablation studies (Reviewer DhYj).
  5. The experimental evaluation is extensive across diverse real-world domains and modeling paradigms (Reviewers Gs2J, DhYj).

 

To address concerns raised by the reviewers, we have made the following changes and updated the manuscript accordingly (all changes are highlighted in red in the updated paper):

  1. We expanded our OpenTSLM and SensorLM baselines with additional variants in Table 16, further strengthening OpenSLA's advantage, as suggested by Reviewer BNt5, Gs2J.
  2. We added gradient boosting on structured clinical context in Table 16 to compare OpenSLA against classical ML methods, as suggested by Reviewer Gs2J.
  3. We added context-only, sensor-only, and sensor-masked ablations in Tables 24–25 to quantify each component's contribution, as suggested by Reviewer BNt5, Gs2J, DhYj.
  4. We added analyses of content sensitivity and template effects in Tables 26–27 to assess our evaluation methods, as suggested by Reviewer BNt5, Gs2J, DhYj.
  5. We added harder negatives, another unseen action, and new baselines in Tables 19–21 to test action generalization, as suggested by Reviewers BNt5, Gs2J.
  6. We added forecasting baselines and OR future-state evaluations in Tables 22–23, supporting predictive physiological structure in SLA representations. We also retitled Sec. 6 and revised its two findings (Figs. 6–7), as suggested by Reviewer BNt5, Gs2J.
  7. We added 95% bootstrap confidence intervals to action prediction results in Tables 2–4 and refined Sec. 5, as suggested by Reviewer DhYj.
  8. We clarified threshold selection, external cohort construction, data privacy, methodological details, and OpenSLA-H compression efficiency in Sec. 4, Appendices A, C.1, C.2, D.1, D.3, D.4 & E.6, and Tables 7, 9–13 & 28–29, as suggested by Reviewer BNt5, Gs2J, DhYj.
  9. Beyond the points above, we rewrote the description of the unified action space (Sec. 3) and added a limitation paragraph on clinical validation and deployment (Sec. 7). We redrew Figs. 3 & 8, compared alignment strategies in Fig. 4 and Table 17, and updated Table 1. We also moved the caption-facet ablation into the main text and added six caption-facet combinations (Table 8). Table 18 gives the full results of Fig. 8. HCSM is now called the Hierarchical Memory.

 

We hope our responses have addressed all concerns from the reviewers adequately. We thank all reviewers again for their time and feedback, and please feel free to let us know if there are other clarifications or experiments we can offer. We would really appreciate it if the reviewers could consider raising their scores after evaluating our updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.