acceptodds
Under review as a conference paper at ICLR 2027

PPG-LM: A Photoplethysmography–Language Model Family with Multi-Level Clinical Alignment

17 views

Abstract

Photoplethysmography (PPG) is widely recorded by clinical monitors and consumer wearables, providing a scalable source of continuous physiological information. These recordings offer an opportunity for physiological assessment at scale, but realizing this potential requires models to learn from both signal-derived physiological supervision and broader clinical context captured in electronic health records (EHRs). This involves aligning information spanning local observations, care events, and entire visits with PPG representations at corresponding temporal scales. However, existing PPG foundation models primarily rely on task-specific prediction heads, while the medical knowledge of large language models does not necessarily translate into waveform understanding. To bridge this gap, we introduce PPG-LM, the first PPG–language model family to learn physiological representations from both signal-derived supervision and broader clinical context captured in EHRs. To construct clinically grounded captions, we develop an automatic captioning pipeline that generates segment-, event-, and visit-level descriptions from signal measurements and structured EHR records. We then learn from these pairs through a two-stage framework that first establishes segment–language correspondence through contrastive learning and waveform-conditioned captioning, then extends alignment to events and visits through time-aware aggregation and temporal statement matching. Pretrained on approximately 73k hours of PPG, PPG-LM supports language-based recognition, cross-modal retrieval, and segment captioning. Experiments on MC-MED, MIMIC-III, and VitalDB show improved retrieval and caption factuality over language-model baselines and gains over PPG and time-series foundation models on multiple clinical prediction tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Waveform-Language Model, Foundation Models, Multimodal Learning, Healthcare, Physiological Waveforms

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TZSKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors have developed PPG-LM, a photoplethysmography-language model which links clinical descriptions of segments, events, or visits to the physiological waveforms recorded during those periods. PPG-LM uses an automated captioning pipeline based on signal-derived features and structured electronic health record data. The training process for PPG-LM is divided into two phases. The first phase uses contrastive alignment to link waveform segments to their corresponding descriptions, together with waveform-conditioned captioning to generate segment-level descriptions. In the second phase, PPG-LM extends alignment to event- and visit-level clinical descriptions. In addition to these two phases, PPG-LM uses time-aware aggregation to combine individual waveform segment features into representations of events or visits. Finally, PPG-LM matches temporal statements describing visits with the corresponding visit representations. PPG-LM is pretrained using approximately 73,700 hours of PPG recordings from MC-MED. PPG-LM is evaluated on MC-MED, MIMIC-III, and VitalDB. Evaluation of PPG-LM indicates that it outperforms several existing models in terms of language-based recognition of physiology, cross-modal retrieval, caption generation, and frozen-representation clinical prediction.

Critical strengths

A meaningful formulation of multi-level physiological–clinical alignment.

The authors address a key challenge in physiological foundation models: how to relate characteristics of individual waveforms to broader clinical data. Their approach of organizing clinical supervision at three levels (segment, event, visit) is well reasoned; also, the combination of automated caption generation with temporal aggregation provides a logical framework for using diverse sources of clinical data. As such, this represents a valuable extension of current approaches to signal-language representation learning.

Substantial experimental coverage and encouraging transfer performance.

The study evaluated the performance of the model on three clinical datasets, which have different data collection environments and involve recognition, retrieval, captioning, and clinical prediction tasks. It was promising to observe performance improvements on both MIMIC-III and VitalDB, as the model had been pretrained on MC-MED alone. Furthermore, the evaluation involved the use of independent ECG reference signals to assess heart rate, frozen-representation probes, and component analyses; this helped to support the empirical findings.

Critical weaknesses

The methodological contribution made by multi-level alignment has yet to be adequately demonstrated.

The proposed architecture combines multiple existing methods, such as contrastive learning, CoCa-style captioning, attention-based temporal aggregation, and late-interaction matching. While the integration of these methods across clinical temporal levels provides value, the experimental design does not clearly demonstrate how much each method contributes to the overall performance. Specifically, the Stage 1 vs. Stage 2 experiment alters three factors (supervision, training duration, and temporal aggregation architecture) at once. Additionally, the component ablation studies in Table 8 utilize a CLIP-based configuration instead of the primary CoCa model. The lack of direct comparison with CAP, which addresses PPG-EHR clinical alignment, further restricts the ability to evaluate the improvements over similar prior work. These limitations are significant as multi-level clinical alignment represents the core methodological contribution of the manuscript.

The evaluation does not fully demonstrate the generalizability of the learned clinical language representations beyond structured descriptions.

A large amount of the training supervision is created through automated descriptions that come from both physiological measurements and structured clinical data. Several evaluation tasks involve attributes and annotation methods closely related to those used to create the pretraining data. Therefore, the zero-shot results reported here primarily demonstrate how well the model can apply its knowledge of the relationship between physiology and language without task-specific model fine-tuning, although some visit-level tasks use threshold calibration. However, there is less evidence that the model can understand clinical information expressed independently. Because of the use of automatically generated reference descriptions as well as the sensitivity shown to different prompt formulations, there remain unanswered questions about whether the model will be able to generalize beyond the structured description format.

Some qualifications to the baseline comparisons should be stated. The Qwen3-4B supervised fine-tuning baseline uses significantly fewer training windows than PPG-LM, and general-purpose LLMs use downsampled waveforms (represented numerically as text), while PPG-LM uses a dedicated waveform encoder. Although these qualifications do not invalidate the reported findings, they do restrict our ability to make conclusions regarding the comparative performance of the proposed architecture.

Questions for authors

Can the authors perform controlled studies where they hold constant the primary CoCa architecture, the training budget, and the temporal aggregation method and systematically remove event-level and visit-level clinical supervision? This will allow us to assess whether improved performance is due to multi-level alignment or simply additional training and/or increased aggregation capacity.

Can the authors provide a quantitative comparison with CAP using a common evaluation protocol, or explain why they cannot do so due to methodological constraints? Also, the use of a more closely matched signal-language baseline would allow us to see if the disparity in pretraining exposure has an impact.

When PPG-LM is tested against independently written clinical descriptions or clinical attributes that have not been included in the pretraining captions, can we determine whether the model has learned more general clinical concepts beyond those represented in the training data? This testing will allow us to assess whether PPG-LM is capable of understanding clinical semantics at a higher level than just what it was trained on.

Since the waveform encoder standardizes each window of data, and perfusion strength is based on an AC/DC amplitude measurement, could the authors please provide more explanation of how they preserve the necessary information (or indirectly infer it) when they remove the absolute offset and scale information from the data during normalization?

10 comments
AuthorsOpenReview ↗

Response [1/8]

W1-1: The methodological contribution made by multi-level alignment has yet to be adequately demonstrated. The proposed architecture combines multiple existing methods, such as contrastive learning, CoCa-style captioning, attention-based temporal aggregation, and late-interaction matching. While the integration of these methods across clinical temporal levels provides value, the experimental design does not clearly demonstrate how much each method contributes to the overall performance. Specifically, the Stage 1 vs. Stage 2 experiment alters three factors (supervision, training duration, and temporal aggregation architecture) at once. Additionally, the component ablation studies in Table 8 utilize a CLIP-based configuration instead of the primary CoCa model.

Q1: Can the authors perform controlled studies where they hold constant the primary CoCa architecture, the training budget, and the temporal aggregation method and systematically remove event-level and visit-level clinical supervision? This will allow us to assess whether improved performance is due to multi-level alignment or simply additional training and/or increased aggregation capacity.

Thank you for highlighting this experimental confound. To strictly disentangle clinical supervision, training duration, and temporal aggregation on our primary CoCa model (rather than CLIP), we ran one-factor-at-a-time ablations on held-out MC-MED patients from the same Stage-1 CoCa checkpoint and fixed Stage-2 training budget—showing that extra duration and aggregator capacity alone yield no cross-scale gains without both multi-level supervision and physiological time modeling.

  1. Isolating Multi-Level Clinical Supervision: Holding the primary CoCa architecture, aggregators, and Stage-2 budget constant (Table R10), segment captions alone (w/o higher-level losses) leave Event AUROC (0.517), Visit R@1 (0.001), and Temporal Acc. (0.532) at chance, ruling out training duration or aggregator capacity as the source of gains. Removing individual losses selectively collapses only the corresponding scale, whereas dropping Stage-2 segment retention cuts Segment R@1 to 0.200 (0.016 without Stage-1 anchoring) and flattening EHR text onto 30s windows (Flat CoCa 1-seed control, same budget) degrades pure-signal retrieval (0.417 → 0.216).

Table R10: Primary CoCa Supervision Ablations (Fixed Architecture, Aggregators & Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
PPG-LM0.4190.7140.079230.917
w/o higher-level losses (segment only)0.4290.5170.00110540.532
w/o event loss0.4250.4930.082220.915
w/o visit & temporal loss0.4280.7180.00210770.544
w/o temporal loss0.4230.7180.089230.490
w/o segment loss0.2000.7220.083230.921
  1. Isolating Temporal Aggregation on Primary CoCa: Holding all three supervision levels and the training budget fixed on primary CoCa (Table R11), removing signed event offsets drops Event AUROC from 0.714 to 0.619 (intra-visit paired accuracy: 0.708 → 0.589). Likewise, removing visit continuous-time and gap encodings halves Visit R@1 (0.079 → 0.049; Visit MedR: 23 → 40; overall Temporal Acc.: 0.917 → 0.913, with order-sensitive rate-ordering accuracy dropping 0.907 → 0.834).

Table R11: Primary CoCa Temporal Aggregation Ablations (Fixed 3-Level Supervision & Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
PPG-LM0.4190.7140.079230.917
w/o event signed offsets0.4230.6190.087210.913
w/o visit time & gap encoding0.4210.7240.049400.913
AuthorsOpenReview ↗

Response [2/8]

  1. Disentangling Stage 1 vs. Stage 2 Downstream Representations: Probing frozen features without text prompts across 8 MC-MED, 3 MIMIC-III, and 4 VitalDB classification tasks on official splits (Table R12) shows that extra Stage-2 duration and capacity with segment supervision alone yield no gain over Stage 1 on dynamic events (0.613 vs. 0.640 Stage-1 window mean) or external transfer (0.731 vs. 0.733 on MIMIC-III). In contrast, multi-level supervision lifts Event Dynamic Probe AUROC to 0.712 and improves cross-center transfer on MIMIC-III (0.731 → 0.742) and VitalDB (0.702 → 0.708), while under matched 4-window pooling, MC-MED static probes saturate in Stage 1 (0.785 vs. 0.784).

Table R12: Non-Text Frozen Representation & Cross-Center Transfer Probes (AUROC ↑; Bold = Notable Degradation)

Model / VariantEvent Dynamic Probe ↑ (MC-MED)MC-MED Static Probe ↑MIMIC-III Transfer ↑VitalDB Transfer ↑
PPG-LM0.7120.7850.7420.708
w/o higher-level losses (segment only)0.6130.7840.7310.702
w/o event loss0.6150.7840.7350.703
w/o visit & temporal loss0.7140.7840.7400.706
w/o Stage-2 (Stage-1 init)0.6400.7840.7330.701
AuthorsOpenReview ↗

Response [3/8]

W1-2: The lack of direct comparison with CAP, which addresses PPG-EHR clinical alignment, further restricts the ability to evaluate the improvements over similar prior work. These limitations are significant as multi-level clinical alignment represents the core methodological contribution of the manuscript.

Q2: Can the authors provide a quantitative comparison with CAP using a common evaluation protocol, or explain why they cannot do so due to methodological constraints? Also, the use of a more closely matched signal-language baseline would allow us to see if the disparity in pretraining exposure has an impact.

Thank you for raising this important comparison. Because CAP's pretrained weights and processed data are not publicly released, we retrained CAP on MC-MED following its official architecture and training objective (pairing 10 consecutive 30s windows with a single visit summary; 5,000-step pre-divergence checkpoint). CAP's downstream protocol probes only its single-scale PPG encoder without a generative decoder or event/visit temporal aggregators, precluding caption generation and multi-scale event or trajectory evaluation. Under a common evaluation protocol using matched 4-window linear probing across all 32 clinical prediction tasks (17 MC-MED, 6 MIMIC-III, and 9 VitalDB) on patient-disjoint splits (Table R13, matching Tables 7 and 22) and 50-shot adaptation on MC-MED (Table R14), PPG-LM achieves state-of-the-art performance, outperforming CAP across all six dataset AUROC and R² group means and all 32 of 32 clinical prediction tasks (with Stage 1 alone leading CAP on 29 of 32, and Stage 2 improving over Stage 1 on 30 of 32)—as well as across all 50-shot Macro-F1 tasks—while uniquely supporting multi-level clinical language interfaces (for the signal–language pretraining-exposure comparison, please see our response to W3 immediately below).

Table R13: Full Per-Task Clinical Prediction Comparison Between CAP and PPG-LM on Patient-Disjoint Splits (4-Window Protocol; Bold = Best)

Dataset / TaskCAPPPG-LM (Stage 1)PPG-LM (Stage 2)
MC-MED (ED) — AUROC ↑
Sex0.7690.8130.856
ED admission0.7290.7650.787
Atrial fibrillation0.8250.8570.870
Heart failure0.7810.8360.860
Cardiac device0.7710.8360.857
Chronic kidney disease0.7370.7740.792
Diabetes0.7020.7280.740
Anemia0.6690.6860.709
Group Mean (n = 8)0.7480.7870.809
MC-MED (ED) — R² ↑
Age0.5880.6910.782
eGFR0.2760.3560.404
Hemoglobin0.1730.2380.274
Albumin0.1970.2390.286
BUN0.1290.1780.213
Potassium0.0380.0670.086
Calcium0.0350.0510.076
Sodium0.0500.0680.091
Glucose0.0790.0710.083
Group Mean (n = 9)0.1740.2180.255
MIMIC-III (ICU) — AUROC ↑
Sex0.7060.7300.733
In-hospital mortality0.7430.7860.793
Circulatory diagnosis0.6860.7200.726
Group Mean (n = 3)0.7120.7450.751
MIMIC-III (ICU) — R² ↑
Age0.3940.5020.507
BUN0.0830.0810.086
Hemoglobin0.0400.0920.095
Group Mean (n = 3)0.1720.2250.229
VitalDB (OR) — AUROC ↑
Sex0.6350.6840.691
ASA ≥ 30.6900.6990.704
Hypertension0.6300.6560.655
ICU admission0.7010.7330.751
Group Mean (n = 4)0.6640.6930.700
VitalDB (OR) — R² ↑
Age0.3650.4180.435
BMI0.0420.0410.043
Albumin0.0950.1340.138
EtCO₂ (window)0.1230.1530.152
BIS (window)0.3800.4070.408
Group Mean (n = 5)0.2010.2310.235

Table R14: Few-Shot Adaptation Comparison on MC-MED (k = 50 Support Examples per Class, Macro-F1 ↑; Bold = Best)

ModelRhythm ↑HR Class ↑Sex ↑
CAP0.5070.8730.602
PPG-LM0.5970.9210.710
Reviewer_8JRrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PPG-LM, a framework for aligning photoplethysmography (PPG) signals with clinical language at three temporal levels: individual waveform segments, medication events, and clinical visits. The authors generate captions from signal-derived physiological measurements and structured electronic health records, then train the model in two stages. The first stage learns segment-level signal–text correspondence using contrastive learning and waveform-conditioned captioning. The second stage incorporates time-aware aggregation and alignment objectives for event- and visit-level representations. The model is pretrained on approximately 73,700 hours of PPG recordings from MC-MED and evaluated on MC-MED, MIMIC-III, and VitalDB. The evaluation covers physiological recognition and estimation, cross-modal retrieval, caption generation, medication-event recognition, and downstream clinical prediction using frozen representations.

Critical strengths
  1. A relevant problem beyond task-specific PPG prediction. Linking waveform segments to clinical care events and descriptions of patient encounters is a worthwhile direction. A shared interface between physiological signals and language could make physiological recordings more useful for retrieval and clinical data exploration, rather than limiting representations to predefined prediction tasks.
  2. Broad and encouraging empirical evaluation. The experiments cover emergency, intensive care, and surgical datasets, including evaluations on external datasets. Improvements in segment-level recognition and clinical prediction using frozen representations are relatively consistent. In particular, the clinical prediction results in Table 7 suggest that the learned representations contain useful information beyond the immediate captioning task.
Critical weaknesses
  1. The methodological contribution is difficult to separate from established components and additional supervision. The framework combines contrastive learning and caption generation following CoCa, temporal aggregation, and a late interaction matching objective. These choices are reasonable, but the paper does not clearly establish which aspect of their combination constitutes a distinct methodological advance. More importantly, the transition from Stage 1 to Stage 2 introduces changes in both the available supervision and the aggregation mechanism. Although the ablations in Table 8 examine individual losses and temporal features, they do not include a simpler baseline trained with the same supervision to isolate the contribution of the proposed alignment design. As a result, it remains difficult to determine whether the improvements primarily come from the proposed method or from the richer supervision provided by the constructed training corpus.
  2. The evaluation provides limited evidence for broader clinical language understanding. The training descriptions are largely generated from predefined waveform measurements and structured EHR fields using fixed templates. Several recognition and caption factuality targets are closely related to these same attributes. While the results demonstrate that the model can recover structured physiological information, they provide less convincing evidence of general clinical language understanding. The results at higher levels are also relatively modest. Table 6 reports visit level balanced accuracy between 0.520 and 0.570. In addition, Table 18 shows a substantial decline in retrieval performance on external datasets when the candidate pool increases to 2,000. For example, the signal to text R@1 on VitalDB drops to 0.108. These observations suggest that the paper could benefit from more carefully qualified claims and additional evaluations using independently written clinical descriptions or more challenging semantic comparisons.
Questions for authors

Q1. Is the improvement due to temporal alignment or additional supervision? Could the authors compare Stage 2 with a simpler event/visit aggregation baseline trained on the same segment-, event-, and visit-level captions and with a comparable training budget? This would materially clarify the claimed methodological contribution. Q2. Does the learned alignment generalize beyond the caption-generation rules? Have the authors tested independently written clinical descriptions or retrieval candidates with closely matched physiological attributes? Such evidence would help distinguish recovery of predefined caption attributes from more general signal–language correspondence. Q3. What waveform information supports the perfusion-related prediction after normalization? Appendix C.1 states that each input window is standardized using its own mean and standard deviation, while Appendix B.1 describes perfusion using pulsatile amplitude relative to the baseline. Since window-wise standardization removes absolute mean and scale, could the authors clarify which preserved signal features support this target, and how the evaluation should be interpreted?

7 comments
AuthorsOpenReview ↗

Response [1/4]

W1: The methodological contribution is difficult to separate from established components and additional supervision. The framework combines contrastive learning and caption generation following CoCa, temporal aggregation, and a late interaction matching objective. These choices are reasonable, but the paper does not clearly establish which aspect of their combination constitutes a distinct methodological advance. More importantly, the transition from Stage 1 to Stage 2 introduces changes in both the available supervision and the aggregation mechanism. Although the ablations in Table 8 examine individual losses and temporal features, they do not include a simpler baseline trained with the same supervision to isolate the contribution of the proposed alignment design. As a result, it remains difficult to determine whether the improvements primarily come from the proposed method or from the richer supervision provided by the constructed training corpus.

Q1: Is the improvement due to temporal alignment or additional supervision? Could the authors compare Stage 2 with a simpler event/visit aggregation baseline trained on the same segment-, event-, and visit-level captions and with a comparable training budget? This would materially clarify the claimed methodological contribution.

Thank you for this thoughtful comment and constructive suggestion. Rather than relying on heavy module capacity, PPG-LM couples scope-matched multi-level supervision with physiological time conditioning in one shared waveform encoder without overwriting local pulse fidelity. Evaluating controlled ablations on the primary CoCa backbone under the same Stage-1 checkpoint and fixed Stage-2 budget on held-out MC-MED patients, we find that multi-level supervision unlocks higher-level interfaces and cross-center transfer, while time conditioning—not aggregator size—grounds them onto clinical and waveform dynamics.

  1. Simpler Aggregation & Matching Baselines: Under identical three-level captions and training budget (Table R4), a half-parameter simpler time-conditioned aggregator and single-vector matching (w/o late interaction) stay within 0.01 of PPG-LM across all four alignment rates—including Event AUROC (0.709 vs. 0.714; with only a modest dynamic-probe gap in Table R6, 0.688 → 0.712)—and match Visit MedR (21–22 vs. 23). Conversely, removing signed event offsets (Event AUROC: 0.714 → 0.619) or visit time/clock and gap encodings (Visit R@1: 0.079 → 0.049; Visit MedR: 23 → 40; and while overall Temporal Acc. stays 0.913 on order-invariant statements, order-sensitive rate-ordering accuracy drops 0.907 → 0.834) sharply degrades the corresponding scale.

Table R4: Simpler Aggregation & Temporal Design Baselines (Same Stage-1 Init & Fixed Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
PPG-LM0.4190.7140.079230.917
Simpler time-conditioned aggregator (half params)0.4210.7090.085210.914
w/o late interaction (single global latent)0.4220.7140.084220.910
w/o event signed offsets (unsigned pooling)0.4230.6190.087210.913
w/o visit time & gap encoding0.4210.7240.049400.913
  1. Multi-Level Supervision Ablations: Holding the full architecture and budget fixed (Table R5), segment captions alone (w/o higher-level losses) leave Event AUROC (0.517), Visit R@1 (0.001), and Temporal Acc. (0.532) at chance, and dropping individual losses collapses only the corresponding level. Meanwhile, removing Stage-2 segment retention cuts Segment R@1 to 0.200 (0.016 without Stage-1 anchoring), whereas flattening EHR text onto 30s windows (Flat CoCa 1-seed control, same budget) degrades pure-signal retrieval (Segment R@1: 0.417 → 0.216).

Table R5: Multi-Level Supervision Ablations (Fixed Architecture & Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
Chance< 0.0010.5000.0005~10000.500
PPG-LM0.4190.7140.079230.917
w/o higher-level losses (segment only)0.4290.5170.00110540.532
w/o event loss0.4250.4930.082220.915
w/o visit & temporal loss0.4280.7180.00210770.544
w/o temporal loss0.4230.7180.089230.490
w/o segment loss0.2000.7220.083230.921
AuthorsOpenReview ↗

Response [2/4]

  1. Non-Text Downstream Representations: Probing frozen features across 8 MC-MED, 3 MIMIC-III, and 4 VitalDB classification tasks on official splits (Table R6) shows that event supervision drives the main Event Dynamic Probe AUROC gain from 0.613 (segment only) and 0.640 (Stage-1 window mean) to 0.688 (simpler aggregator) and 0.712 (PPG-LM). Under matched 4-window pooling, in-domain MC-MED saturates in Stage 1 (0.785 vs. 0.784), whereas multi-level supervision improves cross-center transfer on MIMIC-III (0.731 → 0.742) and VitalDB (0.702 → 0.708).

Table R6: Non-Text Frozen Representation & Cross-Center Transfer Probes (AUROC ↑; Bold = Notable Degradation)

Model / VariantEvent Dynamic Probe ↑ (MC-MED)MC-MED Static Probe ↑MIMIC-III Transfer ↑VitalDB Transfer ↑
PPG-LM0.7120.7850.7420.708
Simpler time-conditioned aggregator (half params)0.6880.7850.7410.707
w/o higher-level losses (segment only)0.6130.7840.7310.702
w/o event loss0.6150.7840.7350.703
w/o visit & temporal loss0.7140.7840.7400.706
w/o Stage-2 (Stage-1 init)0.6400.7840.7330.701

W2: The evaluation provides limited evidence for broader clinical language understanding. The training descriptions are largely generated from predefined waveform measurements and structured EHR fields using fixed templates. Several recognition and caption factuality targets are closely related to these same attributes. While the results demonstrate that the model can recover structured physiological information, they provide less convincing evidence of general clinical language understanding. The results at higher levels are also relatively modest. Table 6 reports visit level balanced accuracy between 0.520 and 0.570. In addition, Table 18 shows a substantial decline in retrieval performance on external datasets when the candidate pool increases to 2,000. For example, the signal to text R@1 on VitalDB drops to 0.108. These observations suggest that the paper could benefit from more carefully qualified claims and additional evaluations using independently written clinical descriptions or more challenging semantic comparisons.

Q2: Does the learned alignment generalize beyond the caption-generation rules? Have the authors tested independently written clinical descriptions or retrieval candidates with closely matched physiological attributes? Such evidence would help distinguish recovery of predefined caption attributes from more general signal–language correspondence.

Thank you for this thoughtful comment and constructive suggestion. First, rather than using a single rigid string, our pretraining pipeline follows SensorLM and SleepLM by pairing 10 diverse templates per physiological attribute with randomized selection, sentence-order shuffling, and missing-field omission—preserving rich compositional stochasticity while ensuring hallucination-free grounding—and we evaluate template-free clinical descriptions manually verified by humans and attribute-matched hard negatives below to confirm generalization beyond template rules (Table R7). Second, the modest scores in Table 6 arise because chronic outpatient medications take time to manifest hemodynamically and peripheral PPG captures only partial cardiovascular signatures—which is why all baselines in Table 6 hover near chance (0.421–0.540) while PPG-LM still leads 7 of 8 tasks—and we confirm this by benchmarking against supervised 4-window probes across all 8 encounter EHR categories (Table R8), while VitalDB comorbidities additionally reflect cross-center label shift. Third, we clarify below why strict exact-index R@1 drops when scaling zero-shot external retrieval pools to 2,000 candidates in Table 18 despite strong top-k ranking, and we will qualify our large-pool retrieval discussion accordingly.

  1. Out-of-Template Generalization & Hard Negatives: Across 900 patient-disjoint windows, our deterministic signal/EHR pipeline extracts ground-truth physiological fact sheets from which clinical descriptions are synthesized strictly without template access and manually verified by humans for factual fidelity (Table R7). Frozen PPG-LM retains 76%–90% of template S2T R@1 across clinical charting and narrative prose, achieves 0.763–0.947 accuracy on attribute-matched hard negatives and 0.690–0.778 on single-attribute minimal pairs, and generalizes to events (AUROC 0.685 vs. 0.713) and visits (200-way narrative R@1 0.205, MedR 5). While qualitative rounding (100-way R@1 0.210–0.440) and unseen clinical shorthand reduce retrieval, fixed inference-time abbreviation expansion lifts shorthand R@1 by 3×–5×, and paraphrase-diversified training resolves quality contrast (0.445–0.525 → 0.688–0.721), leaving pulse morphology (0.535) as the main boundary.
AuthorsOpenReview ↗

Response [3/4]

Table R7: Out-of-Template Retrieval & Hard-Negative Evaluation (Frozen PPG-LM, 300 Windows / Center)

Evaluation ProtocolMC-MEDMIMIC-IIIVitalDB
Template reference (100-way S2T R@1 ↑, chance 0.010)0.9730.8470.583
Clinical charting note (template-free, 100-way R@1 ↑)0.7400.6870.500
Clinical narrative prose (template-free, 100-way R@1 ↑)0.7870.7230.527
Attribute-matched hard negatives (charting / narrative Acc. ↑, chance 0.200)0.917 / 0.9470.870 / 0.9030.777 / 0.763
Single-attribute minimal-pair contrast (2-way Acc. ↑, chance 0.500)0.7410.7780.690
Unseen clinical shorthand: raw → dict. expanded (100-way R@1 ↑)0.107 → 0.3200.050 → 0.2330.060 → 0.293
  1. Visit Fact Readout vs. Supervised 4-Window Probes: Benchmarking pretrained visit-fact readout against supervised 4-window logistic probes across all 8 EHR categories on the MC-MED test split (Table R8) shows inherently weak resting-PPG signal on home medications (0.566 AUROC) versus physiological traits (0.664–0.849). Our readout matches supervised probing across all 8 categories without task fitting (0.580 on medications; 0.669–0.850 on physiological traits), while VitalDB comorbidities trail probing under cross-center label shift (0.530–0.590 vs. 0.580–0.697).

Table R8: Supervised 4-Window Probes vs. Pretrained Visit Fact Readout (MC-MED Test Split, AUROC ↑)

Clinical EHR Fact Category (# Facts)Supervised 4-Window Probe ↑PPG-LM Fact Readout (No Task Fitting) ↑
Age group (5) / Sex (2)0.849 / 0.8120.850 / 0.813
Triage vitals (16) / Medical history (24)0.760 / 0.6840.764 / 0.696
ESI acuity (4) / Arrival mode (2) / Chief complaints (10)0.664 / 0.680 / 0.6700.681 / 0.672 / 0.669
Outpatient home medications (16 across 6 classes)0.566 (0.532–0.621)0.580 (0.539–0.652)
  1. Clarification on 2,000-Pool External Retrieval (Table 18): Scaling the candidate pool 20× from N = 100 to 2,000 lowers chance R@1 to 0.0005 (R@10: 0.005); even at N = 2,000, zero-shot PPG-LM in Table 18 exceeds chance by >215× on VitalDB (S2T R@1: 0.108; R@5: 0.344; R@10: 0.510) and >500× on MIMIC-III (S2T R@1: 0.258; R@5: 0.671; R@10: 0.853). Relative to in-domain MC-MED (R@1: 0.780), strict exact-index R@1 drops externally for three compounding reasons: (i) many-to-one state collisions under instance-level grading—30s captions describe physiological state rather than unique patient identity, so across 2,000 windows many distinct candidates share nearly or completely identical physiological descriptions, yet exact-index R@1 counts retrieving any clinically equivalent non-self window as an error (consistent with the sharp recovery by R@5 and R@10); (ii) fewer external fields and narrower steady-state ranges—MIMIC-III and VitalDB provide 6 common caption fields (vs. 8 on MC-MED, lacking perfusion and morphology), while ICU sedation, mechanical ventilation, general anesthesia, and vasoactive titration clamp heart rate, respiratory rate, and rhythm into tight steady-state ranges that amplify candidate overlap; and (iii) zero-shot domain shift from awake emergency-department PPG (MC-MED) to ICU and anesthetized operating-room monitors, where anesthesia and vasoactive drugs substantially alter peripheral pulse morphology.
Reviewer_WBMUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a family of PPG-language models that align PPG signals with clinical information at segment, event, and visit levels. The authors develop an automatic captioning pipeline using signal measurements and EHR data, followed by a two-stage alignment framework. Experiments on three clinical datasets show improvements in language-based tasks and downstream clinical prediction compared to existing baselines.

Critical strengths
  1. The proposed multi-level alignment framework is well motivated, and the ablation results support the effectiveness of its temporal components.
    1. The paper provides a comprehensive evaluation across three clinical datasets, covering both language-based tasks and downstream clinical prediction. The results show promising performance compared to existing baselines, including PPG foundation models.
Critical weaknesses
  1. My major concern is that the framework integrates several established approaches. Segment-level training follows CoCa, combining contrastive signal–text alignment with waveform-conditioned captioning. Event aggregation adds signed temporal embeddings to attention pooling, while visit aggregation builds on Perceiver-style learned queries and continuous-time RoPE. Temporal statement matching adopts ColBERT-style late interaction, matching each statement to its most compatible visit latent. The main contribution appears to lie in organizing these components into a segment–event–visit PPG–language framework.
  2. The contributions of multi-level clinical supervision and temporal aggregation to downstream performance are not clearly disentangled. Since multi-level clinical alignment is a central contribution of the paper, this makes it difficult to determine how much of the reported improvement can be attributed to the proposed clinical supervision itself.
Questions for authors

Could the authors provide a more controlled ablation to isolate the benefits of event- and visit-level clinical supervision from those of temporal aggregation? For example, how would a model using the same architecture and training budget perform with segment-level supervision alone versus supervision at all three levels?

5 comments
AuthorsOpenReview ↗

Response [1/3]

W1: My major concern is that the framework integrates several established approaches. Segment-level training follows CoCa, combining contrastive signal–text alignment with waveform-conditioned captioning. Event aggregation adds signed temporal embeddings to attention pooling, while visit aggregation builds on Perceiver-style learned queries and continuous-time RoPE. Temporal statement matching adopts ColBERT-style late interaction, matching each statement to its most compatible visit latent. The main contribution appears to lie in organizing these components into a segment–event–visit PPG–language framework.

Thank you for this thoughtful comment. We would like to clarify that rather than a plug-and-play assembly of standalone modules, PPG-LM introduces a coupled cross-scale supervision and temporal alignment design for a setting where standard vision–language and retrieval primitives break down off the shelf: jointly grounding continuous sub-second waveforms and asynchronous, multi-horizon EHRs within one shared encoder. Whereas CoCa, Perceiver, RoPE, and ColBERT assume pre-paired inputs at a single granularity over discrete sequences, bedside monitoring pairs continuous physiological streams with heterogeneous clinical records spanning seconds to hours under frequent state collisions and irregular sampling. Bridging this gap requires two non-incremental designs (whose empirical necessity we isolate with controlled experiments in W2 & Q1 below):

  • Why Multi-Level Clinical Alignment Is Necessary (Scope-Matched Supervision & Two-Stage Grounding): Raw monitors and EHRs do not provide aligned waveform–text pairs; clinical information spans distinct evidential scopes—beat-level morphology (segment), acute intervention responses (event), and longitudinal encounter context (visit)—with frequent identical-state collisions across patients. Flattening encounter-level EHR text onto short PPG windows couples alignment to static EHR shortcuts (degrading standalone signal retrieval), whereas dropping segment supervision in Stage 2 degrades local grounding. We therefore pair scope-matched clinical descriptions with collision-aware contrastive masking in a two-stage framework: Stage 1 grounds local physiological primitives, and Stage 2 extends the shared encoder to events and visits while retaining segment alignment and anchoring so coarse EHR context does not degrade local signal–language grounding.
  • Why Clinical Dynamics Go Beyond Standard Pooling (Directional Events & Irregular Visits): Standard attention pooling and discrete positional encodings treat inputs as uniform or order-invariant sequences, obscuring clinical dynamics. Over the shared window representations, we introduce domain-specific temporal modeling tailored to bedside physiology: (i) signed anchor-relative time offsets paired with intra-visit control sampling to separate pre- vs. post-medication hemodynamic shifts; (ii) continuous-time rotary encoding with clinical arrival/monitoring clocks and explicit gap/coverage tokens to model discontinuous, dropout-prone multi-hour visits without imputing missing signals; and (iii) counterfactual temporal statement matching (ranking true trajectories against single-statement corruptions) to verify genuine state progression, duration, and rate ordering rather than shortcutting on static patient traits.
AuthorsOpenReview ↗

Response [2/3]

W2: The contributions of multi-level clinical supervision and temporal aggregation to downstream performance are not clearly disentangled. Since multi-level clinical alignment is a central contribution of the paper, this makes it difficult to determine how much of the reported improvement can be attributed to the proposed clinical supervision itself.

Q1: Could the authors provide a more controlled ablation to isolate the benefits of event- and visit-level clinical supervision from those of temporal aggregation? For example, how would a model using the same architecture and training budget perform with segment-level supervision alone versus supervision at all three levels?

Thank you for this constructive suggestion. To cleanly disentangle multi-level clinical supervision from temporal aggregation, we ran controlled ablations on the primary CoCa backbone using the same Stage-1 checkpoint and fixed Stage-2 training budget on held-out MC-MED patients. Across three controlled settings below, we find that multi-level clinical supervision is essential to unlock higher-level clinical interfaces and improve dynamic and cross-center representations, while directional and continuous-time aggregation is required to ground those targets onto clinical and physiological dynamics.

  1. Isolating Multi-Level Clinical Supervision: Holding the full architecture and training budget fixed (Table R1), training with segment captions alone (w/o higher-level losses) leaves Event AUROC (0.517), Visit R@1 (0.001), and Temporal Acc. (0.532) at chance, confirming that extra training steps and aggregator capacity alone cannot induce cross-scale alignment. Removing individual losses selectively collapses only the corresponding level, whereas removing Stage-2 segment retention cuts Segment R@1 from 0.419 to 0.200 (and 0.016 without Stage-1 anchoring). Conversely, flattening encounter EHR text directly onto 30s windows (Flat CoCa 1-seed control, same budget) degrades pure-signal retrieval (Segment R@1: 0.417 → 0.216).

Table R1: Multi-Level Supervision Ablations (Fixed Architecture & Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
Chance< 0.0010.5000.0005~10000.500
PPG-LM0.4190.7140.079230.917
w/o higher-level losses (segment only)0.4290.5170.00110540.532
w/o event loss0.4250.4930.082220.915
w/o visit & temporal loss0.4280.7180.00210770.544
w/o temporal loss0.4230.7180.089230.490
w/o segment loss0.2000.7220.083230.921
  1. Isolating Temporal Aggregation Designs: Holding all three levels of clinical supervision and the training budget fixed (Table R2), removing signed event offsets drops Event AUROC from 0.714 to 0.619 (intra-visit paired accuracy: 0.708 → 0.589). Similarly, removing visit time and gap encodings cuts Visit R@1 nearly in half (0.079 → 0.049; Visit MedR: 23 → 40; overall Temporal Acc.: 0.917 → 0.913, with order-sensitive rate-ordering accuracy dropping 0.907 → 0.834). Clinical supervision alone is therefore insufficient without directional and continuous-time modeling.

Table R2: Temporal Aggregation Ablations (Same Stage-1 Init & Fixed Budget, MC-MED Test; Bold = Notable Degradation)

Model / VariantSegment R@1 ↑Event AUROC ↑Visit R@1 ↑Visit MedR ↓Temporal Acc. ↑
PPG-LM0.4190.7140.079230.917
w/o event signed offsets0.4230.6190.087210.913
w/o visit time & gap encoding0.4210.7240.049400.913
AuthorsOpenReview ↗

Response [3/3]

  1. Disentangling Non-Text Downstream Representations: To evaluate frozen representations independent of text prompts and visit-length pooling differences, we fit linear probes on dynamic event features and matched 4-window-mean static features across 8 MC-MED, 3 MIMIC-III, and 4 VitalDB classification tasks on official splits (Table R3). Multi-level alignment lifts Event Dynamic Probe AUROC from 0.613 (segment only) and 0.640 (Stage-1 window mean) to 0.712, showing that clinical supervision directly reshapes dynamic intervention representations. Under matched 4-window pooling, in-domain MC-MED static performance already saturates in Stage 1 (0.785 vs. 0.784), whereas multi-level supervision consistently improves cross-center transfer on MIMIC-III (0.731 → 0.742) and VitalDB (0.702 → 0.708).

Table R3: Non-Text Frozen Representation & Cross-Center Transfer Probes (AUROC ↑; Bold = Notable Degradation)

Model / VariantEvent Dynamic Probe ↑ (MC-MED)MC-MED Static Probe ↑MIMIC-III Transfer ↑VitalDB Transfer ↑
PPG-LM0.7120.7850.7420.708
w/o higher-level losses (segment only)0.6130.7840.7310.702
w/o event loss0.6150.7840.7350.703
w/o visit & temporal loss0.7140.7840.7400.706
w/o Stage-2 (Stage-1 init)0.6400.7840.7330.701

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition that our “multi-level alignment framework is well motivated,” that linking PPG segments to clinical events and patient encounters is “a worthwhile direction” with “well reasoned” segment-, event-, and visit-level supervision, and that “the ablation results support the effectiveness of its temporal components.” They also highlighted our “comprehensive evaluation across three clinical datasets” spanning emergency, intensive care, and surgical settings (including independent ECG heart-rate assessment), noting that our frozen representations contain “useful information beyond the immediate captioning task” and achieve “encouraging transfer performance” on MIMIC-III and VitalDB after pretraining on MC-MED alone.

We have addressed each reviewer's concerns individually below and incorporated all new experiments and clarifications into our revised manuscript (highlighted in orange). Below is a summary of the major revisions:

  1. Multi-Level Supervision and Temporal Aggregation Ablations: Holding the primary CoCa backbone and training budget fixed, hierarchical supervision unlocks event, visit, and temporal interfaces, lifts dynamic event probing, and modestly improves external static transfer (while in-domain static probes remain unchanged). Separately, physiological time conditioning—rather than aggregator size—is essential for the respective event and visit/temporal interfaces (Tables R1–R6, R10–R12).

  2. Generalization to Template-Free Clinical Prose and Hard Negatives: Across all three datasets, frozen PPG-LM generalizes well to human-verified template-free clinical charting and narrative prose, attribute-matched hard negatives, and single-attribute minimal pairs across segments, events, and visits. Inference-time abbreviation expansion and paraphrase-diversified training further mitigate sensitivity to unseen clinical shorthand and phrasing (Tables R7, R21).

  3. Comparisons with CAP and Full-Corpus Qwen3-4B (SFT-L): Training CAP on MC-MED and scaling Qwen3-4B (SFT-L) to all 8.84M windows and 42,856 visits, PPG-LM achieves state-of-the-art performance, outperforming CAP across all six clinical prediction group means (all 32/32 tasks) and 50-shot adaptation. Meanwhile, full-corpus Qwen3-4B (SFT-L) narrows single-window gaps over SFT-S yet remains near chance on event and visit tasks (Tables R13–R19).

  4. Alternative Waveform Input Modalities for General-Purpose LLMs: Replacing numerical text serialization with handcrafted signal features or waveform plot images leaves off-the-shelf GPT-5.6-luna similarly weak across captioning, signal quality, and retrieval, confirming the need for an aligned waveform encoder (Table R20).

  5. Threshold-Free Visit Evaluation vs. Supervised Linear Probes: On held-out MC-MED visits, threshold-free AUROC of our pretrained visit-fact readout matches supervised 4-window linear probes across all eight EHR categories, confirming that modest home-medication scores reflect weak resting-PPG signal (Tables R8, R22). On VitalDB comorbidities, cross-center readout trails supervised probing under label-definition shift.

  6. Semantic Scope on Unseen Clinical Outcomes: On clinical outcomes and diagnoses absent from pretraining captions across MIMIC-III, VitalDB, and MC-MED, zero-shot readout remains bounded to grounded physiological states. However, frozen PPG-LM embeddings preserve strong predictive signal unlocked by linear probing and 16- to 64-shot adaptation on MIMIC-III (Table R23).

  7. Standardization Invariance and Pulsatile Amplitude Audit: Auditing per-window z-score normalization confirms strict offset and gain invariance with zero scale leakage. Frozen PPG-LM representations indirectly infer monitor-scale pulsatile amplitude class from scale-invariant waveform morphology and noise characteristics (Tables R9, R24).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.