acceptodds
Under review as a conference paper at ICLR 2027

Learning Transferable Sensor Models via Language-Informed Pretraining

Abstract

Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured from diverse healthcare sensors. However, existing sensor-language models can only process a fixed number of sensor channels at a fixed sampling rate, making them difficult to adapt to new or different sensor configurations. This inflexibility prevents models from generalizing across heterogeneous health sensing ecosystems, where data spans high-frequency wearable streams to sparse, day-scale clinical signals. To bridge this gap, we introduce (ensor anguage-nformed retraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing a flexible patch embedder, SLIP transfers to new sensor configurations at inference without retraining, regardless of their native sampling rate or input length. Across 11 datasets, SLIP demonstrates superior performance in retrieval, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 3.56% improvement over the strongest baseline (SensorLM). Beyond classification, SLIP supports open-vocabulary sensor captioning and question answering without any architectural modifications. Our work offers broad implications for the development of sensor-language models, utilizing cross-modal supervision to ensure more robust generalizability in health sensing applications.

Primary area:
learning on time series and dynamical systems
Keywords:
Representation Learning, Time Series Language Models, Digital Health, Time series, Wearables, Sensors, Chain of Thought

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KdBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing sensor-language models are often constrained by fixed channel configurations, sampling rates, and input lengths, limiting their transfer to heterogeneous sensor data. Moreover, pretraining based on signal forecasting or reconstruction may not capture the semantic information required by downstream tasks. To address these challenges, this paper proposes SLIP, which uses language as a shared supervision signal to learn general representations that accommodate different sensor configurations. Specifically, SLIP incorporates FlexMLP, which dynamically resizes the weight matrices of its patch-embedding MLP to handle sensor inputs with varying sampling rates and sequence lengths. The authors evaluate SLIP on classification, zero-shot sensor-to-text retrieval, caption generation, and question answering, with results demonstrating SLIP’s effectiveness relative to the baselines.

Critical strengths

(1) Effective Method. Enabling LLMs to understand diverse sensor data is an important problem. The framework accommodates heterogeneous channel configurations, while FlexMLP supports flexible patch sizes for inputs with varying sampling rates and sequence lengths. Through extensive pretraining, SLIP aligns sensor and language representations to support classification and sensor-to-text retrieval, as well as question answering after fine-tuning.

(2) Comprehensive Results. The evaluation covers classification, sensor-to-text retrieval, and question answering across multiple sensor domains and datasets, with comparisons against a broad range of relevant baselines. The reported improvements across multiple tasks support the effectiveness and transferability of the learned sensor representations. Ablation studies further demonstrate the complementary roles of contrastive learning and caption generation, providing empirical support for the main design choices.

Critical weaknesses

(1) Motivation. FlexMLP is a key component for handling heterogeneous sensor inputs by resizing MLP weights to accommodate different patch sizes, but its advantages over directly applying adaptive average pooling to the input require further justification. For example, in Algorithm 1 of Figure 2, when base_patch=16 and patch_size=64, one could apply adaptive average pooling along the last dimension of the concatenated z_p, reducing it from 192 to 48 dimensions before passing it to the base MLP without weight resizing. This alternative preserves the token count determined by patch_size=64 while matching the required input dimension. The authors should clarify the key differences between these approaches and provide explicit comparisons with this pooling baseline to establish FlexMLP’s incremental benefits.

(2) Evaluation Reliability. Benchmark evaluations across multiple sensing tasks provide key evidence for the paper’s main contributions, making the reliability and consistency of the reported results particularly important. However, the main results differ substantially from the results across multiple random seeds in the appendix: SLIP’s zero-shot accuracy on WESAD is 56.95% in Table 3 versus 40.45 ± 0.87% in Table 13, while Sundial’s linear probing accuracy on WISDM is 81.25% in Table 2 versus 41.39 ± 0.12% in Table 13. These discrepancies far exceed the reported standard deviations, yet the paper does not clarify whether the tables use different experimental settings, checkpoints, or aggregation procedures. The authors should explain these discrepancies, specify the experimental settings underlying each table, and clarify whether they affect the paper’s main conclusions.

Questions for authors

Please refer to the critical weakness.

3 comments
AuthorsOpenReview ↗

Response to reviewer KdBY

We thank the reviewer for recognizing the contribution of our language-aligned sensor encoder. The suggested ablation and the careful reading of our supplementary materials further strengthen our work, and we address each point below.

W1 FlexMLP Ablation. Our contribution lies in FlexMLP enabling unified sensor pretraining regardless of the specific choice of patch-embedding operator. Intuitively, FlexMLP resizes the weights according to the input, whereas adaptive pooling resizes the patchified input according to the fixed weights. However, we agree with the reviewer that adaptive pooling is also a viable alternative approach. We have included this ablation in our Table 5b) in our revised manuscript. The results have shown that FlexMLP outperforms Adaptive Pooling by +2.41% in Linear Probing evaluation, +9.15% in zero-shot evaluation, and +1.42% in QA. Because the AdaptivePooling is essentially similar to resampling the input signal (or patches) that will cause information loss that we showed in our controlled case study (Appendix A.5), which further strengthen our motivation of using FlexMLP.

Table 5(b): FlexMLP vs. AdaPool (average accuracy over all datasets; Δ relative to FlexMLP)

Patch embeddingSensor PerceptionRetrievalQA
FlexMLP (ours)77.1439.3668.87
AdaPool74.73 (−2.41)30.21 (−9.15)67.45 (−1.42)

W2 Random-Seed Table. We thank the reviewer for catching this inconsistency. The error is in Table 13, not in Tables 2 and 3. We mistakenly ran the multi-seed experiments in Table 13 in a different conda environment from the one used for all other results in the paper, and this environment was not compatible with our codebase, which introduced an unexpected bug. We have rerun all multi-seed experiments in Table 13 in the same environment as the other results, and we will update the table in the revised version. We have also rechecked our codebase and all reported results, and we found no other inconsistencies. We have updated the manuscript with the corrected Table 13, since the significant test was calculated from the original correct Table 2 and Table 3, our main argument and conclusion will remain the same. We now also provide our code to support reproduce all results at: https://anonymous.4open.science/r/SLIP_ICLR2027-62CC/

We thank the reviewer for the careful attention to implementation details. The corrected multi-seed results are consistent with the single-seed results and confirm that our main conclusions hold. Together with the additional ablation study on FlexMLP, we hope these clarifications address the reviewer’s concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_KdBYOpenReview ↗

Reply to Authors Rebuttal

Thank you for the detailed rebuttal and additional experiments.

For my first concern, the comparison with Adaptive Pooling addresses it and demonstrates the effectiveness of FlexMLP, particularly for zero-shot retrieval. However, the improvements in linear probing and QA remain relatively modest. Although App. A.5 shows that aggressive downsampling can lead to information loss, this is different from Adaptive Pooling. AdaPool first extracts patches from the signal at its original sampling rate and then applies pooling within each patch. Although pooling may also discard high-frequency information, FlexMLP similarly compresses longer patches into fixed-dimensional embeddings, which may also result in information loss. Therefore, the analysis in App. A.5 does not directly establish the advantages of weight resizing over Adaptive Pooling.

For my second concern, I also appreciate the explanation of the inconsistent multi-seed results in Tab. 13.

Overall, while the rebuttal partly addresses my original concerns, I remain concern about the limited methodological novelty. Although FlexMLP shows a notable improvement in zero-shot retrieval, its gains over the relatively simple Adaptive Pooling baseline remain modest in linear probing and QA, making its broader benefits less convincing. Moreover, the evaluation focuses on non-visual sensor data. It would be valuable to further investigate whether FlexMLP offers similar advantages over Adaptive Pooling on more common modalities, such as video, using vision-language benchmarks. This evidence could further establish the significance and general applicability of the proposed design.

Considering the paper's motivation, methodological novelty, and experimental evidence, I am inclined to maintain my original rating.

Best,

Reviewer KdBY

AuthorsOpenReview ↗

Response to reviewer KdBY

We thank the reviewer for the discussion. First, we note that FlexMLP is clearly more effective than AdaPool on zero-shot retrieval, our most challenging setting. Moreover, it consistently outperforms AdaPool on 10 of 11 perception benchmarks and 3 of 4 QA benchmarks, which is not a modest gain. We also want to point out that AdaPool does not take in the signal at its original sampling rate, because it pools each patch to a fixed length (e.g., from 64 samples to 16) before the embedding, which is mathematically identical to resampling the signal and then patchifying. In contrast, FlexMLP resizes the embedding weights to the patch length, so it operates on every sample at the original sampling rate.

Beyond this, we would like to bring the discussion back from implementation details to the paper's main contribution, which extends far beyond FlexMLP to a framework that no prior model offers. SLIP is a sensor time-series paper, and evaluation on video-language benchmarks would not strengthen our argument in any sense and falls outside the scope of this paper. We therefore remind the reviewer to assess the paper from a broader perspective, including its position within the related work, rather than focusing on a single design choice within it.

Reviewer_HWwJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLIP is a sensor-language pretraining framework for heterogeneous sensor signals. It combines flexible patch embedding with a pretrained language model and joint contrastive and captioning objectives. Evaluation includes linear probing and zero-shot label-text matching on 11 tasks, plus supervised adaptation for captioning and question answering. The main tables report 77.14% average linear-probing accuracy, 3.56 percentage points above the SensorLM reimplementation. Component ablations, an equal-token downsampling comparison, and five-seed results are also provided.

Critical strengths

The framework addresses a practical obstacle to sensor-language pretraining: accommodating different input configurations within a common model. Flexible patch embedding, cross-channel modeling, and language-conditioned generation form a coherent system. Evaluation across several sensing domains, with downstream datasets excluded from pretraining as reported, supports its potential relevance. Several controlled comparisons support the design. SensorLM is reimplemented on the same pretraining corpus, and the main instruction-following baselines share the Gemma-3-270M backbone. Training-objective and component ablations provide useful evidence, while Appendix A.5 directly compares flexible patching with downsampling at matched token counts.

Critical weaknesses
  1. The main results and repeated-run results contain substantial unexplained discrepancies. Sundial achieves 81.25 on WISDM in Table 2 but 41.39 ± 0.12 in Table 13. SLIP's zero-shot WESAD result changes from 56.95 in Table 3 to 40.45 ± 0.87 in Table 13. These differences greatly exceed the reported standard deviations. Table 13 still indicates average gains, but the inconsistent reporting prevents a reliable assessment of their magnitude and statistical support.
  2. The central input-adaptation mechanism and evaluation protocol are insufficiently specified. Figure 2 and Appendix A.3.2 do not define the exact resizing operator, time-index normalization, or dataset-specific patch-selection rules. Appendix A.5 provides a useful control, but its clearest benefit occurs under strong compression on one dataset. The no-retraining contribution statement also needs to distinguish fixed-encoder input adaptation from the linear-head training and generative finetuning explicitly described in Section 4.
Questions for authors
  1. What explains the discrepancies between Tables 2-3 and Table 13? Which checkpoints, configurations, and sources of seed variation were used, and do the headline gains and statistical conclusions hold under a consistent protocol?
  2. What exact weight-resizing operator is used, and how are time_index values defined and normalized across sampling rates?
  3. How were downstream patch sizes selected, including any use of validation labels? Was the linear probe refitted for each Appendix A.5 setting, or was the same predictor retained?
  4. What are the subject, recording, or window splitting rules, and how were evaluation data excluded from the aggregated pretraining sources? Please clarify the treatment of the three PPG tasks derived from the same source.
1 comment
AuthorsOpenReview ↗

Response to Reviewer HWwJ

We thank the reviewer for the careful reading of our work and our supplementary results. Our consolidated experimental results confirm that the main conclusions hold. We would like to answer the reviewer’s questions below.

W1&Q1 Random-Seed Table. We thank the reviewer for catching this inconsistency. The error is in Table 13, not in Tables 2 and 3. We mistakenly ran the multi-seed experiments in Table 13 in a different conda environment from the one used for all other results in the paper, and this environment was not compatible with our codebase, which introduced an unexpected bug. We have rerun all multi-seed experiments in Table 13 in the same environment as the other results, and we will update the table in the revised version. We have also rechecked our codebase and all reported results, and we found no other inconsistencies. We have updated the manuscript with the corrected Table 13, since the significant test was calculated from the original correct Table 2 and Table 3, our main argument and conclusion will remain the same. We now also provide our code to support reproduce all results at: https://anonymous.4open.science/r/SLIP_ICLR2027-62CC/

W2&Q2 Implementation details. We clarify the detailed adaptation mechanism as follows: 1)Resize operator: FlexMLP stores its linear projection weight at a base patch size of 16 (16 × d, where d is the hidden dimension). For a runtime patch size P during adaptation phase, it resizes this weight to P × d by 2D interpolation. 2) time-index normalization. Following Chronos-2, the time index is fixed rather than learned: t_j = j / L, where j is the position of the time step and L is the window length. 3)dataset-specific patch technique. Each dataset uses one fixed patch size, chosen from the candidate list for its sampling-frequency class (Appendix A.3.2) so that the input stays within 384 tokens for efficiency. These details will also be visible in our anonymized codebase provided above.

Appendix A.5 is a motivation study with well defined controlled, different dataset will have different interest of sampling rate.

We will clarify that the "no retraining" claim is supported by our extensive linear probing and zero-shot retrieval experiments, where the sensor encoder stays frozen throughout.

Q3 Patch Size. The patch size was chosen heuristically according to Appendix A.3.2 with no validation labels involved. Appendix A.5 is meant to show that FlexMLP is a better way to shorten long inputs than naive resampling. For a fair comparison, we fit a separate linear probe for each setting in A.5 rather than reusing one predictor. These details can also be found in our codebase provided above.

Q4 Data Splitting. All data splits follow the official splits of the original source datasets, and there is no data leakage between pretraining and evaluation. The pretraining corpus and the downstream benchmarks come from different datasets. The only shared source is Beijing AQI, which accounts for only 0.22% of the 1.7B time steps in the pretraining corpus, so its influence on the downstream results is negligible. Following NormWear (ACM Health 2025), we treat stroke, diabetes and four-stage hypertension as three separate tasks from one multi-task PPG dataset that labels each recording with several clinical outcomes. We have added these clarifications in section A1.

We thank the reviewer for the careful attention to implementation details. The corrected multi-seed results are consistent with the single-seed results and confirm that our main conclusions hold. To further assure the reviewer that our experiments are faithful and fair, we have provided our complete codebase along with detailed explanation. We hope these clarifications address the reviewer’s concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_bzE3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

SLIP trains a sensor-language model using contrastive alignment and sensor-conditioned captioning on a heterogeneous paired corpus. It combines a shared transformer sensor encoder with a patch MLP whose weights resize for different input lengths, and partitions a pretrained LM into text and multimodal decoding stages. The paper reports 77.14% average linear-probe accuracy over 11 tasks, 39.36% zero-shot class-text retrieval accuracy, and 68.87% mean QA accuracy after supervised finetuning. Objective and architecture ablations support combining the two losses and flexible tokenization.

Critical strengths
  1. Solving important problems in the sensor space (multi-resolution). Using FlexMLP in the sensor tasks is an interesting idea.
  2. Comphrensive experiments across various domains and datasets.
Critical weaknesses
  1. Limited novelty relative to SensorLM Given that SLIP largely follows SensorLM's hierarchical caption generation and sensor-language alignment paradigm, what is the fundamental methodological novelty beyond flexible patch embedding and incorporating a pretrained language model?

  2. Insufficient evidence of reasoning capabilities While the paper reports improvements on sensor QA benchmarks, these tasks primarily evaluate final-answer accuracy after supervised fine-tuning. It remains unclear whether the proposed architecture (with a reasoning model such as Gemma 3) enables more sophisticated sensor reasoning beyond recognition and language generation.

  3. Limited innovation in caption construction The pretraining captions follow SensorLM's statistical, structural, and semantic hierarchy, with additional paraphrasing through Qwen2. It is unclear whether the proposed data-generation pipeline contributes anything fundamentally new beyond expanding the corpus to additional sensor domains.

Questions for authors

How to demostrate reasoning capabilties of the proposed system? The primary advantages of the proposed system over SensorLM are in two folds 1) the way it handles multi-sequence/sampling-rate, 2) the language bakcbone it uses. The author did great job answering the first question, and would be good to understand how helpful is Gemma 3 model in the system.

1 comment
AuthorsOpenReview ↗

Response to Reviewer bzE3

We thank the reviewer for the positive assessment and the helpful questions. We clarify each point below.

W1 Novelty Concern. SLIP's main methodological novelty lies in being a unified framework that can train on cross-domain sensor-language datasets during contrastive pretraining. In detail, previous methods (e.g., SensorLM, IMU2CLIP, MERL) can only do domain- or modality-specific pretraining, because their sensor encoders cannot handle flexible inputs with different lengths, numbers of channels, and sampling rates (Table 1). In contrast, SLIP utilizes FlexMLP to relax this fundamental constraint, allowing pretraining on a large-scale cross-domain corpus for the first time. This yields a more transferable sensor encoder, which lays the groundwork for sensor-language models.

To show that this transferability comes from our framework rather than from the corpus alone, we compare against SensorLM in a controlled setting. The original SensorLM is pretrained only on Fitbit data and does not generalize to unseen sensor configurations, so we reimplement it and pretrain it on the same cross-domain corpus as SLIP, matching pretraining data and objectives. Even with this upgrade, it falls short of SLIP across every task (Tables 2, 3, and 4). SLIP improves average linear-probing accuracy by 3.56% (77.14% vs. 73.58%), average retrieval accuracy by 8.75% (39.36% vs. 30.61%), and average QA accuracy by 3.88% (68.87% vs. 64.99%). This gap shows the necessity of our framework modifications and the effectiveness of our cross-domain pretraining design.

W2 Reasoning Capabilities. We agree with the reviewer that our evaluation on SensorQA mainly focuses on sensor recognition, since sensor reasoning remains underexplored and the field still lacks benchmarks that evaluate complex reasoning over sensor data. Therefore, SLIP follows the standard benchmark from OpenTSLM [ICML2026] to evaluate reasoning ability of the model. We would like to further clarify that SLIP does not aim to improve the reasoning ability of the language model. Instead, SLIP provides a sensor encoder that learns representations the language model can better understand, which elicits better reasoning and lays a strong foundation for future sensor reasoning models (as shown our Table 4). Same as other LLMs and TSLMs, SLIP’s reasoning capability is elicited through CoT-SFT. We have revised Sec. 5.3 in the manuscript to justify the performance gain in our QA benchmark. We agree that studying the reasoning capability of TSLMs is promising, and we have added this to our future work section.

W3 Caption Generation. We agree that our caption generation follows the SensorLM recipe and do not claim it as a contribution. Our contribution is the resulting dataset, which is, to our knowledge, the first public multi-domain sensor-caption corpus (600K pairs across seven domains), and we will release it with the pipeline as an open foundation for general-purpose sensor-language research.

Q1 Language Backbone. We mainly demonstrate the reasoning capabilities of SLIP through comparisons on the current sensor reasoning benchmarks released by OpenTSLM [ICML2026]. To better ablate the contribution of the pretrained Gemma-3-270M backbone, we add an ablation that replaces it with a randomly initialized Transformer text encoder, as in SensorLM (Table 5e). Replacing the pretrained Gemma-3-270M with a randomly initialized Transformer text encoder (as in SensorLM) during SLIP pretraining lowers zero-shot retrieval accuracy by 9.33% and sensor perception accuracy by 2.08%, supporting our hypothesis that a well-structured language representation helps learn a good sensor encoder. On QA, the performance difference narrows to 0.74%, which we want to clarify that reasoning ability is elicited mainly by SFT on CoT data, as is standard for language models. Gemma-3-270M thus provides the language representation space for pretraining, while reasoning comes from the SFT.

Language ModelSensor Perception (Acc.)Δ\DeltaRetrieval (Acc.)Δ\DeltaQA (Acc.)Δ\Delta
Gemma-3-270M (default)77.1439.3668.87
w/o LM pretraining75.06(−2.08)30.03(−9.33)68.13(−0.74)

In summary, we hope these clarifications show that SLIP offers a principled approach to a fundamental constraint in contrastive sensor-language pretraining, making such pretraining possible for the first time on heterogeneous sensor corpora of any length, sampling rate, and channel layout. We thank the reviewer again for the constructive feedback, and if these clarifications address the concerns, we would be grateful if the reviewer would consider raising the score.

Comments on the paper

2 comments

We have uploaded a revised manuscript that includes all the changes and corrections described in our responses, and we thank all reviewers for their valuable feedback.

AuthorsOpenReview ↗

General Note

We want to thank all reviewers for their time, reading and feedback.

We summarize the main additions and clarification we did in our rebuttal:

Reviewer bzE3: We clarified that SLIP’s novelty is a unified framework whose FlexMLP encoder enables contrastive pretraining on heterogeneous cross-domain sensor corpora for the first time. We also added an ablation showing that the pretrained Gemma-3-270M backbone improves zero-shot retrieval by 9.33%.

Reviewer HWwj: We reran the multi-seed experiments in Table 13 in the same environment as all other results. The corrected results agree with Tables 2 and 3 and support the same conclusions. We also specified the FlexMLP resizing operator, time-index normalization, patch-size selection and data splits, and we released the full codebase for reproducibility.

Reviewer KdBY: We added the adaptive pooling ablation the reviewer suggested (Table 5b). FlexMLP outperforms it by 2.41% in linear probing, 9.15% in zero-shot retrieval and 1.42% in QA. We also reran the multi-seed experiments in Table 13 in the same environment as all other results, and they now agree with Tables 2 and 3.

General note. During the rebuttal we added ablations against adaptive pooling and a randomly initialized text encoder, which further substantiate the contribution of FlexMLP and the pretrained language model. We hope reviewers will consider that SLIP provides a single language-aligned sensor encoder that transfers across heterogeneous domains and can serve as the sensor backbone for future sensor-language models, as shown by its gains on the OpenTSLM reasoning benchmarks when paired with a language model.

We would greatly appreciate it if reviewers could let us know whether our responses address their concerns, and we are happy to provide any further clarification before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.