- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Existing sensor-language models are often constrained by fixed channel configurations, sampling rates, and input lengths, limiting their transfer to heterogeneous sensor data. Moreover, pretraining based on signal forecasting or reconstruction may not capture the semantic information required by downstream tasks. To address these challenges, this paper proposes SLIP, which uses language as a shared supervision signal to learn general representations that accommodate different sensor configurations. Specifically, SLIP incorporates FlexMLP, which dynamically resizes the weight matrices of its patch-embedding MLP to handle sensor inputs with varying sampling rates and sequence lengths. The authors evaluate SLIP on classification, zero-shot sensor-to-text retrieval, caption generation, and question answering, with results demonstrating SLIP’s effectiveness relative to the baselines.
Critical strengths
(1) Effective Method. Enabling LLMs to understand diverse sensor data is an important problem. The framework accommodates heterogeneous channel configurations, while FlexMLP supports flexible patch sizes for inputs with varying sampling rates and sequence lengths. Through extensive pretraining, SLIP aligns sensor and language representations to support classification and sensor-to-text retrieval, as well as question answering after fine-tuning.
(2) Comprehensive Results. The evaluation covers classification, sensor-to-text retrieval, and question answering across multiple sensor domains and datasets, with comparisons against a broad range of relevant baselines. The reported improvements across multiple tasks support the effectiveness and transferability of the learned sensor representations. Ablation studies further demonstrate the complementary roles of contrastive learning and caption generation, providing empirical support for the main design choices.
Critical weaknesses
(1) Motivation. FlexMLP is a key component for handling heterogeneous sensor inputs by resizing MLP weights to accommodate different patch sizes, but its advantages over directly applying adaptive average pooling to the input require further justification. For example, in Algorithm 1 of Figure 2, when base_patch=16 and patch_size=64, one could apply adaptive average pooling along the last dimension of the concatenated z_p, reducing it from 192 to 48 dimensions before passing it to the base MLP without weight resizing. This alternative preserves the token count determined by patch_size=64 while matching the required input dimension. The authors should clarify the key differences between these approaches and provide explicit comparisons with this pooling baseline to establish FlexMLP’s incremental benefits.
(2) Evaluation Reliability. Benchmark evaluations across multiple sensing tasks provide key evidence for the paper’s main contributions, making the reliability and consistency of the reported results particularly important. However, the main results differ substantially from the results across multiple random seeds in the appendix: SLIP’s zero-shot accuracy on WESAD is 56.95% in Table 3 versus 40.45 ± 0.87% in Table 13, while Sundial’s linear probing accuracy on WISDM is 81.25% in Table 2 versus 41.39 ± 0.12% in Table 13. These discrepancies far exceed the reported standard deviations, yet the paper does not clarify whether the tables use different experimental settings, checkpoints, or aggregation procedures. The authors should explain these discrepancies, specify the experimental settings underlying each table, and clarify whether they affect the paper’s main conclusions.
Questions for authors
Please refer to the critical weakness.
3 comments
Reply to Authors Rebuttal
Thank you for the detailed rebuttal and additional experiments.
For my first concern, the comparison with Adaptive Pooling addresses it and demonstrates the effectiveness of FlexMLP, particularly for zero-shot retrieval. However, the improvements in linear probing and QA remain relatively modest. Although App. A.5 shows that aggressive downsampling can lead to information loss, this is different from Adaptive Pooling. AdaPool first extracts patches from the signal at its original sampling rate and then applies pooling within each patch. Although pooling may also discard high-frequency information, FlexMLP similarly compresses longer patches into fixed-dimensional embeddings, which may also result in information loss. Therefore, the analysis in App. A.5 does not directly establish the advantages of weight resizing over Adaptive Pooling.
For my second concern, I also appreciate the explanation of the inconsistent multi-seed results in Tab. 13.
Overall, while the rebuttal partly addresses my original concerns, I remain concern about the limited methodological novelty. Although FlexMLP shows a notable improvement in zero-shot retrieval, its gains over the relatively simple Adaptive Pooling baseline remain modest in linear probing and QA, making its broader benefits less convincing. Moreover, the evaluation focuses on non-visual sensor data. It would be valuable to further investigate whether FlexMLP offers similar advantages over Adaptive Pooling on more common modalities, such as video, using vision-language benchmarks. This evidence could further establish the significance and general applicability of the proposed design.
Considering the paper's motivation, methodological novelty, and experimental evidence, I am inclined to maintain my original rating.
Best,
Reviewer KdBY
Response to reviewer KdBY
We thank the reviewer for the discussion. First, we note that FlexMLP is clearly more effective than AdaPool on zero-shot retrieval, our most challenging setting. Moreover, it consistently outperforms AdaPool on 10 of 11 perception benchmarks and 3 of 4 QA benchmarks, which is not a modest gain. We also want to point out that AdaPool does not take in the signal at its original sampling rate, because it pools each patch to a fixed length (e.g., from 64 samples to 16) before the embedding, which is mathematically identical to resampling the signal and then patchifying. In contrast, FlexMLP resizes the embedding weights to the patch length, so it operates on every sample at the original sampling rate.
Beyond this, we would like to bring the discussion back from implementation details to the paper's main contribution, which extends far beyond FlexMLP to a framework that no prior model offers. SLIP is a sensor time-series paper, and evaluation on video-language benchmarks would not strengthen our argument in any sense and falls outside the scope of this paper. We therefore remind the reviewer to assess the paper from a broader perspective, including its position within the related work, rather than focusing on a single design choice within it.
Response to reviewer KdBY
We thank the reviewer for recognizing the contribution of our language-aligned sensor encoder. The suggested ablation and the careful reading of our supplementary materials further strengthen our work, and we address each point below.
W1 FlexMLP Ablation. Our contribution lies in FlexMLP enabling unified sensor pretraining regardless of the specific choice of patch-embedding operator. Intuitively, FlexMLP resizes the weights according to the input, whereas adaptive pooling resizes the patchified input according to the fixed weights. However, we agree with the reviewer that adaptive pooling is also a viable alternative approach. We have included this ablation in our Table 5b) in our revised manuscript. The results have shown that FlexMLP outperforms Adaptive Pooling by +2.41% in Linear Probing evaluation, +9.15% in zero-shot evaluation, and +1.42% in QA. Because the AdaptivePooling is essentially similar to resampling the input signal (or patches) that will cause information loss that we showed in our controlled case study (Appendix A.5), which further strengthen our motivation of using FlexMLP.
Table 5(b): FlexMLP vs. AdaPool (average accuracy over all datasets; Δ relative to FlexMLP)
W2 Random-Seed Table. We thank the reviewer for catching this inconsistency. The error is in Table 13, not in Tables 2 and 3. We mistakenly ran the multi-seed experiments in Table 13 in a different conda environment from the one used for all other results in the paper, and this environment was not compatible with our codebase, which introduced an unexpected bug. We have rerun all multi-seed experiments in Table 13 in the same environment as the other results, and we will update the table in the revised version. We have also rechecked our codebase and all reported results, and we found no other inconsistencies. We have updated the manuscript with the corrected Table 13, since the significant test was calculated from the original correct Table 2 and Table 3, our main argument and conclusion will remain the same. We now also provide our code to support reproduce all results at: https://anonymous.4open.science/r/SLIP_ICLR2027-62CC/
We thank the reviewer for the careful attention to implementation details. The corrected multi-seed results are consistent with the single-seed results and confirm that our main conclusions hold. Together with the additional ablation study on FlexMLP, we hope these clarifications address the reviewer’s concerns, and we would be grateful if the reviewer would consider raising the score.