acceptodds
Under review as a conference paper at ICLR 2027

MUST: Multimodal Unified Sensor Tokenization

Abstract

The development of general time-series foundation models is fundamentally constrained by the asynchronous, multi-rate nature of physiological data. Traditional early-fusion architectures attempt multimodal integration by forcing diverse streams onto rigid, synchronous interpolation grids. However, this artificial alignment incurs severe computational overhead, wastes parameter capacity on imputed noise, and causes cross-modal contamination where high-variance signals dominate. To overcome these limitations, we introduce Modality-Specific Early Tokenization coupled with semantic latent predictive alignment. Each sensor stream is processed by an independent patch encoder tailored to its native frequency, structurally isolating noise and bypassing artificial data imputation. By dynamically concatenating only valid active tokens, a shared transformer organically aligns multi-rate sequences with routing compute overhead. Evaluated on large-scale, real-world datasets comprising over 3.7 billion minutes of continuous sensor data from 50,000 users across nine modalities and 15 distinct activity classes, our framework consistently outperforms generic tokenization baselines across micro- and macro-level tasks. For fine-grained kinematics, it achieves a relative 18.2% Macro F1 score improvement on complex, long-tail activities. Furthermore, within the Digital Well Being (DWB) dataset, our architecture drives significant improvements in predicting socioeconomic factors related to health (69.7% vs. 65.7% average accuracy) while also achieving a higher average continuous tracking score across five physiological metrics (Pearson vs. ). Ultimately, we establish native modality tokenization as a highly resilient, architecture-agnostic primitive for multi-resolution time-series. To facilitate future research, we will open-source the DWB benchmark.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Time-Series Foundation Models, Self-Supervised Learning, Multimodal Representation Learning, Wearables, Digital Health

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y7taOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUST is a tokenisation scheme for multi-rate wearable data. Each of nine streams (44 channels) gets its own patch encoder, with patch length matched to the stream’s native rate. The valid tokens are concatenated into one ViT-B sequence over 13-minute windows, with no resampling or imputation of missing patches. Tokens carry positional, timestamp and modality embeddings, and the modality embedding is a frozen Gemini text embedding of a short description. Pretraining uses JEPA (and an MAE variant) on about 3.7B minutes of data from 50k users. Evaluation is by linear probing on activity recognition (5 core and 10 long-tail classes) and on the DWB cohort (five regression and four classification targets, including employment, Medicaid and race/ethnicity), against an “early resampling” version of the same model and a raw-feature baseline. Headline JEPA results are A1 macro-F1 from 0.33 to 0.39, and DWB average r from 0.154 to 0.177 and accuracy from 65.7 to 69.7.

Critical strengths

The setting is realistic and large: real wearable data with genuinely different sampling rates and lots of missingness, pretraining splits by subject, and an evaluation that covers short-window activities and person-level outcomes. A solid answer on how to tokenise multi-rate sensor data would be useful to practitioners. I also appreciated that the authors show where the method loses, such as MAE in Table 2 and Dep/Anx, and flag the accuracy that only equals the majority class.

Critical weaknesses

The experiments don’t show why the method helps. From Fig. 2, the early-resampling baseline puts every stream on a 13-step grid with 1×1 patches, so it differs from MUST in resolution, token count and modality isolation all at once. The paper asks which of these matters (Sec. 5.1) but answers with an argument about a 1 Hz grid that wasn’t actually run. The claims about noise isolation, missing-data robustness and “O(1) routing” have no direct test either. Sec. 5.2 repeats numbers from the main tables with no missingness sweep or baseline under the same dropout, no compute figures are given, and there are no ablations of the components even though the discussion says the ablations “confirm” the claims.

The evidence for the gains is also thin. Results look single-run with no intervals, while many differences are 0.01–0.02 in r. The “consistent, architecture-agnostic” framing doesn’t match the tables: under MAE, MUST is worse on three of four activity metrics in Table 2, and under JEPA it is worse on Age, GAD and Dep/Anx. The only baselines are the authors’ own resampling variant and logistic regression, with no comparison to LSM-2 or the MoE methods in the related work. Person-level labels like age and race/ethnicity are predicted from 13-minute windows, but the paper doesn’t say whether the probe split is by participant. If it isn’t, part of the large Race/Ethnicity gain could be user identification.

Questions for authors

Clarifications:

  1. What exactly is the early-resampling baseline (grid, fill method, patch shape, token count), and what are the token counts and compute for both models?

  2. For DWB, are probe train and test splits by participant? How many participants, and what are the class counts and majority rates for each categorical target? Please report balanced accuracy or AUROC.

  3. Sec. 4.3 uses a 50% target mask for activity and 80% for DWB. Are Tables 1 and 2 from different checkpoints? On what data were mask ratios and learning rate chosen? Where are the combined A0+A1 numbers (0.73/0.75) from Sec. 5.2? Is there a held-out test set for activities, or are results on the validation cohort used for both selection and reporting?

  4. Appendix A says DWB is released as de-identified feature aggregations. Can MUST (which needs native-rate streams) be trained or evaluated on the release? What are the provenance, consent and IRB status of the 50k-user pretraining set?

Extra experiments: 5. A controlled ablation separating resolution/token granularity from modality isolation, as described in weakness 1.

  1. A missingness sweep and artefact injection into one stream, for both tokenisations.

  2. At least one external baseline (LSM-2, or a multi-patch-size time-series model).

Reviewer_e6tCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the challenges posed by heterogeneous sampling rates, temporal asynchrony, and missing data in wearable physiological sensor data. The authors propose using modality-specific encoders tailored to the native sampling rate of each modality to map different data streams into tokens of a shared dimensionality. Valid, unmasked tokens are then concatenated and fed into a shared Transformer, with JEPA employed for predictive learning in the latent space.

The paper evaluates the proposed approach on multimodal activity recognition and a range of clinical, demographic, and lifestyle prediction tasks using DWB. The experiments compare early resampling with modality-specific tokenization under both JEPA and MAE backbones. The results demonstrate performance improvements across multiple tasks. In addition, the authors propose making the DWB dataset available to academic researchers.

Critical strengths
  1. The data resource has significant potential value for future research. The paper leverages large-scale, real-world wearable data covering nine sensor modalities and evaluates the proposed approach across a diverse range of tasks, including activity recognition, clinical outcomes, and lifestyle-related outcomes. The authors' initiative to make DWB available to the research community is a valuable contribution of this work.

  2. The method is designed to address practical challenges in multi-rate sensor fusion. The authors consider the loss of fine-grained information that can result from uniform resampling. They address this issue by using independent encoders tailored to individual modalities, followed by joint representation learning through a shared Transformer and JEPA. This design is well suited to the asynchronous and incomplete nature of real-world wearable data. The experimental results also provide empirical support for the proposed approach.

Critical weaknesses
  1. The evidence supporting the proposed mechanism for improving cross-modal interactions remains insufficient. The paper attributes the observed performance gains to cross-modal semantic alignment, more effective exploitation of complementary features, and reduced cross-modal noise contamination. However, the current experiments primarily demonstrate differences in final downstream performance.

  2. Furthermore, the effectiveness of cross-modal learning is mainly supported by comparisons against raw-data baselines without pretraining. These comparisons simultaneously involve changes in both pretraining and representation strategies, making it difficult to isolate the contribution of cross-modal interactions. Consequently, the current results do not clearly distinguish whether the performance improvements primarily arise from better preservation of temporal information within individual modalities or from more effective exploitation of complementary information across modalities during fusion.

Questions for authors
  1. Can the authors provide more direct evidence to disentangle the contributions of preserving native temporal resolution and improving cross-modal interactions? For example, can the authors provide quantitative analyses of representations before and after fusion, or controlled experiments that retain modality-specific encoders while restricting cross-modal interactions? Such analyses would help determine whether the observed performance gains can be attributed to the cross-modal alignment mechanism emphasized in the paper.

  2. Which modalities does the model actually rely on for key downstream tasks, and do different modalities provide complementary benefits? For representative tasks, can the authors report results for individual modalities, combinations of modalities, or leave-one-modality-out ablations, and compare them with the full model? Such experiments would help establish whether the model effectively integrates information from multiple modalities and whether the observed gains are primarily driven by one dominant modality.

1 comment

Dear authors, we would appreciate your response to the questions raised in our review. We remain open to discussion and would be happy to clarify any of our points. Thank you.

Reviewer_iJ1MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Modality-Specific Early Tokenization combined with Joint-Embedding Predictive Architecture (JEPA) alignment to tackle the asynchronous, multi-rate nature of physiological sensor data in foundation models. By using independent patch encoders matched to native sampling frequencies, the framework isolates noise and bypasses artificial imputation by dynamically routing only valid tokens into a shared Vision Transformer. Evaluated on large-scale wearable datasets, the model demonstrates superior performance over generic early-resampling baselines, notably improving complex activity recognition and continuous tracking for health and socio-demographic factors.

Critical strengths
  • Novel Asynchronous Alignment: Solves multi-rate sensor integration using native, modality-specific encoders and dynamic concatenation instead of rigid, computationally heavy resampling grids.

  • Strong Empirical Results: Outperforms early-resampling baselines on massive real-world data, yielding an 18.2% Macro F1 boost in long-tail activity recognition and superior health tracking.

  • Architecture-Agnostic Utility: Demonstrates that modality-specific bottlenecks consistently improve downstream performance across both JEPA and MAE backbones.

  • Valuable Benchmark Contribution: Evaluates on over 3.7 billion minutes of data across 9 modalities and open-sources the DWB benchmark for the community.

Critical weaknesses
  • Impact of Synthetic Gemini Embeddings: Given that modality embeddings are generated from static text descriptions via a frozen Gemini encoder, did you evaluate whether simpler learnable modality embeddings perform similarly, or test how sensitive the model is to the specific phrasing of those text descriptions?

  • Generalizability Across Unseen Devices: How well does the modality-specific tokenization framework adapt to new or unseen sensor hardware that operates at different sampling frequencies or channel configurations without requiring full pre-training from scratch?

Questions for authors

same as weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.