acceptodds
Under review as a conference paper at ICLR 2027

MoJEPA: Learning Transferable Motion Representations via Structured Dual-View Joint-Embedding Prediction

Abstract

A fundamental challenge in sensor-based Human Activity Recognition (HAR) is learning transferable motion representations from unlabeled inertial sensor signals for effective cross-domain adaptation. Existing self-supervised methods often rely on signal reconstruction or consistency across augmented views, which may capture low-level signal variations rather than the underlying physical structure of motion, limiting the transferability of the learned representations. To address this problem, Joint-Embedding Predictive Architectures (JEPAs) offer a framework for learning representations through latent-space prediction. Building on this framework, we propose MoJEPA, a dual-view joint-embedding predictive framework that learns transferable motion representations by explicitly modeling the underlying motion structure. MoJEPA introduces two structured masking views along the channel and temporal dimensions, guiding the model to predict masked latent representations from complementary motion cues. The channel view encourages the representation to capture cross-channel physical dependencies, while the temporal view captures temporal motion dynamics. Jointly optimizing these complementary objectives encourages representations that capture shared motion structure. Extensive experiments with pretraining on two large HAR datasets and evaluation across seven target datasets show that MoJEPA outperforms existing self-supervised baselines under frozen-encoder evaluation with few-shot and low-label adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Inertial Sensors, Human Activity Recognition, Cross-Domain Transfer, Joint-Embedding Predictive Learning, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SsJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoJEPA, a self-supervised representation learning method for IMU-based human activity recognition.

Critical strengths

The authors use two datasets for pre-training. I acknowledge that MoJEPA achieves strong results across various label budgets, with particularly notable advantages in the 5% and 10% label settings.

Critical weaknesses
  1. The author’s research focuses on motion representations; however, beyond action classification and feature clustering, it does not demonstrate what specific motion variations the model captures across different movement states. I believe such variations could be visualized.
  2. For instance, can MoJEPA learn the cyclical variations of the left and right legs during walking, or the differences in movement rhythm between running and walking? I have not yet found any direct evidence of this in the paper, nor is there any proof that the features retain kinematic information.
  3. Most importantly, channel-level masking, cross-channel dependency modeling, and temporal masked prediction are not new concepts. While the specific combination of these techniques may be logical, the paper fails to adequately explain what novel learning capabilities this design offers compared to existing approaches (such as SWISS, RCSMR, and RobustHAR). Furthermore, improved action classification accuracy does not in itself prove that the model has learned a more authentic physical structure of the motion.
Questions for authors
  1. I am curious how the authors demonstrate that the model has truly learned the physical structure of the motion, rather than merely learning the statistical correlations between the six sensor channels.
  2. I would prefer to see a rigorous split across subjects, along with a re-run of key baselines under identical settings.
2 comments
AuthorsOpenReview ↗

Response to Reviewer SsJa 1/2

Thank you for all your valuable comments and suggestions. We provide our responses to your comments below and have revised the manuscript accordingly. All changes in the revised manuscript are highlighted in blue.

W1 & W2:

We have added a visualization of raw inertial signals and the corresponding latent representations for representative walking and running samples from USC-HAD (Figure 4 in Appendix B.2, Representation Visualization). The visualization shows that walking exhibits slower and generally lower-amplitude signal variations than running in the raw inputs. Importantly, distinct temporal variations between the two activities remain visible in the corresponding latent representations, providing qualitative evidence that MoJEPA learns discriminative motion features associated with different movement states. The input data consist of six-axis inertial signals collected using devices such as smartphones and wearable inertial sensors. These signals do not provide separate measurements of the left and right legs and therefore do not allow us to directly examine limb-specific coordination patterns.

W3:

Channel-level masking, temporal masking, and cross-channel dependency modeling have been explored in prior work. MoJEPA's key insight is the joint design of structured masking and prediction target space. It combines complementary channel-view and temporal-view masking with latent-space prediction rather than raw-signal reconstruction. As shown in Table 3, this dual-view design performs better with latent-space prediction, highlighting the importance of jointly designing the masking strategy and prediction target.

The masking strategies adopted by SWISS and RCSMR differ from ours in both masking design and prediction mechanism. Although both methods exploit dependencies across channels or sensors, neither adopts MoJEPA's complementary full-axis masking design: channel-wise masking across the entire temporal extent of randomly selected channels and temporal-wise masking across all channels within contiguous time regions. SWISS applies element-wise random or Markov-based long-sequence masking to individual channel-wise time series before RNN encoding. Its RNN embedding layer independently encodes each channel into a signal token, and the resulting tokens are processed by self-attention to capture inter-channel interactions. RCSMR learns cross-sensor invariant motion representations robust to sensor placement and orientation variations. Given two paired body-worn sensors X and Y, it applies identical time-domain alteration and frequency-domain masking patterns to both sensors, using the masked spectrogram of X to reconstruct the corresponding masked regions of Y. RobustHAR employs location-inspired spatial--temporal 3D-variation modeling with three masking strategies. It combines masked signal reconstruction with a semantic-consistency objective to learn multi-scale spatial-temporal representations. Unlike RobustHAR, MoJEPA predicts masked representations directly in the latent space and incorporates channel identity embeddings and temporal positional encodings to preserve channel-specific and temporal information. We have additionally evaluated SWISS, RCSMR, and RobustHAR under the same cross-domain transfer protocol. As shown in the revised Table 1, MoJEPA outperforms these baselines across all evaluation settings.

We have additionally evaluated SWISS, RCSMR, and RobustHAR under the same cross-domain transfer protocol. As shown in the revised Table 1, MoJEPA outperforms these baselines across all evaluation settings.

MethodMetric5-shot10-shot5%10%
SWISSAcc.0.564 ± 0.0050.638 ± 0.0060.733 ± 0.0010.759 ± 0.006
F10.531 ± 0.0070.614 ± 0.0040.698 ± 0.0010.727 ± 0.006
RCSMRAcc.0.644 ± 0.0110.706 ± 0.0060.763 ± 0.0060.778 ± 0.007
F10.625 ± 0.0130.690 ± 0.0060.742 ± 0.0050.760 ± 0.007
RobustHARAcc.0.660 ± 0.0190.703 ± 0.0320.705 ± 0.0240.714 ± 0.018
F10.634 ± 0.0180.679 ± 0.0410.636 ± 0.0290.641 ± 0.025
MoJEPAAcc.0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002
F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0080.920 ± 0.005
AuthorsOpenReview ↗

Response to Reviewer SsJa 2/2

Q1:

MoJEPA aims to learn representations that capture dependencies across the six inertial signal channels (three accelerometer axes and three gyroscope axes) and their temporal variations for human activity recognition. It is designed to learn motion-related representations from inertial signals, rather than explicitly modeling the underlying physical structure of human motion.

Q2: Thank you for this valuable suggestion. We have conducted an additional cross-subject evaluation in Appendix B.3 (Cross-Subject Study, Table 10), using subject-level splits to assess generalization to unseen subjects. We repeat the frozen linear-probe evaluation, ensuring that subjects in the test set are excluded from the training set. Specifically, a subset of subjects is used to form the labeled training pool, while the remaining subjects are held out for testing. As shown in Table 10, MoJEPA outperforms all baselines in both Accuracy and Macro-F1 across the four settings, demonstrating its strong generalization to unseen subjects. Table 10: Cross-dataset performance with a frozen backbone and linear probe under limited labeled data using subject-level splits. Results are averaged over seven datasets.

MethodMetric5-shot10-shot5%10%
LIMU-BERTAcc.0.313 ± 0.0130.341 ± 0.0070.358 ± 0.0080.357 ± 0.018
F10.202 ± 0.0180.219 ± 0.0050.231 ± 0.0110.227 ± 0.016
CrossHARAcc.0.373 ± 0.0610.416 ± 0.0050.434 ± 0.0030.445 ± 0.004
F10.265 ± 0.0640.293 ± 0.0120.300 ± 0.0130.301 ± 0.005
PatchTSTAcc.0.620 ± 0.0200.707 ± 0.0030.734 ± 0.0090.754 ± 0.007
F10.586 ± 0.0220.681 ± 0.0080.708 ± 0.0100.735 ± 0.003
TimeMAEAcc.0.533 ± 0.0250.555 ± 0.0080.593 ± 0.0140.595 ± 0.013
F10.446 ± 0.0270.461 ± 0.0050.501 ± 0.0310.504 ± 0.023
HAR-FMAcc.0.601 ± 0.0110.667 ± 0.0120.707 ± 0.0190.722 ± 0.029
F10.557 ± 0.0110.626 ± 0.0180.671 ± 0.0180.683 ± 0.028
TS-JEPAAcc.0.627 ± 0.0190.682 ± 0.0220.714 ± 0.0110.707 ± 0.028
F10.571 ± 0.0230.631 ± 0.0260.656 ± 0.0110.644 ± 0.028
BioBankSSLAcc.0.578 ± 0.0110.602 ± 0.0070.620 ± 0.0120.627 ± 0.008
F10.487 ± 0.0160.517 ± 0.0090.536 ± 0.0090.547 ± 0.013
TS-TCCAcc.0.621 ± 0.0090.661 ± 0.0100.672 ± 0.0080.691 ± 0.006
F10.555 ± 0.0180.599 ± 0.0080.610 ± 0.0120.624 ± 0.002
UniMTSAcc.0.672 ± 0.0130.714 ± 0.0030.719 ± 0.0110.729 ± 0.003
F10.641 ± 0.0220.695 ± 0.0040.696 ± 0.0050.704 ± 0.004
SWISSAcc.0.573 ± 0.0140.656 ± 0.0250.698 ± 0.0220.711 ± 0.023
F10.532 ± 0.0190.622 ± 0.0270.666 ± 0.0230.678 ± 0.027
RCSMRAcc.0.624 ± 0.0130.689 ± 0.0210.720 ± 0.0190.732 ± 0.010
F10.586 ± 0.0100.657 ± 0.0110.697 ± 0.0120.710 ± 0.003
RobustHARAcc.0.602 ± 0.0320.648 ± 0.0290.672 ± 0.0440.675 ± 0.026
F10.542 ± 0.0410.578 ± 0.0410.601 ± 0.0510.600 ± 0.035
MoJEPA-MPAcc.0.681 ± 0.0130.723 ± 0.0260.751 ± 0.0140.760 ± 0.008
F10.624 ± 0.0240.660 ± 0.0260.686 ± 0.0170.697 ± 0.008
MoJEPAAcc.0.726 ± 0.0040.781 ± 0.0250.831 ± 0.0160.846 ± 0.018
F10.694 ± 0.0110.757 ± 0.0280.814 ± 0.0200.831 ± 0.021

We appreciate the reviewer’s helpful suggestions and hope our response has addressed the concerns. Please feel free to let us know if any questions or concerns remain.

Reviewer_7JJtOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoJEPA, a JEPA-style self-supervised method for IMU-based human activity recognition. It uses two masking views: masking whole sensor channels, and masking one contiguous temporal block across all channels. The model predicts EMA-encoder latents at the masked positions. It is pretrained on DSA and WISDM and evaluated with a frozen encoder and linear probe on seven target datasets under few-shot and low-label settings.

Critical strengths
  1. Clear Motivation: Channel masking and time-block masking fit the coupled, periodic nature of IMU signals. The design is simple and the model is small.

  2. Useful Finding: Table 3 shows that structured masking helps latent prediction much more than raw-signal reconstruction.

  3. Broad Evaluation: The paper covers seven unseen datasets, four label budgets, many baselines, and thorough ablations.

  4. Clear Writing: The method and figures are easy to follow.

Critical weaknesses
  1. Gain Limited: With the same backbone, random-mask reconstruction already beats all baselines (Table 3 vs. Table 1). The dual-view design adds a marginal 1–3 points on top.

  2. Mechanism Only Indirectly Supported: The paper claims the two views learn cross-channel dependency and temporal dynamics. There is no direct test, such as noisy inputs or missing channels. Add experiments with noisy inputs and missing channels at test time will help.

  3. Unclear Split: Section 4.1 does not say whether the 10%/10%/80% split is by subject or by window.

Questions for authors

I am not familiar with this topic. Pls check the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7JJt

Thank you for all your valuable comments and suggestions. We provide our responses to your comments below and have revised the manuscript accordingly. All changes in the revised manuscript are highlighted in blue.

W1:

Thank you for this comment. Random-mask reconstruction is indeed a strong baseline, benefiting from the same Transformer backbone and preserving channel-specific token representations. Its strong performance highlights the effectiveness of the underlying representation architecture. MoJEPA further improves performance by combining complementary channel-view and temporal-view masking with latent-space prediction.

W2:

Thank you for this valuable suggestion. We have added a test-time input perturbation analysis in Section 4.6 (Table 6), evaluating the frozen encoder under missing channels, missing temporal patches, and Gaussian noise. The results show that removing one channel has a greater impact on performance than the other perturbations across all evaluation settings, highlighting the importance of complementary channel information. MoJEPA maintains relatively stable performance when up to 20% of temporal patches are missing or under Gaussian noise. These findings provide further evidence supporting the role of channel and temporal structures in MoJEPA.

PerturbationsSetting5-shot10-shot5%10%
Missing channels10.723 ± 0.0130.769 ± 0.0080.819 ± 0.0020.837 ± 0.004
Missing time patches10%0.780 ± 0.0110.836 ± 0.0060.889 ± 0.0030.905 ± 0.001
20%0.779 ± 0.0130.833 ± 0.0050.886 ± 0.0020.903 ± 0.002
Gaussian noiseσ = 0.010.778 ± 0.0120.834 ± 0.0060.889 ± 0.0040.906 ± 0.003
σ = 0.030.746 ± 0.0150.813 ± 0.0010.869 ± 0.0110.891 ± 0.004
Full input---0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002

W3:

The default 10%/10%/80% split is by window. We have clarified this in Section 4.1 (Data Processing and Splits). In addition, to evaluate generalization to unseen subjects, we have included an additional subject-level split experiment in Appendix B.3 (Cross-Subject Study, new Table 10).

Table 10: Cross-dataset performance with a frozen backbone and linear probe under limited labeled data using subject-level splits. Results are averaged over seven datasets.

MethodMetric5-shot10-shot5%10%
LIMU-BERTAcc.0.313 ± 0.0130.341 ± 0.0070.358 ± 0.0080.357 ± 0.018
F10.202 ± 0.0180.219 ± 0.0050.231 ± 0.0110.227 ± 0.016
CrossHARAcc.0.373 ± 0.0610.416 ± 0.0050.434 ± 0.0030.445 ± 0.004
F10.265 ± 0.0640.293 ± 0.0120.300 ± 0.0130.301 ± 0.005
PatchTSTAcc.0.620 ± 0.0200.707 ± 0.0030.734 ± 0.0090.754 ± 0.007
F10.586 ± 0.0220.681 ± 0.0080.708 ± 0.0100.735 ± 0.003
TimeMAEAcc.0.533 ± 0.0250.555 ± 0.0080.593 ± 0.0140.595 ± 0.013
F10.446 ± 0.0270.461 ± 0.0050.501 ± 0.0310.504 ± 0.023
HAR-FMAcc.0.601 ± 0.0110.667 ± 0.0120.707 ± 0.0190.722 ± 0.029
F10.557 ± 0.0110.626 ± 0.0180.671 ± 0.0180.683 ± 0.028
TS-JEPAAcc.0.627 ± 0.0190.682 ± 0.0220.714 ± 0.0110.707 ± 0.028
F10.571 ± 0.0230.631 ± 0.0260.656 ± 0.0110.644 ± 0.028
BioBankSSLAcc.0.578 ± 0.0110.602 ± 0.0070.620 ± 0.0120.627 ± 0.008
F10.487 ± 0.0160.517 ± 0.0090.536 ± 0.0090.547 ± 0.013
TS-TCCAcc.0.621 ± 0.0090.661 ± 0.0100.672 ± 0.0080.691 ± 0.006
F10.555 ± 0.0180.599 ± 0.0080.610 ± 0.0120.624 ± 0.002
UniMTSAcc.0.672 ± 0.0130.714 ± 0.0030.719 ± 0.0110.729 ± 0.003
F10.641 ± 0.0220.695 ± 0.0040.696 ± 0.0050.704 ± 0.004
SWISSAcc.0.573 ± 0.0140.656 ± 0.0250.698 ± 0.0220.711 ± 0.023
F10.532 ± 0.0190.622 ± 0.0270.666 ± 0.0230.678 ± 0.027
RCSMRAcc.0.624 ± 0.0130.689 ± 0.0210.720 ± 0.0190.732 ± 0.010
F10.586 ± 0.0100.657 ± 0.0110.697 ± 0.0120.710 ± 0.003
RobustHARAcc.0.602 ± 0.0320.648 ± 0.0290.672 ± 0.0440.675 ± 0.026
F10.542 ± 0.0410.578 ± 0.0410.601 ± 0.0510.600 ± 0.035
MoJEPA-MPAcc.0.681 ± 0.0130.723 ± 0.0260.751 ± 0.0140.760 ± 0.008
F10.624 ± 0.0240.660 ± 0.0260.686 ± 0.0170.697 ± 0.008
MoJEPAAcc.0.726 ± 0.0040.781 ± 0.0250.831 ± 0.0160.846 ± 0.018
F10.694 ± 0.0110.757 ± 0.0280.814 ± 0.0200.831 ± 0.021

We appreciate the reviewer’s helpful suggestions and hope our response has addressed the concerns. Please feel free to let us know if any questions or concerns remain.

Thank you for the response. After reading the other reviewers’ comments and your response, novelty has become my main concern: channel masking and temporal masking both come from prior works. If the paper is positioned as a recombination plus a new finding, the only insight is that structured masking helps latent prediction, which is too thin to support a full paper. I hope the authors can add more insights. Based on this, I will lower my score from 3 to 2.

AuthorsOpenReview ↗

Second-Round Response to Reviewer 7JJt

Thank you for raising this important concern. Our contribution is not the combination of existing masking operations, but the identification and validation of how predictive contexts should be structured to learn transferable representations from multichannel inertial signals.

  • Identifying the problem. Existing JEPA approaches often employ random channel-time block masking to construct predictive contexts. However, this design may be insufficient for multichannel inertial signals, which exhibit strong temporal periodicity and cross-channel physical dependencies. Local masking can leave neighboring temporal patches or correlated channels visible, potentially allowing latent targets to be inferred through local interpolation or redundant contextual cues without adequately capturing the underlying latent motion patterns. We argue that transferable motion representation learning requires predictive contexts that account for both dimensions of inertial motion: cross-channel physical dependencies and temporal dynamics.

  • Establishing the design principle through complementary full-axis prediction. To address this mismatch, MoJEPA separates the two dependency dimensions into complementary prediction views: channel masking over the full temporal extent and temporal masking across all channels. By masking the selected channels across the full temporal extent or a continuous temporal segment across all channels, the model is encouraged to exploit complementary dependencies in the remaining context rather than relying on local observations along the masked dimension. Their joint application imposes complementary constraints on latent-space prediction, aligning the predictive context with the dual structure of inertial motion. Our controlled experiments support this design: dual-view masking substantially benefits latent prediction, neither view alone matches their joint application, and partial coverage of either axis degrades performance. Channel-wise constraints provide the primary contribution, while temporal constraints offer additional benefits.

  • Revealing the source of transferable motion representations. Our findings suggest that transferable motion representations benefit from jointly capturing shared cross-channel physical dependencies and temporal dynamics, rather than relying primarily on domain-specific signal patterns. The results further highlight the importance of considering the dependency structure of multivariate time series when designing predictive contexts for latent-space representation learning. This insight extends beyond inertial HAR and provides guidance for developing JEPA-based methods for structured multivariate time series.

In response to this concern, we have revised the Abstract, Introduction, contribution statements, and Conclusion to clarify both the conceptual motivation and methodological contribution of MoJEPA.

Reviewer_tvQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies the JEPA recipe to inertial sensor data for Human Activity Recognition (HAR). The contribution is the masking: two views of the same window, one masking a few sensor channels entirely (so the model has to predict them from the other channels) and one masking a long contiguous time block across all channels (so it has to use longer-range temporal context). Losses averaged, shared encoder. Downstream is a frozen encoder with a linear probe on per-channel pooled features. Pretrained on DSA+WISDM, evaluated on 7 held-out datasets under 5/10-shot and 5%/10% labels, where it beats a decent set of baselines. The main ablation crosses masking strategy with prediction target and finds the structured masking matters under latent prediction but barely under raw-signal reconstruction.

Critical strengths
  1. One of the strengths of the paper is the masking-by-target interaction experiment (Table 3). Random vs dual-view masking is a ~6.6 point gap at 5-shot under latent prediction but only ~2.8 under signal reconstruction. That's the actual finding of the paper and it plausibly generalizes beyond HAR; it says masking design and target space are coupled choices. This is what keeps me on the positive side.
  2. The evaluation protocol is careful: pretraining data strictly separate from the 7 target datasets, frozen-encoder probing rather than fine-tuning, 4 label budgets, 3 seeds with std, per-dataset numbers in the appendix.
Critical weaknesses
  1. The novelty is thinner than the framing suggests. The learning machinery is I-JEPA/V-JEPA unchanged (EMA targets, narrow predictor, L1 on latents, full-input target encoder). The channel view moves V-JEPA's full-temporal-extent masking principle from spatial position to sensor channel; it has the same geometry, the same information-leakage justification, and whole-channel masking already appears in reconstruction-based HAR work (RCSMR [1], SWISS [2]). What's actually new is pairing the two complementary full-axis views in a time-series JEPA, plus the Table 3 interaction. That's a real but narrow contribution, and the paper presents it as a new framework rather than a masking study.
  2. MoJEPA gets a channel-pool probe (per-channel pooling + concatenation, so a C-times larger probe input), while several baselines ( including TS-JEPA, which is the closest minimal pair (same EMA+L1 latent prediction, different masking) ) get mean pooling because their tokens mix channels. There is no MoJEPA-with-mean-pool number anywhere in the paper. So I can't tell how much of the 0.614 to 0.780 gap over TS-JEPA is representation vs probe.
  3. The masking ablation doesn't test the comparison the design argument needs. The dual-view is only compared against random token masking and against dropping one view. The natural alternative, a contiguous block partial in both axes (time block over a channel subset, i.e. the V-JEPA-style port), is never run. Nor does any check of whether the masks need to span the full axis or maybe 80-90% coverage do the same thing. As written, the case for the specific dual-view design is an inductive-bias argument plus a weak baseline, not evidence.

[1] Logacjov, Aleksej and Kerstin Bach. “Self-supervised learning with randomized cross-sensor masked reconstruction for human activity recognition.” Eng. Appl. Artif. Intell. 128 (2024). [2] Lim, Sae Rin et al. “SWISS: Signal-wise self-supervised learning to learn a multi-signal interaction for human activity recognition.” Knowl. Based Syst. 287 (2024).

Questions for authors
  1. Can you report MoJEPA with a plain mean-pool probe (and TS-JEPA with a channel-structured probe if feasible)? How much of the gap survives?
  2. Can you add a contiguous masking baseline that's partial in both axes (temporal block over a random channel subset), matched mask ratio, latent targets?
  3. Accuracy drops monotonically with pretraining batch size (128->512, ~4 pts at 5-shot) and longer pretraining hurts few-shot. Was the LR retuned per batch size? Is something unstable here?
  4. Can you do a comparison of computing both views every step vs sampling one view per step at matched compute?
  5. And it would be good to say how much the channel-identity embeddings alone contribute. Also please scope the claim: for eg., UniMTS and BioBankSSL are pretrained on different corpora/modalities, and UniMTS actually wins 3 of 7 datasets at 5-shot.
7 comments
AuthorsOpenReview ↗

Response to Reviewer tvQJ 1/5

Thank you for all your valuable comments and suggestions. We provide our responses to your comments below and have revised the manuscript accordingly. All changes in the revised manuscript are highlighted in blue.

W1:

  1. MoJEPA provides a framework for cross-domain motion representation learning, where representations are first learned through structured dual-view self-supervised pretraining on unlabeled source datasets and then transferred to unseen target datasets with limited labeled data. Unlike general image and video data, inertial signals exhibit strong inter-channel dependencies, such as those among accelerometer and gyroscope axes. To capture inter-channel dependencies and temporal motion dynamics, MoJEPA introduces two complementary masking views that differ from the masking designs in I-JEPA/V-JEPA. The two views are jointly optimized through a unified predictive learning objective. To validate this design, we compare MoJEPA with a V-JEPA-style channel–time block masking baseline under the same evaluation protocol (revised Table 3). MoJEPA consistently outperforms this baseline across all evaluation settings and prediction targets. In the 5-shot setting, it achieves gains of 18.9 and 20.8 percentage points in Accuracy and Macro-F1 with latent prediction, and 9.5 and 10.6 percentage points with raw-signal prediction. These results demonstrate the benefit of the complementary channel-wise and temporal-wise masking design.

Comparison with V-JEPA-style channel–time block masking.

Ablation settingPrediction targetMetric5-shot10-shot5%10%
Channel–time block maskingRaw signalAccuracy0.614 ± 0.0020.676 ± 0.0080.771 ± 0.0030.813 ± 0.003
Macro-F10.592 ± 0.0050.655 ± 0.0060.736 ± 0.0040.786 ± 0.004
LatentAccuracy0.591 ± 0.0060.656 ± 0.0070.739 ± 0.0040.775 ± 0.001
Macro-F10.559 ± 0.0090.629 ± 0.0090.693 ± 0.0080.736 ± 0.002
MoJEPARaw signalAccuracy0.709 ± 0.0190.796 ± 0.0070.870 ± 0.0020.901 ± 0.003
Macro-F10.698 ± 0.0160.786 ± 0.0050.851 ± 0.0020.892 ± 0.003
LatentAccuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0080.920 ± 0.005
  1. The masking strategies adopted by SWISS and RCSMR differ from ours in both masking design and prediction mechanism. Although both methods exploit dependencies across channels or sensors, neither adopts MoJEPA's complementary full-axis masking design: channel-wise masking across the entire temporal extent of randomly selected channels and temporal-wise masking across all channels within contiguous time regions. SWISS applies element-wise random or Markov-based long-sequence masking to individual channel-wise time series before RNN encoding. Its RNN embedding layer independently encodes each channel into a signal token, and the resulting tokens are processed by self-attention to capture inter-channel interactions. RCSMR learns cross-sensor invariant motion representations robust to sensor placement and orientation variations. Given two paired body-worn sensors X and Y, it applies identical time-domain alteration and frequency-domain masking patterns to both sensors, using the masked spectrogram of X to reconstruct the corresponding masked regions of Y. We further evaluate SWISS and RCSMR under the same cross-domain transfer protocol. As shown in the revised Table 1, MoJEPA achieves better performance than both baselines across all evaluation settings.
MethodMetric5-shot10-shot5%10%
SWISSAccuracy0.564 ± 0.0050.638 ± 0.0060.733 ± 0.0010.759 ± 0.006
Macro-F10.531 ± 0.0070.614 ± 0.0040.698 ± 0.0010.727 ± 0.006
RCSMRAccuracy0.644 ± 0.0110.706 ± 0.0060.763 ± 0.0060.778 ± 0.007
Macro-F10.625 ± 0.0130.690 ± 0.0060.742 ± 0.0050.760 ± 0.007
MoJEPAAccuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0080.920 ± 0.005
AuthorsOpenReview ↗

Response to Reviewer tvQJ 2/5

W2 & Q1:

We have added a mean pooling variant of MoJEPA to the revised Table 1. For all baselines, we use channel pooling when their architectures preserve separate channel representations, and mean pooling when channels are already mixed during feature extraction. In particular, TS-JEPA mixes multichannel inputs during temporal tokenization, making channel-wise pooling incompatible with its original architecture. MoJEPA-MP outperforms TS-JEPA across all four evaluation settings, with Accuracy gains of 5.7–10.4 percentage points. It also outperforms most other baselines. UniMTS achieves higher Accuracy than MoJEPA-MP in the 5-shot and 10-shot settings, by 2.2 and 0.3 percentage points, respectively. However, MoJEPA-MP outperforms UniMTS in the 5% and 10% settings, with gains of 9.6 and 11.7 percentage points, respectively. This may be attributed to UniMTS leveraging both sensor signals and textual activity descriptions during pretraining, which provides additional semantic knowledge that is particularly beneficial when labeled data are extremely limited. Compared with full MoJEPA, mean pooling in MoJEPA-MP may smooth out channel-specific representations, weakening the distinctive features captured by individual channels. These results suggest that explicitly accounting for channel structure during representation aggregation benefits downstream activity recognition.

MethodMetric5-shot10-shot5%10%
TS-JEPAAccuracy0.614 ± 0.0000.674 ± 0.0140.767 ± 0.0060.805 ± 0.005
Macro-F10.591 ± 0.0030.648 ± 0.0150.724 ± 0.0060.763 ± 0.007
UniMTSAccuracy0.740 ± 0.0120.750 ± 0.0120.738 ± 0.0080.745 ± 0.006
Macro-F10.733 ± 0.0130.744 ± 0.0110.722 ± 0.0080.733 ± 0.00
MoJEPA-MPAccuracy0.718 ± 0.0100.747 ± 0.0110.834 ± 0.0010.862 ± 0.007
Macro-F10.698 ± 0.0130.724 ± 0.0100.797 ± 0.0040.832 ± 0.007
MoJEPAAccuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0080.920 ± 0.005
AuthorsOpenReview ↗

Response to Reviewer tvQJ 3/5

W3 & Q2:

We have added experiments comparing MoJEPA with a V-JEPA-style channel–time block masking baseline (revised Table 3) and investigating the effect of partial-axis masking in both views (Appendix B.4, Analysis of Masking Strategies).

  1. The channel–time block masking baseline randomly selects 5 of the 6 sensor channels and masks a contiguous block of 18 temporal tokens in each selected channel. The masking ratio is 75% (5/6 × 18/20), closely matching the 73.3% average masking ratio of MoJEPA's default dual-view masking (4/6 channels and 16/20 temporal tokens). We report results for both raw-signal and latent prediction targets.

Comparison with V-JEPA-style channel–time block masking.

Ablation settingPrediction targetMetric5-shot10-shot5%10%
Channel–time block maskingRaw signalAccuracy0.614 ± 0.0020.676 ± 0.0080.771 ± 0.0030.813 ± 0.003
Macro-F10.592 ± 0.0050.655 ± 0.0060.736 ± 0.0040.786 ± 0.004
LatentAccuracy0.591 ± 0.0060.656 ± 0.0070.739 ± 0.0040.775 ± 0.001
Macro-F10.559 ± 0.0090.629 ± 0.0090.693 ± 0.0080.736 ± 0.002
MoJEPARaw signalAccuracy0.709 ± 0.0190.796 ± 0.0070.870 ± 0.0020.901 ± 0.003
Macro-F10.698 ± 0.0160.786 ± 0.0050.851 ± 0.0020.892 ± 0.003
LatentAccuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0040.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0080.920 ± 0.005
  1. We investigate whether masking in each view needs to span the full axis of the other dimension. For the temporal view, we vary the number of masked channels within each temporal hole while keeping the hole length fixed at L=16L=16. For the channel view, we vary the length of the contiguous masked temporal block while keeping the number of masked channels fixed at ∣C∣=4|\mathcal{C}|=4. As shown in Tables 11and 12, masking across the full channel axis in the temporal view achieves the best overall performance, while reducing channel coverage generally leads to performance degradation. For the channel view, reducing temporal mask coverage from 20/2020/20 also leads to substantial performance degradation across all evaluation settings. In particular, masking only 18/2018/20 temporal patches already causes a notable decline. These results highlight the importance of full-axis masking in both views.

Effect of Channel Coverage in Temporal View Masking

Masked channelsMetric5-shot10-shot5%10%
6/6 (100%)Accuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0030.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0070.920 ± 0.004
5/6 (≈83%)Accuracy0.757 ± 0.0070.823 ± 0.0060.896 ± 0.0000.923 ± 0.001
Macro-F10.742 ± 0.0070.812 ± 0.0070.880 ± 0.0030.916 ± 0.001
4/6 (≈67%)Accuracy0.733 ± 0.0150.794 ± 0.0060.870 ± 0.0050.902 ± 0.001
Macro-F10.718 ± 0.0170.781 ± 0.0060.848 ± 0.0080.890 ± 0.001
3/6 (50%)Accuracy0.728 ± 0.0060.786 ± 0.0090.873 ± 0.0010.904 ± 0.003
Macro-F10.713 ± 0.0050.772 ± 0.0100.856 ± 0.0020.892 ± 0.002

Effect of Temporal Coverage in Channel View Masking

Masked time-block lengthMetric5-shot10-shot5%10%
20/20 (100%)Accuracy0.780 ± 0.0060.837 ± 0.0060.903 ± 0.0030.929 ± 0.002
Macro-F10.767 ± 0.0080.827 ± 0.0060.886 ± 0.0070.920 ± 0.004
18/20 (90%)Accuracy0.671 ± 0.0100.711 ± 0.0080.765 ± 0.0020.797 ± 0.002
Macro-F10.645 ± 0.0060.687 ± 0.0100.718 ± 0.0020.756 ± 0.003
16/20 (80%)Accuracy0.661 ± 0.0060.719 ± 0.0090.795 ± 0.0040.826 ± 0.004
Macro-F10.632 ± 0.0080.693 ± 0.0100.760 ± 0.0060.797 ± 0.004
14/20 (70%)Accuracy0.530 ± 0.0170.605 ± 0.0180.710 ± 0.0060.754 ± 0.001
Macro-F10.511 ± 0.0150.586 ± 0.0120.667 ± 0.0070.717 ± 0.003

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.