- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MoJEPA, a self-supervised representation learning method for IMU-based human activity recognition.
Critical strengths
The authors use two datasets for pre-training. I acknowledge that MoJEPA achieves strong results across various label budgets, with particularly notable advantages in the 5% and 10% label settings.
Critical weaknesses
- The author’s research focuses on motion representations; however, beyond action classification and feature clustering, it does not demonstrate what specific motion variations the model captures across different movement states. I believe such variations could be visualized.
- For instance, can MoJEPA learn the cyclical variations of the left and right legs during walking, or the differences in movement rhythm between running and walking? I have not yet found any direct evidence of this in the paper, nor is there any proof that the features retain kinematic information.
- Most importantly, channel-level masking, cross-channel dependency modeling, and temporal masked prediction are not new concepts. While the specific combination of these techniques may be logical, the paper fails to adequately explain what novel learning capabilities this design offers compared to existing approaches (such as SWISS, RCSMR, and RobustHAR). Furthermore, improved action classification accuracy does not in itself prove that the model has learned a more authentic physical structure of the motion.
Questions for authors
- I am curious how the authors demonstrate that the model has truly learned the physical structure of the motion, rather than merely learning the statistical correlations between the six sensor channels.
- I would prefer to see a rigorous split across subjects, along with a re-run of key baselines under identical settings.
2 comments
Response to Reviewer SsJa 2/2
Q1:
MoJEPA aims to learn representations that capture dependencies across the six inertial signal channels (three accelerometer axes and three gyroscope axes) and their temporal variations for human activity recognition. It is designed to learn motion-related representations from inertial signals, rather than explicitly modeling the underlying physical structure of human motion.
Q2: Thank you for this valuable suggestion. We have conducted an additional cross-subject evaluation in Appendix B.3 (Cross-Subject Study, Table 10), using subject-level splits to assess generalization to unseen subjects. We repeat the frozen linear-probe evaluation, ensuring that subjects in the test set are excluded from the training set. Specifically, a subset of subjects is used to form the labeled training pool, while the remaining subjects are held out for testing. As shown in Table 10, MoJEPA outperforms all baselines in both Accuracy and Macro-F1 across the four settings, demonstrating its strong generalization to unseen subjects. Table 10: Cross-dataset performance with a frozen backbone and linear probe under limited labeled data using subject-level splits. Results are averaged over seven datasets.
| Method | Metric | 5-shot | 10-shot | 5% | 10% |
|---|---|---|---|---|---|
| LIMU-BERT | Acc. | 0.313 ± 0.013 | 0.341 ± 0.007 | 0.358 ± 0.008 | 0.357 ± 0.018 |
| F1 | 0.202 ± 0.018 | 0.219 ± 0.005 | 0.231 ± 0.011 | 0.227 ± 0.016 | |
| CrossHAR | Acc. | 0.373 ± 0.061 | 0.416 ± 0.005 | 0.434 ± 0.003 | 0.445 ± 0.004 |
| F1 | 0.265 ± 0.064 | 0.293 ± 0.012 | 0.300 ± 0.013 | 0.301 ± 0.005 | |
| PatchTST | Acc. | 0.620 ± 0.020 | 0.707 ± 0.003 | 0.734 ± 0.009 | 0.754 ± 0.007 |
| F1 | 0.586 ± 0.022 | 0.681 ± 0.008 | 0.708 ± 0.010 | 0.735 ± 0.003 | |
| TimeMAE | Acc. | 0.533 ± 0.025 | 0.555 ± 0.008 | 0.593 ± 0.014 | 0.595 ± 0.013 |
| F1 | 0.446 ± 0.027 | 0.461 ± 0.005 | 0.501 ± 0.031 | 0.504 ± 0.023 | |
| HAR-FM | Acc. | 0.601 ± 0.011 | 0.667 ± 0.012 | 0.707 ± 0.019 | 0.722 ± 0.029 |
| F1 | 0.557 ± 0.011 | 0.626 ± 0.018 | 0.671 ± 0.018 | 0.683 ± 0.028 | |
| TS-JEPA | Acc. | 0.627 ± 0.019 | 0.682 ± 0.022 | 0.714 ± 0.011 | 0.707 ± 0.028 |
| F1 | 0.571 ± 0.023 | 0.631 ± 0.026 | 0.656 ± 0.011 | 0.644 ± 0.028 | |
| BioBankSSL | Acc. | 0.578 ± 0.011 | 0.602 ± 0.007 | 0.620 ± 0.012 | 0.627 ± 0.008 |
| F1 | 0.487 ± 0.016 | 0.517 ± 0.009 | 0.536 ± 0.009 | 0.547 ± 0.013 | |
| TS-TCC | Acc. | 0.621 ± 0.009 | 0.661 ± 0.010 | 0.672 ± 0.008 | 0.691 ± 0.006 |
| F1 | 0.555 ± 0.018 | 0.599 ± 0.008 | 0.610 ± 0.012 | 0.624 ± 0.002 | |
| UniMTS | Acc. | 0.672 ± 0.013 | 0.714 ± 0.003 | 0.719 ± 0.011 | 0.729 ± 0.003 |
| F1 | 0.641 ± 0.022 | 0.695 ± 0.004 | 0.696 ± 0.005 | 0.704 ± 0.004 | |
| SWISS | Acc. | 0.573 ± 0.014 | 0.656 ± 0.025 | 0.698 ± 0.022 | 0.711 ± 0.023 |
| F1 | 0.532 ± 0.019 | 0.622 ± 0.027 | 0.666 ± 0.023 | 0.678 ± 0.027 | |
| RCSMR | Acc. | 0.624 ± 0.013 | 0.689 ± 0.021 | 0.720 ± 0.019 | 0.732 ± 0.010 |
| F1 | 0.586 ± 0.010 | 0.657 ± 0.011 | 0.697 ± 0.012 | 0.710 ± 0.003 | |
| RobustHAR | Acc. | 0.602 ± 0.032 | 0.648 ± 0.029 | 0.672 ± 0.044 | 0.675 ± 0.026 |
| F1 | 0.542 ± 0.041 | 0.578 ± 0.041 | 0.601 ± 0.051 | 0.600 ± 0.035 | |
| MoJEPA-MP | Acc. | 0.681 ± 0.013 | 0.723 ± 0.026 | 0.751 ± 0.014 | 0.760 ± 0.008 |
| F1 | 0.624 ± 0.024 | 0.660 ± 0.026 | 0.686 ± 0.017 | 0.697 ± 0.008 | |
| MoJEPA | Acc. | 0.726 ± 0.004 | 0.781 ± 0.025 | 0.831 ± 0.016 | 0.846 ± 0.018 |
| F1 | 0.694 ± 0.011 | 0.757 ± 0.028 | 0.814 ± 0.020 | 0.831 ± 0.021 |
We appreciate the reviewer’s helpful suggestions and hope our response has addressed the concerns. Please feel free to let us know if any questions or concerns remain.
Response to Reviewer SsJa 1/2
Thank you for all your valuable comments and suggestions. We provide our responses to your comments below and have revised the manuscript accordingly. All changes in the revised manuscript are highlighted in blue.
W1 & W2:
We have added a visualization of raw inertial signals and the corresponding latent representations for representative walking and running samples from USC-HAD (Figure 4 in Appendix B.2, Representation Visualization). The visualization shows that walking exhibits slower and generally lower-amplitude signal variations than running in the raw inputs. Importantly, distinct temporal variations between the two activities remain visible in the corresponding latent representations, providing qualitative evidence that MoJEPA learns discriminative motion features associated with different movement states. The input data consist of six-axis inertial signals collected using devices such as smartphones and wearable inertial sensors. These signals do not provide separate measurements of the left and right legs and therefore do not allow us to directly examine limb-specific coordination patterns.
W3:
Channel-level masking, temporal masking, and cross-channel dependency modeling have been explored in prior work. MoJEPA's key insight is the joint design of structured masking and prediction target space. It combines complementary channel-view and temporal-view masking with latent-space prediction rather than raw-signal reconstruction. As shown in Table 3, this dual-view design performs better with latent-space prediction, highlighting the importance of jointly designing the masking strategy and prediction target.
The masking strategies adopted by SWISS and RCSMR differ from ours in both masking design and prediction mechanism. Although both methods exploit dependencies across channels or sensors, neither adopts MoJEPA's complementary full-axis masking design: channel-wise masking across the entire temporal extent of randomly selected channels and temporal-wise masking across all channels within contiguous time regions. SWISS applies element-wise random or Markov-based long-sequence masking to individual channel-wise time series before RNN encoding. Its RNN embedding layer independently encodes each channel into a signal token, and the resulting tokens are processed by self-attention to capture inter-channel interactions. RCSMR learns cross-sensor invariant motion representations robust to sensor placement and orientation variations. Given two paired body-worn sensors X and Y, it applies identical time-domain alteration and frequency-domain masking patterns to both sensors, using the masked spectrogram of X to reconstruct the corresponding masked regions of Y. RobustHAR employs location-inspired spatial--temporal 3D-variation modeling with three masking strategies. It combines masked signal reconstruction with a semantic-consistency objective to learn multi-scale spatial-temporal representations. Unlike RobustHAR, MoJEPA predicts masked representations directly in the latent space and incorporates channel identity embeddings and temporal positional encodings to preserve channel-specific and temporal information. We have additionally evaluated SWISS, RCSMR, and RobustHAR under the same cross-domain transfer protocol. As shown in the revised Table 1, MoJEPA outperforms these baselines across all evaluation settings.
We have additionally evaluated SWISS, RCSMR, and RobustHAR under the same cross-domain transfer protocol. As shown in the revised Table 1, MoJEPA outperforms these baselines across all evaluation settings.