- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a foundation model that uses the DINO self supervised learning model and adapts it to inertial sensor data. The model used is a transformer encoder with frequency aware positional encoding. The training objective of the proposed IMUDINO model combines a global DINO loss, a patch level (iBOT++) loss and a KoLeo regularisation loss. IMUDINO's performance is evaluated on the HAR (Human Activity Recognition) task, the motion capture task and the inertial odometry task.
Critical strengths
- a step towards general purpose inertial foundation models via an adaptation of dino self distillation to inertial sensing
- competitive performance on a broad range of tasks (global and dense prediction tasks)
- able to learn with a simple model and orders of magnitude less training data
Critical weaknesses
-
the adaptation of dino to inertial sensing seems straight forward - e.g. applying ideas of patch level objective and KoLeo regularization loss previously applied in visual tasks
-
In some tasks, e.g. in the Inertial MoCap results, it does not beat the sota, DynaIP (though the authors explain that IMUDINO is not aided by physical priors or does not have access to raw gyro readings).
Questions for authors
-
One of your claims is that IMUDINO could learn with orders of magnitude less data than competing techniques. It would be interesting to show how the SOTA and IMUDINO's ability to learn depends on the size of the data.
-
The impact of some elements of the training, e.g. the KoLeo regularizer is not always positive, eg. in MoCap performance you observed a drop, while in HAR there was a small improvement. Similarly channel masking led to improved HAR performance but reduced MoCap performance. Frequency-aware position encoding was also found to improve HAR and odometry but hinder slightly MoCAP. Could you explain the reasons / provide some intuition of why the MoCap task is not as much improved by these elements of the training recipe while the other tasks are? Does it have to do with the task itself, or other aspects of training or the datasets used for training?
2 comments
Response to Reviewer S1EE (2/2)
Q2 Effect of Certain IMUDINO Elements on MoCap: Here are some intuition regarding the observed ineffectiveness of certain IMUDINO elements on the MoCap task.
- KoLeo Loss: the KoLeo regularizer is applied to the CLS token only and its intended role is to encourage the global representations (CLS tokens) to spread out. As a result, it is not intended to improve the dense features used by the MoCap task. We also hypothesize that, as a regularizer, it may mildly interfere with the optimization of the patch-level objective.
- Channel Masking: Channel masking creates training samples where some channels are zeroed out, and may induce to model to adapt to such samples. Such adaptations, while useful for mitigating overreliance on certain channels for global tasks like HAR, do have the potential to harm MoCap performance. Our hypothesis is that MoCap is a very fine-grained task requiring information from all 6 channels, and the learned adaptation to missing channels may take model capacity away from learning fine-grained 6-channel features.
- Frequency-aware PE: We hypothesize that the ineffectiveness of frequency-aware positional encoding on MoCap could be attributed to MoCap datasets having a sampling rate (60 Hz) closest to the native sampling rate of the pretraining dataset (100 Hz). In contrast, most HAR datasets are recorded at 25 to 50 Hz, and the TartanIMU Challenge use 200 Hz inputs. While resampling augmentation may mitigate the difference, it is only applied at a probability of , and the underlying real data is still 100 Hz. As a result, adaptation to varying sampling rate could be less useful in MoCap than in HAR or odometry, since MoCap input is closest to the training corpus in terms of native sampling rate.
We leave more concrete exploration of how elements of inertial SSL recipes interact to future work, which could lead to more optimal combinations of training objectives.
Response to Reviewer S1EE (1/2)
Thanks for the insightful and constructive review. We hope the results and discussion below address your questions.
Q1 Effect of Data Scaling: To further explore the effect of data scaling on IMUDINO and RelCon, we repeated the experiment in Section 4.2 using subset with different sizes of our training set using the IMUFormer-6M model. The pre-training compute budget was fixed (i.e., smaller subsets got more training epochs to match the 60-epoch full dataset protocol in terms of compute).
The results show that IMUDINO is more data efficient, as it nearly reached full potential at just 800 hours of training data, while RelCon saw more improvement when we scale from 800 hrs to 11600 hrs. However, it has to be noted that IMUDINO-6M nearly saturates some of the downstream benchmarks (e.g. ~98 mF1 on MotionSense, ~95 mF1 on HHAR, >99 mF1 on Shoaib et al.'s dataset), which may dampen the observed overall gain in HAR performance.
[1] Xu, Maxwell, et al. "Relcon: Relative contrastive learning for a motion foundation model for wearable data." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Yuan, Hang, et al. "Self-supervised learning for human activity recognition using 700,000 person-days of wearable data." NPJ digital medicine 7.1 (2024): 91.