1. The objective comparison is mixed up with architecture.
Only Inrtlwrist gets alternating attention structure .The five baselines all use plain attention. The paper treats the plain-attention InrtlWrist as the control for this, so I compared that variant with LIMU-BERT, which is also masked reconstruction with plain attention. Under frozen probing, it doesn't come out ahead: LIMU-BERT wins on arm pose (8.14 vs. 8.36 cm), body pose (98.55 vs. 99.96 mm), odometry (0.208 vs. 0.210 m), HHAR (61.2 vs. 60.2) and PADS (50.5 vs. 48.1), and the variant wins the other five. Even leaving out PADS, which I don't think can rank encoders (point 4), it is 4 to 5, so the numbers are very close. So my understanding is that the 9/10 result comes mostly from the architecture, which the baselines never got, rather than from the pretraining objective. I recognize these are single runs, so I may be wrong, but that’s how it looks to me. I’d like to see one of two things: either retrain LIMU-BERT and SimMTM with alternating attention, or use the plain-attention Inrtlwrist as the entry in the matched comparison.
That said, The architectural benefit itself also looks inconsistent. Comparing the two InrtlWrist versions directly, the frozen differences are mostly within 2 points, except on PADS (+7.1), which again can't really rank encoders (See in point 4). After fine-tuning, they swing both ways by large amounts, the alternating version is ahead by 7.7 on HHAR and 7.5 on TransfHAR, but behind by 16.1 on gesture and 9.1 on PADS.
2. Several key details are missing, which matters for a benchmark paper.
Related to point 1, I couldn't work out what actually differs between InrtlWrist-plain and LIMU-BERT. Sec. 4.2 says InrtlWrist's encoder only sees the visible patches, but the paper doesn't say how the LIMU-BERT reimplementation masks its input, which loss it uses, or whether the random-length cropping/trimming applies to all objectives. And several training details are missing:
- Pretraining: the mask ratio, masking scheme, learning rate, batch size, number of pretraining steps and per-source sampling are not shown.
- Downstream: the learning rate, number of steps and selection rule are not given either.
Sec. 4.3 says "under the following optimization settings" but then only gives Eq. 1. The Reproducibility Statement says Appendix A has the optimization settings, but it doesn't. For a benchmark that wants outside submissions, these details are essential. I feel like there's a inconsistency here, and I hope author could add these details.
3. The kinematic transfer result may be about knowing the data, not the task.
The paper’s main transfer claim, that supervision transfers within kinematic estimation, rests on the three Nymeria tasks. These share one dataset and one device, and on the main split they also share participants. All 82 evaluation participants also appear in training. So when an encoder trained on arm pose helps with odometry, I can’t tell whether it’s the arm-pose labels that help, or just that the encoder has already gotten used to this device, this recording setup and these group of people.
To their credit, the authors check part of this in App. E by splitting the participants into disjoint thirds. But the result actually weakens the story. The gains over the never-trained encoder roughly halve, and Inrtlwrist, which a kinematic source matches or beats on the shared split, now leads on body pose and odometry. In other words, about half of the original effect seems to have come from participant overlap, although App. E also cuts each source's training data to a third, so some of the drop may simply be less data. The paper does mention this in the introduction and Sec. 6 , but Fig. 3 and statements such as "each kinematic task's best frozen source is another kinematic task" still present the shared-split result as the main finding. I'd suggest making the participant-disjoint split the primary one for the kinematic block.
The half that survives isn’t clean either. The sources still come from the same dataset and device, and they are compared against an encoder that has never seen any data, so familiarity with Nymeria alone could explain the remaining gain. Two experiments would help separate this out. The first is an MAE pretrained only on Nymeria IMU, without labels: if it gets the same gain, the gain comes from the domain, not the task. The second is kinematic source–target pairs drawn from different datasets. One smaller point: in the transfer study, activity recognition is represented only by PRISM, even though the benchmark has four HAR datasets, and the gesture, clinical and retrieval families each have a single dataset. Conclusions such as the 16–18% PRISM gain from pose and odometry sources therefore rest on one dataset each, and PRISM in particular has only six test subjects and a dominant class (Table 11).
4. Some comparisons lean on weak references, especially on PADS.
On PADS, training from scratch reaches only 29.9 macro-F1, barely above the 25.4 of a head that always predicts the majority class, yet it still counts toward the claim that "fine-tuning improves on training from scratch on nine of ten tasks". For its Parkinson's conclusion the paper instead relies on the comparison with the dataset authors' BOSS pipeline, but that pipeline is itself beaten by a never-trained encoder (48.1 vs. 51.7), so under this protocol PADS doesn't seem able to rank encoders at all, and I’d either mark it as unranked or use the binary tasks from the original PADS paper.