- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes UniMotion for human activity recognition (HAR) using heterogeneous sensors. The key insight is to leverage motion as the alignment layer for different sensory signals. It proposes a two-stage optimization pipeline. The results on two datasets demonstrate sota results.
Critical strengths
- The idea to leverage the motion as the common ground for heterogeneous sensors is interesting.
- The research problem is well-motivated and the overall writing is clear.
- The evaluations are extensive and reach SOTA results.
Critical weaknesses
-
OctoNet [1] is neither discussed nor evaluated, despite being a particularly relevant test of the proposed idea. OctoNet provides a substantially broader range of heterogeneous sensors, activities, and synchronized motion-captured 3D pose labels. Unfortunately, it does not appear in the related-work discussion or evaluation. The authors are highly recommended to include comparative results on OctoNet, as it can be more informative for the cross-user and cross-scene evaluation.
-
While the core insight is motion-grounded multi-modal alignment, the motions involved are predefined activity classes. It raises two concerns:
- The motion anchors have been optimized with the downstream labels. Therefore, it remains unclear whether models have learnt useful features beyond the common representations of these classes. Does motion alignment improve recognition of activities with closely related body movements, or mainly help separate coarse classes? Figure 5 reports activity-wise representation discrepancies, but the paper does not show pairwise class confusion or identify which similar activities become easier to distinguish after alignment. Such an analysis is important because a common motion representation must preserve task-relevant fine distinctions. The authors are also recommended to consider including more experiments that indicate how much of the gain depends on activity supervision in the skeleton teacher.
- What is the performance on unseen activities? The representation may encode motion patterns that transfer beyond the supervised classes, but the current experiments cannot establish this. The authors are suggested to include more experiments with unseen activities when training the motion anchors.
-
How to distinguish motions from streams? Current experiments use annotated activity clips. However, it is unclear how the method would behave with different levels of granularity of motions. For example, what would be the atomic motions required for these tasks, including some compound actions?
-
The paper would benefit from identifying when motion grounding is most effective and how to handle the uneven modality quality. As we can tell from the results, the current methods show imbalanced performance gains on different modalities. More analysis is preferred.
[1] Yuan, Dongsheng, et al. "OctoNet: A large-scale multi-modal dataset for human activity understanding grounded in motion-captured 3D pose labels." Advances in Neural Information Processing Systems 38 (2026).
Questions for authors
See above.
6 comments
Response to Reviewer 5dxJ – Part 2/3
W2.1 — Fine-grained recognition of closely related motions
A2.1: To directly test whether motion alignment preserves fine-grained motion differences beyond coarse class separation, we analyze the nine left/right activity pairs in MM-Fi. Each pair has nearly the same overall motion pattern but differs mainly in which side of the body performs the action, making them the most closely related movements in the dataset. We report the mirror-confusion rate, i.e., how often one side is predicted as its mirrored counterpart.
Table R3. Mirror-confusion rate within left/right activity pairs on MM-Fi (%, lower is better).
| Activity pair | UniMotion w/o | UniMotion, label-free anchor | UniMotion |
|---|---|---|---|
| Throwing | 10.50 | 2.39 | 2.08 |
| Twist | 12.91 | 6.62 | 6.70 |
| Lunge (side) | 11.93 | 11.50 | 6.18 |
| Kicking | 6.85 | 3.58 | 5.06 |
| Lunge (front) | 5.73 | 2.40 | 4.12 |
| Raising hand | 4.68 | 3.36 | 2.21 |
| Waving hand | 2.37 | 1.48 | 1.18 |
| Body extension | 2.15 | 1.58 | 1.05 |
| Limb extension | 0.14 | 0.00 | 0.00 |
| Mean | 6.36 | 3.66 | 3.18 |
Motion alignment reduces the mean mirror-confusion rate from 6.36% to 3.18%, with lower confusion on eight of the nine pairs and no degradation on the remaining pair. This shows that the learned motion space preserves task-relevant, side-specific cues needed to distinguish closely related activities, rather than only separating coarse activity classes. The label-free anchor also reduces the mean confusion to 3.66%, suggesting that much of this fine-grained discrimination comes from structured motion itself rather than activity-label supervision.
W2.2 — Dependence on activity supervision in the skeleton anchor
A2.2: To directly quantify how much UniMotion depends on activity supervision in the skeleton teacher, we replace the HAR-fine-tuned anchor with a label-free anchor pretrained only with MAMP, while keeping all other settings unchanged. Results are mean ± std over three seeds on MM-Fi.
Table R2. Effect of activity supervision in the skeleton anchor on MM-Fi (accuracy %).
| Anchor | CS Depth | CS LiDAR | CS mmWave | CS Avg. | CE Depth | CE LiDAR | CE mmWave | CE Avg. |
|---|---|---|---|---|---|---|---|---|
| Label-free (MAMP only) | 96.50±0.36 | 91.67±0.91 | 93.83±0.45 | 94.00 | 95.10±1.12 | 60.34±1.28 | 68.60±1.08 | 74.68 |
| HAR-fine-tuned | 97.27±0.25 | 93.47±0.71 | 95.30±0.26 | 95.35 | 99.83±0.21 | 63.63±0.93 | 71.23±1.00 | 78.23 |
| from activity supervision | +0.77 | +1.80 | +1.47 | +1.35 | +4.73 | +3.29 | +2.63 | +3.55 |
Activity supervision adds +1.35 / +3.55 points on average under CS / CE, but the label-free anchor already achieves 94.00% / 74.68%. Together with W2.1, this shows that the motion space captures useful and fine-grained structure without activity labels; HAR supervision further improves task relevance rather than defining the representation itself.
W2.3 — Generalization to unseen activities
A2.3: To test whether the learned motion representation transfers beyond the activities used for motion pretraining, we evaluate MM-Fi → XRF55 anchor transfer. The motion encoder is pretrained on MM-Fi skeletons without activity labels, then fine-tuned on XRF55 activity labels and used as the anchor for XRF55 sensors. Thus, the motion prior is learned from MM-Fi’s 27 activities and transferred to XRF55’s largely different 55-activity set.
Table R4. MM-Fi → XRF55 anchor transfer, Top-1 accuracy (%).
| Anchor pretraining | Depth X-Sub | Depth X-Env | Infrared X-Sub | Infrared X-Env | RFID X-Sub | RFID X-Env | WiFi X-Sub | WiFi X-Env |
|---|---|---|---|---|---|---|---|---|
| XRF55 | 48.3 | 13.9 | 41.4 | 35.8 | 4.9 | 37.4 | 17.9 | 41.8 |
| MM-Fi → XRF55 | 46.7 | 12.6 | 40.8 | 34.3 | 4.6 | 36.1 | 17.2 | 40.7 |
| −1.6 | −1.3 | −0.6 | −1.5 | −0.3 | −1.3 | −0.7 | −1.1 |
The transferred anchor is only 0.3–1.6 points below the XRF55-pretrained anchor across all settings, despite the substantially different activity sets. This indicates that the learned motion representation transfers beyond the activities seen during pretraining, rather than being tied to a fixed set of supervised classes.
Response to Reviewer 5dxJ – Part 3/3
W3 — Motion granularity, compound actions, and continuous streams
A3: We address this from three aspects.
(1) Atomic and compound motions. UniMotion does not require predefined semantic atomic motions. Its basic unit is a short temporal–joint motion patch that captures local body dynamics; longer or compound activities are represented by combining these local patterns over time.
(2) Continuous streams. We follow the standard clip-level protocols of MM-Fi and XRF55 to ensure fair comparison with prior work. Following the reviewer’s suggestion, we additionally evaluate UniMotion on pseudo-continuous MM-Fi streams without retraining. For each test subject/environment, all 27 activities are concatenated and processed with sliding windows at a 0.5 s stride, without using activity boundaries, skeletons, or labels at inference.
Table R5. Continuous-stream recognition with a 2 s window, Top-1 accuracy (%).
| Protocol | Modality | Pre-segmented | Frame-level | Window-level |
|---|---|---|---|---|
| CS | LiDAR | 93.6 | 92.2 | 83.0 |
| CS | mmWave | 95.4 | 95.4 | 91.4 |
| CE | LiDAR | 63.7 | 55.7 | 47.1 |
| CE | mmWave | 71.6 | 69.4 | 63.6 |
With a 2 s window, frame-level accuracy remains within 2.2 points of clip-level performance in three of four settings, showing that UniMotion can operate effectively without pre-segmented activity boundaries.
(3) Temporal granularity. We further vary the window length to test how much temporal context is required.
Table R6. Effect of window length, frame-level / window-level accuracy (%).
| Protocol | Modality | 1 s | 2 s | 3 s |
|---|---|---|---|---|
| CS | LiDAR | 80.3 / 70.5 | 92.2 / 83.0 | 92.7 / 83.3 |
| CS | mmWave | 89.1 / 84.3 | 95.4 / 91.4 | 96.1 / 91.9 |
| CE | LiDAR | 42.2 / 38.2 | 55.7 / 47.1 | 56.5 / 48.6 |
| CE | mmWave | 61.5 / 57.4 | 69.4 / 63.6 | 70.2 / 64.6 |
Performance improves substantially from 1 s to 2 s and then largely saturates at 3 s. This suggests that UniMotion does not rely on a fixed definition of “atomic motion,” but benefits from sufficient temporal context to compose local motion patterns into an activity.
W4 — When motion grounding is most effective and handling uneven modality quality
A4: We observe two clear patterns.
(1) When motion grounding helps most. The largest gains occur on geometric and RF modalities such as LiDAR, mmWave, RFID, and WiFi, whose signals are closely tied to body geometry and dynamics but provide limited semantic or appearance cues. The gains are also more evident under cross-subject and cross-environment shifts, where sensor-specific statistics transfer less reliably.
(2) Handling uneven modality quality. This is the motivation for our RDMF module. Rather than averaging all modalities equally, RDMF estimates time-varying reliability for each modality and forms a weighted motion consensus, reducing the influence of unreliable inputs. Under MM-Fi CE, RDMF consistently improves over uniform fusion:
| MM-Fi CE (Fig. 4) | All modalities | w/o RGB | w/o Depth |
|---|---|---|---|
| Uniform fusion | 75.3 | 69.6 | 44.2 |
| RDMF | 81.1 | 78.9 | 53.5 |
Thus, motion grounding is most effective for modalities that directly encode body motion, while our RDMF module addresses their uneven reliability during multimodal fusion.
Response to Reviewer 5dxJ – Part 1/3
W1 — Evaluation on OctoNet
A1: We thank the reviewer for this suggestion. We have added an evaluation on OctoNet, which provides motion-captured 3D poses and a broader set of heterogeneous sensing modalities.
Setup. We follow the official 62-class HAR benchmark and its in-distribution (ID), cross-user (CU), and cross-scene (CS) protocols. UniMotion uses the same ResNet18 sensor backbone as the official baseline, with the motion-captured 3D poses used only as the training-time anchor.
Table R1. OctoNet Top-1 accuracy (%).
UniMotion improves over the official baseline in all 15 modality–protocol settings, with average gains of +7.8 / +6.1 / +2.9 points under ID / CU / CS. The gains extend across depth, thermal, infrared, mmWave, and WiFi, including the cross-user and cross-scene settings highlighted by the reviewer. This further supports motion grounding as a common representation across heterogeneous sensors and unseen users/scenes. OctoNet is now discussed in Related Work and evaluated in Sec. 5.4 (Table 3a).