acceptodds
Under review as a conference paper at ICLR 2027

Motion as Common Ground: Aligning Heterogeneous Sensing Modalities for Human Activity Recognition

Abstract

Human activity recognition (HAR) using heterogeneous sensors is challenging because modalities such as RGB, depth, LiDAR, mmWave, and WiFi capture the same activity through different physical signals and produce data with distinct structures. Existing methods typically bridge this heterogeneity through modality-invariant learning or LLM-based semantic alignment. We take a different view: *despite different sensing mechanisms, these modalities observe the same underlying human motion*. Since motion is shared across sensors and directly tied to the activity, it provides a natural basis for cross-modal alignment. In this paper, we introduce **UniMotion**, a motion-grounded framework that first establishes a shared temporal-joint motion space using a skeleton-pretrained motion model. UniMotion is then optimized in two stages. In Stage 1, modality-specific Motion Adapters with learnable joint queries map each sensor's native features into joint-indexed tokens in this space, aligning heterogeneous modalities within a common temporal-joint representation. Skeleton sequences are used only during training. In Stage 2, **Reliability- and Disagreement-Aware Motion Fusion (RDMF)** integrates the aligned modalities for multimodal recognition. RDMF estimates time-varying modality reliability to form a weighted motion consensus and uses cross-modal disagreement to retain complementary evidence beyond the consensus. Evaluated on MM-Fi and XRF55 across seven sensing modalities, UniMotion achieves state-of-the-art average accuracy under random-split, cross-subject, and cross-environment evaluation protocols and remains effective across varying modality combinations. These results support the use of structured human motion as a physically grounded, task-relevant shared representation for multimodal HAR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Activity Recognition, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5dxJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniMotion for human activity recognition (HAR) using heterogeneous sensors. The key insight is to leverage motion as the alignment layer for different sensory signals. It proposes a two-stage optimization pipeline. The results on two datasets demonstrate sota results.

Critical strengths
  • The idea to leverage the motion as the common ground for heterogeneous sensors is interesting.
  • The research problem is well-motivated and the overall writing is clear.
  • The evaluations are extensive and reach SOTA results.
Critical weaknesses
  • OctoNet [1] is neither discussed nor evaluated, despite being a particularly relevant test of the proposed idea. OctoNet provides a substantially broader range of heterogeneous sensors, activities, and synchronized motion-captured 3D pose labels. Unfortunately, it does not appear in the related-work discussion or evaluation. The authors are highly recommended to include comparative results on OctoNet, as it can be more informative for the cross-user and cross-scene evaluation.

  • While the core insight is motion-grounded multi-modal alignment, the motions involved are predefined activity classes. It raises two concerns:

    • The motion anchors have been optimized with the downstream labels. Therefore, it remains unclear whether models have learnt useful features beyond the common representations of these classes. Does motion alignment improve recognition of activities with closely related body movements, or mainly help separate coarse classes? Figure 5 reports activity-wise representation discrepancies, but the paper does not show pairwise class confusion or identify which similar activities become easier to distinguish after alignment. Such an analysis is important because a common motion representation must preserve task-relevant fine distinctions. The authors are also recommended to consider including more experiments that indicate how much of the gain depends on activity supervision in the skeleton teacher.
    • What is the performance on unseen activities? The representation may encode motion patterns that transfer beyond the supervised classes, but the current experiments cannot establish this. The authors are suggested to include more experiments with unseen activities when training the motion anchors.
  • How to distinguish motions from streams? Current experiments use annotated activity clips. However, it is unclear how the method would behave with different levels of granularity of motions. For example, what would be the atomic motions required for these tasks, including some compound actions?

  • The paper would benefit from identifying when motion grounding is most effective and how to handle the uneven modality quality. As we can tell from the results, the current methods show imbalanced performance gains on different modalities. More analysis is preferred.

[1] Yuan, Dongsheng, et al. "OctoNet: A large-scale multi-modal dataset for human activity understanding grounded in motion-captured 3D pose labels." Advances in Neural Information Processing Systems 38 (2026).

Questions for authors

See above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5dxJ – Part 1/3

W1 — Evaluation on OctoNet

A1: We thank the reviewer for this suggestion. We have added an evaluation on OctoNet, which provides motion-captured 3D poses and a broader set of heterogeneous sensing modalities.

Setup. We follow the official 62-class HAR benchmark and its in-distribution (ID), cross-user (CU), and cross-scene (CS) protocols. UniMotion uses the same ResNet18 sensor backbone as the official baseline, with the motion-captured 3D poses used only as the training-time anchor.

Table R1. OctoNet Top-1 accuracy (%).

ModalityID: ResNet18ID: UniMotionCU: ResNet18CU: UniMotionCS: ResNet18CS: UniMotion
Depth86.692.111.128.03.96.8
Thermal85.097.025.730.313.419.9
Infrared array1.87.82.69.00.82.9
mmWave (FMCW)24.034.28.910.75.37.6
WiFi91.196.53.43.92.43.4
Average57.765.510.316.45.28.1

UniMotion improves over the official baseline in all 15 modality–protocol settings, with average gains of +7.8 / +6.1 / +2.9 points under ID / CU / CS. The gains extend across depth, thermal, infrared, mmWave, and WiFi, including the cross-user and cross-scene settings highlighted by the reviewer. This further supports motion grounding as a common representation across heterogeneous sensors and unseen users/scenes. OctoNet is now discussed in Related Work and evaluated in Sec. 5.4 (Table 3a).

AuthorsOpenReview ↗

Response to Reviewer 5dxJ – Part 2/3

W2.1 — Fine-grained recognition of closely related motions

A2.1: To directly test whether motion alignment preserves fine-grained motion differences beyond coarse class separation, we analyze the nine left/right activity pairs in MM-Fi. Each pair has nearly the same overall motion pattern but differs mainly in which side of the body performs the action, making them the most closely related movements in the dataset. We report the mirror-confusion rate, i.e., how often one side is predicted as its mirrored counterpart.

Table R3. Mirror-confusion rate within left/right activity pairs on MM-Fi (%, lower is better).

Activity pairUniMotion w/o LalignL_{\text{align}}UniMotion, label-free anchorUniMotion
Throwing10.502.392.08
Twist12.916.626.70
Lunge (side)11.9311.506.18
Kicking6.853.585.06
Lunge (front)5.732.404.12
Raising hand4.683.362.21
Waving hand2.371.481.18
Body extension2.151.581.05
Limb extension0.140.000.00
Mean6.363.663.18

Motion alignment reduces the mean mirror-confusion rate from 6.36% to 3.18%, with lower confusion on eight of the nine pairs and no degradation on the remaining pair. This shows that the learned motion space preserves task-relevant, side-specific cues needed to distinguish closely related activities, rather than only separating coarse activity classes. The label-free anchor also reduces the mean confusion to 3.66%, suggesting that much of this fine-grained discrimination comes from structured motion itself rather than activity-label supervision.


W2.2 — Dependence on activity supervision in the skeleton anchor

A2.2: To directly quantify how much UniMotion depends on activity supervision in the skeleton teacher, we replace the HAR-fine-tuned anchor with a label-free anchor pretrained only with MAMP, while keeping all other settings unchanged. Results are mean ± std over three seeds on MM-Fi.

Table R2. Effect of activity supervision in the skeleton anchor on MM-Fi (accuracy %).

AnchorCS DepthCS LiDARCS mmWaveCS Avg.CE DepthCE LiDARCE mmWaveCE Avg.
Label-free (MAMP only)96.50±0.3691.67±0.9193.83±0.4594.0095.10±1.1260.34±1.2868.60±1.0874.68
HAR-fine-tuned97.27±0.2593.47±0.7195.30±0.2695.3599.83±0.2163.63±0.9371.23±1.0078.23
Δ\Delta from activity supervision+0.77+1.80+1.47+1.35+4.73+3.29+2.63+3.55

Activity supervision adds +1.35 / +3.55 points on average under CS / CE, but the label-free anchor already achieves 94.00% / 74.68%. Together with W2.1, this shows that the motion space captures useful and fine-grained structure without activity labels; HAR supervision further improves task relevance rather than defining the representation itself.


W2.3 — Generalization to unseen activities

A2.3: To test whether the learned motion representation transfers beyond the activities used for motion pretraining, we evaluate MM-Fi → XRF55 anchor transfer. The motion encoder is pretrained on MM-Fi skeletons without activity labels, then fine-tuned on XRF55 activity labels and used as the anchor for XRF55 sensors. Thus, the motion prior is learned from MM-Fi’s 27 activities and transferred to XRF55’s largely different 55-activity set.

Table R4. MM-Fi → XRF55 anchor transfer, Top-1 accuracy (%).

Anchor pretrainingDepth X-SubDepth X-EnvInfrared X-SubInfrared X-EnvRFID X-SubRFID X-EnvWiFi X-SubWiFi X-Env
XRF5548.313.941.435.84.937.417.941.8
MM-Fi → XRF5546.712.640.834.34.636.117.240.7
Δ\Delta−1.6−1.3−0.6−1.5−0.3−1.3−0.7−1.1

The transferred anchor is only 0.3–1.6 points below the XRF55-pretrained anchor across all settings, despite the substantially different activity sets. This indicates that the learned motion representation transfers beyond the activities seen during pretraining, rather than being tied to a fixed set of supervised classes.

AuthorsOpenReview ↗

Response to Reviewer 5dxJ – Part 3/3

W3 — Motion granularity, compound actions, and continuous streams

A3: We address this from three aspects.

(1) Atomic and compound motions. UniMotion does not require predefined semantic atomic motions. Its basic unit is a short temporal–joint motion patch that captures local body dynamics; longer or compound activities are represented by combining these local patterns over time.

(2) Continuous streams. We follow the standard clip-level protocols of MM-Fi and XRF55 to ensure fair comparison with prior work. Following the reviewer’s suggestion, we additionally evaluate UniMotion on pseudo-continuous MM-Fi streams without retraining. For each test subject/environment, all 27 activities are concatenated and processed with sliding windows at a 0.5 s stride, without using activity boundaries, skeletons, or labels at inference.

Table R5. Continuous-stream recognition with a 2 s window, Top-1 accuracy (%).

ProtocolModalityPre-segmentedFrame-levelWindow-level
CSLiDAR93.692.283.0
CSmmWave95.495.491.4
CELiDAR63.755.747.1
CEmmWave71.669.463.6

With a 2 s window, frame-level accuracy remains within 2.2 points of clip-level performance in three of four settings, showing that UniMotion can operate effectively without pre-segmented activity boundaries.

(3) Temporal granularity. We further vary the window length to test how much temporal context is required.

Table R6. Effect of window length, frame-level / window-level accuracy (%).

ProtocolModality1 s2 s3 s
CSLiDAR80.3 / 70.592.2 / 83.092.7 / 83.3
CSmmWave89.1 / 84.395.4 / 91.496.1 / 91.9
CELiDAR42.2 / 38.255.7 / 47.156.5 / 48.6
CEmmWave61.5 / 57.469.4 / 63.670.2 / 64.6

Performance improves substantially from 1 s to 2 s and then largely saturates at 3 s. This suggests that UniMotion does not rely on a fixed definition of “atomic motion,” but benefits from sufficient temporal context to compose local motion patterns into an activity.


W4 — When motion grounding is most effective and handling uneven modality quality

A4: We observe two clear patterns.

(1) When motion grounding helps most. The largest gains occur on geometric and RF modalities such as LiDAR, mmWave, RFID, and WiFi, whose signals are closely tied to body geometry and dynamics but provide limited semantic or appearance cues. The gains are also more evident under cross-subject and cross-environment shifts, where sensor-specific statistics transfer less reliably.

(2) Handling uneven modality quality. This is the motivation for our RDMF module. Rather than averaging all modalities equally, RDMF estimates time-varying reliability for each modality and forms a weighted motion consensus, reducing the influence of unreliable inputs. Under MM-Fi CE, RDMF consistently improves over uniform fusion:

MM-Fi CE (Fig. 4)All modalitiesw/o RGBw/o Depth
Uniform fusion75.369.644.2
RDMF81.178.953.5

Thus, motion grounding is most effective for modalities that directly encode body motion, while our RDMF module addresses their uneven reliability during multimodal fusion.

Reviewer_pGWCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of human activity recognition using multiple sensing devices and proposes a method to migrate different types of signals for activity recognition. More specifically, the proposed method employs human skeleton sequences as privileged information available only during training and aligns tokens from sensor signals with those from the skeleton sequences. The proposed method also predicts "reliability" for each sensing signal and time slot and weights the encoded features according to these reliability scores. Experimental evaluations with popular benchmark datasets for multi-modal activity recognition and pose estimation demonstrate that the proposed method outperformed several existing methods for multi-modal human activity recognition in many settings.

Critical strengths

S1. The problem dealt with in this paper is significant. Human activity recognition (using RGB camera images) is a well-established task in computer vision. More recently, the use of different types of sensors is becoming popular, such as IR light (depth cameras, LiDAR point clouds) and electromagnetic waves (WiFi CSI and millimeter-wave point clouds). However, every sensor has disadvantages for achieving accurate activity recognition. The multi-modal approach that accounts for the significance of each sensor is promising.

S2. The proposed method, when 3D human pose sequences are available during training, is reasonable. 3D pose sequences are strong "privileged information" for human activity recognition. If we can use this type of privileged information, the proposed method, which aligns tokens obtained from each sensing signal with those from the pose sequences, is simple and reasonable. (Note that simplicity should be appreciated, especially from the practical viewpoint.)

S3. The experimental results demonstrated in Section 5 are reasonably good.

Critical weaknesses

W1. The novelty of the proposed method relative to several existing methods should be presented more clearly. This paper considers the case where 3D human pose sequences synchronized with sensing signals are available during training. As far as I understand, this setting has already been explored in several previous papers. Also, we can see several types of usage of this privileged information, for example, (1) employing 3D human poses as intermediate representations and training a part of the network for accurate pose estimation (e.g., [Li+ ICCV2019, Making the Invisible Visible]), (2) setting 3D pose estimation as an auxiliary task (e.g. [Reilly+ CVPR2024, Just add ¥pi]), and (3) distilling knowledge from skeleton-based predictors (e.g. [Sinha+ AAAI2025, SKI models]). The current paper is related to the first and third approaches. Although the proposed method is new, it does not seem novel (i.e., it shows no substantial differences or surprises).

W2. The current formulation predicts the pose difference between two consecutive frames. Although this formulation is reasonable for ensuring the continuity of motion, it accumulates errors, leading to pose prediction breakdown.

Questions for authors

None.

[Post-rebuttal comments]

After the post-rebuttal discussions, most of the concerns presented in my initial review have been resolved. The value of this paper would be enhanced if the authors revise the paper according to the comments from all the reviewers.

3 comments
AuthorsOpenReview ↗

Response to Reviewer pGWC

W1 — Novelty relative to existing uses of privileged pose

A1: We agree that using pose as privileged training information has been explored before. The key difference, summarized in Table R1, is how pose is used. Prior works use pose as an intermediate representation, auxiliary supervision, or representation guidance. UniMotion instead uses structured motion as a shared physical coordinate space that aligns heterogeneous sensors at corresponding temporal patches and anatomical joints, which further enables motion-space multimodal fusion.

Table R1. Comparison with existing uses of privileged pose.

RF-Action (ICCV'19)π-ViT (CVPR'24)SKI (AAAI'25)UniMotion (ours)
Role of poseIntermediate 3D skeletonAuxiliary pose supervisionSkeleton-guided representationShared motion space for alignment
Unified frameworkRF-focusedRGB-focusedRGB / vision-language7 heterogeneous modalities
Cross-sensor structureLimited✗✗Shared (patch, joint)-indexed tokens
Fusion in shared space✗✗✗Reliability- and disagreement-aware fusion

Thus, our novelty is not the availability of privileged pose itself, but using its temporal–joint structure as the common physical space for heterogeneous-sensor alignment and fusion.


W2 — Pose-difference prediction and error accumulation

A2: UniMotion does not recursively accumulate predicted pose differences to reconstruct later poses. Errors in one predicted displacement therefore do not propagate into subsequent predictions.

(1) Pretraining. The signed displacement ΔXt=Xt+δ−Xt\Delta X_t=X_{t+\delta}-X_t is used only as the masked-prediction target for pretraining the skeleton motion anchor (Eq. 4–5). All masked targets are predicted directly from visible tokens in one pass; predicted displacements are never fed back to generate subsequent poses. The pretraining decoder is discarded afterward. Thus, each displacement is predicted without depending on earlier predictions, avoiding temporal error accumulation.

(2) Recognition and inference. During Stage 1, Stage 2, and testing, UniMotion operates directly on temporal–joint motion representations and predicts activity labels. It does not reconstruct pose sequences or autoregressively propagate pose predictions, so there is no mechanism for pose-prediction errors to accumulate over time.

(3) Empirical evidence. Fig. 5 further supports this behavior: sensor-to-skeleton motion discrepancy remains stable across MM-Fi activities and modalities, including large-motion actions such as lunges and jumping. We observe no degradation pattern indicative of accumulated pose-prediction errors.

Dear authors,

The current author feedback has resolved many of the concerns I raised in the initial review. However, I still have a question related to the novelty of the proposed method.

I could not understand the meaning (or intention) of "Cross-sensor structure" and "fusion in shared space". My understanding is the following:

  • "Cross-sensor structure" is a "structure" of feature representations that can be applied to "various sensing signals". In this sense, [Li+ ICCV2019] has a cross-sensor structure, but I am not sure why this is "limited".
  • Related to the above, [Li+ ICCV2019] has shared [patch, joint]-indexed tokens in its spatio-temporal attention module. One could argue that the difference between the two lies in where the module boundaries are.
  • All the methods presented in my initial review have no interest in merging several sensing signals. This is why they lack "fusion in Shared space".
AuthorsOpenReview ↗

Response to Reviewer pGWC's follow-up questions

We thank the reviewer for the thoughtful follow-up. We clarify the terminology and the specific distinctions from RF-Action below.

(1) What does “cross-sensor structure” mean, and why was RF-Action described as “Limited”?
By “cross-sensor structure,” we mean a common temporal–joint organization for representations from different sensing modalities. RF-Action provides this by processing RF-derived and vision-derived skeletons with the same recognition network.

“Limited” referred to the evaluated sensor coverage: RF and vision-derived skeletons, compared with seven heterogeneous modalities in UniMotion. We did not intend to imply that RF-Action lacks shared structure or cannot be extended to other sensors. We will replace this label with explicit sensor coverage, separating this scope difference from the architectural comparison.

(2) Is the difference only a matter of module boundaries?
Both methods exploit joint–temporal structure, and RF-Action also learns latent spatio-temporal features. The distinction lies in what each sensing branch predicts and how it connects to the shared recognition model:

RF-Action [Li+ ICCV'19]UniMotion
Common recognition input3D joint coordinates and confidenceLearned temporal–joint motion tokens
Sensor-side supervisionReconstruct 3D joint coordinatesMatch corresponding tokens from a frozen skeleton anchor
Training procedureJointly optimize skeleton generation and action recognitionFreeze sensor and motion encoders; train adapters and classifiers
Explicit pose at inferencePredicted skeleton feeds the recognition networkNo pose reconstruction; recognition uses adapted motion tokens
Evaluated sensor scopeRF and vision-derived skeletonsSeven modalities: RGB, Depth, LiDAR, mmWave, infrared, RFID, and WiFi

Even when RF-Action’s skeleton-generation and attention modules are grouped together, the model retains an explicitly supervised 3D-pose intermediate. UniMotion instead trains each adapter to match learned motion tokens and feeds these directly into the frozen motion encoder. The distinction therefore concerns the prediction target and training objective, rather than simply the placement of a module boundary.

(3) Fusion in shared space.
The cited works do not target simultaneous multi-sensor fusion. This row was intended to describe an additional task addressed by UniMotion, not a shortcoming of those methods.

For this task, our contribution is the RDMF design, rather than the inclusion of fusion alone. RDMF combines corresponding temporal–joint tokens using patch-level modality reliability and a disagreement-conditioned correction. With the same aligned representations, it improves MM-Fi cross-environment accuracy from 75.3% to 81.1% over uniform fusion with all modalities (Fig. 4). This provides evidence for the fusion contribution separately from the comparison with RF-Action.


In summary, UniMotion builds on skeleton-based cross-modal representations through direct sensor alignment to a learned, frozen motion space, together with reliability- and disagreement-aware fusion within that space.

Reviewer_QFeEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using structured human motion, rather than a language/semantic space or a learned modality-invariant latent, as the alignment target for heterogeneous-sensor (RGB, depth, LiDAR, mmWave, infrared, RFID, WiFi) Human Activity Recognition. A skeleton motion model (MAMP-style masked motion prediction, then fine-tuned for HAR and frozen) defines a temporal-joint token space. In Stage 1, each modality keeps its frozen sensor encoder (from HoloLLM) and a per-modality adapter with 17 learnable "joint queries" cross-attends to sensor tokens to produce joint-indexed motion tokens, trained with a cosine+SmoothL1 alignment loss to the skeleton tokens plus CE through the frozen motion encoder. In Stage 2, a fusion module (RDMF) estimates per-patch modality reliability, builds a weighted consensus, and adds a disagreement-conditioned residual correction. Evaluated on MM-Fi and XRF55 under three protocols (random, cross-subject, cross-environment), reporting per-modality accuracy and the across-modality average.

Critical strengths
  1. The framing is a worthwhile counterpoint to the prevailing semantic-alignment approaches, and the system is well executed. Skeleton supervision for individual sensors isn't new (vision-derived poses already supervise WiFi/radar sensing), but using a structured temporal-joint space as the common anchor for seven heterogeneous modalities at once, with inference on arbitrary subsets, is a useful reframing; and Table 4 shows sensor features adapted into the space are directly usable by the skeleton-trained classifier, which is good evidence the sensors land in the motion space.
  2. Breadth of evaluation: seven modalities, three protocols, two datasets, arbitrary modality subsets at inference, plus a fusion module that trains 0.069M parameters vs 5.8M for X-Fi-style fusion with better accuracy. As a systems result this is thorough.
Critical weaknesses
  1. The comparison conflates alignment target with extra supervision. The anchor is a skeleton model fine-tuned on the HAR labels, so Stage 1 is largely distillation from a task-supervised teacher using a privileged modality the baselines never see; they align to generic semantic spaces (CLIP/ImageBind embeddings, LLM features) with no comparable teacher. The whole system therefore admits a mundane re-description, i.e. cross-modal distillation from the best available teacher, in which the motivation (physical grounding, temporal-joint structure) plays no role. The experiment that would separate the two readings (same supervision through an unstructured/pooled skeleton teacher, or skeleton distillation added to a semantic-alignment baseline) doesn't exist; Table 2 only toggles the alignment loss.

  2. The empirical support is weaker than the headline tables suggest: everything comes from a single training run, and the across-modality average conceals per-modality losses that contradict the uniform-SOTA story. MM-Fi margins over HoloLLM are +2.9/+2.8/+3.1, the random split is near ceiling anyway (97.5%), and per-modality results are mixed where it matters: HoloLLM wins cross-environment MM-Fi RGB by 13.7 points (80.6 vs 66.9), undiscussed, and visual modalities also trail under cross-subject. Gains concentrated on RF/geometric modalities would be a defensible finding ("motion helps where semantics are weak"), but the paper claims uniform SOTA via averages instead of engaging with the trade-off. With one seed and 2-3 point margins I can't rule out noise on MM-Fi.

Questions for authors
  1. Can you run an anchor-structure ablation: align to a pooled/unstructured skeleton teacher (same supervision, no temporal-joint structure) vs your structured anchor? And give one semantic-alignment baseline (e.g., HoloLLM) the same skeleton distillation?
  2. Please report mean$\pm$std over $\geq$3 seeds for Tables 1-2, at least on the cross-subject and cross-environment protocols where margins are small.
  3. Why does UniMotion lose ~14 points to HoloLLM on RGB under cross-environment? Does motion grounding discard appearance cues that matter for vision, and if so, shouldn't the claim be scoped to RF/geometric modalities?
  4. For XRF55 the motion anchor comes from 2D poses extracted from the dataset's own RGB. Doesn't that make the RGB branch partially circular (aligning RGB features to a target derived from RGB), and how does anchor quality from pseudo-poses limit the approach?
  5. Can an anchor trained on one dataset's skeletons align the other dataset's sensors (MM-Fi anchor for XRF55, or vice versa)? Any cross-dataset result would directly test the claim that physical grounding yields a transferable common space.
  6. What enforces that joint query j corresponds to anatomical joint j for RFID/WiFi?
5 comments
AuthorsOpenReview ↗

Response to Reviewer QFeE – Part 1/3

We thank the reviewer for the constructive comments. Revisions in the updated PDF are highlighted in blue. We address each point below.

W1 & Q1 — Ablations: Structured vs. unstructured skeleton anchor; HoloLLM with matched skeleton supervision

A1: We run both requested controls to separate the effect of temporal–joint structure from that of privileged skeleton supervision. All comparisons use the same pretrained HoloLLM sensor encoders, MM-Fi data, and evaluation protocols. We report mean ± std over three seeds on Depth, LiDAR, and mmWave under cross-subject (CS) and cross-environment (CE) settings; these therefore differ slightly from the single-run results in Table 1 (seed 3407).

(1) Structured vs. unstructured skeleton anchor. Using the same HAR-fine-tuned anchor, we compare pooled alignment over the K×JK \times J joint tokens with our token-wise temporal–joint alignment. The anchor, architecture, and training settings are held fixed; only the alignment granularity changes.

Table R1. Pooled vs. token-wise skeleton alignment on MM-Fi (accuracy %). CS: cross-subject; CE: cross-environment.

AlignmentCS DepthCS LiDARCS mmWaveCE DepthCE LiDARCE mmWave
Pooled (K×JK \times J average)96.93±0.3192.47±0.7894.63±0.4796.93±0.8361.10±1.2169.40±1.11
Token-wise (full UniMotion)97.27±0.2593.47±0.7195.30±0.2699.83±0.2163.63±0.9371.23±1.00

Token-wise alignment improves all six settings, averaging +0.67 percentage points under cross-subject and +2.42 under cross-environment evaluation. Because both variants use the same HAR-supervised skeleton teacher and differ only in whether temporal–joint structure is retained, the improvement cannot be explained by privileged supervision alone. This directly shows that UniMotion benefits from the structured motion representation itself, rather than merely distilling from a task-supervised skeleton teacher.

(2) HoloLLM with matched skeleton distillation.
We augment HoloLLM with the same HAR-supervised skeleton teacher used by UniMotion. Specifically, we freeze each pretrained HoloLLM modality encoder and train an MLP classifier with either cross-entropy (CE) alone or CE plus logit-level knowledge distillation (KD) from the skeleton teacher. This directly tests whether the same privileged supervision is sufficient to reproduce UniMotion’s gains.

Table R2. HoloLLM encoder without vs. with skeleton distillation on MM-Fi (accuracy %).

MethodCS DepthCS LiDARCS mmWaveCE DepthCE LiDARCE mmWave
HoloLLM + MLP90.03±0.3184.23±0.9685.63±0.4082.83±0.8650.70±1.3562.23±1.30
HoloLLM + MLP + skeleton distillation91.37±0.3285.23±0.9587.13±0.5083.43±0.8052.47±1.0763.43±1.05
UniMotion97.27±0.2593.47±0.7195.30±0.2699.83±0.2163.63±0.9371.23±1.00

Skeleton distillation improves HoloLLM in all six settings, but the gains are modest and remain well below UniMotion. This shows that the same HAR-supervised skeleton teacher provides useful supervision, but privileged supervision alone is insufficient to account for UniMotion’s gains.

The above comparisons are added to Appendix B.1 (Table 7) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer QFeE – Part 2/3

W2 & Q2 — Multi-seed evaluation under cross-subject (CS) and cross-environment (CE) protocols

A2: We rerun UniMotion with and without Lalign\mathcal{L}_{align} over three seeds on MM-Fi under cross-subject (CS) and cross-environment (CE) protocols. Table 1 reports a single run (seed 3407), so the three-seed average differs slightly.

Table R3. MM-Fi single-modality accuracy (%, mean ± std over three seeds).

ProtocolMethodVDMLWAvg.
CSHoloLLM (reported)98.499.093.876.69.375.4
CSUniMotion w/o Lalign\mathcal{L}_{align}94.77±0.6196.60±0.3693.73±0.3191.13±0.937.63±0.4076.77±0.52
CSUniMotion95.83±0.4097.27±0.2595.30±0.2693.47±0.718.77±0.3578.13±0.40
CEHoloLLM (reported)80.693.261.053.09.559.5
CEUniMotion w/o Lalign\mathcal{L}_{align}59.37±1.4687.17±0.9767.03±1.0655.57±1.278.87±0.4555.60±1.04
CEUniMotion66.73±1.1699.83±0.2171.23±1.0063.63±0.9310.67±0.4562.42±0.75
  • Seed robustness. The three-seed variance is small relative to the observed margins: UniMotion remains +2.73 points above the reported HoloLLM average under CS and +2.92 under CE.
  • Effect of Lalign\mathcal{L}_{align}. Adding Lalign\mathcal{L}_{align} consistently improves all five modalities under both protocols, showing that its benefit is robust to seed variation.
  • Per-modality trade-off. We agree that the gains are not uniform across modalities—most notably, RGB remains below HoloLLM under CE. We discuss this trade-off explicitly in Q3 rather than relying only on the modality average.

These multi-seed results are added to Appendix B.2 (Table 8).


W2 & Q3 — Per-modality trade-offs, particularly RGB under cross-environment evaluation

(1) Why does RGB drop on MM-Fi cross-environment (CE)?
A3-(1): MM-Fi CE trains on E01–E03 and tests on the held-out E04 environment, introducing a substantial shift in visual appearance and background. First, HoloLLM relies on a 7B language model with much greater visual-semantic capacity, whereas UniMotion uses a compact 6M motion model. Second, motion grounding prioritizes body dynamics over scene appearance. Under a strong visual shift, these choices can make the RGB-to-motion representation less effective at exploiting environment-specific visual cues. We therefore view the −13.7-13.7 point RGB gap as a capacity–representation trade-off.

(2) Does motion grounding discard appearance cues, and should the claim be scoped?
A3-(2): Partly, and we have scoped the claim more precisely in the revision. The MM-Fi CE result highlights a trade-off under strong visual shift, but does not suggest a general loss of useful appearance cues, since UniMotion remains competitive on RGB in most settings (Table 1). More broadly, UniMotion’s clearest gains are on geometric and RF modalities, which are also well suited to privacy-preserving and non-identity-revealing sensing.

Sec. 5.2 has been revised to make this modality-specific scope explicit.


Q4 — XRF55 motion anchor: circularity and pseudo-pose quality

A4: XRF55 does not provide ground-truth skeletons, so we extract 17-joint 2D poses from the synchronized RGB videos and use them only during training.

RGB circularity. The RGB input and 2D pose anchor originate from the same synchronized frames, but they follow separate representation pathways. The RGB branch uses the frozen HoloLLM visual encoder, whereas the anchor is produced by an independent pose estimator and then encoded by the skeleton motion model. The poses are used only during training and are absent at inference. Thus, the shared source introduces training-time dependence, but not a shared representation or test-time leakage.

Anchor quality. Pseudo-pose quality directly affects the motion anchor. The extracted 2D poses lack depth and may contain keypoint or occlusion errors, introducing noise into the temporal–joint targets. This is an inherent limitation of the XRF55 supervision setting rather than of UniMotion itself. Nevertheless, motion alignment improves every XRF55 modality under all three protocols, increasing average accuracy by +0.7 / +1.1 / +2.6 points under Random / CS / CE; under CE, the gains span +1.3 to +3.7 points. Better 2D poses or ground-truth 3D motion could further strengthen the anchor.

AuthorsOpenReview ↗

Response to Reviewer QFeE – Part 3/3

Q5 — Can a motion anchor trained on one dataset transfer to another?
A5: Yes. We test MM-Fi → XRF55 by pretraining the motion encoder on MM-Fi skeletons without labels, then fine-tuning it with XRF55 activity labels and using it as the anchor for XRF55 sensors. To remove dataset-specific skeleton differences, we use MM-Fi’s released 2D keypoints, map both datasets to their shared joints, and normalize by body center and torso length.

Table R5. MM-Fi → XRF55 anchor transfer, Top-1 accuracy (%).

Anchor pretrainingDepth X-SubDepth X-EnvInfrared X-SubInfrared X-EnvRFID X-SubRFID X-EnvWiFi X-SubWiFi X-Env
XRF5548.313.941.435.84.937.417.941.8
MM-Fi → XRF5546.712.640.834.34.636.117.240.7
Δ−1.6−1.3−0.6−1.5−0.3−1.3−0.7−1.1

The transferred anchor is only 0.3–1.6 points below the XRF55-pretrained anchor, retaining 91–99% of its accuracy across modalities and protocols. Despite differences in subjects, environments, sensors, and activities, the MM-Fi motion prior remains effective for aligning XRF55 sensors, supporting the transferability of the physically grounded motion space. This result is added to Sec. 5.4 (Table 3b).


Q6 — What enforces joint-query correspondence for RFID/WiFi?
A6: Joint correspondence is learned through token-wise alignment. Each adapter output Zk,jmZ^m_{k,j} is directly aligned with the skeleton token Zk,jsZ^s_{k,j} at the same temporal patch and joint index, encouraging query jj to represent anatomical joint jj. The frozen skeleton-trained motion encoder further reinforces this consistent joint ordering. Thus, explicit joint localization in raw RFID/WiFi signals is not required.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair and Reviewers

We thank the AC and reviewers for the constructive feedback. We have substantially strengthened the paper with additional experiments, controlled ablations, and broader evaluations. All revisions are highlighted in blue in the revised paper PDF.

  • New benchmark evaluation (5dxJ). We add OctoNet, covering five sensing modalities and three protocols. UniMotion improves over the official baseline in all 15 modality–protocol settings, including cross-user and cross-scene evaluation.

  • Controlled tests of novelty and privileged supervision (QFeE, pGWC). Token-wise temporal–joint alignment consistently outperforms pooled alignment, while matched skeleton distillation alone cannot reproduce UniMotion’s gains.

  • Robustness and new transfer experiments (QFeE, 5dxJ). Three-seed results confirm the MM-Fi improvements, while MM-Fi → XRF55 anchor transfer shows only 0.3–1.6 points degradation across settings.

  • Fine-grained motion and activity supervision (5dxJ). Motion alignment reduces mirror confusion on closely related left/right activities from 6.36% to 3.18%, while a label-free anchor already achieves 3.66%.

  • New continuous-stream evaluation and modality quality (5dxJ, QFeE). UniMotion operates on pseudo-continuous streams without retraining, while our RDMF module consistently improves over uniform fusion under uneven or missing modalities.

Overall, these results provide stronger evidence that structured human motion offers a useful and transferable common representation for heterogeneous sensing, while the revised paper more clearly states the method’s scope and modality-specific trade-offs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.