- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HUPER, a multimodal human sensing framework that represents each modality as a Gaussian posterior over a shared latent human state. It combines human/environment decomposition, reliability-weighted posterior fusion, and uncertainty-conditioned flow refinement. Experiments on MM-Fi and XRF55 report improvements in pose estimation, activity recognition, and missing-modality settings.
Critical strengths
- The paper addresses a practical problem: fusing heterogeneous sensors under missing or unreliable observations.
- Results show substantial improvements over X-Fi in several settings. Component, refinement-target, and synthetic-corruption analyses provide useful supporting evidence.
- The paper is written in a good manner.
Critical weaknesses
- While the author claims a human-centered posterior learning framework, the inherent human-state interpretation is insufficiently validated. The proposed state is a learned 256-dimensional latent. Cross-modal alignment, reconstruction, and orthogonality do not establish that it excludes environmental information. Existing ablations demonstrate component utility but do not isolate semantic disentanglement from additional capacity or regularization.
- MM-Fi uses a random split, and no dedicated cross-subject or cross-environment results are reported. The authors are encouraged to provide more dedicated results in different data splits.
- Octonet [1] is relevant literature, yet it is never cited, discussed, or compared. Actually, Octonet provides a rich source of modalities, activity classes, and 3D pose signals. The missing comparison with this dataset would weaken the contributions and scientific conclusions.
- Several reported results are inconsistent. XRF55 dual-modality accuracy is 91.3% in Table 6 but 93.1% in Tables 3/7/8. MM-Fi full-modality DDPM MPJPE is 65.3 mm in Tables 1/5 but 70.8 mm in Table 4. Some “All Avg.” values also disagree with the stated cardinality-balanced averaging rule.
- The benefit of flow refinement is not sufficiently isolated. The refinement target is derived from a stronger, full-modality representation. It remains unclear whether the improvement comes from the flow formulation or from this additional supervision.
[1] Yuan, Dongsheng, et al. "OctoNet: A large-scale multi-modal dataset for human activity understanding grounded in motion-captured 3D pose labels." Advances in Neural Information Processing Systems 38 (2026).
Questions for authors
See above.
8 comments
Friendly follow-up
Thank you again for recognizing the practical importance of robust heterogeneous sensing, the substantial improvements over X-Fi, and the value of our component and corruption analyses. We hope the new semantic probes, capacity-matched control, cross-subject/environment experiments, OctoNet evaluation, corrected aggregates, and matched refinement comparison address your concerns. If you have any further questions, we would be pleased to discuss them and respond promptly.
I appreciate the authors’ detailed rebuttal and additional experiments. The linear probes provide useful evidence that the content and environment branches learn different information.
My central concern, however, remains the interpretation of the “human state,” which is fundamental to the paper’s contribution. It is unclear whether this state is intended to represent instantaneous pose, motion over time, identity and body shape, or more broadly any information useful for downstream tasks. HPE and HAR require information at different temporal scales, yet the paper does not specify which factors the shared representation is meant to retain or suppress.
The rebuttal explains the intended roles of and through their training objectives and use in fusion. This is an operational definition, but it does not establish that the learned variables consistently follow that semantic division. The new probes support partial separation on the evaluated data; they do not yet show whether it persists for unseen subjects or environments.
Since the human-state interpretation is central to the paper’s claims, I maintain my original assessment pending stronger justification and evidence.
Response to Reviewer qHni: Semantic Validation, Generalization, and Flow Refinement
We sincerely thank the reviewer for the careful assessment.
W1. Human-state interpretation. Thank you for highlighting the need for direct semantic validation beyond task accuracy. We define “human-centered” operationally: c is the only latent aligned across physical observations of the same event, fused across modalities, refined, and supervised by HAR/HPE and skeleton losses; sm is modality-specific and never fused. To distinguish this semantics from added capacity, we use linear probes and a parameter-matched control that removes sm but reallocates its parameters to the content branch:
For the human representations (μm, z~), higher action and lower environment/modality accuracy are desired. The capacity-matched model without sm reaches 83.0 All Avg., comparable to the 83.7 of the standard variant without sm (Table 8) and far below 88.2 for full HUPER, showing that the gain is not explained by parameter count alone.
W2. Cross-subject/environment generalization. We appreciate this important suggestion on evaluating generalization beyond the random split. Protocol-3 random split is used in the main tables to match X-Fi and prior baselines. We additionally evaluate unchanged models on MM-Fi cross-subject and cross-environment partitions:
W3. OctoNet. Thank you for pointing us to this highly relevant benchmark. OctoNet provides synchronized heterogeneous observations together with both activity labels and motion-captured 3D poses, which matches the setting HUPER is designed for: aligning WiFi, mmWave, and vision-like depth through a shared human posterior, then serving HAR and HPE from the same refined latent. We therefore evaluate HUPER on OctoNet using the overlapping WiFi, mmWave, and depth modalities. All four multimodal subsets below are obtained from one checkpoint, without changing the architecture or retraining a specialist for each mix.
Two regularities follow from this table. First, HAR is strong whenever WiFi is present (95.0–98.6). Second, HPE stays in a narrow band (141.4–148.4 mm) across all four subsets: WiFi+Depth (141.5) essentially matches the three-modality result (141.4), whereas WiFi+mmWave is the weakest pose setting (148.4). Depth therefore contributes more to geometric pose than to action identity, which is consistent with reliability-calibrated fusion: a modality is weighted by how informative it is for the current human-state estimate, rather than being concatenated uniformly. The three-modality subset is best on both tasks, and the dual-modality numbers show that the same human posterior remains usable when one sensor is removed. We have also cited OctoNet in the revised related work.
W4. Reporting consistency. We thank the reviewer for the careful numerical check and apologize for the ambiguity it revealed. The correct XRF55 dual accuracy is 91.3%, confirmed over five runs; 93.1% was a transcription error and has been corrected throughout. The DDPM values refer to different settings: 65.3 mm in Tables 1/5 is the separately optimized final HUPER-DDPM model, whereas 70.8 mm in Table 4 is the controlled residual-refiner replacement using the shared Flow ablation backbone and budget. We will label this distinction explicitly in the captions of Tables 4 and 5. All aggregate columns are regenerated from subset logs under one rule: within-cardinality means for Single/Dual, all incomplete multimodal subsets for Partial, and cardinality macro-average for All.
W5. Flow vs. stronger target. We appreciate this precise observation, which helps separate the benefit of the transport objective from that of stronger supervision. To isolate architecture from target supervision, all variants below use identical source samples, detached full-modality targets, training budget, and downstream head. The parameter-matched residual MLP receives exactly the same privileged target as Flow: