acceptodds
Withdrawn from ICLR 2027

Kinematics-Induced Multimodal 3D Human Pose Estimation with Subject-Level Privacy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal 3D Human Pose Estimation (3D HPE) combines complementary information from RGB, LiDAR, and mmWave radar, but models trained on correlated observations from the same individuals, raise privacy risks overlooked by record level analysis. We present a unified framework for multimodal 3D HPE that couples kinematics-induced sensor fusion with subject level privacy auditing and private training. First, our multimodal model aligns modality specific joint representation, injects skeletal structure and adaptively aggregates complementary sensor evidence for accurate pose prediction. Second, we formulate a black-box subject membership inference attack for 3D HPE, complemented by an empirical pointwise maximal leakage analysis, which characterizes how individual attack score outcomes change inference about the membership outcome. Third, we instantiate user-level differential privacy via Action Temporal Stratification, a population weighted within-subject sampling strategy that enforces action and temporal coverage. We evaluate our framework on the MM-Fi dataset across three diverse experimental protocols. Source-code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mulltimodal 3D Human Pose Estimation, Differential Privacy, Pointwise Maximal Leakage, Memebrship Inference, Multimodal Sensor Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tFcjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscripts argues for a gap in 3D pose estimation where the input modalities are a combination of RGB, lidar, and mmWave, while accounting for their fusion and subject-level privacy aware training. The authors claim for three contributions: modality alignment, subject membership inference attack for 3D HPE with leakage analysis, and a sampling strategy ATS for more unbiased subject-level differential privacy learning. THe proposed framework is validated on the MM_Fi dataset.

Critical strengths

The problem posed by the paper is very relevant. THe privacy preserving problem is well-motivated. The core strength is the integration of multimoda learning for pose estimation, privacy preserving auditing and training.

Critical weaknesses

THe paper could be strengthened with better support in additional ablation studie. This contribution of the kinematic structure and adaptive gating mechanism could be illustrated as why this multimodal fusion design is useful. The ATS sampling strategy could also be compared with other alternative sampling strategies to highlight the advantage of the unbiased training motivation. These would be espeically useful in considered jointly with and arguing why the framework is still novel and effective, when the tradeoff against keeping privacy budget low causing a considerable drop in accuracy (would an joint error estimate still be useful if it's on the order of 1000mm, if so, in what context?)

Questions for authors

it would help if the authors could motivate why this privacy training requires this multimodal fusion framework and vice versa.

for suggestions for additional ablation studies, please see critical weaknesses. Establihshing the novelty and effectivei of ATS would be convincing for me to raise my score, followed by the fusion strategy.

How sensitive are the values in table 2? would it vary a lot with some other subset of participants?

Reviewer_XenYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper has two claimed contributions: 1) a kinematics-induced fusion module and 2) a subject-level privacy framework. It was evaluated on MM-Fi which is mainly single-subject in each data frame and in a relatively open space.

Critical strengths

This paper has good motivation in terms of combining different sensor modalities and the need to limit what the trained model reveals about whether a subject’s data was used in training. The proposed pipeline appears to be designed to address both issues but with significant concerns on the novelty, its limitation, and evaluation.

Critical weaknesses
  1. Novelty: a) the fusion module includes modality alignment, kinematic injection, and shared-token track and feedback. Each step is based on somewhat existing literature such as multi-head attention, affinity matrix, and residual connection. The main distinction appears to be shared-token gating and feedback, but the paper lacks of clear ablation study on each of these steps and its contribution to the performance (otherwise, one could argue if the shared-token gating and feedback is necessary or critical). b) the suject-level DP module follows established user-wise DP in Chua et al. (2024) with an action-and-time sampling scheme for each subject’s gradient.

  2. Limitation: both fusion and DP modules appear to be designed towards a single-subject scenario (one subject in the scene) and an open-space environment. It appears that the design maybe brittle in the case of multiple subjects in the scenario. It is unclear that how the fusion and DP modules can be extended to account for these cases. To the reviewer's perspective, it requires significant efforts to account for association across potential pairs of keypoint features. Also radar may exhibit ghost targets, cluttter, multipath in the cluttered indoor space. It is unclear how these effects can be taken care.

  3. Evaluation: This is probably the weakest part of this paper. MM-Fi is quite limited in terms of the data granularity (sparse radar point clouds), single-subject-oriented, and open space. There exist more comprehensive datasets on multi-modal sensor datasets including the recent CUHK-X/S or OctoNet. For radar datasets, high-resolution radar datasets for multi-subject and cluttered space are also available such as RT-Pose, HIBER, MMVR, etc. A single dataset evaluation maybe overfitting and the generalization capability is not well tested. This paper lacks of significant ablation studies on each step of the two modules. How does each step in both modules contribute to the performance over different datasets, different indoor environments, different subjects, and different sensor granularity (different LiDAR specs, different RGB-D specs, and different radar specs)?

Questions for authors
  1. The alignment module can simply learn a fusion feature vector in (1) without any alignment. What is the analysis or experiment evidence to support the claim of "This interaction encourages the three streams to align at identical anatomical locations, allowing each modality to incorporate complementary evidence before the primary fusion stage." Can you show the impact without the alignment?
  2. What are the contributions to the performance for 1) Am and 2) Gm in Eq. (2)? Is it possible to mask either one out in (2) or both and directly use Hm=Xm in (3) to see their impact?
  3. Similarly, how important is the shared-token track and feedback? The authors need to provide more theoretical or experimental studies. The design appears to be reasonable but it is unclear why it maybe effective or not.
  4. Any ablation study on the action-and-time sampling scheme for each subject’s gradient?
Reviewer_2xjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework for multimodal 3D human pose estimation (HPE) that also includes methods for privacy auditing and private training. For multimodal fusion, the framework aligns the representations by injecting skeletal structure and using adaptive aggregation. For privacy auditing, a black-box subject membership inference attack is used, and action temporal stratification is used to provide user-level differential privacy. The framework is evaluated with three protocols on the MM-Fi dataset.

Critical strengths
  • The framework aims at subject-level privacy, which is crucial when multiple samples across frames are correlated.
  • The proposed fusion method improves HPE performance compared to other baselines.
Critical weaknesses
  • The novelty of the proposed DP method is unclear due to the lack of comparison with any other prior work focusing on DP for HPE.
  • The improvement in HPE performance compared to prior fusion methods mainly comes from stronger encoder backbones, and the actual effect of the proposed fusion methods is not clear.
Questions for authors
  • A comparison with DP baseline methods should be included.
  • A more informative comparison with prior fusion methods will be beneficial for readers to understand the merit of the proposed fusion method. For example, using a fusion method similar to prior works with the latest encoders.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.