acceptodds
Under review as a conference paper at ICLR 2027

PT: Prototype-Guided Self-Supervised Pretraining for Generalizable mmWave Human Point Cloud Representations

Abstract

Multi-source millimeter-wave (mmWave) observations offer an opportunity to learn human representations reusable across sensing tasks. However, sparse detections and multipath effects make these observations unstable, challenging existing point cloud pretraining methods. We propose PT, a prototype-guided self-supervised pretraining framework for mmWave human point clouds. PT represents local states in spatial regions as sparse compositions of shared prototypes and learns their temporal evolution through autoregressive prediction. Observation consistency further stabilizes representations under radar perturbations. After pretraining on unlabeled sequences from multiple sources, the same frozen encoder supports action classification, dense point generation, joint pose estimation, and future motion prediction on datasets not used for pretraining. Across these tasks, PT outperforms the compared pretraining baselines and achieves competitive or superior performance to task-specific models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
mmWave sensing, human mmwave point cloud

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c9AUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PT3 learns representations of sparse mmWave human point clouds from unlabeled, multi-source radar sequences. A device-conditioned observation encoder maps short frame groups to spatial slots, which are represented through a shared prototype bank. Pretraining combines prediction of future prototype compositions, balanced prototype learning and consistency under radar perturbations. The frozen encoder is evaluated with task-specific heads for action classification, dense point generation, pose estimation and motion prediction on datasets excluded from pretraining.

Critical strengths

S1. The problem is useful, and the evaluation is broad. Reusing one frozen radar encoder across four tasks and several subject/environment splits is a substantial practical contribution.

S2. Results are consistently strong among the evaluated pretrained encoders.

S3. Several design choices address radar-specific difficulties, including device conditioning and perturbation consistency. Shared downstream heads and splits improve comparability, and the appendices document validation-based selection for several heads.

Critical weaknesses

W1. The gain attributable to prototype prediction is not fully isolated from the encoder and readout. Table 6's same-framework Slot-AR variant reaches 89.26 mm Random pose MPJPE versus 66.74 for Full, but its retained losses and prediction target need a precise definition. The cross-method results also change architecture, tokenization and feature readout. A matched same-encoder SSL control would clarify which part of the framework produces the improvement.

W2. The component effects differ sharply across transfer tasks. Removing consistency raises Random pose MPJPE from 66.74 to 91.04, while improving C-Sub classification from 81.35 to 82.28; Slot-AR also retains Full's dense-generation CD of 0.91. Why is pose transfer particularly sensitive, and are optimization and downstream procedures otherwise matched across these variants?

W3. Some margins are small, and training variability is unmeasured: C-Sub accuracy is 81.35 versus DAPNet's 80.72, and generation EMD is 4.70 versus mmWaveFlow's 4.76. Only one pretraining seed is reported. Representative repeated-run estimates and the selection rules for the remaining loss coefficients would help assess these differences.

W4. The fixed 3×4 slot grid is defined in radar coordinates, but robustness to subject distance and lateral position is not characterized. Please explain how coordinate preprocessing and slot routing handle such changes, and state the intended input scope for scenes containing multiple people.

Questions for authors

Pls refer to the weakness, I would like recommand boaderline accept. The consistent frozen-transfer gains across four tasks provide sufficient practical value for a positive recommendation. My main reservations are incomplete isolation of the prototype objective and unexplained pose-specific sensitivity in the component ablations. Clarifying these would strengthen the scientific account of why the framework transfers.

Reviewer_N1vQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised pretraining framework for mmWave human point clouds. The main idea is to represent local human states using spatial slots and shared prototypes, rather than directly modeling unstable radar detections. The method uses autoregressive prediction to learn temporal changes of prototype compositions and a consistency loss to improve robustness to radar noise. The model is pretrained on four unlabeled mmWave datasets and evaluated on four downstream tasks, action recognition, dense point generation, pose estimation, and motion prediction. The results show that the frozen encoder outperforms the compared pretraining baselines across these tasks.

Critical strengths
  1. The paper addresses an important challenge in mmWave human sensing, where point clouds are sparse and unstable across frames and devices. Using spatial slots and shared prototypes is a reasonable way to reduce the dependence on individual radar detections.
  2. The paper evaluates the same frozen encoder on four different downstream tasks and datasets not used during pretraining. PT3 consistently outperforms the compared pretraining baselines, with particularly good results on cross-scene action recognition and cross-environment pose estimation. The ablation result also show that the main components contribute to the performance.
Critical weaknesses

1.The novelty of the pretraining objective is not fully clear. Prototype-based learning and latent prediction have been studied in previous self-supervised methods. PT3 combines these ideas for mmWave point clouds, which is useful, but it is not clear what advantage the sparse prototype composition provides over other latent prediction approaches. The paper could better explain this difference. 2. The experiments do not fully separate the effects of different design choices. Table 6 shows that removing prototype modeling reduces performance, which is encouraging. However, this ablation also changes the prediction target from prototype distributions to continuous features. It is therefore difficult to tell whether the improvement comes from sparse prototype composition or from using a different prediction objective. Also, the main baselines use different encoder architectures, so part of the improvement may come from the spatial-slot encoder rather than the pretraining method.

Questions for authors
  1. Table 6 shows that prototype modeling improves performance compared with direct slot prediction. Could the authors further clarify whether this improvement comes from the sparse prototype composition itself or from the discrete prediction target? A comparison with other latent prediction objectives using the same encoder would help.
  2. How much does multi-source pretraining contribute to generalization? Since this is one of the main motivations of the paper, it would be useful to compare single-source and multi-source pretraining with a similar amount of training data.
  3. What is the contribution of device-conditioned FiLM to cross-device generalization? Does the model still perform well without device metadata conditioning, especially on unseen devices?
Reviewer_p7EhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PT3, a self-supervised pretraining framework for mmWave human point clouds. It combines spatial slots, sparse prototype compositions, autoregressive prediction, and observation consistency. The pretrained encoder is frozen and evaluated on four downstream tasks using datasets excluded from pretraining.

Critical strengths
  1. The method is well motivated by the sparsity and instability of radar detections. Learning latent states rather than reconstructing individual detections is reasonable.
  2. The evaluation covers four downstream tasks, and the reported results demonstrate promising performance.
Critical weaknesses
  1. The methodological novelty needs clearer positioning. The framework combines multiple modules. The prototype ablation and top-k analysis support the usefulness of the design, but the paper could more clearly explain the methodological advance of sparse prototype compositions over existing prototype-based representation learning approaches.
  2. The paper uses four datasets for pretraining, but does not directly show how much using multiple sources helps. Comparing single-source and multi-source pretraining with the same amount of data would help answer this.
  3. The motion prediction setup should be clearer in the main text. The main text describes a TCN decoder, but Appendix I. This should be stated in the main text so that readers can see which parts of the prediction system come from PT3.
Questions for authors

The paper presents a reasonable radar-specific pretraining framework with broad downstream evaluation. However, the methodological advance remain insufficiently established. I lean toward Weak Reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.