acceptodds
Under review as a conference paper at ICLR 2027

Point-S2AE: Structural and Semantic Masked AutoEncoders for Point Cloud Pre-training

Abstract

Masked autoencoders have been widely explored in point cloud self-supervised learning, whereby the point cloud is generally divided into visible and masked patches for reconstruction. These methods typically reconstruct masked patches independently based on visible representations, driving the encoder to learn local geometric structures. However, they treat the reconstruction of each masked patch as an isolated task, neglecting the inherent structural dependencies between adjacent patches. Furthermore, their strict reliance on local geometric reconstruction often fails to capture the holistic semantic consistency of 3D objects, limiting the robustness of the learned representations. To overcome these limitations, we propose a novel pre-training framework that augments the standard Masked Point Modeling (MPM) paradigm with two simple yet effective components: Latent Autoregressive Alignment (LAA) and Global Self-Distillation (GSD). Specifically, LAA explicitly models the spatial transitions between masked patches by enforcing an autoregressive predictive relationship between shallow and deep latent representations in the decoder, utilizing an InfoNCE contrastive loss to prevent representation collapse. Concurrently, GSD extracts consistent global semantics by applying a bidirectional self-distillation loss on the class tokens from two distinct masked views of the same point cloud in the encoder. Extensive experiments on standard benchmarks demonstrate that Point-S2AE significantly enhances the learned representations while maintaining high pre-training efficiency. Notably, Point-S2AE achieves great improvement over the Point-MAE baseline, particularly surpassing it by 5.16% on OBJ_BG, 5.34% on OBJ_ONLY, and 5.35% on PB_T50_RS for 3D object classification on the ScanObjectNN dataset, and establishing new state-of-the-art results on ModelNet40 few-shot learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point Cloud Pre-training, Self-Supervised Learning, Masked Autoencoders, Masked Point Modeling, Latent Autoregressive Alignment, Global Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tPLwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies two key limitations in existing 3D point cloud self-supervised learning methods: (1) the lack of explicit modeling of geometric dependencies between masked patches, and (2) the tendency of masked point modeling objectives to focus primarily on local geometric reconstruction while insufficiently capturing object-level semantic consistency.

Critical strengths
  1. The paper presents a clear and intuitive problem. It separates the limitations of masked point modeling into structural dependency modeling and global semantic consistency, and designs LAA and GSD to address the two aspects respectively.
  2. The proposed framework is simple and minimally intrusive to existing masked point modeling architectures. Both LAA and GSD are introduced as auxiliary pre-training objectives without substantially changing the backbone, and the additional modules are discarded during downstream fine-tuning.
Critical weaknesses
  1. The motivation behind LAA is not fully convincing. LAA treats the (t−1)(t-1)-th and tt-th masked tokens as structurally related simply because they are consecutive in a predefined ordering. However, consecutive tokens are not necessarily geometric neighbors. For example, under FPS ordering, two consecutively sampled patches may lie on distant parts of an object rather than on the same local surface, so predicting the previous token does not necessarily correspond to modeling a meaningful geometric transition. Moreover, since the decoder uses bidirectional self-attention, the predictor may already access information from the target token, introducing potential information leakage into the autoregressive prediction.

  2. The ablation on LAA is not fully controlled. In Table 5, the Point-MAE baseline and the LAA variant appear to use different augmentation settings. Since the proposed model uses rotation augmentation while the original Point-MAE does not, it is unclear how much of the reported improvement can actually be attributed to LAA.

  3. The comparison with prior work is outdated. Most of the baselines are from 2022–2023, while recent methods from 2024–2026 are largely missing. This makes the claimed state-of-the-art performance difficult to assess.

Questions for authors
  1. Could the authors clarify why consecutive masked tokens under the chosen ordering should correspond to meaningful geometric neighbors? It would also be helpful to compare alternative orderings and explain how information leakage is avoided under bidirectional self-attention.
  2. Did the LAA variant in Table 5 use rotation augmentation? If so, please provide a Point-MAE baseline trained with the same augmentation rather than directly using the originally reported result.
  3. Could the authors include comparisons with more recent methods from 2024–2026 in the experimental section?

My current rating places the score between 1 and 2 (closer to 1). If the authors can clarify the data augmentation and include comparisons with more recent literature, I would consider raising the score to 2 or 3.

Reviewer_dLWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Point-S2AE, a masked autoencoding framework for point cloud pre-training that combines geometric reconstruction with two auxiliary objectives. LAA align deep decoder features with shallow features of preceding patches to capture structural dependencies, while GSD encourages consistent global representations across differently masked views. The method is evaluated on ScanObjectNN and ModelNet40 classification, ModelNet40 few-shot classification, and ShapeNetPart part segmentation.

Critical strengths
  1. The proposed solution aligns well with the motivation. LAA and GSD directly address the stated goals of modeling inter-patch structural relationships and encouraging global consistency across masked views, respectively.

  2. The experimental results provide supporting evidence for the approach. Improvements across several downstream tasks, together with the component ablations, suggest that the two auxiliary objectives can benefit point-cloud representation learning.

Critical weaknesses
  1. The evaluation remains limited to object-level tasks, with modest improvements over strong baselines despite introducing multiple auxiliary training objectives. Reporting pre-training time and memory consumption, together with comparisons under matched computational budgets, would clarify whether the added complexity yields sufficient benefits.

  2. The core ideas overlap with existing work such as Point-LGMask [1] and Point-RAE [2], and the novelty requires further clarification. In particular, the paper should explain how LAA differs substantively from existing latent feature alignment methods and why it should be considered autoregressive modeling.

  3. The ablations support the effectiveness of the method, but evidence for the robustness of the gains and the underlying mechanisms remains limited. Given the narrow evaluation scope, numerical results alone do not sufficiently distinguish the claimed benefits. Patch-order controls, local feature visualizations, and cross-mask consistency analyses would help establish whether the improvements indeed arise from better structural modeling and semantic consistency.

[1] Point-LGMask: Local and Global Contexts Embedding for Point Cloud Pre-Training With Multi-Ratio Masking. IEEE Transactions on Multimedia, 26: 8360–8370, 2024.

[2] Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning. Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pp. 1738–1749, 2023.

Questions for authors
  1. How much additional pre-training time and peak GPU memory do LAA and GSD require compared with the baseline? Do the gains persist under matched computational budgets?

  2. What distinguishes LAA from the latent feature alignment in Point-RAE, and GSD from the global consistency objective in Point-LGMask? In LAA, can the deep token access its shallow prediction target through decoder attention, and what makes this objective autoregressive?

  3. Which patches ordering is used in the reported experiments?

Reviewer_qPQcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Point-S2AE, which introduces Latent Autoregressive Alignment (LAA) and Global Self-Distillation (GSD) into the masked point modeling framework. These components align deep and shallow decoder features and global representations across differently masked views, respectively. The method is evaluated on point cloud classification, few-shot learning, and part segmentation.

Critical strengths
  1. The framework is relatively simple, extending existing masked reconstruction methods with auxiliary objectives. The associated auxiliary modules are removed during downstream fine-tuning.
  2. The experiments cover multiple datasets and downstream tasks. The full model improves upon the Point-MAE baseline, and the component ablations provide some evidence of the proposed modules’ effectiveness.
Critical weaknesses
  1. The central motivation lacks direct evidence. The paper does not directly diagnose whether existing methods actually suffer from insufficient modeling of structural dependencies and global semantic inconsistency. The ablation studies mainly show that additional objectives improve downstream performance, which is insufficient to establish the proposed failure mechanisms.

  2. The definition of GSD is internally inconsistent. Section 3.2 initially describes GSD as distillation between a student and a momentum-updated teacher, but subsequently states that no separate teacher network is maintained, instead using a shared network with bidirectional stop-gradient operations. These two training mechanisms are not equivalent, and the current description leaves the actual implementation unclear.

  3. LAA may admit a prediction shortcut. Section 3.3 requires deep features to predict shallow features at another position but does not explain how the predictor’s access to the target features is restricted. Under the described standard self-attention mechanism, deep features may directly access the shallow targets; stopping gradients does not block this forward information path. It therefore remains unclear whether LAA actually encourages the model to learn spatial transitions.

  4. The performance claims and figure presentation need improvement. Existing methods, such as Point-PQAE [64], have reported higher or comparable performance in some settings. The blanket “state-of-the-art” claims should therefore be removed and replaced with precise descriptions of the comparisons. In addition, Figure 1 is crowded, and some text and connections hinder readability. Its layout, font sizes, and visual hierarchy require further manual refinement.

Questions for authors
  1. Is there direct evidence that representations learned by existing MPM models fail to capture structural dependencies or maintain semantic consistency across masked views? How does this evidence go beyond downstream performance improvements from additional objectives?

  2. Does GSD use a momentum teacher or a shared-parameter network with bidirectional stop-gradient operations? Please clarify the parameter update rules and how the target distributions are computed.

  3. Can the deep features in LAA access their corresponding shallow targets through self-attention? If so, what evidence rules out a shortcut based on directly accessing the target features and supports the interpretation that LAA learns spatial transitions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.