acceptodds
Under review as a conference paper at ICLR 2027

MS-Point: Multi-Modal Hierarchical Spectral Learning for Point Cloud Understanding

Abstract

Object-level point–image alignment leaves a structural observability gap: agreement between pooled descriptors does not directly constrain intermediate representations or distinguish neighborhood arrangements that produce the same readout. We introduce MS-POINT, a cross-modal pretraining framework that makes internal structure an explicit target of representation learning. At selected encoder stages, complementary neighborhood-consensus and deviation responses expose distinct structural statistics before pooling. Branch-specific visual targets, crosslevel coordination, and adaptive fusion jointly preserve shared object semantics and encourage useful specialization. A common interface of intermediate features and spatial neighborhoods supports both graph encoders and point Transformers. Evaluations across object classification, few-shot adaptation, dense prediction, and low-label scene learning demonstrate the transfer value of this design. Same-backbone interventions and capacity- and computation-matched controls connect these improvements to the structure of supervision. Together, our findings establish supervision of intermediate structural states as a concrete design principle for learning transferable, geometry-aware representations across modalities.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Cross-modal alignment, 3D representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U2HdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the central challenge of single-global-representation alignment in cross-modal self-supervised 3D point cloud understanding. The authors argue that existing methods overlook the hierarchical semantic structure and spectral properties of point cloud representations, leading to suboptimal transfer performance. To address this, they propose MS-POINT, a hierarchical-spectral structured cross-modal pretraining framework. The method extracts multi-level intermediate features from the 3D backbone (DGCNN by default), performs graph-filter-inspired low/high frequency decomposition on selected stages, and integrates these features via an adaptive fusion module. On the image side, they construct hierarchical representations with coarse/fine decomposition to provide structurally matched supervision. Extensive experiments demonstrate that MS-POINT outperforms previous cross-modal baselines such as CrossPoint and MM-Point across linear classification, few-shot learning, and segmentation tasks.

Critical strengths

The paper explores a potentially interesting direction of combining hierarchical feature learning and spectral analysis for cross-modal 3D pretraining. The experimental evaluation covers multiple standard benchmarks and downstream tasks, providing a basic assessment of the proposed method. The authors also include a series of ablation studies on the components of their framework, which offers some insight into the contribution of each module.

Critical weaknesses

This study has shortcomings in terms of technical rigor and experimental validity: the benchmark comparisons are outdated and incomplete; unsupervised state-of-the-art classical monomodal methods (such as Inter-MAE) have been overlooked; the ablation experiments failed to control for interference from parameters and computational costs, and thus cannot prove that the performance improvement stems from the spectral decomposition itself rather than from additional model capacity; and no comparisons were made with other spectral methods. (The paper is quite extensive; if I have overlooked anything during my review, I suggest the auth

Questions for authors
  1. Could you add comparisons with Inter-MAE(TMM'23) and TPM(TCSVT'25) on ScanObjectNN and ModelNet40? If you can demonstrate that MS-POINT significantly outperforms these latest baselines with a reasonable computational cost, I would be willing to reconsider my score.

  2. The current ablation studies do not isolate the effect of the spectral decomposition from the increase in parameters. Could you perform a controlled ablation where you compare the proposed graph-filter decomposition with a simple MLP or linear projection that has exactly the same number of parameters and computational cost? This would help determine whether the performance improvement comes from the spectral decomposition itself or just the additional model capacity.

  3. The paper only evaluates on object-level tasks and small-scale indoor segmentation. Could you provide results on a large-scale outdoor dataset such as SemanticKITTI for semantic segmentation? This would demonstrate that your method generalizes to more complex real-world scenarios.

  4. The authors claim that {A} acts as a low-pass filter and I-{A} acts as a high-pass filter, but they provide no theoretical analysis or references to support this. Could you provide a detailed explanation of why this formulation corresponds to spectral filtering, including a formal connection to the graph Fourier transform? Without this theoretical justification, the spectral decomposition component appears arbitrary and unmotivated.

  5. The proposed method increases training time and parameters compared to MM-Point for a 1.8% accuracy improvement on ScanObjectNN. Could you justify this trade-off? Are there any optimizations you can make to reduce the computational cost without sacrificing performance?

Reviewer_x8kTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MS-Point proposes a point–image pretraining framework that directly supervises intermediate point-cloud representations. It constructs neighborhood-consensus and deviation responses before pooling, aligns these branches with depth-specific visual targets, and combines them through adaptive fusion. Five joint objectives encourage consistency, cross-modal alignment, and branch diversity.

Critical strengths
  1. Clear motivation: The paper identifies limitations of global pooled alignment and proposes a concrete interface for supervising intermediate structural statistics.
  2. The framework reuses intermediate features and supports both graph-based encoders and point Transformers.
  3. The reported same-backbone studies, capacity-matched controls, multi-seed results, and downstream evaluations support the usefulness of the complete framework.
Critical weaknesses
  1. Primarily an integration contribution: The operators and learning objectives are established components. The paper could more clearly establish the benefit beyond a simpler, well-tuned multi-level alignment framework.
  2. Incomplete mechanism attribution: Point-cloud responses and image targets change together, making it difficult to isolate the contribution of the 3D neighborhood operator. The main experimental model is not specified sufficiently to reproduce the core result.
Questions for authors
  1. Can you independently enable point-cloud responses and image responses while keeping heads, losses, and computation fixed, to separate their contributions?
  2. How does the full objective compare with a simpler, retuned multi-level alignment objective under matched capacity and training budgets?
Reviewer_VUy2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduced MS-POINT, a framework that treats immediate geometry as an explicit target, with structural response and visual semantics coordinated in one objective. This paper proposed a structured supervision interface that exposes encoder depth information and neighborhood change to point-image learning before object aggregation, and enables transfers across graph encoders and point Transformers by complementary graph responses, semantic-depth targets and joint branch/fusion objectives. This paper provided extensive experiments to demonstrate the effectiveness of MS-POINT. The main concerns are the outdated compared methods, insufficient qualitative results, verbose description, unclear theory.

Critical strengths

-The proposed MS-POINT outperforms the listed methods in most quantitative experiments.

-Comprehensive quantitative experiments are conducted to support the effectiveness.

Critical weaknesses

-The manuscript is overly overbose with more than 60 pages, making it really difficult to identify its main ideas, contributions and purposes.

-Most compared methods are relatively outdated. For example, in Table 1, the supervised reference methods were published in or before 2017.

-The main paper and its appendix have more than 60 pages with 80 tables, but only five visual results are provided. Compared with its a large number of quantitative results, visual results are relatively limited.

-The theoretical part is not solid as many symbols are introduced without proper definition, and some are inconsistent. For instance, the symbol P in equation (3) (12) etc does not denote a point cloud but represent two different concepts. However, in page 36, P is used again to denote a point cloud.

-It is recommended to use appropriate diagramming tools, instead of relying solely on AI-assisted images, which are really difficult to interpret, and certain visual elements are not explained, e.g., the blue squares in Figure (3).

Questions for authors

-Could the author provide more comparison results with recent works published after 2025 unfer fair setting, especially the visual results?

-Could the author explain the main ideas and points of this paper in their own words, instead of verbose AI-polished text that is difficult to understand.

-Could the author re-deduce the core theoretical part of the paper in a rigorous and clear manner, ensuring that every symbol only has a unique meaning with explicit definition upon its first occurence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.