acceptodds
Under review as a conference paper at ICLR 2027

Ada3D: Adaptive Cross-Dimensional Alignment for Universal 3D Representation

Abstract

Motivated by the significant performance of contrastive learning models in vision-language tasks, recent efforts have extended such architecture into 3D scenes for richer representation. However, existing approaches typically utilize the pretrained 2D ViT models to encode original 3D group features, which overlooks the inherent domain gap between these two modalities and leads to suboptimal performance. Observing that 3D point clouds and 2D images share geometric correspondences that can be explicitly correlated through cross-dimension projection, we propose Ada3D, a framework that focuses on the adaptive transfer from the 3D to 2D feature spaces to enable universal 3D representation. We achieve this transfer capability via a dual mechanism comprising a pre-adapter and a post-adapter. The pre-adapter bridges unordered point cloud groups and regular image patch sequences through the point-level and patch-level fusion paradigm. Subsequently, the post-adapter utilizes a mix-attention mechanism to capture the latent features in the refined 3D groups and transforms them into the ViT input space. Furthermore, we introduce a specialized Gaussian masked autoencoder and a shallow Transformer layer finetuning strategy to preserve the generalization and reduce computation. Extensive experiments demonstrate that our approach achieves remarkable results on diverse 3D universal representation tasks, including zero-shot open-vocabulary classification, segmentation, and visual question answering.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D Representation, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_uXEbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Basically they say if you just dump 3D group tokens into a pretrained 2D ViT it doesnt work well, because FPS/KNN groups are messy and ViT wants a regular patch grid. So they add a pre-adapter (project to a few views, sample the ViT pos embeddings, then some masked attn at patch level) and a post-adapter with learnable queries. Train with InfoNCE on points/image/text, plus this “Gaussian MAE” thing that multiplies tokens by N(0,1), and they unfreeze the first two ViT layers. Eval is zero-shot cls on LVIS / ModelNet40 / ScanObjectNN, some retrieval viz, ScanNet200 seg after finetuning a head, and a few VQA examples with InstructBLIP

Critical strengths
  • the mismatch they point at is real, Pipeline-2 stuff like ViT-Lens/Uni3D does have this issue. Table 1 and Table 4 at least show a plain MLP or cross-attn adapter is weaker than their stack
  • cls setup is the usual OpenShape/ViT-Lens style on a big mixed pretrain set, and they do ablate the adapter bits / ft strategy / scale / mask type, which I appreciated
Critical weaknesses
  • pretty incremental. adapters are just multi-view PE sampling + empty-patch mask + query cross attn glued together. full-ensemble Top-1 vs RECON++/Uni3D/ViT-Lens is only +0.3 / +1.2 / +0.5 on LVIS / MN40 / ScanObj (Table 2). hard to sell “universal 3D representation” off that, and they dont even match compute / same ViT / same cached I,T features
  • a few things just dont line up. Table 1 says PointNet but sec 3.1 is FPS+KNN+PointBERT, 512 groups, and 54.0/88.5/65.9 show up again as Ada3D-G in Table 2?? so what did Table 1 actually run. Table 3 “OV seg” looks like they finetune tokenizer+head on ScanNet200, thats not really zero-shot. Fig 5 VQA is cherry picked and some panels look like normal 2D photos. also they wrote “fastest point sampling” lol
  • Gaussian MAE with mean-0 multiply is a weird regularizer (can just zero the token). Table 6 is +0.9 on ModelNet vs zero-mask, not enough for the robustness talk
Questions for authors
  1. Table 1 vs 3.1 — PointNet or PointBERT? which ViT / data gives 54.0 / 88.5 / 65.9
  2. can you rerun Uni3D / ViT-Lens / RECON++ with the same OpenCLIP, same cached embeddings, same train set. how much gain is left
  3. is Table 3 actually zero-shot OV or just ft on ScanNet labels. if ft, say the split and compare to a ft Uni3D/OpenScene under the same proto
  4. any automatic VQA number at all? how many Qs, acc vs ViT-Lens
  5. Eq1 views — fixed orthographic at train/test? assume gravity aligned objects? what if you throw random SO(3) at it
Reviewer_m7utOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Ada3D, a framework for transferring pretrained 2D Vision Transformer (ViT) representations to 3D point-cloud understanding.

Critical strengths

The mismatch between unordered point-cloud features and pretrained image-transformer representations is a meaningful issue.

The results suggest that the adaptation strategy can improve the effectiveness of pretrained vision-language representations for 3D understanding.

Critical weaknesses

The authors repeatedly claim that Ada3D bridges the discrepancy between 3D point features and pretrained 2D ViT features. However, improvements in downstream classification accuracy do not necessarily demonstrate that the underlying representation spaces have become better aligned. The appendix includes t-SNE visualisations, but these are qualitative and cannot by themselves establish reduced domain discrepancy. More direct evidence would strengthen the claim, such as representation similarity measurements, cross-modal retrieval analysis, or feature distribution distances.

Some implementation details are missing or not clearly presented. The results are encouraging, but it is difficult to determine how much of the improvement comes from the proposed adapters versus backbone capacity, training configuration, and computational resources.

Questions for authors

What makes the proposed Gaussian modulation an autoencoding method rather than a regularisation technique, and what specific objective reconstructs masked information?

Reviewer_6SaNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Ada3D adapts pre-trained 2D vision transformers to learn 3D point-cloud representations. Its pre-adapter combines positional information from multiple views and facilitates interactions among image patches, while a query-based post-adapter produces the final features. Training combines point–image–text contrastive alignment, Gaussian token modulation, and fine-tuning of shallow transformer layers. The method reports competitive zero-shot classification results and is also evaluated on regional retrieval, open-vocabulary segmentation, and multimodal question answering. However, its distinction from existing adaptation methods remains unclear, the ablations do not fully establish the contribution of the proposed geometric mechanism, and the downstream experiments require clearer descriptions and more thorough validation.

Critical strengths
  1. The problem of improving alignment between point-cloud features and pre-trained vision transformers is well-motivated. The projection-based pre-adapter presents a concrete mechanism for integrating positional and patch-level structural signals into learned 3D representations.
  2. The work is validated on three classification benchmarks under full-data and no-LVIS training configurations. It yields consistent improvements over ViT-Lens, notably a 3.3 percentage point gain on ScanObjectNN in the no-LVIS setup.
Critical weaknesses
  1. The statement that earlier methods overlook the mismatch between 3D inputs and ViTs needs qualification. ViT-Lens addresses this issue through latent-query cross-attention and self-attention, and Ada3D’s post-adapter appears similar. The authors should distinguish Ada3D’s contribution—particularly the projection-based MVA and PCI modules—and support it with controlled comparisons.

  2. Although Table 4 reports improvements as components are added, the parameter count also rises from 15.2M to 23.5M. A post-adapter-only baseline and capacity-matched controls are needed. Comparing pretrained 2D positional embeddings with random, sinusoidal, and direct 3D coordinate encodings would help determine whether the gains stem from transferred 2D priors, positional information, model capacity, or fine-tuning.

  3. Section 4.4 lacks essential details about supervision, category splits, the definitions of f-mIoU and f-mAcc, which pretrained components are retained, and how instance masks are generated. These omissions limit reproducibility and make it difficult to assess whether comparisons are fair. If the adapters and ViT are trainable, further ablations should separate the adapters’ effects on segmentation.

  4. Figures 5 and 10 present illustrative cases but no defined evaluation set, scoring protocol, or aggregate results. They therefore suggest potential applications rather than establish consistent improvements in spatial reasoning or hallucination reduction. Figure 5 also includes image-only comparisons despite frozen image and language branches, making it difficult to attribute the observed differences to the 3D adapters.

Questions for authors
  1. Could the authors clarify the exact architectural distinctions between their post-adapter and the cross/self-attention lens adopted in ViT-Lens? Is it feasible to benchmark against ViT-Lens under identical shallow fine-tuning and noise regularization, so that the individual impact of MVA/PCI can be disentangled? (Lei W, Ge Y, Yi K, et al. Vit-lens: Towards omni-modal representations[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 26637-26647.)
  2. The authors are encouraged to include a post-adapter-only baseline, and compare pre-trained 2D positional embeddings against random, sinusoidal, and raw 3D coordinate encodings with matched model capacity and training configurations. Additionally, we ask whether the reported performance gains are consistent across multiple repeated runs.
  3. Could the authors supply quantitative results on a fixed evaluation benchmark, with shared downstream models, prompts and decoding hyperparameters? We also seek explanations for the observed discrepancies in image-only cases, together with the evaluation protocol for spatial accuracy and hallucination.
1 comment
Reviewer_6SaNOpenReview ↗

Clarification Regarding AI Use in This Review

All substantive technical assessments, concerns about the paper, and the final score were independently formulated by me before using AI.

After reading the paper and drafting my own concerns, I used an AI assistant to refine the wording and organize those concerns into questions for the authors. The AI also suggested more specific experimental comparisons and reporting details. I reviewed these suggestions and independently decided which to include. I take full responsibility for the final review.

For reference, my original concerns, written before using AI, were as follows:

  1. The claim that prior work ignores the 3D-ViT mismatch should be revised. ViT-Lens resolves this mismatch with latent-query attention, and Ada3D’s post-adapter is similar. The authors need to clarify the novelty of MVA and PCI and verify them via controlled experiments.

2.Table 4 shows better results when adding modules, yet parameters increase from 15.2M to 23.5M. A post-adapter-only baseline and capacity-matched comparisons are required. Comparisons across different positional encodings are also needed to identify the real source of performance gains.

3.Section 4.4 misses key experimental details, hurting reproducibility and fair comparison. Additional ablations are required to isolate the adapter’s effect on segmentation.

4.Figures 5 and 10 only show qualitative examples without quantitative metrics. They cannot prove consistent improvements on spatial reasoning or hallucination reduction. The image-only comparisons in Figure 5 cannot be attributed to the proposed 3D adapters.

I would be glad to provide further clarification if needed.

Reviewer_ggbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Ada3D studies how to adapt pretrained 2D vision transformers for 3D point-cloud representation learning. Its pre-adapter combines multi-view sampling of pretrained positional embeddings with patch-level feature interaction, while a post-adapter uses learnable queries and cross-/self-attention to produce ViT input tokens. The framework is trained through image–text–point contrastive alignment, multiplicative Gaussian token perturbation, and shallow-layer finetuning. Experiments report improvements on three zero-shot classification benchmarks, with additional evaluations on open-vocabulary segmentation and qualitative regional retrieval and multimodal question answering.

Critical strengths

1, Geometry-aware pre-adapter. Multi-view positional embedding sampling and patch-level feature interaction provide a concrete mechanism for incorporating 2D spatial priors into 3D representations.

2,Competitive classification results. Ada3D performs well across three benchmarks, including a 3.3-point improvement over ViT-Lens on ScanObjectNN under the “no LVIS” setting.

3,Useful component analysis. Ablations investigate the adapters, projection-view counts, masking strategies, and finetuning configurations, supporting the effectiveness of the overall pipeline.

Critical weaknesses

1,Insufficient isolation of the claimed innovation. Comparisons with random/shuffled 2D positional embeddings and learned 3D positional encodings are missing, making it unclear whether pretrained 2D priors drive the improvements.

2,Limited incremental benefit from PCI. Adding PCI to MVA plus the post-adapter improves accuracy by only 0.23 points in Table 4, despite introducing additional parameters and computation.

3,Unclear spatial correspondence. The post-adapter uses learned queries that attend globally to point groups, but the paper does not explain how their outputs are mapped back to 3D centers for regional localization.

4,Underspecified segmentation evaluation. The supervision, category splits, metric definitions, and instance-mask generation procedure are insufficiently described, limiting reproducibility and assessment of comparison fairness.

5,Limited evidence for broader capabilities. Regional retrieval and VQA rely on selected qualitative examples, which do not adequately substantiate claims of universal representation or improved spatial reasoning.

6,Ambiguous Gaussian MAE formulation. The described method applies multiplicative Gaussian noise with a contrastive objective, without specifying a reconstruction task. Its distinction from standard noise regularization is therefore unclear.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.