acceptodds
Under review as a conference paper at ICLR 2027

Point clouds as visual tokens for pretrained multimodal large language models

Abstract

Multimodal large language models (MLLMs) for 3D scene understanding typically rely on multi-view 2D images or dedicated 3D encoders. However, the former incurs heavy visual computation and represents metric geometry indirectly, whereas the latter requires separate feature extractors and alignment with language under limited paired 3D-text supervision. To bridge this gap, we present a compact, geometry-informed 3D tokenizer that directly reuses a pretrained MLLM's visual encoder and connector. Our tokenizer pools volumetric patches into a bounded set of 3D regions, projecting their descriptors into visual tokens via a single linear layer. We use 3D rotary position embeddings (3D-RoPE) to make visual attention sensitive to relative positions between region centroids, alongside spatially constrained spectral ordering to sequence contextualized features for the causal decoder based on geometric proximity and visual similarity. Training proceeds in two stages: first aligning region features with RGB-D teacher targets via cross-partition distillation using shared point membership, then fine-tuning response generation with attention supervision on annotated referents. Experiments across 3D question answering and dense captioning benchmarks demonstrate strong performance with only 3.6-4.0M tokenizer parameters across two pretrained backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D Large Multimodal Model, Vision Language Models, Multi-modal Large Language Models, 3D Tokenization, Geometry-Informed Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n1kfOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReTok3D, a 3D MLLM that reuses the vision encoder of a pretrained 2D VLM as the point cloud encoder. The contributions of this manuscript are twofold: (1) Curve-stride pooling which pools the number of 3D tokens built by VOLT to the limited token budget defined by the 2D VLMs, and performs feature aggregation on the groups; (2) Spectral Ordering module which re-orders the encoded visual tokens by a joint metric of distance and feature similarity, which only has a slight effect of changing the causal attention order between the visual tokens; Effects on downstream language tokens are limited since the RoPE do not change with the spectral order (L261). The model achieves comparable performance with contemporary models.

Overall, it seems that the mature and well-refined writing style does not precisely match the depth and rigorousness of the method and evaluation. While the topic itself is very interesting and promising, the paper failed to provide justifications for many designs and proposed a very complex module with little to none theoretical chance of improvement (Spectral Ordering). Overall, I would not recommend publication of this paper without a substantial overhaul and a more careful choice of the design elements.

Critical strengths
  1. The paper is well-written in a very detailed with all-round logical guards. While it adds reading burden and increases paper length, this style in and of itself is a good way of writing.
  2. The path of reusing frozen 2D vision encoders to process 3D point clouds directly is a promising direction which might probably achieve the balance of performance and efficiency, which is sadly less explored since EPCL (Huang et al., 2024)
Critical weaknesses
  1. The authors wrote the introduction as if they were the first to investigate so-called ‘third paradigm’ (L54, Figure1 rightmost subfigure). Plenty of papers have investigated this direction, some are cited by the authors in the related work section, including 3D-LLM (Hong et al., 2023), LLaVA-3D (Zhu et al., 2025), EPCL (Huang et al., 2024), and Any2Point (Tang et al., 2024), but the most vital counterpart that also fall into the ‘third paradigm’ (EPCL) is not mentioned in the introduction. There are also some related work that the authors may find as interesting additions to the related work, including LAMM (Yin et al., 2023) which integrates EPCL to indoor VQA task, PointClip (Zhang et al., 2022) which pseudo-renders point cloud as images, and Qwen3D (Lin et al., 2026) which ensembles 2D multiview ViT features into pooled 3D features and adds 3D RoPE. See references below for the exact info of these three papers.

  2. The contribution of the geometry-aware tokenization (i.e., the delta upon VOLT, L188-202) is not ablated at all. I have many questions on its design choices, see Q1, Q2, Q3, and Q4. If the optimality of this module can be well-proved, then the authors are recommended to reframe such a pooling strategy as a necessary step for compatibility with VLMs with strict input budget, which could be written as a contribution.

  3. Limited contribution of Spectral Ordering. The performance increments of the spectral ordering module is not significant according to Table 3, where applying it upon baseline (compare row 1 to row 3) decreases performance on ScanQA, and deducting it from Full (i.e., comparing row2 with row 4) brings marginal decrease on most metrics.

    This can be expected, because the order of input tokens to the LLM decoder mainly affects the attention scheme between the visual tokens, while affecting less on the later language tokens, since the visual tokens’ RoPE are not affected by the spectral ordering (L260). Given an arbitrary permutation of the visual token, the later-generated language tokens attend to all visual tokens regardless. The only slight difference on the language tokens is that the visual token K&V become slightly different due to order-sensitivity of causal self-attention. Therefore, it seems that the second core contribution (the only one with a novelty claim) has limited theoretical and practical value.

References

Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36: 26650–26685, 2023.

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8542–8552. IEEE, 2022.

Lucy Lin, Ayush Jain, Yifan Liu, and Katerina Fragkiadaki. Qwen-3d: A generalist 3d vision-language model for spatial understanding. In European Conference on Computer Vision, pages 186–204. Springer, 2026.

Questions for authors
  1. The grouping operation is sensitive to point cloud size (L192-193) and is not experimented on. When the number of points is large, i.e., N>>Rmax, many patches will be assigned to the same group; Otherwise, if N < Rmax, each group will only have one patch which degrades the following distance mask to a no-op. It is critical that the authors provide actual data statistics on the size of the groups Qr\mathcal{Q}_r which proves rationale for such designs in L194. The authors are also suggested to report such statistics when increasing the number of patches to extremely large numbers (e.g., using whole s3dis buildings as the input)
  2. The choice rule of br\textbf{b}_r is unclear (L195). What is the ‘lower-middle’ location in 3D space?
  3. Are there any ablation regarding the choice of space-filling curve in curve-stride pooling, e.g., using z-order, or their variants with different coordinate priority?
  4. The averaging operation (L197) on the raw patch descriptors uq\textbf{u}_q makes little sense. This operation directly averages the feature (color, normal) of a point with another (or another few) point, located multiples of a fixed stride away in metric 3d space due to being in the same index in a different, neighbouring patch. Besides, some point colors will be averaged with a zero color (zero-filled vacancies, L187) because that other patch does not have a corresponding point in that location within the patch (L187), resulting in the raw colors being darkened. While the patches are indeed highly likely local (due to Hilbert serialization), the relative locations are not exploited for finer-grained feature encoding. It is unclear if the current implementation is optimal; How about adding some encoding of the relative position between these points?
  5. The definition and use of W\mathrm{W} (L246) is unnecessary because AA should be already symmetric according to Equation 3, which implies W=(A+A⊺)/2=AW=(A+A^\intercal)/2 = A.
  6. What is the rationale for Spectral Ordering, where highly-ranked tokens are seen first? Maybe because the higher-similarity and widely-connected tokens can be attended to by more other tokens?
  7. Following the last question, what is the sorting rule of Spectral Ordering, less or more? Specifically, are the larger-score ones placed ahead in the visual token sequence, or is it the reverse order? This should be a very important distinction, but neither did I find such definition anywhere, nor is this distinction ablated in the experiments.
  8. There seems to be an inappropriate allocation of paper length. The real delta upon existing work, i.e., L188-202 and L230-256, are represented in little space, while less relevant information (e.g., Section 3.2 and L257-L262) might as well be put into the appendix, leaving only a shorter summary of main idea.
Reviewer_poPUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReTok3D adapts point clouds to a pretrained MLLM’s visual encoder and connector through a lightweight geometry-aware tokenizer. It combines centroid-based 3D positional encoding, spectral ordering of contextualized region features, and two-stage training with RGB-D feature distillation and grounded attention supervision. Experiments on ScanQA, SQA3D, and ScanRefer with two Qwen backbones demonstrate competitive performance using a 3.6–4.0M-parameter tokenizer. Ablations show substantial benefits from distillation and overall improvements from combining attention supervision with spectral ordering.

Critical strengths
  1. Useful interface design. The method connects point clouds to an MLLM’s existing visual encoder and connector without introducing a dedicated deep 3D encoder. Separating geometric positions from host-required token layouts makes this a practical approach to reusing pretrained visual representations.

  2. Consistent evidence for the overall training pipeline. The approach works with two pretrained hosts across three benchmarks. Stage 1 initialization improves all reported metrics at a fixed Stage 2 schedule, while the combined model outperforms its baseline across all three tasks on both hosts.

Critical weaknesses
  1. The benefit of visual-pathway reuse is not isolated. Cross-system comparisons differ in backbone, training data, and adaptation, and there is no same-host direct point-to-language control. Moreover, relevant visual-pathway methods such as LLaVA-3D and Video-3D LLM are discussed but absent from the benchmark tables. Table 8 provides a synthetic multi-view cost reference rather than an accuracy–cost comparison. Consequently, the experiments establish feasibility more clearly than the advantage of the central interface choice.

  2. The evidence for spectral ordering is incomplete. This highlighted contribution is compared only against Hilbert order, without controls separating geometric and feature-based affinities or estimates of run-to-run variability. Its standalone effect is mixed: on Qwen3-VL-8B, it lowers ScanQA CIDEr from 91.5 to 89.4 while adding approximately 0.17 seconds per scene. The combined gains are encouraging, but do not establish when the proposed ordering is preferable to simpler alternatives.

Questions for authors
  1. Interface attribution: Can you provide a same-host direct point-to-language comparison with matched data and region budget? Please also contextualize performance against LLaVA-3D and Video-3D LLM, explicitly identifying protocol differences.

  2. Ordering contribution: Does the spatial-semantic graph consistently outperform geometry-only spectral ordering and alternative geometric serialization? Repeat-run results on the smaller host would help distinguish reliable gains from training variability.

  3. Result consistency: Why does the default 2,048-region configuration obtain 93.4 ScanQA CIDEr in Table 4 but 95.1 in Tables 3 and 5? Please specify any differences in training, checkpoint selection, or evaluation.

Reviewer_FBMAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to feed point clouds directly into the visual encoder and connector of a pretrained multimodal large language model, using its existing visual representations for 3D scene understanding. The authors propose ReTok3D, which first voxelizes point clouds into cubic patches and compresses them into a bounded number of regions through Hilbert-order grouping, distance-based filtering, and pooling. Linear projection and centroid-based 3D positional signals then produce tokens compatible with the pretrained visual pathway. After processing by the visual encoder and connector, the method constructs a graph using spatial proximity and contextualized feature similarity, and determines the region order for the language decoder through recursive spectral partitioning.

Training proceeds in two stages. The first aligns student regions with RGB-D teacher superpoint features across different partitions using shared point membership. The second performs instruction tuning by combining language supervision with attention supervision based on referent-point proportions. Teacher observations and referent annotations are used only during training.

The authors evaluate the method on Qwen2.5-VL-3B and Qwen3-VL-8B. The results indicate that a compact additional point-cloud interface enables the complete pretrained vision–language pathway to support 3D question answering and captioning, achieving competitive SQA3D and ScanRefer captioning results among the compared systems.

Critical strengths
  1. ReTok3D maps irregular point clouds into bounded visual tokens and processes them through a pretrained MLLM’s visual encoder and connector. The separation between token layout and physical coordinates, together with point-membership-based alignment between student regions and teacher superpoints, addresses concrete integration challenges.
  2. The method achieves strong results across three ScanNet-based tasks, including 60.3 EM on SQA3D and 88.0 CIDEr on ScanRefer captioning with Qwen3-VL-8B.
Critical weaknesses
  1. The main benchmark comparisons involve systems with different backbones, pretraining, and adaptation procedures, as the authors acknowledge. Meanwhile, the baselines in Tables 3 and 7 already retain the pretrained visual encoder, connector, and Stage 1 initialization. Consequently, the experiments establish the effectiveness of the complete system but do not isolate the benefit of retaining the visual pathway relative to a matched direct point-to-language interface. This is consequential because visual-pathway reuse is the paper’s central contribution; its incremental value remains unresolved. This is an evidence gap rather than a demonstrated correctness flaw.
  2. Section 3.3 motivates ordering through joint geometric proximity and semantic similarity, but the experiments primarily compare the complete spectral procedure against Hilbert order. They do not separate the contributions of geometric graph structure, semantic edge weights, and sequence permutation. Results are also host- and task-dependent: spectral ordering alone reduces all four ScanQA metrics on the 8B host, including CIDEr from 91.5 to 89.4, while improving all nine metrics on the 3B host. Although the combined method performs well, these observations leave the proposed mechanism and its applicability uncertain.

I also have the impression that the figures throughout the paper may have been generated using AI. While I have no objection to using AI in research or scientific illustration, I believe some elements would benefit from manual refinement to improve their clarity and presentation.

Questions for authors
  1. My main concern is how much benefit comes from reusing the pretrained visual encoder itself. The paper’s central design is to pass point clouds through this encoder, but the comparisons also differ in their backbone models and training data. Some baselines already use visual information, which further complicates my understanding of where the gains come from. I would therefore like to see a more controlled ablation that isolates the contribution of reusing the visual encoder under otherwise comparable conditions.

  2. My second concern comes from the ablation results for spectral ordering, which is presented as one of the paper’s main contributions. In the ScanQA ablation for the 8B model, adding spectral ordering alone reduces performance, whereas combining it with grounded attention produces better results. This leaves me uncertain about the contribution of spectral ordering itself and the rationale for its design. In particular, what would happen if feature similarity were removed and the ordering were constructed using geometric relationships alone? This comparison would help clarify whether the semantic information used in the ordering is actually responsible for the improvement.

I am also concerned that the design rationale for the geometric representation and sequence organization is not sufficiently aligned with the requirements of 3D relational reasoning. Specifically, averaging corresponding voxel slots across multiple patches before encoding and using region centroids for positional information may cause different spatial arrangements of patches to produce identical regional representations. Could this lead the model to prioritize coarse regional geometry while overlooking fine-grained information? Furthermore, spectral ordering favors grouping spatially nearby regions with similar features, whereas 3D understanding often depends on relationships between semantically distinct objects, such as the support relationship between a cup and a table or the directional relationship between a chair and a cabinet. A sequence organized by feature similarity does not necessarily facilitate relational reasoning. The paper therefore needs a clearer explanation of why this ordering principle is appropriate for the spatial relationships required by the target tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.