- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReTok3D, a 3D MLLM that reuses the vision encoder of a pretrained 2D VLM as the point cloud encoder. The contributions of this manuscript are twofold: (1) Curve-stride pooling which pools the number of 3D tokens built by VOLT to the limited token budget defined by the 2D VLMs, and performs feature aggregation on the groups; (2) Spectral Ordering module which re-orders the encoded visual tokens by a joint metric of distance and feature similarity, which only has a slight effect of changing the causal attention order between the visual tokens; Effects on downstream language tokens are limited since the RoPE do not change with the spectral order (L261). The model achieves comparable performance with contemporary models.
Overall, it seems that the mature and well-refined writing style does not precisely match the depth and rigorousness of the method and evaluation. While the topic itself is very interesting and promising, the paper failed to provide justifications for many designs and proposed a very complex module with little to none theoretical chance of improvement (Spectral Ordering). Overall, I would not recommend publication of this paper without a substantial overhaul and a more careful choice of the design elements.
Critical strengths
- The paper is well-written in a very detailed with all-round logical guards. While it adds reading burden and increases paper length, this style in and of itself is a good way of writing.
- The path of reusing frozen 2D vision encoders to process 3D point clouds directly is a promising direction which might probably achieve the balance of performance and efficiency, which is sadly less explored since EPCL (Huang et al., 2024)
Critical weaknesses
-
The authors wrote the introduction as if they were the first to investigate so-called ‘third paradigm’ (L54, Figure1 rightmost subfigure). Plenty of papers have investigated this direction, some are cited by the authors in the related work section, including 3D-LLM (Hong et al., 2023), LLaVA-3D (Zhu et al., 2025), EPCL (Huang et al., 2024), and Any2Point (Tang et al., 2024), but the most vital counterpart that also fall into the ‘third paradigm’ (EPCL) is not mentioned in the introduction. There are also some related work that the authors may find as interesting additions to the related work, including LAMM (Yin et al., 2023) which integrates EPCL to indoor VQA task, PointClip (Zhang et al., 2022) which pseudo-renders point cloud as images, and Qwen3D (Lin et al., 2026) which ensembles 2D multiview ViT features into pooled 3D features and adds 3D RoPE. See references below for the exact info of these three papers.
-
The contribution of the geometry-aware tokenization (i.e., the delta upon VOLT, L188-202) is not ablated at all. I have many questions on its design choices, see Q1, Q2, Q3, and Q4. If the optimality of this module can be well-proved, then the authors are recommended to reframe such a pooling strategy as a necessary step for compatibility with VLMs with strict input budget, which could be written as a contribution.
-
Limited contribution of Spectral Ordering. The performance increments of the spectral ordering module is not significant according to Table 3, where applying it upon baseline (compare row 1 to row 3) decreases performance on ScanQA, and deducting it from Full (i.e., comparing row2 with row 4) brings marginal decrease on most metrics.
This can be expected, because the order of input tokens to the LLM decoder mainly affects the attention scheme between the visual tokens, while affecting less on the later language tokens, since the visual tokens’ RoPE are not affected by the spectral ordering (L260). Given an arbitrary permutation of the visual token, the later-generated language tokens attend to all visual tokens regardless. The only slight difference on the language tokens is that the visual token K&V become slightly different due to order-sensitivity of causal self-attention. Therefore, it seems that the second core contribution (the only one with a novelty claim) has limited theoretical and practical value.
References
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36: 26650–26685, 2023.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8542–8552. IEEE, 2022.
Lucy Lin, Ayush Jain, Yifan Liu, and Katerina Fragkiadaki. Qwen-3d: A generalist 3d vision-language model for spatial understanding. In European Conference on Computer Vision, pages 186–204. Springer, 2026.
Questions for authors
- The grouping operation is sensitive to point cloud size (L192-193) and is not experimented on. When the number of points is large, i.e., N>>Rmax, many patches will be assigned to the same group; Otherwise, if N < Rmax, each group will only have one patch which degrades the following distance mask to a no-op. It is critical that the authors provide actual data statistics on the size of the groups which proves rationale for such designs in L194. The authors are also suggested to report such statistics when increasing the number of patches to extremely large numbers (e.g., using whole s3dis buildings as the input)
- The choice rule of is unclear (L195). What is the ‘lower-middle’ location in 3D space?
- Are there any ablation regarding the choice of space-filling curve in curve-stride pooling, e.g., using z-order, or their variants with different coordinate priority?
- The averaging operation (L197) on the raw patch descriptors makes little sense. This operation directly averages the feature (color, normal) of a point with another (or another few) point, located multiples of a fixed stride away in metric 3d space due to being in the same index in a different, neighbouring patch. Besides, some point colors will be averaged with a zero color (zero-filled vacancies, L187) because that other patch does not have a corresponding point in that location within the patch (L187), resulting in the raw colors being darkened. While the patches are indeed highly likely local (due to Hilbert serialization), the relative locations are not exploited for finer-grained feature encoding. It is unclear if the current implementation is optimal; How about adding some encoding of the relative position between these points?
- The definition and use of (L246) is unnecessary because should be already symmetric according to Equation 3, which implies .
- What is the rationale for Spectral Ordering, where highly-ranked tokens are seen first? Maybe because the higher-similarity and widely-connected tokens can be attended to by more other tokens?
- Following the last question, what is the sorting rule of Spectral Ordering, less or more? Specifically, are the larger-score ones placed ahead in the visual token sequence, or is it the reverse order? This should be a very important distinction, but neither did I find such definition anywhere, nor is this distinction ablated in the experiments.
- There seems to be an inappropriate allocation of paper length. The real delta upon existing work, i.e., L188-202 and L230-256, are represented in little space, while less relevant information (e.g., Section 3.2 and L257-L262) might as well be put into the appendix, leaving only a shorter summary of main idea.