- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies object-centric 3D multimodal large language models by decomposing their visual interface into three components: object proposals, the underlying 3D representation, and object-token construction. By holding the proposal set fixed, the authors compare dense scene representations, independently encoded object representations, and detector-query representations. They also compare several object-token aggregation mechanisms, including mean/max pooling, MLP-based aggregation, single-query attention, and Perceiver IO. The main empirical conclusion is that the choice of pretrained 3D representation has a substantially larger effect than the complexity of the object-token construction mechanism. Strong frozen point representations combined with simple mean pooling perform competitively across five ScanNet-based benchmarks. The strongest configuration achieves the best reported result on eight of the nine comparable metrics in Table 1.
Critical strengths
-
The object-token-based, object-centric representation formulation presented in this paper leads to substantial performance improvements in 3D understanding. By separating the object proposals from the representation supplied to the language model, the proposed formulation enables these two factors to be studied independently. Its effectiveness is further demonstrated across multiple downstream tasks.
-
The paper is well written, clear, and easy to follow.
-
The ablation studies are well designed. In particular, the object-token construction function is evaluated through comprehensive ablations.
Critical weaknesses
- The core formulation consists of three components: the object proposals , the 3D representation encoder , and the object-token construction function . However, both the object proposal generators and the 3D representation encoders are adopted from existing open-source works, while is investigated using several relatively standard aggregation mechanisms. Consequently, the algorithmic novelty of the paper is limited, and the work does not introduce a novel theory or learning principle for representation learning.
Questions for authors
-
Can the authors provide an experiment using a representation model whose pretraining data excludes ScanNet, or evaluate on a genuinely different scene distribution?
-
What are the proposal recall and oracle grounding performance at IoU thresholds of 0.25 and 0.5?
-
Are the small differences among object-token construction methods stable across multiple random seeds?
-
How much of the improvement from the Vicuna-7B configuration to the Qwen3.5-9B configuration is attributable to language-model capacity rather than the 3D representation?
Response to Reviewer MzoT (Part I)
Thank you for the detailed and constructive comments. We have addressed all your questions below with additional experiments to provide further explanation and clarification.
[W1, Q1]: How does SAVR differ from previous works, such as DeCo, EPIC and 3DRS?
DeCo [1] uses parameter-free 2D adaptive average pooling as the projector of a generic image MLLM (CLIP ViT-L/14 with Vicuna-7B) to reduce the number of visual tokens. It conducts evaluations only on 2D image-text benchmarks. We also utilize adaptive average pooling, but the differences lie in the setting and in how aggressive the token compression can go:
EPIC [2] is a self-distillation method that trains a LLaVA-1.5 to adapt token pruning inside the LLM (FastV, DART, or random pruning) and evaluates the model on 2D image-text benchmarks. EPIC uses one shared MLLM that acts as both teacher and student, while the student undergoes more aggressive and stronger visual token compression and the teacher operates under moderate compression, forcing the student to match the teacher's output distribution during training. Due to this self-distillation mechanism, EPIC can not activate the informative cues if the teacher is not able to highlight or student easily loses that under the aggressive visual token compression. Moreover, the target is the output of the same MLLM that the teacher cannot supply visual information that the MLLM itself has lost or can not encode. Our proposed SAVR instead builds on the dense, per-patch representations of vision foundation models, e.g., VGGT, C-RADIO and Wan2.1. The teachers are frozen expert models that differ from the MLLM and from each other: VGGT (3D geometry), C-RADIOv3 (semantics) and Wan2.1 (video generation). They encode heterogeneous and complementary visual or context information, which is then aligned with their features and pooled over the same region to be distilled to the student for the 3D task. At 64 token retention (11.1%), the distillation term of EPIC raises the average score from 58.1 / 58.7 to 59.4 (EPIC Tables 3–4, TCD / LCD), while the full-token model scores 61.4, leading to 26–39% of the remaining gap. However, our SAVR improves the averaged 3D tasks performance up to 99.63%, relatively close to the vanilla model. Moreover, EPIC employs token compression within LLM layers before the pruning layer, which incurs substantial computation overhead, while our SAVR does not.
3DRS shows that a feed-forward 3D geometry model (VGGT or FLARE) can improve a 3D MLLM performances. 3DRS explores only whether 3D supervision improves a full-token 3D model while leaving several questions that SAVR studies. Firstly, 3DRS processes all visual tokens of its dense sampled video frames and does not penalize visual token efficiency. Secondly, 3DRS compares 3D teachers (e.g., VGGT, FLARE) with 2D teachers (DINOv2, MAE, SigLIP), and concludes that 3D geometry teachers are best. But we explore several representative vision foundation models that contain heterogeneous and complementary information, e.g., VGGT provides geometry-aware and cross-view information, C-RADIO contributes general dense semantic cues, and Wan2.1 supplies spatio-temporal and generative geometry priors. Our SAVR instead penalizes visual token inefficiency under the dense multi-view input and investigates the redundancy and compression capacity in terms of visual token space.