acceptodds
Withdrawn from ICLR 2027

From 3D Representations to Object Tokens in Object-Centric MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) provide a general interface for perception and language-based reasoning, yet it remains unclear how the underlying 3D representation affects the performance. Existing systems often couple object discovery, representation source, and subsequent object-token construction, making their individual contributions difficult to isolate. In this paper, we study these factors under a controlled object-centric formulation that separates object proposals from the representation used for downstream reasoning, and compare multiple representation sources and object-token construction mechanisms. Under our strongest configuration, the resulting model surpasses prior point-cloud-only methods on most comparable metrics across five ScanNet benchmarks. Our experiments further show that object discovery and object representation can be treated as distinct design choices, and that the choice of frozen 3D representation has a substantial effect on downstream performance, whereas increasing the complexity of the object-token construction mechanism provides only limited and task-dependent gains. These results highlight representation source as a central design factor in object-centric 3D MLLMs and show that effective object representations can be constructed with comparatively lightweight aggregation mechanisms.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D multimodal large language models, object-centric 3D understanding, 3D representations, point foundation models, object-token construction, 3D grounding, 3D question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ndNNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

I used AI to assist with searching the relevant literature in order to assess whether the related-work comparisons were sufficiently comprehensive and whether the reported results indeed achieved state-of-the-art performance on the public benchmarks. However, all assessments and opinions expressed in the review, as well as the review itself, were written independently by me.

Critical strengths
  1. The object-token-based, object-centric representation learning presented in this paper leads to substantial performance improvements in 3D understanding. By separating the object proposals from the representation supplied to the language model, the proposed formulation enables these two factors to be studied independently. Its effectiveness is further demonstrated across multiple downstream tasks.

  2. The paper is well written, clear, and easy to follow.

  3. The ablation studies are well designed. In particular, the object-token construction function gθg_\theta is evaluated through comprehensive ablations.

Critical weaknesses

The core formulation consists of three components: the object proposals MM, the 3D representation encoder ϕ\phi, and the object-token construction function gθg_\theta. However, both the object proposal generators and the 3D representation encoders are adopted from existing open-source works, while gθg_\theta is investigated using several relatively standard aggregation mechanisms. Consequently, the algorithmic novelty of the paper is limited, and the work does not introduce a novel theory or learning principle for representation learning.

Questions for authors

All five evaluation benchmarks are based on ScanNet. Therefore, the experiments demonstrate generalization across tasks, but not across data distributions. The paper also states that region descriptions of ScanNet instances are used during the object-token alignment stage. Could the observed performance gains be partly attributable to using alignment data drawn from the same underlying dataset or distribution as the evaluation benchmarks?

Reviewer_vLmPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how 3D features and object-token construction affect object-centric MLLMs. It keeps object proposals fixed and compares features from object encoders, detection backbones, and point foundation models. It also compares pooling and attention-based methods for turning point features into object tokens. Across five ScanNet-based benchmarks, point foundation features perform best, while mean pooling remains competitive with more complex aggregation methods.

Critical strengths
  1. Keeping object proposals fixed makes the representation comparison useful. The results show that proposal features do not need to come from the model that detects the objects. Features from separately pretrained point models consistently outperform the tested detector features.

  2. Mean pooling provides a strong and simple baseline. The paper compares several aggregation methods under the same setup and shows that adding more parameters does not consistently improve performance. This is useful guidance for building object-centric 3D MLLMs.

Critical weaknesses
  1. The point foundation models were pretrained on ScanNet scenes across dataset splits, including evaluation scenes. The authors acknowledge this, and it does not mean downstream language labels were used. However, it leaves unclear how much of the advantage comes from better transferable features versus prior exposure to the evaluated scenes. Since all five benchmarks use ScanNet, the paper does not resolve this question with an independent dataset or a checkpoint trained without evaluation scenes.

  2. The conclusion about aggregation complexity is based on one backbone and one training setup. All aggregation modules are frozen during instruction tuning, which may limit the benefit of learned aggregation. Table 3 also reports small differences without results across seeds or uncertainty estimates. The evidence supports mean pooling as a good baseline in this setup, but is not yet enough to establish when more complex aggregation is unnecessary.

Questions for authors
  1. Can you provide results on scenes excluded from backbone pretraining, or with a checkpoint trained without the evaluation scenes? Please also clarify the scene overlap for each representation source.

  2. What happens if the aggregation module remains trainable during instruction tuning? Does the ranking in Table 3 hold with another backbone?

  3. How much do the Table 3 results vary across training seeds? Are the differences between mean pooling and attention consistent?

  4. Section 4.1.1 mentions an alternative proposal-source ablation using Mask3D, but I could not find its results in the provided PDF. Do the representation rankings hold with Mask3D proposals?

Reviewer_2fXPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how 3D feature selection and aggregation influence performance in object-centric multimodal large language models. The authors compare the choice of 3D encoder and features under fixed object proposals, then compare feature aggregation methods using the same frozen encoder. The resulting object representations are provided to the language model as a sequence of object identifiers and visual tokens. Across five ScanNet-based benchmarks, features from pretrained 3D encoders (Sonata, Concerto, and Utonia) outperform detector backbone features, detector query features, and object features extracted separately for each proposal using Uni3D. attention blocks and Perceiver IO for feature aggregation provide limited gains over mean pooling. The authors also report higher scores than prior point-cloud-only methods.

Critical strengths
  1. The paper is well organized, making the methodology and experimental comparisons easy to follow.
  2. The discussion of prior work provides useful context for understanding object-centric 3D multimodal large language models.
Critical weaknesses
  1. The paper primarily combines and compares existing encoders, aggregation methods, and language-model components. The reviewer finds the contribution beyond these comparisons unclear.
  2. The finding that stronger pretrained encoders improve downstream performance is unsurprising without further analysis of what drives the gains. Several baseline MLLMs also use Vicuna-7B (given that MLLM performance depends heavily on the underlying LLM), and comparisons with more recent models would help establish the relevance of the findings.
  3. The finding that learned aggregation methods offer limited gains over mean pooling depends on freezing the aggregation module during instruction tuning. This prevents joint adaptation with the language model and may affect the relative performance of these methods. So, the authors could consider jointly adapting the 3D encoder, multimodal projector, and language model, potentially using LoRA and a lower encoder learning rate.
  4. As acknowledged in lines 419–424, the evaluated point foundation models were pretrained on ScanNet scenes across dataset splits. The reviewer is concerned that prior exposure to evaluation scenes may partly explain their performance. Although the paper highlights confounded comparisons in prior work, its own comparisons are subject to a similar concern.
Questions for authors
  1. Would the authors consider adjusting the layout to keep the conclusion within the nine-page limit?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.