acceptodds
Withdrawn from ICLR 2027

What Should 3D-LMMs See? Scene-State Curation for 3D Large Multimodal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in unified 3D Large Multimodal Models (3D-LMMs) has benefited substantially from increasing training-side investment, including large-scale 3D vision-language supervision, stronger cross-modal alignment, and multi-stage training. These advances enable models to acquire rich perceptual evidence from complex 3D scenes. Yet this evidence must still be organized into a shared scene state before it can support downstream reasoning. We identify a systematic mismatch in this process: scene-state construction uses candidate-level perceptual cues to decide which evidence should be prioritized or treated as redundant, but these cues do not necessarily align with the physical structure that the scene state should preserve. In particular, saliency is systematically coupled with physical support, so large-support candidates tend to receive greater representation priority, while semantic similarity alone can cause spatially distinct instances to be treated as redundant, suppressing independent evidence. We therefore propose **Physics-Aware 3D Scene Representation Curation (PARC-3D)**, a training-free framework for scene-state curation. PARC-3D calibrates the support-associated saliency advantage and models redundancy jointly through semantic and spatial criteria, yielding scene states that better preserve relevance, complementarity, and physical separability. Evaluations on referring segmentation, 3D question answering, and dense captioning show gains across all three task families while holding model parameters and scene-state size fixed, with the largest improvements on referring segmentation. These results highlight scene-state curation as a distinct representation axis complementary to data scaling, representation learning, and cross-modal optimization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Large Multimodal Models, Scene Understanding, Point Cloud Processing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6UhNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PARC-3D, a training-free framework for selecting visual candidates for 3D multimodal models. It combines support-size-based saliency calibration, spatial–semantic redundancy estimation, and adaptive greedy selection. Experiments show improvements across referring segmentation, question answering, and dense captioning, with the largest gains on referring segmentation.

Critical strengths
  • Controlled evaluation: Comparisons keep the pretrained checkpoint, candidate pool, scene-state size, and downstream pipeline fixed, isolating the effect of candidate selection.
  • Practical utility: PARC-3D requires no additional training and achieves substantial referring-segmentation gains, particularly on Multi3DRefer.
  • Complementary analyses: Selection baselines, component ablations, and cross-interface experiments provide useful evidence beyond aggregate performance.
Critical weaknesses
  • Limited methodological novelty: The method combines support-based score calibration, spatial–semantic redundancy suppression, and greedy selection. Its distinction from established diversity-aware selectors remains insufficiently demonstrated.
  • Incomplete mechanism validation: Support–saliency correlations do not establish harmful bias, and increased spatial dispersion does not directly demonstrate preservation of distinct instances.
  • Missing targeted ablations: Multi3DRefer shows the largest improvement, but lacks component ablations and single-/multi-target analyses needed to substantiate the proposed explanation.
  • Insufficiently matched baselines: Comparisons do not establish whether MMR or DPP supplied with the same support calibration and spatial–semantic information could achieve similar results.
  • Weak statistical evidence: Variability across heavily overlapping evaluation subsets does not establish the significance of paired improvements over competing methods.
Questions for authors
  1. Can you directly measure whether PARC-3D retains more distinct instances, rather than merely increasing spatial dispersion?
  2. Can you provide component ablations on Multi3DRefer and separate results for single- and multi-target expressions?
  3. What evidence distinguishes harmful support-size bias from a legitimate preference for more informative regions?
Reviewer_RchhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PARC-3D is a training-free method for selecting a fixed-size scene representation from pretrained 3D-LMM candidates. It combines support-size-based saliency calibration, semantic–spatial redundancy estimation with an effective-rank-adaptive scale, and greedy multiplicative selection. Experiments on five benchmarks show improvements primarily in referring segmentation, with smaller gains in question answering and captioning. Additional experiments evaluate alternative selectors, component ablations, computational overhead, and transfer to an object-centric interface.

Critical strengths
  1. Well-controlled improvements without retraining. The main comparison fixes the checkpoint, candidate pool, representation size, and downstream pipeline, isolating scene-state selection as the intervention. The improvements on ScanRefer and Multi3DRefer (+2.9 and +9.4 mIoU) demonstrate meaningful benefits from reorganizing existing evidence.

  2. Useful supporting analysis and transfer evidence. The paper includes relevance–diversity baselines, component and routing-scale ablations, structural diagnostics, and runtime measurements. Applying the same configuration to Chat-Scene produces improvements across all six reported metrics, supporting applicability beyond the primary interface.

Critical weaknesses
  1. The incremental advantage over generic selectors needs stronger validation. Table 2 reports relatively small gains over DPP: 0.5 ScanRefer mIoU and 0.5 ScanQA CIDEr. However, the DPP kernel and baseline tuning protocols are insufficiently specified, and MMR uses semantic rather than joint semantic–spatial redundancy. Comparisons providing generic selectors with the same calibrated utilities and spatial information would clarify the contribution of the proposed selection rule. Multi3DRefer, which shows the largest gain, is also absent from the selector comparisons and component ablations.

  2. The mechanism interpretation is stronger than the direct evidence. Table 4 does isolate VSC’s contribution to downstream performance. However, support-composition changes and small-target gains are measured only for the full method versus vanilla, leaving their attribution to VSC rather than CSR unresolved. Similarly, increased pairwise distances demonstrate dispersion but do not directly establish preservation of distinct same-category instances. These limitations concern the explanation of the gains, not the existence of the reported improvements.

Questions for authors
  1. Can you extend the selector comparisons and VSC/CSR ablations to Multi3DRefer, including zero-, single-, and multiple-target subsets? Which component accounts for its particularly large improvement?

  2. Please specify the DPP kernel, baseline hyperparameters, and tuning protocol. How do MMR/DPP perform when supplied with the same calibrated utilities and joint semantic–spatial information?

  3. Can you report support-size composition and target-size-stratified performance for Full versus w/o VSC? An instance-level coverage analysis would also help distinguish preserved physical instances from greater spatial dispersion.

  4. Can you provide paired scene-level confidence intervals for improvements over vanilla and DPP? Appendix E.5 measures variability in PARC-3D scores across overlapping subsets, rather than uncertainty in the differences between methods.

Reviewer_25uvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates which evidence 3D large multimodal models should retain from existing candidate regions before reasoning. The authors argue that upstream saliency correlates with candidate support size, potentially favoring larger regions in scene-state construction. Meanwhile, identifying redundancy solely through semantic similarity may incorrectly suppress spatially distinct instances.

To address these issues, the authors propose PARC-3D, a training-free framework comprising three steps: attenuating saliency according to each candidate’s relative support size; estimating redundancy using semantic and spatial distances, with the suppression scale adapted through the effective rank of the feature matrix; and constructing a scene state under a fixed budget through iterative selection and multiplicative score decay.

The authors evaluate the framework on five benchmarks using frozen 3D-LLaVA and further test its transfer to Chat-Scene. The results suggest that changing the composition of scene evidence can improve performance on certain 3D understanding tasks without updating model parameters, with the largest gains observed in referring segmentation.

Critical strengths

I consider the reported performance improvements to be the paper’s only substantive strength.

Critical weaknesses
  1. The methodological design itself is problematic. Assigning higher saliency to larger candidate regions does not inherently indicate an undesirable representational bias. Greater support may reflect more reliable observations, more complete objects, or more important scene context. Likewise, the original proportions of small and large regions in the candidate pool do not automatically constitute an appropriate allocation criterion for an ideal scene state.

  2. The paper appears to make extensive use of AI-generated figures. However, the authors seem to disclose only AI-assisted writing and language polishing, without mentioning the use of AI for figure generation.

  3. The baseline models used in the paper are somewhat outdated relative to recent developments in the field, and the evaluation does not adequately reflect the latest advances.

Questions for authors
  1. Method design and motivation: What justifies interpreting higher saliency in larger-support candidates as an undesirable bias rather than a useful signal of observation reliability, object completeness, or contextual importance? On what basis should a monotonic support penalty produce a more appropriate allocation of the scene-state budget?
  2. Baseline selection and applicability: What motivated the choice of 3D-LLaVA and Chat-Scene as evaluation platforms? Please clarify whether the proposed mechanism’s assumptions remain applicable to more recent 3D-LMM interfaces and summarize any relevant comparisons already available.

In addition, substantial portions of the manuscript appear to be AI-generated. I understand that such use may be permitted, but it considerably weakened my initial impression of the paper, as I could not identify clear evidence of careful author revision or thoughtful reasoning behind the design choices in those passages.

I therefore did not raise a long list of specific questions. My concerns are more fundamental: the proposed design is insufficiently developed, the motivation is not clearly established, and the overall reasoning is difficult to follow. These are substantial issues, and I find it difficult to see how my concerns could be adequately addressed during the rebuttal period.

Even the design choices already incorporated into the model lack clear experimental justification. For example, why should high saliency necessarily be regarded as an undesirable bias? And what evidence supports the claim that this bias leads to better scene partitioning?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification of PARC-3D’s Motivation and Contributions

We thank the reviewers for their time and feedback. We would like to clarify two points central to our work.

1. Support-associated representation priority. Our candidate-level audit reveals a systematic association between physical support, upstream saliency, and representation priority. The key distinction lies between candidate-level relevance and scene-level representation requirements: upstream saliency characterizes individual candidates, whereas the shared scene state must organize evidence across different physical scales and spatial locations. Directly inheriting support-associated saliency advantages can therefore leave fine-grained but informative evidence under-represented. VSC addresses this mismatch by calibrating the support-associated saliency advantage while preserving upstream learned relevance.

2. The contribution of scene-state curation. PARC-3D studies how learned perceptual evidence is organized into a shared scene state before downstream reasoning. It addresses two coupled decisions: which evidence receives representation priority and which evidence is considered redundant. VSC calibrates priority, while CSR models redundancy jointly in semantic and physical space, distinguishing semantic similarity from physical redundancy. Residual-utility construction integrates these two signals into the shared scene state. Controlled experiments, component ablations, and cross-interface evaluations support this formulation while keeping the pretrained model and scene-state size fixed within each comparison. These findings highlight scene-state curation as a representation problem complementary to representation learning and cross-modal optimization.

We have decided to withdraw this submission and appreciate the reviewers’ consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.