acceptodds
Withdrawn from ICLR 2027

Inject3D: Re-Accessing Dense Geometric Features with Compact Tokens for Fine-Grained 3D Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained 3D understanding requires access to local geometric evidence, yet existing 3D multimodal large language models typically compress dense point clouds into a small set of point tokens before language modeling. This compression can discard geometric details that subsequent LLM layers can no longer directly access, while increasing the point-token budget adds computational and memory overhead and complicates optimization. We introduce Inject3D, which feeds a compact set of spatially anchored point tokens into the LLM and retains high-resolution point-wise features as external geometric memory. At selected LLM layers, each anchor token retrieves features from its local 3D neighborhood and integrates them into its hidden state via cross-attention. To encourage effective use of the retrieved geometric evidence, we further introduce task-specific reinforcement post-training across captioning, visual question answering, and grounding. Experiments on object- and part-level tasks show that Inject3D effectively enhances fine-grained 3D understanding with a compact point-token sequence and substantially lower computational cost than increasing the input token budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLMs, Point Cloud Understanding, Fine-Grained 3D Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G3UjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Inject3D is a 3D multimodal large language model (MLLM) designed to preserve fine-grained geometric details while maintaining a compact visual token sequence. The framework uses a pretrained Utonia point-cloud encoder to extract dense point-wise representations and introduces Anchor-Indexed Feature Re-Access (AFRA), which uses layer-specific learnable queries and cross-attention to inject localized geometric features into selected LLM layers. The authors complement AFRA with GRPO-based task-specific reinforcement post-training across 3D captioning, visual question answering (VQA), and part grounding. Experiments demonstrate competitive performance across several object-centric benchmarks.

Critical strengths

Solid Empirical Performance: Demonstrates strong performance across 3D captioning, VQA, and fine-grained part grounding benchmarks.

Structured Ablations: Provides systematic ablations evaluating geometric injection strategies, token budgets, and reinforcement post-training stages.

Critical weaknesses

1. Limited Methodological Novelty

  • Overlap with Prior Work: AFRA conceptually aligns with existing local-global 3D feature aggregation frameworks, such as PerLA (CVPR 2025) and LSceneLLM (CVPR 2025).
  • Static Feature Retrieval: In Eq. (3), the learnable queries and local neighborhoods are independent of evolving LLM hidden states. Because features are not dynamically conditioned on intermediate LLM representations, they could effectively be precomputed offline.

2. Marginal Incremental Improvements

  • AFRA vs. Input Injection: Table 3 shows that AFRA improves grounding mIoU by only +0.014 over input-level feature injection (0.603 vs. 0.589).

  • Reinforcement Post-Training: Table 2 indicates that GRPO post-training yields a modest +0.011 average grounding mIoU gain (0.638 vs. 0.649), requiring multi-run statistical validation to confirm robustness.

3. Influence of the Pretrained Encoder

  • Strong Encoder Prior: Inject3D relies on the powerful Utonia point-cloud encoder.

  • Confounding Gains: Table 9 shows that replacing Point-BERT with Utonia alone increases grounding mIoU from 0.482 to 0.520 (+0.038) under identical settings. Controlled comparisons with matched backbones are necessary to isolate AFRA's actual contribution.

4. Insufficient Efficiency and Memory Analysis

  • Inference Overhead: Table 3 reports an increase in time to first token (TTFT) from 136.86 ms to 154.76 ms upon introducing AFRA.

  • Missing Peak VRAM: Peak GPU memory consumption is not reported, leaving the true memory footprint of retaining dense point-wise features across multi-layer cross-attention uncharacterized.

5. Limited Evaluation on Large-Scale Point Clouds

  • Object-Centric Scope: Experiments are restricted to object-centric point clouds containing 8,192 points.

  • Unverified Scalability: The effectiveness and computational scalability of fixed local neighborhoods remain unproven for large-scale indoor scenes (e.g., ScanQA, ScanRefer) or outdoor LiDAR point clouds.

Questions for authors
  1. Methodological Novelty: How does AFRA technically and conceptually differ from local geometric aggregation in PerLA or adaptive preference retrieval in LSceneLLM?
  2. Dynamic Feature Access: Given that queries QlQ^l and neighborhoods Na\mathcal{N}_a are independent of current LLM hidden states, why is layer-wise cross-attention superior to precomputing layer-specific features offline?
  3. Statistical Significance: Are the gains observed for AFRA over input-level injection (+0.014 mIoU) and GRPO post-training (+0.011 mIoU) statistically significant across multiple random seeds?
  4. Controlled Baselines: Can you provide baseline comparisons using identical point encoders (Utonia), LLM backbones (Qwen3-VL-4B), and training data to isolate the true contribution of AFRA?
  5. Memory Overhead & Scalability: What is the peak GPU VRAM consumption during inference and training with and without AFRA? How do memory and latency scale when increasing point resolution beyond 8,192 points?

References

  • Guofeng Mei, Wei Lin, Luigi Riz, Yujiao Wu, Fabio Poiesi, Yiming Wang. "PerLA: Perceptive 3D Language Assistant." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
  • Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, Chuang Gan. "LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Reviewer_zMMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Inject3D, a framework for fine-grained 3D understanding that addresses the information loss caused by compressing dense point clouds into a limited number of visual tokens. The proposed Anchor-Indexed Feature Re-Access (AFRA) module enables compact point tokens to retrieve local geometric features from external memory at selected LLM layers. The authors further introduce task-specific reinforcement post-training for 3D captioning, VQA, and part grounding. Experiments demonstrate improvements in fine-grained understanding while maintaining computational efficiency.

Critical strengths
  1. Motivated problem setting. The paper addresses the trade-off between preserving fine-grained geometric information and maintaining a compact visual token sequence, which is important for efficient 3D MLLMs.
  2. Effective geometric feature retrieval. AFRA provides a reasonable mechanism for accessing dense local geometric features without increasing the LLM input sequence length.
  3. Comprehensive experiments. The authors evaluate multiple 3D understanding tasks and provide ablation studies on injection strategies, token budgets, and computational efficiency.
Critical weaknesses
  1. My primary concern is the limited novelty of the proposed framework. Its core components, including spatial neighborhood retrieval, cross-attention, feature injection, and reinforcement post-training, are well-established. Similar intermediate-layer feature injection strategies have been explored in DeepStack and Cambrian-1. The novelty of AFRA beyond adapting existing techniques to 3D data remains unclear.

  2. The experiments focus primarily on object-centric point clouds and part-level understanding, leaving the generalization to complex multi-object scenes and spatial reasoning tasks unexplored.

  3. The improvements over Inject3D-SFT are relatively modest. Comparisons with standard RL objectives and reward designs are needed to better justify the proposed task-specific rewards.

  4. Although the authors leave scene-level understanding for future work, relevant studies on 3D LLMs and scene understanding should still be discussed to better position the proposed method within the broader 3D MLLM literature. Representative works include:

    • Leo-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
    • Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
    • 3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
Questions for authors

What are the key methodological differences between AFRA and existing intermediate-layer visual feature injection methods?

What is the peak memory consumption when storing and repeatedly accessing dense geometric features?

Reviewer_56CZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Inject3D targets fine-grained understanding of colored point clouds by retaining dense point-wise features as external geometric memory. A compact set of spatial anchors forms the LLM input. At selected early layers, AFRA uses layer-specific learned queries to aggregate features from each anchor’s fixed local neighborhood and adds the results to its hidden state. These queries are independent of the current LLM hidden states, and the retrieved features do not extend the main token sequence. Training combines feature alignment, instruction tuning, and task-specific GRPO post-training. Experiments evaluate captioning, VQA, part grounding, and computational efficiency.

Critical strengths
  • The proposed method is intuitive and logically sound.

  • AFRA provides an effective way to preserve access to dense local geometry without expanding the LLM sequence. At the same 512-token budget, it improves grounding mIoU from 0.514 to 0.603 without RL.

  • The controlled token-budget study demonstrates a favorable accuracy–efficiency trade-off: 512-token AFRA achieves comparable grounding performance to the 4096-token baseline with substantially lower prefill cost and first-token latency.

Critical weaknesses
  • Appendix A.5 describes different IoU definitions and mask-to-box conversions for different methods, making the grounding comparisons in Table 2 difficult to interpret.

  • The object-level independence of the ablation validation split is unclear, particularly given multiple annotations and rotation augmentations per object.

  • Although input-only and intermediate injection have similar reported computation, their aggregation capacities are not clearly matched. The specific benefit of injecting features across layers therefore remains partially unresolved.

  • Insufficient positioning against closely related work. LSceneLLM’[1] Scene Magnifier already accesses dense point-cloud features within LLM layers to enrich sparse visual representations. AFRA differs through fixed anchor neighborhoods and learned queries independent of the current hidden states, but the paper neither discusses this closely related work nor compares against its mechanism.

[1] LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences. (CVPR 2025)

Questions for authors
  • Could all grounding methods be evaluated using the same IoU definition and multi-instance matching protocol, with the mask-to-box conversion clarified?

  • Are the ablation validation objects disjoint from training objects, including rotated copies and annotations from other tasks?

  • Does intermediate injection retain its advantage over input-only injection when aggregation capacity and parameter count are matched?

  • Could the authors clarify the core differences between the proposed AFRA and LSceneLLM[1]'s Scene Magnifier?

  • Could the authors discuss training-time geometric supervision methods such as 3D-IDE[2] in the related work, alongside the existing discussion of 3DRS[3]? This would clarify the distinction between learning geometry-aware representations during training and retaining dense geometric features for repeated access during inference.

[1] LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences. (CVPR 2025)

[2] 3D-IDE: 3D Implicit Depth Emergent (CVPR 2026)

[3] 3drs: mllms need 3d-aware representation supervision for scene understanding (NIPS 2025)

I will raise my rating if the authors address my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.