- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether 3D MLLMs use explicit 3D positional information. The authors find that shuffling or zeroing positional inputs leads to minimal performance degradation across five models, despite probing results showing that relative 3D offsets can be recovered from the hidden states of four models. Motivated by this observation, the authors propose Anchor3D, which masks non-target visual tokens to create a question-conditioned information bottleneck. Anchor3D achieves 44.5 on R3D-Bench, compared with 32.6 without the bottleneck, and reaches scores comparable to GPT-5.5 and Gemini 3.1 Pro. Furthermore,shuffling its positions reduces the score to 33.5, demonstrating increased reliance on positional information.
Critical strengths
- Systematic investigation of positional dependence. The paper evaluates five 3D MLLMs with different geometry injection mechanisms across multiple benchmarks. Position shuffling, zero-position controls, and text-only comparisons provide complementary evidence about their reliance on explicit 3D inputs.
- Effective intervention supported by controlled ablations. Anchor3D introduces a straightforward localization bottleneck while reusing an existing positional encoding. With the same backbone and training data, it improves R3D-Bench performance from 32.6 to 44.5; positional shuffling then reduces its score to 33.5, supporting increased dependence on the supplied coordinates.
Critical weaknesses
- The source of the localization bottleneck’s gains is not fully isolated. The ablation in Table 5 does not distinguish explicit grounding from restricting non-target information. A full-scene baseline with target markers, using the same backbone, training data, and localization supervision, would help clarify whether background masking is necessary.
Questions for authors
- The performance changes in Tables 1 and 2 are often small. Could the authors report means and standard deviations across multiple shuffling seeds?
- Anchor3D uses Grounding DINO, SAM2, and SigLIP for preprocessing. Could the authors clarify whether “without tool use” specifically refers to the absence of dynamic tool calls by the LLM?
4 comments
Author Response to Reviewer MBEP (2/2)
2. Means and standard deviations across multiple shuffling seeds for the performance changes in Tables 1 and 2
The performance changes in Tables 1 and 2 are often small. Could the authors report means and standard deviations across multiple shuffling seeds?
Response: We thank the reviewer for this suggestion. We ran the 3D position shuffle with two additional random permutation seeds (Seeds 2 and 3) beyond the original seed reported in Table 1 (Seed 1) for all five models on ScanQA (4,675 questions) and SQA3D (3,519 questions). The table below reports the EM@1 results for the original seed and the two new seeds together with the mean standard deviation across all three seeds:
| Model | ScanQA Seeds 1 / 2 / 3 | ScanQA Mean SD | SQA3D Seeds 1 / 2 / 3 | SQA3D Mean SD |
|---|---|---|---|---|
| LLaVA-3D | 26.3 / 26.20 / 26.40 | 26.30 0.10 | 54.4 / 54.96 / 54.73 | 54.70 0.28 |
| Video-3D LLM | 30.1 / 29.58 / 29.43 | 29.70 0.35 | 57.8 / 58.00 / 58.28 | 58.03 0.24 |
| Spatial-MLLM | 23.5 / 23.89 / 24.49 | 23.96 0.50 | 53.2 / 53.42 / 53.06 | 53.23 0.18 |
| Proxy3D | 24.9 / 25.07 / 24.75 | 24.91 0.16 | 56.2 / 55.41 / 55.07 | 55.56 0.58 |
| QuatRoPE | — | — | 53.2 / 54.22 / 54.33 | 53.92 0.62 |
(1) High stability across random permutation seeds. Across all evaluated models and both benchmarks, the standard deviation across the three shuffling seeds is at most 0.50 EM@1 points on ScanQA (–) and 0.62 EM@1 points on SQA3D (–).
(2) Comparison with scene removal and Anchor3D's shuffle drop. By contrast, removing the visual scene altogether (Text-only in Table 1) reduces EM@1 by – points, and applying the same position shuffle to Anchor3D reduces its score by 11.0 points on R3D-Bench and 3.9 EM on SQA3D (Table 5)—well beyond the seed-to-seed standard deviation.
3. Clarification of "without tool use" regarding dynamic tool calls by the LLM
Anchor3D uses Grounding DINO, SAM2, and SigLIP for preprocessing. Could the authors clarify whether "without tool use" specifically refers to the absence of dynamic tool calls by the LLM?
Response: Yes—"without tool use" specifically refers to the absence of dynamic tool calls and externally computed geometric measurements provided to the LLM. We clarify this distinction and our manuscript update below:
(1) Definition of "tool use" following R3D-Bench.
We follow the definition of tool use in R3D [1], where explicit spatial measurement functions (e.g., get_distance, get_object_size, get_object_volume) are exposed to the LLM as callable function definitions; the LLM invokes them dynamically over multiple reasoning steps and receives pre-computed geometric quantities (in meters or cubic meters) as text outputs. "Without tool use" means that the LLM makes no such function calls and receives no computed geometric measurement.
(2) Role of perception modules in Anchor3D. In Anchor3D, Grounding DINO, SAM2, and the SigLIP-based matcher form a fixed visual preprocessing stage executed once per question rather than dynamically invoked by the LLM. Crucially, they only determine which visual tokens are retained; they compute and return no distance, size, or volume. The metric answer is predicted directly by the LLM from the retained visual tokens and their 3D positional encodings. R3D makes the exact same distinction, noting that 3D LLMs operating in latent space typically still rely on external 2D/3D object detectors or segmenters to provide input representations.
(3) Manuscript revision. We agree with the reviewer that this term was not explicitly defined in the original submission and that Anchor3D relies on external perception modules during preprocessing (and thus is not a single end-to-end network from raw pixels). We have added this explicit definition to Section 5.1 and the caption of Table 3 (marked in red).
References
[1] R3D-Bench: Benchmarking Quantitative 3D Spatial Reasoning from Egocentric RGB-D Videos.
Thank you for your detailed rebuttal. Your explanations have addressed my concerns and questions. Therefore, I will maintain my positive score.
Thank you so much for your prompt reply and for your positive support of our paper, Reviewer MBEP! We are glad that our responses addressed all of your questions, and we truly appreciate your constructive suggestions throughout the review process.
Author Response to Reviewer MBEP (1/2)
We sincerely appreciate Reviewer
MBEPfor the positive assessment and constructive suggestions, and for recognizing the value of our systematic investigation across five 3D MLLMs and the controlled ablations supporting Anchor3D.As a brief overarching note: our work is primarily a diagnostic study, and Anchor3D is intended as an initial, proof-of-concept intervention rather than a final architectural standard. Our goal is to expose what current 3D LLMs lack—binding the entities named in a question to their visual tokens and 3D coordinates—and to clarify what is required to make models consult their injected 3D positions. All wording clarifications and definitions added to the revised manuscript are marked in red. Below, we address each concern and question in detail.
1. Full-scene baseline with target markers under matched settings to assess the necessity of background masking
Response: We thank the reviewer for suggesting this exact control experiment. To isolate the effect of explicit target grounding from restricting non-target background tokens, we trained and evaluated a full-scene target-marker baseline (Highlight) using the exact same Qwen3.5-4B backbone, training data, and target masks as Anchor3D: instead of zeroing out non-target tokens, all 4,096 scene tokens are kept visible and the matched target tokens are explicitly marked with a learned highlight indicator. The table below compares Highlight on R3D-Bench (all 3,033 questions across 15 question types) against the unmarked full-scene model retrained on the same GPU machine for a strictly matched side-by-side comparison, as well as Anchor3D:
(1) Overall comparison: soft target markers alone are insufficient. Marking the target tokens while keeping the background visible yields almost no overall gain over the unmarked full-scene baseline (35.4 vs. 35.3 overall; 25.5 vs. 23.8 on distances), whereas masking out the non-target background tokens in Anchor3D improves the overall score by +9.1 points (to 44.5) and the distance average by +14.1 points (to 39.6).
(2) Per-type breakdown: why soft markers fail when the background remains visible. Examining individual question types clarifies why soft target markers are insufficient:
(3) Diagnostic takeaway. This control experiment directly supports our central diagnostic thesis: even when explicit grounding markers tell the LLM which visual tokens correspond to the named objects, keeping the remaining scene tokens visible allows the model to fall back on global scene appearance and contextual shortcuts rather than reading the target 3D coordinates. Restricting non-target information via a hard bottleneck is therefore necessary to force coordinate readout in current architectures, and designing training objectives or attention mechanisms that prevent this shortcut without hard background masking is a key direction for future 3D LLMs.