acceptodds
Under review as a conference paper at ICLR 2027

Anchor3D: Do 3D LLM Really Use 3D Positions?

Abstract

Recent 3D large language models (3D LLMs) inject explicit geometry into a language backbone through 3D positional embeddings, voxel-aligned features, or specialized rotary encodings, and report steady gains on 3D question answering and grounding benchmarks. We ask whether these models actually use the geometry they are given. Across five representative 3D LLMs, including a model with a dedicated 3D rotary encoding, we find that randomly permuting the 3D positions assigned to visual tokens leaves performance essentially unchanged across the benchmarks we evaluate, and so does setting them all to zero. Simple MLP probes show that, the LLM's hidden states can still carry accurate relative 3D offsets between tokens, so the geometry is encoded but not consulted: current benchmarks can be solved from appearance and language cues, and a purely autoregressive decoder is never forced to bind the entities named in a question to their 3D locations. Motivated by this, we propose to make question-conditioned localization an explicit bottleneck. Our model first detects and segments the objects named in the question, then selects one mask per named object with a question-conditioned matcher, and passes only the patch tokens covered by these masks, together with their 3D positions, to the LLM for metric reasoning. Trained on ScanQA, SQA3D, and R3D-Bench-style metric questions built on ScanNet, the resulting model, Anchor3D, becomes sensitive to positional perturbation, matches GPT-5.5 and Gemini 3.1 Pro on R3D-Bench with 4B parameters, and improves ScanQA and SQA3D over the same pipeline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Large Language Models, 3D Positional Encoding, Spatial Reasoning, Probing, 3D Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MBEPOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether 3D MLLMs use explicit 3D positional information. The authors find that shuffling or zeroing positional inputs leads to minimal performance degradation across five models, despite probing results showing that relative 3D offsets can be recovered from the hidden states of four models. Motivated by this observation, the authors propose Anchor3D, which masks non-target visual tokens to create a question-conditioned information bottleneck. Anchor3D achieves 44.5 on R3D-Bench, compared with 32.6 without the bottleneck, and reaches scores comparable to GPT-5.5 and Gemini 3.1 Pro. Furthermore,shuffling its positions reduces the score to 33.5, demonstrating increased reliance on positional information.

Critical strengths
  • Systematic investigation of positional dependence. The paper evaluates five 3D MLLMs with different geometry injection mechanisms across multiple benchmarks. Position shuffling, zero-position controls, and text-only comparisons provide complementary evidence about their reliance on explicit 3D inputs.
  • Effective intervention supported by controlled ablations. Anchor3D introduces a straightforward localization bottleneck while reusing an existing positional encoding. With the same backbone and training data, it improves R3D-Bench performance from 32.6 to 44.5; positional shuffling then reduces its score to 33.5, supporting increased dependence on the supplied coordinates.
Critical weaknesses
  • The source of the localization bottleneck’s gains is not fully isolated. The ablation in Table 5 does not distinguish explicit grounding from restricting non-target information. A full-scene baseline with target markers, using the same backbone, training data, and localization supervision, would help clarify whether background masking is necessary.
Questions for authors
  1. The performance changes in Tables 1 and 2 are often small. Could the authors report means and standard deviations across multiple shuffling seeds?
  2. Anchor3D uses Grounding DINO, SAM2, and SigLIP for preprocessing. Could the authors clarify whether “without tool use” specifically refers to the absence of dynamic tool calls by the LLM?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer MBEP (1/2)

We sincerely appreciate Reviewer MBEP for the positive assessment and constructive suggestions, and for recognizing the value of our systematic investigation across five 3D MLLMs and the controlled ablations supporting Anchor3D.

As a brief overarching note: our work is primarily a diagnostic study, and Anchor3D is intended as an initial, proof-of-concept intervention rather than a final architectural standard. Our goal is to expose what current 3D LLMs lack—binding the entities named in a question to their visual tokens and 3D coordinates—and to clarify what is required to make models consult their injected 3D positions. All wording clarifications and definitions added to the revised manuscript are marked in red. Below, we address each concern and question in detail.


1. Full-scene baseline with target markers under matched settings to assess the necessity of background masking

The source of the localization bottleneck's gains is not fully isolated. The ablation in Table 5 does not distinguish explicit grounding from restricting non-target information. A full-scene baseline with target markers, using the same backbone, training data, and localization supervision, would help clarify whether background masking is necessary.

Response: We thank the reviewer for suggesting this exact control experiment. To isolate the effect of explicit target grounding from restricting non-target background tokens, we trained and evaluated a full-scene target-marker baseline (Highlight) using the exact same Qwen3.5-4B backbone, training data, and target masks as Anchor3D: instead of zeroing out non-target tokens, all 4,096 scene tokens are kept visible and the matched target tokens are explicitly marked with a learned highlight indicator. The table below compares Highlight on R3D-Bench (all 3,033 questions across 15 question types) against the unmarked full-scene model retrained on the same GPU machine for a strictly matched side-by-side comparison, as well as Anchor3D:

SettingMC AvgDist AvgVol AvgOverall Avg
Full scene without markers64.023.814.235.3
Full scene with target markers63.825.510.935.4
Anchor3D67.339.617.844.5

(1) Overall comparison: soft target markers alone are insufficient. Marking the target tokens while keeping the background visible yields almost no overall gain over the unmarked full-scene baseline (35.4 vs. 35.3 overall; 25.525.5 vs. 23.823.8 on distances), whereas masking out the non-target background tokens in Anchor3D improves the overall score by +9.1+9.1 points (to 44.5) and the distance average by +14.1+14.1 points (to 39.6).

(2) Per-type breakdown: why soft markers fail when the background remains visible. Examining individual question types clarifies why soft target markers are insufficient:

  • The model does learn to read the target markers: On How Far (object-to-object distance), adding target markers jumps MRA from 25.225.2 to 42.7 (+17.5+17.5 points), matching Anchor3D (42.542.5) and confirming that the LLM successfully recognizes the highlight indicators.
  • However, visible background tokens re-enable appearance and contextual shortcuts on other tasks: On almost all other metric and relational question types, the target-marker model remains close to the unmarked baseline and far below Anchor3D—for example, on How Long (4.04.0 vs. 31.231.2), Much Taller (17.417.4 vs. 36.636.6), Much Longer (19.319.3 vs. 35.035.0), Far From Me (20.720.7 vs. 35.735.7), Walk Distance (35.835.8 vs. 48.448.4), and Nearest (40.440.4 vs. 60.560.5).

(3) Diagnostic takeaway. This control experiment directly supports our central diagnostic thesis: even when explicit grounding markers tell the LLM which visual tokens correspond to the named objects, keeping the remaining scene tokens visible allows the model to fall back on global scene appearance and contextual shortcuts rather than reading the target 3D coordinates. Restricting non-target information via a hard bottleneck is therefore necessary to force coordinate readout in current architectures, and designing training objectives or attention mechanisms that prevent this shortcut without hard background masking is a key direction for future 3D LLMs.

AuthorsOpenReview ↗

Author Response to Reviewer MBEP (2/2)


2. Means and standard deviations across multiple shuffling seeds for the performance changes in Tables 1 and 2

The performance changes in Tables 1 and 2 are often small. Could the authors report means and standard deviations across multiple shuffling seeds?

Response: We thank the reviewer for this suggestion. We ran the 3D position shuffle with two additional random permutation seeds (Seeds 2 and 3) beyond the original seed reported in Table 1 (Seed 1) for all five models on ScanQA (4,675 questions) and SQA3D (3,519 questions). The table below reports the EM@1 results for the original seed and the two new seeds together with the mean ±\pm standard deviation across all three seeds:

ModelScanQA Seeds 1 / 2 / 3ScanQA Mean ±\pm SDSQA3D Seeds 1 / 2 / 3SQA3D Mean ±\pm SD
LLaVA-3D26.3 / 26.20 / 26.4026.30 ±\pm 0.1054.4 / 54.96 / 54.7354.70 ±\pm 0.28
Video-3D LLM30.1 / 29.58 / 29.4329.70 ±\pm 0.3557.8 / 58.00 / 58.2858.03 ±\pm 0.24
Spatial-MLLM23.5 / 23.89 / 24.4923.96 ±\pm 0.5053.2 / 53.42 / 53.0653.23 ±\pm 0.18
Proxy3D24.9 / 25.07 / 24.7524.91 ±\pm 0.1656.2 / 55.41 / 55.0755.56 ±\pm 0.58
QuatRoPE——53.2 / 54.22 / 54.3353.92 ±\pm 0.62

(1) High stability across random permutation seeds. Across all evaluated models and both benchmarks, the standard deviation across the three shuffling seeds is at most 0.50 EM@1 points on ScanQA (0.100.10–0.500.50) and 0.62 EM@1 points on SQA3D (0.180.18–0.620.62).

(2) Comparison with scene removal and Anchor3D's shuffle drop. By contrast, removing the visual scene altogether (Text-only in Table 1) reduces EM@1 by 5.15.1–12.812.8 points, and applying the same position shuffle to Anchor3D reduces its score by 11.0 points on R3D-Bench and 3.9 EM on SQA3D (Table 5)—well beyond the seed-to-seed standard deviation.


3. Clarification of "without tool use" regarding dynamic tool calls by the LLM

Anchor3D uses Grounding DINO, SAM2, and SigLIP for preprocessing. Could the authors clarify whether "without tool use" specifically refers to the absence of dynamic tool calls by the LLM?

Response: Yes—"without tool use" specifically refers to the absence of dynamic tool calls and externally computed geometric measurements provided to the LLM. We clarify this distinction and our manuscript update below:

(1) Definition of "tool use" following R3D-Bench. We follow the definition of tool use in R3D [1], where explicit spatial measurement functions (e.g., get_distance, get_object_size, get_object_volume) are exposed to the LLM as callable function definitions; the LLM invokes them dynamically over multiple reasoning steps and receives pre-computed geometric quantities (in meters or cubic meters) as text outputs. "Without tool use" means that the LLM makes no such function calls and receives no computed geometric measurement.

(2) Role of perception modules in Anchor3D. In Anchor3D, Grounding DINO, SAM2, and the SigLIP-based matcher form a fixed visual preprocessing stage executed once per question rather than dynamically invoked by the LLM. Crucially, they only determine which visual tokens are retained; they compute and return no distance, size, or volume. The metric answer is predicted directly by the LLM from the retained visual tokens and their 3D positional encodings. R3D makes the exact same distinction, noting that 3D LLMs operating in latent space typically still rely on external 2D/3D object detectors or segmenters to provide input representations.

(3) Manuscript revision. We agree with the reviewer that this term was not explicitly defined in the original submission and that Anchor3D relies on external perception modules during preprocessing (and thus is not a single end-to-end network from raw pixels). We have added this explicit definition to Section 5.1 and the caption of Table 3 (marked in red).

References

[1] R3D-Bench: Benchmarking Quantitative 3D Spatial Reasoning from Egocentric RGB-D Videos.

Thank you for your detailed rebuttal. Your explanations have addressed my concerns and questions. Therefore, I will maintain my positive score.

Thank you so much for your prompt reply and for your positive support of our paper, Reviewer MBEP! We are glad that our responses addressed all of your questions, and we truly appreciate your constructive suggestions throughout the review process.

Reviewer_JyHwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates into a question about whether the current 3D LLMs really use the 3D positions encoded into the inputs. The authors first find out that randomly permuting or zeroing out the 3D positions encoded for the visual tokens has little impact on the performance of current 3D LLMs on the 3D VQA or grounding tasks. Furthermore, the authors find out that the 3D information is actually encoded in the features, but just not used by the 3D LLMs to answer the questions. To solve this issue, the paper proposes Anchor3D, a method to finetune the 3D LLMs to focus only on the visual tokens that are mentioned in the questions, to force the model to reason over the regions corresponding to the questions. Experimental results demonstrate that the proposed solution can indeed make 3D LLMs more sensitive to the input 3D position information, indicating that they get instructed to use the information.

Critical strengths
  • The investigation provided by the paper is valuable for a deeper understanding of the spatial reasoning capability of 3D LLMs. Previously there are existing studies like [1, 2] that find out that the visual input is problematic as the 3D LLMs can ignore the visual input and still get decent results on many questions and tasks. Compared to the existing works, this paper reveals a more detailed aspect for the problem in the visual input, which lies 3D LLMs’ leverage for the 3D positions. The diagnosis of the 3D positional information gets encoded but not used can be very helpful to the community, and can further serve as method design guidelines for future works.

  • Besides the diagnosis, the paper also proposes a method Anchor3D to fix this issue. With experimental results compared with existing methods (Table 3) and ablation studies (Table 5), Anchor3D is demonstrated to force the 3D LLMs to bind the concepts in the questions to the correct parts in the scene, and thus answer the questions more accurately.

The above two points are the critical strengths. The other minor strengths include the clear presentation of the paper, and the detailed explanation on the implementation to foster reproducibility.

[1] Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs. arXiv:2506.05318.

[2] Do 3D Large Language Models Really Understand 3D Spatial Relationships? ICLR 2026.

Critical weaknesses
  • Although Anchor3D serves as a remedy to fix the issue of 3D LLMs not using 3D positional information, the proposed method seems to use the "matcher" module to enable the 3D positional/spatial understanding capability, instead of really empowering the 3D LLMs the capability to use the positional information. Basically, the proposed "matcher", which is a preprocessing module prior to the input of 3D LLMs, is responsible of binding concepts mentioned in the questions to the visual tokens representing the 3D scene. Then, the unrelated part will be zeroed out so they cannot interfere with the reasoning process of 3D LLMs. This does not seem like a fundamental fix to the diagnosed problem, as the current capability is powered by the preprocessing module, instead of granting the 3D LLMs to understand the different positional encoding values from all visual tokens.

  • The solution proposed by Anchor3D does not feel like an ultimate method for effectively processing 3D positions, and instead feels like a temporary fix. In the current implementation, the localization is forced by setting the visual tokens that do not correspond to concepts in the questions to zero, but the method still feeds all these zero-value vectors into the model for processing, which costs waste for compute. Although I understand that this paper is not for efficiency improvement, but this design which causes redundant computation may not be an ultimate design choice for future 3D LLMs.

The above two points are the critical weaknesses. The other minor weaknesses include that it seems to have performance compromise on standard 3D VQA benchmarks like ScanQA and SQA3D (Table 4) after finetuning with Anchor3D.

Questions for authors
  • Is the purpose of Anchor3D to exclude the unrelated visual tokens so the 3D LLMs will not focus on these tokens, or it is targeting at empowering 3D LLMs to understand, and furthermore, utilize the 3D positional information encoded in the feature? If the purpose is the latter (which makes better sense for this paper’s motivation), then the designed Anchor3D seems not to well accomplish this goal, because it seems to enforce this by only allowing visual tokens that are related to the questions into the model.

  • Are there other underlying reasons for not discarding the zero-out token besides the reason mentioned in the paper for "the sequence length and the position IDs of the text being unchanged" (Line 331)? Because the current solution does not seem like a permanent efficient solution and cannot serve as future standard.

  • What is the inference latency introduced by the preprocessing steps like Grounding DINO and SAM2, and the matcher before processing by the 3D LLMs?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer JyHw (1/2)

We sincerely appreciate Reviewer JyHw for the positive and constructive feedback, and especially for recognizing the value of our diagnostic finding—that 3D positional information is encoded inside current 3D LLMs but not consulted at prediction time—as well as the clarity and reproducibility of our paper.

We fully share the reviewer's perspective on the role of Anchor3D: our work is primarily a diagnostic study, and Anchor3D is intended as an initial, proof-of-concept remedy rather than an ultimate architectural standard. Its purpose is to demonstrate what current 3D LLMs lack—binding the entities named in the question to their visual tokens and 3D coordinates—and to motivate future 3D LLMs that learn this binding internally. All corresponding clarifications in the revised manuscript are marked in red. Below, we address each question in detail.


1. Unclear distinction between filtering irrelevant visual tokens and improving the understanding and use of 3D positional information

Is the purpose of Anchor3D to exclude the unrelated visual tokens so the 3D LLMs will not focus on these tokens, or it is targeting at empowering 3D LLMs to understand, and furthermore, utilize the 3D positional information encoded in the feature? If the purpose is the latter (which makes better sense for this paper's motivation), then the designed Anchor3D seems not to well accomplish this goal, because it seems to enforce this by only allowing visual tokens that are related to the questions into the model.

Response: We thank the reviewer for this insightful question. The purpose of Anchor3D is indeed the latter: to make the LLM utilize the 3D positional information that is already encoded in its input features. Excluding unrelated visual tokens is the means, not the end goal. We clarify how the mechanism works, why the empirical gains come from reading 3D coordinates rather than merely removing distractors, and how we view its scope:

(1) Why excluding unrelated tokens unlocks 3D coordinate readout. Our diagnosis in Section 3 shows that 3D positions are already encoded across layers (recoverable by simple MLP probes with R2R^2 up to 0.940.94) but are not read during answer generation, because the model can satisfy the training objective by matching the question to the semantic appearance of visual tokens anywhere in the scene. The localization bottleneck removes this shortcut so that the question can only be answered from the tokens of the named objects and their 3D coordinates.

(2) Evidence that the gain comes from using 3D positions rather than fewer distracting tokens. Our ablation in Section 5.3 (Table 5) directly separates the effect of having fewer distractor tokens from the effect of reading 3D coordinates:

  • If the bottleneck merely helped the LLM avoid distracting background tokens, the performance gain would persist when 3D positions are shuffled among the retained tokens, since the selected tokens and their visual appearance remain identical.
  • Instead, Anchor3D drops sharply from 44.5 to 33.5 on R3D-Bench under position shuffling—falling back to the level of the same model without the bottleneck (w/o match, 32.6), which itself is insensitive to shuffling (32.6→32.332.6 \to 32.3).
  • Crucially, Anchor3D reuses the 3D positional encoding of Video-3D LLM unchanged: the exact same positional encoding goes from ignored to actively consulted once entity-to-token binding is enforced.

(3) Scope of Anchor3D and direction for future 3D LLMs. We fully agree with the reviewer that imposing this binding externally via hard masking is a restrictive mechanism: it is enforced outside the LLM rather than learned end-to-end, and it applies specifically when the question names target objects (while questions requiring global scene context bypass it). Because our work is primarily diagnostic, Anchor3D is designed as a controlled intervention to prove that what current 3D LLMs lack is entity-to-token binding rather than a stronger 3D positional encoding. Empowering 3D LLMs to learn this binding internally—without hiding the rest of the scene—is precisely the future research direction we hope our diagnosis motivates.

AuthorsOpenReview ↗

Author Response to Reviewer JyHw (2/2)


2. Justification for retaining zeroed-out tokens and concerns about long-term efficiency and general applicability (Line 331)

Are there other underlying reasons for not discarding the zero-out token besides the reason mentioned in the paper for "the sequence length and the position IDs of the text being unchanged" (Line 331)? Because the current solution does not seem like a permanent efficient solution and cannot serve as future standard.

Response: We thank the reviewer for this question.

(1) Why non-selected tokens were zeroed rather than dropped. There is no other underlying architectural reason: non-selected visual tokens are zeroed out rather than discarded solely to keep the visual sequence length and the 1D/3D position IDs of the subsequent text tokens identical to the unmasked baseline, which provided the cleanest and most controlled implementation for our diagnostic study.

(2) Efficiency and future direction. We completely agree with the reviewer that feeding zero-valued vectors into the LLM is computationally redundant and should not be taken as a permanent efficiency standard. Consistent with the diagnostic focus of our paper, zero-masking was chosen to isolate whether enforcing entity-to-token binding makes the model consult its 3D coordinates, rather than to propose a final inference-efficient architecture. Physically discarding the non-selected tokens is the natural next step and would substantially shorten the visual token sequence; we will train a variant of Anchor3D in which non-selected visual tokens are removed rather than zeroed out in the future.


3. Inference latency introduced by Grounding DINO, SAM2, and the matcher before 3D LLM processing

What is the inference latency introduced by the preprocessing steps like Grounding DINO and SAM2, and the matcher before processing by the 3D LLMs?

Response: We thank the reviewer for this question. The table below reports the exact wall-clock latency breakdown of our localization preprocessing pipeline across all 57 scenes of R3D-Bench (1,824 frames and 1,594 questions, generating 43,782 candidate masks) measured on 4×RTX 40904 \times \text{RTX 4090} GPUs:

Pipeline StageTotal TimePer SceneAmortized Per Question
Grounding DINO + SAM2 candidate generation419.4 s7.36 s0.26 s
SigLIP feature extraction229.2 s4.02 s0.14 s
Matcher scoring & mask export39.0 s0.68 s0.02 s
Total localization preprocessing687.7 s12.06 s0.43 s
3D LLM inference——∼9.0 s\sim 9.0\text{ s}

Across all 57 scenes, the full localization pipeline takes 687.7 s (11 min 28 s) in total: 419.4 s for Grounding DINO + SAM2 candidate generation (0.26 s/question0.26\text{ s/question}), 229.2 s for SigLIP feature extraction (0.14 s/question0.14\text{ s/question}), and 39.0 s for matcher scoring and mask export (0.02 s/question0.02\text{ s/question}). Because candidate generation and feature extraction are run once per unique object name in a scene and shared across questions querying that scene, the amortized preprocessing latency is 0.43 s of wall-clock time per question (1.73 GPU-s/question1.73\text{ GPU-s/question}, or 12.06 s12.06\text{ s} per 32-frame scene), which is small (∼4.8\sim 4.8\\%) relative to the ∼9.0 s\sim 9.0\text{ s} per question required by the 3D LLM inference step itself.

Dear authors,

Thank you for providing the rebuttal! I agree with the authors' explanations, and they addressed my concerns. Therefore, I would like to keep my score to stay positive on this paper.

Thank you so much for your prompt reply and continued positive support of our paper, Reviewer JyHw! We are glad that our responses addressed your questions, and we truly appreciate your constructive feedback throughout the review process.

Reviewer_n2pnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether 3D LLMs actually use the geometric 3D information they are given. To diagnose this, the authors apply three inference-time perturbations: (1) removing the visual input to the model; (2) randomly permuting the 3D positions; and (3) setting all 3D positions to zero. These perturbations are applied to five state-of-the-art models, which are evaluated on ScanQA, SQA3D, and R3D. The main finding is that the perturbations barely affect the models’ performance, while an MLP probe can still decode 3D offsets from their hidden states. This suggests that the injected geometry survives inside the LLM, but the model’s answers do not depend on it.

To address this problem, the authors propose Anchor3D, a model that keeps only the visual tokens of the objects named in the input question. These tokens are extracted using Grounding DINO, SAM2, and a SigLIP-based matcher. Anchor3D uses Qwen3.5-4B as its LLM, fine-tuned on SQA3D, ScanQA, and a self-built training set of metric questions on ScanNet. Anchor3D is shown to outperform all open-weight models without tool use and to perform comparably to GPT-5.5, Gemini 3.1 Pro, and Gemini 3 Flash, while becoming sensitive to the aforementioned perturbations.

Critical strengths
  • The paper makes a valuable contribution.
  • The diagnostic experiments are simple and informative.
  • The diagnostic experiments cover a range of 3D LLMs that use different strategies to inject 3D information.
  • The probing-head experiments (Section 3.3) effectively address the question of whether 3D geometry is encoded within the LLM.
Critical weaknesses
  1. The localization bottleneck greatly simplifies the task. Keeping only the visual tokens of the objects mentioned in the question is a significant shortcut. I believe MLLMs should be able to identify the visual tokens relevant to the question on their own. Moreover, the bottleneck is bypassed when no object name is extracted (lines 329–331), and on the ADT questions of R3D-Bench, the matcher selects the correct masks in only 63.6% of cases (Appendix C). This limits the generality of Anchor3D.
  2. Frame selection. An object visible in several frames may be more useful for answering the question in one frame than in another. For example, if the model is asked for the distance between a chair and a table, a specific viewpoint showing both objects may be more informative than others for computing their distance. However, in Anchor3D, masks are not associated across frames (line 305), and the matcher picks one candidate per name using only the name phrase, without the rest of the question (lines 311–312). As a result, the LLM sees each object from a single view, and this view is chosen without regard to the task the LLM must perform. In addition, there may be cases where merging information from multiple views is crucial to answering the question correctly, which Anchor3D does not do by design. This may also explain Anchor3D’s weakness on question types such as “How Long” and “Volume” (Table 3), where multi-view information can help because a single view may be occluded.
  3. Role of training data vs. architecture. All perturbations are applied only at inference time (line 145). As the authors suggest in lines 267–268 (“We interpret this as a supervision problem rather than an architectural one”), the blindness to 3D positions may be caused by the training data. However, the paper’s own ablation seems to contradict this interpretation: the “w/o match” model is trained on the same data as Anchor3D but uses all visual tokens, and it remains insensitive to shuffling. This suggests that supervision alone does not solve the problem.
  4. The Related Work section is too brief. It does not explain in sufficient depth how competing models work or how Anchor3D differs from them. In particular, Real-3DQA (Ma et al., 2026) is cited only for its text-only finding (lines 44–45), but that paper also proposes a 3D-reweighted training objective to make 3D LLMs rely on 3D input. This is a training-based solution to the same problem addressed in this submission and should be discussed.
  5. Fairness of the main comparison. The claim of matching GPT and Gemini (line 106) compares Anchor3D, which uses depth, camera poses, and in-distribution data, against zero-shot, RGB-only models. The claim should be adjusted accordingly.
  6. Presentation. The paper is generally understandable but requires effort to follow in places. For instance, the zero-position control described in Section 3.3 is not clearly explained.

Overall: The diagnostic study is a useful contribution, but the proposed method relies on a hand-engineered shortcut with the limitations described above, and the link between the diagnosis and the proposed fix is not fully supported. I would consider raising my score if the authors address the questions below.

Questions for authors
  1. Cost of matcher errors. The matcher selects fully correct masks on only 63.6% of ADT questions (Appendix C). Could the authors report R3D-Bench results using ground-truth masks at inference, as well as R3D accuracy broken down by whether the matcher’s selection was correct? This would show how much performance depends on the localization stage.
  2. Bypass rate. What fraction of ScanQA and SQA3D questions have no extracted object name, such that the bottleneck is bypassed (lines 329–331)? Could the authors report EM separately on the bypassed and non-bypassed subsets, along with the drop under position shuffling for each?
  3. Train/test mask mismatch. At training time, S is replaced by the ground-truth masks of the referred objects. Do these masks cover the object in every frame where it is visible, or only in a single frame? If the former, the LLM sees multi-view object tokens during training but only a single top-1 mask at inference. Could the authors clarify this and discuss the effect of such a mismatch?
  4. Single view vs. positional resolution. Section 5.2 attributes the weakness on “How Long”, “Volume”, and “Gap Fit” questions to patch-level coordinate averaging. An alternative explanation is that a single mask from a single frame often shows a partially occluded or truncated object. Could the authors test this by keeping the masks of the selected instance across all frames (or the top-k candidates) and reporting the effect on the extent-based question types?
  5. Use of the question. The matcher scores candidates using only the name phrase, without the rest of the question. Have the authors considered conditioning it on the full question, e.g., to select a view in which both named objects are visible? Given its current inputs, would “name-conditioned matcher” be a more accurate name than “question-conditioned matcher”?
  6. Supervision vs. architecture. Could the authors apply the 3D position shuffle to the Video-3D LLM fine-tuned on their training data (34.9 in Table 3)? Separately, the w/o-match model is trained with the same metric supervision but remains insensitive to shuffling (32.6 → 32.3, Table 5). How do the authors reconcile this with their interpretation that the problem stems from supervision rather than architecture (lines 267–268)?
  7. Relation to Real-3DQA. Real-3DQA also proposes a training-based remedy (a 3D-reweighted objective) to make 3D LLMs rely on 3D input. How does Anchor3D compare with this approach conceptually and, if feasible, empirically?
  8. Fairness of the comparison. Anchor3D uses depth and camera poses and is trained on R3D-Bench-style question templates, whereas GPT-5.5 and Gemini are evaluated zero-shot on RGB input only. Among systems with access to depth, the tool-use pipelines score considerably higher (61.9 and 73.5). Would the authors revise the claim of matching GPT-5.5 and Gemini 3.1 Pro to reflect these differences?
  9. Cost. What is the inference overhead of running Grounding DINO, SAM2, and SigLIP over 32 frames per named object, compared with the baselines? How do the training time and resources compare with those of the fine-tuned Video-3D LLM?

Post-rebuttal update: the authors’ response addressed my concerns; rating raised from 2 to 3.

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer n2pn (1/6)

We sincerely appreciate Reviewer n2pn for the constructive feedback and the time dedicated to evaluating our work, as well as for recognizing the value of our diagnostic study, its coverage across diverse 3D injection strategies, and our probing-head analysis (Section 3.3).

Before addressing each point in detail, we would like to emphasize the overarching positioning of our work: this paper is primarily a diagnostic study, and Anchor3D is intended as an initial, proof-of-concept remedy rather than a final architectural standard. Our central goal is to reveal what current 3D LLMs lack—namely, that injected 3D coordinates survive inside the network but are ignored at prediction time due to a failure of entity-to-token binding—and to point out a clear direction for future 3D LLMs. All wording revisions, clarified explanations, and expanded discussions in the updated manuscript are marked in red. Below, we address each concern and question in detail.


1. Apparent contradiction between the supervision-based explanation (Lines 267–268) and the "w/o match" ablation

As the authors suggest in lines 267–268 ("We interpret this as a supervision problem rather than an architectural one"), the blindness to 3D positions may be caused by the training data. However, the paper's own ablation seems to contradict this interpretation: the "w/o match" model is trained on the same data as Anchor3D but uses all visual tokens, and it remains insensitive to shuffling. This suggests that supervision alone does not solve the problem.

Response: We thank the reviewer for pointing this out. We agree that the original sentence in lines 267–268 was imprecise: it read as a claim that training data alone cause the positional insensitivity, which, as the reviewer rightly notes, our own ablation rules out. We clarify our interpretation and manuscript revision below:

(1) Clarifying "not architectural" vs. "supervision" (binding vs. encoding). By "not architectural," we meant that the bottleneck does not lie in the design of the 3D geometry encoding: five models with distinct injection mechanisms are equally insensitive to position perturbations, and relative 3D offsets remain decodable from the hidden states of four of them (Section 3.3). By "supervision," we meant that the standard end-to-end training signal never forces the model to bind the entities named in the question to their corresponding visual tokens and coordinates. We did not mean that simply adding geometry-dependent QA pairs is sufficient.

(2) Consistent evidence from our "w/o match" ablation. The ablation cited by the reviewer is precisely our evidence for this distinction (Section 5.3: "the change is not due to the training data"). Trained on the exact same metric data, the w/o match model stays insensitive to 3D position shuffling (32.6→32.332.6 \to 32.3 on R3D-Bench), whereas with the localization bottleneck the same backbone and Video-3D LLM position encoding drop from 44.544.5 to 33.533.5 under shuffling. Metric supervision is therefore necessary (30.230.2 without it) but not sufficient on its own.

We have revised lines 267–268 (Section 3.3) and the corresponding sentence in the Conclusion (marked in red) to frame this precisely as a failure of binding rather than of encoding.

AuthorsOpenReview ↗

Author Response to Reviewer n2pn (2/6)

2. Insufficient discussion of competing methods in Related Work and comparison with Real-3DQA's 3D-reweighted training objective

The Related Work section is too brief. It does not explain in sufficient depth how competing models work or how Anchor3D differs from them.

In particular, Real-3DQA (Ma et al., 2026) is cited only for its text-only finding (lines 44–45), but that paper also proposes a 3D-reweighted training objective to make 3D LLMs rely on 3D input. This is a training-based solution to the same problem addressed in this submission and should be discussed. Real-3DQA also proposes a training-based remedy (a 3D-reweighted objective) to make 3D LLMs rely on 3D input. How does Anchor3D compare with this approach conceptually and, if feasible, empirically?

Response: We thank the reviewer for this constructive suggestion. We have substantially expanded the Related Work section in the revised manuscript (marked in red in Section 2) to discuss in detail how each family of 3D LLMs injects geometry, as well as prior work on shortcut mitigation, metric supervision, and grounding before reasoning. Below, we clarify how our work differs from the closest lines of research:

(1) Geometry injection methods (LLaVA-3D, Video-3D LLM, Spatial-MLLM, Proxy3D, QuatRoPE). These works propose increasingly expressive mechanisms to encode 3D coordinates into visual tokens or attention layers. Rather than proposing a new encoding, Anchor3D reuses the 3D position encoding of Video-3D LLM unchanged. Our diagnostic question is whether existing encodings are consulted at all, and Section 3 shows that across five representative designs they are largely ignored.

(2) Conceptual and empirical comparison with Real-3DQA (3DR-FT). Real-3DQA demonstrates that many 3D QA items can be answered from text alone, and its 3DR-FT objective reweights training examples based on text-vs.-scene prediction discrepancy so that the model relies on the scene input rather than language priors.

  • Conceptual difference: Our diagnosis operates one level finer. As shown in Section 3.1 (Table 1), the models we study do rely on the scene input ( +7.9+7.9 to +12.8+12.8 EM over text-only on SQA3D), yet they still ignore the 3D positions within that scene. A model can therefore depend on the scene input (as encouraged by 3DR-FT) while answering entirely from 2D visual appearance and semantics rather than 3D coordinates. Separating 2D appearance reliance from 3D geometric reliance requires perturbing the 3D positions while holding the visual features fixed—which is what our position-shuffle and 3D zero-position tests do.
  • Empirical comparison: We would have liked to apply our positional perturbation tests to 3DR-FT checkpoints and compare with 3DR-FT empirically; however, its training pipeline, filtered training data, and fine-tuned model checkpoints have not been publicly released, making direct empirical reproduction infeasible.

(3) Metric supervision (SpatialVLM, SpatialRGPT, Q-Spatial, SpatialBot, SpaceR). These works treat quantitative spatial estimation as a target capability and curate distance/size supervision. Our ablation demonstrates that metric supervision alone does not make a 3D LLM read its injected 3D coordinates: the w/o match baseline is trained on the exact same metric data but remains insensitive to position shuffling (32.6→32.332.6 \to 32.3).

(4) Grounding before reasoning (Chat-Scene, Robin3D, Grounded 3D-LLM, SceneCoT) and token selection (FastV, SparseVLM, VisionZip). The former introduce object identifiers or auxiliary grounding heads while leaving the full scene visible to the LLM; the latter prune tokens primarily for computational efficiency. In contrast, Anchor3D uses question-conditioned localization as a hard information bottleneck—only tokens of the named objects reach the LLM—and verifies via position shuffling that this forces the model to consult their 3D coordinates (44.5→33.544.5 \to 33.5).

AuthorsOpenReview ↗

Author Response to Reviewer n2pn (3/6)

3. Unmatched input modalities and training settings in the performance-parity claim with GPT-5.5 and Gemini 3.1 Pro

The claim of matching GPT and Gemini (line 106) compares Anchor3D, which uses depth, camera poses, and in-distribution data, against zero-shot, RGB-only models. The claim should be adjusted accordingly.

Anchor3D uses depth and camera poses and is trained on R3D-Bench-style question templates, whereas GPT-5.5 and Gemini are evaluated zero-shot on RGB input only. Among systems with access to depth, the tool-use pipelines score considerably higher (61.9 and 73.5). Would the authors revise the claim of matching GPT-5.5 and Gemini 3.1 Pro to reflect these differences?

Response: We fully agree with the reviewer and have revised the claim throughout the paper.

(1) Differences in modality, supervision, and tool use. Anchor3D uses posed RGB-D inputs and is trained on R3D-Bench-style question templates, whereas GPT-5.5 and Gemini 3.1 Pro are evaluated zero-shot on RGB frames only, so the two settings are not directly comparable. We also explicitly note in the text that depth-enabled tool-use pipelines score substantially higher (61.961.9 and 73.573.5). As a minor clarification regarding data distribution: our metric training questions are constructed on ScanNet scenes whereas R3D-Bench is built on Aria Digital Twin (ADT), so the question templates are in-distribution while the test scenes come from an unseen domain.

(2) Manuscript revisions. In the revised manuscript (marked in red), we have removed the claim of matching GPT-5.5 and Gemini 3.1 Pro from the Abstract, Introduction (line 106), and Section 5.2. Instead, we highlight the controlled comparisons under matched modalities and training data: Anchor3D achieves 44.5 vs. 32.6 for the same Qwen3.5-4B backbone and training data without the bottleneck (w/o match), and vs. 34.9 for Video-3D LLM fine-tuned on the same training set. The proprietary zero-shot RGB models remain in Table 3 in gray font strictly for reference. This revision aligns with our diagnostic focus: the key result is not the absolute leaderboard score against proprietary RGB models, but that the exact same 3D positional encoding goes from ignored to actively consulted once entity-to-token binding is enforced.


4. Unclear description of the zero-position control in Section 3.3

For instance, the zero-position control described in Section 3.3 is not clearly explained.

Response: We thank the reviewer for noting this and agree that the original paragraph in Section 3.3 was too terse, as the motivation and setup of the control were previously deferred to Appendix B.6. We have rewritten the paragraph in Section 3.3 (marked in red) to make it self-contained:

  • Why the control is needed: Because 2D visual features themselves carry implicit positional cues (e.g., floors appear low in a room and ceilings appear high), a probing head might recover relative 3D offsets Δxyz\Delta xyz from visual appearance alone rather than from the injected 3D coordinates.
  • How the control works: To separate appearance cues from injected geometry, we re-run each model under the 3D zero-position perturbation (setting all input 3D coordinates to 0\mathbf{0} so that the hidden states contain no injected position) and train a new set of probes from scratch on the resulting frozen hidden states, using the exact same token pairs, probe architecture, and true 3D offsets Δxyz\Delta xyz as regression targets (Pos0 in Figure 2).
  • What it shows: If the probes were inferring offsets from visual appearance, R2R^2 would remain high under Pos0. Instead, R2R^2 collapses well below the original curves across the four models whose geometry enters the hidden states (Δx\Delta x drops near or below zero, with only a partial appearance-driven residual on height Δz\Delta z), confirming that the probes genuinely decode the injected 3D positions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers (n2pn, JyHw, MBEP) and the Area Chair for your time, constructive feedback, and engagement throughout the discussion period. We are glad that all reviewers' concerns have been addressed and that all three reviewers are now positive on our paper (with Reviewer n2pn raising their score to 3, and Reviewers JyHw and MBEP maintaining their scores of 3).

Strengths recognized across reviews:

  • Systematic diagnosis (n2pn, JyHw, MBEP): Simple, informative inference-time perturbations across five representative 3D LLMs with diverse geometry injection mechanisms and multiple benchmarks.
  • Encoded-but-unconsulted geometry (n2pn, JyHw, MBEP): Probing-head experiments and zero-position controls demonstrate that 3D offsets survive in hidden states yet are not read at prediction time.
  • Effective controlled intervention (JyHw, MBEP): Anchor3D reuses Video-3D LLM's positional encoding unchanged, improves R3D-Bench from 32.6 to 44.5, and makes the encoding actively consulted (44.5 → 33.5 under shuffling), with clear and reproducible presentation.

Rebuttal revisions and new experiments:

  • Manuscript updates (marked in red): Expanded Related Work including Real-3DQA's 3DR-FT (Sec. 2); made the 3D zero-position control self-contained and framed the diagnosis precisely as a failure of binding rather than encoding (Sec. 3.3, Sec. 6); updated terminology to name-conditioned matcher and clarified multi-frame GT training masks (Abstract, Sec. 4.2–4.3); replaced the unmatched GPT-5.5/Gemini 3.1 Pro parity claim with controlled comparisons, explicitly defined "without tool use", and added fine-tuned Video-3D LLM shuffle results (Abstract, Sec. 1, Sec. 5.1–5.3).
  • New experiments: Added the matched full-scene target-marker control (Highlight, 35.4 vs. 35.3 unmarked and 44.5 for Anchor3D) (MBEP, JyHw); GT-mask/matcher-correctness breakdowns, top-kk multi-view evaluation, and ScanQA/SQA3D bypass rates (n2pn); fine-tuned Video-3D LLM shuffling (34.93 → 34.63) and 3-seed shuffle variance (SD ≤ 0.50 / 0.62) (n2pn, MBEP); and full latency/training-cost breakdowns (0.43 s/question amortized preprocessing on 4×RTX 40904 \times \text{RTX 4090}) (n2pn, JyHw).

Following Reviewer n2pn's suggestion, we will incorporate all of these rebuttal experiments into the supplementary material of the final version. Thank you all again for helping us make this work stronger!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.