acceptodds
Under review as a conference paper at ICLR 2027

GEAR: An Evidence-Adaptive VLM Agent for Zero-Shot 3D Visual Grounding

Abstract

Zero-shot 3D visual grounding aims to localize objects from natural-language descriptions without task-specific 3D training. Existing approaches largely follow two paradigms: proposal-based methods enable efficient and reliable geometric reasoning but cannot recover missing objects, while RGB-D reconstruction provides stronger visual recoverability at substantially higher computational cost and with additional sources of error. To bridge this gap, we introduce GEAR, an evidence-adaptive ReAct-style agent that unifies proposal-based grounding, selective verification, and on-demand candidate repair. Starting from proposal hypotheses, GEAR adaptively selects and revisits grounding operations based on the query and evolving evidence, invoking semantic or visual verification for ambiguous hypotheses and bounded RGB-D reconstruction only when candidates are missing or unreliable. A candidate-bound memory maintains consistent evidence across grounding operations, while a role-aware geometric solver handles compound ordinal and local-anchor expressions. GEAR achieves 60.4% [email protected] and 52.4% [email protected] on ScanRefer and 56.0% accuracy on Nr3D, surpassing TAB, the previous state-of-the-art method, while reducing token usage and inference time by approximately 50%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Visual Grounding, Zero-shot Scene Understanding, Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EC8ROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GEAR, an evidence-adaptive VLM agent for zero-shot 3D visual grounding. It combines proposal-based geometric reasoning with selective visual verification and on-demand RGB-D reconstruction to repair missing or unreliable candidates. Candidate-bound memory maintains consistent evidence across operations, while a structured geometric solver and terminal validation support grounding decisions. Experiments on 250-query subsets of ScanRefer and Nr3D show improved grounding accuracy and approximately 50% lower token usage and total inference time than the reproduced TAB baseline.

Critical strengths
  1. The paper addresses the trade-off between efficient proposal-based reasoning and the ability to recover missing objects from RGB-D observations. Selectively acquiring additional evidence is a sensible approach to improving grounding without incurring reconstruction costs for every query.

  2. GEAR integrates geometric reasoning, visual verification, and candidate repair within a shared inference loop. Candidate-bound memory explicitly tracks identities, target/anchor roles, and evidence provenance, while terminal validation helps prevent unsupported grounding decisions.

  3. Experiments on ScanRefer and Nr3D show higher grounding accuracy than the reproduced TAB baseline, alongside substantially lower token usage and total runtime. Comparisons with a rule-based controller and component ablations provide supporting evidence for the benefits of adaptive control and persistent candidate memory.

Critical weaknesses
  1. Experiments use only 250-query subsets of ScanRefer and Nr3D. On ScanRefer, the improvements over the reproduced TAB baseline correspond to just three and four additional correct predictions at [email protected] and [email protected], respectively. Larger-scale evaluation and paired uncertainty analysis would help establish the reliability of these gains.

  2. The evaluation matches predicted boxes back to existing Mask3D proposals, making it difficult to distinguish recovery of genuinely missing proposals from recovery of candidates omitted by semantic retrieval. Reporting accuracy before and after proposal matching, together with separate analyses of these two failure cases, would better substantiate the candidate-repair claim.

  3. The ablations demonstrate that visual verification and RGB-D repair are useful, but the paper does not clearly quantify how frequently each route is executed or its success rate and cost. Comparisons with always-on reconstruction and simple conditional repair policies would more directly establish the benefits of evidence-adaptive routing.

Questions for authors

Please see the weakness

4 comments

Response to Reviewer EC8R

We thank the reviewer for the positive assessment of GEAR’s evidence-adaptive design, candidate-bound memory, and accuracy–cost trade-off. We agree that the three requested analyses are important for clarifying the strength and scope of our empirical claims.

W1/Q1. Evaluation scale and paired uncertainty.

Experiments use only 250-query subsets

\.\.\.

. Larger-scale evaluation and paired uncertainty analysis would help establish the reliability of these gains.

On the original 250 ScanRefer queries, GEAR alone was correct on 47 queries and TAB alone on 44 at IoU 0.25, giving a net gain of 1.20 percentage points (95% CI

−64˙0,88˙0−6\.40, 8\.80

). At IoU 0.50, the corresponding counts were 42 and 38, giving a gain of 1.60 points (95% CI

−52˙0,88˙0−5\.20, 8\.80

). We computed percentile intervals from 100,000 paired query bootstrap resamples.

We also evaluated both methods on 750 additional ScanRefer validation queries across three disjoint subsets. GEAR achieved 53.73% / 47.60% at IoU 0.25 / 0.50, compared with 52.67% / 44.80% for TAB. The pooled gains were +1.07 and +2.80 points, with scene-bootstrap 95% CIs of

−32˙0,54˙0−3\.20, 5\.40

and

−11˙4,67˙9−1\.14, 6\.79

, respectively. On a separate 181-query ViGiL3D subset used exclusively for evaluation, GEAR achieved 37.56% mIoU and 45.53% / 37.44% accuracy, compared with 33.09% and 42.54% / 32.60% for TAB.

W2/Q2. Proposal matching and the interpretation of candidate repair.

The evaluation matches predicted boxes back to existing Mask3D proposals, making it difficult to distinguish recovery of genuinely missing proposals from recovery of candidates omitted by semantic retrieval.

a. Before and after matching.

Reporting accuracy before and after proposal matching,

\.\.\.

For a fair comparison with TAB, we adopt the same final proposal-matching rule. The final proposal macthing evaluator replaces a valid predicted box with its highest-IoU Mask3D proposal only when their 3D IoU exceeds 0.25; Otherwise, the prediction is retained. On the same 250 ScanRefer queries:

OutputMethodmIoU[email protected][email protected]
Native, before matchingGEAR0.493860.00%50.40%
Native, before matchingTAB0.347259.20%37.20%
After matchingGEAR0.503860.40%52.40%
After matchingTAB0.498359.20%50.80%

We now report both native and matched outputs on the same 250 ScanRefer queries. Before matching, GEAR exceeds TAB by 13.2 percentage points at [email protected]. Matching increases [email protected] by 2.0 points for GEAR and 13.6 points for TAB, substantially narrowing the difference between the native outputs. These results quantify the contribution of post-processing and show that GEAR’s native localization quality is not solely a consequence of the final matching step.

b. Retrieval omissions and missing proposal coverage.

\.\.\.

, together with separate analyses of these two failure cases,

\.\.\.

.

At each IoU threshold, we compared the maximum ground-truth IoU over the full Mask3D proposal set with that over GEAR's initial target retrieval. A retrieval miss occurs when a qualifying proposal exists in the full set but is absent from initial retrieval, whereas a coverage miss occurs when no original proposal meets the threshold. We evaluated GEAR's native predictions on these two disjoint groups, before final proposal matching.

IoU ThresholdCoverage missesGEAR localizedRetrieval missesGEAR localized
0.1064/6 667˙66\.7%3510/35 286˙28\.6%
0.20179/17 529˙52\.9%278/27 296˙29\.6%
0.50409/40 225˙22\.5%266/26 231˙23\.1%

At the standard IoU thresholds of 0.25/0.50, GEAR successfully localized 9/17 and 9/40 coverage misses, respectively, alongside 8/27 and 6/26 retrieval misses. Even at IoU 0.10, where no original proposal provides minimal target coverage, GEAR localized 4 of 6 cases. These pre-matching results demonstrate that GEAR can localize targets beyond the original proposal coverage, rather than merely recovering candidates omitted during semantic retrieval.

W3/Q3. Routing frequency, success rate, and cost.

The ablations demonstrate that visual verification and RGB-D repair are useful, but

\.\.\.

how frequently each route is executed or its success rate and cost. Comparisons with always-on reconstruction and simple conditional repair policies would

\.\.\.

.

On the original 250 ScanRefer queries, 166 queries 664˙66\.4% used View Probe, while 157 queries 628˙62\.8% executed target RGB-D reconstruction. On their respective query subsets, the boxes selected by View Probe achieved 53.0% / 47.6% [email protected]/0.50, while those produced by RGB-D reconstruction achieved 44.6% / 28.0%. Averaged over all 250 queries, View Probe and Repair requests accounted for 51.6 s/query and 279.0 s/query, respectively.

We further compared alternative routing strategies. The fixed rule-based routing policy (Table 4), which replaces the VLM Controller with predefined evidence-conditioned action selection, achieved 56.4% / 49.2% [email protected]/0.50, compared with 60.4% / 52.4% for GEAR, with comparable inference time 465\.5 vs\. 472\.8 s but fewer tokens 137\.2 K vs\. 375\.2 K. Always-on reconstruction achieved 58.80% / 37.20%, requiring 933.63 s and 683.27 K tokens. A lightweight rule-only baseline achieved 38.78% / 34.18%, requiring only 0.73 s and no VLM tokens.

ModelsmIoU %[email protected] %[email protected] %Time ssTokens KK
Fixed rule-based routing48.0056.4049.20465.50137.20
Always-on reconstruction34.6258.8037.20933.63683.27
Rule-only inference33.1038.7834.180.730.00
GEAR (adaptive)50.3860.4052.40472.83375.17

These results suggest that indiscriminate reconstruction incurs substantial inference overhead without necessarily improving grounding accuracy, while purely rule-based inference trades grounding accuracy for computational efficiency. Furthermore, fixed conditional routing reduces token consumption but yields lower accuracy than GEAR. By selectively acquiring additional evidence, GEAR achieves higher grounding accuracy while avoiding the substantial overhead of always-on reconstruction.

Thank you for the clarifications and additional analyses. They address most of my concerns. I will maintain my positive scores.

We sincerely thank the reviewer for the constructive feedback and positive assessment. We are glad that our additional analyses have addressed most of the concerns. We will incorporate these clarifications and results into the revised manuscript.

Reviewer_jtvGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GEAR, an evidence-adaptive VLM agent for zero-shot 3D visual grounding. Starting from existing 3D proposals, the system dynamically invokes semantic verification, candidate image inspection, geometric reasoning, and on-demand RGB-D repair according to the available evidence. Candidate-bound memory maintains object identities, provenance, target/anchor roles, bounding-box versions, and associated evidence, while terminal validation checks whether the final answer is sufficiently supported. On evaluation subsets of 250 queries each from ScanRefer and Nr3D, GEAR achieves 60.4% [email protected], 52.4% [email protected], and 56.0% accuracy, respectively. Compared with TAB reproduced under the same evaluation protocol, the paper reports lower token usage and total inference time, with component and strategy ablations examining the contribution of individual designs.

Critical strengths

The motivation is sound and the framework is useful. GEAR unifies a revisitable workflow, balancing grounding accuracy against inference cost, while its candidate-bound evidence memory curbs evidence misassociation and target drift.

Clear framework and well-presented results.​ The proposed method is conceptually simple and easy to follow, with each module's role clearly defined, making the technical contribution easy to identify.

Sufficient ablation studies.​ The component-level ablation and strategy-level analysis together validate each design choice.

Critical weaknesses
  1. Although GEAR reduces the average total runtime by approximately 52% relative to TAB, its reported runtime remains around 473 seconds per query, which limits its applicability to embodied tasks requiring prompt responses. The main sources of this runtime overhead should also be further analyzed.
  2. The proposed method incorporates multiple routing rules, verification mechanisms, and thresholds, yet it is evaluated on only two subsets of 250 queries each. The paper does not report how extensively these queries exercise the different execution paths, nor does it establish the robustness of each path.
  3. The current results do not clearly distinguish whether the method genuinely recovers a target that is absent from the proposal set, or merely corrects retrieval or selection errors involving existing proposals. This distinction directly affects the paper's central claim regarding missing-candidate recovery.
Questions for authors
  1. Which data were used to develop the prompts, routing rules, geometric thresholds, and termination criteria? Were either of the public 250-query subsets used for design choices or error analysis that informed system modifications? If so, was the final configuration evaluated on independent queries or scenes not used during development? For the modest ScanRefer gains over TAB, can you provide paired per-query analyses and confidence intervals?
  2. Could you reevaluate both methods under settings aligned with the original TAB paper, including query versions, inference configurations, and post-processing, or compare them under a common protocol on a larger independent subset not used for prompt or rule development? This would help clarify the discrepancy in TAB’s reproduced Nr3D performance and the robustness of GEAR’s gains. If the additional results explain this discrepancy and substantiate GEAR’s improvements, I would consider raising my score.
  3. Is proposal matching applied to every reconstructed output? Please report results before matching and performance on queries for which no proposal in the full original set meets the target IoU threshold, to clarify whether GEAR can recover targets beyond the original proposal coverage.
3 comments

Response to Reviewer jtvG

We thank the reviewer for recognizing the motivation, clarity, and component analysis of GEAR.

W1. Runtime overhead and applicability

\.\.\.

its reported runtime remains around 473 seconds per query, which limits its applicability to embodied tasks requiring prompt responses. The main sources of this runtime overhead should also be further analyzed.

We agree that GEAR’s current latency is too high for prompt closed-loop control.

Reducing inference time while preserving localization accuracy motivated GEAR’s selective use of RGB-D repair. On the 250 ScanRefer queries, GEAR uses 51.9% less time than TAB 9821˙8s/query982\.18 s/query, with [email protected] of 52.4% vs. 50.8%.

RGB-D repair is the largest cost at 279.03 s/query 590˙59\.0%, followed by controller calls 1047˙5s104\.75 s, non-repair Object Analyzer calls 581˙1s58\.11 s, Geometry Solver 156˙1s15\.61 s, Space Snapper 74˙7s7\.47 s, final relation review 24˙4s2\.44 s, and other overhead 54˙1s5\.41 s.

Holding other stage times fixed, removing repair yields an estimated 194 s/query 24˙4×speedup2\.44× speedup. In the measured ablation (Table 5), disabling repair reduces [email protected] and [email protected] by 7.6 and 5.2 percentage points, respectively.

W2. Execution-path coverage and robustness.

The proposed method incorporates multiple routing rules, verification mechanisms, and thresholds,

\.\.\.

. The paper does not report how extensively these queries exercise the different execution paths, nor

\.\.\.

the robustness of each path.

a. Observed execution diversity. On the 250 ScanRefer queries, we observed 85 distinct ordered tool sequences. The median was 6 tool calls per query (range: 2–18). The three most common sequences covered 99 queries 396˙39\.6%, while 64 sequences occurred once.

b. Route usage and final accuracy. Repair was requested on 165 queries 660˙66\.0%, with RGB-D reconstruction executed on 157 628˙62\.8%. Separately, terminal relation verification ran on 52 queries 208˙20\.8%. The table reports final matched accuracy stratified by repair execution.

Observed executionn[email protected][email protected]
RGB-D reconstruction executed15783/157 528˙752\.87%69/157 439˙543\.95%
Repair requested, no reconstruction executed84/8 500˙050\.00%4/8 500˙050\.00%
No repair requested8564/85 752˙975\.29%58/85 682˙468\.24%

These statistics describe observed execution diversity and final matched accuracy. Route selection is evidence-dependent, and many exact sequences are sparsely sampled.

W3. Retrieval and selection errors versus proposal coverage

The current results do not clearly distinguish whether the method genuinely recovers a target that is absent from the proposal set, or merely corrects retrieval or selection errors involving existing proposals. This distinction directly affects the paper's central claim regarding missing-candidate recovery.

GEAR can both correct errors involving existing proposals and recover targets beyond the coverage of the original proposal set. At IoU 0.50, before proposal matching, GEAR localized the target in 6/26 retrieval misses where a qualifying proposal existed in the full set and 30/86 initial ranking errors where a qualifying proposal was already shortlisted. As illustrated in Q3cc, GEAR also recovers targets for which the full proposal set contains no qualifying box. For more details, please refer to Q3cc.

Q1. Development data and independent evaluation

a. Development data and Additional independent evaluation

Which data was used to develop the prompts,

\.\.\.

and termination criteria? Were either of the public 250-query subsets used for design choices or error analysis

\.\.\.

? If so, was the final configuration evaluated on independent queries or scenes not used during development?

We used a random sample from the ScanRefer training split to refine GEAR's inference harness, including its prompts, routing rules, geometric thresholds, and termination criteria. Neither public 250-query evaluation subset was used for any design choices, and error analyses on these subsets did not inform the final configuration.

We further compared the frozen GEAR configuration with TAB on the subset (181 queries) of a different dataset ViGiL3D. We used these queries exclusively for evaluation:

MethodmIoU[email protected][email protected]
TAB33.09%42.54%32.60%
GEAR37.56%45.53%37.44%
Difference pppp+4.47+2.99+4.84

b. Paired analysis of the ScanRefer gains

For the modest ScanRefer gains over TAB, can you provide paired per-query analyses and confidence intervals?

We agree that the gains reported in Table 1 are modest. We conducted a paired analysis of all 250 queries, computing 95% percentile confidence intervals using 100,000 paired bootstrap resamples seed=20261116seed = 20261116. Under the Table 1 scoring protocol, GEAR alone was correct on 47 queries and TAB alone on 44 at IoU 0.25, yielding a gain of **1.20 percentage points (95% CI

−64˙0,88˙0−6\.40, 8\.80

)**. At IoU 0.50, the corresponding counts were 42 and 38, yielding a gain of **1.60 percentage points (95% CI

−52˙0,88˙0−5\.20, 8\.80

)**.

We also compared the two methods under the raw scoring protocol on the same queries. GEAR improved mIoU by 14.66 percentage points (paired 95% CI

89˙8,203˙58\.98, 20\.35

) and [email protected] by 13.20 points (95% CI

56˙0,208˙05\.60, 20\.80

). The difference in raw [email protected] was 0.80 points (95% CI

−68˙0,84˙0−6\.80, 8\.40

). Thus, under raw scoring, the clearest gains were in mean IoU and accuracy at the stricter threshold.

Stage/MetricGEARTABDifference ppQuery-paired 95% CI
raw mIoU49.38%34.72%+14.66
+89˙8,+203˙5+8\.98, +20\.35

| |raw [email protected]|60.00%|59.20%|+0.80|

−68˙0,+84˙0−6\.80, +8\.40

| |raw [email protected]|50.40%|37.20%|+13.20|

+56˙0,+208˙0+5\.60, +20\.80

|

Q2. TAB reproduction and protocol alignment

Could you reevaluate both methods under settings aligned with the original TAB paper,

\.\.\.

or compare them under a common protocol on a larger independent subset not used for prompt or rule development? This would help clarify the discrepancy in TAB’s reproduced Nr3D performance and the robustness of GEAR’s gains.

\.\.\.

a. Larger ScanRefer evaluation under the same settings.

As clarified in Q1, GEAR’s harness was developed on ScanRefer training data. We additionally evaluated the frozen GEAR configuration and TAB on three randomly sampled, mutually non-overlapping 250-query subsets of the full ScanRefer evaluation set, comprising 750 distinct queries used exclusively for evaluation.

All accuracies are percentages. Averaged across the three equally sized subsets, GEAR achieved 53.73% [email protected] and 47.60% [email protected], compared with 52.67% and 44.80% for TAB.

b. Independent ViGiL3D evaluation. On the ViGiL3D queries described in Q1, GEAR achieves a per-query averaged mIoU of 0.3756, compared with 0.3309 for TAB. GEAR also improved [email protected]/[email protected] from 42.54%/32.60% to 45.53%/37.44%.

c. Additional inference costs. Across the three ScanRefer subsets, GEAR uses approximately 391.77k tokens and 52.24 API calls per query, compared with 691.56k tokens and 256.10 calls for TAB. These correspond to reductions of 43.35% in token usage and 79.60% in recorded API calls. These results show higher aggregate accuracy with substantially lower inference costs.

Q3. Proposal matching and recovery beyond original coverage

a. Matching policy.

Is proposal matching applied to every reconstructed output?

For a fair comparison with TAB, we adopt the same final proposal-matching rule. Our submitted ScanRefer protocol maps the predicted box to a Mask3D proposal before scoring. Each valid final box is replaced by the Mask3D proposal with the highest 3D IoU only if that IoU exceeds 0.25; otherwise, it is retained. This matching uses agreement between the predicted box and the proposals rather than ground-truth target overlap.

b. Results before and after matching.

Please report results before matching and

\.\.\.

We now report both native and matched outputs on the same 250 ScanRefer queries. Before matching, GEAR exceeds TAB by 13.2 percentage points at [email protected]. Matching increases [email protected] by 2.0 points for GEAR and 13.6 points for TAB, substantially narrowing the difference between the native outputs. These results quantify the contribution of post-processing and show that GEAR’s native localization quality is not solely a consequence of the final matching step.

OutputMethodmIoU[email protected][email protected]
Native, before matchingGEAR0.493860.00%50.40%
Native, before matchingTAB0.347259.20%37.20%
After matchingGEAR0.503860.40%52.40%
After matchingTAB0.498359.20%50.80%

c. Recovery beyond original proposal coverage.

\.\.\.

and performance on queries for which no proposal in the full original set meets the target IoU threshold, to clarify whether GEAR can recover targets beyond the original proposal coverage.

For each query, we computed the maximum 3D IoU between the ground-truth box and every box in the released per-scene Mask3D proposal table, regardless of category label or retrieval status. A query is uncovered at threshold τ if this maximum is below τ.

IoU threshold ττUncovered queriesGEAR before Matching
IoU 0.000N/A
IoU 0.1064/6 667˙66\.7%
IoU 0.25179/17 529˙52\.9%

At both thresholds, GEAR produced boxes that met the evaluation criterion on queries where every original proposal fell below it. This demonstrates recovery beyond the original proposal coverage on the evaluated subset.

As shown in the table, when no proposal in the full original set meets the target IoU threshold, TAB recovers 66.7% 4/64/6 of targets at 0.10 and 52.9% 9/179/17 at 0.25 before final proposal matching. At 0.00, every target has at least one original proposal with positive overlap, so the recovery rate is undefined.

Reviewer_HYVeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an evidence-adaptive framework for 3D visual grounding that dynamically switches between proposal-based reasoning and RGB-D reconstruction to bound recovery.

Critical strengths

The figures shown in the paper are clear, informative, and illustrative.

Critical weaknesses

1. The motivation for the proposed method is unclear. In particular, the challenges described in lines 95–97, including “A reconstructed object may overlap…,” “evidence supporting an anchor…,” and “updated bounding box may…,” are difficult to interpret without concrete examples. The authors should provide illustrative scenarios or visual examples explaining what each challenge entails, why existing methods struggle with it, and how it motivates the proposed design.

2. The method uses Mask3D to generate the initial object proposals, but its sensitivity to the choice and quality of these proposals remains unclear. Ablation studies with alternative segmentation methods or proposal sources would help establish whether the improvements generalize beyond Mask3D.

3. I am not yet convinced that reduced inference cost is sufficiently substantiated as a major contribution. Existing zero-shot approaches, such as Agent3D-Zero [1], VLM-grounder, perform 3d understanding without relying on 3d pretrained proposal model, saving the cost of training a 3d model. In contrast, the proposed method combines LLM inference with extra cooperation of 3d pretrained model, which actually costs more than Agent3D-Zero using LLM only.

4. As I understand it, the method follows a proposal-based grounding pipeline and supplements the initial proposals with multi-view evidence from video when the target is missing from the proposal set. However, the current presentation makes the implementation difficult to follow. In particular, how are observations of the same object across different views associated with a single persistent memory entry and a consistent object ID? How does the method prevent identity switches, duplicate entries for the same object, or the incorrect merging of distinct objects? These association and memory-update mechanisms should be specified explicitly, preferably through pseudocode and a worked example.

5. The methodological novelty relative to prior work needs to be articulated more clearly. Related components already appear in existing approaches. For example, ZSVG3D also combines object localization with LLM-generated spatial reasoning programs, and S-agent[2] also maintains object-centric memory that aggregates visual and geometric evidence across views, which is pretty similar to this paper. I strongly suggest the authors further clarify the method's novelty and uniqueness.

[1] Zhang, Sha, et al. "Agent3d-zero: An agent for zero-shot 3d understanding." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Dai, Yalun, et al. "S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence." arXiv preprint arXiv:2606.20515 (2026).

Questions for authors

Refer to the weakness section.

3 comments

Response to Reviewer HYVe

We thank the reviewer for the constructive feedback and for appreciating the clarity of our figures. The comments identify important gaps in our explanation of the motivation, association mechanism, and scope of the efficiency and novelty claims.

W1/Q1. Motivation and concrete failure cases.

The motivation for the proposed method is unclear. In particular, the challenges

\.\.\.

, including “A reconstructed object may overlap…,” “evidence supporting an anchor…,” and “updated bounding box may…,” are difficult to interpret without concrete examples.

We have added Figure 4 and Appendix A.3 in the revised manuscript to illustrate the three challenges in lines 95–97.

AA Semantic identity versus geometric overlap.A backpack rests on a chair, and the reconstructed chair box fully contains the backpack proposal 3DIoU02˙84,measuredposthoc3D IoU 0\.284, measured post hoc, although ScanNet annotates them as distinct instances. This shows that spatial overlap alone does not establish object identity. GEAR checks the reconstruction against chair candidates and activates the repaired chair without conflating it with the backpack.

BB Target-anchor evidence separation. For a query referring to a picture above another picture, 97% of TAB's predicted box volume lies inside the annotated lower picture. GEAR instead assigns the lower picture the anchor role and grounds the upper picture as the target, keeping anchor evidence distinct from target evidence.

CC Geometric relation re-evaluation after box updates. TAB's final box spans only 52% of the annotated table length, while GEAR's initial reconstruction also covers only one end. For the same window proposal, the "below" relation scores are 0.471 for GEAR's partial box (offline replay) and 0.623 for its completed box (logged execution). This illustrates why spatial relations must be re-evaluated after bounding-box updates rather than inherited from earlier geometry.

Together, these examples motivate GEAR's candidate-bound evidence memory, which preserves object identity, target/anchor roles, and bounding-box consistency across grounding operations.

W2/Q2. Dependence on Mask3D proposals.

The method uses Mask3D

\.\.\.

, but its sensitivity to the choice and quality of these proposals remains unclear. Ablation studies with alternative segmentation methods or proposal sources would help

\.\.\.

.

We thank the reviewer for suggesting this experiment. To examine whether GEAR’s gains are specific to Mask3D, we additionally evaluate GEAR and TAB using proposals generated by ISBNet, an alternative 3D instance segmentation model. Under the ISBNet configuration, GEAR achieves 63.8% [email protected] and 54.6% [email protected], compared with 60.0% and 53.3% for TAB, yielding improvements of 3.8 and 1.3 percentage points, respectively.

Together with the Mask3D results, where GEAR outperforms TAB by 1.2 and 1.6 percentage points, these results show that the observed advantage is retained across both evaluated proposal sources and both IoU thresholds. This provides direct evidence that GEAR’s effectiveness is not specific to Mask3D.

W3/Q3. Inference Cost Relative to TAB

I am not yet convinced that reduced inference cost is sufficiently substantiated

\.\.\.

. Existing zero-shot approaches

\.\.\.

perform 3d understanding without relying on 3d pretrained proposal model, saving the cost of training a 3d model. In contrast, the proposed method combines LLM inference with

\.\.\.

3d pretrained model, which actually costs more than Agent3D-Zero

\.\.\.

.

Thank you for raising this distinction. Agent3D-Zero is a VLM-based agent for zero-shot 3D scene understanding. Its published evaluation covers tasks such as 3D question answering and semantic segmentation, so it does not provide a task-matched cost comparison for predicting a 3D bounding box from a referring expression. Methods such as VLM-Grounder that do not rely on a pretrained 3D proposal generator report substantially lower grounding accuracy than GEAR on ScanRefer GEAR vs\. VLM-Grounder, **60\.4% / 52\.4% vs\. 51\.6% / 32\.8%** on Acc@0\.25 / Acc@0\.50.

We will explicitly clarify in the paper that our inference-cost claim concerns comparisons with methods that use proposals from a pretrained 3D proposal generator under the evaluation protocol, such as TAB with Mask3D-based output matching. In Table 3, GEAR uses 375.17k vs. 703.04k tokens and takes 472.83 vs. 982.18 seconds per query on average compared with reproduced TAB, corresponding to 46.6% fewer tokens and 51.9% lower elapsed time.

W4/Q4. Cross-view association and memory updates.

a. How are observations associated across views?

In particular, how are observations of the same object across different views associated with a single persistent memory entry and a consistent object ID?

Each ScanRefer or Nr3D query refers to a specific target instance. Accordingly, GEAR performs query-conditioned cross-view association during target repair, rather than global multi-object tracking. It does not maintain persistent entity identities across video frames or create a separate memory entry for each observed object.

Specifically, during RGB-D repair, cross-view association is achieved through reference-conditioned target re-identification rather than matching frame-local mask IDs. Starting from a reference frame with the target explicitly marked, GEAR processes adjacent frames sequentially. Qwen first determines whether the same target remains visible by comparing the new frame against the accumulated target-marked views. If present, SAM generates candidate masks in the new frame, and Qwen selects the mask corresponding to the reference target using the query and visual references. The selected mask is associated with the same target hypothesis, and its marked view is added to the reference set for subsequent frames. If the target is no longer visible, expansion stops in that temporal direction.

GEAR then revisits the retained frames and uses Qwen to reselect one target-consistent mask per frame from the SAM candidates, conditioned on the query and accumulated target-marked references. Frames without valid selections are discarded. The selected depth pixels are back-projected into a common 3D coordinate system using calibrated camera poses. After outlier filtering, GEAR retains the largest DBSCAN cluster when available, excluding points outside the retained cluster and their corresponding 2D pixel contributions. This consolidates the selected observations into a single 3D target hypothesis. For further refinement, GEAR projects the reconstructed centroid into additional views, checks depth occlusion, and uses the projected location to guide second round of SAM mask selection and reconstruction.

b. How are identity switches, duplicate entries, and incorrect merges handled?

How does the method prevent identity switches, duplicate entries

\.\.\.

or the incorrect merging of distinct objects?

  • Identity switches. GEAR does not perform global multi-object tracking, as each ScanRefer/Nr3D query specifies a single target instance. Thus, conventional identity switches between multiple tracked objects do not arise; GEAR only needs to preserve the identity of the query-specific target during repair. To avoid mistakenly selecting a different instance, GEAR employs two safeguards. First, visual verification: Qwen checks whether the observed object matches the intended target using accumulated target-marked views and the attributes and spatial relations in the query, rejecting observations when target identity cannot be confirmed. Second, geometric filtering: outlier removal and DBSCAN clustering exclude spatially disconnected distractor points and their corresponding 2D pixel contributions.

  • Duplicate entries. Since each ScanRefer/Nr3D query specifies a single target instance, GEAR reconstructs one query-specific 3D hypothesis, rather than creating separate instances for observations from different views. Specifically, the selected 2D masks are back-projected into a common 3D coordinate system and consolidated through spatial clustering into a single reconstructed hypothesis. All retained views within one repair contribute to this hypothesis, while subsequent repairs update the same episode-local repaired:video_lifted_target entry rather than creating new IDs. Thus, duplicate repair entries for the same query are avoided by design.

  • Incorrect merges. GEAR employs visual and geometric safeguards to reduce the incorrect merging of distinct objects. First, Qwen selects only the mask corresponding to the query-specific target in each frame, limiting contamination from nearby instances. Second, the selected depth pixels are back-projected into a common 3D coordinate system, where outlier removal and DBSCAN clustering help separate spatially disconnected objects. Third, centroid reprojection with depth-occlusion checks guides a second round of mask selection and reconstruction, improving geometric consistency across views. Finally, GEAR checks the reconstructed box against existing proposals before activating it for grounding, withholding trust when geometric inconsistencies indicate potential target drift.

c. Supplementary material.

\.\.\.

pseudocode and a worked example.

We have added a complete worked example and two pseudocode algorithms in Appendix A.7 (Figure 5 and Algorithms 1–2).

W5/Q5. Novelty relative to ZSVG3D and S-Agent.

The methodological novelty

\.\.\.

needs to be articulated more clearly. Related components already appear in existing approaches. For example, ZSVG3D also combines object localization with LLM-generated spatial reasoning programs, and S-agent[2] also maintains object-centric memory

\.\.\.

across views

\.\.\.

. I strongly suggest the authors further clarify the method's novelty and uniqueness.

We agree that ZSVG3D and S-Agent share important components with GEAR. ZSVG3D uses an LLM-generated visual program to compose localization and spatial operators. S-Agent acquires visual and geometric evidence on demand and accumulates it in entity-centric memory across views.

GEAR’s contribution is to make candidate geometry revisable during proposal-based 3D grounding. The solver starts with existing 3D proposals and reports which constraints remain unresolved. This feedback can trigger a targeted appearance check or RGB-D repair. GEAR records the source and target or anchor role of each hypothesis, checks whether a repaired hypothesis should enter the active candidate set, and can rerun geometric reasoning on the updated box before accepting it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.