acceptodds
Withdrawn from ICLR 2027

DACA: Dual-Anchor Collaborative Agent for Zero-Shot 3D Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D visual grounding (3D-VG) is a task that bridges the natural language and 3D space, serving as a foundational capability for embodied intelligence and human-computer interaction. Although recent 3D-VG agents overcome the reliance on pre-processed 3D point clouds, they essentially employ a unidirectional, pipeline-style tool-calling architecture. This architecture separates semantic understanding from geometric reconstruction, leading to error accumulation and an inability to autonomously recover from failures. To address this flaw, this paper proposes the Dual-Anchor Collaborative Agent (DACA), which reinterprets 3D-VG as a semantic-geometric dual-anchor collaborative reasoning paradigm. DACA comprises two complementary specialized tools: the semantic anchor tool holds the visual semantic features of the target and achieves efficient target tracking via optical flow and VLM local verification; the geometric anchor tool holds the 3D spatial geometric features of the target and performs global spatial reasoning via voxelization and multi-keypoint projection. The two components exchange information bidirectionally via a standardized tool call protocol, autonomously deciding on the next perception-action step to form a closed-loop optimization. Experiments on the ScanRefer and Nr3D benchmarks demonstrate that zero-shot DACA achieves an overall [email protected] of 73.6% on ScanRefer and 70.4% overall accuracy on Nr3D, far surpassing the latest zero-shot 3D-VG methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D Visual Grounding, Zero-Shot Learning, Collaborative Agent, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EBpeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DACA, a zero-shot 3D visual grounding framework built on a VLM and visual tools. It extends TAB with optical-flow-guided temporal tracking, multi-keypoint geometric projection, and semantic-geometric cross-validation. The goal is to collect reliable object masks across views and reconstruct the target’s 3D bounding box. Experiments on 250-example subsets of ScanRefer and Nr3D show improvements over TAB, with ablations on the main components.

Critical strengths
  1. The motivation is reasonable. Semantic tracking can drift, while geometric projection can select the wrong object. Checking the two against each other is a useful way to filter unreliable observations.

  2. The results improve over TAB on both benchmarks using the same reported VLM backbone. The cross-validation ablations also show consistent gains. This provides some evidence that the proposed checks help beyond simply adding more views.

Critical weaknesses
  1. The evaluation is too limited to support the main claims. Each benchmark uses only 250 examples. The 2.4 percentage-point gain over TAB corresponds to six additional correct predictions on each dataset. No repeated runs or uncertainty estimates are reported. Both benchmarks also come from ScanNet, so they provide limited evidence of generalization. The paper uses corrected annotations from TAB, but it is unclear whether all baseline numbers use the same examples and annotations. This also makes the claimed advantage over supervised methods hard to assess.

  2. The ablations do not fully separate the proposed mechanism from extra processing. Table 3(a) mainly compares against a single-frame baseline. The cross-validation ablation is useful, but there is no comparison that matches the number of views, tool calls, and iterations while removing adaptive feedback. There is also no direct ablation of TSDF visibility against the previous depth-based check, or flow-guided tracking against VLM tracking under the same view budget. The current results do not show how much of the gain comes from the claimed collaboration.

  3. Several technical details are inconsistent or insufficiently defined. Eq. (2) uses forward-flow coordinates to sample the reference mask, which appears inconsistent with backward sampling in grid_sample. Eq. (5) compares image-plane coordinates, but the corresponding threshold is reported as 0.4 meters. The definitions of view coverage and reconstruction convergence are also missing. These details affect tracking, validation, and termination, so they matter for reproducing the method.

Questions for authors
  1. How were the 250 examples selected for each benchmark? Did all methods use the same examples and corrected annotations? Which baselines did you rerun, and which results were taken from prior work?

  2. Are the gains over TAB consistent across repeated runs and different evaluation subsets?

  3. Does iterative feedback improve results beyond simply adding more views? Please compare with a fixed pipeline using the same number of views and tool calls, and report accuracy after each iteration.

Reviewer_gDtWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DACA addresses zero-shot 3D visual grounding through semantic–geometric collaboration. It combines optical-flow-guided semantic tracking with keypoint-based geometric view expansion and bidirectional validation. On 250-example subsets, it reports improvements over TAB of 2.4/4.8 percentage points in ScanRefer [email protected]/[email protected] and 2.4 points in Nr3D accuracy.

Critical strengths
  1. Reasonable motivation. Semantic appearance and spatial consistency provide complementary evidence for distinguishing similar objects and maintaining cross-view identity.

  2. Encouraging internal ablations. Removing either validation direction lowers the reported accuracy, providing preliminary evidence for the usefulness of bidirectional checks.

Critical weaknesses
  1. Evaluation comparability is questionable. The paper evaluates only 250 examples per dataset without clearly establishing how they were selected or whether all baselines, particularly fully supervised methods, used identical examples and corrected annotations. Table 1 raises a specific concern: under standard sample-weighted aggregation, the Unique proportion inferred from DACA’s scores is approximately 36.8%, whereas the supervised ScanRefer row implies approximately 19.4%. Unless different aggregation rules were used, these figures suggest different evaluation compositions. This is an inference from the reported numbers and requires explanation. Moreover, a 2.4-point gain on 250 examples corresponds to only six additional correct predictions.
  2. Disagreement handling is underspecified. The paper does not clearly explain which evidence takes precedence when semantic and geometric checks disagree. Requiring both checks could prevent recovery from semantic failures, while accepting either could propagate errors. Since the geometric anchor originates from semantic predictions, both may also reinforce the same incorrect target.
  3. The accuracy–cost trade-off is unknown. Optical flow, TSDF processing, bidirectional checks, and repeated reconstruction introduce additional computation. Local verification might offset some overhead, but no latency or inference-cost comparison is provided. It is therefore unclear whether the accuracy gains justify the increased complexity.
  4. Figure 1 lacks visual clarity. The crowded layout and unclear information flow make it difficult to distinguish the compared frameworks and understand DACA’s key differences.
Questions for authors
  1. How were the 250 examples selected? Were all baselines—including fully supervised methods—evaluated on exactly the same examples and annotation version? Please explain the approximately 36.8% versus 19.4% Unique proportions implied by Table 1, and provide a larger held-out evaluation with uncertainty estimates.
  2. Compared with TAB under identical settings, what are DACA’s end-to-end latency, module-level runtime, VLM calls, and token consumption? Does the improvement persist under a fixed inference budget?
  3. When semantic and geometric checks disagree, which result is accepted, rejected, or revised? How can the system recover when both anchors initially correspond to the wrong instance?
  4. Can you compare bidirectional collaboration with a one-way pipeline using matched view counts and inference budgets? How much accuracy and cost does each additional iteration contribute?
Reviewer_51LPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DACA, a zero-shot 3D visual grounding framework that extends TAB with motion-guided semantic tracking, keypoint-aware geometric extension, and dual-anchor collaboration. It achieves improved performance on ScanRefer and Nr3D.

Critical strengths

The method achieves state-of-the-art performance on two zero-shot 3D visual grounding benchmarks.

Critical weaknesses
  1. Limited Novelty. MG-STE and KA-MGE are incremental extensions of STE and MGE in TAB, mainly incorporating existing optical flow and geometric techniques. The overall methodological novelty is limited.
  2. Missing Critical Ablations. The paper lacks direct comparisons between MG-STE and STE, and between KA-MGE and MGE. The current ablations do not adequately isolate the contributions of the proposed improvements over TAB.
  3. Missing Inference Cost Analysis. The paper does not report the additional computational overhead introduced by dual-anchor cross-validation and iterative refinement compared to TAB. Given the relatively modest performance gains, an efficiency analysis is necessary.
Questions for authors

see weakness 2 and weakness 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.