- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces GEAR, an evidence-adaptive VLM agent for zero-shot 3D visual grounding. It combines proposal-based geometric reasoning with selective visual verification and on-demand RGB-D reconstruction to repair missing or unreliable candidates. Candidate-bound memory maintains consistent evidence across operations, while a structured geometric solver and terminal validation support grounding decisions. Experiments on 250-query subsets of ScanRefer and Nr3D show improved grounding accuracy and approximately 50% lower token usage and total inference time than the reproduced TAB baseline.
Critical strengths
-
The paper addresses the trade-off between efficient proposal-based reasoning and the ability to recover missing objects from RGB-D observations. Selectively acquiring additional evidence is a sensible approach to improving grounding without incurring reconstruction costs for every query.
-
GEAR integrates geometric reasoning, visual verification, and candidate repair within a shared inference loop. Candidate-bound memory explicitly tracks identities, target/anchor roles, and evidence provenance, while terminal validation helps prevent unsupported grounding decisions.
-
Experiments on ScanRefer and Nr3D show higher grounding accuracy than the reproduced TAB baseline, alongside substantially lower token usage and total runtime. Comparisons with a rule-based controller and component ablations provide supporting evidence for the benefits of adaptive control and persistent candidate memory.
Critical weaknesses
-
Experiments use only 250-query subsets of ScanRefer and Nr3D. On ScanRefer, the improvements over the reproduced TAB baseline correspond to just three and four additional correct predictions at [email protected] and [email protected], respectively. Larger-scale evaluation and paired uncertainty analysis would help establish the reliability of these gains.
-
The evaluation matches predicted boxes back to existing Mask3D proposals, making it difficult to distinguish recovery of genuinely missing proposals from recovery of candidates omitted by semantic retrieval. Reporting accuracy before and after proposal matching, together with separate analyses of these two failure cases, would better substantiate the candidate-repair claim.
-
The ablations demonstrate that visual verification and RGB-D repair are useful, but the paper does not clearly quantify how frequently each route is executed or its success rate and cost. Comparisons with always-on reconstruction and simple conditional repair policies would more directly establish the benefits of evidence-adaptive routing.
Questions for authors
Please see the weakness
4 comments
W3/Q3. Routing frequency, success rate, and cost.
\.\.\.The ablations demonstrate that visual verification and RGB-D repair are useful, but
how frequently each route is executed or its success rate and cost. Comparisons with always-on reconstruction and simple conditional repair policies would
\.\.\..
On the original 250 ScanRefer queries, 166 queries used View Probe, while 157 queries executed target RGB-D reconstruction. On their respective query subsets, the boxes selected by View Probe achieved 53.0% / 47.6% [email protected]/0.50, while those produced by RGB-D reconstruction achieved 44.6% / 28.0%. Averaged over all 250 queries, View Probe and Repair requests accounted for 51.6 s/query and 279.0 s/query, respectively.
We further compared alternative routing strategies. The fixed rule-based routing policy (Table 4), which replaces the VLM Controller with predefined evidence-conditioned action selection, achieved 56.4% / 49.2% [email protected]/0.50, compared with 60.4% / 52.4% for GEAR, with comparable inference time 465\.5 vs\. 472\.8 s but fewer tokens 137\.2 K vs\. 375\.2 K. Always-on reconstruction achieved 58.80% / 37.20%, requiring 933.63 s and 683.27 K tokens. A lightweight rule-only baseline achieved 38.78% / 34.18%, requiring only 0.73 s and no VLM tokens.
| Models | mIoU | [email protected] | [email protected] | Time | Tokens |
|---|---|---|---|---|---|
| Fixed rule-based routing | 48.00 | 56.40 | 49.20 | 465.50 | 137.20 |
| Always-on reconstruction | 34.62 | 58.80 | 37.20 | 933.63 | 683.27 |
| Rule-only inference | 33.10 | 38.78 | 34.18 | 0.73 | 0.00 |
| GEAR (adaptive) | 50.38 | 60.40 | 52.40 | 472.83 | 375.17 |
These results suggest that indiscriminate reconstruction incurs substantial inference overhead without necessarily improving grounding accuracy, while purely rule-based inference trades grounding accuracy for computational efficiency. Furthermore, fixed conditional routing reduces token consumption but yields lower accuracy than GEAR. By selectively acquiring additional evidence, GEAR achieves higher grounding accuracy while avoiding the substantial overhead of always-on reconstruction.
Thank you for the clarifications and additional analyses. They address most of my concerns. I will maintain my positive scores.
We sincerely thank the reviewer for the constructive feedback and positive assessment. We are glad that our additional analyses have addressed most of the concerns. We will incorporate these clarifications and results into the revised manuscript.
Response to Reviewer EC8R
We thank the reviewer for the positive assessment of GEAR’s evidence-adaptive design, candidate-bound memory, and accuracy–cost trade-off. We agree that the three requested analyses are important for clarifying the strength and scope of our empirical claims.
W1/Q1. Evaluation scale and paired uncertainty.
\.\.\.. Larger-scale evaluation and paired uncertainty analysis would help establish the reliability of these gains.
On the original 250 ScanRefer queries, GEAR alone was correct on 47 queries and TAB alone on 44 at IoU 0.25, giving a net gain of 1.20 percentage points (95% CI
−64˙0,88˙0). At IoU 0.50, the corresponding counts were 42 and 38, giving a gain of 1.60 points (95% CI
−52˙0,88˙0). We computed percentile intervals from 100,000 paired query bootstrap resamples.
We also evaluated both methods on 750 additional ScanRefer validation queries across three disjoint subsets. GEAR achieved 53.73% / 47.60% at IoU 0.25 / 0.50, compared with 52.67% / 44.80% for TAB. The pooled gains were +1.07 and +2.80 points, with scene-bootstrap 95% CIs of
−32˙0,54˙0and
−11˙4,67˙9, respectively. On a separate 181-query ViGiL3D subset used exclusively for evaluation, GEAR achieved 37.56% mIoU and 45.53% / 37.44% accuracy, compared with 33.09% and 42.54% / 32.60% for TAB.
W2/Q2. Proposal matching and the interpretation of candidate repair.
a. Before and after matching.
\.\.\.For a fair comparison with TAB, we adopt the same final proposal-matching rule. The final proposal macthing evaluator replaces a valid predicted box with its highest-IoU Mask3D proposal only when their 3D IoU exceeds 0.25; Otherwise, the prediction is retained. On the same 250 ScanRefer queries:
We now report both native and matched outputs on the same 250 ScanRefer queries. Before matching, GEAR exceeds TAB by 13.2 percentage points at [email protected]. Matching increases [email protected] by 2.0 points for GEAR and 13.6 points for TAB, substantially narrowing the difference between the native outputs. These results quantify the contribution of post-processing and show that GEAR’s native localization quality is not solely a consequence of the final matching step.
b. Retrieval omissions and missing proposal coverage.
\.\.\., together with separate analyses of these two failure cases,
\.\.\..
At each IoU threshold, we compared the maximum ground-truth IoU over the full Mask3D proposal set with that over GEAR's initial target retrieval. A retrieval miss occurs when a qualifying proposal exists in the full set but is absent from initial retrieval, whereas a coverage miss occurs when no original proposal meets the threshold. We evaluated GEAR's native predictions on these two disjoint groups, before final proposal matching.
At the standard IoU thresholds of 0.25/0.50, GEAR successfully localized 9/17 and 9/40 coverage misses, respectively, alongside 8/27 and 6/26 retrieval misses. Even at IoU 0.10, where no original proposal provides minimal target coverage, GEAR localized 4 of 6 cases. These pre-matching results demonstrate that GEAR can localize targets beyond the original proposal coverage, rather than merely recovering candidates omitted during semantic retrieval.