acceptodds
Withdrawn from ICLR 2027

Select, Compete, Refine: Training-Free 3D Visual Grounding from RGB-D Views

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free 3D visual grounding from calibrated RGB-D views enables language-conditioned object localization without task-specific model training or preconstructed scene point clouds. However, selecting the object that satisfies a full referring expression remains ambiguous, and geometry reconstructed from partial masks and depth can yield incomplete object boxes. We propose Select, Compete, Refine (SCR), a framework that combines clause-scoped object selection and verification with multi-view boundary recovery using frozen pretrained models. Select organizes entity-level options and clause-bound references. Compete separates challenger discovery under withheld candidate evidence from replacement verification after evidence restoration. Refine recovers observed but underrepresented extent through adaptive per-view bounds and face-wise multi-view support, without additional segmentation or vision-language model calls. We evaluate SCR on SeqAlign3DVG, EmbodiedScan and ScanRefer. On SeqAlign3DVG, SCR achieves 62.96% [email protected] and 38.07% [email protected], improving over our VLM-Grounder reproduction using the same Qwen model by 11.38 and 12.45 percentage points, respectively. Ablation studies further support the benefits of object selection, verification, and boundary recovery.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D visual grounding, training-free methods, vision-language models, multi-view RGB-D understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d28jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCR is a training-free pipeline for 3D visual grounding from posed RGB-D views. A Qwen model parses the query into a target phrase and typed clauses; SAM3.1 segments the views; the masks are back-projected and associated across views into candidate boxes and a separate reference bank. Select groups duplicate candidates into answer options, binds reference hypotheses to the clauses they serve, and shows the VLM three planned views plus detail crops. Compete, invoked by a fixed risk router, hides the initial choice's crop and labels to elicit a challenger, then restores both and accepts the challenger only if a structured acceptance gate passes. Refine expands box faces using per-view quantiles of the selected object's dense mask depth, trimmed by about sqrt(N) points per tail, and requires two views per moved face plus safety checks.

Critical strengths

The geometric analysis is careful and instructive. Table 5 fixes the object selections and varies only the box estimator: using all mask depth points with pooled 2-98% quantiles already moves [email protected] from 21.98 to 26.54, raw min/max hurts, and Refine adds a large improvement on top of every reasonable initializer. The paper also says plainly what Refine cannot do (shrink oversized boxes, recover unobserved surfaces). The VLM-Grounder reproduction uses the same Qwen model, which is the right control for the reasoning side.

Critical weaknesses

The headline gain comes mostly from box post-processing rather than from the selection and verification ideas the paper is built around. In Table 4, Select and Compete add 2.81 and 1.44 at [email protected], while Refine adds 7.34 / 16.09; without Refine, SCR is 3.6 points below VLM-Grounder at [email protected], and Refine was never applied to that baseline. Withhold-restore adds 0.61 over plain reconsideration in a single run, and on ScanRefer SCR (41.03 / 24.41) trails the training-free SeeGround (44.1 / 39.4).

The numbers cannot yet be read as held-out results. The system has more than a hundred hand-set constants, including 27 category size prototypes (Table 7); the Compete router was selected on 500 annotated "development rows" whose relation to the 2,916 SeqAlign3DVG evaluation queries is not stated; and the EmbodiedScan run uses a benchmark-specific language adapter.

Questions for authors
  1. Are the 500 development rows part of the 2,916 SeqAlign3DVG evaluation queries? If so, please report results on the disjoint remainder. How were the other constants and prototype sizes chosen?
  2. VLM-Grounder with the same Qwen model and SCR's Refine applied to its selected object, so that the selection comparison is isolated.
  3. EmbodiedScan without the distance adapter, and VLM-Grounder on ScanRefer under a matched subset.
  4. A paired test (e.g., McNemar) for withhold-restore against intact reconsideration (62.96 vs 62.35 on the same routed queries).
  5. Why Qwen3.5-9B on ScanRefer instead of the 27B model used for the other two benchmarks?
Reviewer_oWq7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SCR, a training-free pipeline for 3D visual grounding from posed RGB-D views. It consists of three steps: Select binds reference objects to the query clauses they describe; Compete first withholds the VLM's initial choice to elicit an alternative, then restores the evidence and lets the VLM compare the two candidates; Refine expands the selected box using multi-view depth support. SCR improves over a same-model VLM-Grounder reproduction by about 11-12 points on SeqAlign3DVG, with smaller gains on EmbodiedScan and ScanRefer.

Critical strengths
  1. The Compete step is interesting: by eliciting a challenger under withheld evidence and accepting it only after a restored, anonymous comparison, it gives the VLM a chance to escape its initial preference.

  2. The method is training-free, needs no pre-built point cloud, and is faster than VLM-Grounder (33 s vs. 72 s per query) while being clearly more accurate on SeqAlign3DVG.

Critical weaknesses
  1. The pipeline introduces many hand-crafted hyperparameters, and the routing policy is tuned on annotated SeqAlign3DVG dev rows. It is unclear whether the method generalizes beyond this benchmark: on EmbodiedScan, Top-1 improves over the same-model VLM-Grounder by under 1 point (15.16/6.73 vs. 14.25/6.05), and on ScanRefer the only RGB-D comparison is on a 14-scan subset, where [email protected] is nearly identical (26.42 vs. 26.0).

  2. Most of the gain comes from Refine ([email protected]: ~22 -> ~37), a model-free step that recovers box extent lost during candidate filtering. It could be applied to any pipeline, so the comparison does not isolate the contribution of the proposed selection mechanism. Compete itself adds only ~1.5 points.

Questions for authors

What is the result of VLM-Grounder + Refine, and of VLM-Grounder using the same SAM3.1 candidates?

Reviewer_KDg4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SCR (Select, Compete, Refine), a training-free pipeline for 3D visual grounding from calibrated RGB-D views. It uses no scene point cloud or external proposals. SAM 3.1 masks are back-projected and associated across views to form candidates. Select groups duplicate candidates into entity-level answer options and binds reference objects to the clauses they serve, then a frozen Qwen VLM picks an initial target from joint views and detail crops. Compete is triggered by a hand-designed risk router. It hides the initial choice's crop and labels to elicit a challenger, then restores evidence and accepts the challenger only if a gated pairwise comparison supports it. Refine expands the selected box face by face using per-view quantile bounds with a point-count-adaptive trim, requiring two-view support and safety checks. On SeqAlign3DVG, SCR reaches 62.96 / 38.07 [email protected]/0.50, which is 11.38 / 12.45 points above a same-Qwen VLM-Grounder reproduction. The paper also reports EmbodiedScan Mini and ScanRefer results.

Critical strengths
  1. On SeqAlign3DVG the comparison is controlled. VLM-Grounder is re-run with the same Qwen3.6-27B, and SCR improves [email protected]/0.50 by 11.4 / 12.5 points at about half the recorded cost per query. The paired geometry controls in Table 5 (fixed selections, different initial box estimators, same Refine policy) are a good experimental design. They show that the gain is not simply "use more points".
  2. The appendix specifies the pipeline in enough detail (every threshold, score, and gate) that it could be reimplemented. It is also candid about what each heuristic does and does not guarantee.
Critical weaknesses
  1. The paper's main framing (object selection and verification) is not where most of the gain comes from. In Table 4, Select improves over unscoped context by 2.8 / 1.4 points, and Compete adds 1.4 / 0.5 (or 1.6 / 1.3 on top of Refine). Compete is only 0.6 / 0.7 points better than simple re-asking with intact evidence (62.35 / 37.41), with no variance reported. Refine alone adds 7.1 / 15.3 points. Table 5 also shows that simply fitting pooled 2–98% quantiles to all source-mask points, without Refine, gives 59.40 / 26.54. That beats the paper's own initial boxes (55.62 / 21.98) and is above Select + Compete. The authors acknowledge (Appendix A.7) that Refine's benefit depends on how many points their own candidate construction discarded. A large part of the improvement over VLM-Grounder is therefore box fitting in the authors' front end, not the selection/verification mechanism the paper is built around. On EmbodiedScan, where selection matters more, Top-1 improves over the same-Qwen VLM-Grounder by only 0.91 / 0.68 points.
  2. The method is a large set of hand-tuned rules, and some were tuned with annotations. The pipeline contains dozens of fixed weights and thresholds: relation scores in Eqs. 21–29, a manual table of category size priors (Table 7), the router weights, and a router threshold of 0.33806227953614815. There is also a dataset-specific rewrite for EmbodiedScan that maps "near/far" to "nearest/farthest". Appendix A.5.5 states that the routing policy was selected on 500 "declared development rows" using annotations, but does not say which split these rows came from. If they overlap the SeqAlign3DVG evaluation queries, the Compete numbers are optimistic. Beyond the specific numbers, this design gives limited insight that would carry over to other models or datasets, which matters for an ICLR contribution. The efficiency claim also rests on an offline replay of cached calls (A.5.5), not a measured end-to-end run, so the "33.31 vs. 72.26 s/query" comparison may not be like-for-like.
Questions for authors

Clarifications that could change my assessment:

  1. Which split do the 500 development rows used to select the Compete router come from? Do they overlap the reported SeqAlign3DVG / EmbodiedScan / ScanRefer evaluation queries?
  2. What does VLM-Grounder (same Qwen) achieve if its final box is replaced by your Refine output, or by dense pooled 2–98% quantiles? This would separate the selection contribution from the geometry contribution.
  3. Is the 33.31 s/query measured end to end, or obtained from cached replay? Please report a cold end-to-end measurement for both methods on the same hardware.
  4. How sensitive are the results to the hand-set weights and thresholds (e.g., ±20% on the router threshold and relation weights)? How were the category size prototypes chosen?

Suggestion (not required for my decision): on ScanRefer, the RGB-D comparison relies on a 14-scan subset; comparing with TAB and SceneGraphGrounder on the full validation set would make the claim stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.