acceptodds
Under review as a conference paper at ICLR 2027

Help or Hinder? View Preselection in VLM-Based Zero-Shot 3D Grounding

Abstract

Zero-shot 3D visual grounding often restricts a scanned scene to a small set of query-relevant views before a vision–language model (VLM) makes its grounding decision. This reduces computation, but it also removes visual evidence before the grounding model can assess it. We ask whether the accuracy effect of this restriction changes with the grounding VLM. We introduce Observe-Then-Ground (OTG), a controlled evaluation protocol in which the VLM selects one image and localizes the target in 2D, and a fixed downstream pathway performs segmentation, RGB-D lifting, and 3D instance assignment. Only the observations supplied to the VLM are varied. Across the same nested CLIP-ranked Top-3/6/12/24 inputs, dense Qwen3-VL checkpoints are less accurate as the input expands beyond their best tested compact condition, whereas GPT-5.6 and Gemini improve through Top-24. Paired-query analysis shows why aggregate accuracy alone is insufficient: more views can recover targets absent from a smaller input, but can also change grounding when the target was already visible. From Top-6 to Top-24, target recovery accounts for most of the net gain for GPT and Gemini, while all dense Qwen checkpoints lose final 3D accuracy on the same already-covered queries. Thus, under this controlled CLIP-ranked interface, query-dependent preselection is an accuracy-relevant component of the grounding system, and its effect differs across the VLM configurations evaluated here.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Visual Grounding, 3D Scene Understanding, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qgk8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the quantity of preselected input views affects vision-language models (VLMs) in zero-shot 3D visual grounding by introducing the Observe-Then-Ground (OTG) evaluation protocol, which fixes the downstream 3D instantiation pathway to isolate the impact of the visual evidence budget. Experimental results reveal that while GPT-5.6 and Gemini steadily improve their grounding accuracy as the evidence budget expands to Top-24, dense Qwen3-VL checkpoints experience performance degradation due to losing final 3D accuracy on already-visible targets. Ultimately, this paired-query analysis demonstrates that view preselection is not a model-independent computational shortcut, but rather an accuracy-relevant design choice that interacts deeply with the specific VLM and must be evaluated jointly.

Critical strengths

(1) The paper addresses a clear and practically relevant question: whether view preselection helps or hinders zero-shot 3D grounding across different VLMs. Studying the trade-off between reducing computation and preserving useful visual evidence offers valuable guidance for jointly choosing the grounding model and view budget.

(2) OTG provides a well-controlled evaluation protocol by fixing the VLM output format and downstream 2D-to-3D pipeline while varying only the supplied observations. The nested Top-k inputs derived from a shared ranking further reduce confounding factors, enabling a clearer analysis of view preselection.

(3) The experimental analysis goes beyond aggregate accuracy by distinguishing recovery of initially omitted targets from grounding changes on already-covered queries. Paired-query comparisons, direct 2D localization metrics, and scene-clustered bootstrap intervals provide stronger evidence for the model-dependent effects of view budgets.

Critical weaknesses

(1) The main study is restricted to 250 ScanRefer queries and a single PE-Core ranking. The alternative-selector experiment covers only GPT at Top-6, leaving it unclear whether the observed model-dependent responses persist across datasets, selectors, and sampling strategies. A broader selector × model × budget evaluation would strengthen the conclusions.

(2) Increasing the view budget simultaneously changes target coverage, distractor content, sequence length, and the positions of existing views. Although the paired analysis separates target recovery from changes on already-covered queries, it does not isolate why additional views harm some models. Target-preserving distractor controls and repeated order perturbations across backbones would provide stronger mechanistic insight.

(3) Each model–budget condition uses only one saved generation per query. The scene-clustered bootstrap intervals quantify uncertainty from scene resampling, but do not capture stochastic variation in model outputs. This is particularly relevant for small differences, such as GPT’s one additional correct prediction from Top-24 to Full. Repeated runs would help establish the stability of these findings.

(4) Lack of discussion of recent and highly relevant works [1]-[3]

[1] Visual programming for zero-shot open-vocabulary 3d visual grounding, 2024 CVPR

[2] Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding, 2026 EMNLP

[3] Solving zero-shot 3d visual grounding as constraint satisfaction problems, arXiv:2411.14594

Questions for authors

Please see the weakness.

12 comments
AuthorsOpenReview ↗

Response to Reviewer Qgk8

We thank you for highlighting the factors that change simultaneously as the view budget increases. Your observations have helped guide the additional analyses presented below.

W1. Selector × model × budget and broader scope

You ask whether the model-dependent responses to increasing view budgets persist across different view selectors and suggest a broader evaluation spanning view selectors, models, and view budgets. To address this, we have extended our experiments to compare two view selection methods, CLIP (the PE-Core ranking used in the original submission) and SigLIP2, across all six VLMs at Top-3/6/12/24. Both selectors are evaluated using the same ScanRefer evaluation protocol and stride-30 candidate pool. In the original submission, SigLIP2 was evaluated only for GPT at Top-6 (Table 2).

The results demonstrate that the contrasting model-dependent trends persist across both view selection methods. As the view budget increases from Top-6 to Top-24, accuracy decreases for all four Qwen models (2B/4B/8B/32B), by 10.3/9.5/6.3/2.8 percentage points (pp) with CLIP and 14.0/8.8/8.4/4.4 pp with SigLIP2. In contrast, accuracy increases slightly for the more powerful VLMs, GPT and Gemini, by a mean of 1.0/1.9 pp with CLIP and by 1.2/1.2 pp with SigLIP2.

These findings confirm that the contrasting responses to increasing view budgets are consistent across the two view selectors evaluated, supporting our central conclusion that the influence of view budget on grounding accuracy depends primarily on the VLM rather than the choice of view selector.

Table 2. Selector × model × budget results with the SigLIP2 view selector: 3D bbox [email protected] (%).

ModelTop-3Top-6Top-12Top-24Top-24 minus Top-6 (pp)
Qwen3-VL-2B42.842.037.228.0−14.0
Qwen3-VL-4B48.450.044.441.2−8.8
Qwen3-VL-8B49.649.248.040.8−8.4
Qwen3-VL-32B54.054.852.450.4−4.4
GPT-5.656.058.860.060.0+1.2
Gemini 3.8 Flash57.260.862.062.0+1.2

3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. The last column is Top-24 minus Top-6 in percentage points (pp). One run per condition. View selector: SigLIP2. Views in chronological order.

W2. Coverage, distractors, sequence length, and presentation order

You ask how we can identify why additional views degrade performance in some models, given that increasing the view budget simultaneously changes target coverage, distractor content, sequence length, and the positions of existing views. To distinguish these effects, we have extended our analysis to all six VLMs through three complementary comparisons, each controlling for different factors. Previously, only target coverage was examined separately, and the only presentation-order control involved a single shuffled ordering for GPT at Full.

The first comparison holds target coverage fixed by considering the 235 queries whose targets are already visible at Top-6 and therefore remain visible at Top-24. The second focuses on the 66 Unique queries, for which the target is the only object of its class in the scene, thereby excluding the possibility that additional views introduce same-class distractors. The third examines the influence of presentation order by comparing two arrangements of the same selected images: chronological order, which follows the temporal sequence of the scan, and relevance order, which ranks views by their similarity to the query. Importantly, relevance ordering preserves the original six views in the first six positions as additional views are introduced. Since both image ordering methods use exactly the same selected images at each view budget, distractor content and sequence length remain unchanged between them.

The results demonstrate that the performance degradation of the smaller Qwen models (2B/4B/8B/32B) persists even when target coverage is held fixed and when same-class distractors are absent. However, changing only the presentation order substantially reduces this degradation in direct 2D accuracy for all four Qwen models, with corresponding improvements in final 3D accuracy most pronounced for Qwen 2B, 4B, and 8B.

When target coverage is held fixed, increasing the view budget from Top-6 to Top-24 still reduces mean final 3D accuracy by 11.1/10.5/7.5/3.7 percentage points (pp) for Qwen 2B/4B/8B/32B, respectively (Table 3). For the Unique queries, where same-class distractors are also excluded, the corresponding decreases are 14.6/13.1/8.6/9.6 pp (Table 7), demonstrating that the degradation is at least as pronounced in the absence of such distractors.

W2 (continued)

In contrast, presentation order has a substantial influence on performance: relevance ordering considerably reduces the performance degradation as the view budget increases to Top-24. For the 235 queries with fixed target coverage, the two orderings produce similar accuracy at Top-6, while the decreases in direct 2D accuracy from Top-6 to Top-24 are reduced from 7.2/15.0/11.5/17.9 pp under chronological ordering to just 0.4/6.7/4.3/4.0 pp under relevance ordering. Similarly, the decreases in final 3D accuracy are reduced from 11.1/10.5/7.5/3.7 pp to 4.5/4.3/3.1/2.6 pp (Table 4). The effect is particularly striking for the Unique queries, where the changes in direct 2D accuracy improve from −4.0/−16.2/−21.2/−26.8 pp under chronological ordering to +2.0/−3.0/−3.5/−3.0 pp under relevance ordering (Table 7).

By contrast, the more powerful VLMs, Gemini and GPT, remain substantially more robust to increasing view budgets, with changes in final 3D accuracy of no more than 1.3 pp from Top-6 to Top-24 across these comparisons. Together, these findings indicate that neither target coverage nor same-class distractors adequately explain the performance degradation of the smaller VLMs, while the substantial improvements achieved through relevance ordering highlight the importance of how additional views are presented.

The comparison of presentation orders also provides further insight into the source of the performance degradation observed in the smaller VLMs. Under both ordering methods, Qwen 4B, 8B, and 32B continue to select images containing the target almost as frequently at Top-24 as at Top-6, indicating that their loss in 2D accuracy arises primarily from less accurate bounding-box localisation rather than incorrect image selection (Table 4).

For all four Qwen models, bounding-box localisation is consistently more accurate when the selected image occupies one of the first three positions in the input sequence, regardless of presentation order or view budget. At Top-24, this occurs for 54–76% of selected images under relevance ordering, compared with only 18–28% under chronological ordering (Table 5).

In contrast, although Gemini and GPT also select images from later positions as the view budget increases, their bounding-box accuracy is considerably less sensitive to the position of the selected image. At Top-24, the difference in accuracy between selections from the first three positions and those from later positions is less than 8 percentage points (pp) for Gemini and GPT, compared with 12–30 pp for the Qwen models.

Since relevance ordering keeps the original six views in the first six positions and only appends the additional views, the performance degradation that persists under relevance ordering reflects the combined effects of increased sequence length and additional visual content. These findings demonstrate that additional views can degrade the performance of smaller VLMs even when target coverage is fixed and same-class distractors are absent, primarily through less accurate bounding-box localisation for Qwen 4B, 8B, and 32B. However, this degradation is substantially reduced when the additional views are appended after the more relevant views. In contrast, neither Gemini nor GPT shows a comparable gain from relevance ordering. These contrasting responses demonstrate that the same additional views affect the two groups of VLMs differently, reinforcing our central conclusion that view preselection should be evaluated in conjunction with the grounding VLM.

Table 3. Change from Top-6 to Top-24 for queries whose targets are visible at Top-6 and for queries whose targets are visible only at Top-24.

ModelTargets visible at Top-6 (235 queries): 2D bbox [email protected] Top-24 minus Top-6 (pp)Targets visible at Top-6 (235 queries): 3D bbox [email protected] Top-24 minus Top-6 (pp)Targets visible only at Top-24 (15 queries): 3D bbox [email protected] at Top-24 (%)All 250 queries: 3D bbox [email protected] Top-24 minus Top-6 (pp)All 250 queries: of which from targets visible at Top-6 (pp)All 250 queries: of which from targets visible only at Top-24 (pp)
Qwen3-VL-2B−7.2 ± 0.9−11.1 ± 3.42.2 ± 3.8−10.3 ± 3.3−10.4 ± 3.2+0.1 ± 0.2
Qwen3-VL-4B−15.0 ± 0.9−10.5 ± 0.96.7 ± 0.0−9.5 ± 0.8−9.9 ± 0.8+0.4 ± 0.0
Qwen3-VL-8B−11.5 ± 0.4−7.5 ± 1.513.3 ± 0.0−6.3 ± 1.4−7.1 ± 1.4+0.8 ± 0.0
Qwen3-VL-32B−17.9 ± 4.5−3.7 ± 4.011.1 ± 3.8−2.8 ± 4.0−3.5 ± 3.8+0.7 ± 0.2
GPT-5.6−0.9 ± 2.7−0.8 ± 2.128.0 ± 5.6+1.0 ± 1.9−0.7 ± 1.9+1.7 ± 0.3
Gemini 3.8 Flash+0.9 ± 1.3−0.4 ± 0.938.7 ± 3.0+1.9 ± 0.7−0.4 ± 0.8+2.3 ± 0.2

Targets visible at Top-6: the target appears in the Top-6 input, and therefore also in the Top-24 input (N=235). Targets visible only at Top-24: the target first enters the input at Top-24 (N=15); at Top-6 the accuracy on these queries is zero. [email protected]: the percentage of the queries of a group whose bbox has IoU ≥0.50 with the target. The last two columns split the change over all queries into the part that comes from each group: the change in the number of correct 3D bboxes in that group, as a percentage of all 250 queries. They add up to the change over all queries before rounding. Values are means ± sample SD over three runs for Qwen and five runs for GPT/Gemini; changes are computed within each run. View selector: CLIP. Views in chronological order.

Reviewer_mCWUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how view preselection affects zero-shot VLM-based 3D grounding. It introduces the Observe-Then-Ground (OTG) evaluation protocol, which fixes the downstream grounding pipeline while varying only the visual observations provided to the VLM. Experiments across Qwen3-VL, GPT-5.6, and Gemini show that the effect of increasing the view budget is model-dependent: more views can recover previously unseen targets but can also hurt grounding on already-covered queries. The paper concludes that view preselection should be treated as an accuracy-relevant system design choice rather than merely a computational optimization.

Critical strengths

The main strength is the controlled experimental design, which isolates the effect of view preselection by fixing the downstream grounding pipeline and varying only the observations given to the VLM. This makes the empirical conclusions relatively easy to interpret.

Critical weaknesses
  1. The scope is tied to one specific OTG pipeline. OTG requires the VLM to select one image and localize the target in 2D, followed by a fixed segmentation, RGB-D lifting, and 3D instance-assignment pipeline. Existing zero-shot 3D grounding methods use substantially different evidence construction and downstream grounding mechanisms, so it is unclear whether the conclusions transfer to those settings.

  2. The main conclusion may depend on the choice of view selector. Most experiments use a fixed CLIP/PE-Core ranking. Since the selector determines which views are added as the budget increases, the observed model–budget interaction may partly reflect the properties of this particular ranking strategy.

  3. The empirical findings are largely model-specific. The main observation is that Qwen, GPT, and Gemini respond differently to increasing view budgets. While useful, the paper does not identify a general mechanism that predicts when additional views will help or hurt a given model.

  4. There is limited methodological novelty. The paper does not introduce a new grounding model, training objective, or view-selection algorithm. Its main technical contribution is the controlled OTG evaluation protocol and associated diagnostic analysis.

  5. The work does not provide an actionable selection strategy. Although the paper shows that evidence budget should be considered jointly with the VLM, it does not propose an adaptive rule or selector for deciding which views to keep, how many views to use, or when more evidence is likely to become harmful.

Questions for authors

See weakness

12 comments
AuthorsOpenReview ↗

Response to Reviewer mCWU

We thank you for recognizing the value of our controlled experimental design and for encouraging us to clarify the broader significance of our findings beyond the performance of individual models.

W1. Dependence on the OTG pipeline

You ask whether our conclusions are specific to the OTG pipeline, in which the VLM selects an image and localises the target in 2D, followed by a fixed procedure for segmentation, RGB-D lifting, and 3D assignment. To determine whether the observed model-dependent differences originate in the VLM itself or arise from the downstream processing, we have extended our analysis to examine the VLM's direct outputs: the selected image and predicted 2D bounding box, before any segmentation or 3D conversion. This analysis now covers all six models across repeated runs and under two presentation orders, whereas the original submission reported these results from a single run using one presentation order.

The results demonstrate that the model-dependent differences are already evident in the VLM's direct 2D outputs, independently of the downstream processing. For the 235 queries whose targets are already visible at Top-6, increasing the view budget from Top-6 to Top-24 reduces direct 2D accuracy by 7.2/15.0/11.5/17.9 percentage points (pp) for Qwen 2B/4B/8B/32B under chronological ordering, compared with smaller decreases of 0.4/6.7/4.3/4.0 pp under relevance ordering (Table 4). In contrast, the more powerful VLMs, Gemini and GPT, exhibit changes in direct 2D accuracy of no more than 0.9 pp under either presentation order.

These findings establish that the contrasting model responses to increasing view budgets arise at the VLM grounding stage, where multiple preselected views are processed in a single call, rather than being introduced by the subsequent segmentation and 3D reconstruction procedures. They therefore support the broader conclusion that the influence of view budget on grounding accuracy is fundamentally dependent on the VLM itself, and is not an artifact of OTG's fixed downstream processing pipeline.

Table 4. Image selection accuracy, 2D bbox accuracy and 3D bbox accuracy at Top-6 and Top-24, and the change between them, under the two presentation orders on the 235 queries with targets visible at Top-6.

ModelOrder of input viewsImage selection accuracy: Top-6 (%)Image selection accuracy: Top-24 (%)Image selection accuracy: Top-24 minus Top-6 (pp)2D bbox [email protected]: Top-6 (%)2D bbox [email protected]: Top-24 (%)2D bbox [email protected]: Top-24 minus Top-6 (pp)3D bbox [email protected]: Top-6 (%)3D bbox [email protected]: Top-24 (%)3D bbox [email protected]: Top-24 minus Top-6 (pp)
Qwen3-VL-2BChronological order80.1 ± 1.766.7 ± 0.7−13.5 ± 2.115.5 ± 0.58.2 ± 1.3−7.2 ± 0.944.5 ± 2.133.5 ± 1.4−11.1 ± 3.4
Relevance order80.6 ± 2.877.3 ± 3.0−3.3 ± 5.715.0 ± 1.514.6 ± 1.8−0.4 ± 2.146.5 ± 2.842.0 ± 3.1−4.5 ± 5.6
Qwen3-VL-4BChronological order84.0 ± 1.479.9 ± 1.8−4.1 ± 2.124.7 ± 0.49.6 ± 1.2−15.0 ± 0.952.5 ± 1.542.0 ± 1.0−10.5 ± 0.9
Relevance order84.3 ± 1.181.7 ± 2.6−2.6 ± 1.527.0 ± 2.920.3 ± 4.1−6.7 ± 5.852.1 ± 3.347.8 ± 1.7−4.3 ± 1.9
Qwen3-VL-8BChronological order86.2 ± 1.583.1 ± 1.9−3.1 ± 0.526.0 ± 0.714.5 ± 0.4−11.5 ± 0.453.5 ± 1.546.0 ± 0.0−7.5 ± 1.5
Relevance order84.8 ± 0.285.5 ± 2.0+0.7 ± 1.926.2 ± 1.522.0 ± 0.7−4.3 ± 2.051.8 ± 2.748.7 ± 0.7−3.1 ± 3.0
Qwen3-VL-32BChronological order86.8 ± 3.085.8 ± 0.5−1.0 ± 2.652.1 ± 2.334.2 ± 2.7−17.9 ± 4.558.7 ± 2.655.0 ± 2.3−3.7 ± 4.0
Relevance order85.7 ± 0.987.2 ± 2.2+1.6 ± 2.952.1 ± 2.148.1 ± 1.1−4.0 ± 1.059.6 ± 0.757.0 ± 0.9−2.6 ± 1.5

Table 4 (continued)

ModelOrder of input viewsImage selection accuracy: Top-6 (%)Image selection accuracy: Top-24 (%)Image selection accuracy: Top-24 minus Top-6 (pp)2D bbox [email protected]: Top-6 (%)2D bbox [email protected]: Top-24 (%)2D bbox [email protected]: Top-24 minus Top-6 (pp)3D bbox [email protected]: Top-6 (%)3D bbox [email protected]: Top-24 (%)3D bbox [email protected]: Top-24 minus Top-6 (pp)
GPT-5.6Chronological order86.9 ± 0.886.2 ± 0.9−0.7 ± 1.170.6 ± 1.169.8 ± 1.7−0.9 ± 2.764.5 ± 0.963.7 ± 1.4−0.8 ± 2.1
Relevance order86.085.5−0.468.569.4+0.962.160.9−1.3
Gemini 3.8 FlashChronological order86.5 ± 0.688.6 ± 0.4+2.1 ± 0.971.2 ± 0.572.1 ± 1.3+0.9 ± 1.363.7 ± 0.663.3 ± 0.6−0.4 ± 0.9
Relevance order88.189.4+1.373.273.6+0.464.764.3−0.4

All values are on the 235 queries whose targets are visible at both budgets. Each cell is a percentage of these queries. Image selection accuracy: the image selected by the model contains the target. [email protected]: the predicted bbox has IoU ≥0.50 with the target, for the 2D bbox in the selected image and for the final 3D bbox. The change is Top-24 minus Top-6 in percentage points (pp), computed within each run. Values are means ± sample SD over three runs for Qwen; for GPT/Gemini, over five runs in chronological order and one run in relevance order. Chronological order: views in the order they were recorded. Relevance order: views sorted by similarity to the query, most similar first. View selector: CLIP.

Reviewer_FQAqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OTG as a controlled evaluation pathway for studying view preselection in VLM-based zero-shot 3D grounding. Models select a frame and produce a category and 2D localization; a shared segmentation, RGB-D lifting, and proposal-assignment pipeline produces the 3D prediction. On a fixed ScanRefer panel of 250 expressions from 115 scenes, the study varies nested PE-Core-ranked view budgets across dense Qwen checkpoints and hosted VLMs. Dense Qwen models decline beyond compact tested inputs, whereas GPT and Gemini improve through Top-24, with Gemini declining at Full. Paired analyses distinguish recovery of previously unseen targets from changes on already-covered targets.

Critical strengths
  1. The controlled within-model budget sweeps provide an informative empirical result. Inputs are nested, presentation is chronological, and the downstream pathway is fixed. The cross-model difference in the Top-3-to-Top-24 budget effect is supported by paired scene-clustered intervals: GPT minus Qwen-2B is +20.0 points with a 95% interval of [+12.4, +27.6] (Appendix G.1). This shows that view selection is an accuracy-relevant system decision, rather than merely an efficiency setting.

  2. The intermediate and paired diagnostics are particularly valuable. On the same 235 queries with a target already visible in Top-6, all four dense Qwen models lose both direct 2D and final 3D accuracy at Top-24, with scene-clustered intervals below zero (Table 3 and Appendix G.2). The paper also shows that most of GPT's and Gemini's net improvement from Top-6 to Top-24 comes from target recovery, not a net improvement on already-covered queries. This avoids conflating visibility, localization, and downstream 3D success.

Critical weaknesses
  1. Each model–budget condition uses only one generation per query. The scene-clustered bootstrap appropriately measures uncertainty across scenes but cannot establish stability across repeated model generations, as the paper acknowledges. This matters most for fine-grained recommendations: GPT's Full result exceeds Top-24 by only one query, while Gemini's Top-24-to-Full drop is six queries. The broad interaction has substantially stronger support than the exact ranking of nearby budgets, and repeated-generation evidence would help separate these conclusions.

  2. The empirical scope remains limited to one 250-expression panel and one selector for the full sweep. The SigLIP2 check uses only GPT at Top-6, so it tests neither a budget response nor whether the opposite model responses survive a different selector. The paper explicitly limits its claims, which mitigates this concern, but the evidence supports a useful result for these interfaces rather than a general rule for choosing budgets across VLM systems. This limits the strength of my acceptance recommendation.

Questions for authors
  1. Decision-relevant additional evidence: Do you have repeated-generation results for the key Top-6/Top-24/Full comparisons, or an estimate of within-condition variability? Please distinguish scene-resampling uncertainty from generation variability. If reruns are unavailable, the one-query GPT difference and six-query Gemini difference should remain descriptive rather than evidence for a stable optimal budget.

  2. Clarification: For the Qwen Top-6 and Top-24 conditions, what were the rates of invalid or incomplete outputs, exhausted output budgets, and any input/context truncation? Appendix B fixes nominal per-image settings, but reporting these execution outcomes would help determine how much of the direct 2D decline reflects localization versus interface or output failures.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FQAq

We thank you for recognizing the value of the controlled comparisons and paired analyses, and for separating the broad model-dependent interaction from the exact ranking of nearby budgets. The repeated runs below support this separation.

W1 / Q1. Stability across repeated generations

You ask whether the reported budget effects are stable across repeated model generations, and how generation variability differs from scene-resampling uncertainty. To establish this, we now repeat every model–budget condition, with three runs for each of the Qwen 2B/4B/8B/32B models and five runs for GPT and Gemini, including Full, whereas previously each condition was evaluated only once (Table 1). The work demonstrates that the broad interaction is retained: from Top-6 to Top-24, mean final 3D accuracy decreases for the smaller models Qwen 2B/4B/8B/32B (−10.3/−9.5/−6.3/−2.8 percentage points, pp) and does not decrease for the more powerful VLM models, GPT and Gemini (+1.0/+1.9 pp). Qwen 2B, 4B, and 8B decrease in every run, and Qwen 32B decreases on average. The Top-3-to-Top-24 contrast that you cite also holds in the repeated means, with +4.7 pp for GPT and −13.9 pp for Qwen 2B. Changes are computed within each run and then averaged, so they can differ by 0.1 pp from differences between the rounded means in Table 1.

Table 1. Model × budget results with the CLIP view selector: 3D bbox [email protected] (%) on ScanRefer.

ModelTop-3Top-6Top-12Top-24Full
Qwen3-VL-2B45.5 ± 3.441.9 ± 2.035.7 ± 0.831.6 ± 1.429.6†
Qwen3-VL-4B49.2 ± 1.149.3 ± 1.442.5 ± 0.839.9 ± 0.9—
Qwen3-VL-8B49.3 ± 1.450.3 ± 1.446.9 ± 1.044.0 ± 0.0—
Qwen3-VL-32B54.0 ± 0.855.2 ± 2.452.0 ± 0.852.4 ± 2.4—
GPT-5.656.9 ± 0.560.6 ± 0.962.2 ± 1.061.6 ± 1.460.9 ± 1.6
Gemini 3.8 Flash56.0 ± 0.359.9 ± 0.561.7 ± 1.061.8 ± 0.560.6 ± 0.5

3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. Values are means ± sample SD over three runs for Qwen at Top-3/6/12/24 and over five runs for GPT/Gemini at every budget, including Full. Full: all sampled views of the scene. † The Qwen-2B Full value is the single run reported in the submission. A dash marks a condition that was not run. View selector: CLIP. Views in chronological order.

The repeated runs also clarify the two differences between nearby view budgets that you highlighted. For GPT, Full outperforms Top-24 in two of the five runs and underperforms in the remaining three, indicating that the difference between these budgets is within the observed run-to-run variation, with the highest mean accuracy occurring at Top-12. For Gemini, the decrease in accuracy from Top-24 to Full is consistently observed across all five runs, with a mean reduction of 1.3 percentage points (pp). The standard deviations (SDs) reported in Table 1 quantify variability across repeated runs, whereas the scene-clustered intervals in the original submission quantify uncertainty associated with resampling scenes within a single run. These findings reinforce your observation that the broader model-dependent trends are more robust than the precise ranking of nearby view budgets, and further support the central conclusion of the submission that the influence of view budget on grounding accuracy depends on the VLM being evaluated.

W2. Selector and evaluation scope

To address your question of whether the contrasting model responses to increasing view budgets persist with a different view selector, we have extended our experiments to evaluate SigLIP2 across all six VLMs at Top-3/6/12/24, using the same 250-expression ScanRefer panel and stride-30 candidate pool. In the original submission, SigLIP2 was evaluated only for GPT at Top-6.

The results confirm that the model-dependent trends persist across both view selection methods (Table 2). With SigLIP2, increasing the view budget from Top-6 to Top-24 reduces the final 3D accuracy by 14.0, 8.8, 8.4, and 4.4 percentage points (pp) for Qwen 2B, 4B, 8B, and 32B, respectively, while improving accuracy by 1.2 pp for both GPT and Gemini. These trends are consistent with those observed across repeated runs using CLIP (the PE-Core ranking employed in the original submission): mean accuracy decreases for all four Qwen models but increases for GPT and Gemini.

These findings demonstrate that the contrasting responses to increasing view budgets are not specific to the view selector, but persist across both CLIP and SigLIP2. They further support the central conclusion of the submission that the choice of view budget should be made in conjunction with the grounding VLM, rather than independently of it.

Table 2. Selector × model × budget results with the SigLIP2 view selector: 3D bbox [email protected] (%).

ModelTop-3Top-6Top-12Top-24Top-24 minus Top-6 (pp)
Qwen3-VL-2B42.842.037.228.0−14.0
Qwen3-VL-4B48.450.044.441.2−8.8
Qwen3-VL-8B49.649.248.040.8−8.4
Qwen3-VL-32B54.054.852.450.4−4.4
GPT-5.656.058.860.060.0+1.2
Gemini 3.8 Flash57.260.862.062.0+1.2

3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. The last column is Top-24 minus Top-6 in percentage points (pp). One run per condition. View selector: SigLIP2. Views in chronological order.

Q2. Invalid outputs, output exhaustion, and input truncation

To address your question of how much of the observed decline in direct 2D accuracy is attributable to localisation errors rather than interface or output failures, we have extended our analysis to examine every Qwen run at Top-6 and Top-24 in the setting of the submitted version (CLIP selector, chronological order). Specifically, we investigate the frequency of invalid outputs (Table 6), the circumstances under which responses terminate, and whether any inputs exceed the model's context limit.

The results demonstrate that interface and output failures are too infrequent to explain the observed performance degradation. Invalid JSON or invalid bounding boxes occur in only 0–1.2% of queries per run, whereas direct 2D accuracy decreases by 6.7–16.8 percentage points (pp) across the Qwen models when increasing the view budget from Top-6 to Top-24, evaluated over all 250 queries.

Of the twelve invalid outputs, ten contain incomplete JSON responses, primarily caused by repetitive generation loops consistent with reaching the output limit. The remaining two contain valid JSON with a zero-area bounding box. All other responses terminate normally. Furthermore, even the longest input, including the full output token allowance, remains within the smallest model context limit, confirming that no inputs were truncated.

These findings indicate that the decline in direct 2D accuracy is primarily attributable to localisation errors rather than interface or output failures. The analysis presented in item 3 of the general response further distinguishes errors in image selection from those arising in bounding-box localisation.

Table 6. Number of invalid outputs in each run of the Qwen models.

ModelTop-6 (run 1 / run 2 / run 3)Top-24 (run 1 / run 2 / run 3)
Qwen3-VL-2B0/0/12/0/0
Qwen3-VL-4B0/0/00/0/0
Qwen3-VL-8B0/0/01/0/0
Qwen3-VL-32B1/0/02/3/2

Each cell lists three counts, one for each of the three repeated runs of that model (the runs averaged in Table 1). A count is the number of queries, out of 250, whose output is invalid JSON or has no valid bbox; these queries are scored as incorrect. Of the twelve such outputs, ten contain no complete JSON object and two have a zero-area bbox.

Comments on the paper

10 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and for recognizing the controlled design of the study. We have added clarifying experimental work and supporting evidence to the paper to address several issues raised. The submission examines a choice that zero-shot 3D grounding systems usually fix in advance, namely how many preselected views are supplied to the grounding VLM, and concludes that this choice affects accuracy in a model-dependent manner: from Top-6 to Top-24, mean accuracy decreases for the smaller Qwen models (2B/4B/8B/32B), whereas it does not for the more powerful VLMs, Gemini and GPT. The reviews raise three recurring concerns about this conclusion: (1) whether the reported budget effects are stable across repeated generations, since each condition used a single generation (Reviewers FQAq and Qgk8); (2) whether the contrasting model responses persist with a different view selector, since the full sweep used only one selector (Reviewers FQAq, Qgk8, and mCWU); and (3) why additional views harm some models, since a larger budget simultaneously changes target coverage, distractor content, sequence length, and the positions of existing views (Reviewers Qgk8 and mCWU).

To address these concerns, we have extended the presented experiments and analysis in four ways. Items 1 and 2 below address concerns (1) and (2) through repeated runs and a second view selector for all six models. Items 3 and 4 address concern (3): item 3 separates the effect of target coverage, while item 4 examines the effect of view ordering while keeping the selected images, and therefore the distractor content and sequence length, fixed. The additional work demonstrates that the model-dependent conclusion holds across repeated runs, two view selectors, and two presentation orders. It further shows that both model groups benefit when additional views reveal previously unseen targets, but that the smaller models lose accuracy on targets that are already visible. For these smaller models, bounding-box localisation is most accurate when the selected image occupies one of the first three positions in the input sequence. These findings support the central conclusion of the submission that view preselection should be evaluated together with the grounding VLM, rather than treated as a model-independent computational shortcut. The remaining points raised by each reviewer are addressed in the individual responses.

AuthorsOpenReview ↗

1. Repeated model × budget evaluation

To establish whether the reported trends are consistent across repeated runs, we have extended the experiments to include three runs for each Qwen model (2B/4B/8B/32B) and five runs for GPT and Gemini, compared with a single run per condition in the original submission. All models are evaluated with view budgets of Top-3/6/12/24, with GPT and Gemini additionally evaluated using all available views (Full).

The results confirm that the model-dependent trends reported in the original submission persist across repeated runs (Table 1). For the smaller Qwen models (2B/4B/8B/32B), mean accuracy decreases as the number of views increases from Top-6 to Top-24. In contrast, the more powerful VLMs, Gemini and GPT, achieve higher mean accuracy as the number of views increases from Top-6 to Top-12, with performance remaining broadly stable at Top-24.

The repeated experiments also provide two refinements to our original observations. First, GPT achieves its highest mean accuracy at Top-12 rather than at Full, with comparable performance at Top-24 and Full. Second, the decreases for Qwen 2B, 4B, and 8B are observed consistently in every run, whereas Qwen 32B decreases on average and in two of its three runs.

Together, these results strengthen the central conclusion that the contrasting responses to increasing view budgets reflect systematic differences between the two groups of VLMs, rather than being artifacts of a single experimental run.

Table 1. Model × budget results with the CLIP view selector: 3D bbox [email protected] (%) on ScanRefer.

ModelTop-3Top-6Top-12Top-24Full
Qwen3-VL-2B45.5 ± 3.441.9 ± 2.035.7 ± 0.831.6 ± 1.429.6†
Qwen3-VL-4B49.2 ± 1.149.3 ± 1.442.5 ± 0.839.9 ± 0.9—
Qwen3-VL-8B49.3 ± 1.450.3 ± 1.446.9 ± 1.044.0 ± 0.0—
Qwen3-VL-32B54.0 ± 0.855.2 ± 2.452.0 ± 0.852.4 ± 2.4—
GPT-5.656.9 ± 0.560.6 ± 0.962.2 ± 1.061.6 ± 1.460.9 ± 1.6
Gemini 3.8 Flash56.0 ± 0.359.9 ± 0.561.7 ± 1.061.8 ± 0.560.6 ± 0.5

3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. Values are means ± sample SD over three runs for Qwen at Top-3/6/12/24 and over five runs for GPT/Gemini at every budget, including Full. Full: all sampled views of the scene. † The Qwen-2B Full value is the single run reported in the submission. A dash marks a condition that was not run. View selector: CLIP. Views in chronological order.

AuthorsOpenReview ↗

2. View selector × model × budget evaluation

To better understand the influence of view selection on model performance, we have extended our experiments to compare two view selection methods, CLIP (the PE-Core ranking used in the original submission) and SigLIP2, across all six VLMs: Qwen 2B/4B/8B/32B, GPT, and Gemini. Whereas the original submission primarily reported results using CLIP, the extended experiments evaluate all six models with both view selectors at Top-3/6/12/24 view budgets.

The results demonstrate that the model-dependent performance trends are consistent across both view selection methods (Tables 1 and 2). For the smaller Qwen models (2B/4B/8B/32B), accuracy decreases as the number of views increases from Top-6 to Top-24, irrespective of the view selector. In contrast, the more powerful VLMs, Gemini and GPT, achieve higher accuracy as the number of views increases from Top-6 to Top-12, with performance remaining broadly stable at Top-24 under both selection methods.

These findings strengthen the conclusion that the contrasting responses to increasing view budgets arise from differences in the VLMs themselves, rather than being specific to the view selection method. In particular, they suggest that the decline is intrinsic to the Qwen models evaluated here rather than to the view selector.

Table 2. Selector × model × budget results with the SigLIP2 view selector: 3D bbox [email protected] (%).

ModelTop-3Top-6Top-12Top-24Top-24 minus Top-6 (pp)
Qwen3-VL-2B42.842.037.228.0−14.0
Qwen3-VL-4B48.450.044.441.2−8.8
Qwen3-VL-8B49.649.248.040.8−8.4
Qwen3-VL-32B54.054.852.450.4−4.4
GPT-5.656.058.860.060.0+1.2
Gemini 3.8 Flash57.260.862.062.0+1.2

3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. The last column is Top-24 minus Top-6 in percentage points (pp). One run per condition. View selector: SigLIP2. Views in chronological order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.