acceptodds
Under review as a conference paper at ICLR 2027

SplatScout: Query-Driven Part-Level Referring Segmentation in Gaussian Scenes

Abstract

Existing referring segmentation methods on Gaussian scenes primarily target whole objects, relying on segmenter-generated candidates from which a vision- language model (VLM) selects. Such pre-determined fixed candidates may omit fine-grained parts needed as targets or cues for identifying objects. We intro- duce SplatScout, a query-driven, closed-loop approach that places perception un- der VLM control. The model actively selects views, directs segmentation, and inspects rendered 3D results to refine its predictions. For part-level evaluation, we introduce PartRef-ScanNet++, comprising 261 manually verified queries across 12 ScanNet++ scenes with mesh-derived ground truth. On PartRef-ScanNet++, SplatScout achieves 0.538 mIoU, substantially outperforming existing methods. Ablations demonstrate the benefits of iterative refinement, while comparisons across VLM backbones highlight the importance of effective tool-feedback uti- lization. SplatScout also achieves higher scores on four existing referring bench- marks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D visual grounding, referring segmentation, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wUopOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a query-based closed-loop method called SplatScout, which is used for "part-level referential segmentation" in Gaussian scenarios. Existing referential segmentation methods usually rely on predefined or one-time segmentation candidates, making it difficult to capture fine-grained object parts and lacking the flexibility to adjust the segmentation precision according to specific query requirements. To address this limitation, the authors propose a perception framework driven by a visual-language model (VLM), which consists of three core components: a "navigator" for selecting informative viewpoints, a "segmenter" for generating masks for specific queries, and an "updater" for constructing 3D Gaussian instances. The VLM iteratively optimizes its predictions by evaluating the intermediate segmentation and rendering feedback. Additionally, the authors have proposed the PartRef-ScanNet++ benchmark dataset, which contains 12 ScanNet++ scenes, covering 261 manually verified queries, and provides ground truth annotations from the mesh.

Critical strengths
  1. This paper points out a limitation of the candidate-based anaphora resolution method: when the target component is not included in the candidate set, the visual language model (VLM) cannot retrieve it solely through candidate selection. Allowing the VLM to actively control the perception process is a reasonable and well-grounded solution to this limitation. The focus on "component granularity dependent on queries" and "different roles of components in referential expressions" in the paper is particularly notable.
  2. The PartRef-ScanNet++ dataset covers various types of anaphora, including those for complete objects, those for component targets, discriminative descriptions based on components, relationship anchors, and referential expressions for multiple targets. This dataset uses manually verified query statements and ground truth generated based on meshes to provide an effective evaluation environment for fine-grained 3D anaphora segmentation tasks.
Critical weaknesses

1.The paper claims that SplatScout is superior to methods based on fixed candidate objects, but this comparison may conflate two distinct concepts: query-adaptive candidate generation and closed-loop perception. Although SplatScout can dynamically generate segmentation candidates based on the query, its Navigator retrieves views only from a pre-existing, fixed image pool. Therefore, the ability to recover previously missed regions or objects may primarily come from the query-conditioned SAM3 prompting, rather than from the proposed VLM-controlled iterative perception process. This raises an important question: if all available images were exhaustively traversed and the corresponding object points were simply identified/localized, followed by a single SAM3 invocation, would such a much simpler pipeline already achieve comparable performance?

2.The overall pipeline involves multiple interactions with both the VLM and SAM3, yet the paper does not provide sufficient details regarding the computational cost, inference time, number of VLM/SAM3 calls, or overall resource consumption. This makes the comparison with other methods potentially unfair, especially if those baselines operate under substantially different computational budgets. More importantly, how does SplatScout perform when compared with other methods under a comparable computational/resource budget?

3.The Navigator, Segmenter, and Lifter modules are largely built upon existing techniques, including geometric view selection, SAM3, and FlashSplat. The paper would therefore benefit from a clearer discussion of what fundamentally distinguishes SplatScout from existing agentic grounding systems, such as OpenGround. In particular, the authors should clarify which components or capabilities are genuinely novel, and how SplatScout differs conceptually and technically from these prior agent-based grounding frameworks beyond the specific combination of existing modules.

4.As the paper also introduces a benchmark, the benchmark itself should be described in substantially more detail. In particular, the paper currently provides limited information about the benchmark composition, task distribution, and difficulty levels, and lacks sufficient qualitative visualization of the benchmark examples. Providing representative visualizations would make it easier to understand what the benchmark actually evaluates and how challenging the proposed tasks are. In addition, the failure cases deserve a more thorough analysis, particularly for part-level targets.

Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Response to Reviewer wUop

Response to Reviewer wUop

We sincerely thank the reviewer for the detailed review, which covers the method, its cost and the benchmark. The one-pass pipeline proposed in Weakness 1 is now in our ablations (SplatScout, Table 4). We answer each point below, and would be happy to discuss further if any concern remains.

Re Weakness 1 (Query-adaptive candidate generation vs. closed-loop perception)

We thank the reviewer for decoupling two parts of our contribution: segmentation at query time, and the loop that controls it. This helped us understand the position of our paper more clearly. Following the suggestion, we built this pipeline. The VLM gives a phrase from the query, SAM3 segments the frames with it (up to 200 frames per query, since segmenting every frame would take far longer), the masks are grouped and lifted in 3D with our tools, and the VLM selects once among the candidates. We ran it with five VLMs (Table wUop-1).

Table wUop-1: mIoU on PartRef-ScanNet++ (one-pass: phrases from Qwen3.5-397B, selecting VLM varied).

MethodQwen3-VL-30BQwen3.5-35BQwen3.5-397BQwen3.8-FlashGPT-5.6
One-pass0.3960.3930.4120.4260.458
SplatScout0.1280.3530.4500.4920.538

Analysis Query-time segmentation alone is a strong baseline: under every backbone it is better than all the other baselines (SplatScout, Table 3). However, it is exactly the two-stage process described in C1, and it is limited by its first stage. It stays within 0.39–0.46 however strong the VLM is, whereas SplatScout rises from 0.128 to 0.538 and overtakes it with the stronger backbones (significantly with Qwen3.8-Flash and GPT-5.6).

Conclusion Query-time segmentation and the loop therefore complement each other. Segmenting at query time provides candidates that a fixed table lacks, and the loop lets the VLM decide, from what it has observed, what to segment next, where, and with which phrase. It is this control that takes the result beyond what the first stage provides. We also agree that the Navigator only selects among captured frames. This is the common setting of this line of methods, most of which work from the captured images.

Re Weakness 2 (Computational cost and comparison under a comparable budget)

We thank the reviewer for pointing this out. The time, the VLM and SAM3 calls, and the tokens per query of all methods under GPT-5.6 are reported in the Common Response (C2, Table G-2). In short, SplatScout runs under a budget comparable to the baselines: it takes less time per query than OpenGround and ZeroSplat, and makes fewer VLM calls and segments far fewer frames than ZeroSplat.

Re Weakness 3 (What distinguishes SplatScout from agentic systems such as OpenGround)

We thank the reviewer for this question. The key difference is who makes each perception decision (Common Response, C1).

Our contributions are:

  • Control by the VLM. The VLM decides every perception action and can revise each result, instead of only selecting. With the same segmentation and lifting tools under two-stage control (one-pass, Weakness 1), GPT-5.6 reaches 0.458, against 0.538 with our loop.
  • Tools designed for this control in 3D. The VLM can zoom on any box it draws and see the same surface from other views, restrict a cut to a parent, merge cuts from different views by their 3D overlap, and inspect each lifted instance rendered back into the images.
  • A shared scene representation. The tools are not merely combined: every box, mask and instance refers to the same Gaussians, so what one tool returns can be passed directly to another. This is what closes the loop, and lets a stronger VLM go further (0.128 → 0.538 across backbones).

Re Weakness 4 (Benchmark details, visualizations, and failure analysis)

We thank the reviewer for these suggestions. We have added to the paper example queries of each role with their ground-truth targets (Fig. 4), the results for each role (Appendix B.4, Table 9), and a failure analysis for part targets (Appendix G).

Thank you again for your detailed review. As the discussion period is closing soon, we would like to kindly follow up on our response. On your two main questions:

  • One-pass pipeline (W1): we built the pipeline you proposed. It is a strong baseline, but it stays within 0.39–0.46 mIoU across five backbones, while the closed loop overtakes it with the stronger ones (0.538 vs. 0.458 with GPT-5.6; Table wUop-1).
  • Cost (W2): under the same GPU and VLM, SplatScout takes 64 s per query, versus 89 s for OpenGround and 208 s for ZeroSplat (Common Response, Table G-2).

W3 and W4 are addressed in the revision (Sec. 2, Fig. 4, Appendix B.4 and G).

We would be grateful to know whether this addresses your concerns, and we are happy to clarify anything further.

Reviewer_VVCvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SplatScout, a VLM-controlled framework for referring segmentation of objects and parts in Gaussian scenes. Given posed images and a scene reconstructed with 2D Gaussian Splatting, the VLM coordinates three tools: 1. a Navigator that selects complementary views and provides local inspection, 2. a SAM3-based Segmenter that generates masks from text and box prompts, and 3. a Lifter that aggregates the multi-view masks into subset of Gaussian instances. For part-level queries, segmentation can be constrained to a previously identified parent object. Mask overlays and rendered instances are returned to the VLM, allowing it to inspect intermediate results, revise segmentation prompts, and request further observations or lifting. This process essentially returns query-specific targets (subset of gaussians) through iterative interaction. The authors also introduce PartRef-ScanNet++, containing 261 manually verified queries across 12 scenes, and report improvements on this dataset and four of five existing referring benchmarks.

Critical strengths
  • A relevant problem setting and benchmark: Query-dependent segmentation addresses targets missing from fixed object proposals. PartRef-ScanNet++ covers parts as targets, discriminators, and relational anchors, alongside whole-object and multi-target queries. Manual verification, mesh-derived ground truth, and held-out evaluation views provide useful controls.

  • Strong demonstrated grounding results: SplatScout reaches 0.538 mIoU versus 0.253 for the strongest baseline in Table 1. With the same Qwen3.5-397B backbone, Table 3 reports 0.450 versus 0.180 for OpenGround and 0.088 for ZeroSplat. These controls support the pipeline's effectiveness beyond using a stronger VLM alone.

Critical weaknesses
  • The contribution beyond closely related methods needs clearer positioning. Think, Act, Build already uses a VLM agent for grounding and projects reconstructed 3D geometry into additional views to prompt segmentation. OpenGround also selects views, segments new targets, and extends its candidate table online. Section 2's characterization of these methods, and OpenGround's placement among fixed-candidate methods in Table 1, obscure this overlap. SplatScout's rendered-instance inspection and part-level Gaussian segmentation output remain meaningful distinctions, but the effect of this addition seem to be minor (table 4 - No lifting feedback).

  • The contribution of interleaved 3D construction beyond VLM-guided 2D mask refinement is insufficiently isolated. SAM 3 Agent already iterates between language-driven segmentation and visual mask inspection. In Table 4, removing lifting feedback changes mIoU from 0.450 to 0.440 (Part: 0.453 to 0.429), while the oracle experiments demonstrate substantial sensitivity to mask quality with lifting fixed. A useful baseline would let the VLM iteratively refine the masks using only image and mask feedback, then lift the final masks into 3D as a final single step. This would clarify the benefit of the proposed coupling, and if the setup using intermediate 3D instances improves over completing the mask refinement in 2D first.

Questions for authors
  • Could you provide some details about controller prompts, execution budget, and average runtimes?
  • Which VLM was used for the SplatScout results in Table 2?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VVCv

Response to Reviewer VVCv

We sincerely thank the reviewer for the thoughtful review. The baseline suggested in Weakness 2 led to a new ablation that isolates what 3D adds inside the loop (Table VVCv-1), and we believe it makes the paper stronger. We answer each point below, and refer to the Common Response where a point is shared with the other reviewers. We would be happy to discuss further if any concern remains.

Re Weakness 1 (Positioning relative to TAB and OpenGround)

We thank the reviewer for pointing out this overlap, which helps us position our contribution more clearly. We apologize that our unclear writing in Sec. 2 and Table 1 obscured it. We agree with the reviewer's description of both methods, and fully recognize their contributions. Our point is that both still follow a two-stage process: the VLM selects among what a segmenter returns for a category fixed in advance, and cannot revise it. In SplatScout, the VLM decides every perception action from what it has observed (Common Response, C1). We have rewritten Sec. 2 (SplatScout) and regrouped Table 1 (SplatScout) accordingly.

On the small effect of “No lifting feedback” (Table 4) We agree with the reviewer, and we realize that this ablation was too narrow. It only removes the check of the rendered instance after lifting, whereas 3D also takes part earlier in the loop, when masks from different views are grouped and when further views are searched. Following the reviewer's suggestion in Weakness 2, we redesigned the ablation of 3D. The result is given below, and Table 4 (SplatScout) is revised accordingly.

Re Weakness 2 (Isolating interleaved 3D construction from 2D mask refinement)

We thank the reviewer for this suggestion. Letting the VLM refine masks iteratively, as SAM 3 Agent does, addresses exactly the weakness of the two-stage process (Common Response, C1). Our view differs in one respect: we believe that 3D information also helps to correct 2D masks. Inspired by the reviewer, we therefore ablated 3D within our own framework.

The new ablation Every path from masks to 3D and back is removed from the loop: no lifting, no grouping of masks across views in 3D, and no search for further views from a mask or a box. What remains is 2D refinement: the VLM can still select views, zoom in, segment with a phrase and a box, inspect the masks, segment again, and track a mask across frames. It answers with 2D masks, which are lifted once at the end. The backbone (Qwen3.5-397B), prompts, budget, and evaluation are those of Table 4 (SplatScout).

Table VVCv-1: ablation of 3D inside the loop (mIoU on PartRef-ScanNet++, Qwen3.5-397B)

SettingObjPart-targetPart-discr.Part-anchorMultiAll
No 3D in the loop0.3890.3520.5290.3930.4010.395
SplatScout0.4690.4050.5390.5100.3850.450

Using 3D inside the loop improves the overall result (Table VVCv-1). A 3D instance gives the VLM a view-consistent result to check its masks against. Without it, the VLM more often gives up, or returns masks in only one or two frames. We have added this ablation to Table 4 (“No 3D in the loop”), with details in Appendix F.

Re Question 1 (Controller prompts, execution budget, and runtimes)

We have added the controller prompt and tool definitions to the appendix (SplatScout, Appendix H). The budget is 16 steps or 150k input tokens per query, whichever comes first, and the termination rule is given in our response to Reviewer 6b7x (Weakness 2). The average time is 64 s per query with GPT-5.6. Calls, tokens, and the comparison with other methods are in the Common Response (C2, Table G-2).

Re Question 2 (VLM used in Table 2)

GPT-5.6 with medium reasoning, the same configuration as the last row of Table 1. Each of the other methods uses its default model. This is now stated in the captions of Tables 1–3.

Thanks for the clarifications and additional experiments, they do help isolate the individual components better. However, While I recognize the practical value of the work, I remain unconvinced that the scientific insight beyond the integration of existing tools is sufficient to change my overall assessment, therefore after careful consideration, I maintain my original score.

Thank you for considering our response. We are glad the ablation you suggested helped isolate the components.

We would only like to summarize what we see as the paper's finding beyond the system itself. With the tools held fixed, the result depends on who controls them: the same segmentation and lifting tools reach 0.458 mIoU under two-stage control and 0.538 under closed-loop control with GPT-5.6 (Table wUop-1). Across five backbones, the two-stage methods vary by at most about 0.10 mIoU, while the closed loop rises from 0.128 to 0.538 and overtakes them once the VLM is strong enough (Tables G-1 and wUop-1). For part-level grounding, this suggests that the limit lies in the two-stage structure rather than in the tools.

Thank you again for your time and reconsideration.

Reviewer_6b7xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SplatScout, which uses a vision-language model to coordinate view selection, query-conditioned segmentation, and Gaussian instance construction, refining its predictions through intermediate visual feedback. The method combines local inspection, parent-object constraints, and feedback from rendered 3D instances to handle part-level referring expressions at different granularities. The authors also introduce PartRef-ScanNet++, containing 12 scenes and 261 manually verified referring expressions. SplatScout achieves 0.538 mIoU on this benchmark and higher scores than the reported baselines on four of five external benchmarks.

Critical strengths
  1. The method addresses practical difficulties in part-level grounding, with clear overall gains. Constructing instances at query time, together with parent-object constraints, local zooming, and result inspection, helps address missing part candidates and segmentation at the wrong granularity. Figure 2 shows a concrete example of how an incorrect segmentation is revised. With Qwen3.5-397B as the backbone for all three methods, SplatScout achieves 0.450 mIoU, compared with 0.180 for OpenGround and 0.088 for ZeroSplat. Restricting SplatScout to a single round of observation and segmentation lowers its mIoU to 0.209. These comparisons support the value of the interactive design.

  2. The new benchmark is well suited to the problem, and the evaluation includes useful diagnostic experiments. The dataset distinguishes parts used as targets, discriminators, and relational anchors, covering a broader range of part-related referring tasks. Ground truth is derived from laser-scanned meshes, the 216 test images are excluded from perception and preprocessing, and the evaluation accounts for occlusion. Interventions using ground-truth boxes, views, and masks help separate the effects of localization, view selection, and segmentation quality, pointing to specific areas for improvement.

Critical weaknesses
  1. The distinction from existing dynamic perception methods needs to be stated more accurately. Section 2 describes TAB as augmenting an existing candidate set, while Table 1 places OpenGround in the “candidates fixed before the query arrives” group. However, TAB already uses a VLM to dynamically coordinate multi-view segmentation and 3D reconstruction, with adjustments in response to failures. OpenGround also supports online candidate expansion guided by a task chain. The current categorization could mislead readers about the capabilities of these methods. The paper should explain more precisely what it adds in query-dependent granularity control, parent-object constraints, and correction using rendered 3D feedback. It should also clarify whether OpenGround’s online perception module was fully enabled in the comparisons.

  2. The execution settings and runtime of the closed-loop system need more detail. The appendix reports an average of 11 interaction steps and approximately 118k tokens per query, but does not specify the execution budget B or the end-to-end query time. Appendix E also reports that the 30B model exhausted its budget without a valid terminal action on 65 queries, indicating that the budget and termination rules can directly affect the results. Providing these settings and timings would make it easier to reproduce the method and assess its practical cost.

Questions for authors
  1. Were OpenGround’s task-chain planning and online perception modules fully enabled in the comparisons? If so, why does Table 1 place it in the group whose candidates are fixed before the query arrives?

  2. What execution budget B is used, and what conditions must be met for valid termination? What is the average end-to-end query time for the full system?

  3. Table 1 pools results for parts used as targets, discriminators, and relational anchors. How does the method perform on each group? In particular, what are the gains over the main baselines when the part itself is the segmentation target?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6b7x

Response to Reviewer 6b7x

We are grateful to the reviewer for the encouraging assessment of our method and benchmark, and for the precise questions, which helped us make the paper clearer and more complete. We answer each point below and would be happy to discuss further.

Re Weakness 1 & Question 1 (Distinction from TAB and OpenGround)

We thank the reviewer for pointing this out, which helps us state our claim and our position relative to these two methods more clearly. We apologize that our unclear writing caused misunderstanding. We fully recognize that TAB and OpenGround were among the first to rethink and improve on earlier table-based methods. Going one step further, we argue that most existing zero-shot 3D visual grounding methods, including these two, follow a two-stage process (Common Response, C1).

TAB We agree that TAB does not augment an existing candidate set, and that it segments at query time. However, as TAB itself states, its VLM follows a fixed procedure at every step, and this procedure is the two-stage process above. It can accept or reject the first-stage masks, but cannot revise them.

OpenGround OpenGround has a similar limitation. We agree that it extends its candidate table online, but its structure is still two-stage: the VLM only selects from the table that the first stage provides. Its online perception was enabled in every run, in the submission and in the new experiments, with its released code and default configuration. We have corrected the inaccurate statements in the paper, including its grouping in Table 1.

SplatScout Our method breaks this fixed two-stage structure: the VLM is not bound to a fixed procedure and can adapt to hard part queries. For the aspects the reviewer lists, the VLM sets the phrase and box of each cut after seeing the previous result, can restrict a cut to an object it has already built, and can use 3D information to check and correct its 2D masks.

Re Weakness 2 & Question 2 (Execution settings and runtime)

We thank the reviewer for pointing out these missing details, which we have added (Sec. 3.3 and Appendix A.1). The average time is 64 s per query with GPT-5.6. Calls, tokens, and the comparison with other methods are in the Common Response (C2, Table G-2).

On the budget B and valid termination B is 16 steps or 150k input tokens per query, whichever comes first. A query ends when the VLM calls answer with lifted instances, or report (not found or ambiguous).

On the 65 queries of the 30B model (Appendix E) Small models such as Qwen3-VL-30B are weak at tool use, and often exhaust all steps without reaching an answer: 65 queries with Qwen3-VL-30B, at most 10 with the other backbones, and none with GPT-5.6. This is the main reason for its low score.

Re Question 3 (Results for each role and gains when the part itself is the target)

The per-role results were left out of the main text because of page width and readability. They are now in Appendix B.4 (Table 9) for the methods of Table 1, and Table 6b7x-1 below adds the baselines under the same backbones.

Table 6b7x-1: mIoU by role on PartRef-ScanNet++ (evaluable queries: 76 / 91 / 31 / 29 / 29). The ZeroSplat numbers are the corrected ones (Common Response, C3).

Method (backbone)ObjPart-targetPart-discr.Part-anchorMultiAll
GaussDet (Qwen2.5-VL-7B)0.2930.1710.3740.2970.2300.253
OpenGround (GLM-4.5V)0.1950.0980.3260.1550.2290.176
ZeroSplat (Qwen3-VL-30B)0.3210.1890.4110.3190.2500.277
OpenGround (Qwen3.5-397B)0.2360.0970.3750.0950.1650.180
ZeroSplat (Qwen3.5-397B)0.3270.1950.4450.3590.2400.288
SplatScout (Qwen3.5-397B)0.4690.4050.5390.5100.3850.450
OpenGround (GPT-5.6)0.2410.0710.2700.1250.1660.162
ZeroSplat (GPT-5.6)0.3390.2740.4480.3880.2550.325
SplatScout (GPT-5.6)0.5720.4490.6210.6310.5480.538

Under the same backbone, SplatScout is ahead of both baselines in every role. When the part itself is the target, SplatScout reaches 0.405 with Qwen3.5-397B and 0.449 with GPT-5.6, against 0.195 and 0.274 for ZeroSplat, the strongest baseline on this role.

Thank you again for your encouraging review and precise questions. As the discussion period is closing soon, we would like to kindly follow up on our response. In brief:

  • Positioning (W1/Q1): we corrected the description of TAB and OpenGround and regrouped Table 1. OpenGround's online perception was enabled in every run.
  • Budget and runtime (W2/Q2): the budget is 16 steps or 150k input tokens per query, and the average time is 64 s per query (89 s for OpenGround, 208 s for ZeroSplat).
  • Per-role results (Q3): when the part itself is the target, SplatScout reaches 0.449 mIoU with GPT-5.6, against 0.274 for ZeroSplat, the strongest baseline on this role.

We would appreciate knowing whether these changes resolve your concerns, or whether anything remains that we can clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

Common response

We thank the three reviewers for their constructive reviews. A revised paper has been uploaded, with the changes marked in blue. We answer two shared questions (C1–2) and correct one baseline (C3).

C1. Our claim, and its relation to OpenGround and Think, Act, Build (TAB)

We apologize that the submission described TAB and OpenGround inaccurately (Reviewers 6b7x and VVCv) and did not state our claim clearly enough, which may have led to misunderstandings about it. The related parts have been corrected in the revision (Abstract, Sec. 1–2, Fig. 1 and Table 1). We sincerely thank the reviewers for these comments, which helped us position our work more precisely.

We fully recognize that both works move beyond a fixed candidate table, but they are like most existing zero-shot visual grounding methods sharing a two-stage process: a detector or segmenter first extracts candidates of the target category, and a VLM then selects the target among them. Our claim is that their limitation lies not in the fixed table alone, but in this two-stage process. OpenGround and TAB also follow this process:

  • OpenGround can extend its table online, but the extension is itself a first stage: the VLM parses a label from the query, and if the table has no match, a segmenter is run with this label and the result joins the table, from which the VLM then selects. The VLM thus only provides the label, and a detection that returns nothing or bad instance is not retried.
  • TAB replaces the table with a tool-calling agent, but its procedure is still fixed: its skill document tells the agent to "strictly follow this sequential execution path" (TAB, Appendix A.1). The path is again two-stage: a segmenter is run on every frame for the target category, then the VLM filters the frames and selects among numbered candidates. Its autonomy is limited to adjustments such as relaxing a threshold when no frame remains.

In both, the VLM can accept or reject what the first stage provides, but cannot revise it. The two-stage process may work for whole objects, but has three clear limitations for parts.

(i) Candidates are cut once, with a word fixed in advance Part names are long-tail vocabulary, and a single cut often yields no correct candidate, so the first stage limits the final result.

(ii) The VLM is limited to selection A zero-shot method draws all of its capability from the VLM, and selection leaves much of it unused. Across backbones (Table G-1), the two-stage methods stay within a narrow range, while SplatScout rises from 0.128 to 0.538.

Table G-1: mIoU across backbones (Table 3 in the revision).

MethodQwen3-VL-30BQwen3.5-35BQwen3.5-397BQwen3.8-FlashGPT-5.6
ZeroSplat0.2770.2690.2880.2220.325
OpenGround0.1610.1730.1800.1740.162
SplatScout0.1280.3530.4500.4920.538

(iii) A fixed procedure cannot adapt to parts Parts are far more varied than whole objects: one is found only through its parent object, another must be zoomed in on, a third needs a different phrase. This requires adapting to each case, and the VLM does so when left to decide.

To our knowledge, SplatScout is the first zero-shot method for 3D part-level referring segmentation in which the VLM is not bound to a fixed procedure.

C2. Running time and cost

We measured all methods under one setting: one RTX PRO 5000 Blackwell GPU, GPT-5.6 (gpt-5.6-sol, medium reasoning).

Table G-2: average per query with GPT-5.6 (Table 6 in the revision, * estimated).

MethodmIoUVLM callsTokens inTokens outSAM3 framesTime
OpenGround0.1626.923k2.7k0.189 s
ZeroSplat0.32530.753k7.2k247208 s
TAB–>450*>1M*–>300*>30 min*
SplatScout0.5389.374k1.3k2664 s

Input tokens of SplatScout Most of the input hits the prompt cache: of the 74k input tokens per query, only about 12k are billed at full price.

OpenGround Its online extension is enabled but rarely triggered: following the official prompt, GPT-5.6 maps a part to its parent's label, so no segmentation is requested.

TAB TAB could not be run on the full benchmark, so its row is an estimate. It queries the VLM on every frame in which the target category is segmented, and a part such as a handle is visible in almost every frame, so the VLM is called very often.

C3. Correction on ZeroSplat

Our reproduction of ZeroSplat (code unavailable at submission time) read the VLM's box coordinates on the wrong scale. With this fixed, ZeroSplat rises from 0.090 to 0.277 with its own backbone (Qwen3-VL-30B) and from 0.088 to 0.288 with Qwen3.5-397B. All ZeroSplat numbers in our responses are corrected, and Tables 1, 3 and 10 of the revision are updated. No other method is affected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.