acceptodds
Under review as a conference paper at ICLR 2027

Vision Harnessing Agent for Open Ad-hoc Segmentation

10 views

Abstract

Segmentation has become easy when the concept is known, requiring retrieval of a learned visual grounding from text. It remains hard for open ad-hoc concepts, where the grounding may not exist as one learned mask and must often be constructed from image evidence through parts, relations, exclusions, and collections. We propose a Vision-guided Ad-hoc Segmentation Agent (VASA), the first vision harnessing agent for open ad-hoc segmentation. VASA is training-free and couples a VLM agent, a segmentation foundation model, and a visual harness that maintains a working mask to make visual progress persistent, inspectable, and editable. Rather than revising text prompts alone, it plans visual operations, invokes segmentation tools, inspects results, edits the mask, and recovers from errors. We construct PARS, a new benchmark that turns part-level labels into open ad-hoc concepts through long-form definition queries. We show that VASA is consistently effective across six VLMs with varying capabilities. Using Qwen3-VL 32B Thinking as the VLM, VASA outperforms various baselines on PARS, surpassing SAM3 Agent by 13.5%–25.3%. On RefCOCOm, VASA improves over SAM3 Agent by 4.8%–8.8% and over other agentic baselines by more. VASA also remains competitive with SAM3 Agent on ReasonSeg for common, named concepts. These results validate VASA's agentic visual construction for open ad-hoc segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic segmentation, open ad-hoc concepts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DpnjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies open ad-hoc segmentation, where a target region is defined through combinations of parts, relations, and exclusions rather than a familiar category name. It proposes VASA, a training-free framework combining a vision-language model with SAM3 and a visual execution harness. The framework maintains a persistent working mask that the agent can inspect and modify through addition, removal, and replacement operations. Planning, execution constraints, visual scrutiny, and error recovery support this iterative construction process. The authors also introduce PARS, a benchmark derived from PartImageNet annotations with detailed textual definitions of target regions. Experiments across six vision-language models, together with evaluations on RefCOCOm and ReasonSeg, report substantial improvements on fine-grained segmentation and competitive performance on conventional reasoning segmentation.

Critical strengths
  1. The persistent working mask provides a practical mechanism for constructing targets through multiple segmentation operations while preserving intermediate progress. This directly addresses a limitation of repeatedly searching for a single prompt that retrieves the entire target.
  2. Improvements across six vision-language models and on RefCOCOm support the usefulness of the framework beyond one backbone or the proposed benchmark. The RefCOCOm results are particularly consequential because they use the dataset’s original expressions.
Critical weaknesses
  1. For Qwen3-VL 32B Thinking, Table 1 reports SAM3 Agent’s overall scores as 46.4/48.1/48.0, which Table 4 identifies as short-query results, whereas VASA’s corresponding scores match its long-query results. The paper should clarify whether the main comparison uses identical queries or method-specific query settings, as this affects the interpretation and attribution of the reported gains.
  2. PARS focuses primarily on body-region definitions derived from existing PartImageNet annotations, limiting the evidence for generalization to broader open-ended compositions and relations. Additionally, the annotated examples used to generate its descriptions overlap the evaluation set, leaving the effect of this overlap on the reported performance unresolved.
Questions for authors
  1. Which query settings were used for each method in Table 1, and why do the Qwen3-VL 32B Thinking results for SAM3 Agent match the short-query row of Table 4? Does VASA retain its advantage when both methods receive identical queries?
  2. How many evaluation images were used as annotated demonstrations for description generation? Can you report the existing results separately for demonstration images and the remaining evaluation images to assess whether performance differs materially between these groups?
  3. Can you provide an ablation that removes persistent mask state from the complete framework while retaining the other harness components where feasible and using a comparable computational budget?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Dear Reviewer Dpnj,

We appreciate your recognition of 1) the practical value of VASA’s harness design built around a persistent working mask, and 2) VASA's effectiveness across six VLMs and multiple benchmarks. We provide a detailed response to your concerns below. We will incorporate these clarifications into the revised manuscript.


R1. Clarification on the query form

[W1] The paper should clarify whether the main comparison uses identical queries or method-specific query settings, as this affects the interpretation and attribution of the reported gains.

[Q1] Which query settings were used for each method in Table 1, and why do the Qwen3-VL 32B Thinking results for SAM3 Agent match the short-query row of Table 4? Does VASA retain its advantage when both methods receive identical queries?

We report the query form used for each method in the table below. We use short queries for methods designed to accept short noun phrases, such as OVSeg, Semantic-SAM, and SAM3. The only exception is SAM3 Agent, where we also use short queries to compare against its stronger performance, as long queries considerably reduce its accuracy.

MethodQuery form
OVSegshort
Semantic-SAMshort
SAM3short
SAM3-Ilong
LISA/LISA++long
VisionReasonerlong
SegZerolong
RESAnythinglong
CoReSlong
SegAgentlong
GuidelineSeglong
Evol-SAM3long
SAM3 Agentshort
VASA (ours)long

Table 1 compares the two methods under their respective best-performing query settings, while Table 4 compares them under identical query conditions:

Query formSAM3 Agent gIoUVASA gIoUGain
Short46.451.0+4.6
Long24.356.9+32.6

VASA outperforms SAM3 Agent under both matched query conditions. The larger long-query gap largely reflects SAM3 Agent’s performance drop. Table 1 therefore uses SAM3 Agent’s stronger short-query result, yielding a 10.5-point gain. Although this comparison uses different query forms, Table 4 confirms that VASA’s advantage persists with identical inputs.

VASA handles customized target queries that SAM3 Agent struggles with. Such queries reflect how users specify targets in real-world open ad-hoc segmentation. Our long-query experiments capture this setting. SAM3 Agent scores 22.1 gIoU points lower with detailed descriptions than with short category names, while VASA scores 5.9 points higher.


R2. PARS scope and description-generation overlap

[W2] PARS focuses primarily on body-region definitions derived from existing PartImageNet annotations, limiting the evidence for generalization to broader open-ended compositions and relations.

Current evaluations do include cross-object relations and various compositional targets. Cross-object relationships are already represented in our RefCOCOm, such as "torso of the guy on the right" (Figure 8), and VASA outperforms SAM3 Agent on this dataset. PARS includes different body parts for different object classes, which are unconventional and unnamable, such as "segment all cylindrical engine parts, including the propellers if present, located under the wings, excluding the wings…" (Figure 4), and "Annotate the entire animal from the neck down, excluding all limbs and the head, which is the part with the face, eyes, nose, mouth, and the hair on the head" (Figure 9). We therefore construct PARS in an efficient way that ensures broad coverage without needing labor-intensive manual mask annotations.

Our work introduces a practical, understudied task and an effective solution, with PARS providing an initial benchmark comparable in size to ReasonSeg. Together with evaluations on RefCOCOm and ReasonSeg, PARS demonstrates VASA’s effectiveness and establishes the foundation for further research. Further expanding its scale and coverage is valuable but requires substantial annotation and validation effort. We leave this important direction to future work.

AuthorsOpenReview ↗

Rebuttal (2/2)

[Q2] How many evaluation images were used as annotated demonstrations for description generation? Can you report the existing results separately for demonstration images and the remaining evaluation images to assess whether performance differs materially between these groups?

We first note that the annotated demonstrations are used to generate class-level descriptions, not as inputs to the segmentation agent during evaluation. Nevertheless, we recognize the concern that these descriptions may fit the demonstration images better than the remaining images. For each open ad-hoc concept in PARS, we use 5 unique images out of 100–150 images as annotated demonstrations. As requested, we report the Table 1 results separately for these two groups below, with Qwen3-VL 32B Thinking as the VLM. The results show little performance difference between demonstration images and the remaining images, with VASA’s gains persisting in both groups.

Evaluation groupSAM3 Agent gIoUVASA gIoUGain
Demonstration images47.057.7+10.7
Remaining images46.356.9+10.6

We also validate PARS’s text queries by randomly sampling 30 examples covering 30 different classes and asking a third-party annotator to draw a segmentation mask from each image and its long-form description. Comparing these masks with the ground truth assesses how well the descriptions convey the intended open ad-hoc concepts. The masks achieved 75.6 gIoU, 76.6 cIoU, and 12.9 xIoU, supporting that PARS’s text queries communicate the intended target regions.

VASA’s gains also generalize beyond PARS and its query-generation procedure. VASA outperforms SAM3 Agent across all three splits of RefCOCOm, which uses human-written referring expressions, and remains competitive on ReasonSeg. These results provide complementary evidence across datasets and query sources.


R3. Ablation of persistent mask state

[Q3] Can you provide an ablation that removes persistent mask state from the complete framework while retaining the other harness components where feasible and using a comparable computational budget?

We use cumulative ablation because the components are designed around a shared working mask and persistent states. Removing that state would require redesigning the planning, scrutiny, and recovery instructions that depend on it. Retaining those instructions without their underlying state would not preserve their functionality. Table 3 therefore evaluates how these components progressively enable effective visual construction.

VASA’s success comes from coordinating the working-mask mechanism with the other harness components. Expanding an agent’s action space without adequate guidance introduces more ways to fail. Adding a working mask alone to SAM3 Agent reduces gIoU from 46.4 to 40.0, whereas the complete harness achieves 56.9. The working-mask mechanism alone is not enough. With the complete harness, VASA substantially outperforms the baselines, combining the flexibility of working-mask operations with the reliability of guided execution. This cumulative analysis supports their coordinated operation, although it does not isolate the contribution of persistent state through a leave-one-out comparison.

Both methods already receive the same maximum interaction budget of 20 rounds. SAM3 Agent often stops early after unsuccessful prompt refinement, leaving much of its budget unused. In contrast, VASA uses the available budget effectively through working-mask updates, visual scrutiny, and error recovery, enabling continued progress toward an accurate segmentation.

Dear Reviewer Dpnj,

As the discussion period ending soon, we would like to thank you again for your time and energy you have devoted to our paper so far. We will continue monitoring this thread and respond promptly to any additional questions or concerns. We look forward to hearing from you.

Best regards,

The Authors

Thank you for clarifying the query settings in Table 1 and showing that the demonstration overlap in PARS does not materially affect the results. However, the ablation still does not isolate the persistent working mask from the other harness components, which leaves the source of the improvement unclear since adding the working mask alone lowers gIoU from 46.4 to 40.0 in Table 3. Because PARS also remains limited to the PartImageNet "body" category and Table 1 still compares methods under different query forms, I will keep my original rating.

Reviewer_tCAcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VASA is a segmentation agent framework that requires no additional training. It maintains intermediate states in a separate working mask and combines regions from repeated VLM-guided SAM3 calls through add, remove, and replace operations. The authors construct PARS using existing PartImageNet part masks and expanded textual descriptions, and evaluate on PARS, RefCOCOm, and ReasonSeg.

Critical strengths
  1. The separate working mask supports region composition and exclusion across calls, allowing intermediate results to be retained and revised during iterative segmentation. The method reuses existing VLMs and SAM3 without additional training, giving it practical engineering value.

  2. The experimental coverage is reasonably broad, including six VLMs, evaluations on PARS, RefCOCOm, and ReasonSeg, component ablations, and comparisons between short and long queries. Gains on RefCOCOm with its original queries provide further evidence of effectiveness beyond the authors' own benchmark.

Critical weaknesses
  1. The methodological novelty is limited. SAM3 Agent already supports planning and visual feedback, and SegAgent tracks and refines masks, so the claim that prior agents cannot accumulate visual progress is an overgeneralization. VASA’s core novelty lies in maintaining a dedicated working mask with explicit Boolean operations across tool calls. While this design is practically useful, it largely extends the state management and tool interfaces of existing iterative segmentation frameworks, yielding only incremental technical advances. Furthermore, query settings in the main table are underdocumented: Table 1’s overall Qwen SAM3 Agent results align with its short-query entries in Table 4, and VASA’s numbers correspond to its long-query setting. Although comparisons with matched query formats in Table 4 still show VASA’s improvements, the query format and selection criteria for each method should be clearly stated in the main table. See SAM3[1] and SegAgent[2].

  2. The PARS benchmark is inadequate for evaluating open-ended compositional segmentation. PartImageNet mostly contains one dominant foreground object; all annotated parts within an image belong to that single object, and part vocabularies are shared across its 11 supercategories. The fine-grained class count does not equate to diverse scenes or complex part arrangements. PARS inherits these original masks and only enriches textual descriptions of regions, without introducing new compositional annotations. It mainly assesses description-guided segmentation of predefined part categories and lacks rigorous tests for cross-object relationships and arbitrary compositional prompts. The authors may consider supplementary evaluations using compositional or exclusion queries from datasets such as PACO[3] to verify whether the observed improvements hold under identical models and compute budgets.

[1] SAM3: https://arxiv.org/html/2511.16719v1

[2] SegAgent: https://arxiv.org/html/2503.08625

[3] PACO: https://github.com/facebookresearch/paco

Questions for authors
  1. Method clarification: Compared with SAM3 Agent's planning and visual feedback and SegAgent's mask updates, what substantive capability differences arise from VASA's separate working mask and Boolean operations? Which reported results or examples support this distinction?

  2. Configuration clarification: Do the Qwen SAM3 Agent and VASA results in Table 1 use the short and long queries from Table 4, respectively? What rule determines the query format for each method in the main table?

  3. Evidence clarification: PARS retains existing part masks. Do the current evaluations include cross-object relations or compositional targets that cannot be represented by a single existing part category? Could the authors also create annotations for PACO and evaluate the method on this dataset?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Dear Reviewer tCAc,

We appreciate your recognition of 1) the practical value of VASA, and 2) our broad experimental coverage. We provide a detailed response to your concerns below. We will incorporate these clarifications into the revised manuscript.


R1. Methodological contribution and distinction from prior agents

[W1] SAM3 Agent already supports planning and visual feedback, and SegAgent tracks and refines masks, so the claim that prior agents cannot accumulate visual progress is an overgeneralization.

[Q1] What substantive capability differences arise from VASA’s separate working mask and Boolean operations? Which reported results or examples support this distinction?

VASA directly edits the accumulated segmentation solution. As discussed in Related Work, SAM3 Agent revises text prompts, while SegAgent refines geometric prompts through positive and negative clicks. In both cases, the complete target must ultimately be produced by the segmenter from those prompts, which is rarely possible for open ad-hoc concepts. Their updates guide the segmenter’s prediction, without explicitly composing independently obtained regions into a solution. In contrast, VASA maintains a working mask separately from segmentation candidates and combines them through region-level operations. The segmenter only needs to identify the constituent regions, without producing the entire requested composition in one prediction. This allows VASA to preserve accepted regions while progressively constructing and correcting the target mask, making steady visual progress.

VASA retains maximum flexibility for the agent while supporting reliable execution through vision harnessing. There is often a tradeoff between an agent’s action space and execution reliability. The larger the action space, the more ways an agent can fail. SegAgent learns its click-selection behavior through fine-tuning on annotation trajectories, which stabilizes execution but limits its capabilities. SAM3 Agent offers greater flexibility in iterative prompt optimization, yet lacks sufficient guidance and constraints to reliably handle complex queries defining open ad-hoc concepts. In contrast, VASA supports flexible visual construction through a complete vision harness, enabling the agent to perform more complex operations reliably. In addition, unlike SegAgent’s fine-tuning approach, this modular harness accommodates new tools, VLMs, and segmenters without additional training. Results across six VLMs and on RefCOCOm with its original expressions support the framework’s applicability beyond one backbone or benchmark.


R2. Clarification on the query form

[Q2] Do the Qwen SAM3 Agent and VASA results in Table 1 use the short and long queries from Table 4, respectively? What rule determines the query format for each method in the main table?

We report the query form used for each method in the table below. We use short queries for methods designed to accept short noun phrases, such as OVSeg, Semantic-SAM, and SAM3. The only exception is SAM3 Agent, where we also use short queries to compare against its stronger performance, as long queries considerably reduce its accuracy.

MethodQuery form
OVSegshort
Semantic-SAMshort
SAM3short
SAM3-Ilong
LISA/LISA++long
VisionReasonerlong
SegZerolong
RESAnythinglong
CoReSlong
SegAgentlong
GuidelineSeglong
Evol-SAM3long
SAM3 Agentshort
VASA (ours)long

[W1] Table 1’s overall Qwen SAM3 Agent results align with its short-query entries in Table 4, and VASA’s numbers correspond to its long-query setting. Although comparisons with matched query formats in Table 4 still show VASA’s improvements, the query format and selection criteria for each method should be clearly stated in the main table.

Table 1 compares the two methods under their respective best-performing query settings, while Table 4 compares them under identical query conditions:

Query formSAM3 Agent gIoUVASA gIoUGain
Short46.451.0+4.6
Long24.356.9+32.6

VASA outperforms SAM3 Agent under both matched query conditions. The larger long-query gap largely reflects SAM3 Agent’s performance drop. Table 1 therefore uses SAM3 Agent’s stronger short-query result, yielding a 10.5-point gain. Although this comparison uses different query forms, Table 4 confirms that VASA’s advantage persists with identical inputs.

VASA handles customized target queries that SAM3 Agent struggles with. Such queries reflect how users specify targets in real-world open ad-hoc segmentation. Our long-query experiments capture this setting. SAM3 Agent scores 22.1 gIoU points lower with detailed descriptions than with short category names, while VASA scores 5.9 points higher.

AuthorsOpenReview ↗

Rebuttal (2/2)

R3. Benchmark scope and compositional evaluation

[W2] PARS inherits these original masks and only enriches textual descriptions of regions, without introducing new compositional annotations. It mainly assesses description-guided segmentation of predefined part categories and lacks rigorous tests for cross-object relationships and arbitrary compositional prompts.

[Q3] Do the current evaluations include cross-object relations or compositional targets that cannot be represented by a single existing part category? Could the authors also create annotations for PACO and evaluate the method on this dataset?

Current evaluations do include cross-object relations and various compositional targets. Cross-object relationships are already represented in our RefCOCOm, such as "torso of the guy on the right" (Figure 8), and VASA outperforms SAM3 Agent on this dataset. PARS includes different body parts for different object classes, which are unconventional and unnamable, such as "segment all cylindrical engine parts, including the propellers if present, located under the wings, excluding the wings…" (Figure 4), and "Annotate the entire animal from the neck down, excluding all limbs and the head, which is the part with the face, eyes, nose, mouth, and the hair on the head" (Figure 9). We therefore construct PARS in an efficient way that ensures broad coverage without needing labor-intensive manual mask annotations.

Our work introduces a practical, understudied task and an effective solution, with PARS providing an initial benchmark comparable in size to ReasonSeg. Together with evaluations on RefCOCOm and ReasonSeg, PARS demonstrates VASA’s effectiveness and establishes the foundation for further research. Further expanding its scale and coverage is valuable but requires substantial annotation and validation effort. We leave this important direction to future work.

Dear Reviewer tCAc,

As the discussion period ending soon, we would like to thank you again for your time and energy you have devoted to our paper so far. We will continue monitoring this thread and respond promptly to any additional questions or concerns. We look forward to hearing from you.

Best regards,

The Authors

Regarding novelty, I view the contribution primarily as a systems and engineering contribution. The framework integrates existing techniques, with refinements that make the integration effective. To be clear, I do not dismiss pipeline-based work, and I recognize the system’s practical value. However, I leave the final judgment on whether its methodological novelty meets the venue’s standards to the AC. I will maintain my current score.

Reviewer_8dPtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines open ad-hoc segmentation — the query is a long description of an unconventional concept, and the model must construct the region rather than match a class name. VASA is a training-free agent that couples a VLM, SAM3 and a visual harness maintaining a persistent, inspectable, editable working mask, so that only an explicit update_working_mask changes the solution. The authors also build PARS (from PartImageNet body parts; 937 ad-hoc + 1,084 common concepts, 2,021 images) and introduce xIoU for cross-concept confusion. On PARS VASA beats SAM3 Agent across six VLMs (same-backbone ad-hoc gIoU +8.4 to +19.1), with part-only gains of +4.8/+8.8/+6.3 on RefCOCOm and parity on ReasonSeg.

Critical strengths
  • A clean abstraction. Separating the textual reasoning history from the visual construction state, and making the latter first-class and editable, is a practically valuable system view; the split between segment_phrase (refreshes candidates) and update_working_mask (the only entry point that changes the solution) is sharp.
  • A well-controlled cross-VLM comparison. Six VLM families are evaluated under the same harness and SAM3 Agent is re-run under the same VLMs, so "effectiveness depends on harnessing, not VLM strength" is supported: SAM3 Agent stays in a narrow 39.6%–40.8% band while VASA climbs from 48.0 to 59.8. RefCOCOm is positive on all three splits and ReasonSeg is reported as parity rather than spun.
  • Useful problem statement and benchmark surface. PARS's long-form definitional queries push evaluation from "retrieve a learned concept" toward "construct a mask from a definition", and the five visual-operation categories in Appendix B give follow-up work a reusable vocabulary.
Critical weaknesses
  1. Tables 1 and 4 use inconsistent query conditions, so the baseline point of the main claim cannot be fixed. Table 4's "SAM3 Agent short" (46.4/48.1/48.0) is identical to Table 1's SAM3 Agent row, while Table 1's VASA row (56.9/66.5/22.9) is identical to Table 4's "VASA long". Either Table 1 ran the baseline on short queries (a reading that is conservative for the authors, not inflating), or Table 4's labels are wrong; Table 3 inherits the problem by using 46.4 as reference. This must be clarified before the main comparison can be read.

  2. The magnitude of the result is driven mainly by the query form. Table 4: under short queries VASA leads 51.0 vs 46.4 (+4.6); under long queries 56.9 vs 24.3 (+32.6). VASA's own absolute improvement is only +5.9 gIoU, so the large separation comes mostly from long queries disabling the phrase-based baseline — a point the paper needs to address rather than attribute to precision of specification alone.

  3. The paper's own ablation contradicts the attribution in its abstract. In Table 3's cumulative ablation, "working mask only" scores 40.0 gIoU — below SAM3 Agent's 46.4 (the paper acknowledges the working mask alone is insufficient), and the final step contributes +8.6 in one go (48.3 → 56.9). Because the ablation is cumulative rather than leave-one-out, that +8.6 bundles error recovery, verification/structured output, stopping conditions and workflow instructions. Yet the abstract and introduction attribute the contribution to "maintaining a working mask". There is also no control with "equally detailed prompts but no persistent state", and the appendix gives no system prompt or tool JSON schema, so the method itself is not reproducible.

  4. PARS's construct validity needs external validation, and no variance is reported anywhere. Appendix D states that long queries were generated by Qwen3-VL 8B from five annotated examples that overlap the evaluation set ("The demonstration images overlap the evaluation set."), that the ad-hoc/common split was decided by the same VLM from ground-truth-overlaid images with no human-verification report, and Figure 5's ablation has no numeric table. Combined with a full-text absence of std / variance / seed / significance, this leaves both the benchmark's difficulty distribution and every small comparison undecidable.

Questions for authors
  1. Please clarify the query conditions: which rows of Table 1 use PARS long queries and which use short ones, and provide a Table 1 version in which both methods use long queries.
  2. Please provide leave-one-out ablations (especially "full harness − working mask" and "full harness − error recovery") plus a prompt-matched control with equally detailed instructions but no persistent state, and explain how "working mask only = 40.0 < 46.4" is consistent with the abstract's attribution.
  3. Please report human verification of PARS (fraction of queries corrected, agreement), and run one batch of queries rewritten by third-party annotators who have not seen the ground-truth masks, to test construct validity.
  4. Please give the exact formula behind "13.5%–25.3%", and report run variance or repeated runs for the main results; can you also add an equal-round-budget comparison against SAM3 Agent and a PARS subset containing relations / collections?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Dear Reviewer 8dPt,

We appreciate your recognition of 1) the value of open ad-hoc segmentation and PARS, 2) the separation between VASA’s reasoning history and visual construction, and 3) the strong cross-VLM results. We provide a detailed response to your concerns below. We will incorporate these clarifications into the revised manuscript.


R1. Clarification on the query form

[Q1] ... which rows of Table 1 use PARS long queries and which use short ones ...

We report the query form used for each method in the table below. We use short queries for methods designed to accept short noun phrases, such as OVSeg, Semantic-SAM, and SAM3. The only exception is SAM3 Agent, where we also use short queries to compare against its stronger performance, as long queries considerably reduce its accuracy.

MethodQuery form
OVSegshort
Semantic-SAMshort
SAM3short
SAM3-Ilong
LISA/LISA++long
VisionReasonerlong
SegZerolong
RESAnythinglong
CoReSlong
SegAgentlong
GuidelineSeglong
Evol-SAM3long
SAM3 Agentshort
VASA (ours)long

There is often a tradeoff between an agent’s action space and execution reliability. The larger the action space, the more ways an agent can fail. Earlier segmentation agents generally rely on predefined procedures, which stabilize execution but limit their ability. SAM3 Agent offers greater flexibility in iterative prompt optimization, yet lacks sufficient guidance and constraints to reliably handle complex queries defining open ad-hoc concepts. In contrast, VASA supports flexible visual construction through a complete vision harness, enabling the agent to perform more complex operations reliably. This also helps explain why simply adding a working mask to SAM3 Agent reduces performance in Table 3, as noted in [W3].

[W1] Tables 1 and 4 use inconsistent query conditions, so the baseline point of the main claim cannot be fixed.

[W2] The magnitude of the result is driven mainly by the query form.

Table 1 compares the two methods under their respective best-performing query settings, while Table 4 compares them under identical query conditions:

Query formSAM3 Agent gIoUVASA gIoUGain
Short46.451.0+4.6
Long24.356.9+32.6

VASA outperforms SAM3 Agent under both matched query conditions. The larger long-query gap largely reflects SAM3 Agent’s performance drop. Table 1 therefore uses SAM3 Agent’s stronger short-query result, yielding a 10.5-point gain. Although this comparison uses different query forms, Table 4 confirms that VASA’s advantage persists with identical inputs.

VASA handles customized target queries that SAM3 Agent struggles with. Such queries reflect how users specify targets in real-world open ad-hoc segmentation. Our long-query experiments capture this setting. SAM3 Agent scores 22.1 gIoU points lower with detailed descriptions than with short category names, while VASA scores 5.9 points higher.


R2. PARS construction and validation

[W4] PARS's construct validity needs external validation, and no variance is reported anywhere. [Q3] Please report human verification of PARS (fraction of queries corrected, agreement), and run one batch of queries rewritten by third-party annotators who have not seen the ground-truth masks, to test construct validity.

Additional human validation supports the quality of PARS’s text queries. PARS already features human verification and refinement after the automated query generation. Our human annotator manually examined all 158 class-level descriptions and rewrote 19 (12.0%). For independent validation, we randomly sampled 30 examples covering 30 different classes. A third-party annotator received each image and its long-form description and produced a segmentation mask. Compared with the ground truth, these masks achieved 75.6 gIoU, 76.6 cIoU, and 12.9 xIoU, indicating that PARS's text queries indeed communicate the intended target regions.

VASA’s gains generalize beyond PARS and its query-generation procedure. VASA not only outperforms SAM3 Agent on PARS, but also across all three splits of RefCOCOm, which uses human-written referring expressions, and remains competitive on ReasonSeg. These results provide complementary evidence across datasets and query sources.


AuthorsOpenReview ↗

Rebuttal (2/2)

R3. Additional leave-one-out ablation

[W3] The paper's own ablation contradicts the attribution in its abstract. In Table 3's cumulative ablation, "working mask only" scores 40.0 gIoU — below SAM3 Agent's 46.4 ...

VASA’s success comes from coordinating the working-mask mechanism with the other harness components. As discussed in R1, expanding an agent’s action space without adequate guidance introduces more ways to fail. The performance drop with the working mask alone directly illustrates this problem. The working mask mechanism alone is not enough. With the complete harness, VASA substantially outperforms the baselines, combining the flexibility of working-mask operations with the reliability of guided execution.

[Q2] Please provide leave-one-out ablations (especially "full harness − working mask" and "full harness − error recovery") plus a prompt-matched control with equally detailed instructions but no persistent state, and explain how "working mask only = 40.0 < 46.4" is consistent with the abstract's attribution.

We use cumulative ablation because the components are designed around a shared working mask and persistent states. Removing that state would require redesigning the planning, scrutiny, and recovery instructions that depend on it. Matching prompt length would not preserve their functionality. Table 3 therefore evaluates how these components progressively enable effective visual construction. The final gain reflects their coordinated operation, and we do not attribute it exclusively to error recovery.


R4. Additional analysis

[Q4] Please give the exact formula behind "13.5%–25.3%", and report run variance or repeated runs for the main results

The reported “13.5%–25.3%” refers to absolute percentage-point improvements over SAM3 Agent on PARS’s ad-hoc subset using Qwen3-VL-32B-Thinking, as described in the abstract: 54.0 − 40.5 = 13.5 points in gIoU, 56.9 − 39.1 = 17.8 points in cIoU, and a 58.8 − 33.5 = 25.3-point reduction in xIoU. In a repeat evaluation on PARS with the same locally hosted VLM, the mean per-sample IoU difference between runs is 3.32. The harness remains fixed across runs, and the variability can arise from stochastic VLM decoding and nondeterministic GPU operations.

[Q4] can you also add an equal-round-budget comparison against SAM3 Agent ... ?

An agent that uses its available budget to reach an accurate answer is better than one that gives up after only a few unsuccessful attempts. Open ad-hoc segmentation naturally requires multiple rounds of reasoning and scrutiny. In our experiments, we already give both methods the same maximum interaction budget of 20 rounds. SAM3 Agent often stops early after unsuccessful prompt refinement, leaving much of its budget unused. In contrast, VASA uses the available budget effectively through working-mask updates, visual scrutiny, and error recovery, enabling continued progress toward an accurate segmentation.

[Q4] can you also add ... a PARS subset containing relations / collections?

Relations and collections are already represented in our evaluations. RefCOCOm includes relational queries such as "torso of the guy on the right" (Figure 8), and VASA outperforms SAM3 Agent on this dataset. PARS includes collection queries such as "segment all cylindrical engine parts, including the propellers if present, located under the wings, excluding the wings…" (Figure 4). Our work introduces a practical, understudied task and an effective solution, with PARS providing an initial benchmark comparable in size to ReasonSeg. Together with evaluations on RefCOCOm and ReasonSeg, PARS demonstrates VASA’s effectiveness and establishes the foundation for further research. Further expanding its scale and coverage is valuable but requires substantial annotation and validation effort. We leave this important direction to future work.


R5. Clarification of the reviewer's comments

[Q1] ...Please also explain the very different gain magnitudes on MindCube-Tiny (+47.7) and MMSI-Bench (+7.8).

MindCube-Tiny and MMSI-Bench are not among the benchmarks evaluated in our paper. Could the reviewer clarify which results this question refers to?

Flag For Ethics Review: Yes

The reviewer flagged our manuscript for ethics review without detailed comments. Could the reviewer clarify whether this was accidental, or how we can help to address this?

First, I sincerely apologize for mistakenly mentioning MindCube-Tiny and MMSI-Bench in my review. These two benchmarks were actually discussed in another paper I was reviewing, and I inadvertently included them in my comments on this submission. I sincerely apologize to the authors for any confusion or inconvenience this may have caused. I have also updated my ethics review accordingly.

I appreciate the authors' responses to my previous concerns. However, one question remains regarding the necessity and effectiveness of the proposed VASA framework.

Based on the examples presented in the paper, VASA appears to perform a series of set operations (e.g., intersection, union, and complement) on the segmentation masks generated by SAM3. This raises the question of whether a simpler alternative could achieve comparable performance at a lower computational cost.

Specifically, could a VLM directly predict the required set operations among semantic categories, followed by segmenting the corresponding categories and applying these operations to their masks? Such an approach might be more computationally efficient while achieving performance comparable to VASA.

I believe evaluating this alternative as a baseline would help clarify whether VASA provides benefits beyond explicit mask-level set operations, thereby offering stronger evidence for the methodological novelty and necessity of the proposed framework. Moreover, this baseline could provide a useful intermediate point in the trade-off between computational cost and performance, bridging the gap between SAM3 and the full VASA harness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.