acceptodds
Under review as a conference paper at ICLR 2027

Segment Anything In-Context via Visual Demonstrations

Abstract

We present the Segment Anything Model via Visual Demonstrations (SAYD), a foundation model for in-context segmentation. Given visual demos comprising prompt images and their segmentation masks, SAYD segments coherent regions (e.g., objects or their parts) in a query image by establishing visual correspondences between the prompts and the query. SAYD generalizes to previously-unseen segmentation tasks, including those requiring tacit visual knowledge or involving concepts that are difficult to articulate unambiguously in language. We train SAYD on 12 public datasets originally curated for panoptic segmentation using our proposed techniques, including a lightweight mask decoder for prompt-agnostic mask generation and a learning mechanism for recognizing novel query classes absent in the prompts. These innovations enhance both computational efficiency and out-of-domain generalization. Consequently, SAYD significantly outperforms existing methods of in-context segmentation and open-vocabulary segmentation across 24 out-of-domain benchmarks, spanning natural, biological, ecological, cosmological, medical, pathological, and satellite imagery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context segmentation, vision foundation model, scientific fields, tacit visual knowledge

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_x24YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The submission presents SAYD, a framework for in-context segmentation (ICS) that segments query images conditioned on visual demonstrations (image–mask pairs) rather than text prompts. To circumvent the O(K)O(K) computational bottleneck of prior visual ICS methods that re-run forward passes for each category, the authors decouple mask proposal from prompt conditioning. A query-only Transformer decoder predicts class-agnostic binary masks directly from query features, which are then classified in parallel via cosine similarity against class prototypes pooled from the prompt demos.
Key Contributions Prompt-Agnostic Mask Decoder: Generates mask proposals strictly from the query image, reducing multi-class inference cost from O(K)O(K) to O(1)O(1) and avoiding mask boundary degradation caused by heavy prompt cross-attention. Ignota-Aware Calibrated BCE Loss: Formulates mask classification via binary cross-entropy with a learnable temperature τ\tau and bias β\beta, treating query regions absent from the prompt demos ("ignota") as explicit negatives alongside background. Encoder Adaptation for In-Context Matching: Demonstrates that finetuning the pretrained ViT backbone (DINOv3 ViT-L) is necessary for dense visual correspondence, contrasting with prior works that rely on frozen backbones.
Empirical Validation The model is trained on 12 public panoptic datasets (~0.91M images, 11.46M masks) and evaluated across 24 out-of-domain benchmarks spanning semantic, instance, and panoptic segmentation. Evaluated against text-driven models (SAM3) and visual prompting baselines (DINOv, INSID3). SAYD maintains constant runtime across class counts and shows substantial gains on specialized scientific domains (microscopy, astronomy, metallography) where open-vocabulary text models struggle with tacit visual knowledge.

Critical strengths

The evaluation is serious. Testing across 24 genuine out-of-domain scientific benchmarks—cell biology, astronomy, metallurgy—is far more convincing than the customary incremental gains on COCO. It actually substantiates the central thesis: visual demonstrations bypass the vocabulary bottleneck and capture tacit visual knowledge precisely where text-prompted models like SAM3 collapse. On top of that, decoupling the mask decoder from prompt features is just sensible engineering; dropping multi-class inference from O(K)O(K) down to O(1)O(1) makes visual in-context segmentation computationally viable rather than a curiosity.

Critical weaknesses

The empirical validation suffers from selective baseline curation and a fragile evaluation protocol. On baselines, the paper leans heavily on outperforming text-prompted models like SAM3, which is unsurprising since text prompts naturally collapse on specialized scientific jargon. Against native visual in-context segmentation (ICS) methods, however, the comparison is remarkably thin: Painter is an outdated 2023 color-regression heuristic that fails across the board, and INSID3 is restricted to single-class binary masks. The authors actively discuss and ablate against native visual ICS frameworks like SINE (NeurIPS 2024), SegGPT, and SegIC in the text, yet omit them entirely from the 24-benchmark evaluation. Furthermore, on biomedical benchmarks where SAYD claims outsized gains, UniverSeg is conspicuously absent.
Compounding this, the evaluation protocol—randomly drawing ~10 prompt images across just three runs—carries immense variance. In visual in-context learning, exemplar choice heavily dictates performance. Without providing fixed, standardized prompt splits or exact image indices, and given that trained checkpoints are omitted from the submission, these benchmark numbers are virtually impossible for the community to faithfully reproduce or verify.

Questions for authors

How does SAYD compare against peer visual in-context baselines—specifically SINE, SegGPT, UniverSeg (on biomedical tasks), and native SAM 2 using memory-based cross-frame visual prompting—rather than text-prompted models like SAM3?
Given the variance of evaluating on ~10 random visual prompts across only three runs, can you provide standard deviations, the exact prompt splits/IDs used, and an anonymous link with trained checkpoints and evaluation code to verify reproducibility?

5 comments

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our serious and convincing evaluation, and our sensible mask decoder design. Below, we address each of the reviewer’s weaknesses and questions. Should any further clarification be needed, we would be happy to provide additional information.


The reviewer suggests comparing more methods such as SegGPT, SINE, UniverSeg, and SAM 2.

Table 1 in the updated manuscript now includes SINE, SegGPT, UniverSeg, and SAM 2. The comparison shows that our SAYD resoundingly outperforms these methods, corroborating the superior performance of SAYD. We reproduce Table 1 below for reviewing convenience:

ModelGTADRAMCUBLIPYouTNYUiShapeTigDPlantAntLDAJSRTODMGalaCell
SegGPT1.770.420.061.240.240.784.601.090.024.837.619.880.960.781.24
SINE36.4939.230.076.9310.4032.3724.714.730.8925.9731.520.1813.731.522.30
UniverSeg0.030.180.000.620.140.281.450.000.002.221.052.920.110.371.97
SAM20.732.140.814.560.581.121.880.032.388.451.912.973.370.572.04
SAYD (ours)52.9558.0717.0629.6558.5140.8734.9215.5215.6232.0439.9116.8818.7310.1922.43

The reviewer asks us to "provide standard deviations, prompt splits, trained checkpoints and evaluation code."

We are fully committed to reproducibility and address each requested item below:

  • Evaluation code: Our evaluation code is already included in the previously submitted supplementary material (Code-SAYD.zip) via the Jupyter notebook demo_SAYD_eval.ipynb. For convenience, the notebook contains both the Python script and pre-computed intermediate and final outputs, allowing the evaluation pipeline to be inspected directly without executing the code.

  • Model and prompt splits: Because the trained SAYD checkpoint is 1.24GB, exceeding the file size limit for supplementary material, we have hosted it anonymously on HuggingFace https://huggingface.co/Anonymous-27/anonymous/blob/main/ckpt.pth. Upon publication, we will publicly release the model and prompt splits across all benchmarks.

  • Standard deviation: Augmenting Table 1, the table below reports the mean performance and standard deviation across prompt splits for SAYD, SegGPT, and INSID3 on representative benchmarks. To keep the rebuttal concise and readable, we omit the full breakdown across all 20 benchmarked methods and 24 benchmarks here. We will include the complete standard deviation results in the appendix of our next version. As shown below, SAYD exhibits low variance alongside its substantial performance gains. | model | GTA | DRAM | CUB | LIP | YouT | NYU | iShape | | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | SegGPT | 1.77$^{\pm 1.97}$ | 0.42$^{\pm 0.81}$ | 0.06$^{\pm 0.04}$ | 1.24$^{\pm 0.36}$ | 0.24$^{\pm 0.56}$ | 0.78$^{\pm 1.08}$ | 4.60$^{\pm 2.75}$ | | INSID3 | 15.28$^{\pm 3.07}$ | 18.35$^{\pm 2.48}$ | 0.01$^{\pm 0.01}$ | 7.32$^{\pm 1.77}$ | 9.86$^{\pm 1.19}$ | 10.68$^{\pm 3.04}$ | 21.97$^{\pm 1.95}$ | | SAYD | 52.95$^{\pm 2.64}$ | 58.07$^{\pm 1.87}$ | 17.06$^{\pm 1.90}$ | 29.65$^{\pm 1.92}$ | 58.51$^{\pm 0.82}$ | 40.87$^{\pm 0.78}$ | 34.92$^{\pm 2.44}$ |

I appreciate the additional comparisons and variance estimates. Please include the exact sample IDs used to construct the test-time prompts in the code release. Each experiment should use a fixed set of at least three random seeds and report the mean and standard deviation. These details are absent from the current Code-SAYD.zip; if time permits, please provide them during the discussion period. They would help reproducibility and future comparisons.

We thank the reviewer for the follow-up. Indeed, we used a set of three random seeds in each experiment.

The reviewer suggests including the exact sample IDs used to construct the test-time prompts during the discussion period.

We have updated our supplementary material accordingly. Inside the Code-SAYD.zip, the prompts directory contains JSON files for 10 benchmarks (Ant, Cell, CUB, DRAM, Gala, JSRT, LDA, NYU, TigD, and YouT), each listing the three sets of image IDs (corresponding to the three random seeds) used for constructing the test-time prompts. As an example, the contents of JSRT.json are shown below. We will release our model, codebase, and exact data splits to ensure full reproducibility upon acceptance.

    "1":[
        "JPCLN011.png",
        "JPCNN092.png",
        "JPCLN149.png",
        "JPCNN036.png",
        "JPCLN090.png",
        "JPCNN011.png",
        "JPCNN083.png"
    ],
    "2":[
        "JPCLN017.png",
        "JPCLN044.png",
        "JPCNN037.png",
        "JPCNN010.png",
        "JPCLN047.png",
        "JPCNN012.png"
    ],
    "3":[
        "JPCNN035.png",
        "JPCLN066.png",
        "JPCNN073.png",
        "JPCLN050.png",
        "JPCLN101.png",
        "JPCNN027.png"
    ]
Reviewer_LtufOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SAYD, a model for in-context segmentation (ICS). Given a few prompt images with masks that define K classes, SAYD segments a query image. SAYD fine-tunes a DINOv3 ViT-L encoder and uses a Mask2Former-style decoder that sees only the query image and outputs class-agnostic masks. Each mask query is classified by the cosine similarity between its embedding and class prototypes, which are masked-average-pooled features of the prompt images.

Critical strengths
  1. Broad evaluation focused on out-of-domain data, with large gains where text prompting fails.

  2. A simple, efficient design that suits practical use. The decoder conditions only on the query, so prototypes can be precomputed and the cost per query stays essentially constant as the number of classes grows.

Critical weaknesses
  1. SAYD differs from the baselines simultaneously in training data (12 curated panoptic datasets), image encoder (fine-tuned DINOv3 ViT-L vs. frozen DINOv3 in INSID3 or Swin-L in DINOv), and architecture/loss. The ablations isolate SAYD's own design choices but not its gain over prior methods, so it is unclear how much of the improvement comes from the proposed techniques versus data and encoder.

  2. The strongest trained in-context baselines are missing. SegGPT and SINE are cited and open-source but never compared; Painter, SegGPT's predecessor, is used instead. Matcher and SegIC are excluded because they "produce only binary masks". Yet INSID3 has the same limitation and is included by running it once per class. The same could be done for these.

  3. SAM3, the strongest baseline overall, is prompted only with text, although it also accepts visual exemplar prompts. The paper's thesis is that visual demonstrations beat text, so SAM3 given visual prompts is the most relevant counterpart.

  4. The ablations are narrow. All of Tables 4–8 use 100k-iteration models, a single run each, on DRAM and TigD only. Neither is one of the far-domain benchmarks where the paper's advantage is claimed.

Questions for authors
  1. Table 3 compares FLOPs at each method's default resolution (512² for SAYD vs. 1008²/1024² for SAM3/INSID3). Could FLOPs also be reported at a matched resolution?

  2. Prototypes are mask-pooled encoder features, while the query side uses a learned decoder embedding (Q_cls) trained only on natural images. Did the authors compare Q_cls against mask-pooled query features, particularly on far-domain benchmarks?

3 comments
AuthorsOpenReview ↗

Part 1 of our rebuttal

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our broad evaluation, and our simple and efficient design. Below, we address each of the reviewer’s weaknesses and questions. Should any further clarification be needed, we would be happy to provide additional information.


SAYD differs from the baselines simultaneously in training data, image encoder, and architecture/loss. The ablations isolate SAYD's own design choices but not its gain over prior methods, so it is unclear how much of the improvement comes from the proposed techniques versus data and encoder.

To disentangle the contributions of (1) the image encoder, (2) the training data, and (3) our proposed architecture and loss, we break down our evidence across two controlled comparisons.

1. Controlling for the Image Encoder (vs. INSID3): The recent state-of-the-art method INSID3 (CVPR 2026) utilizes the exact same DINOv3 backbone as SAYD, keeping it frozen in a training-free pipeline. While INSID3 demonstrates what a frozen DINOv3 encoder can achieve out-of-the-box, unlocking DINOv3's full potential for cross-domain in-context segmentation via supervised adaptation is non-trivial. As reproduced from Table 1 below, SAYD significantly outperforms INSID3, demonstrating that the DINOv3 backbone alone does not account for SAYD's performance:

ModelGTADRAMCUBLIPYouTNYUiShapeTigDPlantAntLDAJSRTODMGalaCell
INSID315.2818.350.007.329.8610.6821.975.590.0319.8531.1510.5216.391.692.63
SAYD (ours)52.9558.0717.0629.6558.5140.8734.9215.5215.6232.0439.9116.8818.7310.1922.43

2. Isolating Proposed Techniques under Matched Data & Encoder (Tables 6–8): Crucially, simply fine-tuning DINOv3 on our training data with standard segmentation objectives fails to generalize. Under identical training data and the same DINOv3 encoder, our ablations demonstrate that each proposed mechanism is essential:

  • Loss & Calibration (Tables 6 & 8): Without learnable temperature&bias calibration and ignota-aware BCE supervision, fine-tuning DINOv3 suffers severe degradation. This confirms that data scaling alone cannot succeed without our training formulation.

  • Prompt-Decoupled Decoder (Table 4): Replacing existing prompt-coupled decoders with our lightweight, class-agnostic decoder under the exact same data and encoder substantially improves both efficiency and segmentation accuracy.

Together, these comparisons systematically isolate the impact of the backbone, data scale, and algorithmic contributions, demonstrating that SAYD's gains stem primarily from our proposed training and architectural mechanisms.


The reviewer suggests including more methods: SegGPT, SINE, Matcher, SegIC, and SAM3 with visual exemplars as prompts.

We have added all the suggested methods to Table 1 in the updated manuscript. Below, we reproduce the results based on Table 1. Notably, SAM3 does not natively support (cross-image) in-context segmentation; instead, it requires a user-provided spatial prompt (such as a bounding box) to segment similar instances within the same image. To adapt SAM3 to the exemplar-based setting, i.e., in-context segmentation, we concatenate the prompt images and the query image side-by-side, and overlay bounding boxes (derived from segmentation masks of prompts) onto the prompt images. We denote this variant SAM3-V. As reported in Table, SAM3-V significantly underperforms the text prompt-based SAM3. Overall, our SAYD significantly outperforms these added methods.

ModelGTADRAMCUBLIPYouTNYUiShapeTigDPlantAntLDAJSRTODMGalaCell
SegGPT1.770.420.061.240.240.784.601.090.024.837.619.880.960.781.24
SINE36.4939.230.076.9310.4032.3724.714.730.8925.9731.520.1813.731.522.30
Matcher41.5146.9817.3610.9530.0633.8822.6810.869.5627.9023.1614.0317.461.4513.87
SegIC32.5431.221.3213.687.9029.0227.4412.581.6526.2233.720.3318.575.721.58
SAM3-V4.813.4717.8220.092.913.845.871.774.0310.371.126.495.591.281.89
SAYD (ours)52.9558.0717.0629.6558.5140.8734.9215.5215.6232.0439.9116.8818.7310.1922.43
AuthorsOpenReview ↗

Part 2 of our rebuttal

The reviewer suggests expanding ablation studies in Table 4-8, particularly using far-domain benchmarks."

We added three more benchmarks to these tables in the updated manuscript. The added benchmarks are the Ant dataset (Ant), the land-cover dataset (LDA), and the Galaxy dataset (Gala). They focus on biological microscopy, remote sensing, and cosmological imagery, respectively. Therefore, they represent "far-domain benchmarks" relative to natural image training datasets such as COCO and Pascal (see Figure 1 for some visual examples). Across all these benchmarks, our ablation findings remain consistent, further validating the generalizability of our SAYD.


Table 3 compares FLOPs at each method's default resolution (512² for SAYD vs. 1008²/1024² for SAM3/INSID3). Could FLOPs also be reported at a matched resolution?

SAYD can flexibly operate at arbitrary image resolutions by bicubically interpolating its learned positional embeddings (e.g., scaling the grid from 32$\times$32 to 64$\times$64 to shift from 512$\times$512 to 1024$\times$1024 resolution). This allows us to evaluate all models under a matched resolution setting.

The table below compares the FLOPs (T) and wall-clock inference time (in seconds) when processing KK classes presented in the prompt images at a matched resolution of 1024$\times$1024 (alongside SAYD's default 512$\times$512 setting). As demonstrated, SAYD maintains superior computational efficiency and exhibits remarkable scaling stability as the number of prompt classes KK increases.

ModelResolutionFLOPsKK=1KK=10KK=100KK=1000
SAM31008$\times$10081.750.611.4510.3494.06
INSID31024$\times$10242.511.6216.47163.591635.71
SAYD1024$\times$10241.470.430.430.440.46
SAYD512$\times$5120.340.100.100.100.11

Prototypes are mask-pooled encoder features, while the query side uses a learned decoder embedding (Q_cls). Did the authors compare Q_cls against mask-pooled query features?

To evaluate this design choice, we trained a model that represents each mask using the corresponding mask-pooled query feature for classification. We compare this model with our SAYD, which learns Q_cls to represent predicted masks for classification. The table below lists their results on the Hypersim panoptic segmentation benchmark w.r.t metrics Recognition Quality (RQ) and Segmentation Quality (SQ). The two metrics are standard in panoptic segmentation [R1], enabling breakdown analysis on classification accuracy and mask quality.

classifier designRQ ↑\uparrowSQ ↑\uparrow
Q_cls (SAYD)61.0881.66
mask-pooled query feature54.5779.85

As shown in the table, replacing Q_cls, the "mask-pooled query feature" yields slightly lower SQ (79.85 vs. 81.66) but causes a significant drop in RQ (54.57 vs. 61.08). We attribute this degradation to feature contamination from imperfect mask predictions. Because mask-pooled query features are aggregated over predicted masks, boundary errors or background leakage pollute the pooled representation and destabilize the classification loss. In contrast, learning a dedicated Q_cls embedding decouples classification from noisy intermediate mask predictions, resulting in more stable optimization and higher recognition accuracy.

  • [R1] Kirillov, et al. Panoptic segmentation, CVPR, 2019.

We thank the reviewer for confirming that our responses have addressed their concerns and are delighted to see their positive recommendation. As a brief note, the confirmation message was inadvertently posted under the thread of Reviewer cB8Y.

Reviewer_cB8YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAYD defines segmentation classes through image–mask demonstrations. It generates candidate masks from the query image and classifies them using demonstration feature prototypes. The encoder and decoder are jointly fine-tuned, with classes absent from the demonstrations, termed ignota, treated as negatives. BCE supervision and learnable temperature and bias support classification and rejection. The model is trained on 12 public datasets and evaluated on 24 cross-domain benchmarks covering semantic, instance, and panoptic segmentation.

Critical strengths

The design is simple and the training recipe has practical value: Demonstration prototypes can be precomputed and reused, and one framework supports semantic, instance, and panoptic segmentation. Ablations of encoder fine-tuning, negative supervision, and calibration support the overall recipe and provide useful insights for building segmentation systems guided by visual demonstrations.

Critical weaknesses
  1. The methodological advance and evidence for the key mechanism remain insufficiently established. Mask decoding with a separate classification branch is standard in Mask2Former. SAYD should specify the advances introduced by demonstration-prototype matching, training, and calibration. Existing binary ICS methods already exclude non-target regions through background supervision, while CDICS uses negative examples that do not satisfy the prompt, with all-zero target masks, to learn rejection. The added capability of ignota therefore needs to be established relative to these mechanisms, rather than through internal ablations alone. Targeted comparisons are needed to substantiate this claimed advance. Final mIoU in Table 4 combines mask and classification errors. It does not establish that the gains arise from improved boundaries, much less validate the explanation that excessive semantic alignment harms mask quality. These issues affect the assessment of novelty and the central mechanism: the results support effectiveness, but the explanation of why the method works and what capability it adds remains incomplete.

  2. Missing key baselines limits the performance and cross-paradigm conclusions. The main table omits representative ICS methods such as SegGPT and SINE without explanation, leaving insufficient evidence for the claimed advantage over prior ICS methods. Cross-domain generalization is already a goal of the field, so claims about prior limitations should be restricted to the tested methods and settings. Text-based comparisons also focus on open-vocabulary segmentation and do not sufficiently support claims about limitations of linguistic expression. VLM-based referring segmentation with descriptive instructions, evaluated for both accuracy and inference cost, could test this claim and clarify practical value, but cannot replace the missing direct ICS baselines.

Questions for authors
  1. What mechanisms does SAYD add beyond existing mask decoding and prototype matching methods? How are their contributions distinguished from encoder fine-tuning and expanded training data?
  2. Are there mask or boundary quality results controlling for classification errors that support the mechanism proposed for Table 4? If not, should this explanation be restricted to an unverified hypothesis?
  3. What additional rejection capability do ignota supervision and calibration provide over existing background supervision and negative-example mechanisms such as CDICS? Which direct comparisons support this advance?
  4. Why were SegGPT and SINE omitted? Suggested additional validation: Compare them under a consistent demonstration protocol, and report accuracy and inference cost for VLM-based referring segmentation with descriptive instructions on a suitable subset.
4 comments
AuthorsOpenReview ↗

Part 1 of our rebuttal

We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including the simple design and practical value of our training recipe, our comprehensive ablation studies, and our useful insights. Below, we address each of the reviewer’s weaknesses and questions. Should any further clarification be needed, we would be happy to provide additional information.


What mechanisms does SAYD add beyond existing mask decoding and prototype matching methods? How are their contributions distinguished from encoder fine-tuning and expanded training data?

SAYD introduces two distinct mechanisms: (i) a lightweight, prompt-decoupled, class-agnostic mask decoder; (ii) the learnable temperature and bias term within BCE loss. Below, we elaborate on each and clarify how they distinguished from encoder finetuning and expanded training data.

  • Lightweight, prompt-decoupled, class-agnostic mask decoder. Unlike existing mask decoders that rely on prompt feature maps or prompt-derived class prototypes, our decoder predicts masks independently of the prompts (ref. Fig. 4ab). With the identical encoder finetuning and training data, Table 4-left shows that our decoder design reduces both parameter count and FLOPS and improves the final performance; Table 4-right shows that our decoder particularly outperforms the existing prompt-dependent counterparts on Boundary Segmentation Quality metric (76.05 vs. 62.58). We reproduce Table 4-right in our response to your next question.

  • Learnable temperature and bias term within BCE loss. We jointly optimize a learnable temperature and bias term within the BCE loss alongside the model parameters. Table 6 (reproduced below) shows that this calibration mechanism is essential for effective training. Because the loss parameters operate independently of the encoder and dataset, their gains are distinct from and complementary to encoder finetuning and expanded training data. | temp. | bias | DRAM ↑\uparrow | TigD ↑\uparrow | Ant ↑\uparrow | LDA ↑\uparrow | Gala ↑\uparrow | |:-----:|:----:|------:|------:|-----:|-----:|------:| | ✗ | ✗ | fail | fail | fail | fail | fail | | ✓ | ✗ | 10.79 | 7.76 | 5.27 | 10.86 | 4.37 | | ✗ | ✓ | fail | fail | fail | fail | fail | | ✓ | ✓ | 59.20 | 12.69 | 31.76 | 39.85 | 10.54 |


The final mIoU in Table 4 combines mask and classification errors. Are there mask or boundary quality results controlling for classification errors that support the mechanism proposed for Table 4?

Great suggestion! We included Table 4-right in the updated manuscript that evaluates mask and boundary quality. The evaluation measures performance using Boundary Segmentation Quality (Boundary SQ) [R1, R2], which focuses on mask geometry and boundary precision, alongside Recognition Quality (RQ) and Segmentation Quality (SQ). These metrics are standard in the literature of panoptic segmentation; the results are on the Hypersim panoptic segmentation benchmark. The results demonstrate that our prompt-decoupled decoder produces substantially finer-grained boundaries than alternative designs that couple mask prediction with prompt conditioning. For the reviewer's convenience, we reproduce Table 4-right below.

DesignRQ ↑\uparrowSQ ↑\uparrowBoundary SQ ↑\uparrow
a (ours)61.0881.6676.05
b-159.4264.9157.94
b-260.1770.6262.58
  • [R1] Kirillov, et al. Panoptic segmentation, CVPR, 2019.
  • [R2] Cheng, et al. Boundary IoU: Improving Object-Centric Image Segmentation Evaluation, CVPR, 2021.
AuthorsOpenReview ↗

Part 2 of our rebuttal

Existing binary ICS methods already exclude non-target regions through background supervision (e.g., CDICS uses negative examples with all-zero target masks). How is ignota supervision and calibration different from existing background supervision?

The reviewer is spot on that, from the perspective of binary classification, our defined "ignota" and conventionally defined "background" regions provide the same supervision, as both serve as negatives relative to the foreground target regions. However, we explicitly define "ignota" to distinguish it from standard "background" based on its intrinsic nature and the specific advantages it introduces during training, particularly with respect to the mask prediction loss. We clarify these distinctions below.

  • Concept. Ignota regions correspond to (novel) classes present in the query but absent from the visual demos (L233). These regions are annotated in the training dataset, but they arise dynamically even within the exact same training image depending on sampled prompt/query counterparts. By contrast, "background" is conventionally defined as a static, catch-all region for un-annotated pixels. Introducing a distinct term from "background" provides clearer conceptual precision for understanding and formulating ICS training.

  • Mask supervision. Unlike "background" which acts as an undifferentiated catch-all for miscellaneous image regions, "ignota" regions correspond to concrete instance-level and semantic-level masks. Because these ground-truth masks are already provided in the publicly-available training datasets, they can be directly leveraged to supervise the mask prediction loss.

  • Ignota vs. CDICS. CDICS constructs negative training pairs by selecting prompt images containing (novel) classes absent from the query image, which sets the entire target mask to be all-zero. In contrast, our ignota formulation allows us to randomly sample images to form prompt-query pairs (§4.2), where region-level ignota arise naturally that circumvent the need to explicitly mine negative counterparts with all-zero masks. This substantially increases the diversity of prompt-query combinations during training. We attribute the flexibility of prompt-query training pair construction to the ignota concept.

Lastly, we clarify that, with KK-way classification loss, inter-class masks serve as negatives to each other. Strictly speaking, this is independent of ignota or background, which, yet, provides more negatives for all classes. Other than this, we note that learning a temperature and bias term alongside model parameter updates is essential to successful training, as demonstrated by Table 6, way beyond calibration.


The reviewer suggests comparing more methods such as SegGPT and SINE.

Table 1 in the updated manuscript now includes SegGPT and SINE. The comparison shows that our SAYD resoundingly outperforms these methods, corroborating the superior performance of SAYD. We reproduce Table 1 below for reviewing convenience:

ModelGTADRAMCUBLIPYouTNYUiShapeTigDPlantAntLDAJSRTODMGalaCell
SegGPT1.770.420.061.240.240.784.601.090.024.837.619.880.960.781.24
SINE36.4939.230.076.9310.4032.3724.714.730.8925.9731.520.1813.731.522.30
SAYD (ours)52.9558.0717.0629.6558.5140.8734.9215.5215.6232.0439.9116.8818.7310.1922.43
AuthorsOpenReview ↗

Part 3 of our rebuttal

The reviewer suggests reporting the accuracy and inference cost for VLM-based referring segmentation with descriptive instructions.

To address this suggestion, we evaluate the recently released open-source WeVisOne-VL-4B [R1], a state-of-the-art Vision-Language Model (VLM) supporting segmentation. We test two prompting strategies:

  1. WeV-cls: Prompts containing only the class name.
  2. WeV-ins: Prompts combining the class name with a descriptive instruction. Specifically, we use Qwen3.8 to generate a concise one-sentence description for each class. For example, for "person", it generates "a human being, including any man, woman, or child in the scene".

We have incorporated these results into Table 1 (segmentation accuracy) and Table 3 (inference efficiency) of the revised manuscript, and reproduce them below for convenience. Overall, SAYD substantially outperforms WeVisOne-VL-4B across these benchmarks while operating at a small fraction of the computational cost.

1. Segmentation Accuracy Comparison (extracted from Table 1).

ModelGTADRAMCUBLIPYouTNYUiShapeTigDPlantAntLDAJSRTODMGalaCell
WeV-cls39.2749.451.5314.406.7621.268.9814.050.1321.8621.7313.801.970.323.01
WeV-ins38.6258.332.7617.746.2119.788.9114.761.1231.0924.529.081.640.410.96
SAYD (ours)52.9558.0717.0629.6558.5140.8734.9215.5215.6232.0439.9116.8818.7310.1922.43

2. Inference Cost Comparison (extracted from Table 3). Below we compare parameter count, peak memory consumption, and FLOPs. Because WeV-cls and WeV-ins exhibit negligible differences in runtime overhead, we report them jointly as WeV. Here, KK denotes the number of target classes in the prompt images; unlike SAYD, which processes all KK classes in a single forward pass, WeV must execute KK separate forward passes per query image.

ModelParamsMemoryFLOPsImage runsPrompt runs
WeV4.10B22.62G14.52TKK
SAYD0.31B1.56G0.34T11
  • [R1] Liang, et al. WeVisOne-VL: Unified Visual Perception from Objects to Pixels, arxiv: 2610.31329, 2026.

I appreciate the additional experiments and analysis. My concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.