- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The submission presents SAYD, a framework for in-context segmentation (ICS) that segments query images conditioned on visual demonstrations (image–mask pairs) rather than text prompts. To circumvent the computational bottleneck of prior visual ICS methods that re-run forward passes for each category, the authors decouple mask proposal from prompt conditioning. A query-only Transformer decoder predicts class-agnostic binary masks directly from query features, which are then classified in parallel via cosine similarity against class prototypes pooled from the prompt demos.
Key Contributions
Prompt-Agnostic Mask Decoder: Generates mask proposals strictly from the query image, reducing multi-class inference cost from to and avoiding mask boundary degradation caused by heavy prompt cross-attention. Ignota-Aware Calibrated BCE Loss: Formulates mask classification via binary cross-entropy with a learnable temperature and bias , treating query regions absent from the prompt demos ("ignota") as explicit negatives alongside background. Encoder Adaptation for In-Context Matching: Demonstrates that finetuning the pretrained ViT backbone (DINOv3 ViT-L) is necessary for dense visual correspondence, contrasting with prior works that rely on frozen backbones.
Empirical Validation
The model is trained on 12 public panoptic datasets (~0.91M images, 11.46M masks) and evaluated across 24 out-of-domain benchmarks spanning semantic, instance, and panoptic segmentation. Evaluated against text-driven models (SAM3) and visual prompting baselines (DINOv, INSID3). SAYD maintains constant runtime across class counts and shows substantial gains on specialized scientific domains (microscopy, astronomy, metallography) where open-vocabulary text models struggle with tacit visual knowledge.
Critical strengths
The evaluation is serious. Testing across 24 genuine out-of-domain scientific benchmarks—cell biology, astronomy, metallurgy—is far more convincing than the customary incremental gains on COCO. It actually substantiates the central thesis: visual demonstrations bypass the vocabulary bottleneck and capture tacit visual knowledge precisely where text-prompted models like SAM3 collapse. On top of that, decoupling the mask decoder from prompt features is just sensible engineering; dropping multi-class inference from down to makes visual in-context segmentation computationally viable rather than a curiosity.
Critical weaknesses
The empirical validation suffers from selective baseline curation and a fragile evaluation protocol.
On baselines, the paper leans heavily on outperforming text-prompted models like SAM3, which is unsurprising since text prompts naturally collapse on specialized scientific jargon. Against native visual in-context segmentation (ICS) methods, however, the comparison is remarkably thin: Painter is an outdated 2023 color-regression heuristic that fails across the board, and INSID3 is restricted to single-class binary masks. The authors actively discuss and ablate against native visual ICS frameworks like SINE (NeurIPS 2024), SegGPT, and SegIC in the text, yet omit them entirely from the 24-benchmark evaluation. Furthermore, on biomedical benchmarks where SAYD claims outsized gains, UniverSeg is conspicuously absent.
Compounding this, the evaluation protocol—randomly drawing ~10 prompt images across just three runs—carries immense variance. In visual in-context learning, exemplar choice heavily dictates performance. Without providing fixed, standardized prompt splits or exact image indices, and given that trained checkpoints are omitted from the submission, these benchmark numbers are virtually impossible for the community to faithfully reproduce or verify.
Questions for authors
How does SAYD compare against peer visual in-context baselines—specifically SINE, SegGPT, UniverSeg (on biomedical tasks), and native SAM 2 using memory-based cross-frame visual prompting—rather than text-prompted models like SAM3?
Given the variance of evaluating on ~10 random visual prompts across only three runs, can you provide standard deviations, the exact prompt splits/IDs used, and an anonymous link with trained checkpoints and evaluation code to verify reproducibility?
5 comments
I appreciate the additional comparisons and variance estimates. Please include the exact sample IDs used to construct the test-time prompts in the code release. Each experiment should use a fixed set of at least three random seeds and report the mean and standard deviation. These details are absent from the current Code-SAYD.zip; if time permits, please provide them during the discussion period. They would help reproducibility and future comparisons.
We thank the reviewer for the follow-up. Indeed, we used a set of three random seeds in each experiment.
The reviewer suggests including the exact sample IDs used to construct the test-time prompts during the discussion period.
We have updated our supplementary material accordingly. Inside the Code-SAYD.zip, the prompts directory contains JSON files for 10 benchmarks (Ant, Cell, CUB, DRAM, Gala, JSRT, LDA, NYU, TigD, and YouT), each listing the three sets of image IDs (corresponding to the three random seeds) used for constructing the test-time prompts. As an example, the contents of JSRT.json are shown below. We will release our model, codebase, and exact data splits to ensure full reproducibility upon acceptance.
"1":[
"JPCLN011.png",
"JPCNN092.png",
"JPCLN149.png",
"JPCNN036.png",
"JPCLN090.png",
"JPCNN011.png",
"JPCNN083.png"
],
"2":[
"JPCLN017.png",
"JPCLN044.png",
"JPCNN037.png",
"JPCNN010.png",
"JPCLN047.png",
"JPCNN012.png"
],
"3":[
"JPCNN035.png",
"JPCLN066.png",
"JPCNN073.png",
"JPCLN050.png",
"JPCLN101.png",
"JPCNN027.png"
]
We appreciate the reviewer’s constructive feedback, which helped us significantly improve the quality of our manuscript; we have uploaded a revised version with all changes highlighted in blue. The reviewer recognized the strengths of our work, including our serious and convincing evaluation, and our sensible mask decoder design. Below, we address each of the reviewer’s weaknesses and questions. Should any further clarification be needed, we would be happy to provide additional information.
Table 1 in the updated manuscript now includes SINE, SegGPT, UniverSeg, and SAM 2. The comparison shows that our SAYD resoundingly outperforms these methods, corroborating the superior performance of SAYD. We reproduce Table 1 below for reviewing convenience:
We are fully committed to reproducibility and address each requested item below:
Evaluation code: Our evaluation code is already included in the previously submitted supplementary material (
Code-SAYD.zip) via the Jupyter notebookdemo_SAYD_eval.ipynb. For convenience, the notebook contains both the Python script and pre-computed intermediate and final outputs, allowing the evaluation pipeline to be inspected directly without executing the code.Model and prompt splits: Because the trained SAYD checkpoint is 1.24GB, exceeding the file size limit for supplementary material, we have hosted it anonymously on HuggingFace https://huggingface.co/Anonymous-27/anonymous/blob/main/ckpt.pth. Upon publication, we will publicly release the model and prompt splits across all benchmarks.
Standard deviation: Augmenting Table 1, the table below reports the mean performance and standard deviation across prompt splits for SAYD, SegGPT, and INSID3 on representative benchmarks. To keep the rebuttal concise and readable, we omit the full breakdown across all 20 benchmarked methods and 24 benchmarks here. We will include the complete standard deviation results in the appendix of our next version. As shown below, SAYD exhibits low variance alongside its substantial performance gains. | model | GTA | DRAM | CUB | LIP | YouT | NYU | iShape | | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | SegGPT | 1.77$^{\pm 1.97}$ | 0.42$^{\pm 0.81}$ | 0.06$^{\pm 0.04}$ | 1.24$^{\pm 0.36}$ | 0.24$^{\pm 0.56}$ | 0.78$^{\pm 1.08}$ | 4.60$^{\pm 2.75}$ | | INSID3 | 15.28$^{\pm 3.07}$ | 18.35$^{\pm 2.48}$ | 0.01$^{\pm 0.01}$ | 7.32$^{\pm 1.77}$ | 9.86$^{\pm 1.19}$ | 10.68$^{\pm 3.04}$ | 21.97$^{\pm 1.95}$ | | SAYD | 52.95$^{\pm 2.64}$ | 58.07$^{\pm 1.87}$ | 17.06$^{\pm 1.90}$ | 29.65$^{\pm 1.92}$ | 58.51$^{\pm 0.82}$ | 40.87$^{\pm 0.78}$ | 34.92$^{\pm 2.44}$ |