acceptodds
Withdrawn from ICLR 2027

What You Show Is Not What You Get: Rethinking Demonstration Following in Visual In-Context Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual in-context segmentation uses visual demonstrations to guide predictions for a new image. We identify background color mapping deviation, where non-target regions sharing a background label receive different output colors, and part-to-parent expansion, where predictions extend beyond the requested part into its parent object. These deviations can occur even when the relevant regions remain distinguishable in internal features. Controlled changes to demonstration colors and requested targets support an account in which demonstrations guide role and color assignments over familiar semantic regions, influenced by learned semantic and color preferences. Building on these findings, we introduce Semantic Contrast Prompt Selection (SCPS). Its semantic contrast encoding assigns separate colors to the requested part and the rest of its object while keeping the remaining scene black. SCPS combines existing retrieval scores with quality estimates from labeled development images to select demonstrations and their binary or semantic contrast encodings. Averaged across seven retrieval methods on three benchmarks, SCPS improves target IoU by 0.054 and reduces leakage into the rest of the object by 22.7% relative to the original binary selectors. Our findings offer a new perspective on understanding and improving demonstration following in visual in-context segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual In-Context Learning, Visual In-Context Segmentation, Demonstration Following, Representation Analysis, Prompt Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8ZA5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether visual in-context segmentation models actually follow the mapping specified by a demonstration. Using SegGPT, the authors identify two failure modes. First, regions that share the same background label in the demonstration can receive different colors in the output, which they call background color mapping deviation. Second, when only a part of an object is requested, the prediction can expand to the rest of the object, which they call part-to-parent expansion.

The paper argues that these errors do not simply arise because the model fails to distinguish the relevant regions. Controlled changes to demonstration colors and requested targets, together with linear probes and feature interventions, show that the requested part and the rest of its parent object can remain distinguishable in late features even when they receive the same foreground prediction. For example, across 1,085 expansion cases, the requested part and parent-only region are separated by linear probes with an average AUROC of 0.926.

Based on this observation, the authors propose Semantic Contrast Prompt Selection (SCPS). Instead of giving the target and the rest of its parent object the usual foreground/background encoding, semantic contrast gives the parent-only region a separate color. SCPS then uses labeled development images to estimate the quality of each demonstration and chooses both the demonstration and its encoding. Across three part-segmentation benchmarks and seven retrieval methods, SCPS improves target IoU and Strict Target Success while reducing Parent Leakage compared with the original binary selectors.

Critical strengths
  1. The diagnostic experiments go beyond simply reporting a new failure case. The most convincing part of the paper is that the authors try to separate representation failure from output failure. In particular, the requested part can remain clearly distinguishable from the rest of the parent object in the model's late features even when both regions are predicted as foreground. The paper also changes the requested part while keeping the images fixed, which gives stronger evidence that the demonstration affects the internal representation rather than the probe only recovering object identity.

I find this important because it changes the interpretation of part-to-parent expansion. The error is not well explained by saying that the model simply cannot tell the two regions apart. The additional feature interventions and matched controls make this point more convincing than a linear-probe result alone. This is a useful diagnostic result for understanding visual in-context segmentation.

  1. SCPS is evaluated with unusually careful controls for a relatively simple method. The main experiments cover three benchmarks and seven different demonstration selectors, rather than showing that the method works only with one retrieval strategy.

The paper also separates the effects of semantic contrast and target-quality calibration, checks different output decoding choices, and compares against recoloring the full background rather than specifically marking the parent-only region. These controls make it easier to understand which part of SCPS is responsible for the observed behavior.

This matters because semantic contrast by itself involves a very simple change to the demonstration. The paper provides enough controlled evidence that the reduction in parent leakage is not merely an artifact of adding another output color.

Critical weaknesses
  1. The broader mechanistic claim is still supported mainly by one model.

Most of the analysis, including both identified failure modes and all SCPS experiments, is conducted on SegGPT. The paper itself acknowledges that the main analysis primarily concerns SegGPT. The additional SINE experiment is useful, but it establishes a narrower result: semantic regions can remain distinguishable internally even when the final binary output treats them the same. It does not reproduce the background color mapping behavior, nor does it establish the same explanation for part-to-parent expansion.

This makes me hesitant about the broader framing around demonstration following in visual in-context segmentation. The results convincingly characterize SegGPT, but it is not yet clear whether the proposed account reflects a common property of visual in-context segmentation models or behavior that is strongly tied to SegGPT's RGB output formulation and training procedure. I would view the contribution more favorably if the claim were scoped more explicitly to SegGPT, or if similar failure behavior could be demonstrated in another model.

  1. Much of SCPS's headline performance gain comes from labeled target-quality calibration rather than semantic contrast itself.

SCPS does more than introduce the semantic contrast encoding. It also evaluates candidate demonstrations on labeled development images and uses their target IoU to change the retrieval score. The method therefore requires target and parent annotations for demonstrations as well as task-specific labeled development data.

This distinction is important when interpreting the main improvement. The original binary selectors have a mean target IoU of 0.2412, while SCPS reaches 0.2952. However, the paper's own ablation shows that much of this improvement is already obtained by applying target-quality calibration to the binary encoding. Semantic contrast without calibration substantially reduces parent leakage but decreases target IoU. Thus, the strongest evidence for semantic contrast itself is the reduction in leakage, while its additional contribution to target segmentation quality is considerably smaller once the labeled calibration signal is controlled for.

This does not invalidate SCPS, since the authors do provide the relevant ablation. However, it weakens the connection between the paper's mechanistic analysis and the headline IoU improvement, and it also makes the amount of additional supervision an important part of the practical comparison.

Questions for authors
  1. Clarification on the scope of the mechanistic claim. Do the authors intend the proposed account of demonstration following to describe SegGPT specifically, or visual in-context segmentation more generally? The SINE experiment shows that internal semantic distinctions can survive the output stage, but it does not appear to establish either background color mapping deviation or the same mechanism behind part-to-parent expansion. Evidence of a comparable failure on another VICL model, or a narrower statement of the claim, would materially affect my assessment of the generality of the contribution.

  2. Clarification on the supervision required by SCPS. How sensitive is SCPS to the amount of labeled development data used for target-quality calibration? Since semantic contrast without calibration reduces leakage but does not improve target IoU, while calibrated binary selection already accounts for a substantial fraction of the final performance gain, it would be useful to understand how SCPS behaves when only a small labeled development set is available. This would clarify how much of the practical gain comes from the proposed encoding and how much comes from additional labeled calibration.

Reviewer_eYUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the issue in Visual In-Context Segmentation where models fail to strictly adhere to provided examples. By manipulating the colors of the examples and the target regions—and employing linear probes to analyze the model’s internal features—the authors discovered that even when a model produces an incorrect final output, its internal representations may still retain the correct semantic regions and target information. Building on this, the authors propose SCPS, a method that assigns distinct colors to the target region, other parts of the same object, and the background. It then selects the optimal example and encoding strategy based on quality estimates derived from validation data. Evaluated across three segmentation datasets, this approach yields an average improvement of 0.054 in target IoU and a 22.7% relative reduction in parent leakage. While the findings are intriguing, the conclusions rely heavily on the SegGPT model, and the proposed method requires additional annotation and calibration.

Critical strengths
  • The perspective presented by the authors is valuable for research into the mechanisms of Visual In-Context Learning.
  • The experiment makes the results relatively complete and credible.
  • SCPS does not require the training of an additional, complex segmentation network, making it convenient to use.
Critical weaknesses
  • Although the author discusses the general problem of Visual In-Context Segmentation, the primary experiments rely solely on SegGPT. The experiments for SINE do not fully validate the bias related to background color mapping or the effectiveness of SCPS. This raises significant concerns regarding generalizability.
  • The methodological innovation of SCPS is limited, and the final scoring function is merely a simple weighted sum.
  • One argument presented in this paper is that errors in the model’s output stem from the interplay between semantic recognition, role assignment, and color preferences. However, a linear probe only demonstrates that the internal representations contain certain decodable information; it does not directly prove that the model actually employs the specific mechanism proposed by the authors.
  • SCPS requires the use of labeled development set images for a given task to estimate the quality of each demonstration-encoding pair. This means that SCPS is not strictly a general-purpose prompt selection method that operates entirely without additional supervision.
Questions for authors

Refer to weakness part.

Reviewer_wPxmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates demonstration-following failures in visual in-context segmentation, identifying background color mapping deviation and part-to-parent expansion. Based on these observations, the authors propose Semantic Contrast Prompt Selection (SCPS), which combines semantic contrast encoding with quality-calibrated demonstration selection to improve target segmentation and reduce parent leakage. Experiments on three part segmentation benchmarks demonstrate consistent improvements across seven retrieval methods.

Critical strengths
  1. The motivation is good. The paper identifies meaningful demonstration-following failures in visual in-context segmentation and provides useful insights through controlled color experiments and feature-level analyses.

  2. Consistent experimental improvements. SCPS improves target segmentation and reduces parent leakage across multiple retrieval methods and benchmarks. The experiments include several ablations and diagnostic analyses that help explain the observed behavior.

Critical weaknesses
  1. Limited methodological novelty and unclear contribution of semantic contrast encoding. SCPS mainly combines a simple three-color encoding strategy with existing quality-calibrated demonstration selection. Table 2 shows that calibrated binary selection already achieves 0.2875 Target IoU, while SCPS only improves it to 0.2952. The authors should better clarify the methodological novelty and demonstrate the distinctive contribution of semantic contrast encoding.

  2. Dependence on additional annotations and calibration. SCPS requires target and parent annotations for demonstrations, along with labeled development images for task-specific quality estimation. This additional supervision may limit its practical applicability compared with conventional demonstration retrieval. The authors should clarify the annotation costs and evaluate performance under more limited development-data budgets.

  3. Limited generalization beyond SegGPT. The main analysis and SCPS evaluation rely almost entirely on SegGPT. Although Appendix A.5 provides complementary feature analysis on SINE, the effectiveness of SCPS on other visual in-context segmentation architectures remains unclear. Additional cross-model experiments would strengthen the generality of the findings.

  4. Insufficient evidence for the proposed mechanistic explanation. The linear probing results demonstrate that semantic regions remain distinguishable in internal features, but this does not fully establish the proposed mechanism involving learned semantic and color preferences. More direct evidence is needed to clarify how these internal representations cause the observed demonstration-following deviations.

Questions for authors

Please see the weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.