acceptodds
Under review as a conference paper at ICLR 2027

Beyond Selection : Integrating Complementary Information for Training-Free Visual In-Context Segmentation

Abstract

Recent visual foundation models provide strong visual representations and segmentation capabilities, enabling training-free visual prompt-based in-context segmentation (VP-ICS) for novel segmentation tasks. However, existing VP-ICS methods predominantly rely on winner-take-all selection strategies, which retain only a limited subset of retrieved evidence and can discard complementary information distributed across multiple support shots. We propose Integra-ICS, a training-free VP-ICS framework that integrates support evidence to exploit such distributed information. For each query token, Integra-ICS retrieves the global top- support tokens from a unified support token pool containing both foreground and background. By applying masked softmax aggregation to query-support token similarities, Integra-ICS integrates complementary evidence across support shots. Candidate foreground regions are subsequently localized and filtered by a minimum-evidence criterion to obtain reliable predictions. Across eight diverse benchmarks, Integra-ICS outperforms the strongest baseline by 3.60 and 3.56 mIoU points on average in the 5-shot and 10-shot settings, respectively, using a single hyperparameter configuration without dataset-specific tuning. The improvement comes from integrating complementary cross-shot evidence, while the minimum-evidence criterion retains reliable predictions supported by foreground tokens from the support set.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
In context Segmentation, Training-Free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WcazOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Integra-ICS, a training-free visual in-context segmentation framework built on frozen DINOv3 features. It aggregates foreground and background evidence through global top-k retrieval across support images, followed by candidate-guided query clustering and minimum-evidence filtering. Experiments on eight benchmarks show improved average segmentation performance in both 5-shot and 10-shot settings using a single hyperparameter configuration.

Critical strengths
  1. The paper presents a clear motivation for aggregating multiple support correspondences rather than relying on a single nearest neighbor, addressing a plausible source of missed foreground regions.

  2. Table 1 demonstrates strong average performance across eight diverse benchmarks, with improvements of 3.60 and 3.56 mIoU points over the strongest baseline on average in the 5-shot and 10-shot settings, respectively.

  3. The framework requires no task-specific training and uses a single hyperparameter configuration across datasets. The runtime analysis also shows inference speed close to INSID3, suggesting that the additional aggregation introduces limited overhead under the evaluated setting.

Critical weaknesses
  1. The related-work discussion overlooks relevant retrieval-based segmentation methods, such as “Shift the Lens: Environment-Aware Unsupervised Camouflaged Object Detection” (CVPR 2025) and “Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection” (ICCV 2025). In particular, the latter constructs foreground/background prototype libraries and uses top-k nearest-neighbor retrieval and voting for segmentation, which is technically relevant to the proposed evidence aggregation. Although these methods address a different task setting, discussing their connections and differences would help clarify the novelty of the proposed approach.

  2. The methodological novelty appears incremental. The framework combines global top-k retrieval, similarity-weighted aggregation, clustering, and filtering, but the manuscript does not sufficiently explain what distinguishes this combination from a straightforward nearest-neighbor label aggregation pipeline. The use of standard operations is not itself a weakness; the concern is that the distinctive technical insight and the necessity of the proposed combination remain insufficiently established.

  3. The contribution over “INSID3: Training-Free In-Context Segmentation with DINOv3” (CVPR 2026) should be more clearly isolated through controlled comparisons. In particular, the core aggregation mechanism lacks simple baselines, such as uniform voting over the same top-k neighbors and per-shot top-k aggregation. Replacing only the correspondence module of INSID3 while keeping the remaining pipeline unchanged would also help determine how much of the improvement comes from the proposed evidence integration.

Questions for authors
  1. What distinguishes the proposed retrieval and aggregation mechanism from prior retrieval-based segmentation approaches, particularly “Shift the Lens: Environment-Aware Unsupervised Camouflaged Object Detection” and “Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection”? Beyond the different task setting, which technical contribution is specific to Integra-ICS?

  2. Does similarity-weighted aggregation outperform uniform voting over the same top-k neighbors, and does global retrieval outperform per-shot top-k aggregation under otherwise identical conditions? Such comparisons would help establish whether the proposed aggregation provides benefits beyond considering additional neighbors.

  3. Which components are inherited from INSID3, and which are newly introduced? Can the contribution of evidence integration be isolated by replacing only INSID3’s correspondence module while keeping the remaining pipeline unchanged?

  4. What evidence distinguishes genuine cross-shot complementarity from redundant support or the general benefit of retrieving more tokens? In particular, does the CSCR analysis establish that contributions from different shots are necessary for the recovered predictions, rather than merely present among the retrieved neighbors?

3 comments

[W1] Relation to retrieval-based segmentation methods (EASE, RISE)

W1: The related-work discussion overlooks relevant retrieval-based segmentation methods, such as EASE (CVPR 2025) and RISE (ICCV 2025), which constructs FG/BG prototype libraries and uses top-k nearest-neighbor retrieval and voting; discussing their connections and differences would help clarify the novelty.

  • Answer: We thank the reviewer for pointing out these relevant methods. We agree that EASE and RISE should have been cited and discussed, and we will add them to the related-work section in the revision. Although all three methods use foundation-model retrieval for foreground/background discrimination, EASE and RISE are developed for unsupervised camouflaged object detection, and neither is formulated around episodic visual-prompt inference or the loss of complementary evidence caused by winner-take-all (WTA) correspondence across annotated support shots. The main distinctions are summarized below.

    AspectEASERISEIntegra-ICS
    ProblemDetect camouflaged objects by identifying and inverting the salient environmentGenerate pseudo-masks for COD using dataset-level informationRecover cross-shot foreground evidence discarded by WTA selection in multi-shot VP-ICS
    Retrieval referenceGenerated environment prototype libraryFG/BG prototype libraries mined from unlabeled training imagesRaw FG/BG tokens from the annotated support images of the current episode
    Retrieval / aggregationGlobal-to-local environment retrieval with KDE-based thresholds, refined by self-retrieval with image-level FG/BG prototypesTop-KK (K=512K=512) prototype label votingGlobal top-kk (k=5k=5) retrieval across all support shots
    Role of retrievalEnvironment mask, inverted to obtain the object maskPseudo-masks for training a downstream COD modelCandidate mask that guides prototype extraction and cluster scoring, followed by minimum-evidence filtering

    EASE's self-retrieval also pools foreground/background prototypes under a coarse mask, but it builds a single image-level prototype per class and re-classifies each pixel by their contrast; CMPE instead computes a prototype for each query cluster from only its candidate tokens and uses it for cluster-level scoring against the support prototype.

    We do not claim top-kk retrieval itself as a new generic operation. Integra-ICS is an observation-driven contribution: we identify and quantify a specific bottleneck in multi-shot VP-ICS, i.e., foreground evidence distributed across support shots being discarded by WTA selection, and introduce our problem-specific mechanism to resolve it. Our controlled comparison (see [W3]) shows that what matters is where the retrieval is performed rather than how the retrieved labels are weighted: global retrieval over all shots improves over 1-NN with either softmax weighting (63.48) or uniform voting (63.64), whereas shot-wise aggregation does not (62.43 vs. 62.52). We will revise the related-work and contribution descriptions accordingly.

[W2] Distinctive insight beyond nearest-neighbor label aggregation

W2: The framework combines global top-k retrieval, similarity-weighted aggregation, clustering, and filtering, but the manuscript does not sufficiently explain what distinguishes this combination from a straightforward nearest-neighbor label aggregation pipeline.

  • Answer: Thank you for this comment. We agree that the token-level aggregation is simple; in fact, our controlled study shows that uniform label voting over the same global top-kk neighbors performs comparably to similarity weighting (63.64 vs. 63.48). The distinctive insight therefore lies not in the voting rule but in two points that a straightforward nearest-neighbor label aggregation pipeline does not address. (1) Which neighbors compete. The bottleneck we identify is that per-shot WTA correspondence discards foreground evidence held by the other shots; ranking FG/BG tokens from all shots in a single list resolves it, whereas aggregating more neighbors within each shot does not (shot-wise top-kk: −0.09 despite five times more tokens). (2) How token evidence becomes a region decision. Token-level voting alone leaves foreground regions without direct support, so CMPE uses the candidate mask to guide cluster-level prototype extraction and reduce false negatives (Appendix Fig. 6), and because this expansion can introduce false positives, ME requires direct foreground support (Appendix Fig. 7). Each step addresses the error left by the previous one, which the 8-benchmark averages reflect: 1-NN 62.52 → global top-kk 63.48 → with CMPE and ME 64.47. We will make this progression and its rationale explicit in Section 3 of the revision.

[W3] Controlled comparison with INSID3 and simple aggregation baselines

W3: The contribution over INSID3 should be isolated through controlled comparisons, including uniform voting over the same top-k neighbors and per-shot top-k aggregation, and by replacing only the correspondence module while keeping the remaining pipeline unchanged.

  • Answer: Thank you for this suggestion; we performed exactly this study. We disable ME and CMPE, replace only the correspondence module while keeping all other components fixed, and extend the evaluation from the five benchmarks of Table 3 to all eight.

    Note: All variants share our base pipeline, and Variant A is not identical to INSID3. It already includes shot-independent refinements: (i) features are L2-normalized both before and after positional debiasing, so that all similarities (matching, prototypes, and clustering) are cosine similarities in the same normalized debiased space, whereas INSID3 uses original features for clustering and debiased features for matching; (ii) the forward-mask filtering of the official INSID3 implementation is removed, which we found to slightly hurt performance; and (iii) each shot's foreground prototype is normalized before averaging, so that every shot contributes equally regardless of its mask size. These refinements improve Variant A over INSID3† by +3.05 mIoU (62.52 vs. 59.47) with the same clustering threshold (Appendix). We report them transparently; every gain attributed to our method is measured on top of this stronger baseline.

    Variant (5-shot)CorrespondenceAvg mIoU (8 benchmarks)Δ vs. A
    INSID3† (reference)Per-shot backward 1-NN59.47–
    APer-shot backward 1-NN62.52–
    CShot-wise top-kk (5 per shot)62.43−0.09
    DGlobal top-kk, softmax63.48+0.96
    BGlobal top-kk, uniform63.64+1.12
    FullD + CMPE + ME64.47+1.95

    Shot-wise top-kk (C) retrieves 25 tokens per query token but yields no gain, whereas global top-kk (D, B) retrieves only 5 and improves by +0.96–+1.12. Since C and D differ only in whether candidates from different shots compete in a single ranking, the gain comes from integrating evidence across shots rather than from retrieving more tokens. Uniform voting (B) performs comparably, showing that the gain stems from cross-shot integration itself rather than a particular weighting scheme. We adopt softmax weighting so that each neighbor contributes according to its matching confidence, and will extend the comparison of weighting schemes in the revision. CMPE and ME add a further +0.99 mIoU on top of D (64.47). We will add this study to the ablation section.

  • Q1: What technically distinguishes Integra-ICS from EASE and RISE?

    • Answer: Please see [W1]. Unlike EASE/RISE, which retrieve from external or mined libraries, Integra-ICS retrieves from the annotated FG/BG tokens of each episode's support shots, jointly across shots, to recover evidence that per-shot WTA discards.
  • Q2: Does similarity weighting beat uniform voting, and does global retrieval beat per-shot top-k?

    • Answer: Please see the table in [W3] (ME/CMPE disabled, all else fixed). Global retrieval outperforms per-shot top-kk (63.48/63.64 vs. 62.43), whereas similarity weighting and uniform voting are comparable (63.48 vs. 63.64). The gain thus comes from cross-shot integration, not the weighting rule; we will state this explicitly.
  • Q3: Which components are inherited from INSID3, and can evidence integration be isolated?

    • Answer: Inherited: the frozen DINOv3 backbone with positional debiasing, query-token clustering, and cluster-level scoring. New: global top-kk retrieval over a unified FG/BG pool (replacing per-shot 1-NN), CMPE, and ME. Replacing only the correspondence module improves the 8-benchmark average by +0.96–+1.12 mIoU, while shot-wise top-kk does not ([W3]). We will add a component breakdown table.
  • Q4: Does CSCR show genuine cross-shot complementarity, rather than redundancy or simply more tokens?

    • Answer: CSCR shows the presence of cross-shot evidence, not its complementarity. We therefore additionally measure the cosine similarity between FG tokens retrieved from different shots. On X-Ray, CSCR is highest (74.39%), but this evidence is nearly identical (cosine 0.952), and global retrieval has almost no effect (79.71 → 79.78). On the other datasets the evidence is less redundant and global retrieval helps on most of them. Retrieving more tokens alone is not enough: shot-wise top-kk (25 tokens) does not beat 1-NN (62.43 vs. 62.52), whereas global top-kk with 5 tokens does (63.48). We will add the retrieved-cosine column to Table 2.

Thank you for the response. Most of my concerns have been addressed, and I have raised my score to 3.

Reviewer_8UoAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Integra-ICS, a training-free visual in-context segmentation framework that integrates complementary information across multiple support shots through global top-k retrieval and similarity-based aggregation. It also introduces a minimum-evidence mechanism to improve segmentation reliability. Experiments on eight diverse benchmarks demonstrate improvements over existing methods in both 5-shot and 10-shot settings, using a single set of hyperparameters.

Critical strengths
  1. The method achieve good performance across diverse benchmarks. Integra-ICS achieves consistent improvements over existing baselines across eight datasets without additional training or dataset-specific hyperparameter tuning, demonstrating its effectiveness and generalizability.
  2. The method integrates support information through a relatively mechanism without requiring additional training. The overall framework is clearly presented, and the experimental analyses are helpful.
Critical weaknesses
  1. The authors claim in the abstract that the improvements stem from integrating complementary cross-shot evidence, the one-shot results in Table 5 (appendix) show that Integra-ICS already achieves competitive performance across several benchmarks even without cross-shot complementary information. Furthermore, the ablation study only examine minimal evidence(ME) and candidate mask-guided prototype extraction(CMPE) without directly isolating the contribution of the proposed global top-k aggregation mechanism. Additionally, the current experiments do not sufficiently distinguish the benefits of cross-shot aggregation from those of retrieving multiple neighboring tokens within a single support image.
  2. The authors introduce a "cross-shot complementary rescue rate" (CSCR) metric and quantify it for eight datasets in Table 2. This metric provide the presence of complementary evidence, but it remains unclear to what extent such evidence translates into improvements in final segmentation accuracy. The X-Ray dataset exhibits the highest CSCR score yet fails to show a corresponding advantage in segmentation performance. The authors should provide a clearer analysis of the link between the cross-shot evidence and the final segmentation results.
  3. I noticed that the segmentation visualizations for INSID3 and Integra-ICS in the fourth image of Figure 3 appear to be identical; upon zooming in, they are nearly pixel-for-pixel identical. I’m wondering whether these two methods actually produced the same segmentation masks, or if the authors accidentally duplicated the image when creating the figure. If the prediction results are indeed identical, then this example fails to demonstrate the qualitative advantages of Integra-ICS over INSID3 as described by the authors.
Questions for authors
  1. [Additional experiment] Could the authors provide a controlled comparison between global top-k aggregation and shot-wise top-k aggregation to better demonstrate the specific contribution of cross-shot complementary information?

  2. [Clarification] Could the authors further explain how the token-level improvements measured by CSCR translate into improvements in final segmentation performance, particularly for the X-Ray dataset?

  3. [Clarification] In the last row of Figure 3, the segmentation results of INSID3 and Integra-ICS appear to be identical. Could the authors clarify whether the two methods actually produce identical segmentation masks or whether this is an unintended duplication during figure preparation?

4 comments

[W1] Isolating the contribution of global top-k aggregation

W1: The one-shot results in Table 5 (appendix) are already competitive without cross-shot information; the ablation only examines ME and CMPE without isolating global top-k aggregation; and the experiments do not distinguish cross-shot aggregation from retrieving multiple neighboring tokens within a single support image.

  • Answer: Thank you for these three points, which we address in turn.

    One-shot performance. The one-shot results do not rely on cross-shot information; they stem from shot-independent refinements to the base pipeline: (i) all features are L2-normalized both before and after positional debiasing, so that all similarities (matching, prototypes, and clustering) are cosine similarities in the same normalized debiased space, whereas INSID3 uses original features for clustering and debiased features for matching; and (ii) we remove the forward-mask filtering of the official INSID3 implementation, which we found to slightly hurt performance. The one-shot results are therefore not evidence for cross-shot integration; instead, we isolate its contribution on top of these refinements in the controlled study below, and will describe the refinements separately in Section 3 of the revision.

    Isolating global top-kk aggregation. We added a controlled study that disables ME and CMPE and replaces only the correspondence module, keeping all other components fixed, including the refinements above, and extends the evaluation from the five benchmarks of Table 3 to all eight. Variant A uses 1-NN correspondence, but it is not INSID3: it is our base pipeline including the refinements above. INSID3 is reported separately as a reference (INSID3†, evaluated with the same clustering threshold).

    Variant (5-shot)CorrespondenceAvg mIoU (8 benchmarks)Δ vs. A
    INSID3† (reference)Per-shot backward 1-NN59.47–
    A1-NN62.52–
    CShot-wise top-kk (5 per shot)62.43−0.09
    DGlobal top-kk, softmax63.48+0.96
    BGlobal top-kk, uniform63.64+1.12
    FullD + CMPE + ME64.47+1.95

    Cross-shot aggregation vs. multiple tokens within a shot. Shot-wise top-kk (C) retrieves the top-5 neighbors within each shot, i.e., 25 tokens per query token, yet yields no gain over 1-NN (62.43 vs. 62.52). Global top-kk (D) retrieves only 5 tokens per query token but ranks candidates from all shots in a single list, and improves by +0.96. Since C and D differ only in whether candidates from different shots compete in a single ranking, and D uses five times fewer tokens, the gain comes from integrating evidence across shots rather than from retrieving more tokens. That uniform voting (B) performs comparably further shows that it does not depend on a particular weighting scheme. We will add this study to the ablation section of the revision.

[W3] Identical masks of INSID3 and Integra-ICS in Figure 3

W3: The INSID3 and Integra-ICS masks in the fourth image of Figure 3 appear pixel-for-pixel identical; if so, this example fails to demonstrate the qualitative advantage of Integra-ICS.

  • Answer: Thank you for catching this detail. This is not a figure-preparation error: both methods genuinely produce identical masks on this example. Although the two methods differ in how they establish correspondences and score regions, both form the final mask by selecting query regions at patch resolution and upsampling the result without CRF post-processing († in Table 1). This example contains multiple target instances, and both methods capture all of them with the same set of patches, which yields pixel-identical masks after upsampling. While this row was originally chosen to contrast against Matcher and GF-SAM, we agree that it does not illustrate our advantage over INSID3, and we will replace it in the revised Figure 3 with an example where the difference between the two methods is visible.

[W2] Link between CSCR and final segmentation accuracy, particularly on X-Ray

W2: CSCR shows the presence of complementary evidence, but it is unclear to what extent it translates into final segmentation accuracy; X-Ray has the highest CSCR yet shows no corresponding advantage.

  • Answer: Thank you for this comment. CSCR measures the opportunity for cross-shot rescue rather than the gain itself. The Complementary Recovery Rate in Table 2 (Rescue Rate × CSCR) shows that, on average, 4.73% of the 1-NN false-negative tokens are recovered specifically through evidence from other shots. These tokens affect mIoU not by their count alone but through cluster-level scoring (Eq. 10), where a few rescued tokens can change the decision on an entire cluster. How much this helps therefore depends on whether the cross-shot evidence is new. As shown in the table below, X-Ray has by a clear margin the highest similarity both between the foreground evidence retrieved from different shots and between random foreground tokens of different shots, reflecting the homogeneous appearance of lung regions together with the strong self-similarity of DINOv3 features. Its high CSCR (74.39%) thus largely reflects near-duplicate evidence across shots rather than new information, which is consistent with the near-zero effect of global retrieval on X-Ray in our controlled study (Variant A 79.71 vs. B 79.78). X-Ray's large final gain instead comes from the subsequent ME and CMPE stages (Table 3: 79.21 → 86.72), which work well there because its top-1 FN ratio is relatively low (6.02%). On the other datasets, the evidence retrieved from different shots is markedly less similar and thus carries more complementary information, consistent with the retrieval-level gains observed on most of them in our controlled study. We will add this analysis to Section 4.3 and revise its discussion of X-Ray, clarifying that a high CSCR indicates rescue opportunity rather than information gain.

    DatasetCSCR (%)Compl. Recovery Rate (%)Retrieved cos.*Random cos.*
    COCO-20i21.393.510.6450.480
    PASCAL-Part15.161.800.7110.536
    iSAID22.283.100.7850.671
    ISIC20.621.690.7910.672
    LVIS-92i29.663.940.8160.652
    PACO-Part21.972.020.8000.631
    SUIM22.542.460.7060.551
    X-Ray74.3919.280.9520.806

    *Retrieved cos.: mean cosine similarity between foreground support tokens retrieved from different shots for the same rescued query token; Random cos.: mean cosine similarity between randomly sampled foreground tokens from the same pairs of shots, as a control.

Questions

  • Question 1: [Additional experiment] Could the authors provide a controlled comparison between global top-k aggregation and shot-wise top-k aggregation to better demonstrate the specific contribution of cross-shot complementary information?

    • Answer: Yes; please see the controlled study in our response to [W1]. Under identical conditions with ME and CMPE disabled, shot-wise top-kk does not improve over 1-NN (62.43 vs. 62.52) despite using five times more tokens, whereas global top-kk improves by +0.96 (softmax) and +1.12 (uniform voting). Since the two differ only in whether candidates from different shots compete in a single ranking, this difference directly reflects the contribution of cross-shot complementary information.
  • Question 2: [Clarification] Could the authors further explain how the token-level improvements measured by CSCR translate into improvements in final segmentation performance, particularly for the X-Ray dataset?

    • Answer: Please see our response to [W2]. In short, rescued tokens affect mIoU through cluster-level scoring rather than by their count, so the benefit depends on whether the cross-shot evidence is new. On X-Ray, the foreground evidence retrieved from different shots is nearly identical (cosine 0.952), so global retrieval has almost no effect (79.71 → 79.78), and its large final gain comes from ME and CMPE instead.
  • Question 3: [Clarification] Please see our response to [W3].

Dear reviewer,

Thank you again for your careful reviews. As the rebuttal period closes today, we wanted to gently check whether our answers sufficiently address your concerns and questions.

Please let us know if any further clarification would be helpful.

Reviewer_CUkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Integra-ICS, a training-free framework for visual in-context segmentation, which aims to better exploit complementary information distributed across multiple support examples. To solve the limitations that prior approaches largely rely on top-1 correspondence selection, Integra-ICS retrieves global top-k support tokens from a unified foreground-background token pool and aggregates their similarities to construct a foreground evidence map. The resulting candidate mask is further used for query prototype extraction, while a minimum-evidence criterion suppresses predictions without direct foreground support. Query clusters are finally scored using support-level semantic affinity, intra-query contextual affinity, and candidate coverage. Experiments on eight segmentation benchmarks show average improvements of 3.60 and 3.56 mIoU over the strongest baseline in the 5-shot and 10-shot settings, respectively. More analysis on complementary evidence, the number of retrieved neighbors and support shots, component ablations, and inference efficiency proves the effectiveness of proposed methods.

Critical strengths
  • The motivation of the paper is reasonable and meaningful. The author briefly describe the drawbacks of previous methods that ignores or drops the foreground evidence.
  • The method is simple yet effective, which purely relies on foundation model without further training and redundant modules. The efficiency is also remarkable.
  • The evaluation covers most of the in-context segmentation benchmarks, and the experiments are enough for the validation of effectiveness.
  • The methods achieves better results compared to the previous methods, especially the approaches with the same DINOv3 backbone.
Critical weaknesses
  • The current methods closely inherited from INSID3, where the core contribution focuses on the global top-k soft evidence aggregation and lightweight filtering heuristics. It is more likely an incremental design.
  • The benefit of cross-shot complementary evidence is not explicitly isolated by ablation. While the paper shows that top-k aggregation improves over top-1 matching, it remains unclear whether the gain comes from using more support tokens in general, from multiple tokens within the same shot, or specifically from integrating evidence across different support shots.
  • S_i and A_i involved in Eq.10 are cosine similarities, where numbers are in [-1,1], they may have a higher score with two negative similarity values.
  • The comparison in Tab.3 mostly has limited improvements in most of the tasks.
  • Minor typos. E.g., K or k, X-Ray or Lung.
Questions for authors
  • Is your method still effective with other vision backbones, e.g., DINOv2?
  • Experiments comparison in Tab. 3 are mostly dominated by X-Ray tasks. Why this task has a significant difference compared to other tasks?
7 comments

[W1 & W2] Relation to INSID3 and isolating the source of the gain

W1 & W2: The method closely inherits from INSID3 and may be an incremental design (W1), and the cross-shot benefit is not isolated, i.e., whether the gain comes from (i) more support tokens in general, (ii) multiple tokens within the same shot, or (iii) integrating evidence across different shots (W2).

  • Answer: We address both with a single controlled study.

    Relation to INSID3. We agree that Integra-ICS inherits much of INSID3's high-level design, namely frozen DINOv3 features with positional debiasing, query clustering, and cluster-level scoring. However, our starting point is a problem that INSID3 does not address: in the multi-shot setting, winner-take-all correspondence keeps only the single best match per query token and discards foreground evidence distributed across the other support shots, which we term the multi-shot information bottleneck. Global top-k aggregation over a unified FG/BG support pool is directly motivated by this observation.

    Controlled study. To isolate the gain from the top-k aggregation, we disable ME and CMPE and replace only the correspondence module while keeping all other components fixed, including the refinements described in the note below, and extend the evaluation from the five benchmarks of Table 3 to all eight (table below). Note: *Variant A is not identical to INSID3. Variant A is our base pipeline with 1-NN correspondence, and it already differs from INSID3 through shot-independent refinements: (i) all features are L2-normalized both before and after positional debiasing, so that all similarities (matching, prototypes, and clustering) are cosine similarities in the same normalized debiased space, whereas INSID3 uses original features for clustering and debiased features for matching; and (ii) we remove the forward-mask filtering of the official INSID3 implementation, which we found to slightly hurt performance. For multiple shots, we additionally normalize each shot's foreground prototype before averaging, so that every shot contributes equally regardless of its mask size. Together, these refinements improve Variant A over INSID3† by +3.05 mIoU on average under 5-shot (62.52 vs. 59.47), with the same clustering threshold (Appendix). We report them transparently; every gain attributed to our method is measured on top of this stronger baseline.

    C retrieves the top-5 neighbors within each shot, i.e., 25 tokens per query token in the 5-shot setting. D (ours) and B retrieve only 5 tokens per query token but rank candidates from all shots in a single list, weighted by softmax (D) or by uniform voting (B).

    • (i) More support tokens in general and (ii) multiple tokens within the same shot are both tested by C, which yields no gain over 1-NN (62.43 vs. 62.52, 8-benchmark average).
    • (iii) Integrating evidence across different shots is tested by D and B, which improve over 1-NN by +0.96 and +1.12 mIoU, respectively.

    Since D uses five times fewer tokens than C, and the two differ only in whether candidates from different shots compete in a single ranking, the gain cannot be attributed to using more tokens or multiple tokens within a shot, but to integrating evidence across shots. That uniform voting (B) performs comparably further shows that the gain stems from cross-shot integration itself rather than from a particular weighting scheme. We adopt similarity-weighted (softmax) aggregation in Integra-ICS so that each retrieved neighbor contributes according to its matching confidence; given the comparable results of the two schemes, we will extend this comparison with additional experiments on the weighting scheme and include it in the ablation study of the revision.

    Variant (5-shot)CorrespondenceAvg mIoU (8 benchmarks)Δ vs. A
    INSID3† (reference)Per-shot backward 1-NN59.47–
    A1-NN62.52–
    CShot-wise top-k (5 per shot)62.43−0.09
    DGlobal top-k, softmax63.48+0.96
    BGlobal top-k, uniform63.64+1.12
    FullD + CMPE + ME64.47+1.95

    Role of ME and CMPE. ME and CMPE then target the two error types that remain after integration, rather than serving as generic heuristics: CMPE reduces false negatives by guiding prototype extraction with the candidate mask (Appendix Fig. 6), and ME suppresses false positives from cluster over-expansion by requiring direct foreground support (Appendix Fig. 7), adding a further +0.99 mIoU (64.47).

    In summary, while we build on INSID3's high-level design, our contribution lies in identifying and quantifying the multi-shot information bottleneck and resolving it under a single universal configuration, which we believe goes beyond an incremental design. We will add this study to the ablation section and clarify our positioning in the revision.

[W3] Sign of the multiplicative score in Eq. (10)

W3: SiS_i and AiA_i involved in Eq.10 are cosine similarities, where numbers are in [-1,1], they may have a higher score with two negative similarity values.

  • Answer: The reviewer makes a valid mathematical point: because cosine similarity spans [−1,1][-1, 1], Eq. (10) as written could in principle yield a positive product from two negative values. In practice, negative values of SiS_i or AiA_i do occur for some clusters, but their magnitudes are small, so the product of two negative values stays far below the aggregation threshold, and we have not observed a case where such a product exceeds it. Since Ri≤1R_i \le 1, the final score is reduced further. Notably, INSID3 adopts the exact same multiplicative scoring (skcross⋅skintras^{\text{cross}}_k \cdot s^{\text{intra}}_k) without sign clipping. Nevertheless, we agree that a more rigorous formulation is to explicitly apply max⁡(⋅,0)\max(\cdot, 0), and we will revise Eq. (10) to Scorei=max⁡(Si,0)×max⁡(Ai,0)×Ri\text{Score}_i = \max(S_i, 0) \times \max(A_i, 0) \times R_i in the manuscript. Because a cluster with a negative factor never passes the threshold in our experiments, this revision leaves all reported results unchanged.

[W4] Limited improvements in Table 3

W4: The comparison in Tab.3 mostly has limited improvements in most of the tasks.

  • Answer: While we agree that the gain in Table 3 appears concentrated on X-Ray, ME and CMPE are auxiliary safeguard and refinement modules on top of our core global top-kk aggregation (which already achieves 67.33 avg. mIoU over the five benchmarks of Table 3 without them, outperforming INSID3 at 61.34 and RSRM at 65.41), and they still yield clear gains on three of the five benchmarks (X-Ray +8.14, PASCAL-Part +1.05, and ISIC +0.81). Specifically, ME serves as a precision safeguard against semantic drift during clustering, which is particularly severe in homogeneous chest radiographs (X-Ray), whereas on complex scenes (COCO-20i, iSAID) its net gain appears small because strict filtering occasionally trims valid cluster expansions lacking direct top-kk support, as shown in Appendix Figure 7. Meanwhile, Appendix Figure 6 shows that CMPE consistently reduces false negatives (ΔTP>ΔFP\Delta\text{TP} > \Delta\text{FP}) across datasets and shot settings. Together, CMPE (reducing FNs) and ME (suppressing FPs) work complementarily—turning the unguarded CMPE variant (67.04) into a +1.53 mIoU average improvement (68.86) over the base aggregation (67.33) under a single universal configuration. Why X-Ray differs so markedly from the other benchmarks is detailed in our answer to Question 2.

[W5] Minor typos

W5: Minor typos. E.g., K or k, X-Ray or Lung.

  • Answer: Thank you for pointing out these typos. We will correct them in the revision.

Questions

  • Question 1: Is your method still effective with other vision backbones, e.g., DINOv2?

    • We re-ran both INSID3 and Integra-ICS with a DINOv2 ViT-L/14 (reg4) backbone on COCO-20^i (5-shot, 4 folds) under an identical protocol (same resolution, seed, and episodes). Integra-ICS achieves 41.85 mIoU vs. 40.02 for INSID3 (+1.8 mIoU), and improves on every fold (+1.5 / +2.3 / +1.0 / +2.5). This indicates that the gains of our framework are not tied to a specific backbone but stem from the proposed components themselves. We will add this result (with per-fold numbers) to the revised manuscript.
  • Question 2: Experiments comparison in Tab. 3 are mostly dominated by X-Ray tasks. Why this task has a significant difference compared to other tasks?

    • Answer: As shown in Table 3, the large difference on X-Ray (+8.14 mIoU) stems from the decisive role of ME. By definition (Eq. 4), ME is a relaxed criterion requiring at least one foreground token among the top-KK retrieved neighbors (Vq≥1V_q \ge 1), designed to suppress false positives (FPs) when semantic clustering over-expands into visually similar background. Following our response above, ME has a trade-off: besides suppressing FPs, it can also discard true foreground query tokens that were correctly grouped by clustering despite lacking direct top-KK foreground support, which causes the marginal negative side effect observed on complex dataset such as COCO-20i (Appendix Figure 7). The reason X-Ray behaves so differently is explained by Table 2: because lung regions are visually consistent across radiographs, true foreground tokens almost always receive direct foreground support (X-Ray has by far the lowest Top-1 FN ratio at 6.02%, vs. 18.52%–29.45% on the other datasets), meaning ME rarely suffers from this false-negative side effect on X-Ray while still pruning large background cluster expansions, resulting in a substantial net gain.

Thank you for the detailed response and the additional controlled experiments. The comparison between shot-wise and global top-k aggregation substantially addresses my concern.

However, the new results also reveal that the majority of the improvement over INSID3 comes from the additional pipeline refinements in Variant A (+3.05 mIoU), rather than the proposed global top-k aggregation (+0.96 mIoU) that is nominated as the main contribution. Besides, uniform voting still has limited improvements compared to softmax aggregation. Could you also report the results of 1-NN with ME and CMPE, to clarify whether these components provide similar gains independently of global top-k aggregation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.