acceptodds
Withdrawn from ICLR 2027

DINO-KNN: Training-Free Multiclass In-Context Segmentation via Nearest Neighbors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In-context segmentation (ICS) segments a target image using one or more annotated reference images. Existing approaches often require task-specific training, combine multiple pretrained models, or rely on multi-stage inference pipelines. Several methods predict one foreground concept at a time. Applying these methods to multiclass segmentation requires combining independently predicted masks, which may assign conflicting labels to the same pixel. We ask whether direct classification of frozen visual features is sufficient for accurate multiclass ICS. We introduce DINO-KNN, a training-free approach that applies \(k\)-NN classification to frozen DINOv3 features without a learned segmentation decoder. Given a single reference image annotated with one or more foreground concepts, DINO-KNN classifies target patches using labeled reference features. Given classes compete within a single multiclass decision, producing mutually exclusive predictions without separate per-class segmentation or post-hoc conflict resolution as in state-of-the-art ICS methods. On multiclass episodes from seven datasets spanning natural images, aerial imagery, underwater scenes, and medical images, DINO-KNN achieves the highest average mIoU among the evaluated SOTA methods, surpassing INSID3, the strongest evaluated baseline, by \(6\) mIoU points. DINO-KNN assigns exactly one semantic label to each pixel. For INSID3, the average percentage of predicted foreground pixels assigned to more than one class is \(42%\). These results establish direct \(k\)-NN classification over frozen DINOv3 features as a simple yet powerful method for training-free multiclass ICS. The code and full setup will be made public.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
In-Context Segmentation, Multiclass Segmentation, One-Shot Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_u1VsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents DINO-KNN, a training-free method for one-shot multiclass in-context segmentation based on frozen DINOv3 features and nearest-neighbor classification. The method adopts the positional debiasing technique from INSID3, constructs a labeled feature bank from a single reference image, and assigns each target patch a semantic label through inverse distance weighted k-NN voting. DINO-KNN also jointly predicts multiple foreground classes and background without the learned segmentation decoder and class-wise inference.

Critical strengths

The proposed method requires no task-specific training, additional segmentation decoder, or auxiliary foundation model. Its straightforward design makes it easy to implement and potentially useful as a practical baseline for in-context segmentation.

The experiments cover seven datasets from different visual domains. The results demonstrate that simple feature-based classification can be competitive with more elaborate training-free segmentation pipelines.

DINO-KNN substantially outperforms INSID3 on CAMUS, FASS, and PSFHS. These results suggest that direct feature matching can be preferable to clustering-based segmentation in certain domains.

Critical weaknesses

The main method is based on standard k-NN classification over frozen DINOv3 features, with positional debiasing adopted from INSID3. The main contribution appears to be a simpler application of existing techniques rather than a substantially new method. Therefore, I believe this paper lacks novelty.

Section 4.1 in this paper states that the foreground label set contains only the classes shared by the reference and target images. This raises a concern about whether target ground-truth information is used to construct the evaluation episodes. In a practical one-shot ICS setting, the classes present in the target image are not known beforehand.

The zero CCR is guaranteed by assigning a single label to each patch. Moreover, DINO-KNN-Binary already achieves a CCR of approximately 1% and nearly identical mIoU. Therefore, it is unclear how much the proposed joint multiclass prediction contributes beyond enforcing mutually exclusive labels.

Questions for authors

How is the foreground label set determined for each evaluation episode? Does the construction require access to the target annotations, and how would the results change if all classes in the reference annotation were retained?

Can the authors compare DINO-KNN with INSID3 using a post-hoc conflict resolution strategy? This would help clarify whether joint multiclass prediction offers an advantage beyond eliminating overlapping predictions.

DINO-KNN underperforms INSID3 on COCO and LVIS, while most of the average improvement comes from the ultrasound datasets. What explains this difference? How sensitive are the results to the choice of reference image?

Reviewer_LFujOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed DINK-KNN, a training-free in-context segmentation framework that produces multi-class segmentation masks with a non-parametric classification module on top of pretrained-DINOv3 backbones. The overall idea follows INSID but extends its capability to generate mutually exclusive semantic predictions from a single annotated reference image. Experiments on several datasets ranging from natural images to medical images show the overall performance gain of proposed method.

Critical strengths

1.DINO-KNN presents a simple yet effective extension of INSID to multi-class segmentation. Instead of performing multi-pass binary foreground-background segmentation with potential overlapping labels, DINO-KNN adopt similarity-driven KNN voting to acquire single class labels per target image patch without much computational overhead.

2.Experimental analysis are conducted on several heterogeneous datasets, of which the advantages of DINO-KNN standout in medical imaging data, while underporm INSID on natural images.

Critical weaknesses

1.Reference–target construction. It is mentioned that ‘For each evaluation episode, the foreground label set consists of the semantic classes shared by the reference and target images. All other categories, including those present in only one of the two images, are treated as background. All evaluated methods use exactly the same reference images, target images, and semantic classes.’ I assume this is a relatively strict assumption asking for strong similarity between reference-target pairs and the final evaluation happens only on co-exist classes. I am wondering if the refined DINOV3 features could hold effectiveness when all annotated reference categories remain valid target classes, ,regardless of whether they actually appear in the target images. Will there be false positive segmentation?

2.It is acknowledged that DINO-KNN could achieves multi-class segmentation in one-shot compared to multiple inferences of binary ICS methods. Still, i am curious what is the performance when fusing multiple binary-ICS segmentation in a post-doc heuristic manner to see the advantages of proposed K-NN strategy.

3.The reported performance improvement mainly comes from medical datasets, while on natural-images (COCO,LVIS) the performance remains inferior to the strongest baseline. More in-depth analysis are expected to explain such characteristics.

4.Some popular fine-grained part-level datasets like PASCAL and PACO included in INSID are missing to show the advantages of DINO-KNN on small objects/parts. As all target image features are cached in a global pool, there may exist trade-off between searching neighbour areas and fine-grained precision. More explanations are expected.

Questions for authors

Please see the comments above. Especially,

  • what is the performance when fusing multiple binary-ICS segmentation in a post-doc heuristic manner to see the advantages of proposed K-NN strategy.

  • what is the performance when all annotated reference categories remain valid target classes, regardless of whether they actually appear in the target images

  • More in-depth analysis on the fact that the performance improvement mainly comes from medical datasets, while on natural-images (COCO,LVIS) the performance remains inferior to the strongest baseline.

Reviewer_ETHgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DINO-KNN, a training-free one-shot multiclass in-context segmentation method that applies k-NN classification directly to frozen DINOv3 patch features. Given one annotated reference image, it builds a class-wise reference feature bank, retrieves the k nearest reference patches for each target patch across all classes, aggregates inverse-distance votes, and takes the argmax over classes to produce a mutually exclusive multiclass mask. It performs well on several datasets.

Critical strengths
  1. The motivation is clear. The observation that training-free ICS methods are fundamentally per-class binary predictors requiring post-hoc merging, and can assign conflicting labels to the same pixel, is a legitimate and well-articulated gap.

  2. The method is clean and reproducible. It reduces to “positional debiasing (borrowed from INSID3) + k-NN voting”. The formulation is crisp, and the ablations (prototype vs. 1-NN vs. k-NN; sensitivity to k) and the efficiency comparison are carefully designed.

Critical weaknesses
  1. Algorithmic novelty is very thin. The core is k-NN classification on frozen DINOv3 features, a classic non-parametric baseline. The closest prior work, Hummingbird, is discussed in related work but not compared experimentally, and FROST is likewise mentioned but not evaluated. Omitting these closest baselines undermines the novelty claim.

  2. The headline claim of “highest average mIoU” is a consequence of averaging over heterogeneous domains. The 6-point average improvement is driven almost entirely by the three ultrasound datasets, while DINO-KNN loses on the two most standard and competitive benchmarks (−6 on COCO, −8 on LVIS). Moreover, the ultrasound absolute values are extremely low for both DINO-KNN and the baselines, so the “improvement” is from near-failure to still poor performance.

  3. The “42% conflict” motivation is similarly domain-dominated. The CCR of INSID3 is 2, 4, 4, and 6 on COCO, LVIS, SUIM, and iSAID, respectively, but 100, 83, and 96 on the three ultrasound sets, so the conflict problem emphasized in the introduction is marginal on mainstream datasets.

  4. Learning-based comparisons and scale are missing. The abstract’s claim of “highest average mIoU among the evaluated SOTA methods” should be qualified as referring only to training-free methods.

Questions for authors
  1. Why are Hummingbird and FROST absent from the experiments? Please evaluate them under the same single-reference protocol; this will largely determine whether the contribution holds up against the closest prior work.

  2. Please report per-domain-normalized/unweighted results and the average mIoU excluding the three ultrasound datasets.

  3. Given the quadratic scaling of brute-force k-NN, please provide a scalability analysis at higher resolution (e.g., 2048²) or with larger reference sets.

Reviewer_paWgOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DINO-KNN, a training-free approach for one-shot multiclass in-context segmentation. The method uses frozen DINOv3 features with positional debiasing and applies k-nearest-neighbor classification to predict target patch labels directly from an annotated reference image. By jointly classifying foreground and background categories, it avoids separate binary predictions and cross-class conflicts. Experiments on seven datasets show an average improvement of 6 mIoU points over INSID3, along with improved inference efficiency in the multiclass setting.

Critical strengths

1.The approach is simple and easy to understand. It removes several intermediate stages required by INSID3 and does not introduce additional trainable parameters. This makes the method relatively easy to reproduce and potentially useful as a lightweight baseline.

2.The paper addresses a relevant practical limitation of binary in-context segmentation. Joint prediction naturally avoids assigning multiple foreground labels to the same pixel. The reduction in repeated inference is also useful when multiple classes must be segmented.

3.The cross-domain evaluation is a positive aspect. In particular, the improvements on the three ultrasound datasets are interesting. The comparison among prototype matching, 1-NN, and k-NN also provides some evidence for retaining individual reference features rather than representing each class with a single prototype.

Critical weaknesses
  1. Limited methodological novelty. The proposed method mainly combines existing components, including frozen DINOv3 features, positional debiasing, and standard k-NN classification. Moreover, the joint multiclass formulation achieves the same average mIoU as DINO-KNN-Binary, making its additional contribution less convincing.

  2. Performance gains are not consistent across datasets. The reported 6-point average mIoU improvement over INSID3 is largely driven by the ultrasound datasets, while DINO-KNN performs worse on COCO and LVIS. The paper lacks sufficient analysis of these substantial cross-domain differences.

  3. Potential limitations in the evaluation protocol. The foreground classes are defined using categories shared by the reference and target images, which assumes knowledge of the target's class presence. It remains unclear how the method performs when reference classes are absent from the target image.

  4. Insufficiently controlled baseline comparisons. The zero CCR is guaranteed by the joint classification design, but the paper does not compare against binary baselines with conflict resolution. Similarly, the reported 2.8× speedup may partly result from repeated feature extraction in the baselines, making the efficiency comparison less conclusive.

Questions for authors
  1. How exactly is DINO-KNN-Binary constructed? When predicting one foreground class, are patches from the other foreground classes relabeled as background or removed from the reference bank? Please also provide the unrounded average mIoU for the binary and joint variants so that the size of their difference is clear.

  2. Does the INSID3 runtime in Table 2 reuse image features across foreground classes? Could the authors provide a comparison using shared features and a documented conflict-resolution rule on a representative dataset?

  3. How were the ultrasound reference images selected, and are reference and target images subject-disjoint? Does the improvement persist when a few alternative valid references are used on one of these datasets?

  4. How are reference-only classes handled in the current implementation? What happens when the candidate label set is fixed by the reference annotation, including classes absent from the target?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.