acceptodds
Withdrawn from ICLR 2027

Rethinking Over-Segmentation in Few-Shot Object Detection with Vision Foundation Models: A Training-Free Prototype Refinement and Occlusion-Aware Discrimination Approach

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot object detection pipelines built on frozen vision foundation models achieve strong cross-domain performance without task-specific training, yet still suffer from over-segmentation: a single object is fragmented into multiple proposals, and non-maximum suppression can discard the correct whole-object proposal in favour of a higher-confidence fragment. Our diagnostic study shows that this failure mode differs qualitatively from classical over-segmentation: proposal coverage is rarely the bottleneck; instead, classification confidence becomes unreliable in the few-shot regime, where biased support sets shift the class prototype toward part-views. Building on this insight, we propose two training-free, complementary modules. Training-free Prototype Refinement analytically performs a one-step contrastive prototype update against an adjacency set of negative samples, bounded by a trust region derived from the support set. Occlusion-Aware Discrimination distinguishes whole-object proposals from fragments by their confidence decline rate under partial occlusion, and additionally retains genuine small objects that NMS would otherwise delete. On eight FSOD benchmarks, our method consistently improves over VFM-based detectors while adding no trainable parameters and only modest inference overhead.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Few-Shot Object Detection, Frozen Vision Foundation Models, Over-segmentation, Training-free Methods, Prototype Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UdqiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose POISE, a training-free solution designed to address over-segmentation in few-shot object detection (FSOD) pipelines built from frozen vision foundation models. The method operates in two phases: first, repelling per-class prototypes from retrieved hard negatives within a trust region via Training-free Prototype Refinement (TPR); second, adjudicating nested same-class proposals using confidence decline under structured partial occlusion via Occlusion-Aware Discrimination (OAD). To protect genuine small objects from being falsely eliminated, the authors leverage this structured occlusion to evaluate confidence variations. Additionally, the paper evaluates POISE across multiple FSOD and CD-FSOD benchmarks and complete splits, demonstrating consistent improvements over recent VFM-based baselines, particularly in 1-shot and small-object regimes.

Critical strengths
  1. The paper addresses a relevant problem: mitigating over-segmentation in few-shot object detection (FSOD) pipelines built from frozen vision foundation models (VFMs) without requiring additional training. This motivation is clearly presented and justified by targeted diagnostics, reframing over-segmentation as a proposal selection issue rather than merely a bounding-box coverage problem.

  2. The proposed POISE framework is modular and well organized. The formulation is easy to follow. The paper first introduces Training-free Prototype Refinement (TPR) to repel class prototypes from hard negatives within a defined trust region, and then employs Occlusion-Aware Discrimination (OAD) to adjudicate nested proposals using occlusion-decline asymmetry. The inclusion of mathematical expressions for the trust-region projection, a concise OAD scoring formula, and algorithmic pseudocode makes the overall workflow—repairing the confidence signal and then the selection rule—easy to understand.

  3. The paper is well written and easy to read. The introduction provides a clear problem setup with intuitive figures, and the experimental section demonstrates the method's performance across Pascal-5i, COCO-20i, and six CD-FSOD target domains. Ablations and controlled comparisons isolate the contributions of TPR and OAD, demonstrating improvements, particularly in the 1-shot and small-object regimes.

Critical weaknesses
  1. TPR updates a class prototype by moving it away from the mean of the retrieved hard negatives. However, increasing the distance from negative samples does not necessarily make the updated prototype more representative of the target class. When the support set is small or biased, the update may also move the prototype away from valid positive examples that are not covered by the support set. Although the trust region limits the update magnitude, it does not verify whether the update direction is suitable.

  2. TPR retrieves a fixed number of nearest negative samples even when all other classes are already well separated from the target class. In this case, the retrieved samples may not be genuinely confusing, but they still produce a nonzero update because the update magnitude is not explicitly controlled by their absolute similarity to the prototype. This may unnecessarily move an already suitable prototype, particularly when unrelated negative features produce an unreliable mean direction.

  3. OAD computes the relative confidence decline as $ \Delta(b)=\frac{1}{T}\sum_{t=1}^{T} \frac{s^{(0)}(b)-s^{(t)}(b)}{s^{(0)}(b)}. $ Dividing by the unoccluded confidence �0��0� may amplify small fluctuations for low-confidence proposals while reducing the effect of comparable absolute changes for high-confidence proposals. Since cosine similarity can approach zero or become negative, this ratio may also become unstable or have an unexpected sign unless score filtering or denominator protection is applied.

Questions for authors

Please see the weaknesses.

Reviewer_A6hQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the over-segmentation problem in few-shot object detection (FSOD) using frozen vision foundation models (VFMs). The authors highlight the issue of over-segmentation, where a proposal containing only a part of the object may receive a higher confidence score than one containing the entire object and be incorrectly selected by NMS. To address this issue, the authors propose POISE, which consists of two training-free modules, Training-free Prototype Refinement (TPR) and Occlusion-Aware Discrimination (OAD). TPR is designed to improve classification confidence by pushing class prototypes away from hard negatives retrieved from the memory bank. Furthermore, OAD distinguishes between whole-object proposals and fragments by leveraging the drop in confidence observed when partial occlusion is applied to a proposal, thereby mitigating the issue of small, genuine objects being discarded. POISE achieves improved performance over FSOD-VFM and CD-ViTO on eight FSOD benchmarks, and the authors analyze each module's contribution to overall performance through module-level ablations and comparisons with alternative selection rules.

Critical strengths
  • Consistent performance gains. The paper reports improvements over FSOD-VFM and CD-ViTO across eight benchmarks, particularly in low-shot settings. Component ablations also show benefits from both TPR and OAD.

  • Useful diagnostic analysis. The paper examines proposal coverage and confidence ranking separately, showing that whole-object proposals can be suppressed in favor of fragments. This identifies a concrete failure mode in frozen VFM-based FSOD and motivates improving proposal selection.

Critical weaknesses
  • Inconsistencies between the stated method and reported results. Under the default POISE settings (η=0.1\eta=0.1), since both (p^c\hat p_c) and (a^c\hat a_c) are unit vectors, the L2 norm of the TPR update before projection cannot exceed 0.2. Therefore, assuming all other settings remain fixed, the same prototype should be generated for δ\delta=0.3, 0.5, and 0.7 in the 1-shot setting. However, the accuracies reported in Table 16(b) are not the same. For Pascal-5i's 1-shot, Split 1, seed 33, Table 16(d-e) shows an nAP50 value of 80.1 under the default POISE settings, whereas Table 25 presents a value of 77.9 under the same conditions. However, the authors do not address the reason for these numerical differences. Furthermore, regarding the COCO-20i 30-shot setting in Table 4, while the paper reports a standard deviation of 0.3–0.4 per split, it reports a standard deviation of 0.8 for the average performance across the four splits. These standard deviations are mathematically inconsistent.

  • Insufficient evidence that TPR corrects the diagnosed failure mode. While TPR improves AP scores, it is unclear whether this improvement stems from mitigating the over-segmentation problem claimed in the paper. Since TPR uses negatives from other classes, the gains could instead come from better class discrimination. Since most of the performance gains are due to TPR (Tables 5-6), direct evidence that TPR helps select proposals covering the entire object rather than its fragments is needed to attribute these gains.

  • Unclear advantage over backbone scaling. The authors show that stronger backbones mitigate the problem of over-segmentation (Table 9(b)). While POISE enhances AP even with a fixed backbone, it is still unclear whether it is better to allocate additional computational resources to POISE's modules or to a more powerful backbone.

Questions for authors

Why do (\delta=0.3,0.5,0.7) produce different results in Table 16(b) when none should constrain the TPR update?

What explains the different results for seed 33 in Tables 16(d–e) and 25?

Why does Table 4 report a larger standard deviation for the four-split mean than for any individual split at 30-shot?

How does moving a class prototype away from negatives of other classes favor proposals covering the whole object over fragments of the same class?

What evidence links TPR's AP gains to reduced over-segmentation?

What advantage does POISE offer over using a stronger backbone alone when both accuracy and computational cost are considered?

Reviewer_ubQwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that over-segmentation in VFM-based few-shot object detection is a selection problem rather than a coverage problem. It introduces two training-free modules: TPR retrieves hard negatives from a frozen memory bank to adjust class prototypes within a trust region, while OAD partially occludes nested, same-class proposal pairs and uses the drop in confidence to distinguish whole objects from fragments. The combined method, POISE, is compared with FSOD-VFM and CD-ViTO on CD-FSOD, Pascal-5i, COCO-20i, and other benchmarks. However, I have some concerns about the assumptions, how the proposed modules work, whether the evidence supports the claims, etc.

Critical strengths
  1. The authors argue that proposal coverage is usually sufficient. The problem is that part-view support biases few-shot prototypes, making confidence scores unreliable and thus causing NMS to discard whole-object proposals. F1-F4 largely follow this argument. This diagnosis provides some basises of the proposed method - TPR and OAD.

  2. The control experiments suggest that the gains are not simply due to perturbing the prototypes or relaxing suppression. In Table 7, Soft-NMS offers little benefit, keep-largest hurts performance on DIOR-small, and random prototype perturbations with the same step size do not help. TPR’s hard-negative direction and OAD’s occlusion signal provide a good performance.

Critical weaknesses
  1. The paper attributes over-segmentation mainly to support bias, but the proposed modules do not clearly based on on that mechanism. TPR pushes away from external hard negatives, and OAD compares confidence difference under occlusion, and that difference is itself based on box size and occlusion coverage. As a result, it is hard to tell whether the gains come from the claimed mechanism or from two post-processing steps that happen to work.

    (1) The paper argues that support bias pulls the prototype toward part-level views. However, TPR corrects the prototype using inter-class hard negatives, not by addressing the intra-class coverage bias in the support set. The paper does not show that pushing away from external negatives is equivalent to correcting this internal support bias. The ablations in Table 7 rule out “random perturbation” and “relaxed suppression,” but they do not rule out a basic way based on the diagnosis. The mechanism is mis-specified, even if the resulting operation is empirically helpful.
    (2) OAD has a similar issue. Its decision statistic is entangled with box size, occlusion coverage, and the geometric asymmetry of nested proposals. A smaller box will naturally lose a larger fraction of its visible evidence when 1/4 is occluded, so its confidence can drop more sharply even without reflecting the whole object. Without a size-matched control or normalization by occluded area, the 89.4% decision accuracy in Table 12 does not rule out the possibility that OAD is largely exploiting box-size effects rather than the proposed completeness signal.
    (3) F1 uses the IoU between the largest proposal and the ground truth to argue that the proposal set contains sufficiently complete masks, which implicitly treats largest as the whole object. F4 later acknowledges that the smaller member of a nested pair can sometimes be a genuine distinct object. This weakens the diagnosis because the evaluation partly relies on the same assumption it is supposed to test.
    (4) The paper presents TPR and OAD as two complementary repairs. However, the results do not really support this balanced view. In Table 5, combining OAD with TPR brings only +0.3 gain. In addition, the full method also has larger variance than either component alone. This suggests that most of the improvement comes from TPR rather than from the proposed two-part design. As a result, the method looks like a prototype refinement based on negative repulsion, and its difference from existing prototype negative-repulsion methods is not clear.

  2. Several presentations are unreadable or insufficiently specified. It prevents the reader from evaluating whether the paper’s claim (diagnosis plus training-free repair) is actually supported.

    (1) Findings F1–F4 seem to rely on Table 9 as their most important evidence, but this table is missing from the main text (this table is appeared at page 14). Even if it is placed in the appendix, putting the central evidence for these findings outside the main text is a serious presentation problem.
    (2) Isn’t p~c−p^c\tilde{p}_c - \hat{p}_c simply equal to γ(pc−−a^c)\gamma(p_c^- - \hat{a}_c) in Eq. (1)? Why redefine p~c\tilde{p}_c? \

Questions for authors

See Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.