acceptodds
Withdrawn from ICLR 2027

When Do Local Features Help? Patch-Area Matching in Fine-Grained Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive vision–language models (VLMs) align a single global image embedding with text, capturing whole-image structure but discarding local detail that fine-grained recognition depends on. Frozen patch features retain localizable structure that text-based scoring misses, while gradient-based adaptation risks eroding transferable features. We propose GL-VLM, a training-free, few-shot method that keeps the backbone frozen and augments the global embedding with a local "patch-area" descriptor: per class, we cluster the support patches into region prototypes and score a query by how its patches fall into each class's areas, combined with a global prototype. Stacked on a strong Tip-Adapter, it improves texture/part/scene categories (mean +1.4 points over 15 backbones, up to +5.4 at k=16) and never hurts (a β=0 fallback). The benefit is interpretable and stated a priori: local patch-areas help where distinctions are textural, part-based, or scene-level, and are neutral where text is near-perfect or the cue is global shape text priors and local structure are complementary, and which wins is predictable from the category. Because the backbone is never updated, GL-VLM preserves zero-shot transfer exactly, at zero training cost. We do not claim state of the art covariance-based methods match or exceed it but characterize when local features help.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision-language models, few-shot learning, fine-grained recognition, training-free adaptation, patch features, open-set recognition.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NahkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free few-shot classification method that clusters frozen VLM patch features into class-specific prototypes and combines local patch matching with global image similarity. The authors find that local features mainly improve texture-based recognition.

Critical strengths
  • A training-free and straightforward method that might apply to many models.
  • Extensive evaluation across 15 backbones.
  • Some improvements on texture-based datasets.
Critical weaknesses

Generally, I don't like to comment on novelty, but I think this paper truly has "limited novelty", as the method mainly combines existing techniques, and the necessity of k-means clustering is unclear.

Regarding the method, I'm concerned that the background introduces bias, and I hope the authors will explore this. Averaging all patch similarities may introduce background correlations rather than capturing discriminative object features.

Furthermore, I think the authors' approach isn't limited to VLM. After all, it's already a few shots; why use VLM instead of a pure vision encoder (such as the Dino family)? Therefore, I believe the paper lacks comparisons with direct patch-to-patch matching and non-VLM visual encoders.

Last, the paper has poor presentation. The paper is repetitive and poorly organized, with an overly dense abstract, limited methodological explanation, and unclear distinctions between experimental protocols.

Questions for authors

Since the method relies on labeled few-shot examples rather than text supervision, why is a VLM necessary? How would it compare with DINOv3 under the same setting?

How does the method compare with direct patch-to-patch matching without k-means?

Have the authors investigated whether background patches affect the improvements?

What is the additional inference latency and computational cost compared with existing training-free baselines?

Reviewer_FEJdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when frozen vision–language model patch features improve few-shot fine-grained classification. It proposes GL-VLM, a gradient-free add-on that clusters labeled support-image patch embeddings into M class-specific prototypes and computes a late-interaction query score by averaging the maximum patch-to-prototype similarities. This local score is combined with either a global support prototype or Tip-Adapter using a validation-selected fusion weight β, while the backbone remains frozen. Across seven datasets, the authors report gains for texture- and scene-oriented recognition than for shape-dominant or readily nameable categories.

Critical strengths

The proposed method integrates local patch features into frozen VLMs without gradient-based training, making it straightforward to implement. The paper evaluates multiple datasets and 15 backbones, providing useful observations about when local features benefit fine-grained recognition.

Critical weaknesses

The proposed approach combines class-wise patch clustering with late-interaction matching, both of which build on established techniques. More importantly, Table 4 shows that using a single prototype M=1 already improves DTD accuracy from 0.718 to 0.748, compared with 0.767 for M=8. This suggests that a substantial portion of the improvement does not require multiple patch-area prototypes or a nontrivial max-over-regions operation. Consequently, the incremental contribution of the proposed patch-area mechanism remains insufficiently established.

Table 1 shows that GL-VLM underperforms several existing methods, including GDA and Tip-Adapter-F, on most reported benchmarks. Furthermore, Section 6 explicitly acknowledges that the proposed cue-type prediction is supported on a little datasets. The primary evaluation also selects the fusion weight using 16 additional labeled validation images per class, making its applicability under a strict K-shot label budget less directly established.

The paper's organization could be more concise. The current nine-section structure is somewhat fragmented, with several sections discussing overlapping findings.

Questions for authors

For M=1, Equation (3) reduces algebraically to matching the mean query patch feature to one class patch centroid. precisely how is the “mean-pool” ablation in Table 4 implemented? Please report a matched baseline that averages exactly the same dense patch tokens on both query and support sides, and explain the M=1 versus mean-pool discrepancy.

Please reconcile the differences between the gains implied by Table 1 and those in Table 2 at k=16, especially DTD (+8.2 vs +5.3 points) and Flowers (+3.8 vs +0.6). Which figures use full test sets, distinct support draws, or episode-level tests? What is the primary headline effect size under one consistently specified protocol?

Could you provide the full strict k-shot results without 16 extra validation labels per class, together with per-dataset paired gain distributions or confidence intervals, and the actual per-backbone figures for the claimed 15-model generalization?

What are the exact uncertainties of the paired gains used for the “larger than one episode standard deviation” success criterion in Section 6? Can the cue-type prediction be evaluated on additional datasets with mixed texture/shape cues, rather than only seven dataset-level labels, to distinguish a prospective prediction from an explanatory post hoc grouping?

Reviewer_VGcbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes GL-VLM, a training-free method for few-shot recognition that combines global matching with localized patch matching. For each class, the patch features of the support set are clustered with k-means and used as an additional matching signal for patches from images at test time. While the methods improves the Tip-Adapter baseline on DTD, it does not show any significant gains on other datasets. The paper claims that it works well on texture-based datasets but not on shape-based ones.

Critical strengths

S1: The paper openly and transparently reports negative results and its ablation study (Tab. 4) reveals which configurations of the method work and which don't.

Critical weaknesses

W1: The paper is hard to follow. Not because the method is complex, but because it is densely written, with a lot of overloaded phrases, and the results could be better organized (they span multiple protocols, backbones, and shot counts, with key results deferred to the supplementary).

  • The introduction does a poor job in motivating the research direction. Why do we want to predict where local features help fine-grained recognition (as opposed to just optimizing the model)? Why does this paper tackle few-shot recognition in this context? These core questions are not answered.
  • The motivations that are given are not supported either. The paper does not show that the compared methods suffer from forgetting (all keep the backbone frozen, including Tip-Adapter-F), nor that text scoring misses fine-grained structure (in Table 4, the text anchor actually gives the highest DTD accuracy).

W2: The claims are poorly supported.

  • The main research question of when local features help relies entirely on 7 data points (the evaluated benchmarks) and self-assigned labels such as "DTD is a texture dataset". There could be many other reasons why this method works better on this dataset and barely or not at all on others. Moreover, the neutral results on the "shape" datasets are partially because the method simply defaults to beta=0, where it recovers the original baseline. There is no real evidence that texture vs. shape vs. nameable is the underlying reason.
  • The experimental results are poor. The method does not improve over existing training-free methods such as GDA and TAMP+LDA. While the paper is transparent about it, there is little value in the approach as a result.

W3: There is little novelty and scientific insight.

  • As reported by the paper and also shown in the ablation, much of the improvement on DTD is already obtained with a single centroid per class (M=1). So the main method contribution is not providing much benefit beyond using MaskCLIP features with a scoring similar to FILIP.
  • When M=1, the message contradicts the main premise of fine-grained matching as it acts more like another global feature.
  • With M=1, the patch score is mathematically equivalent to comparing mean-pooled patch features to a class centroid, i.e., another global feature. This contradicts the main premise of fine-grained local matching.
Questions for authors

I'd requests the authors to take a thorough look at the weaknesses and address them. Moreover, these questions are particularly relevant.

  • It seems that M=1 is the same as mean-pooled patch features in the ablation Tab. 4. Why does the result of M=1 differ from the "mean-pool" ablation which goes back to the baseline?
  • Can you provide evidence that texture, rather than other dataset properties, drives the gains?
  • Which beta is selected on each dataset? Without this information is it not clear which results are fallbacks or weak results with the provided method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.