acceptodds
Withdrawn from ICLR 2027

HEADA: Hybrid Embedding ADAptation for Open-set Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting pretrained representations to downstream tasks is more challenging in an open world, where models often face unseen categories. In this setting, even strong pretrained embeddings (e.g., DINOv3 and SigLIP2) can remain under-adapted, as standard linear probing keeps the embedding space unchanged rather than adapting it for known–unknown separation. To reduce this gap, we propose Hybrid Embedding Adaptation (HEAda), a lightweight nonlinear projection head trained jointly with classification and metric-learning objectives. HEAda uses only known-class examples and operates directly on embeddings. By adapting both the embedding space and logits, HEAda supports wide range of open- and close-set methods within the same framework. HEAda consistently improves both open- and closed-set recognition over linear probing. It reduces mean AUROC error by , , and for embedding-, logit-based, and hybrid OSR methods, respectively, while improving closed-set accuracy by percentage points on average. But, more importantly, once the representation is properly adapted, simple KNN and Mahalanobis scores outperform more elaborate OSR methods, without additional detector training or auxiliary unknown data. This suggests that representation adaptation can matter more than the choice of OSR scoring rule.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
open-set recognition, representation learning, frozen vision encoders, post-hoc open-set detection, out-of-distribution detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YAugOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that even pretrained representations remain under-adapted for open-set recognition. The authors propose HEAda, a lightweight nonlinear projection head that combines classification and metric-learning objectives to adapt the embedding space. Extensive experiments demonstrate consistent improvements across various encoders, datasets, and OSR methods, and show that simple KNN and Mahalanobis scores can outperform more sophisticated detectors after proper representation adaptation.

Critical strengths

The strengths of this paper are clear. (1) It achieves effective improvements in open-set recognition with large pretrained models through a combination of two existing losses, and the resulting representation improvements are consistently beneficial across a wide range of OSR methods. (2) The experimental results are convincing, and the findings provide useful insights for future OSR research, particularly for practical applications. (3) The experiments and the analyses based on them are quite comprehensive.

Critical weaknesses

The weaknesses of this paper are also clear. (1) Although the proposed method and experimental analysis are useful, most of the results are not particularly surprising. Fine-tuning a large pretrained model to obtain more compact representations is a relatively straightforward idea, and most of the experimental findings are largely consistent with expectations. (2) The proposed method is essentially a brute-force combination of existing techniques, with limited methodological novelty. (3) The results could serve as a useful new baseline for future OSR research, but they provide limited academic inspiration for future work, since learning more compact representations is already a common and well-established idea in the field, while the novelty of the proposed method itself is relatively limited.

Questions for authors
  1. The choice of loss functions appears somewhat ad hoc, particularly the motivation for using the long-tailed loss. I suggest that the authors clearly explain the rationale for this choice at the first point where the loss is introduced.

  2. The description of the proposed method in the abstract and introduction is very limited. After reading these sections, it is still difficult for the reader to understand what the proposed approach actually does. I suggest providing a clearer and more informative overview of the method in the abstract and introduction.

  3. I believe this work has both significant strengths and weaknesses. Although I like this paper, my Weak Reject recommendation is mainly based on its limited novelty. However, I leave the final decision to the AC, as I am not fully certain about ICLR’s criteria for evaluating the level of contribution.

Reviewer_5VL2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HEAda, a feature adaptation framework for open-set recognition (OSR) that operates directly on frozen pretrained embeddings. HEAda uses a nonlinear projection head trained with a hybrid objective combining a classification loss (Seesaw Loss) and a metric-learning loss (Proxy-Anchor Loss). The adapted embeddings and classifier logits are subsequently used by existing post-hoc OSR methods. The approach does not require access to the pretrained encoder during adaptation or unknown-class samples for training. Experiments verify the advantages of the proposed method.

Critical strengths
  1. The paper considers a relevant setting where only pretrained embeddings are available and fine-tuning the backbone is not feasible. The proposed approach is simple and can be readily applied to different encoders and OSR detectors.
  2. The experiments cover six pretrained encoders, multiple datasets, and twelve OSR methods. The improvements are notable, especially for KNN (+10.1 AUROC points) and Mahalanobis (+7.8 points), showing the potential of feature adaptation for distance-based OSR.
Critical weaknesses
  1. The method essentially combines a standard MLP adapter with existing classification and metric-learning losses, followed by off-the-shelf OSR detectors. Using metric learning to obtain more compact and discriminative known-class representations is not a new idea in OSR. I find it difficult to identify a clear methodological contribution beyond combining these existing components.
  2. The main comparisons are against linear probing and classification-only MLPs. While these baselines are useful for understanding the source of the improvements, they are not sufficient to establish an advantage over existing feature adaptation and metric-learning approaches for OSR. I would like to see more direct comparisons under the same pretrained features and evaluation settings.
  3. Proxy-Anchor Loss operates only on known-class features and does not directly enforce separation between known and unknown samples. The experimental gains are clear, but the paper does not explain why this particular loss should be especially effective for OSR. It is also unclear whether a simpler prototype-based objective could achieve similar results.
  4. The authors observe that the choice of unknown classes can have a larger impact than the training seed, but the additional split experiments are restricted to Textures and two encoders. It remains unclear how consistently the improvements would hold under different unknown-class partitions.
Questions for authors
  1. How does HEAda differ from existing frozen-feature adaptation and OSR representation-learning methods? Can the authors provide direct comparisons under the same pretrained features and evaluation protocol?
  2. Why is Proxy-Anchor Loss chosen for this task? Would a simpler learnable-prototype loss achieve similar performance under the same training setup?
  3. Since the training objective only involves known classes, why does it improve known–unknown separation? Can the authors provide further evidence beyond the current correlation analysis?
  4. How do the parameter counts and computational costs of the standard and parameter-matched HEAda models compare with linear probing? Is the standard configuration still lightweight?
Reviewer_X4hkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of open-set recognition. The authors propose to add a trainable MLP after pretrained models, and adapt the feature space of the pretrained model to better facilitate OSR.

Critical strengths
  1. The proposed method is intuitive.

  2. The proposed method is evaluated on different benchmarks with different pretrained encoders and detection methods. The experimental analysis is comprehensive.

Critical weaknesses
  1. The novelty and technical contribution of this paper are limited. The technical components, e.g., MLP adapter, classification loss, proxy-anchor loss, are extensively studied in previous works, and combining them does not sufficiently justify technical contributions to this field.

  2. The paper does not adequately compare with state-of-the-art methods in OSR. Though the authors evaluated with post-hoc OSR scoring methods, comparisons with other relevant approaches are missing. While the authors briefly discussed previous metric-learning/feature-adaptation methods such as CIDER, PALM, SSD/SupCon in the related works, the experiments do not compare the proposed method with these previous works. Therefore, the experiment results can only demonstrate the superiority over the baselines rather than advanced methods in OSR.

Questions for authors

My main concerns are with the limited novelty and inadequate evaluation. A few other questions are listed below.

  1. One of the claimed contributions is that HEAda "uses the same or fewer trainable parameters than a linear probe in our setting", but this may be misleading. HEAda uses a two-layer MLP with a width of 4096. For small-class datasets such as CUB, the number of parameters is far greater than that of linear probing. This is only true for datasets with several thousand classes, such as FungiTastic.

  2. In abstract, the improvement of accuracy is reported as 4.3, while in section 4.2 it is reported as 3.89. It would be better if the authors could keep the numbers (and how the numbers are obtained) consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.