HEADA: Hybrid Embedding ADAptation for Open-set Recognition

Adapting pretrained representations to downstream tasks is more challenging in an open world, where models often face unseen categories. In this setting, even strong pretrained embeddings (e.g., DINOv3 and SigLIP2) can remain under-adapted, as standard linear probing keeps the embedding space unchanged rather than adapting it for known–unknown separation. To reduce this gap, we propose Hybrid Embedding Adaptation (HEAda), a lightweight nonlinear projection head trained jointly with classification and metric-learning objectives. HEAda uses only known-class examples and operates directly on embeddings. By adapting both the embedding space and logits, HEAda supports wide range of open- and close-set methods within the same framework. HEAda consistently improves both open- and closed-set recognition over linear probing. It reduces mean AUROC error by $27\%$, $13\%$, and $21\%$ for embedding-, logit-based, and hybrid OSR methods, respectively, while improving closed-set accuracy by $4.29$ percentage points on average. But, more importantly, once the representation is properly adapted, simple KNN and Mahalanobis scores outperform more elaborate OSR methods, without additional detector training or auxiliary unknown data. This suggests that representation adaptation can matter more than the choice of OSR scoring rule.