acceptodds
Withdrawn from ICLR 2027

Koo-Fu CLIP: Closed-Form Adaptation of Vision–Language Models via Fukunaga–Koontz Linear Discriminant Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models such as CLIP provide general-purpose representations, but their embeddings are not explicitly optimized for class separation or compactness in downstream supervised tasks. We propose Koo-Fu CLIP, a supervised adaptation method based on Fukunaga–Koontz Linear Discriminant Analysis. The method combines regularized within-class whitening with a rotation that orders projection directions by between-class scatter. The resulting closed-form transformation supports multiple embedding sizes through truncation, without iterative optimization. Experiments with CLIP, SigLIP2, and DINOv3 on seven datasets show improvements over the original embeddings in prototype and nearest-neighbor classification across most evaluated settings. Dimensionality reduction often further improves nearest-neighbor accuracy while reducing storage requirements and distance-computation costs. For example, on ImageNet-1k, Koo-Fu CLIP raises CLIP 15-nearest-neighbor accuracy from 79.42% to 82.18% while reducing embedding dimensionality from 768 to 192. Although learned linear projections generally achieve higher accuracy, Koo-Fu CLIP obtains all output dimensionalities from a single fitted transformation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
VLM adaptation, Fukunaga-Koontz transform, LDA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ET3JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Koo-Fu CLIP adapts frozen visual embeddings using labeled within-class and between-class scatter matrices. A regularized whitening step followed by an ordered discriminative rotation gives a linear transform that can be truncated to different output dimensions without refitting for a fixed regularization setting. Experiments compare nearest-visual-prototype and 15-NN classification across three encoders and seven datasets, including learned linear projection baselines.

Critical strengths

I like the practical simplicity of fitting one transform and then choosing among several embedding sizes without retraining. The evaluation across CLIP, SigLIP2, and DINOv3 shows that the approach can be useful beyond one particular encoder, with especially substantial gains on FungiTastic-Mini. The learned linear comparisons are also helpful, and the paper is clear that those methods often achieve higher accuracy. That makes the proposed method an interesting option when a compact representation and a simple fitting procedure matter.

Critical weaknesses
  1. I'm a little unsure how I'd choose a compact configuration before seeing its final evaluation results. Section 4.1 uses validation data to select the regularization and also to report performance, while Tables 1–2 report the best tested reduced dimension. The splits and the source/regularization within each dimension sweep are fixed, which is helpful; my question is whether selection and final evaluation are separate. Could you clarify that and check one existing feature setting with a separate selection split? That would make the reported compression gains easier to interpret as something a user can expect after choosing a configuration.

  2. Could we unpack what the between-class ordering adds to the compression gain? At full dimension, the rotation in Eq. (15) leaves cosine similarities unchanged, so those results already capture the corresponding whitening effect. The reduced representations do improve further, but I'm wondering how much depends on that particular ordering of directions. Keeping the whitening matrix fixed and comparing against PCA ordering of the whitened features at one existing nonzero-regularization setting, such as CLIP/ImageNet-1k at lambda = 500 and 192 dimensions, would help separate discriminative ordering from a more general benefit of compression.

  3. The learned linear comparisons in Table 5 are useful, and I'd love to see the other half of that tradeoff: how much fitting time does Koo-Fu save? At 192 dimensions on CLIP/ImageNet-1k, for example, it gets 82.18% versus 82.52% for the stronger learned baseline, so the practical case depends on the fitting savings. Could you add timing for one existing setting, showing the cost of producing one embedding size and several sizes, including regularization selection and keeping shared feature extraction separate? Existing timing logs may be enough, and this would make the benefit of reusing one fit much easier to judge.

Questions for authors

Could you clarify the selection/evaluation separation, the added value of between-class ordering under fixed whitening, and the fitting-time tradeoff for one versus several output sizes? I'm referring to the same three points above; these are the pieces that would most change my assessment.

Reviewer_12a8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper applies regularized Fukunaga–Koontz discriminant analysis to frozen image embeddings. It whitens within-class scatter and orders directions using between-class scatter. The resulting projection can be truncated to different embedding sizes without refitting. The paper evaluates nearest-prototype and nearest-neighbor classification with CLIP, SigLIP2, and DINOv3 on seven datasets, and compares with two learned linear projection baselines.

Critical strengths

The method is simple and can produce several embedding sizes from one fitted transformation. This could be useful when storage or search cost matters. The experiments also cover three encoders and report gains over the original embeddings in many settings. These are useful observations, even though they do not resolve my concerns about the contribution and comparisons.

Critical weaknesses
  1. The novelty is very limited. Section 3 mainly applies classical regularized LDA through whitening and a discriminative rotation to frozen image features. Both the construction and truncating ordered directions are established ideas. Section 2 also acknowledges close prior work on discriminative whitening and discriminant analysis of CLIP features. At full dimensionality, the orthogonal rotation does not change cosine similarities, so the gains over raw features do not by themselves establish a benefit beyond within-class whitening. I do not see a sufficiently clear new technical contribution or empirical insight beyond applying this familiar transform to newer encoders.

  2. The baseline comparisons are too limited. Tables 1 and 2 mainly compare original embeddings with full-dimensional and reduced versions of the proposed method. These are variants of the same pipeline. Figure 3 and supplementary Table 5 do include two learned linear projections, so there are additional comparisons, but they still leave out important simple alternatives. PCA, PCA whitening, and within-class whitening alone would help establish what the method adds. A matched comparison with a visual discriminant classifier, such as the relevant component of GDA-based CLIP adaptation, would also be informative. The learned projections already outperform Koo-Fu on many settings. The paper needs a stronger comparison at matched embedding sizes and label access to establish why this approach is useful.

  3. The writing does not make the actual implementation clear enough. Section 3.1 spends considerable space deriving standard LDA, while Section 3.2 leaves basic preprocessing choices unclear. For example, the text around Eq. 12 describes a centered input, but Eqs. 17–18 apply the transformation directly to encoder embeddings without explicitly subtracting the training mean. Please state whether both reference and query features are centered, which mean is used, and where normalization is applied. This matters because the evaluation uses cosine similarity. A short, explicit algorithm would make the method much easier to follow and reproduce.

  4. The figures are difficult to read and interpret. Figure 2 uses small labels and packs fitting, transformation, storage, and classification into one diagram. Figure 3 overlays many dataset and method curves, with small axis labels and a dense legend. This makes the comparisons harder to follow than necessary. Please enlarge the text and use simpler panels with fewer curves. The main reasons for my rejection recommendation are the limited novelty and insufficient experimental comparisons; the presentation problems add to the difficulty of assessing the work.

Questions for authors
  1. What new technical result or empirical finding distinguishes this method from standard regularized LDA and earlier discriminative whitening methods? What does within-class whitening alone achieve under the same evaluation?

  2. Can you compare against simple dimensionality-reduction and discriminant-analysis baselines with the same labeled data and output dimensions? Since avoiding iterative training is a main practical argument, please also report measured fitting costs for the relevant alternatives.

  3. Are reference and query embeddings centered before projection? Please specify the centering and normalization steps used to obtain Tables 1 and 2.

Reviewer_nyfVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes using Fukunaga–Koontz Linear Discriminant Analysis (Koo-Fu) to produce embeddings of multiple embedding sizes given a classification dataset. These embeddings can be used for nearest-neighbor and prototype classification, often reducing dimensionality and improving classification accuracy.

Critical strengths

Given an image classification dataset, the paper provides a simple closed-form method to reduce embedding dimensionality while improving nearest-neighbor and prototype classification accuracy. The method is evaluated across 7 datasets and with embeddings from 3 models. The method could help to efficiently compute similarities and store high-dimensional embeddings.

Critical weaknesses

The method is only compared to linear projections and performs slightly worse on most datasets. The main advantage of the proposed method is that Koo-Fu doesn’t need to be refit for a different embedding size. It is not clear when the proposed method helps in practical applications.

Furthermore, there are other baselines that can also provide multiple output dimensions without refitting. PCA, whitened PCA, standard regularized LDA, and a linear projection trained on the same frozen embeddings with a Matryoshka-style objective could enable the same capabilities. The proposed method should be compared to such baselines. Also, the method should be compared in more detail to closely related methods from Radenovic et al. (2019) and Mensink et al. (2012). In addition, an ablation study on which component leads to the improvements would be interesting, including, for example, regularized within-class whitening alone.

The authors “select the regularization parameter λ\lambda using associated validation data” (l. 305), but also “evaluate on the validation split of the target dataset” (l. 315). If these validation sets are the same, this could increase the score on the validation split. The authors should clarify the protocol and, if the same split is used, report results on an independent held-out split after selecting hyperparameters.

Questions for authors
  1. How does the proposed method perform to other related methods?
  2. In which practical settings, should Koo-Fu be used compared to a linear projection?‌ How much faster or better is it in such scenarios?
  3. Are the reported accuracies measured on the same split that is used to tune the hyperparameter?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.