acceptodds
Under review as a conference paper at ICLR 2027

GSSP: Elevating Under-Performing Classes in Zero-Shot Image Classification

Abstract

Vision Language models exhibit strong generalized zero-shot image classification capabilities, however, they frequently suffer from severe performance disparities with accuracy in which under-performing classes often fall to near zero accuracy. To address this without the computational burden of retraining, we propose the Gram-Schmidt Subspace Projection (GSSP) a training-free, lightweight framework that operates directly in the post-projection embedding space. Our method constructs orthonormal bases using the corresponding text embeddings of the under-performing classes, applying targeted geometric transformations to the visual features. Because raw projection can cause catastrophic forgetting of a model's generalized knowledge, we introduce the extension GSSP-Boost, which utilizes a parametrized blending mechanism to regularize the method and maintain the overall performance. Through comprehensive empirical evaluations across a diverse suite of datasets, we demonstrate that GSSP-Boost significantly improves the under-performing classes' accuracy while preserving the model's overall performance. Notably, our approach consistently outperforms existing baselines across different architectures, offering a highly effective flexible method for deploying robust zero-shot image classification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Zero-shot Image Classification, CLIP, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_UnY5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gram-Schmidt Subspace Projection (GSSP) and its parameterized extension, GSSP-Boost, to improve the accuracy of under-performing classes in zero-shot image classification using Vision-Language Models (VLMs). The method constructs orthonormal bases using the text embeddings of the worst-performing classes and projects image embeddings into this subspace. To prevent catastrophic forgetting of general class knowledge, GSSP-Boost blends the projected representation with the original image embeddings using a parameterized mechanism. The method is training-free and is evaluated on 10 datasets across different architectures, including CLIP, PE, and SigLIP2.

Critical strengths
  1. The proposed Gram-Schmidt Subspace Projection (GSSP) framework operates directly in the post-projection embedding space using geometric operations, completely avoiding costly backpropagation computations or model fine-tuning.
  2. The method is effective not only on the foundational CLIP architecture but is also validated across multiple vision-language model architectures, including Perception Encoder (PE) and SigLIP2. Furthermore, its extended version, GSSP-Boost Prompted (GSSP-BP), successfully demonstrates that this geometric subspace framework can directly absorb rich text descriptions (such as CuPL) generated by external Large Language Models (LLMs) to achieve compounded performance gains.
  3. The GSSP-Boost method, which introduces the blending parameter λ\lambda, effectively improves the classification accuracy of tail (under-performing) classes while significantly mitigating the catastrophic forgetting caused by the original GSSP method, successfully maintaining the model's overall Harmonic Mean performance.
Critical weaknesses
  1. When identifying "under-performing classes," the paper's similarity-based selection strategy requires reserving 10% of the target dataset's images in advance as an independent calibration subset. This reliance on target-domain unlabeled data for preliminary statistics essentially deviates from a strict zero-shot image classification setting, introducing fairness flaws when directly compared to baselines that require absolutely no target-domain samples (such as UPE).

  2. Although Appendix A provides the limit derivations for the projection and boosting mechanisms, the paper admits that up-weighting directions spanned by under-performing classes is "tested empirically rather than guaranteed by the construction". Additionally, in Equations (2) and (3), the authors compute the global mean μ\mu across all classes in the dataset but only apply mean-centering and orthogonalization to the N%N\% tail classes. There is a lack of rigorous mathematical boundary analysis explaining why global mean-centering holds theoretical optimality in local subspace projection.

  3. The effectiveness of the method is highly dependent on the designated percentage of tail classes (N%N\%). As shown in Figure 3, as NN increases from 5% to 30%, the relative performance boost across all architectures exhibits a significant decline. However, in the primary experimental setup, the authors uniformly adopted a fixed value of N%=20%N\%=20\% for datasets with drastically different total class counts (e.g., 102 classes in Flowers102 vs. 397 classes in SUN397), which limits the method's robustness across heterogeneous datasets.

Questions for authors
  1. To ensure experimental fairness across different baseline methods, since GSSP-BP incorporates CuPL's text descriptions, could the authors supplement the performance comparisons of other training-free feature alignment methods (such as IsoCLIP or CALIP) when combined with CuPL descriptions?

  2. When constructing the subspace for text features, the paper explicitly mentions that mean-centering is not applied to the image embeddings themselves. From a mathematical derivation perspective, if the same centering mapping used for text features were applied to the image features, how would the cosine similarity distribution and theoretical error bounds change before and after projection?

  3. Considering that in a strict, pure zero-shot application scenario, the model would have absolutely no prior access to 10% of the target dataset's samples for calibration, could the authors provide a TUPT_{UP} selection strategy that does not require any target image involvement and is based purely on text semantic topology or class similarity structures?

4 comments

Thank you for your review, below is our response to your questions. An updated version of the paper with the mentioned changes will be uploaded before the end of the discussion period

W1

The GSSP transformation itself is zero-shot. Given a set of target classes, it is built entirely from their text embeddings and applied to the image features without labels, training or target images. The calibration split is used only for deciding which classes to target. Regarding fairness, all baselines and compared methods use the same calibration to find the under-performing classes -- GSSP and its variants are not utilizing any additional information in an unfair manner.

W3

We use a percentage rather than a fixed number of classes precisely because the datasets differ so much in size. We did not mean to suggest that 20% is the optimum value for the bottom N as that will differ for each dataset. The decline in gain as N grows is expected and is not specific to GSSP. A larger N adds performant classes to the selection leading to a smaller increase in performance as these classes are already performing well. The decline also follows from the design of the method: as the selection grows towards all classes, the boost adds nearly the same term to every class score and the prediction returns to those of the baselines. GSSP is designed to target a minority of classes. Figure 3 shows the full range from 5% to 30% for every dataset and shows that as more classes are included in the bottom N selection there are less gains in performance. We will clarify this in the results section.

Q1

We ran two new baselines which combine CALIP and IsoCLIP with CuPL and the comparison with CuPL and GSSP-BP is presented in the table below. Adding CuPL to CALIP gives almost the same result as CuPL alone, and adding it to IsoCLIP is worse than CuPL alone. GSSP-BP gives the highest under-performing class accuracy of all CuPL combinations. We will add these results to Table 1 in the main paper.

MethodTUPT_{UP}HM
CLIP35.643.7
CALIP51.750.2
IsoCLIP33.640.0
CuPL58.158.2
CALIP-CuPL59.658.3
IsoCLIP-CuPL52.650.6
GSSP-B44.950.5
GSSP-BP63.755.8

Q2

The projection in Eq. (5) is linear, so centring the image by the class mean μ\mu of Eq. (2) gives πUP(x−μ)=πUP(x)−πUP(μ)\pi_{UP}(x - \mu) = \pi_{UP}(x) - \pi_{UP}(\mu). The boosted score for class ii (Eq. 7) then becomes

(x+λπUP(x))⋅ti−(μ+λπUP(μ))⋅ti∥(x−μ)+λπUP(x−μ)∥2\frac{\big(x + \lambda \pi_{UP}(x)\big) \cdot t_i - \big(\mu + \lambda \pi_{UP}(\mu)\big) \cdot t_i}{\lVert (x - \mu) + \lambda \pi_{UP}(x - \mu) \rVert_2}

The first term of the numerator is the GSSP-Boost score of the uncentred image. The second term, bi=(μ+λπUP(μ))⋅tib_i = \big(\mu + \lambda \pi_{UP}(\mu)\big) \cdot t_i, depends only on the class, not on the image. The denominator is the same for every class, so it does not change the prediction in Eq. (1). Centring the images is therefore equivalent to GSSP-Boost with a fixed per-class offset −bi-b_i. Each image's similarities are shifted by these class-dependent constants and rescaled by a common factor. A prediction can change only when the margin between two classes is smaller than the difference of their offsets, which is at most (1+λ)∥μ∥2∥ti−tj∥2(1+\lambda) \lVert \mu \rVert_2 \lVert t_i - t_j \rVert_2 ; all other predictions are unchanged. Because of the modality gap, μ\mu is the mean of the text embeddings and not of the image embeddings, so subtracting it does not centre the images. For these reasons we apply the centring only when constructing the text basis. We will add this as an extended discussion of implementation details in the appendix.

Q3

We have implemented two further selections that use only the Class Names:

The first method, text clustering uses the default prompt, i.e., “a photo of a {class}”, which can be compared using cosine similarity with the prompts of all other classes. A class is scored by its average cosine similarity to its nearest neighboring prompts. Classes surrounded by very similar prompts are ranked as under-performing, since their prototypes have small margins to competing classes.

The second method uses the CuPL descriptions which are classified with the zero-shot classifier. In detail, for each description, we compute the margin between its similarity to the class's own prompt and its highest similarity to any other class prompt. A class is scored by the average margin over its descriptions. Classes whose descriptions the model cannot clearly separate from other classes are ranked as under-performing. In effect, the descriptions act as text stand-ins for target images.

As observed in the table below Text clustering recovers under-performing classes somewhat better than a random selection. The CuPL description margin is considerably more effective: 44%-54% of the classes it selects are true under-performing classes, approaching our image-based selection. GSSP can therefore be applied when no target images are available at the cost of a less precise selection.

We will add these results into the appendix regarding different methods on finding under-performing classes.

SelectionCLIPPESigLIP2
Random18.320.321.3
Text clustering31.438.133.0
CuPL description margin46.043.554.3
Image-based (similarity)59.051.360.2

We just wanted to follow up and check whether you have had a chance to review our responses. We would greatly appreciate any feedback you may have. Please also let us know if you require any further information or clarification from our side.

Reviewer_2GGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GSSP constructs a subspace from selected under-performing classes' text embeddings and projects or boosts image features within that subspace. The method is training-free and evaluated across ten datasets and three visual-language encoders, with a prompted extension using richer class descriptions.

Critical strengths

The transformation is simple, lightweight, and compatible with multiple pretrained encoders. Moreover, the experiments provide evidence of improved selected-class accuracy, and the paper investigates the projection-strength trade-off.

Critical weaknesses
  1. The main performance-preservation claim is too strong. A higher harmonic mean does not establish preservation of the remaining classes' accuracy. For CLIP on Flowers102 in Table 1, the baseline has HM = 24.0 and selected-class accuracy = 14.3, whereas GSSP-BP has 38.2 and 41.1. Under the stated HM definition, the remaining-class accuracy is approximately 74.6% versus 35.7%. The paper discusses trade-offs, but this substantial cost should be reported explicitly. The claim of consistently outperforming CuPL also needs qualification: Flowers102 HM is 38.2 for GSSP-BP versus 48.2 for CuPL.
  2. The class-selection protocol is insufficiently specified. Section 3.1 reserves 10% for calibration and samples 5% of that subset, but describes each draw as approximately 5% of the entire dataset; the literal calculation gives 0.5%. It is also unclear whether class-level similarities average over shared unlabeled images, true-class subsets, or pseudo-labeled subsets. These alternatives have different implications for the zero-shot setting.
Questions for authors
  1. Could the authors report selected-class, remaining-class, and overall accuracy separately for GSSP-B and GSSP-BP, and qualify the claims of performance preservation and consistent superiority?
  2. What are the exact calibration sample counts, and which labels, if any, are accessed? How well does this ranking identify genuinely under-performing classes?
4 comments

Thank you for your review, below is our response to your questions. An updated version of the paper with the mentioned changes will be uploaded before the end of the discussion period

Q1

We have included the table below which reports the under-performing classes accuracy TUP_{TUP}, remaining classes accuracy, R, overall accuracy, O, and Harmonic Mean, HM separately. On PE and SIGLIP2, GSSP-BP raises the under-performing accuracy by 14.8 and 15.2 points on average across all datasets. Overall accuracy changes by 0.3 and -0.9 points and remaining class accuracy falls by 3.4 and 4.8 points. On CLIP the cost is much larger: a 28.1 increase in Bottom-N but remaining class accuracy falls 14.5 points and overall accuracy 5.4.

Regarding the claims of performance preservation and superiority, we primarily focus on the accuracy of the under-performing classes, in which GSSP-BP consistently achieves higher under-performing accuracy than CuPL. We report the harmonic mean of the under-performing classes and the remaining classes accuracy to show whether the gain on the under-performing classes comes with a balanced overall result, thereby showing the trade-off. We note that the λ\lambda parameter (eq.7 in the main paper) controls this trade-off, which we did not tune for these results instead choosing the middle ground of 0.5. Figures 2 and 5 (Appendix B.1) show how the bottom-N gain and overall accuracy change across the lambda sweep.

For Flowers-102, you are correct in that the harmonic mean for GSSP-Boost using CLIP performs worse than CUPL, however, in all cases for SigLIP2 and PE GSSP-BP outperforms CUPL apart from UCF101/FGVCAircraft for SigLIP2. We will update the discussion in Section 4.2 accordingly and will add the table below to the appendix for reference to the overall and remaining accuracies.

DatasetBaseline TUPT_{UP}Baseline RBaseline OBaseline HMCuPL TUPT_{UP}CuPL RCuPL OCuPL HMGSSP-B TUPT_{UP}GSSP-B RGSSP-B OGSSP-B HMGSSP-BP TUPT_{UP}GSSP-BP RGSSP-BP OGSSP-BP HM
CLIP
Caltech10177.790.882.383.783.989.184.186.582.689.183.185.786.983.382.185.0
OxfordPets61.388.083.272.393.584.586.388.878.284.483.581.296.773.177.683.2
StanfordCars32.264.858.343.157.852.854.055.239.760.756.548.060.347.150.052.9
Flowers10214.376.263.824.038.863.559.048.224.267.258.935.641.135.737.738.2
SUN39739.067.860.049.554.564.160.758.948.564.460.655.462.058.258.860.1
EuroSAT21.337.734.927.246.435.538.740.226.636.334.930.748.433.737.739.7
UCF10132.768.862.444.466.458.460.662.246.361.258.952.773.445.351.556.0
Food10165.882.279.073.174.580.779.577.577.079.178.778.082.574.375.978.2
FGVCAircraft4.421.918.47.424.119.120.121.37.319.316.910.633.114.318.120.0
DTD6.950.942.512.140.745.244.442.918.748.142.527.052.238.941.444.5
Mean35.664.958.543.758.159.358.758.244.961.057.550.563.750.453.155.8
PE
Caltech10189.197.889.093.288.497.888.992.990.197.388.793.689.897.088.793.2
OxfordPets90.796.595.493.590.496.195.093.295.095.295.295.193.494.093.993.7
StanfordCars89.194.893.691.889.994.393.492.090.794.193.492.490.593.693.092.0
Flowers10230.397.485.646.353.595.888.368.644.793.484.860.563.187.983.673.5
SUN39760.682.377.269.861.081.977.069.969.378.976.773.869.677.776.073.4
EuroSAT4.264.050.37.912.963.952.221.48.363.851.014.629.064.756.340.0
UCF10169.489.685.878.278.489.187.383.477.288.186.382.382.786.486.184.5
Food10189.895.594.492.686.895.793.991.094.094.294.294.193.094.494.193.7
FGVCAircraft35.174.966.947.850.674.469.660.244.772.166.655.266.172.270.969.0
DTD33.777.268.947.052.874.870.661.952.373.069.060.962.968.367.365.5
Mean59.287.080.766.866.586.481.673.566.685.080.672.274.083.681.077.9
SigLIP2
Caltech10188.697.388.392.885.297.688.291.090.096.287.293.089.496.888.793.0
OxfordPets90.694.493.792.483.994.992.889.195.890.591.693.191.792.792.692.2
StanfordCars90.393.392.891.889.293.292.591.294.192.292.693.192.692.292.392.4
Flowers10229.494.983.144.936.494.683.452.635.287.278.750.254.388.081.367.1
SUN39751.078.572.361.841.379.769.754.460.174.071.866.459.875.171.666.6
EuroSAT0.151.940.40.11.551.840.73.00.251.740.30.42.051.040.33.8
UCF10139.981.774.053.667.875.774.471.558.478.675.067.072.767.569.070.0
Food10182.291.789.886.779.592.089.685.388.689.989.689.387.690.890.189.1
FGVCAircraft0.044.035.20.040.142.942.441.50.142.333.90.153.232.936.940.6
DTD35.268.462.146.543.165.761.352.051.466.363.557.955.960.859.958.2
Mean50.779.673.257.156.878.873.563.157.476.972.461.065.974.872.367.3

Q2

We apologize for this being unclear. We will add an extended description of the class-selection protocol as given below and additional calibration methods in the appendix.

We reserve 10% of each dataset as a calibration split, the remaining 90% is used as the evaluation set. The similarity score uses three draws from the calibration split. Each draw takes 50% of the calibration images, representing 5% of the full dataset --- not 5% of the calibration subset.

We then, for each class cc, take the calibration images labeled cc in a draw and compute the cosine similarity between each of these images and the text embedding of cc. The class score is the mean of these similarities, averaged over the three draws. Classes with the lowest scores form the Bottom N %. The score therefore measures how well a class's own images align with its text embeddings. It requires the ground-truth labels of the small calibration split but never uses the evaluation images or evaluation labels.

Across three calibration splits and Bottom-N from 5% to 30%, 51%-60% of the classes selected by the similarity ranking are among the true under-performing classes (CLIP 59.0 ±\pm 0.4, PE 51.3 ±\pm 2.3, SigLIP2 60.2 ±\pm 1.3). A random search reaches 18%-21%.

We have implemented two further selections that use only the Class Names:

The first method uses the default prompt, i.e., “a photo of a {class}”, which can be compared using cosine similarity with the prompts of all other classes. A class is scored by its average cosine similarity to its nearest neighboring prompts. Classes surrounded by very similar prompts are ranked as under-performing, since their prototypes have small margins to competing classes.

The second method uses the CuPL descriptions which are classified with the zero-shot classifier. In detail, for each description, we compute the margin between its similarity to the class's own prompt and its highest similarity to any other class prompt. A class is scored by the average margin over its descriptions. Classes whose descriptions the model cannot clearly separate from other classes are ranked as under-performing. In effect, the descriptions act as text stand-ins for target images.

Regardless of the method of finding the underperforming classes, we wish to highlight the following: Firstly, all baselines and compared methods use the same calibration to find the under-performing classes -- GSSP and its variants are not utilizing any additional information in an unfair manner. Secondly, that GSSP improves upon the baselines regardless of how the underperforming classes are chosen.

In addition, in appendix B, we show the oracle results if the under-performing classes are known a priori, but excluded this from the main results since it requires knowing class specific knowledge across the evaluation set.

SelectionCLIPPESigLIP2
Random18.320.321.3
Text clustering31.438.133.0
CuPL description margin46.043.554.3
Image-based (similarity)59.051.360.2

We just wanted to follow up and check whether you have had a chance to review our responses. We would greatly appreciate any feedback you may have. Please also let us know if you require any further information or clarification from our side.

Comments on the paper

5 comments

Thank you for your review, below is our response to your questions. An updated version of the paper with the mentioned changes will be uploaded before the end of the discussion period

Q1/Q/2/Q3

We reserve 10% of each dataset as a calibration split, the remaining 90% is the evaluation set. The similarity score uses three draws from the calibration split. Each draw takes 50% of the calibration images of every class, about 5% of the full dataset not 5% of the calibration subset. Across three calibration splits and Bottom-N from 5% to 30%, 51%-60% of the classes selected by the similarity ranking are among the true under-performing classes (CLIP 59.0 ±\pm 0.4, PE 51.3 ±\pm 2.3, SigLIP2 60.2 ±\pm 1.3). A random search reaches 18%-21%.

For the similarity score, for each class cc we take the calibration images labeled cc in a draw and compute the cosine similarity between each of these images and the text embedding of cc. The class score is the mean of these similarities, averaged over the three draws. Classes with the lowest scores form the Bottom N %. The score therefore measures how well a class's own images align with its text embeddings. It requires the ground-truth labels of the calibration split but never uses the evaluation images, evaluation labels or model predictions.

We also include below a table for two evaluated selection strategies that use no target images and no labels at all. Text clustering ranks classes by how similar their prompt is to the prompt of their nearest neighbor classes. The CuPL description margin classifies each class's CuPL descriptions with the zero-shot classifier and ranks classes by the average margin between their own class and the best competing class. Both identify under-performing classes better than chance but neither matches the similarity ranking. Without labels, a larger share of boosted classes are not actually under-performing. Boosting such classes spends the method's effect on classes that do not need it and adds to the cost on the remaining classes. This may not not fully reflect the distribution of the whole dataset but it identifies the under-performing classes considerably more reliably. This choice does not change the zero-shot nature of GSSP itself. Given a set of target classes, the transformation is built entirely from their text embeddings and is applied to the image features without any labels, training or target images. The same transformation applies unchanged when that set comes from another source.

Regardless of the method of finding the underperforming classes, we wish to highlight the following: Firstly, all baselines and compared methods use the same calibration to find the under-performing classes -- GSSP and its variants are not utilizing any additional information in an unfair manner. Secondly, that GSSP improves upon the baselines regardless of how the underperforming classes are chosen.

In addition, in appendix B, we show the oracle results if the under-performing classes are known a priori, but excluded this from the main results since it requires knowing class specific knowledge across the evaluation set.

SelectionCLIPPESigLIP2
Random18.320.321.3
Text clustering31.438.133.0
CuPL description margin46.043.554.3
Image-based (similarity)59.051.360.2

Q4

We repeated all experiments with three different random calibration splits. Each split re-draws which 10% forms the calibration set, and therefore which classes are selected as under-performing classes. As shown in the table below the variance is 2--3 orders of magnitude smaller compared to the gains in TUPT_{UP} and HM we see for GSSP-B and GSSP-BP. We will add these results in the appendix.

BackboneMethodTUPT_{UP}HM
CLIPGSSP-B+9.2 ± 0.2+6.7 ± 0.1
GSSP-BP+28.5 ± 0.4+12.5 ± 0.5
PEGSSP-B+7.9 ± 0.5+5.9 ± 0.4
GSSP-BP+15.4 ± 0.5+11.5 ± 0.4
SigLIP2GSSP-B+6.8 ± 0.2+4.4 ± 0.4
GSSP-BP+16.9 ± 1.6+11.5 ± 1.1

Q5

We have run our experiments against a new logit scaling baseline which multiplies the Bottom-N logits by (1+s). As can be seen from the results shown in the table below even though the logit scale provides comparable performance. That is the case with scaling being tuned for each model. Because the logits are scaled cosine similarities, multiplying the Bottom-N logits by (1+s) changes predictions abruptly and an optimal value of ss strongly depends on the backbone. GSSP-Boost shows more consistent results for a constant value of lambda across different backbones and therefore can be used without per-model tuning. Moreover, GSSP-Boost changes the image representation rather than the classifier's scores so it combines directly with other classifiers. Combined with CuPL descriptions (GSSP-BP), it gives the largest gains on the under-performing classes. We will add the Logit scale baseline to Table 1.

BackboneMethodBottom-NRemainingOverallHM
CLIPBaseline35.664.958.543.7
GSSP-B (λ=0.5)44.961.057.550.5
GSSP-BP (λ=0.5)63.750.453.155.8
Logit scale (s=0.01)38.964.458.746.4
Logit scale (s=0.02)42.063.759.048.7
Logit scale (s=0.05)49.061.158.452.7
Logit scale (s=0.1)55.054.654.753.6
PEBaseline59.287.080.766.8
GSSP-B (λ=0.5)66.685.080.672.2
GSSP-BP (λ=0.5)74.083.681.077.9
Logit scale (s=0.01)60.886.780.868.2
Logit scale (s=0.02)62.186.480.969.3
Logit scale (s=0.05)65.985.680.972.3
Logit scale (s=0.1)71.084.080.775.9
SigLIP2Baseline50.779.673.257.1
GSSP-B (λ=0.5)57.476.972.461.0
GSSP-BP (λ=0.5)65.974.872.367.3
Logit scale (s=0.01)51.479.573.257.6
Logit scale (s=0.02)52.179.473.358.0
Logit scale (s=0.05)53.978.973.259.1
Logit scale (s=0.1)56.477.772.860.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.