- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Gram-Schmidt Subspace Projection (GSSP) and its parameterized extension, GSSP-Boost, to improve the accuracy of under-performing classes in zero-shot image classification using Vision-Language Models (VLMs). The method constructs orthonormal bases using the text embeddings of the worst-performing classes and projects image embeddings into this subspace. To prevent catastrophic forgetting of general class knowledge, GSSP-Boost blends the projected representation with the original image embeddings using a parameterized mechanism. The method is training-free and is evaluated on 10 datasets across different architectures, including CLIP, PE, and SigLIP2.
Critical strengths
- The proposed Gram-Schmidt Subspace Projection (GSSP) framework operates directly in the post-projection embedding space using geometric operations, completely avoiding costly backpropagation computations or model fine-tuning.
- The method is effective not only on the foundational CLIP architecture but is also validated across multiple vision-language model architectures, including Perception Encoder (PE) and SigLIP2. Furthermore, its extended version, GSSP-Boost Prompted (GSSP-BP), successfully demonstrates that this geometric subspace framework can directly absorb rich text descriptions (such as CuPL) generated by external Large Language Models (LLMs) to achieve compounded performance gains.
- The GSSP-Boost method, which introduces the blending parameter , effectively improves the classification accuracy of tail (under-performing) classes while significantly mitigating the catastrophic forgetting caused by the original GSSP method, successfully maintaining the model's overall Harmonic Mean performance.
Critical weaknesses
-
When identifying "under-performing classes," the paper's similarity-based selection strategy requires reserving 10% of the target dataset's images in advance as an independent calibration subset. This reliance on target-domain unlabeled data for preliminary statistics essentially deviates from a strict zero-shot image classification setting, introducing fairness flaws when directly compared to baselines that require absolutely no target-domain samples (such as UPE).
-
Although Appendix A provides the limit derivations for the projection and boosting mechanisms, the paper admits that up-weighting directions spanned by under-performing classes is "tested empirically rather than guaranteed by the construction". Additionally, in Equations (2) and (3), the authors compute the global mean across all classes in the dataset but only apply mean-centering and orthogonalization to the tail classes. There is a lack of rigorous mathematical boundary analysis explaining why global mean-centering holds theoretical optimality in local subspace projection.
-
The effectiveness of the method is highly dependent on the designated percentage of tail classes (). As shown in Figure 3, as increases from 5% to 30%, the relative performance boost across all architectures exhibits a significant decline. However, in the primary experimental setup, the authors uniformly adopted a fixed value of for datasets with drastically different total class counts (e.g., 102 classes in Flowers102 vs. 397 classes in SUN397), which limits the method's robustness across heterogeneous datasets.
Questions for authors
-
To ensure experimental fairness across different baseline methods, since GSSP-BP incorporates CuPL's text descriptions, could the authors supplement the performance comparisons of other training-free feature alignment methods (such as IsoCLIP or CALIP) when combined with CuPL descriptions?
-
When constructing the subspace for text features, the paper explicitly mentions that mean-centering is not applied to the image embeddings themselves. From a mathematical derivation perspective, if the same centering mapping used for text features were applied to the image features, how would the cosine similarity distribution and theoretical error bounds change before and after projection?
-
Considering that in a strict, pure zero-shot application scenario, the model would have absolutely no prior access to 10% of the target dataset's samples for calibration, could the authors provide a selection strategy that does not require any target image involvement and is based purely on text semantic topology or class similarity structures?
4 comments
Q1
We ran two new baselines which combine CALIP and IsoCLIP with CuPL and the comparison with CuPL and GSSP-BP is presented in the table below. Adding CuPL to CALIP gives almost the same result as CuPL alone, and adding it to IsoCLIP is worse than CuPL alone. GSSP-BP gives the highest under-performing class accuracy of all CuPL combinations. We will add these results to Table 1 in the main paper.
| Method | HM | |
|---|---|---|
| CLIP | 35.6 | 43.7 |
| CALIP | 51.7 | 50.2 |
| IsoCLIP | 33.6 | 40.0 |
| CuPL | 58.1 | 58.2 |
| CALIP-CuPL | 59.6 | 58.3 |
| IsoCLIP-CuPL | 52.6 | 50.6 |
| GSSP-B | 44.9 | 50.5 |
| GSSP-BP | 63.7 | 55.8 |
Q2
The projection in Eq. (5) is linear, so centring the image by the class mean of Eq. (2) gives . The boosted score for class (Eq. 7) then becomes
The first term of the numerator is the GSSP-Boost score of the uncentred image. The second term, , depends only on the class, not on the image. The denominator is the same for every class, so it does not change the prediction in Eq. (1). Centring the images is therefore equivalent to GSSP-Boost with a fixed per-class offset . Each image's similarities are shifted by these class-dependent constants and rescaled by a common factor. A prediction can change only when the margin between two classes is smaller than the difference of their offsets, which is at most ; all other predictions are unchanged. Because of the modality gap, is the mean of the text embeddings and not of the image embeddings, so subtracting it does not centre the images. For these reasons we apply the centring only when constructing the text basis. We will add this as an extended discussion of implementation details in the appendix.
Q3
We have implemented two further selections that use only the Class Names:
The first method, text clustering uses the default prompt, i.e., “a photo of a {class}”, which can be compared using cosine similarity with the prompts of all other classes. A class is scored by its average cosine similarity to its nearest neighboring prompts. Classes surrounded by very similar prompts are ranked as under-performing, since their prototypes have small margins to competing classes.
The second method uses the CuPL descriptions which are classified with the zero-shot classifier. In detail, for each description, we compute the margin between its similarity to the class's own prompt and its highest similarity to any other class prompt. A class is scored by the average margin over its descriptions. Classes whose descriptions the model cannot clearly separate from other classes are ranked as under-performing. In effect, the descriptions act as text stand-ins for target images.
As observed in the table below Text clustering recovers under-performing classes somewhat better than a random selection. The CuPL description margin is considerably more effective: 44%-54% of the classes it selects are true under-performing classes, approaching our image-based selection. GSSP can therefore be applied when no target images are available at the cost of a less precise selection.
We will add these results into the appendix regarding different methods on finding under-performing classes.
| Selection | CLIP | PE | SigLIP2 |
|---|---|---|---|
| Random | 18.3 | 20.3 | 21.3 |
| Text clustering | 31.4 | 38.1 | 33.0 |
| CuPL description margin | 46.0 | 43.5 | 54.3 |
| Image-based (similarity) | 59.0 | 51.3 | 60.2 |
We just wanted to follow up and check whether you have had a chance to review our responses. We would greatly appreciate any feedback you may have. Please also let us know if you require any further information or clarification from our side.
Thank you for your review, below is our response to your questions. An updated version of the paper with the mentioned changes will be uploaded before the end of the discussion period
W1
The GSSP transformation itself is zero-shot. Given a set of target classes, it is built entirely from their text embeddings and applied to the image features without labels, training or target images. The calibration split is used only for deciding which classes to target. Regarding fairness, all baselines and compared methods use the same calibration to find the under-performing classes -- GSSP and its variants are not utilizing any additional information in an unfair manner.
W3
We use a percentage rather than a fixed number of classes precisely because the datasets differ so much in size. We did not mean to suggest that 20% is the optimum value for the bottom N as that will differ for each dataset. The decline in gain as N grows is expected and is not specific to GSSP. A larger N adds performant classes to the selection leading to a smaller increase in performance as these classes are already performing well. The decline also follows from the design of the method: as the selection grows towards all classes, the boost adds nearly the same term to every class score and the prediction returns to those of the baselines. GSSP is designed to target a minority of classes. Figure 3 shows the full range from 5% to 30% for every dataset and shows that as more classes are included in the bottom N selection there are less gains in performance. We will clarify this in the results section.