acceptodds
Withdrawn from ICLR 2027

Do Shared VLM Coordinates Imply Shared Calibration? A Study through Few-Shot Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained vision-language models place visual and textual class references in shared coordinates, but alignment does not determine their compatibility with downstream geometric calibration. We expose this distinction through training-free few-shot class-incremental learning, where support-derived visual estimates and pretrained textual anchors are evaluated against the same image queries. Crossing visual- and text-derived diagonal calibration transforms with the two isolated scorers reveals a reproducible role-dependent calibration asymmetry. Across three encoders and four benchmarks, visual-derived calibration is more favorable to visual than textual scoring in all 12 settings. Textual accuracy decreases in every setting, even when the calibration uses textual statistics from base classes or all currently known classes. Our geometric analysis and source-matched counterexample explain the key distinction: calibration replaces native cosine with an affine Mahalanobis-like cosine, and matching reference marginals does not ensure preservation of image-to-reference class ordering. These results motivate source-preserving scoring: calibrate the empirical visual estimate, preserve the pretrained textual anchor, and compose their class evidence. We instantiate this principle as Dual-Space Product-of-Experts (DS-PoE), combining a base-calibrated visual expert and a native textual expert through fixed logit addition, without gradient updates, replay, or dataset-specific modality weights. Across four benchmarks, DS-PoE achieves state-of-the-art performance, attaining the best mean session accuracy on every benchmark and a four-benchmark mean of 78.6% under the evaluated training-free protocol. Our code is included in the supplementary material.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
few-shot class-incremental learning, continual learning, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9FBPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the challenge of improving performance in inference for vision-language models (VLMs), such as CLIP, by applying statistical calibration using image or text prototypes. Although VLMs share a coordinate between images and text, this paper experimentally shows that calibrating using statistics from text prototypes leads to a significant degradation in performance. The paper points out that this is because statistics based on the text prototypes are calculated without using information from the image query, and there are cases where calibration does not yield a positive effect. Based on these observations, the paper proposes DS-PoE, a calibration method that utilizes both image and text prototypes. DS-PoE applies calibration to scores derived from image prototypes, while calculating scores from text prototypes without calibration. It computes the final score by summing these two scores and then calculates the final class classification prediction probability. Experiments confirm that DS-PoE achieves higher accuracy than existing methods in a training-free class-incremental learning setting.

Critical strengths
  • The paper analyzed the effectiveness of image and text prototypes in the statistical calibration of VLMs and demonstrated that while calibration is useful for image prototypes, it actually leads to a decline in performance for text prototypes.
  • The paper proposes a simple calibration-based classification method that combines image and text prototypes. This method is easy to implement.
Critical weaknesses
  • There is a weak connection between the motivation, the proposed methodology, and the problem formulation. As stated at the beginning of the introduction, the focus of this paper is on whether visual and text prototypes can be used compatibly in the statistical calibration of VLM features. However, the specific problem addressed in this paper is unnecessarily limited to FSCIL. The paper’s research focus and research questions should be addressed within a simple problem framework that excludes noisy and complex assumptions. At a minimum, the paper should identify the fundamental challenges of validation via zero-shot inference and explain the necessity of validating calibration within the FSCIL framework.
  • The theoretical analysis in Section 3 is not directly related to FSCIL and fails to provide clear insights into the reasons why the text prototype fails. While the analysis in Section 3.4 demonstrates that the order of candidate classes may change when using the text prototype, this does not imply a clear degradation, as the paper itself acknowledges in the Limitations section. Furthermore, these analyses do not explain why the method works well when using a visual prototype.
  • The experimental findings in this paper overlap with those of Qi et al. [a], and thus lack novelty. Furthermore, the theoretical analysis in Qi et al. [a] demonstrates that using visual prototypes rather than text is optimal for image classification. In this sense, this paper can be viewed as reaffirming the findings in [a] through the lens of statistical calibration. However, the paper fails to cite [a], an important prior study, nor does it include it in its discussion or experimental baselines.
  • There is significant room for improvement in the writing of this paper. For example, the following concerns can be raised.
    • Overall, the logical relationships between paragraphs, sentences, and words are unclear. The writing does not meet the standard level expected for academic writing.
    • Coined terms are being overused without clear definitions, e.g., "scoring role," "reference statistics," "source-matched calibration," and "native geometry."
    • The explanation mixes issues related to VLMs with those related to FSCIL, making the relationship unclear.
    • Figure 1 does not aid intuitive understanding.

[a] Qian, Qi, Yuanhong Xu, and Juhua Hu. "Intra-modal proxy learning for zero-shot visual categorization with clip." Advances in Neural Information Processing Systems 36 (2023): 25461-25474.

Questions for authors
  • Why is it necessary to discuss the problem on FSCIL? What is the connection between VLMs' geometry and FSCIL?
  • What makes the proposed method theoretically superior? Why is it necessary to use text prototypes, which result in a loss of accuracy, as an expert for PoE with equal weights?
  • Which is better: InMaP [a] or the proposed method?
Reviewer_vBjgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether visual and text classifiers in a shared VLM space can share a calibration WR(z)=Norm((z−μR)⊘u~R)W_R(z) = \mathrm{Norm} \big( (z - \mu_R) \oslash \sqrt{\tilde{u}_R} \big) in training-free FSCIL. A crossed source × scorer experiment shows a consistent asymmetry: visual statistics help the visual scorer but hurt the text scorer, and text statistics hurt both, with Γ=Δ(V←V)−Δ(T←V)>0\Gamma = \Delta(V \leftarrow V) - \Delta(T \leftarrow V) > 0 in all 12 settings. Based on this, DS-PoE scores with s=svcal+stnatives = s_v^{\text{cal}} + s_t^{\text{native}} and reaches 78.6 mean AavgA_{\text{avg}} on four benchmarks versus 75.4 for BiMC.

Critical strengths

The experimental design is clean. Crossing source with scorer, holding the reference banks and queries fixed within each row, and reporting bootstrap intervals over 12 settings makes the asymmetry hard to dismiss as noise. As a result, it is a useful caution for training-free methods that apply one calibration to both modalities. The method is simple and insensitive to ρ. The ImageNet-1K gain over the reproduced baselines (+6.0) is substantial.

Critical weaknesses
  1. The source of the CUB gain is not isolated. On Pets and ImageNet-100, UVA is within about one point of ABS+WCA (+0.33+0.33 and +0.99+0.99) and of CLIP-Adapter. On CUB, however, it is +19.12+19.12 over ABS+WCA and +13.99+13.99 over CLIP-Adapter. A gain that appears on only one dataset needs an explanation, and the "difficulty-scaling rule" is fitted to just three points. The key control is missing: the same adapters Av,AtA_v, A_t and the same description-prototype InfoNCE loss applied to the global CLS feature, without ABS regions. Without it, the gain may come from training against attribute-description prototypes, which CLIP-Adapter does not do, rather than from region alignment, which is the paper's thesis. In addition, the few-shot baselines are limited to CoOp, CLIP-Adapter, and Tip-Adapter. MaPLe and PLOT are discussed but not compared, and the paper does not state whether the Table 1 baselines were re-run on the same splits.

  2. Hyperparameters appear to be selected on CUB test accuracy. The sweeps in Sec. 5.2 peak at λ=0.2\lambda = 0.2 and K=30K = 30 with 80.6080.60, which the limitations section identifies as the best random draw (range 79.9879.98–80.6080.60). These values are then used on all datasets. The 44-shot validation split exists but is used only for checkpoint selection. The effect is probably large enough to survive proper selection, but the paper should redo the selection on validation data.

Questions for authors
  1. What accuracy does the global-feature variant (adapters Av,AtA_v, A_t plus description-prototype InfoNCE, without ABS regions) achieve on CUB?
  2. Were the Table 1 few-shot baselines re-run on your 1616-shot splits and ViT-B/16 checkpoint, or copied from prior work? Could you add MaPLe or PLOT under the same protocol?
  3. If λ\lambda and KK are selected on the 44-shot validation split instead of CUB test accuracy, which values are chosen and what CUB accuracy results?
  4. What are the agreement-gate statistics (agreement rate, and accuracy of agreeing vs. disagreeing predictions) on CUB? Please also report the standard deviation on ImageNet-100.
Reviewer_xbMSOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether geometric calibration affects visual and textual representations similarly, given that both share the same embedding space in vision-language models. The authors find that geometric calibration consistently degrades image-to-text classification performance, even when the calibration statistics are derived from textual embeddings. Based on this observation, they propose Dual-Space Product-of-Experts (DS-PoE), which combines a geometrically calibrated visual classifier with a textual classifier that preserves the original embedding geometry. The proposed approach achieves improved performance in training-free few-shot class-incremental learning.

Critical strengths

The proposed DS-PoE is simple yet effective, achieving strong performance without requiring additional training or replay.

The calibration analysis is comprehensive, covering 12 encoder–dataset settings across three vision-language models and four benchmarks.

The geometric analysis provides valuable insights into why calibration can degrade image-to-text classification performance despite the shared embedding space.

Critical weaknesses
  1. Limited methodological novelty:The main contribution of the paper lies in its calibration analysis, while the proposed DS-PoE is essentially a simple addition of logits from a calibrated visual classifier and an uncalibrated textual classifier.

  2. Limited theoretical explanation: The geometric analysis provides a counterexample demonstrating that calibration can reverse class ordering. However, this does not establish that textual calibration necessarily degrades classification performance. Furthermore, the theoretical analysis does not explain why geometric calibration tends to improve visual classification while degrading textual classification.

  3. Limited textual representation diversity: The visual prototypes are constructed from multiple images per class, whereas each textual prototype is derived from a single description. It would be valuable to investigate whether using multiple diverse textual descriptions per class improves textual calibration. This could help determine whether the observed degradation is partly caused by limited textual representation diversity rather than an inherent difference between the two modalities.

  4. Novelty of the observed calibration asymmetry: The authors should more clearly establish how the observed role-dependent calibration asymmetry differs from findings in prior studies on CLIP embedding geometry, modality gaps, and feature calibration. In particular, it is important to clarify whether similar behavior has already been observed or explained in existing work.

Questions for authors
  1. Choice of evaluation setting: Why did the authors choose few-shot class-incremental learning (FSCIL) as the primary evaluation setting? Since the proposed geometric analysis is not specific to incremental learning, could the authors provide additional results on standard zero-shot and few-shot classification to demonstrate the generality of their findings?

  2. Explanation of calibration asymmetry: Could the authors provide a stronger theoretical or empirical explanation for why geometric calibration generally benefits visual classification while consistently degrading textual classification? The current counterexample demonstrates that calibration can alter class ordering but does not fully explain the observed modality-dependent behavior.

  3. Novelty relative to prior work: Do existing methods typically apply the same geometric calibration to both visual and textual representations? If not, what specific gap in the existing literature does the proposed analysis address? The authors should more clearly distinguish their findings from prior studies on modality-specific calibration and CLIP embedding geometry to establish the novelty of the observed role-dependent asymmetry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.