- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the challenge of improving performance in inference for vision-language models (VLMs), such as CLIP, by applying statistical calibration using image or text prototypes. Although VLMs share a coordinate between images and text, this paper experimentally shows that calibrating using statistics from text prototypes leads to a significant degradation in performance. The paper points out that this is because statistics based on the text prototypes are calculated without using information from the image query, and there are cases where calibration does not yield a positive effect. Based on these observations, the paper proposes DS-PoE, a calibration method that utilizes both image and text prototypes. DS-PoE applies calibration to scores derived from image prototypes, while calculating scores from text prototypes without calibration. It computes the final score by summing these two scores and then calculates the final class classification prediction probability. Experiments confirm that DS-PoE achieves higher accuracy than existing methods in a training-free class-incremental learning setting.
Critical strengths
- The paper analyzed the effectiveness of image and text prototypes in the statistical calibration of VLMs and demonstrated that while calibration is useful for image prototypes, it actually leads to a decline in performance for text prototypes.
- The paper proposes a simple calibration-based classification method that combines image and text prototypes. This method is easy to implement.
Critical weaknesses
- There is a weak connection between the motivation, the proposed methodology, and the problem formulation. As stated at the beginning of the introduction, the focus of this paper is on whether visual and text prototypes can be used compatibly in the statistical calibration of VLM features. However, the specific problem addressed in this paper is unnecessarily limited to FSCIL. The paper’s research focus and research questions should be addressed within a simple problem framework that excludes noisy and complex assumptions. At a minimum, the paper should identify the fundamental challenges of validation via zero-shot inference and explain the necessity of validating calibration within the FSCIL framework.
- The theoretical analysis in Section 3 is not directly related to FSCIL and fails to provide clear insights into the reasons why the text prototype fails. While the analysis in Section 3.4 demonstrates that the order of candidate classes may change when using the text prototype, this does not imply a clear degradation, as the paper itself acknowledges in the Limitations section. Furthermore, these analyses do not explain why the method works well when using a visual prototype.
- The experimental findings in this paper overlap with those of Qi et al. [a], and thus lack novelty. Furthermore, the theoretical analysis in Qi et al. [a] demonstrates that using visual prototypes rather than text is optimal for image classification. In this sense, this paper can be viewed as reaffirming the findings in [a] through the lens of statistical calibration. However, the paper fails to cite [a], an important prior study, nor does it include it in its discussion or experimental baselines.
- There is significant room for improvement in the writing of this paper. For example, the following concerns can be raised.
- Overall, the logical relationships between paragraphs, sentences, and words are unclear. The writing does not meet the standard level expected for academic writing.
- Coined terms are being overused without clear definitions, e.g., "scoring role," "reference statistics," "source-matched calibration," and "native geometry."
- The explanation mixes issues related to VLMs with those related to FSCIL, making the relationship unclear.
- Figure 1 does not aid intuitive understanding.
[a] Qian, Qi, Yuanhong Xu, and Juhua Hu. "Intra-modal proxy learning for zero-shot visual categorization with clip." Advances in Neural Information Processing Systems 36 (2023): 25461-25474.
Questions for authors
- Why is it necessary to discuss the problem on FSCIL? What is the connection between VLMs' geometry and FSCIL?
- What makes the proposed method theoretically superior? Why is it necessary to use text prototypes, which result in a loss of accuracy, as an expert for PoE with equal weights?
- Which is better: InMaP [a] or the proposed method?