- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper re-examines the reliability of zero-shot accuracy for evaluating continual vision-language pretraining using 28 publicly released TiC-CLIP checkpoints across three training strategies. By comparing zero-shot classification with nearest-class-mean classification and linear probing, the authors find that prompt configurations can reverse model rankings and that declines in zero-shot accuracy do not necessarily correspond to degradation in visual discriminability. The experiments further show that linear probing produces smaller and more consistent performance gaps between training strategies, particularly when measuring the benefits of historical data replay. Based on these findings, the authors recommend incorporating image-only evaluations, per-dataset statistical comparisons, and prompt sensitivity analysis into continual pretraining evaluation protocols.
Critical strengths
- The motivation is clearly presented and easy to follow.
- Clear organization and detailed experimental reporting.
Critical weaknesses
-
Limited Novelty and Research Significance. The main findings largely confirm existing observations, including the sensitivity of CLIP's zero-shot accuracy to prompt configurations and the fact that accuracy degradation does not necessarily imply representation forgetting. While the paper demonstrates that prompt sensitivity can reverse checkpoint rankings, it does not further investigate why different training strategies exhibit such sensitivity.
-
Unclear Evaluation Objectives and Questionable Interpretation of Results. The paper does not sufficiently distinguish between evaluating zero-shot classification performance and assessing visual representation quality, which are inherently different objectives. Since zero-shot classification relies on image-text alignment while linear probing requires labeled downstream data, discrepancies between these protocols are not necessarily surprising. So smaller probing gaps do not necessarily imply that zero-shot evaluation overestimates the benefits of replay, as the two protocols measure different downstream capabilities.
-
Limited Practical Significance and Actionable Insights. Although the paper recommends combining zero-shot evaluation with linear probing and prompt sensitivity analysis, it remains unclear how these additional evaluations would meaningfully influence practical continual-pretraining decisions, such as how findings would lead to different model selections.
-
Insufficient Experimental Generalization. The entire experimental evaluation relies on the publicly released TiC-CLIP checkpoints, covering only one model architecture (ViT-B/16) and a single continual-pretraining benchmark. The paper does not investigate whether the conclusion happens across other model architectures, scales, or continual-pretraining methods. Given that the paper draws broader conclusions about evaluating continual vision-language pretraining, experiments beyond this single benchmark are necessary to establish the generality of its findings.
-
The figures are relatively simplistic and do not effectively communicate the experimental findings. For example, Figure 2 contains heavily overlapping scatter points that make individual trends difficult to interpret, while Figure 3 presents only median replay gaps without showing variability or uncertainty across datasets.
Questions for authors
None