acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Zero-shot Evaluation in Continual Vision-Language Pretraining

Abstract

Continual pretraining of vision-language models is commonly evaluated by zero-shot accuracy, which informs whether to retrain from scratch every year at about four times the compute or to keep all past data for replay. Accuracy drops during training are read as forgetting. Yet zero-shot accuracy conflates the image features with their alignment to class-name embeddings, while also depending on evaluator-chosen class names and templates. We disentangle representation quality from zero-shot alignment on all 28 released TiC-CLIP checkpoints by comparing zero-shot classification with nearest class mean and linear probing on the same frozen image features, while varying only the prompt. The checkpoints cover three training strategies: retraining from scratch on all data (Oracle), replaying all past data while continuing from the previous checkpoint (Cumulative), and sequential updating without replay (Sequential), across 2016–2022 and two data-filtering regimes. We find that zero-shot comparisons between training strategies are unstable, imprecise, and inflated, and that zero-shot declines alone do not establish representation forgetting. With model weights fixed, switching between two commonly used templates reverses the sign of the Oracle–Cumulative gap on 6 of 11 datasets under basic filtering. Across datasets, paired tests fail to distinguish Oracle from Cumulative using zero-shot accuracy under either data filter, but separate them using linear probes under both. Under the probe, replay is worth 0.6 to 1.5 points, a third to a fifth of the 3.2 to 4.4 that zero-shot accuracy reports, and its value grows at most a third as fast over time; retraining from scratch adds a further 0.6 to 0.7 points. Along the Cumulative trajectory, where the training pool only grows, zero-shot accuracy drops by at least one point on 10 of 132 one-year steps, and the linear probe on none of them. These findings motivate three checks for comparing continual-pretraining strategies from frozen checkpoints: probe the image features alongside the zero-shot head, test paired per-dataset differences rather than suite-level averages, and verify conclusions across standard prompt configurations. Without these checks, differences attributed to training may instead reflect the evaluation protocol.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wXW2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper re-examines the reliability of zero-shot accuracy for evaluating continual vision-language pretraining using 28 publicly released TiC-CLIP checkpoints across three training strategies. By comparing zero-shot classification with nearest-class-mean classification and linear probing, the authors find that prompt configurations can reverse model rankings and that declines in zero-shot accuracy do not necessarily correspond to degradation in visual discriminability. The experiments further show that linear probing produces smaller and more consistent performance gaps between training strategies, particularly when measuring the benefits of historical data replay. Based on these findings, the authors recommend incorporating image-only evaluations, per-dataset statistical comparisons, and prompt sensitivity analysis into continual pretraining evaluation protocols.

Critical strengths
  1. The motivation is clearly presented and easy to follow.
  2. Clear organization and detailed experimental reporting.
Critical weaknesses
  1. Limited Novelty and Research Significance. The main findings largely confirm existing observations, including the sensitivity of CLIP's zero-shot accuracy to prompt configurations and the fact that accuracy degradation does not necessarily imply representation forgetting. While the paper demonstrates that prompt sensitivity can reverse checkpoint rankings, it does not further investigate why different training strategies exhibit such sensitivity.

  2. Unclear Evaluation Objectives and Questionable Interpretation of Results. The paper does not sufficiently distinguish between evaluating zero-shot classification performance and assessing visual representation quality, which are inherently different objectives. Since zero-shot classification relies on image-text alignment while linear probing requires labeled downstream data, discrepancies between these protocols are not necessarily surprising. So smaller probing gaps do not necessarily imply that zero-shot evaluation overestimates the benefits of replay, as the two protocols measure different downstream capabilities.

  3. Limited Practical Significance and Actionable Insights. Although the paper recommends combining zero-shot evaluation with linear probing and prompt sensitivity analysis, it remains unclear how these additional evaluations would meaningfully influence practical continual-pretraining decisions, such as how findings would lead to different model selections.

  4. Insufficient Experimental Generalization. The entire experimental evaluation relies on the publicly released TiC-CLIP checkpoints, covering only one model architecture (ViT-B/16) and a single continual-pretraining benchmark. The paper does not investigate whether the conclusion happens across other model architectures, scales, or continual-pretraining methods. Given that the paper draws broader conclusions about evaluating continual vision-language pretraining, experiments beyond this single benchmark are necessary to establish the generality of its findings.

  5. The figures are relatively simplistic and do not effectively communicate the experimental findings. For example, Figure 2 contains heavily overlapping scatter points that make individual trends difficult to interpret, while Figure 3 presents only median replay gaps without showing variability or uncertainty across datasets.

Questions for authors

None

Reviewer_yfuEOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper targets the problem of zero-shot evaluation for continual vision-langauge pretraining or CLIP. In particular, this paper identifies accuracy drop can be break down into forgetting on representation or mis-alignment on class embedding and also class names or templates used. The paper disentangles representation quality from alignment on 28 checkpoints by comparing zero-shot classification with nearest class mean and linear probing on frozen representation. The paper targets 3 setups: 'Oracle' that retrains from scratch; 'Cumulative' that continually train with replay on old and new data; 'Sequential' that only on new data. The paper concludes 3 complementary checks: 1) evaluate frozen visual representation; 2) zero-shot performance on paired per-dataset differences, uncertainty; 3) sensitivity to prompts.

Critical strengths

It challenges the ambiguous interpretation of forgetting in continual pretraining of CLIP under 3 setups. Lower zero-shot accuracy can reflect forgetting of visual representation or poorer image–text alignment. The breakdown of the zero-shot accuracy is interesting.

Critical weaknesses

The writing of the paper is the main issue that I have trouble to understand this paper. The title didn't reflect that this is a CLIP paper: 'vision-langauge pretraining' can be MLLMs. Although the breakdown of the zero-shot accuracy is interesting, is zero-shot classification important in 2026 as it's just a proxy eval. for CLIP?

Questions for authors

weird number in abstract: why 'about four times the compute'? how to get a particular number four ?

Reviewer_96WCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to re-evaluate all 28 released TiC-CLIP checkpoints covering three training strategies: Oracle, Cumulative and Sequential, across 2016-2022, under basic and bestpool data-filtering regimes, claiming that the zero-shot image classification is not a sufficient analysis. It uses 11 classification datasets while comparing zeroshot classification accuracy with two image-only evaluations of the same frozen image features, nearest class mean (NCM) and linear probe (LP). The weights are also held fixed while varying the class names and prompt templates. Along with the main study the authors also employ CoOp, k-shot, accuracy-matched and robustness analyses while providing transparent reproducibilty.

Main findings:

  1. Prompt Sensitivity: Prompt choice in itself can flip the sign of the Oracle-Cumulative gap on 6/11 datasets under basic filtering regime
  2. Zero-shot decline: Along the Cumulative trajectory where the training pool grows, zeroshot drops are rarely matched by image-only drops.
  3. Strategy gaps. The probe recovers the reported strategy ordering with smaller and consistent gaps. Replay is worth 0.6–1.5 points under the probe vs 3.2–4.4 under zeroshot, and its value grows at most a third to fifth as fast.
Critical strengths
  1. The paper exposes a real evaluation problem. While holding the weights fixed and changing only the prompt shows that evaluation strategy can reverse the strategy rankings. Under basic filtering, the median O-C gap is 1.8 while the prompt choice itself moves it by a median of 3.9. This directly reflects on how the continual-pretraining works should rethink the evaluation strategies.

  2. The analysis is very throrough and transparent. The paper employs extensive robutstness analysis including paired tests, Hodges–Lehmann intervals, per-step McNemar and bootstrap checks, leave-one-dataset-out sensitivity, accuracy-matched comparisons and full per-checkpoint tables.

Critical weaknesses
  1. Uncertainity is not accounted for at either of the analysis levels
  • Strategy Level: Each training strategy is represented by a singlerun only. The significance tests treat the datasets as the unit of replication, so they show that two particular checkpoints differ across two datasets. However they dont show clearly show that the strategies differ. A 0.6-0.7 probe difference between independantly trained models having different initialisation and schedules, could fall well within the seed variance. Therefore the blanket statements such as "replay is worth 0.6 to 1.5 points" should be attributed to the checkpoints and not the underlying strategies unless backed by seed variance.
  • Step-level: The paper presents the zero-shot drops without the associated image-side drops as "alignment drift and not feature degradation". But it doesnt account for the measurement noise. Under basic filtering, 1 year NCM change has a median width of 1.5 (Table 14) with a half-wdith of about 0.75. Out of the 6 steps where NCM rises while the zeroshot falls, 4 are 0.1-0.6 which lie within the half-width, 1 (0.8) is borderline and only 1 (1.5) is clear rise. Therefore the claim that these points cannot arise from any monotone relation is not backed by enough evidence. Also since the probe changes are in general much smaller compared to the zeroshot changes, showing that the features didnt degrade just because the probe never drops by a point or more seems insufficent and would need an equivalence test.
  1. The evidence covers a narrow setting. The paper mentions zero-shot evaluation yet all the experiments are limited to only classification. In section 2.2, the authors mention that even TiC-CLIP computes backward-transfer on yearwise retrieval tasks but the paper however doesnt include this metric used by their benchmark. Also, the first finding of authors, prompt sensitivity is not applicable to retrieval tasks. The authors acknowledge this in evaluation protocols and limitations however since the checkpoints are frozen, this seems like a necessary metric to be added to this paper unless the authors have a reasonable argument to back this.
Questions for authors
  1. Do the authors have any findings for the retrieval tasks as well. A retrieval counterpart to the probe would be helpful in showing whether the zeroshot vs image-only discrepancy also holds for retrieval tasks. A possible idea would be to fit a linear map from image to caption embeddings and then compare zeroshot and probed retrieval.

  2. Can the authors estimate the seed variance for atleast one comparison, If not then the claims for the strategy level gaps should be changed to attributes of properties.

  3. Can the authors separate the text-encoder drift from cross-modal drift?

  4. Can the authors address the step-level uncertainity concern raised above by adjusting their calculations and seeing how many of the steps still hold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.