acceptodds
Under review as a conference paper at ICLR 2027

When the Teacher Lags: Contrastive Pre-training under Distribution Drift

Abstract

Contrastive pre-training typically assumes that training data follow a stable distribution. When the distribution changes, however, a momentum teacher may retain information from earlier data and provide targets that are poorly aligned with the current samples. We study how this lag affects representation learning and use a structural causal model to characterize the resulting source of bias. Guided by this analysis, we propose Resilient Contrastive Pre-training (RCP), which introduces an intervention-inspired contrastive objective to reduce the influence of drift-induced bias during training. RCP integrates a query–key–value module into a momentum-based student–teacher framework and learns from samples within a drift adaptation window. We evaluate the resulting representations through long-tailed classification, domain-shifted classification, and out-of-distribution detection. Across these evaluations, RCP improves upon standard contrastive pre-training baselines, with particularly consistent gains for underrepresented classes. These findings suggest that accounting for drift during pre-training can improve the robustness and transferability of learned visual representations. Codes are available at https://anonymous.4open.science/r/ResilientCL/.

Primary area:
learning theory
Keywords:
Drift Training, Contrastive Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_PoEWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies contrastive pre-training under distribution drift, focusing on the potential bias introduced by delayed updates of momentum teachers. The authors propose Resilient Contrastive Pre-training (RCP), which combines a causal-intervention-inspired objective with a query–key–value module and a drift adaptation window within a momentum-based student–teacher framework. Experiments on ImageNet-LT, iNaturalist2018, domain-shift benchmarks, and OOD detection tasks report improvements over several contrastive pre-training methods, particularly for underrepresented classes.

Critical strengths
  1. The paper addresses representation learning under non-stationary data distributions, where the stability provided by momentum teachers may conflict with the need for rapid adaptation. This is a meaningful research direction, and the proposed framework attempts to address it at the pre training stage rather than through downstream classifier adjustments.

  2. Tables 1-5 report improvements on long-tailed classification, linear probing, domain-shifted classification, and OOD detection. The gains on underrepresented classes are particularly relevant to the paper's motivation. Appendix B also examines model and adaptation-window length, providing additional empirical evidence.

Critical weaknesses

1.Section 2.2 derives an interventional objective through a structural causal model. However, the approximation in Eq. (7) and its transition to the QKV computation in Section 2.3 are not adequately justified. In particular, it is unclear how the implementation represents the drift contexts and their probabilities in Eq. (5). The normalization dimension of the Softmax operation in the proposed QKV module also needs clarification. Consequently, the evidence does not yet establish that the implemented objective performs the claimed causal adjustment rather than providing a useful attention-based modification of contrastive learning. This uncertainty affects the interpretation of the paper's central contribution.

  1. The experiments do not sufficiently isolate teacher lag as the source of improvement. Although the reported downstream results are encouraging, long-tailed classification and post-training domain-shift evaluation do not independently establish adaptation to temporally evolving distributions or demonstrate that momentum-teacher lag causes the observed degradation. Furthermore, Appendix B.2, Table 7 shows that a larger adaptation window substantially improves RCP, while MoCo v3 is reported only at a smaller window. At the matched window of 4,096, the improvement is more modest. Without direct teacher-lag diagnostics, EMA sensitivity analysis, and stronger window-matched controls, the source of the reported gains remains unclear.
Questions for authors
  1. Could the authors clarify the derivation from Eq. (7) to the implemented QKV objective, including the Softmax dimension and the representation of drift contexts? How much improvement remains when the intervention module is removed or replaced with a standard attention operation?

2.How is the time-varying training distribution constructed? Could the authors provide direct teacher-lag measurements and comparisons across EMA coefficients, together with MoCo v3 results under matched adaptation-window sizes?

1 comment

I have not seen the authors' final response, so I will maintain my original rating.

Reviewer_UjBeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduced a novel mechanism for handling distribution drifts via pre-training. Authors propose Resilient Contrastive Pre-training (RCP) that builds upon observed momentum-teacher lag under drifts. RCP uses query-key-value aggregation with casual model, and evaluated obtained representations on variety of tasks, such as long-tail, domain generalization, and OOD.

Critical strengths

I really like the highlighting of how important problem the teacher lag is under nonstationary settings. The proposed module is simple to follow and evaluation is done nicely over multiple tasks, with finetuning and linear probing. Some reported results are very good, like gains on iNaturalist2018 lin-prob acc increasing from 24.9 to 37.3

Critical weaknesses

I find this paper to have four critical weaknesses that prohibit me from recommending this paper for acceptance.

W1: Lack of any evidence for temporal drift adaptation. While I commend the paper on very nice evaluation on static class imbalance and shifted test distributions; these experiments do not, on their own, demonstrate any nonstationary changes in training distributions over time. The paper do not show any temporal sampling schedules, drift boundaries, or drift rates - standards in any modern papers on distribution changes and concept drift in dynamic settings. Thus the reported results support effective representation learning for shifting test set, rather than the central claim on the method being applicable to non-stationary data.

W2: The casual interpretation part is insufficiently explained. I cannot find sufficient identification of the the latent drift variable and teacher bias. The transition from the formula in Eq. 5 to aggregation in Eq. 8 lacks any convincing derivation. Eq. 7 replaces an expectation of predictions with one prediction at an expected bias without any approximation conditions.

W3: Lack of any ablation of the proposed mechanism. I am missing any direct measurements of teacher lags, momentum sweeps, teacher reset controls, or any more generic ablations. I am also confused by the window size. Table 7 compares the main RCP configuration at 9,600 samples against MoCo v3 at 4,096; matching at 4,096 reduces the ImageNet-LT linear-probing advantage from 5.7 to 1.4 points. I am also missing any comparisons with previously proposed casual-attention mechanisms that already use query–key–value processing for intervention-inspired debiasing.

W4: Limited reproducibility and missing details. I am missing essential details for ensuring reproducibility of this paper. No optimization details, uncertainty/variance from repeated runs, and lack of clear specification of used OOD scoring. I am confused by reported extremely weak results from BYOL. Also, it seems that the base model is not DINOv2, but Zhou et al.’s DINO-WM, based on the reference cited.

Questions for authors

Q1: What exact temporal sampling creates the drift presence during pre-training? How does RCP perform against a shuffled stationary control dataset, using identical class frequencies and samples selected?

Q2: Under what assumptions Eq.5 leads to the attention operator in Eq. 8?

Q3: Do the gains persist when you use matching window / batch size under same seeds, as well as against lower-momentum teachers, periodic teacher resets, or basic attention module?

Q4: Can you provide complete training and OOD protocols, clarify the DINO module confusion, and report multi-seed runs?

Reviewer_261vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at a specific failure mode of student–teacher contrastive pre-training when the data stream changes over time. Because The teacher is an exponential moving average of the student, the EMA teacher become stale targets when the data distribution moves. The proposed fix is a cross-sample attention head with a causal interpretation. It appears to help tail classes under ImageNet-LT pre-training, but the evidence is entangled with batch size, baseline quality, and an undefined notion of "drift".

Critical strengths
  • This paper identifies and targets a real and important problem in contrastive pre-training.
  • The implementation is simple.
Critical weaknesses
  • Drift is never instantiated and what the experiments test is actually stationary imbalance. The paper asks whether contrastive pre-training can learn from non-stationary, drifting data, yet ImageNet-LT and iNaturalist 2018 are just globally, statically imbalanced datasets. The paper says ImageNet-LT "simulates a non-stationary stream" (L74) but never describes the stream ordering, and it also states "we do not assume any type of drift" (L307). If batches are shuffled as usual, the data are IID from a fixed imbalanced distribution, and Eq. 1 holds only trivially through sampling noise. Unless the authors imposed a time-varying class-sampling order in the dataloader and document it, the paper addresses head-class bias accumulating in a momentum teacher on long-tailed data, not concept drift. The framing overstates the contribution.
  • The lagging-teacher hypothesis is not directly or systematically tested. The claim that feature degradation is “attributed to bias accumulation in the momentum-updated teacher” (L84–85) rests mainly on a qualitative t-SNE comparison (Fig. 1), which also confounds class imbalance with pretraining set size (ImageNet-1K ~1.28M vs. ImageNet-LT ~116K). The paper provides no momentum-coefficient sweep, no teacher-free baseline (e.g., SimCLR/SimSiam) under the same stream, and no measurement of teacher–student divergence over time. These are needed to establish the proposed causal mechanism rather than only showing that long-tailed pretraining yields distorted features.
  • No implementation details. Implementation details promised in the appendix are absent.
Questions for authors
  • What should readers take away from Figure 1? In balanced pre-training (Fig. 1a), why do the “tail” classes still collapse into a compact central region? How are tail classes defined in Fig. 1a? Or is this a t-SNE artifact?
  • The paper claims to study how teacher lag affects representation learning, but it remains unclear how exactly the lag affects the learned representations. Could the authors provide ablation studies on (i) the severity of the distribution change (e.g., varying the drift rate or the imbalance ratio), and (ii) the EMA momentum coefficient λ\lambda? In particular, if teacher lag is the cause, a smaller λ\lambda (i.e., a less lagged teacher) should alleviate the degradation for MoCo v3, and the gain of RCP over MoCo v3 should grow as the drift becomes more severe. Do the results show these trends?
  • Can the authors provide comprehensive implementation details about how the distribution drifting is simulated during training? The paper uses “drift” broadly across long-tailed imbalance, domain shift, and OOD detection, but these are distinct phenomena. Distribution shift (e.g., domain/OOD shift) refers to a mismatch between training and target distributions, whereas distribution drift/concept drift requires a time-varying, non-stationary data stream Pt(X,Y)P_t(X,Y). The paper should clearly define which phenomenon each experiment instantiates, and avoid framing static long-tailed datasets or domain-shift benchmarks as evidence of temporal drift unless a time-varying stream is explicitly constructed and documented.
  • The Few column is highlighted incorrectly in Table 2: ReLIC clearly outperforms RCP on Few, so highlighting 7.9 in red is wrong.
Reviewer_td3qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper answers the question of whether constrastive pre-training can learn from non-stationary data, or more specifically, solve concept drift during pretraining. The topic is particularly important for student-teacher networks, where the teacher network is updated with an EMA of the student parameters, which is used to stabilize training. When concept drift is present during pretraining, the teacher network might retain outdated knowledge and fail to adapt to updated data. The provided RCP framework is claimed to solve this issue.

Critical strengths

S1: The experiments involve multiple domains, such as long-tailed classification, OOD detection and domain adaptation. The ablation study covers models of different size, and linear probing on ImageNet-LT with varying window size.

S2: The contrastive drift pre-training and the causal analysis of it is pretty informative and easy to follow/understand. The problem becomes very clear after reading Section 2.1-2.2.

S3: The provided code is highly detailed and covers all experiments provided in the manuscript with exact trained checkpoints, which I specifically appreciate.

Critical weaknesses

W1: In the momentum-updated teacher-student framework, the momentum coefficient λ\lambda is the hyperparameter that directly controls the "memory" of the model just like in any momentum based learning algorithm. For example, a higher λ\lambda makes the memory larger, corresponding to smaller updates/changes to the teacher parameters. Therefore, the authors should definitely provide an ablation study varying the momentum coefficient.

W2: It is not directly clear from the manuscript how the proposed self-attention mechanism fixes the teacher bias, that is claimed to be present by the authors. The authors provide experiments on a wide range of settings, however, none of the experiments directly show the student-teacher lag is solved. The authors should have provide either a metric that measures the lag or results with and without the provided self-attention head. Adding the self-attention head on top of other baselines present in the study would also strengthen this work, and clearly showcase that it fulfills its purpose. On the other hand, the provided RCP framework does not change the EMA update of the teacher, which is claimed to be the main motivation. The teacher is still updated the same way (as given in Equation 2) with the same exact momentum coefficient. Therefore, the teacher could still lag? While RCP reweighs the student features using keys of the teacher, the lag itself may not be fixed as it is not handled directly.

W3: Building on top of W2, while Section 3.2 and 3.3 do broaden the set of experiments, from my understanding, they measure the distribution shift at test time. On the other hand, the main claim of this paper is the concept drift during pretraining. Therefore, I again emphasize that it is unclear how we experience concept drift during pretraining on ImageNet-LT or ImageNet-21k.

Some less critical weaknesses/comments which I feel should still be stated:

W4: The authors say "More detailed experimental implementations are given in Appendix." in Section 3 but it is not provided in the Appendix.

W5: The paper appears to be a bit out of date, as there are no baselines or references dating to 2026, and only a few in 2025 with no provided baselines.

Questions for authors

Q1: The authors provide two different datasets to compare to motivate their work: ImageNet and ImageNet-LT. While I do understand the difference between the two datasets, I do not see how the ImageNet-LT dataset would necessarily cause concept drift. Is it the case that the dataset is provided in a curriculum learning kind of way? I checked the manuscript but did not find any information regarding how the dataset is seen in training. If we simply shuffle the training dataset, then it is unclear to me how we see concept drift. Can the authors clarify this part. From my perspective, the motivation provided by the authors could be more suitable to the topic of continual learning, as we would observe the affect of "teacher lagging" much more clearly in that setting.

While I do see the importance of this work, and the motivation is clear and important, I do not necessarily see the pretraining dataset or in general the long-tailed learning setting to satisfy the provided motivation. Furthermore, I believe an ablation study on the momentum coefficient λ\lambda is absolutely necessary. I am willing to reassess my rating if the authors solve the stated weaknesses and questions, and better explain how their experiments and the proposed self-attention mechanism solves the concept drift problem, and why it is specific to the student-teacher setup.

Q2: A less important question that I also have is, I do not understand why the discussion at the end of Section 2 is present. The provided score component is exactly self-attention, nothing is different except for the use case. What do the authors claim exactly with this paragraph?

Reviewer_ttk5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies contrastive pretraining under concept drift and argues that the EMA teacher in the MoCo-style student-teacher framework accumulates bias. This paper builds a structural causal model, applies backdoor adjustment, and implements the resulting objective as a query-key-value module inside MoCo v3, with infoNCE computed over a drift-adaptation window.

Critical strengths
  1. The module is simple; the code is provided, and the evaluation covers several downstream tasks, including the tail splits.
  2. Window-length and backbone-scaling analyses are included.
Critical weaknesses
  1. Contrastive pre-training is not a particularly important research problem, and the proposed method has limited significance and makes little contribution to the field.
  2. The experiments lack insight: the paper does not analyze in depth why the proposed method works.
  3. The experiments also lack many details. The main text and appendix do not specify the optimizer, epochs, learning rate, EMA schedule, or stream/window construction.
  4. The writing does not reach the standard expected of a top-tier conference paper.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.