- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies contrastive pre-training under distribution drift, focusing on the potential bias introduced by delayed updates of momentum teachers. The authors propose Resilient Contrastive Pre-training (RCP), which combines a causal-intervention-inspired objective with a query–key–value module and a drift adaptation window within a momentum-based student–teacher framework. Experiments on ImageNet-LT, iNaturalist2018, domain-shift benchmarks, and OOD detection tasks report improvements over several contrastive pre-training methods, particularly for underrepresented classes.
Critical strengths
-
The paper addresses representation learning under non-stationary data distributions, where the stability provided by momentum teachers may conflict with the need for rapid adaptation. This is a meaningful research direction, and the proposed framework attempts to address it at the pre training stage rather than through downstream classifier adjustments.
-
Tables 1-5 report improvements on long-tailed classification, linear probing, domain-shifted classification, and OOD detection. The gains on underrepresented classes are particularly relevant to the paper's motivation. Appendix B also examines model and adaptation-window length, providing additional empirical evidence.
Critical weaknesses
1.Section 2.2 derives an interventional objective through a structural causal model. However, the approximation in Eq. (7) and its transition to the QKV computation in Section 2.3 are not adequately justified. In particular, it is unclear how the implementation represents the drift contexts and their probabilities in Eq. (5). The normalization dimension of the Softmax operation in the proposed QKV module also needs clarification. Consequently, the evidence does not yet establish that the implemented objective performs the claimed causal adjustment rather than providing a useful attention-based modification of contrastive learning. This uncertainty affects the interpretation of the paper's central contribution.
- The experiments do not sufficiently isolate teacher lag as the source of improvement. Although the reported downstream results are encouraging, long-tailed classification and post-training domain-shift evaluation do not independently establish adaptation to temporally evolving distributions or demonstrate that momentum-teacher lag causes the observed degradation. Furthermore, Appendix B.2, Table 7 shows that a larger adaptation window substantially improves RCP, while MoCo v3 is reported only at a smaller window. At the matched window of 4,096, the improvement is more modest. Without direct teacher-lag diagnostics, EMA sensitivity analysis, and stronger window-matched controls, the source of the reported gains remains unclear.
Questions for authors
- Could the authors clarify the derivation from Eq. (7) to the implemented QKV objective, including the Softmax dimension and the representation of drift contexts? How much improvement remains when the intervention module is removed or replaced with a standard attention operation?
2.How is the time-varying training distribution constructed? Could the authors provide direct teacher-lag measurements and comparisons across EMA coefficients, together with MoCo v3 results under matched adaptation-window sizes?
I have not seen the authors' final response, so I will maintain my original rating.