acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing and Mitigating Late-Stage Shortcut Emergence in Generative Self-Supervised Learning

Abstract

Generative self-supervised learning is commonly improved by extending pretraining, yet better in-distribution representations do not necessarily translate into uniformly better robustness. We study this discrepancy by tracking frozen representations throughout prolonged masked-image pretraining. Beyond controlled stylized shifts, we find that performance on several style-sensitive natural distribution shifts, including ImageNet-Sketch and a label-aligned subset of ImageNet-R, can peak substantially earlier than in-distribution accuracy and subsequently decline while in-distribution performance continues to improve. This behavior recurs across multiple settings but is shift-dependent, revealing a late-stage divergence between in-distribution fitting and transferable representation quality. Motivated by this observation, we formulate an operational invariance-aligned pretraining design principle: nuisance variation should not only be introduced at the input, but should also be paired with supervision that makes nuisance-dependent features unreliable for minimizing the pretraining objective. We instantiate this idea with HyGDL, which combines cross-perturbation self-distillation, low-rank geometric projection, and residual-conditioned reconstruction to encourage perturbation-stable representations. Across unsupervised domain generalization benchmarks, HyGDL improves robustness over generative SSL baselines and mitigates the late-stage degradation observed in standard pretraining. Representation-level probing further shows that the learned stable projection preserves substantially more semantic predictability than its orthogonal residual, while the residual remains non-trivially domain-predictive. Together, these results characterize a previously underexplored failure mode of prolonged generative pretraining and provide an operational approach for reducing shortcut-sensitive representation drift.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Representation learning, Shortcut learning, Domain generalization, Robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X1niOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tracks frozen representations across pretraining checkpoints and observes that MAE/SimMIM performance on several appearance shifts peaks before ID performance and subsequently declines. HyGDL combines AdaIN perturbations, EMA self-distillation, minibatch cross-view SVD projection, and residual-conditioned reconstruction. It improves average PACS performance and mitigates late-stage deterioration on ImageNet-Sty and ImageNet-Sketch.

Critical strengths
  • a clear and testable empirical observation: evaluation every 200 epochs reveals changes hidden by final-checkpoint reporting. Three runs and two generative SSL objectives provide replication evidence

  • good performance: HyGDL reaches 58.15±0.33 average accuracy on PACS, compared with reproduced DisMAE at 53.74 and MS-UDG at 54.96.

Critical weaknesses
  • In the main DomainNet protocol, 29.37±0.29 is only slightly above MS-UDG's 29.16, while HyGDL is worse on Infograph and Quickdraw; MS-UDG variability is not reported. In reverse transfer, HyGDL's 26.14 trails DARLING's 29.99 and SimCLR v2's 28.59. The acknowledged asymmetry limits claims of broadly improved domain generalization and motivates investigating where AdaIN perturbations may fail to preserve useful semantics.

  • Please supplement the representation analysis with visualizations across training checkpoints and decomposition components. Using a fixed sample set, color features by semantic class and style/domain at early, peak-OOD, and final checkpoints, and compare the original, stable, and residual representations.

  • Please specify the backbone, model size, initialization, and result provenance for each method in Table 2. Although DisMAE and MS-UDG are described as reproduced under the same source-only/from-scratch protocol, it remains unclear whether all baselines use the same ViT-Small backbone as HyGDL. Clarify which methods follow Table 6 and provide comparisons with key baselines under matched backbones and comparable training budgets.

Questions for authors

see Weaknesses

============================================

Thanks for authors' responses. I am willing to keep my initial score.

3 comments
AuthorsOpenReview ↗

We thank the reviewer for the careful reading and constructive feedback. We appreciate the positive assessment of the observation and performance, and respond to each concern below.

W1

In the main DomainNet protocol, 29.37±0.29 is only slightly above MS-UDG's 29.16, while HyGDL is worse on Infograph and Quickdraw; MS-UDG variability is not reported. In reverse transfer, HyGDL's 26.14 trails DARLING's 29.99 and SimCLR v2's 28.59. The acknowledged asymmetry limits claims of broadly improved domain generalization and motivates investigating where AdaIN perturbations may fail to preserve useful semantics.

Response. We thank the reviewer for highlighting this limitation.

Under the main protocol, HyGDL achieves 29.37±0.2929.37\pm0.29\\% average accuracy, compared with 29.1629.16\\% for our protocol-matched MS-UDG reproduction. Since variability is unavailable for MS-UDG and the average difference is only 0.210.21 points, we do not interpret this gap as statistically meaningful. The per-domain results are also heterogeneous: HyGDL improves substantially on Clipart (51.4951.49\\% vs. 47.2347.23\\%) but is lower on Infograph (23.8623.86\\% vs. 26.1226.12\\%) and Quickdraw (12.7612.76\\% vs. 14.1314.13\\%).

The reverse-transfer results further expose this asymmetry: HyGDL obtains 26.1426.14\\%, below DARLING (29.9929.99\\%) and SimCLR v2 (28.5928.59\\%). We therefore view the DomainNet results as evidence that HyGDL's benefit is source--target dependent rather than universal.

A plausible explanation is the perturbation model itself. AdaIN primarily varies appearance statistics while preserving spatial content; this inductive bias is well aligned with style- and texture-related shifts, but may be less appropriate for domains such as Infograph and Quickdraw, where the shift can involve abstraction, sparsity, layout, or structural changes that are not well represented by appearance statistics alone.

Revision. We have revised the Sec 5.3 in manuscript to make this limitation explicit and to avoid broad claims of uniformly improved domain generalization. "However, the per-domain and reverse-transfer results reveal clear source. Target asymmetry, suggesting that the benefit is not uniform across shifts and that the current AdaIN-based perturbation may be less effective when domain differences extend beyond appearance statistics."

AuthorsOpenReview ↗

The response to W2 and W3

W2

Please supplement the representation analysis with visualizations across training checkpoints and decomposition components. Using a fixed sample set, color features by semantic class and style/domain at early, peak-OOD, and final checkpoints, and compare the original, stable, and residual representations.

Response. We thank the reviewer for this helpful suggestion. We have added checkpoint-wise representation visualizations using a fixed PACS sample set at the early (epoch 200), peak-OOD (epoch 800), and final (epoch 1000) checkpoints. For each checkpoint, we compare the original representation, stable projection, and residual, and visualize the same samples when colored by either semantic class or PACS domain. To make within-checkpoint organization visible, we remove checkpoint-wise mean drift; the original and residual representations use shared PCA bases across checkpoints, while the rank-22 stable component is shown in its aligned projection coordinates.

The resulting visualization does not reveal a simple semantic--style factorization. Semantic classes remain substantially intermixed across all three components, whereas clear domain-related organization is visible in the original and stable representations. The residual retains non-trivial domain structure, although its two-dimensional projection becomes less visibly separated at later checkpoints.

We view this result as complementary to the linear-probing analysis rather than contradictory to it. The probe shows that the stable component preserves more semantic predictability than the residual (26.4326.43\\% vs. 17.3117.31\\%), while the visualization shows that this does not imply clean semantic clustering or domain invariance. In particular, the stable projection remains strongly domain-predictive, consistent with our interpretation of "stable" as perturbation-relative rather than as a semantic subspace.

Revision. We have added the new visualization to the Appendix. A.6.1 and revised Sec. 5.5 to make this interpretation explicit.


W3

Please specify the backbone, model size, initialization, and result provenance for each method in Table 2. Although DisMAE and MS-UDG are described as reproduced under the same source-only/from-scratch protocol, it remains unclear whether all baselines use the same ViT-Small backbone as HyGDL. Clarify which methods follow Table 6 and provide comparisons with key baselines under matched backbones and comparable training budgets.

Response. We thank the reviewer for pointing out that the previous version did not make the provenance and comparability of the Table 2 results sufficiently explicit.

We have revised Table 2 to distinguish literature-reported reference results from protocol-matched comparisons. DINO, MAE+BSS, MAE+Sty, DisMAE, and MS-UDG are trained or reproduced by us using the same ViT-Small backbone, source-only/from-scratch initialization, comparable pre-training budget, and downstream evaluation protocol as HyGDL. In contrast, the remaining entries are quoted from the corresponding papers and are not strictly matched: some use different or larger backbones, while others benefit from external ImageNet or MAE pre-training. We therefore retain these results as broader literature references rather than treating them as controlled comparisons.

To make this distinction explicit, we additionally provide a protocol/provenance table in Appendix Sec. A2.6 reporting the backbone, initialization, result source, and whether each method follows our matched protocol. Table 6 summarizes the training and evaluation configuration for the protocol-matched methods.

Under this controlled setting, HyGDL achieves 58.1558.15\\% on PACS, compared with 53.7453.74\\% for DisMAE and 54.9654.96\\% for MS-UDG. On DomainNet, HyGDL obtains 29.37±0.2929.37\pm0.29\\%, compared with 27.4827.48\\% and 29.1629.16\\%, respectively; we therefore characterize the DomainNet result as comparable to MS-UDG rather than as a meaningful improvement.

Revision. We have revised the corresponding discussion so that claims of improvement are based on the protocol-matched comparisons, while literature-reported methods with different model capacity or external pre-training are used only for contextual reference.

Dear Reviewer,

Thank you again for your constructive feedback. We have addressed your questions and added the requested analyses in our rebuttal. As the discussion period is approaching its end, we would greatly appreciate it if you could take a brief look at our responses when convenient and let us know if any concerns remain.

Thank you for your time and consideration.

Reviewer_i5LJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper investigates late-stage robustness divergence in self-supervised learning, where OOD accuracy can fall despite ID accuracy continuing to improve as training progresses. The authors introduce HyGDL: a hybrid generative-discriminative learning framework combining cross-perturbation self-distillation, low-rank geometric decomposition, and residual-conditioned reconstruction.

The authors evaluate HyGDL on PACS, DomainNet, ImageNet-Sty and ImageNet-Sketch, and demonstrate consistently improved OOD performance over self-supervised and unsupervised domain generalisation baselines, and reduced late-stage robustness degradation compared with MAE and SimMIM. With extensive ablation studies they also demonstrate the contributions of each of HyGDL's individual learning objectives.

Critical strengths
  1. The authors showcase HyGDL’s improved OOD performance across several benchmarks over perturbation-based MAE baselines. They also demonstrate improved late-stage robustness degradation which was previously observed with MAE and SimMIM.
  2. The authors provide extensive ablation isolating HyGDL’s learning objective components, and show that removing any learning component degrades performance, as evaluated on PACS.
Critical weaknesses
  1. The novelty of the paper's first listed contribution: identifying late-stage robustness divergence. This has been characterised in more detail by the paper "Understanding the Learning Phases in Self-Supervised Learning via Critical Periods" (Lee et al., ICLR 2026), which the authors don't cite, and also details that intermediate checkpoints can yield stronger OOD generalisation.
  2. The authors do not report the in-distribution accuracy of HyGDL. OOD robustness often comes at the expense of ID accuracy, and omitting the ImageNet-100 accuracy of HyGDL may conceal the tradeoff the algorithm makes against other vanilla/DG methods.
Questions for authors
  1. Can the authors please provide the ID accuracy of HyGDL on ImageNet-100 across the different pertaining checkpoints?
  2. Can the authors ablate mini-batch cross-view SVD decompositions’ contribution? For example in Table 3a, how would HyGDL perform if the residual-conditioned cross-reconstruction uses instead a fixed or learned low-rank projection?
3 comments

Dear Reviewer,

Thank you again for your constructive feedback. We have addressed your questions and added the requested analyses in our rebuttal. As the discussion period is approaching its end, we would greatly appreciate it if you could take a brief look at our responses when convenient and let us know if any concerns remain.

Thank you for your time and consideration.

I want to thank the authors for their responses and added experiments. The revisions made satisfy some of the methodological questions which I’ve raised, regarding the ID–OOD trade-off and the contribution of the low-rank geometric decomposition. With these, I believe the paper’s empirical contributions have been strengthened and I’ll be increasing Soundness from 2->3. However, I’ll be maintaining the original overall rating of 3, given several limitations that remain.

First, the improvements are narrower than the paper frames them to be. For example, HyGDL’s final ImageNet-Sty accuracy improves by just 0.26 percentage points over MAE’s best earlier checkpoint, being within variation across training runs while requiring 1.4 times the GPU-hours and 4 times the peak GPU memory. Similar marginal improvements are observed on DomainNet as well. Second, the authors’ proposed late-stage shortcut emergence hypothesis still seems more suggestive than causal. The perturbation-stable projection still carries domain information, and its advantage over the orthogonal residual in semantic predictability is no longer there after per-sample L2 normalization (14.82% vs. 17.40% after normalization, compared with 26.43% vs. 17.31% before normalization). Finally, HyGDL’s methodological novelty remains limited. The paper’s main observation of late-stage robustness divergence strongly overlaps with what was introduced by Lee et al. (ICLR 2026). Prior methods such as DisMAE and MS-UDG also already combine reconstruction and representation factorization. HyGDL’s low-rank geometric decomposition and residual-conditioned reconstruction objective would be considered incremental compared to prior work.

Thank you for your response, your suggestions and recognition really have greatly helped our work, thank you again!

Reviewer_uK3ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that the transfer performance can peak earlier than ID performance, and gives a possible explanation that the late-stage optimization tends ot use nuisance-sensitive visual statistics,which is related to shortcut learning. The paper identify that this nuisance informaiton,which can be considered as pertuibation information in real world, mostly resides in the residual space, and propose a hybrid generative–discriminative learning framework combining cross-view alignment, low-rank geometric decomposition, and residual-conditioned reconstruction to reduce reliance on perturbation-sensitive information. The framework is evaluated on unsupervised domain generalization benchmark and outperforms other methods.

Critical strengths
  1. The presentation of the paper is clear

  2. Extensive experiments are conducted. From the experiments, the proposed method outperforms other methods.

Critical weaknesses
  1. The paper proposes a research question: Does representation robustness improve monotonically throughout generative self-supervised pre-training? But I do not find a clear answer to this question.

  2. More explanation of the projection in the residual space is required. The authors mentioned that the stable projection (in principal space) captures variation consistently preserved across perturbations. But I think this is different from the use of PCA in classification. For instance, in MNIST, PCA should capture the information of the numbers instead of the background. But the background seems to be the consistently preserved one.

  3. The whole pipeline is similar to DINO, but the comparison with DINO is missing.

Questions for authors
  1. Why use the residual space? Are there any theoretical or experimental explanations?

  2. Is there any evidence for the assumption: late-stage optimization tends ot use nuisance-sensitive visual statistics?

3 comments

Dear Reviewer,

Thank you again for your detailed feedback. We have carefully addressed your main concerns in the rebuttal, including additional experiments and clarifications to the manuscript. As the discussion period is approaching its end, we would greatly appreciate it if you could take another look when convenient and let us know whether any concerns remain.

Thank you for your time and consideration.

Thanks for the response. I think the explanation of residual space is reasonable, while the explanation of the late-stage optimization is still a little weak. Overall, I think the main methodology part is reasonable, and the experimental results show superior performance. Therefore, I raise the score to 3.

Thank you for your response, your suggestions and recognition really have greatly helped our work, thank you again!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to All Reviewers

Dear Reviewers,

We sincerely thank you for your careful reading, constructive comments, and valuable suggestions on our manuscript. We have now completed point-by-point responses to all reviewer comments and have revised the manuscript accordingly. An updated PDF has been prepared and uploaded for your review.

In the revision, we have addressed the major concerns raised across the reviews, including clarifying the answer to our motivating research question, better distinguishing our stable projection from PCA, adding a direct DINO baseline and additional conditioning-source and low-rank-subspace ablations, reporting the in-distribution accuracy trajectory of HyGDL, discussing the DomainNet and reverse-transfer limitations more explicitly, and citing and positioning our work relative to Lee et al. (ICLR 2026). We have also revised the novelty claims and limitation discussions to avoid overstatement.

We hope that the updated manuscript and detailed responses adequately address your concerns. We would be grateful if you could kindly review the revised PDF and the response letters at your convenience.

Thank you again for your time and helpful feedback, which have substantially improved the quality of this work.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.