acceptodds
Withdrawn from ICLR 2027

Understanding and Mitigating Late-Stage OOD Detection Degradation in EMA Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exponential moving average (EMA) is widely used to improve model generalization, but its behavior in out-of-distribution (OOD) detection is not uniformly beneficial throughout training. We observe a consistent late-stage degradation phenomenon: EMA improves substantially during early and middle training, but its OOD performance later declines, while the plain model remains competitive and gradually closes the gap. Our analysis suggests that this behavior is associated with increasing redundancy in the late-stage plain-model trajectory, which reduces the complementary information available to EMA. We propose ReDiv, a simple Rewind-and-Diversify framework for EMA-based OOD detection. ReDiv first detects the onset of late-stage degradation using an ID-only logit-margin signal and rewinds the complete training state to the corresponding checkpoint. It then diversifies the post-rewind trajectory by fixing the learning rate to avoid further scheduler-induced shrinkage and applying sharpness-aware minimization (SAM) to modify the effective update. Across multiple datasets and OOD benchmarks, ReDiv consistently improves over both plain training and standard EMA, requires no OOD validation data or architectural modification, and significantly complements existing training-time and test-time OOD methods. Through careful investigation, we show that the effectiveness of EMA depends not only on checkpoint quality, also on the diversity of the model trajectory, which verifies the late-stage trajectory control in ReDiv as a powerful way to improve performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
out-of-distribution detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EgdWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the OOD detection performance degradation of EMA models in the late stages of training. It proposes ReDiv, which mitigates this degradation by adjusting the late-stage training process without requiring additional OOD data. Extensive experiments demonstrate that the proposed method achieves performance improvements across multiple datasets and post-hoc detection methods.

Critical strengths
  1. OOD detection is an important task for reliable model deployment.
  2. Investigating the late-stage OOD performance degradation of EMA models from the perspective of training dynamics is interesting.
  3. The proposed method requires no additional OOD data, relying solely on ID data to perform diagnosis and training intervention.
Critical weaknesses
  1. Missing essential ablations: This paper lacks the ablations using SAM + EMA from scratch, as well as switching to SAM in the late stage without Rewind. Consequently, the distinct roles of Rewind and delayed SAM remains unclear.

  2. Lack of ID performance evaluation: This paper does not report ID classification accuracy, making it impossible to judge whether the OOD performance improvement comes at the expense of sacrificing original ID performance.

  3. Lack of cross-architecture evaluation: All experiments are conducted solely using ResNet-18, leaving it unverified whether the observed degradation phenomena and method benefits can generalize to other architectures.

  4. Insufficient related work discussion and contribution positioning: This paper lacks adequate discussion on late-stage SAM [1], SAM for OOD detection [2], and weight averaging[3]. Furthermore, it fails to sufficiently clarify the distinction between this work and existing studies on EMA dynamics, making the contribution unclear.

  5. Lack of computational efficiency analysis: This paper omits the computational efficiency analysis compared to standard training.

  6. Missing some post-hoc methods: This paper lacks comparisons with several post-hoc methods, such as [4], and [5].

[1] Zhanpeng Zhou, Mingze Wang, Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness-aware minimization efficiently selects flatter minima late in training. In International Conference on Learning Representations, pages 20949–20980, 2025.

[2] Chi Zhang, Wei Wang, Yao Zhao, Nicu Sebe, and Yue Song. Sharpness-aware fine-tuning for ood detection. IEEE Transactions on Image Processing, 2026.

[3] Daniel Morales-Brotons, Thijs Vogels, and Hadrien Hendrikx. Exponential moving average of weights in deep learning: Dynamics and benefits. Transactions on Machine Learning Research, 2024.

[4] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2023.

[5] Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. Advances in Neural Information Processing Systems, 35:17885–17898, 2022.

Questions for authors

Please see the Weaknesses.

Reviewer_pTzdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OOD detection of plain model keeps on getting better with training time. However, the paper identifies OOD detection performance degradation of EMA with at late stage of training. The paper provides the theoretical analysis of complementary roles of fixed learning rate and SAM in mitigating the late stage degradation. The method identifies the training time where the peak occurs, rewinds to that stage and then fixes the learning rate and applies SAM thereafter. It uses difference between the highest and second highest logit to determine the peak.

Critical strengths

The finding is novel and useful to OOD detection community. The paper identifies the late stage OOD detection performance degradation of EMA. The paper not only identifies the problem but also proposes the fixes to the problem which seem to be very effective in the compared 4 baselines. This is complemented with theoretical insights of both the components introduced to mitigate it. No architectural change is needed and method is supposed to be scorer agnostic.

The phenomena of degradation is easy is easy to follow and well explained.

Critical weaknesses

The paper is weak in terms of empirical evaluation.

The evaluations on mere MSP, Energy, ReAct, NCI baselines are not enough. The authors are recommended to compare with latest state of the art OOD detection methods. If the performance could be substantially improved in high performing methods, then the paper would be much stronger. For instance, the results of compatibility with Logitnorm (Cifar-10) datasets shows the performance improvement is not significant.

Is the architecture use just ResNet-18? The authors are recommended to demonstrate the experimental results on bigger network as well as diverse networks. For especially OOD detection paper, showcasing the results on just one architecture (ResNet-18) is not enough.

Questions for authors

I dont have questions, but I request the authors for additional experiments as asked in the weakness section.

Reviewer_fZvbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the late-stage degradation of out-of-distribution (OOD) detection performance in models trained with exponential moving average (EMA). The authors observe that EMA improves OOD detection during early and middle training, but its performance deteriorates in later stages while the plain model continues to improve. They associate this phenomenon with reduced diversity among late-stage model checkpoints. To address this issue, they propose ReDiv, a training framework that uses the logit margin on in-distribution validation data to identify a rewind point, then resumes training with a fixed learning rate and sharpness-aware minimization (SAM) to preserve trajectory diversity. The paper provides a theoretical analysis of the relationship between checkpoint quality and diversity in EMA and evaluates the proposed method on CIFAR-10, CIFAR-100, ImageNet-50, and ImageNet-200. Experimental results show improvements in OOD detection across multiple datasets and scoring methods, with particularly substantial gains on CIFAR-10.

Critical strengths
  1. The paper identifies an interesting phenomenon in the training dynamics of EMA models. Examining OOD detection performance throughout training, instead of only at the final checkpoint, provides useful insights into how EMA behaves.

  2. The proposed method is relatively simple to implement and does not require OOD validation data or architectural changes.

  3. The experiments cover several datasets and OOD detection methods. The improvements are consistent across most settings, with particularly substantial gains on CIFAR-10. The ablation studies also provide some evidence for the contributions of the individual components.

Critical weaknesses
  1. The research motivation and underlying mechanism are not sufficiently convincing. The authors attribute late-stage EMA degradation to reduced trajectory diversity, but the connection is not well established. Reduced diversity may explain why EMA gradually loses its advantage over the plain model, but it does not necessarily explain why EMA's own OOD performance deteriorates. The theoretical analysis does not directly establish this relationship, and the empirical evidence is largely correlational.

  2. The proposed method lacks a clear methodological justification. ReDiv combines several existing techniques, including rewinding, fixed learning rate, and SAM. Although the combination improves performance, it remains unclear why these particular interventions should address the identified problem. The improvements could also result from the known benefits of SAM or changes to the learning rate schedule, independently of the proposed diversity explanation.

  3. The generalizability of the findings is unclear. All experiments use ResNet-18 with SGD and cosine learning rate decay. It is not clear whether the reported degradation occurs under other architectures, optimizers, or EMA configurations. The magnitude of improvement also varies considerably across datasets, with much smaller gains on CIFAR-100 than CIFAR-10. The paper provides limited insight into when the proposed method should be expected to work.

Questions for authors

My question is mainly on the motivation and theoretical insight of the proposed approach: What explains the substantial variation in improvement across datasets? Are there identifiable characteristics of the training dynamics or data distributions that predict when ReDiv will be effective?

How does ReDiv compare with standard training using SAM or alternative learning rate schedules under matched computational budgets? Could these alternatives achieve similar improvements without rewinding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.