Understanding and Mitigating Late-Stage OOD Detection Degradation in EMA Models

Exponential moving average (EMA) is widely used to improve model generalization, but its behavior in out-of-distribution (OOD) detection is not uniformly beneficial throughout training. We observe a consistent late-stage degradation phenomenon: EMA improves substantially during early and middle training, but its OOD performance later declines, while the plain model remains competitive and gradually closes the gap. Our analysis suggests that this behavior is associated with increasing redundancy in the late-stage plain-model trajectory, which reduces the complementary information available to EMA. We propose ReDiv, a simple Rewind-and-Diversify framework for EMA-based OOD detection. ReDiv first detects the onset of late-stage degradation using an ID-only logit-margin signal and rewinds the complete training state to the corresponding checkpoint. It then diversifies the post-rewind trajectory by fixing the learning rate to avoid further scheduler-induced shrinkage and applying sharpness-aware minimization (SAM) to modify the effective update. Across multiple datasets and OOD benchmarks, ReDiv consistently improves over both plain training and standard EMA, requires no OOD validation data or architectural modification, and significantly complements existing training-time and test-time OOD methods. Through careful investigation, we show that the effectiveness of EMA depends not only on checkpoint quality, also on the diversity of the model trajectory, which verifies the late-stage trajectory control in ReDiv as a powerful way to improve performance.