P-DORM: Parameter-Wise Dynamic Optimizer-State Reset for Test-Time Adaptation

Test-time adaptation (TTA) adapts deployed models to unlabeled test streams by optimizing unsupervised objectives on incoming batches. These updates commonly rely on optimizers that accumulate gradient history. We find that this history can become stale after distribution shifts and hinder subsequent adaptation. We therefore propose the Parameter-wise Dynamic Optimizer-State Reset Mechanism (P-DORM), a lightweight wrapper that detects stale history by comparing the current gradient with the historical direction stored by the optimizer. We prove that raw directional alignment is not comparable across tensors because its center and fluctuation depend on tensor dimension and directional geometry. P-DORM therefore standardizes each tensor's alignment against its own recent history. When parameter-aware thresholds identify unusually large departures, P-DORM rebuilds the accumulated update history of the affected parameter tensors without reverting adapted model parameters or modifying the adaptation objective or network architecture. Extensive experiments across corruption and natural distribution shifts, multiple TTA methods, optimizer families, architectures, and test-stream protocols demonstrate improved adaptation performance and broad plug-and-play compatibility.