- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper starts from the observation that, under domain shift, the history that an optimizer relies on can become unreliable. To address this, the authors propose a plug-in, P-DORM, which measures the cosine similarity between the gradient and the stored momentum and resets the momentum when the resulting standardized score Z exceeds a threshold. In addition, instead of using a fixed threshold, the authors propose a dynamic threshold based on online statistics and parameter size. In experiments across various backbones, datasets, adaptation scenarios, and baselines, P-DORM shows consistent performance improvements.
Critical strengths
(1) Clear motivation: The paper has a clear and intuitively plausible motivation. Optimizer history, such as accumulated momentum, implicitly assumes that all past gradients come from the same data distribution, and this assumption no longer holds in environments where domain shift occurs. Through the Boundary-Oracle experiment (Figure 1), the paper demonstrates that this problem also exists in the TTA setting.
(2) Consistent gain: The paper covers a substantial portion of the available TTA evaluation protocols and reports fairly consistent accuracy improvements given this broad coverage.
Critical weaknesses
(1) Limited verification: As the authors themselves state (Sec. 3.2, Appendix. H.1), thresholding based on sample- or batch-wise measurements may struggle to distinguish within-distribution variation from variation caused by domain shift. To address this, the authors propose standardization and dynamic thresholding based on statistics accumulated over EMA and a sliding window. However, no evidence is presented that this method actually distinguishes within-distribution gradient deviations from global deviations caused by domain shift. The analysis section is devoted mostly to outcome-level (end-metric) analyses such as sensitivity studies, and diagnostic or mechanistic verification of the proposed method is lacking.
(2) Heavy reliance on heuristics: A substantial part of the proposed parameter-aware threshold allocation relies on heuristics. (a) The underlying hypothesis that “smaller modules are more sensitive to domain shift” is only weakly supported by the ablation study in Table 5(a) and Appendix C, and it is adopted without sufficiently strong support, such as a layer-wise sensitivity analysis. (b) The values of several hyperparameters introduced in Eq. (10) (lambda, k_min, k_max) are chosen arbitrarily, and no sufficient sensitivity analysis is provided for lambda and the clipping width (+-0.5).
(3) Lack of optimizer control baselines: Although the main results and the appendix show that P-DORM is effective regardless of the optimizer (SGD, Adam, etc.), and a comparison with a momentum-free optimizer is also provided, negative controls that isolate the effect of the proposed detection are lacking. Specifically, comparing P-DORM with optimizers whose states are reset randomly or periodically would demonstrate its effectiveness more clearly.
(4) Missing discussion of closely related work: GALA (Sahoo et al., AAAI 2025) also measures, per layer, the cosine alignment between the current update and the accumulated parameter displacement in TTA. Although GALA rejects misaligned updates rather than resetting optimizer states, it is the closest TTA work that uses a history-alignment signal. Discussing this work would help position both the detection signal and the standardization contribution.
- Ref) Sahoo, Sabyasachi, et al. "A layer selection approach to test time adaptation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 19. 2025.
Questions for authors
(1) Please provide a quantitative analysis of whether the proposed standardization and dynamic thresholding can indeed distinguish within-domain gradient deviations from gradient deviations caused by domain shift.
(2) Regarding the premise of the parameter-aware threshold allocation, could you provide verification of this premise through a layer-wise domain-shift analysis or additional experiments?
(3) Please provide a comparison between P-DORM and optimizers whose states are reset randomly or periodically. If these controls yield the expected negative results, I expect this would make the contribution of P-DORM more convincing.
(4) The SGD experiments in the paper use momentum = 0.98. To my knowledge, this differs from the commonly used setting of momentum = 0.9. Is there a particular reason for this choice? Is the gain preserved under the standard SGD setting?
I would consider raising my score if these concerns are addressed.
4 comments
Thank you for your thorough response and clear explanations. Below, I detail whether each of my concerns has been addressed.
Transition-related versus within-domain responses (W1/Q1): Largely addressed The authors showed that the proposed detector detected most of the transitions that occurred in the actual experiments and also reduced the false-positive rate. This suggests that the proposed detector is clearly superior in its role of detecting domain shifts. Accordingly, I consider this concern largely addressed.
Testing the size prior and its hyperparameters (W2/Q2): Largely addressed The authors originally hypothesized that smaller modules are more sensitive to distribution shifts and used this hypothesis to motivate the parameter-aware threshold allocation, but their correlation analysis shows only a weak link between module size and sensitivity.
However, this does not invalidate the mechanism proposed in the main paper. As an alternative, the authors present an intervention-scope prior, i.e., that larger modules are more strongly affected by clearing the history, which I think it seems plausible and sufficiently supported by Table 5(a) of the main paper.
In addition, the authors conducted additional hyperparameter sweeps. Although some of the analyses, such as the λ sensitivity analysis, do not guarantee the global optimality of the default setting, the authors state this explicitly, and they show that the hyperparameters except λ are optimal at the default setting.
Taken together, I regard this concern as largely addressed as well.
Momentum choice and the standard 0.9 setting (Q4): Partially addressed The authors explain that they set the SGD momentum higher than the standard setting in order to lengthen the effective history. According to the results they provide, the improvement from P-DORM is reduced with a shorter history (momentum = 0.9) compared to momentum = 0.98.
Given that P-DORM acts mainly on the gradient history stored in the optimizer, this is a natural consequence. However, it raises the question of whether P-DORM is necessary for SGD with the standard setting, since such an optimizer already forgets its history at an appropriate rate. In other words, the momentum coefficient of SGD may already be performing part of the role of P-DORM.
Remaining concern and questions. Nevertheless, it is undeniable that P-DORM shows consistent gains in the momentum = 0.98 experiments. Therefore, my remaining concern is not about the performance or novelty of P-DORM, but about the scope of its claims and its practical necessity.
- (1) Is the error rate of P-DORM with momentum = 0.98 lower than that of the baselines (i.e., without P-DORM) with momentum = 0.9?
- (2) If not, could you refine the claim of plug-and-play compatibility across optimizers to explicitly state the condition under which the gain appears (i.e., a long optimizer history)?
Relationship to GALA (W4): Fully addressed The authors clearly explain how P-DORM differs from GALA. As they have committed to including this discussion in the revision, this concern has been fully addressed.
Periodic and random optimizer-state reset controls (W3/Q3): Largely addressed The optimizer-control experiments provided by the authors serve as sufficiently strong negative controls for the validity of P-DORM. They show that the effect of P-DORM does not come from simply clearing the optimizer history. Accordingly, I consider this concern largely addressed.
(2026.11.15.) Please note that part of my comment on Q4 has been edited.
Thank you for the careful follow-up and for recognizing the additional evidence. We answer your two remaining questions below and provide new results that clarify P-DORM's practical scope.
1. P-DORM at momentum 0.98 versus the baseline at 0.9
We compare P-DORM with SGD momentum 0.98 against Pure SGD with momentum 0.9 and report multi-seed mean error (%).
| Configuration | Pure SGD, momentum 0.9 | P-DORM + SGD, momentum 0.98 |
|---|---|---|
| ImageNet-C / EATA-C / correlated | 66.51 | 66.09 |
| ImageNet-C / RDumb / mixed | 79.90 | 79.05 |
| CIFAR-10-C / BufferTTA / CTTA | 20.12 | 19.48 |
| CIFAR-100-C / CoME / CTTA | 34.09 | 33.52 |
| CIFAR-10-C / CoME / CTTA | 18.45 | 18.63 |
| CIFAR-100-C / EATA-C / correlated | 40.80 | 40.97 |
| CIFAR-100-C / Tent / CTTA | 32.34 | 32.82 |
As shown, P-DORM with momentum 0.98 outperforms Pure SGD with momentum 0.9 in several configurations, reducing error by 0.42–0.85 pp. In the other listed configurations, its error is higher by 0.17–0.48 pp. These results support momentum 0.98 as a meaningful long-history setting for evaluating P-DORM, with practical gains in several configurations but no uniform superiority over the 0.9 baseline.
2. Benefits across optimizers with standard momentum settings
We additionally evaluated AdaBelief [1] and Lion [2] with their default beta coefficients, (0.9, 0.999) and (0.9, 0.99), respectively, alongside RMSProp-M [3] with momentum 0.9 and squared-gradient decay 0.99, retaining the configuration specified in the paper. No momentum or decay coefficient was increased for these additional experiments.
Results on CIFAR-100-C (Pure / P-DORM error %, 3 paired seeds; lower is better):
| CIFAR-100-C configuration | RMSProp-M | AdaBelief | Lion |
|---|---|---|---|
| DeYO / CTTA | 31.41 / 31.35 | 31.48 / 31.24 | 32.64 / 32.29 |
| Tent / CTTA | 32.45 / 32.29 | 32.28 / 31.99 | 33.54 / 33.14 |
| DeYO / correlated | 40.73 / 40.64 | 40.68 / 40.50 | 41.96 / 41.42 |
In the nine comparisons shown, P-DORM reduces mean error by 0.06–0.53 pp, demonstrating practical gains across three optimizers with standard momentum settings. These optimizers use distinct update rules and state structures: RMSProp-M and AdaBelief maintain additional second-moment or belief-variance states for adaptive scaling, whereas Lion uses sign-based updates with a single momentum buffer. The improvements in the reported configurations, obtained without increasing their momentum or decay coefficients, support P-DORM’s practical value across different optimizer designs, rather than only in the deliberately long-history SGD regime.
3. Clarifying the compatibility claim
We agree with your suggestion to clarify the scope of our claims. P-DORM provides plug-and-play compatibility with all evaluated stateful optimizers, requiring no changes to the TTA objective or network architecture. Our experiments demonstrate practical gains in the reported optimizer configurations, ranging from long-history SGD to the standard-momentum settings of RMSProp-M, AdaBelief, and Lion.
We will therefore refine our claim in the revision to state that P-DORM offers empirical benefits across the evaluated stateful optimizers in the reported configurations, with stronger SGD gains observed in the tested long-history regime.
References
[1] Zhuang et al. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. NeurIPS, 2020.
[2] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS, 2023.
[3] Tieleman and Hinton. Lecture 6.5—RMSProp: Divide the Gradient by a Running Average of Its Recent Magnitude. Neural Networks for Machine Learning, Coursera lecture notes, 2012.
I am sincerely grateful for your supplementary explanation. The authors show that P-DORM outperforms SGD with the standard setting on ImageNet-C. Furthermore, the additional evaluation with other optimizers makes the claim that P-DORM is broadly applicable across optimizers more convincing. Hence, all of my concerns, including Q4, have been largely or fully addressed.
Taking the above experiments into account, the gain of P-DORM is somewhat marginal. However, since P-DORM acts on the optimizer state rather than on the adaptation objective or the model itself, I do not consider this a critical flaw. I believe its consistency across various backbones, datasets, optimizers, and adaptation scenarios is more valuable than the magnitude of its gain. Therefore, I have raised my score to weak acceptance.
Thank you for your positive and constructive feedback. Please find our detailed responses below.
1. Transition-related versus within-domain responses (W1/Q1)
We analyzed the detector on ImageNet-C/Tent and CIFAR-100-C/DeYO (5 seeds). Table KqY6-1. Detector diagnostics The transition window (first 5 updates) captures 68/70 and 69/70 transitions. Transition trigger rates (2.52%/3.18%) clearly exceed stable rates (1.33%/1.14%), confirming boundary sensitivity.
Mechanistic controls. Table KqY6-2 replays decision rules with uniform coefficients (λ=0).
First, standardization (Z) nearly doubles the boundary trigger rate vs. raw cosine (e.g., 3.21% → 6.18% for SGD), elevating the contrast ratio from ~4.5× to 6.44×.
Second, although replacing the adaptive window with a fixed threshold on Z yields higher boundary triggers (7.67%), it is due to indiscriminate false alarms: the stable-window trigger rate surges by ~30% (0.96% → 1.32%). These spurious resets destroy useful history and cause negative transfer. Our adaptive window filters out this noise, achieving the highest contrast ratio (6.44×). At λ=0.5, this ordering remains robust (5.79× for Z vs. 4.50× for raw cosine).
2. Testing the size prior and its hyperparameters (W2/Q2)
Transition magnitude and reset benefit are distinct. Table KqY6-3 shows size is not a universal predictor (ρ(size, ΔZ) ranges from −0.22 to +0.02) and correlates strongly with depth. We therefore formulate allocation as an intervention-scope prior: resetting larger tensors clears more coordinates, so conservative thresholds limit collateral damage. Table 5(a) confirms Small (our default) beats Fixed by 0.39, 0.06, 0.39, and 0.22 pp across four configurations. We will revise Section 3.3 to reflect this rationale.
Hyperparameter sensitivity. Figure R10 and Figure R11 show sweeps of λ and clipping width h. Maximum deviations are 0.13 pp on ImageNet-C and 0.04 pp on CIFAR-100-C. Widening bounds to h=0.75 does not degrade performance, confirming robustness.
3. Momentum choice and the standard 0.9 setting (Q4)
Memory decays as μj: μ=0.98 retains a theoretical memory scale of ~50 updates vs. 10 at μ=0.9. After 10 updates, ~82% vs. 35% of a pre-existing buffer remains. We evaluate μ=0.98 as the long-history setting. Table KqY6-4. Momentum comparison At μ=0.9, P-DORM does not degrade the baseline (e.g., Tent: −0.004 pp; DeYO/BufferTTA: +0.072/+0.078 pp, within seed variance). Short memory rarely has stale history, so the detector correctly suppresses intervention.
At μ=0.98, P-DORM delivers consistent reductions (0.786, 0.450, 0.382 pp). 0.98 does not uniformly weaken the baseline (Pure performs better at 0.98 on CIFAR-10-C/BufferTTA). The benefit scales with memory length: P-DORM targets long-history inertia while remaining safe when history is short.
4. Relationship to GALA (W4)
GALA selects layer updates using a shared raw-cosine threshold. P-DORM compares gradients with the pre-update direction and reconstructs state. The distinction is update selection versus optimizer-state reconstruction. Furthermore, P-DORM standardizes drops with tensor-specific online statistics, addressing tensor-dependent variability rather than assuming a uniform raw-cosine drop. We will add this to the revision.
5. Periodic and random optimizer-state reset controls (W3/Q3)
We evaluate PeriodicAll (resets all tensors every Kper updates) and RandomSparse (resets each tensor with probability 1/Ksp), retaining the base method. As strong baselines, we use high-frequency schedules (Kper=32,Ksp=10) found competitive in preliminary validation.
Table KqY6-5. State-reset examples
P-DORM requires only 424–1,392 targeted resets. For inferior results, PeriodicAll needs 2.79–3.96× more resets, and RandomSparse needs 9.04–12.72× more. Blind schedules clear states indiscriminately, destroying useful history and causing negative transfer. P-DORM intervenes only when the gradient-history mismatch is statistically significant, demonstrating a clear efficiency–accuracy advantage.