acceptodds
Under review as a conference paper at ICLR 2027

P-DORM: Parameter-Wise Dynamic Optimizer-State Reset for Test-Time Adaptation

Abstract

Test-time adaptation (TTA) adapts deployed models to unlabeled test streams by optimizing unsupervised objectives on incoming batches. These updates commonly rely on optimizers that accumulate gradient history. We find that this history can become stale after distribution shifts and hinder subsequent adaptation. We therefore propose the Parameter-wise Dynamic Optimizer-State Reset Mechanism (P-DORM), a lightweight wrapper that detects stale history by comparing the current gradient with the historical direction stored by the optimizer. We prove that raw directional alignment is not comparable across tensors because its center and fluctuation depend on tensor dimension and directional geometry. P-DORM therefore standardizes each tensor's alignment against its own recent history. When parameter-aware thresholds identify unusually large departures, P-DORM rebuilds the accumulated update history of the affected parameter tensors without reverting adapted model parameters or modifying the adaptation objective or network architecture. Extensive experiments across corruption and natural distribution shifts, multiple TTA methods, optimizer families, architectures, and test-stream protocols demonstrate improved adaptation performance and broad plug-and-play compatibility.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Optimizer, Test-time adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KqY6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper starts from the observation that, under domain shift, the history that an optimizer relies on can become unreliable. To address this, the authors propose a plug-in, P-DORM, which measures the cosine similarity between the gradient and the stored momentum and resets the momentum when the resulting standardized score Z exceeds a threshold. In addition, instead of using a fixed threshold, the authors propose a dynamic threshold based on online statistics and parameter size. In experiments across various backbones, datasets, adaptation scenarios, and baselines, P-DORM shows consistent performance improvements.

Critical strengths

(1) Clear motivation: The paper has a clear and intuitively plausible motivation. Optimizer history, such as accumulated momentum, implicitly assumes that all past gradients come from the same data distribution, and this assumption no longer holds in environments where domain shift occurs. Through the Boundary-Oracle experiment (Figure 1), the paper demonstrates that this problem also exists in the TTA setting.

(2) Consistent gain: The paper covers a substantial portion of the available TTA evaluation protocols and reports fairly consistent accuracy improvements given this broad coverage.

Critical weaknesses

(1) Limited verification: As the authors themselves state (Sec. 3.2, Appendix. H.1), thresholding based on sample- or batch-wise measurements may struggle to distinguish within-distribution variation from variation caused by domain shift. To address this, the authors propose standardization and dynamic thresholding based on statistics accumulated over EMA and a sliding window. However, no evidence is presented that this method actually distinguishes within-distribution gradient deviations from global deviations caused by domain shift. The analysis section is devoted mostly to outcome-level (end-metric) analyses such as sensitivity studies, and diagnostic or mechanistic verification of the proposed method is lacking.

(2) Heavy reliance on heuristics: A substantial part of the proposed parameter-aware threshold allocation relies on heuristics. (a) The underlying hypothesis that “smaller modules are more sensitive to domain shift” is only weakly supported by the ablation study in Table 5(a) and Appendix C, and it is adopted without sufficiently strong support, such as a layer-wise sensitivity analysis. (b) The values of several hyperparameters introduced in Eq. (10) (lambda, k_min, k_max) are chosen arbitrarily, and no sufficient sensitivity analysis is provided for lambda and the clipping width (+-0.5).

(3) Lack of optimizer control baselines: Although the main results and the appendix show that P-DORM is effective regardless of the optimizer (SGD, Adam, etc.), and a comparison with a momentum-free optimizer is also provided, negative controls that isolate the effect of the proposed detection are lacking. Specifically, comparing P-DORM with optimizers whose states are reset randomly or periodically would demonstrate its effectiveness more clearly.

(4) Missing discussion of closely related work: GALA (Sahoo et al., AAAI 2025) also measures, per layer, the cosine alignment between the current update and the accumulated parameter displacement in TTA. Although GALA rejects misaligned updates rather than resetting optimizer states, it is the closest TTA work that uses a history-alignment signal. Discussing this work would help position both the detection signal and the standardization contribution.

  • Ref) Sahoo, Sabyasachi, et al. "A layer selection approach to test time adaptation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 19. 2025.
Questions for authors

(1) Please provide a quantitative analysis of whether the proposed standardization and dynamic thresholding can indeed distinguish within-domain gradient deviations from gradient deviations caused by domain shift.

(2) Regarding the premise of the parameter-aware threshold allocation, could you provide verification of this premise through a layer-wise domain-shift analysis or additional experiments?

(3) Please provide a comparison between P-DORM and optimizers whose states are reset randomly or periodically. If these controls yield the expected negative results, I expect this would make the contribution of P-DORM more convincing.

(4) The SGD experiments in the paper use momentum = 0.98. To my knowledge, this differs from the commonly used setting of momentum = 0.9. Is there a particular reason for this choice? Is the gain preserved under the standard SGD setting?

I would consider raising my score if these concerns are addressed.

4 comments

Thank you for your positive and constructive feedback. Please find our detailed responses below.

1. Transition-related versus within-domain responses (W1/Q1)

We analyzed the detector on ImageNet-C/Tent and CIFAR-100-C/DeYO (5 seeds). Table KqY6-1. Detector diagnostics The transition window (first 5 updates) captures 68/70 and 69/70 transitions. Transition trigger rates (2.52%/3.18%) clearly exceed stable rates (1.33%/1.14%), confirming boundary sensitivity.

Mechanistic controls. Table KqY6-2 replays decision rules with uniform coefficients (λ=0\lambda=0).

First, standardization (ZZ) nearly doubles the boundary trigger rate vs. raw cosine (e.g., 3.21% →\to 6.18% for SGD), elevating the contrast ratio from ~4.5× to 6.44×.

Second, although replacing the adaptive window with a fixed threshold on ZZ yields higher boundary triggers (7.67%), it is due to indiscriminate false alarms: the stable-window trigger rate surges by ~30% (0.96% →\to 1.32%). These spurious resets destroy useful history and cause negative transfer. Our adaptive window filters out this noise, achieving the highest contrast ratio (6.44×). At λ=0.5\lambda=0.5, this ordering remains robust (5.79× for Z vs. 4.50× for raw cosine).

2. Testing the size prior and its hyperparameters (W2/Q2)

Transition magnitude and reset benefit are distinct. Table KqY6-3 shows size is not a universal predictor (ρ\rho(size, ΔZ\Delta Z) ranges from −0.22 to +0.02) and correlates strongly with depth. We therefore formulate allocation as an intervention-scope prior: resetting larger tensors clears more coordinates, so conservative thresholds limit collateral damage. Table 5(a) confirms Small (our default) beats Fixed by 0.39, 0.06, 0.39, and 0.22 pp across four configurations. We will revise Section 3.3 to reflect this rationale.

Hyperparameter sensitivity. Figure R10 and Figure R11 show sweeps of λ\lambda and clipping width hh. Maximum deviations are 0.13 pp on ImageNet-C and 0.04 pp on CIFAR-100-C. Widening bounds to h=0.75h=0.75 does not degrade performance, confirming robustness.

3. Momentum choice and the standard 0.9 setting (Q4)

Memory decays as μj\mu^j: μ=0.98\mu=0.98 retains a theoretical memory scale of ~50 updates vs. 10 at μ=0.9\mu=0.9. After 10 updates, ~82% vs. 35% of a pre-existing buffer remains. We evaluate μ=0.98\mu=0.98 as the long-history setting. Table KqY6-4. Momentum comparison At μ=0.9\mu=0.9, P-DORM does not degrade the baseline (e.g., Tent: −0.004 pp; DeYO/BufferTTA: +0.072/+0.078 pp, within seed variance). Short memory rarely has stale history, so the detector correctly suppresses intervention.

At μ=0.98\mu=0.98, P-DORM delivers consistent reductions (0.786, 0.450, 0.382 pp). 0.98 does not uniformly weaken the baseline (Pure performs better at 0.98 on CIFAR-10-C/BufferTTA). The benefit scales with memory length: P-DORM targets long-history inertia while remaining safe when history is short.

4. Relationship to GALA (W4)

GALA selects layer updates using a shared raw-cosine threshold. P-DORM compares gradients with the pre-update direction and reconstructs state. The distinction is update selection versus optimizer-state reconstruction. Furthermore, P-DORM standardizes drops with tensor-specific online statistics, addressing tensor-dependent variability rather than assuming a uniform raw-cosine drop. We will add this to the revision.

5. Periodic and random optimizer-state reset controls (W3/Q3)

We evaluate PeriodicAll (resets all tensors every KperK_{\text{per}} updates) and RandomSparse (resets each tensor with probability 1/Ksp1/K_{\text{sp}}), retaining the base method. As strong baselines, we use high-frequency schedules (Kper=32,Ksp=10K_{\text{per}}=32, K_{\text{sp}}=10) found competitive in preliminary validation.

Table KqY6-5. State-reset examples

P-DORM requires only 424–1,392 targeted resets. For inferior results, PeriodicAll needs 2.79–3.96× more resets, and RandomSparse needs 9.04–12.72× more. Blind schedules clear states indiscriminately, destroying useful history and causing negative transfer. P-DORM intervenes only when the gradient-history mismatch is statistically significant, demonstrating a clear efficiency–accuracy advantage.

Thank you for your thorough response and clear explanations. Below, I detail whether each of my concerns has been addressed.

Transition-related versus within-domain responses (W1/Q1): Largely addressed The authors showed that the proposed detector detected most of the transitions that occurred in the actual experiments and also reduced the false-positive rate. This suggests that the proposed detector is clearly superior in its role of detecting domain shifts. Accordingly, I consider this concern largely addressed.

Testing the size prior and its hyperparameters (W2/Q2): Largely addressed The authors originally hypothesized that smaller modules are more sensitive to distribution shifts and used this hypothesis to motivate the parameter-aware threshold allocation, but their correlation analysis shows only a weak link between module size and sensitivity.

However, this does not invalidate the mechanism proposed in the main paper. As an alternative, the authors present an intervention-scope prior, i.e., that larger modules are more strongly affected by clearing the history, which I think it seems plausible and sufficiently supported by Table 5(a) of the main paper.

In addition, the authors conducted additional hyperparameter sweeps. Although some of the analyses, such as the λ sensitivity analysis, do not guarantee the global optimality of the default setting, the authors state this explicitly, and they show that the hyperparameters except λ are optimal at the default setting.

Taken together, I regard this concern as largely addressed as well.

Momentum choice and the standard 0.9 setting (Q4): Partially addressed The authors explain that they set the SGD momentum higher than the standard setting in order to lengthen the effective history. According to the results they provide, the improvement from P-DORM is reduced with a shorter history (momentum = 0.9) compared to momentum = 0.98.

Given that P-DORM acts mainly on the gradient history stored in the optimizer, this is a natural consequence. However, it raises the question of whether P-DORM is necessary for SGD with the standard setting, since such an optimizer already forgets its history at an appropriate rate. In other words, the momentum coefficient of SGD may already be performing part of the role of P-DORM.

Remaining concern and questions. Nevertheless, it is undeniable that P-DORM shows consistent gains in the momentum = 0.98 experiments. Therefore, my remaining concern is not about the performance or novelty of P-DORM, but about the scope of its claims and its practical necessity.

  • (1) Is the error rate of P-DORM with momentum = 0.98 lower than that of the baselines (i.e., without P-DORM) with momentum = 0.9?
  • (2) If not, could you refine the claim of plug-and-play compatibility across optimizers to explicitly state the condition under which the gain appears (i.e., a long optimizer history)?

Relationship to GALA (W4): Fully addressed The authors clearly explain how P-DORM differs from GALA. As they have committed to including this discussion in the revision, this concern has been fully addressed.

Periodic and random optimizer-state reset controls (W3/Q3): Largely addressed The optimizer-control experiments provided by the authors serve as sufficiently strong negative controls for the validity of P-DORM. They show that the effect of P-DORM does not come from simply clearing the optimizer history. Accordingly, I consider this concern largely addressed.

(2026.11.15.) Please note that part of my comment on Q4 has been edited.

Thank you for the careful follow-up and for recognizing the additional evidence. We answer your two remaining questions below and provide new results that clarify P-DORM's practical scope.

1. P-DORM at momentum 0.98 versus the baseline at 0.9

We compare P-DORM with SGD momentum 0.98 against Pure SGD with momentum 0.9 and report multi-seed mean error (%).

ConfigurationPure SGD, momentum 0.9P-DORM + SGD, momentum 0.98
ImageNet-C / EATA-C / correlated66.5166.09
ImageNet-C / RDumb / mixed79.9079.05
CIFAR-10-C / BufferTTA / CTTA20.1219.48
CIFAR-100-C / CoME / CTTA34.0933.52
CIFAR-10-C / CoME / CTTA18.4518.63
CIFAR-100-C / EATA-C / correlated40.8040.97
CIFAR-100-C / Tent / CTTA32.3432.82

As shown, P-DORM with momentum 0.98 outperforms Pure SGD with momentum 0.9 in several configurations, reducing error by 0.42–0.85 pp. In the other listed configurations, its error is higher by 0.17–0.48 pp. These results support momentum 0.98 as a meaningful long-history setting for evaluating P-DORM, with practical gains in several configurations but no uniform superiority over the 0.9 baseline.

2. Benefits across optimizers with standard momentum settings

We additionally evaluated AdaBelief [1] and Lion [2] with their default beta coefficients, (0.9, 0.999) and (0.9, 0.99), respectively, alongside RMSProp-M [3] with momentum 0.9 and squared-gradient decay 0.99, retaining the configuration specified in the paper. No momentum or decay coefficient was increased for these additional experiments.

Results on CIFAR-100-C (Pure / P-DORM error %, 3 paired seeds; lower is better):

CIFAR-100-C configurationRMSProp-MAdaBeliefLion
DeYO / CTTA31.41 / 31.3531.48 / 31.2432.64 / 32.29
Tent / CTTA32.45 / 32.2932.28 / 31.9933.54 / 33.14
DeYO / correlated40.73 / 40.6440.68 / 40.5041.96 / 41.42

In the nine comparisons shown, P-DORM reduces mean error by 0.06–0.53 pp, demonstrating practical gains across three optimizers with standard momentum settings. These optimizers use distinct update rules and state structures: RMSProp-M and AdaBelief maintain additional second-moment or belief-variance states for adaptive scaling, whereas Lion uses sign-based updates with a single momentum buffer. The improvements in the reported configurations, obtained without increasing their momentum or decay coefficients, support P-DORM’s practical value across different optimizer designs, rather than only in the deliberately long-history SGD regime.

3. Clarifying the compatibility claim

We agree with your suggestion to clarify the scope of our claims. P-DORM provides plug-and-play compatibility with all evaluated stateful optimizers, requiring no changes to the TTA objective or network architecture. Our experiments demonstrate practical gains in the reported optimizer configurations, ranging from long-history SGD to the standard-momentum settings of RMSProp-M, AdaBelief, and Lion.

We will therefore refine our claim in the revision to state that P-DORM offers empirical benefits across the evaluated stateful optimizers in the reported configurations, with stronger SGD gains observed in the tested long-history regime.

References

[1] Zhuang et al. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. NeurIPS, 2020.

[2] Chen et al. Symbolic Discovery of Optimization Algorithms. NeurIPS, 2023.

[3] Tieleman and Hinton. Lecture 6.5—RMSProp: Divide the Gradient by a Running Average of Its Recent Magnitude. Neural Networks for Machine Learning, Coursera lecture notes, 2012.

I am sincerely grateful for your supplementary explanation. The authors show that P-DORM outperforms SGD with the standard setting on ImageNet-C. Furthermore, the additional evaluation with other optimizers makes the claim that P-DORM is broadly applicable across optimizers more convincing. Hence, all of my concerns, including Q4, have been largely or fully addressed.

Taking the above experiments into account, the gain of P-DORM is somewhat marginal. However, since P-DORM acts on the optimizer state rather than on the adaptation objective or the model itself, I do not consider this a critical flaw. I believe its consistency across various backbones, datasets, optimizers, and adaptation scenarios is more valuable than the magnitude of its gain. Therefore, I have raised my score to weak acceptance.

Reviewer_eeJrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the persistence of accumulated optimizer states (e.g., momentum buffers in SGD, first- and second-moment buffers in Adam) during Test-Time Adaptation (TTA). The paper make a hypothesis that when deployed models encounter non-stationary test streams (abrupt or gradual distribution shifts), stale historical gradient memory causes tracking lag and optimization inertia. To address this issue, the paper proposes P-DORM, a plug-and-play optimizer wrapper that computes cosine alignment between the current gradient and accumulated optimizer states, standardizes it online (Z^t) to account for dimension-dependent variance (1/d), and triggers tensor-specific optimizer resets using module-size-scaled thresholds (k_l) without reverting model weights. The method is evaluated across four benchmarks (CIFAR-10-C, CIFAR-100-C, ImageNet-C, DomainNet-126), four backbones (WideResNet-28-10, ResNeXt-29, ResNet-50, ViT-B/16), six TTA methods (TENT, DeYO, CoME, BufferTTA, EATA-C, RDumb), five optimizers (Adam, SGD, RMSProp, AdaBelief, Lion), and five adaptation protocols (TTA, CTTA, GTTA, PTTA, Mixed).

Critical strengths
  • The concept of investigating optimizer states as a distinct failure mode in continual test-time adaptation is insightful.
  • The Boundary-Oracle experiment cleanly demonstrates that stale optimizer memory impedes adaptation. The theoretical justification for online standardization reasonably proves that raw cosine variance scales inversely with nominal or effective parameter dimension.
  • The method is plug-and-play and comprenhensively evaluated across a wide range of benchmarks, backbones, adaptation objectives, and non-stationary protocols.
Critical weaknesses
  • The improvements of the method is marginal while it suffers from severe runtime overhead.
  • The performance is sensitive the the batch size and architecture. It significantly degrades when the batch size is smaller than 32. Moreover, for transformer architectures adapting LayerNorm parameters (e.g., ViT-B/16), all monitored tensors share identical dimensions. Consequently, normalized rank weights collapse to w_m=1/2, causing the parameter-aware allocation heuristic to reduce entirely to a uniform threshold. This may limit the utility of this method in practice.
  • The results reported in Tables 2&3 are slightly different from those in Table 7.
  • While momentum-free SGD and Boundary-Oracle are evaluated, the comparison misses the trivial baseline of periodic reset. Without this, it is unclear whether dynamic anomaly detection outperforms simple scheduled cache clearing.
  • P-DORM introduces multiple hyperparameters which are not completely verified.
Questions for authors
  • Please clarify why CIFAR-10-C (CTTA, DeYO), CIFAR-100-C (PTTA, TENT & DeYO), and ImageNet-C (CTTA/GTTA) report divergent numbers between Tables 2&3 and the Table 7.
  • Since TTA frequently handles batch sizes down to B=1, how can the gradient-momentum alignment signal be stabilized when gradient noise dominates? Would temporal filtering across consecutive batches prevent false triggers?
  • How does P-DORM compare to a periodic-reset baseline?
  • Can the 39% runtime overhead be mitigated by grouping tensors into layer blocks or applying fused kernels, and does tensor-grouping compromise detection sensitivity?
2 comments

Thank you for your positive feedback. Please find our detailed responses below.

1. Differences between Tables 2–3 and Table 7 (W3/Q1)

The inconsistency stems from different seed-level run records used for aggregation. Both are five-seed aggregations; Tables 2–3 use the reconciled main-result records, whereas Table 7's exporter read an earlier archive. We will use the main-table records as the canonical source for all repeated entries.

We checked every configuration named in the review. All entries below use momentum SGD and report error (%) as Pure/P-DORM:

Table eeJr-1. Repeated entries in Tables 2–3 and Table 7

The largest displayed discrepancy in Table 7 is 0.03 pp. Recomputing them preserves summary rankings: momentum-free SGD has the highest error in 16/18 settings, and P-DORM the lowest in 15/18. We will regenerate Table 7 from these canonical records so that all repeated Pure/P-DORM entries match Tables 2–3 exactly.

2. Runtime overhead and grouping (W1/Q4)

The overhead of P-DORM is implementation-level, not algorithmic: it adds tensor-wise bookkeeping, but no extra forward/backward pass and no trainable parameters. We evaluate tensor-wise batching and a fused kernel. All variants use Tent with Adam. The table reports full-step time (forward, backward, optimizer), measured with segment-wise synchronization on an RTX 4090.

Table eeJr-2. Runtime and grouping

The paper's 39.1% profiles the reference implementation (166 tensors, 5 optimizers). Our 32.0% measurement here uses Adam and 106 adapted tensors. The main results use the reference; the batched and fused paths are orthogonal deployment optimizations that preserve the algorithm exactly.

On ResNet-50, batching retains 106 detector units and cuts step time from 104.11 to 76.99 ms. Grouping reduces units to 53, saving 0.56 ms more. Most savings occur without coarsening the detector. On CIFAR-100-C, batching matches the reference (Δ\Delta error = +0.00 pp; 95% CI [−0.01, +0.02]); grouping adds a small variation (+0.04 pp). We treat grouping as a runtime option, not a substitute for tensor-wise detection.

The batched path brings the step to near the no-reset cost, showing the overhead is a property of the unoptimized reference, not inherent to P-DORM.

Custom fused kernel. We implemented a Triton kernel fusing dot products and squared-norm reductions, retaining 106 tensor-local detectors. It reduces optimizer-only latency by 35.7% (4.25 to 2.73 ms). The full-step gain is limited by the dominant forward/backward cost; the fused kernel specifically eliminates the optimizer-level cost P-DORM adds.

Table eeJr-3. Custom fused gather timing

In a 320-step closed-loop test, fused vs. batched execution gives 0/320 trigger-set disagreements (305 resets per arm), with identical final parameters and first-/second-moment states. Scores agree within 10−610^{-6}, preserving exact algorithmic behavior.

3. Small batches and equal-sized LayerNorm tensors (W2/Q2)

Small batches. Please refer to our response to Reviewer CcJS, §4.

Equal-sized LayerNorm tensors. ViT-B/16 monitors 25 LayerNorm modules (768 parameters each), yielding w=0.5w = 0.5 and k=2.5k = 2.5 throughout. The size prior contributes no differential allocation here. Nevertheless, a common coefficient is not a global threshold: each tensor retains its own score-window statistics and reset decision.

Figure R7 shows threshold means range from 2.66 to 2.88 due to distinct local statistics. Figure R8 shows the boundary-update trigger rate is 3.40% vs. 1.76% stable, demonstrating a clear transient spike. Existing ViT results already operate with these uniform coefficients, confirming the detection mechanism remains effective.

4. Small gains and their uncertainty (W1)

Please refer to our response to Reviewer CcJS, §1.

5. Periodic optimizer-state reset baseline (W4/Q3)

Please refer to our response to Reviewer KqY6, §5.

6. Hyperparameter verification (W5)

Please refer to our response to Reviewer KqY6, §2.

I thank the authors for your feedback. I keep my initial rating.

Reviewer_CcJSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes P-DORM, a plug-and-play wrapper that addresses stale optimizer history in test-time adaptation. It detects incompatibility between the current gradient and the optimizer's accumulated direction via cosine alignment, standardizes this signal with tensor-specific online statistics, and resets the corresponding tensor's optimizer state when a parameter-aware threshold is exceeded — without reverting the adapted model parameters. The paper also provides an appendix argument that raw alignment is not comparable across tensors. Experiments span four datasets, six TTA methods, five test-stream protocols, multiple backbones (ResNet/ViT) and optimizers (Adam/SGD), reporting average error reductions of roughly 0.05–0.48 percentage points.

Critical strengths

The paper identifies a genuinely under-explored failure mode and validates it systematically. The observation that optimizer history becomes stale after distribution shifts — and that resetting it at known domain boundaries consistently helps (the Boundary-Oracle study across five optimizers) — is a useful and clearly demonstrated insight for the TTA community. The experimental breadth (four datasets, six methods, five stream protocols, multiple backbones and optimizers, five seeds) is commendable and gives the central hypothesis credible empirical support.

Critical weaknesses
  1. The empirical gain is marginal and its significance is not established. The reported improvements are on the order of 0.05–0.48 pp, and the paper's own text notes that among 60 method-level averages, four are worse by up to 0.09 pp — i.e., the improvements and the regressions are of the same magnitude. Moreover, the tables report five-seed means without standard deviations, confidence intervals, or paired significance tests (contrast with comparable CTTA papers that report paired intervals). As a result, it is not possible to assess whether the 0.05–0.48 pp gains exceed seed-level variance. For a wrapper whose entire contribution is an incremental performance improvement, the absence of a significance assessment is a material soundness gap.

  2. The contribution is an engineering increment whose theoretical component restates known results. The central theoretical claim (Appendix B, Proposition B.1) — that the variance of the cosine alignment between two spherically symmetric random vectors is 1/d — is a standard result in high-dimensional probability / random geometry (the variance of a single coordinate of a uniform point on the unit sphere is 1/d). The follow-up propositions (B.2/B.3) rest on the delta method and the participation-ratio effective dimension, which are likewise standard. These are presented in the contributions as "we prove," but they appear to re-derive known results rather than establish new theory. Combined with the fact that optimizer-state management has been explored before (adaptive restart, cautious optimizers, deep-RL optimizer resets), the paper's novelty rests largely on the automatic, tensor-granular, boundary-free detection — which is an engineering increment rather than a conceptual advance.

Questions for authors
  1. The tables report five-seed means without standard deviations or paired significance tests. Can you provide confidence intervals or paired tests to establish that the 0.05–0.48 pp gains are significant beyond seed-level variance? (This would materially affect my assessment of Soundness.)

  2. Proposition B.1 (Var(C_d) = 1/d) is a standard result for uniform points on the unit sphere, and B.2/B.3 rely on the delta method and effective dimension. What is the non-trivial new theoretical contribution of Appendix B beyond these known results? (Clarification.)

  3. Relative to existing optimizer-state management work (adaptive restart, cautious optimizers, MGTTA), what is the substantive novelty of P-DORM beyond "automatic, boundary-free detection"? Please clarify the precise difference. (Clarification.)

  4. The paper acknowledges that gains are inconsistent for very small batches (B < 32). Since real-world TTA deployment is often online with small or single-sample batches, how does this limitation affect the method's practical applicability? (Clarification.)

2 comments

Thank you for your constructive feedback. Please find our detailed responses below.

1. Effect sizes, uncertainty, and regressions (W1/Q1)

We now provide paired statistical evidence for the across-method averages. All ten block-level 95% paired-t confidence intervals lie below zero, and the corresponding two-sided paired t-tests remain significant after Holm correction across the ten reported blocks (largest adjusted p = 0.0191). For each dataset/backbone/optimizer block, we first average errors over the evaluated protocols within each method, then equally across the six TTA methods, retaining the main tables' aggregation weights.

Table CcJS-1. Block-level paired error differences

At the paper's displayed precision, 55 method-level means improve, 1 is unchanged, and 4 regress:

Table CcJS-2. Method-level regressions

Two method-level regressions have unadjusted intervals entirely above zero. We do not dismiss them as noise; they identify specific method-level limitations. However, as shown in the block-level table above, all ten aggregated block-level intervals lie entirely below zero, demonstrating that the average gain across datasets and optimizers is consistent and stable despite these isolated cases.

The exact sign-flip test is bounded by the 5-pair sample size (minimum attainable p = 0.0625); we therefore rely on the parametric paired-t tests and 95% CIs, which establish the statistical significance of the block-level gains.

2. What Appendix B contributes (W2/Q2)

We agree that the spherical variance result 1/d1/d and the delta-method tools in B.1–B.3 are established mathematics. However, applying them to derive the statistical incomparability of raw directional alignment across optimizer tensors is novel, and directly motivates our calibration design.

B.4 further examines a property of our online score: under its location–scale model, scaled initial states, and a zero numerical stabilizer, the tensor-specific alignment center and scale cancel from the score. While the fixed initial variance and nonzero stabilizer limit exact invariance, this conditional analysis provides the exact theoretical rationale for why our calibration is necessary and how it functions under idealized conditions. We will revise the text to frame B.1–B.3 as background and B.4 as conditional invariance analysis, rather than claiming a new mathematical technique.

3. Novelty relative to optimizer-state management (W2/Q3)

The substantive distinction is calibrated intervention in retained optimizer history. Tensor-local calibration matters because alignment signals have different normal levels across tensors: an alignment decrease that is routine for a noisy tensor may be unusual for another tensor with a stable history. P-DORM evaluates each departure relative to that tensor's own online statistics, enabling selective resets that do not discard unflagged histories.

  • Adaptive restart: Restarts momentum based on trajectory conditions. P-DORM calibrates tensor-wise mismatches against background statistics.
  • Cautious optimizers: Mask coordinates of the current update. P-DORM selectively reconstructs history before applying the optimizer.
  • MGTTA: Learns a replacement gradient generator. P-DORM retains the base optimizer and adds a non-learned trigger.
  • Deep-RL resets: Requires external events. P-DORM diagnoses incompatibility locally without boundary annotations.

P-DORM acts as a plug-and-play optimizer component with no added trainable parameters, decoupling "when to intervene" from "how the optimizer updates."

4. Practical applicability at small batch sizes (Q4)

Single-sample adaptation (B=1B=1) under severe corruption represents an extreme edge case. Each update relies on one heavily corrupted image without within-batch averaging, yielding highly variable unsupervised gradients.

Table CcJS-3. Single-sample adaptation

Crucially, the foundational unsupervised TTA objective collapses under this extreme regime (error surges from 43.5% to 90.0% on CIFAR-10-C). This collapse is a fundamental limitation of the TTA task itself, not specific to our wrapper: when both the current gradient and accumulated direction become unreliable, no state management can rescue the optimization. Therefore, P-DORM is designed for standard batch-based deployments (B=200/64) where a meaningful batch-wise signal exists. We will state this applicability boundary explicitly in the revision.

Reviewer_CcJSOpenReview ↗

Official Comment by Reviewer CcJS

I thank the authors for your feedback. I keep my initial rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.