acceptodds
Under review as a conference paper at ICLR 2027

Decomposing the Entropy Minimization Gradient for Calibrated Test-Time Adaptation

Abstract

Entropy minimization (EM) is widely used for test-time adaptation (TTA), but its accuracy improvements under distribution shift often come at the cost of miscalibrated predictions. We study this behavior through the geometry of the EM gradient. After removing the softmax-invariant mean of the logits, we decompose the gradient into a radial component, which increases the centered-logit norm, and a tangential component, which changes relative class scores while preserving the norm to first order. Although these components are orthogonal in logit space, they induce conflicting parameter gradients, particularly in later network layers. Empirically, we find that the tangential component is the main driver of accuracy improvement, whereas the radial component is closely tied to confidence dynamics and becomes relatively stronger in later layers. These observations motivate two complementary interventions: Conflict-Aware Regularized Entropy Minimization (CAREM), which protects the tangential update by projecting conflicting radial gradients onto the tangential gradient's orthogonal complement, and freezing later layers during adaptation to improve calibration. CAREM modifies only the backward pass and introduces no additional hyperparameters or changes to the model architecture or adaptation objective. Across multiple entropy-based TTA methods, architectures, batch sizes, and distribution-shift benchmarks, CAREM improves classification performance, while its combination with late-layer freezing substantially improves calibration and retains strong accuracy. Our results show that accurate and calibrated entropy-based TTA can be achieved with simple changes to its optimization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time adaptation, entropy minimization, calibration, gradient surgery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bnjpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the accuracy and calibration behavior of the entropy minimization for test-time adaptation through the geometry of its gradient. For each sample, the authors deal with the centered logit of the data, especially the radial component parallel to the centered logits zz and a tangential component orthogonal to zz. The authors pull both components back to parameter space and observe the negative cosine similarity between the resulting parameter gradients. Base on the above, they propose CAREM, Conflict-Aware Regularized Entropy Minimization. Also, they combine CAREM with freezing the top layers (FTL), following SAR. They experiment with ImageNet-C, ImageNet-A, ImageNet-R, and VisDA-2021 with Vit-Base and ResNet-50-GN on baselines such as TENT, EATA, SAR, DeYO, and ZeroSiam.

Critical strengths
  • The method is simple, modifies only the backward pass, and introduces no additional hyperparameters. It can be applied to wide range of entropy-based TTA methods.

  • The accuracy gains of CAREM on Vit-Base are consistent across almost all corruptions and baselines.

  • Reinterpreting COME as a manipulation of the EM gradient is an interesting perspective.

Critical weaknesses

W1. It seems that the parameters η\eta and λ(η)\lambda(\eta) are not necessary in Section 4.1. Because the decomposition in Eq. (5) is simply the Euclidean orthogonal projection onto span of zz, when zz is given and the gradient gg is fixed. This implies that the step size η\eta and the scalar λ(η)\lambda(\eta) in Eq. (2) play no role. Moreover, λ(η)\lambda(\eta) always exist, not only for sufficiently small η>0\eta>0.

W2. By Proposition 1, we see that z⊤g=−varp(z)z^{\top} g = -{\mathrm{var}_{p}}(z). Hence, grad=−cz{g_{\mathrm{rad}}}=-c z with c=varp(z)/∣z∣22>0c=\mathrm{var}_p(z)/\\|z\\|_{2}^{2}>0 and so gtan=g+cz=∇z[H(softmax(z))+c2∥z∥22]g_{\mathrm{tan}}=g+c z=\nabla_{z} \Big[H(\mathrm{softmax}(z)) + \frac{c}{2}\|z\|_{2}^{2}\Big] with a constant cc. Therefore, Tangential only TENT is the entropy minimization with a per-sample adaptive logit norm penalty and radial only TENT maximizes a weighted logit norm. This explains the behavior in Figure 3 without relying on the radial and tangential semantics. It also means that natural baselines are missing, namely TENT with a fixed or adaptive logit norm penalty and logit norm style regularization.

W3. Since gθ=gθ,rad+gθ,tang_\theta = g_{\theta,\mathrm{rad}} + g_{\theta,\mathrm{tan}} and both components contain the shared term ∓cJ⊤z\mp c J^\top z, a strongly negative cosine is equivalent to cancellation. That is, gθg_{\theta} is much smaller than the two components. Also r=∥gθ,rad∥/∥gθ,tan∥r = \|g_{\theta,\mathrm{rad}}\|/\|g_{\theta,\mathrm{tan}}\| means that ∥gθ∥2/∥gθ,tan∥2=1+r2+2rρ\|g_\theta\|^{2}/\|g_{\theta,\mathrm{tan}}\|^{2} = 1 + r^{2} + 2r\rho. Whenever J⊤zJ^\top z has a large gain, the cosine is pushed toward −1-1. This is expected in later layers, because the logits are closed to homogenenous in the affine scale of the final normalization layers. The observations in Figure 2 may reflect this structure rather than an empirical property of the entropy minimization.

W4. In Eqs. (7) and (8), the authors say that JJ⊤JJ^\top breaks both the orthogonality and the norm-only role of the components and Figure 3 confirms it. Moreover, the decomposition is defined per sample and the projection is applied to batch aggregated parameter gradients ∑iJi⊤grad,i\sum_{i} J_{i}^{\top} g_{\mathrm{rad},i}, which do not correspond to any geometric object in the parameter space. However, the paper claim that the radial component is central to calibration, nevertheless relies on the logit space interpretation.

Questions for authors

Q1. Could the authors report ∥gθ∥\|g_{\theta}\| together with ρ\rho and rr in Figure 2, both globally and per block? In addition, is the negative cosine specific to the radial and tangerntial split?

Q2. Could the authors confirm the identity gtan=∇z[H+c2∥z∥2]g_{\mathrm{tan}}=\nabla_{z}\big[H+\frac{c}{2}\|z\|^{2}\big] and compare tangential-only TENT and CAREM against TENT with a fixed or adaptive logit norm penalty?

Q3. Did the authors observe batches or layers with gθ⊤gcomb<0g_\theta^\top g_{\mathrm{comb}} < 0?

7 comments
AuthorsOpenReview ↗

Response to Reviewer bnjp (1/2)

Dear Reviewer,

Thank you for the valuable feedback. We appreciate the time you have spent to help us improve our paper, especially pointing out the equivalence between tangential-only EM and EM with an adaptive centered-logit L2 penalty. The suggested fixed or adaptive centered-logit norm penalty baselines achieve strong accuracy and calibration improvements, and have been included in the manuscript as a loss-based alternative to CAREM+FTL. We have updated the manuscript with new experimental results and will now address the issues you have raised:

W1: Orthogonal projection derivation

We agree and have simplified Section 4.1 accordingly.

W2 and Q2: Adaptive logit-norm regularization

Thank you for pointing out this connection. We established it analytically in Eq. 4 and confirmed numerically that the tangential gradient is equivalent to EM with an adaptive centered-logit L2 penalty when the coefficient is detached during differentiation.

We evaluated fixed L2 regularization across all TTA baselines (c=0.01c=0.01) and swept fixed and adaptive coefficients in Appendix G. The adaptive coefficient was defined as c=β⋅varp(z)/∣z∣22c= \beta \cdot \mathrm{var}_p(z) / \\|z\\|_2^2 with β∈[0,1]\beta \in [0, 1]. This formulation retains part of the radial gradient instead of removing it completely. We have also included the sweep's results in Table 1 for fixed cc and Table 2 for adaptive cc.

Table 1: Fixed centered-logit L2 regularization sweep results for Tent on standard ImageNet-C. Entries are mean ± standard deviation across four seeds. Results may differ from the main tables due to fresh runs with different random states.

ccViT-Base Acc. (%)ViT-Base ECE (%)ResNet-50-GN Acc. (%)ResNet-50-GN ECE (%)
060.94 ± 2.3316.88 ± 2.8828.97 ± 0.1340.12 ± 0.19
0.00164.09 ± 2.6312.54 ± 3.2530.89 ± 0.0935.24 ± 0.17
0.00565.59 ± 2.756.83 ± 3.4831.78 ± 0.3731.23 ± 0.87
0.007564.59 ± 2.397.25 ± 2.9132.95 ± 0.5126.89 ± 1.25
0.0166.94 ± 2.415.20 ± 2.8334.99 ± 0.1120.91 ± 0.39
0.012566.85 ± 2.448.61 ± 2.7236.04 ± 0.2219.48 ± 0.41
0.02522.96 ± 2.4113.45 ± 1.1735.74 ± 0.1323.61 ± 2.12
0.0513.10 ± 0.9912.24 ± 0.9826.94 ± 0.7127.27 ± 0.69
0.19.50 ± 0.808.93 ± 0.7919.51 ± 0.1217.65 ± 0.12
0.52.30 ± 0.402.00 ± 0.407.41 ± 0.046.84 ± 0.03

Table 2: Adaptive centered-logit L2 regularization sweep results for Tent on standard ImageNet-C. Entries are mean ± standard deviation across four seeds.

β\betaViT-Base Acc. (%)ViT-Base ECE (%)ResNet-50-GN Acc. (%)ResNet-50-GN ECE (%)
0.1060.77 ± 0.0717.33 ± 0.0930.02 ± 0.2038.20 ± 0.19
0.2061.53 ± 0.0716.36 ± 0.1230.86 ± 0.1536.54 ± 0.11
0.3064.50 ± 2.6912.41 ± 3.3431.52 ± 0.1735.25 ± 0.16
0.4063.98 ± 2.3212.30 ± 2.9032.07 ± 0.1734.01 ± 0.14
0.5066.80 ± 2.337.50 ± 2.9232.18 ± 0.2033.64 ± 0.41
0.6068.33 ± 0.043.82 ± 0.0432.46 ± 0.4032.75 ± 0.83
0.7067.39 ± 2.424.66 ± 2.9632.86 ± 0.5231.33 ± 1.09
0.8067.50 ± 2.388.46 ± 2.7433.57 ± 0.4928.94 ± 1.07
0.9068.61 ± 0.0417.61 ± 0.0334.52 ± 0.5025.66 ± 1.21
1.0068.14 ± 0.0533.99 ± 0.0435.67 ± 0.4421.47 ± 1.00

Fixed regularization achieves strong accuracy and calibration, competitive with CAREM+FTL on ViT-Base and slightly worse on ResNet-50-GN. However, performance is sensitive to the choice of cc and its optimum varies across backbones, as seen in Table 1. Table 2 shows that adaptive regularization performs well on ViT-Base but transfers less consistently to ResNet-50-GN.

These results demonstrate that appropriately controlling logit norm growth can substantially improve both accuracy and calibration. L2 regularization does not require an additional backward pass, as CAREM does, and is therefore more efficient. However, CAREM has the advantage that it is hyperparameter-free and works well for both ViT-Base and ResNet-50-GN without requiring tuning.

AuthorsOpenReview ↗

Response to Reviewer bnjp (2/2)

W3 and Q1: Gradient cancellation and alternative decompositions

We have added the original EM-gradient norm globally and per block to Figure 2, making the cancellation directly observable. We also investigated whether the negative cosine is specific to the radial and tangential split with alternative decompositions in Appendix K (Table 14). We compare projections onto ∣z∣|z|, z2z^2, z∣z∣z|z|, and z3z^3 with the original zz projection. The results are also shown in Table 3.

Table 3: Cosine similarity and performance of alternative projection directions on standard ImageNet-C with ViT-Base. Accuracy / ECE (%) is reported as mean ± standard deviation over four seeds.

DirectionMedian ρρ < 0 (%)Orthogonal only: Acc. / ECE
Original Tent——60.94±2.33 / 16.88±2.88
zz-0.73299.9868.14±0.05 / 33.99±0.04
∣z∣\lvert z\rvert-0.50699.4565.14±2.39 / 12.33±2.88
z2z^2-0.66199.8056.40±1.69 / 18.87±1.74
z∣z∣z\lvert z\rvert-0.85699.9854.82±0.78 / 50.79±0.73
z3z^3-0.74799.5653.00±0.11 / 47.38±0.14

All tested decompositions exhibit negative cosine similarities in more than 99% of diagnostic batches. Thus, negative alignment is not exclusive to our decomposition, and large Jacobian gain along the projected directions may indeed explain the negative cosines. However, negative alignment alone does not identify a useful adaptation direction. Only zz and ∣z∣|z| improve accuracy over Tent with orthogonal-only adaptation.

We appreciate your point about the approximate homogeneity of late normalization layers and believe it is an important property of neural networks for EM. We hypothesize that these layers provide an easy way to reduce entropy by scaling logits without improving classification under distribution shift, resulting in miscalibration. They must therefore be constrained, either through freezing or a penalty on the logit norm, further motivating our approach.

W4: Batch aggregation and geometric interpretation

We agree that the geometric properties of the logit-space decomposition do not generally carry over to batch-aggregated parameter gradients. However, our claim that the radial component plays an important role in calibration does not rely solely on the logit-space interpretation, but also on empirical evidence.

In Figure 3, we observe that tangential-only adaptation improves accuracy but becomes strongly underconfident (32.9% ECE), whereas standard EM retains substantially better calibration (13.8% ECE). Radial-only adaptation exhibits distinctly different confidence dynamics and severely reduces accuracy. These experiments directly investigate the effects of the components after backpropagation.

Furthermore, as you pointed out, the radial component is connected to centered-logit norm maximization, while removing it is equivalent to introducing an adaptive L2 penalty (Eq. 4). The additional regularization experiments in Tables 1 and 2 further demonstrate that controlling this component can substantially improve calibration and accuracy.

Q3: Negative alignment with the original gradient

Yes, but rarely for the global gradient, as seen in Table 4. Layer-wise analysis is still underway and the results will be uploaded as soon as they are done.

Table 4: Percentage of negative cosine between the original and CAREM-combined gradients on standard ImageNet-C with ViT-Base. Results are calculated over four seeds. All runs include CAREM.

MetricTentEATASARDeYOZeroSiam
Cosine < 0 (%)0.0640.0350.1390.0180.000
AuthorsOpenReview ↗

Layer-wise negative alignment with the original gradient

Negative alignment with the original gradient is more common for individual layers, especially later ones, as seen in Table 5. These results are consistent with the larger radial-to-tangential norm ratios and stronger negative cosine similarities observed for later layers in the manuscript's Figure 2. It may therefore be interesting to investigate layer-wise gradient surgery to specifically address the stronger conflicts in these layers in future work.

Table 5: Layer-wise percentages of negative cosine between the original and CAREM-combined gradients on standard ImageNet-C with ViT-Base. -- indicates that the layer was frozen. Results are calculated over four seeds. All runs include CAREM.

LayerTentEATASARDeYOZeroSiam
Block 0, LN10.8440.9311.3110.0990.139
Block 0, LN20.4330.0590.0860.0090.009
Block 1, LN10.0430.2020.2950.0070.011
Block 1, LN20.0190.0280.1310.0040.004
Block 2, LN10.0260.1090.1980.0090.006
Block 2, LN20.0020.0070.0520.0000.004
Block 3, LN10.1810.6100.1840.0930.004
Block 3, LN20.0020.0610.0650.0000.000
Block 4, LN10.9781.5921.1320.1350.030
Block 4, LN20.0000.0200.0010.0000.000
Block 5, LN19.9084.42323.6242.1422.540
Block 5, LN20.0000.0020.0000.0000.000
Block 6, LN13.4481.7244.7820.8830.066
Block 6, LN20.0000.0000.0000.0000.000
Block 7, LN10.0000.0000.0780.0000.000
Block 7, LN20.0000.0000.0000.0000.000
Block 8, LN10.0170.1090.0500.0090.000
Block 8, LN20.0000.0000.0000.0000.000
Block 9, LN10.0771.520--0.0420.002
Block 9, LN20.0000.000--0.0000.000
Block 10, LN114.7513.105--2.9941.449
Block 10, LN259.29235.044--10.88724.659
Block 11, LN119.8360.319--5.8471.984
Block 11, LN21.0343.179--0.4810.023
Reviewer_jrpaOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how entropy minimization in test-time adaptation can improve classification accuracy while harming prediction calibration. The authors decompose the entropy gradient on centered logits into radial and tangential components, then analyze the conflicts between their parameter-space gradients after mapping through the network Jacobian. The experiments are extensive, covering multiple datasets and model architectures.

Critical strengths
  1. The motivation is clear, and the geometric analysis of gradients offers an interesting perspective.
  2. The experiments cover a range of datasets and model architectures.
Critical weaknesses
  1. As a plug-in for the state-of-the-art method ZeroSiam, CAREM provides only limited gains and sometimes degrades performance.
  2. CAREM preserves the tangential gradient but does not guarantee a decrease in the original entropy objective. This seems at odds with the claim that it “does not change the adaptation objective.”
  3. The paper freezes later layers to improve performance but does not adequately explain this choice. It also lacks comparisons with freezing early, middle, or randomly selected layers.
  4. The geometric relationships between gradients in logit space do not carry over to parameter space. The paper itself states that “the parameter gradients do not inherit the geometric behavior of their source logit gradients.” Given this, what is the purpose and practical relevance of the logit-space analysis?
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jrpa

Dear Reviewer,

Thank you for the valuable feedback. We appreciate the time you have spent to help us improve our paper. We have updated the manuscript with new experimental results and will now address the issues you have raised:

W1: Limited gains with ZeroSiam

CAREM's benefits vary across methods and adaptation scenarios. It improves ZeroSiam on standard and imbalanced ImageNet-C, but slightly degrades performance under continuous mixed shifts.

A possible explanation is ZeroSiam's architecture and objective. Unlike pure EM methods, ZeroSiam combines an online branch with a learnable predictor, a stop-gradient target branch, and an auxiliary symmetric KL loss that may already mitigate excessive sharpening and collapse. CAREM decomposes the complete gradient, including both entropy and symmetric KL, so its effect interacts with this additional alignment mechanism. We hypothesize that this limits the additional benefit of gradient surgery.

W2: Descent of the original entropy objective

We agree that CAREM does not guarantee descent of the original entropy objective, and have corrected the formulation in the manuscript. Empirically, we find that the combined CAREM gradient is often aligned with the original gradient (gθ⊤gcomb≥0g_\theta^\top g_{\mathrm{comb}} \geq 0), as seen in Table 1.

Table 1: Percentage of negative cosine between the original and CAREM-combined gradients on standard ImageNet-C with ViT-Base. Results are calculated over four seeds. All runs include CAREM.

MetricTentEATASARDeYOZeroSiam
Cosine < 0 (%)0.0640.0350.1390.0180.000

W3: Choice of frozen layers

We added an ablation in Appendix J (Table 13), comparing early, middle, late, random, and gradient-statistic-based freezing for Tent on standard ImageNet-C with ViT-Base. We also include the table here (Table 2). Each strategy freezes three of the twelve ViT blocks. With CAREM, late freezing achieves the highest accuracy and lowest ECE, outperforming early, middle, and random freezing.

Table 2: Layer-freezing ablations on standard ImageNet-C with ViT-Base. Accuracy / ECE (%), mean ± standard deviation over four seeds. Results may differ from the main tables due to fresh runs with different random states.

PolicyFrozen blocksTentTent+CAREM
No blocks—59.98±0.11 / 18.06±0.1466.60±2.79 / 12.41±3.39
Early0–258.18±0.06 / 18.27±0.1064.70±2.30 / 13.00±2.83
Middle4–659.06±0.07 / 17.94±0.1165.22±2.75 / 12.75±3.35
Late9–1159.37±0.14 / 12.91±0.2167.97±2.40 / 4.92±3.02
RandomVaries by seed61.51±2.57 / 13.41±2.8366.35±2.84 / 10.27±3.12
Highest norm ratio6, 10, 1160.48±2.24 / 10.61±2.9567.75±2.41 / 4.95±3.01
Most negative cosine9–1159.37±0.14 / 12.91±0.2167.97±2.40 / 4.92±3.02
Most negative rρr\rho6, 10, 1160.48±2.24 / 10.61±2.9567.75±2.41 / 4.95±3.01

We also select layers based on their radial-to-tangential norm ratio, cosine similarity, and their product. These statistics were calculated from an EM adaptation run over the full unlabeled ImageNet validation set. The most negative cosine selects exactly blocks 9–11, while the other criteria select blocks 6, 10, and 11, achieving similar results.

These results support late freezing as a simple choice for CAREM without requiring a separate validation set EM run to calculate layer-wise statistics.

W4: Practical relevance of logit-space analysis

The logit-space decomposition identifies two interpretable components of the EM gradient: one associated with logit scaling and another with first-order norm-preserving changes in relative class scores. Some logit-space properties also do survive backpropagation: for a single sample, the radial parameter-gradient update still increases the centered-logit norm to first order because JJ⊤JJ^\top is positive semidefinite.

Moreover, as pointed out by Reviewer bnjp, the decomposition leads directly to an equivalence between tangential-only EM and EM with an adaptive centered-logit L2 penalty (Eq. 4 in the updated manuscript).

Motivated by this connection, we added fixed and adaptive L2 regularization baselines, which achieve strong accuracy and calibration improvements. The logit-space analysis therefore motivates both CAREM and a simple loss-based alternative.

Thank you for the authors’ response. Most of my concerns have been addressed, and I am willing to maintain my score and increase my confidence.

Dear Reviewer,

Thank you for taking the time to read our response and for increasing your confidence. If there are concerns remaining we have not fully addressed, we would be happy to do so.

Best regards, The Authors

Reviewer_YdN7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript studies the accuracy–calibration trade-off in entropy-minimization-based test-time adaptation (TTA). It decomposes the entropy gradient into radial and tangential components with respect to the centered logits, and then examines how their geometry changes after backpropagation. The authors observe strong negative alignment between the induced parameter gradients, with a larger relative radial contribution in later layers. The proposed CAREM prioritizes the tangential gradient by projecting away conflicting radial contributions, and its combination with late-layer freezing. The decomposition and its main Jacobian analysis are mathematically sound. The experiments demonstrate useful gains in accuracy across several configurations and substantial calibration improvements when combining CAREM with freezing.

Critical strengths

S1. The centered-logit decomposition is mathematically correct and conceptually appropriate for separating scale and direction.

S2. The paper explicitly and correctly distinguishes logit-space geometry from parameter-space behavior.

S3. The proposed CAREM has a precise interpretation in optimization and a straightforward integration mechanism.

Critical weaknesses

W1. CAREM is motivated by the idea that radial and tangential gradients can conflict after backpropagation. Its projection removes the part of the radial gradient that opposes the tangential gradient (Eqs. 10–12). However, this also changes the strength of the update along the tangential direction. The results therefore leave an important question open: how much of the improvement comes from resolving gradient conflict, and how much comes from changing the effective update size?

W2. The radial–tangential decomposition is mathematically valid, but the underlying operation is a standard orthogonal projection. Related ideas also appear in earlier methods: COME suppresses a gradient component associated with logit norm, PCGrad projects conflicting gradients, and SAR already uses late-layer freezing. The strongest contribution is therefore the paper’s analysis of entropy minimization, its examination of what happens after backpropagation, and its use of these findings to design a practical adaptation method.

W3. The authors should report how often these failures occur across seeds and whether the same runs account for the large variance. A focused analysis of these cases would be more useful than simply adding more benchmarks.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer YdN7

Dear Reviewer,

Thank you for the valuable feedback. We appreciate the time you have spent to help us improve our paper. We have updated the manuscript with new experimental results and will now address the issues you have raised:

W1: Conflict resolution versus update magnitude

We added two ablations in Appendix I (Table 12): one modifies the CAREM gradient to match the original gradient's total L2 norm, while the other matches its projection magnitude along the tangential gradient. The following Table 1 presents a shortened version of Table 12 in Appendix I, focusing on Tent and EATA with and without FTL.

Table 1: Accuracy / ECE (%) on standard ImageNet-C with ViT-Base. Mean ± standard deviation over four seeds. Original and CAREM results may differ from the main tables due to fresh runs with different random states, particularly for configurations affected by stochastic failures.

MethodOriginalCAREMProjection matchedNorm matched
Tent59.91±0.12 / 18.21±0.1565.39±2.43 / 13.95±2.9664.80±2.09 / 4.97±2.5966.15±2.68 / 10.32±3.27
Tent+FTL59.37±0.14 / 12.91±0.2167.97±2.40 / 4.92±3.0263.57±2.29 / 5.43±2.6967.11±2.34 / 4.94±2.90
EATA66.64±2.14 / 12.92±0.1769.44±2.28 / 11.10±0.2569.07±0.23 / 3.94±0.0869.41±2.24 / 8.85±0.23
EATA+FTL65.25±0.07 / 9.64±0.1370.27±0.35 / 3.14±0.0766.80±2.29 / 3.85±0.0768.99±2.29 / 3.78±1.93

Matching the total norm largely preserves CAREM's improvements. For example, Tent improves from 59.91% to 66.15% accuracy, and EATA from 66.64% to 69.41%. Matching the tangential projection magnitude generally reduces the accuracy gains but often further improves calibration without FTL.

These results show that both the change in gradient direction and the increased tangential contribution matter and therefore support the benefit of resolving gradient conflict beyond simply increasing the update magnitude.

W2: Novelty and relation to prior work

We agree that the main contribution is the analysis of EM's gradient, its behavior after backpropagation, and the use of these findings to design a practical TTA method.

We view the simplicity of CAREM as a strength rather than a weakness. Our goal was to improve accuracy and calibration through a simple intervention without introducing elaborate calibration objectives, additional architectural components, or new hyperparameters. CAREM deliberately builds on established projection techniques, but applies them to two interpretable components of a single EM gradient rather than separate task gradients.

Our experiments show that this simple approach achieves competitive or better accuracy and calibration than existing calibration-aware TTA methods across several settings without introducing additional hyperparameters.

W3: Failure frequency and instability

We added failure counts and affected seed–corruption combinations in Appendix H (Table 11). We also include a compact version of the table here (Table 2). A failure is defined as corruption accuracy being at most 5% or at least 10 percentage points below the corresponding no-adaptation baseline. Failures concentrate on difficult corruptions, particularly Fog and Contrast, but do not consistently occur for the same seeds. Every seed experiences at least one failure.

Table 2: Failure analysis on standard ImageNet-C with ViT-Base. Each entry reports the number of failed seed–corruption runs out of 60 (4 seeds × 15 corruptions), followed by the affected corruptions and seeds. F = Fog, C = Contrast; subscripts indicate seed indices.

MethodOriginal+L2+CAREM+CAREM+FTL
Tent3 (F2,3,4F_{2,3,4})1 (F4F_4)3 (F1,2,4F_{1,2,4})2 (F2,4F_{2,4})
EATA3 (C3,4C_{3,4}, F2F_2)03 (C2,3,4C_{2,3,4})4 (C1,2,3,4C_{1,2,3,4})
SAR002 (C2,4C_{2,4})—
DeYO3 (C3,4C_{3,4}, F2F_2)2 (C4C_4, F3F_3)6 (C1,2,3,4C_{1,2,3,4}, F2,3F_{2,3})5 (C2,3,4C_{2,3,4}, F1,3F_{1,3})
ZeroSiam02 (F3,4F_{3,4})00

We additionally analyze failed runs in Figure 10. EATA, DeYO, and SAR exhibit large early gradient-norm peaks in the range between 137.5 and 304.9 preceding collapse. Gradient clipping at 50 prevents these failures, whereas Tent still collapses, indicating that large gradients are not the only cause of instability. These observations are consistent with SAR's analysis of TTA instability, which identifies high-entropy samples and large, noisy gradients as the main cause of model collapse during TTA. Tent has no filter for high-entropy samples, whereas EATA, DeYO, and SAR do.

However, the appropriate gradient clipping threshold depends on the architecture, objective, data, and optimizer. Gradient clipping can therefore not be used as a universal solution as is. Applying clipping only during the initial adaptation phase and relaxing it after the gradients stabilize may be a possible direction for future work.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would like to kindly ask whether our rebuttal and the revised manuscript have addressed your concerns.

We would appreciate any further feedback and would be happy to clarify any remaining questions.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

Dear Reviewers and Area Chair,

Thank you for your constructive feedback and the discussions throughout the rebuttal period. Your expertise has helped us further improve our paper. We have substantially revised the manuscript and conducted additional experiments to address the raised concerns. We would like to briefly summarize the main additions:

  1. Theoretical analysis (Section 4.1): We established the equivalence between the tangential component of the EM gradient and EM with an adaptive L2 penalty on the centered-logit norm. Thank you to Reviewer bnjp for pointing out this connection.

  2. Centered-logit L2 regularization (Section 5, Appendix G): Based on the aforementioned equivalence, we introduced and extensively evaluated fixed L2 regularization as an alternative to CAREM+FTL, including an analysis of its hyperparameter sensitivity and backbone dependence. L2 regularization also improved accuracy and calibration, and achieved competitive results with CAREM+FTL on ViT-Base and slightly worse results on ResNet-50-GN. However, its performance is sensitive to the choice of its hyperparameter cc and its optimum varies across backbones (Appendix G), while CAREM is hyperparameter-free and works well across backbones.

  3. Additional ablation studies (Appendices I and J): We investigated the effects of gradient direction and magnitude on CAREM's improvements and systematically compared different layer-freezing strategies. The results show that both CAREM's increased tangential contribution and its change in gradient direction contribute to the observed improvements, and that late-layer freezing works best.

  4. Failure case analysis (Appendix H): We quantified adaptation failures and investigated their relationship to gradient norms and the effects of gradient clipping. Consistent with SAR's analysis of TTA instability, we found that large gradient spikes are associated with several failure cases. Gradient clipping prevents failures for EATA, DeYO, and SAR, but not for Tent, possibly due to its lack of high-entropy sample filtering.

  5. Alternative gradient decompositions (Appendix K): We evaluated alternative projection directions with CAREM and analyzed their gradient conflicts, accuracy, and calibration. The results show that strong negative gradient alignment occurs across all decompositions, possibly due to large Jacobian gains along the projected directions. However, negative alignment does not determine their usefulness, as some alternatives lead to severe underconfidence even with CAREM. A useful decomposition must therefore balance constraining logit-norm inflation against excessive suppression.

We hope these revisions addressed the reviewers' concerns and provided a more comprehensive understanding of our work.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.