- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the accuracy and calibration behavior of the entropy minimization for test-time adaptation through the geometry of its gradient. For each sample, the authors deal with the centered logit of the data, especially the radial component parallel to the centered logits and a tangential component orthogonal to . The authors pull both components back to parameter space and observe the negative cosine similarity between the resulting parameter gradients. Base on the above, they propose CAREM, Conflict-Aware Regularized Entropy Minimization. Also, they combine CAREM with freezing the top layers (FTL), following SAR. They experiment with ImageNet-C, ImageNet-A, ImageNet-R, and VisDA-2021 with Vit-Base and ResNet-50-GN on baselines such as TENT, EATA, SAR, DeYO, and ZeroSiam.
Critical strengths
-
The method is simple, modifies only the backward pass, and introduces no additional hyperparameters. It can be applied to wide range of entropy-based TTA methods.
-
The accuracy gains of CAREM on Vit-Base are consistent across almost all corruptions and baselines.
-
Reinterpreting COME as a manipulation of the EM gradient is an interesting perspective.
Critical weaknesses
W1. It seems that the parameters and are not necessary in Section 4.1. Because the decomposition in Eq. (5) is simply the Euclidean orthogonal projection onto span of , when is given and the gradient is fixed. This implies that the step size and the scalar in Eq. (2) play no role. Moreover, always exist, not only for sufficiently small .
W2. By Proposition 1, we see that . Hence, with and so with a constant . Therefore, Tangential only TENT is the entropy minimization with a per-sample adaptive logit norm penalty and radial only TENT maximizes a weighted logit norm. This explains the behavior in Figure 3 without relying on the radial and tangential semantics. It also means that natural baselines are missing, namely TENT with a fixed or adaptive logit norm penalty and logit norm style regularization.
W3. Since and both components contain the shared term , a strongly negative cosine is equivalent to cancellation. That is, is much smaller than the two components. Also means that . Whenever has a large gain, the cosine is pushed toward . This is expected in later layers, because the logits are closed to homogenenous in the affine scale of the final normalization layers. The observations in Figure 2 may reflect this structure rather than an empirical property of the entropy minimization.
W4. In Eqs. (7) and (8), the authors say that breaks both the orthogonality and the norm-only role of the components and Figure 3 confirms it. Moreover, the decomposition is defined per sample and the projection is applied to batch aggregated parameter gradients , which do not correspond to any geometric object in the parameter space. However, the paper claim that the radial component is central to calibration, nevertheless relies on the logit space interpretation.
Questions for authors
Q1. Could the authors report together with and in Figure 2, both globally and per block? In addition, is the negative cosine specific to the radial and tangerntial split?
Q2. Could the authors confirm the identity and compare tangential-only TENT and CAREM against TENT with a fixed or adaptive logit norm penalty?
Q3. Did the authors observe batches or layers with ?
7 comments
Response to Reviewer bnjp (2/2)
W3 and Q1: Gradient cancellation and alternative decompositions
We have added the original EM-gradient norm globally and per block to Figure 2, making the cancellation directly observable. We also investigated whether the negative cosine is specific to the radial and tangential split with alternative decompositions in Appendix K (Table 14). We compare projections onto , , , and with the original projection. The results are also shown in Table 3.
Table 3: Cosine similarity and performance of alternative projection directions on standard ImageNet-C with ViT-Base. Accuracy / ECE (%) is reported as mean ± standard deviation over four seeds.
| Direction | Median ρ | ρ < 0 (%) | Orthogonal only: Acc. / ECE |
|---|---|---|---|
| Original Tent | — | — | 60.94±2.33 / 16.88±2.88 |
| -0.732 | 99.98 | 68.14±0.05 / 33.99±0.04 | |
| -0.506 | 99.45 | 65.14±2.39 / 12.33±2.88 | |
| -0.661 | 99.80 | 56.40±1.69 / 18.87±1.74 | |
| -0.856 | 99.98 | 54.82±0.78 / 50.79±0.73 | |
| -0.747 | 99.56 | 53.00±0.11 / 47.38±0.14 |
All tested decompositions exhibit negative cosine similarities in more than 99% of diagnostic batches. Thus, negative alignment is not exclusive to our decomposition, and large Jacobian gain along the projected directions may indeed explain the negative cosines. However, negative alignment alone does not identify a useful adaptation direction. Only and improve accuracy over Tent with orthogonal-only adaptation.
We appreciate your point about the approximate homogeneity of late normalization layers and believe it is an important property of neural networks for EM. We hypothesize that these layers provide an easy way to reduce entropy by scaling logits without improving classification under distribution shift, resulting in miscalibration. They must therefore be constrained, either through freezing or a penalty on the logit norm, further motivating our approach.
W4: Batch aggregation and geometric interpretation
We agree that the geometric properties of the logit-space decomposition do not generally carry over to batch-aggregated parameter gradients. However, our claim that the radial component plays an important role in calibration does not rely solely on the logit-space interpretation, but also on empirical evidence.
In Figure 3, we observe that tangential-only adaptation improves accuracy but becomes strongly underconfident (32.9% ECE), whereas standard EM retains substantially better calibration (13.8% ECE). Radial-only adaptation exhibits distinctly different confidence dynamics and severely reduces accuracy. These experiments directly investigate the effects of the components after backpropagation.
Furthermore, as you pointed out, the radial component is connected to centered-logit norm maximization, while removing it is equivalent to introducing an adaptive L2 penalty (Eq. 4). The additional regularization experiments in Tables 1 and 2 further demonstrate that controlling this component can substantially improve calibration and accuracy.
Q3: Negative alignment with the original gradient
Yes, but rarely for the global gradient, as seen in Table 4. Layer-wise analysis is still underway and the results will be uploaded as soon as they are done.
Table 4: Percentage of negative cosine between the original and CAREM-combined gradients on standard ImageNet-C with ViT-Base. Results are calculated over four seeds. All runs include CAREM.
| Metric | Tent | EATA | SAR | DeYO | ZeroSiam |
|---|---|---|---|---|---|
| Cosine < 0 (%) | 0.064 | 0.035 | 0.139 | 0.018 | 0.000 |
Layer-wise negative alignment with the original gradient
Negative alignment with the original gradient is more common for individual layers, especially later ones, as seen in Table 5. These results are consistent with the larger radial-to-tangential norm ratios and stronger negative cosine similarities observed for later layers in the manuscript's Figure 2. It may therefore be interesting to investigate layer-wise gradient surgery to specifically address the stronger conflicts in these layers in future work.
Table 5: Layer-wise percentages of negative cosine between the original and CAREM-combined gradients on standard ImageNet-C with ViT-Base. -- indicates that the layer was frozen. Results are calculated over four seeds. All runs include CAREM.
| Layer | Tent | EATA | SAR | DeYO | ZeroSiam |
|---|---|---|---|---|---|
| Block 0, LN1 | 0.844 | 0.931 | 1.311 | 0.099 | 0.139 |
| Block 0, LN2 | 0.433 | 0.059 | 0.086 | 0.009 | 0.009 |
| Block 1, LN1 | 0.043 | 0.202 | 0.295 | 0.007 | 0.011 |
| Block 1, LN2 | 0.019 | 0.028 | 0.131 | 0.004 | 0.004 |
| Block 2, LN1 | 0.026 | 0.109 | 0.198 | 0.009 | 0.006 |
| Block 2, LN2 | 0.002 | 0.007 | 0.052 | 0.000 | 0.004 |
| Block 3, LN1 | 0.181 | 0.610 | 0.184 | 0.093 | 0.004 |
| Block 3, LN2 | 0.002 | 0.061 | 0.065 | 0.000 | 0.000 |
| Block 4, LN1 | 0.978 | 1.592 | 1.132 | 0.135 | 0.030 |
| Block 4, LN2 | 0.000 | 0.020 | 0.001 | 0.000 | 0.000 |
| Block 5, LN1 | 9.908 | 4.423 | 23.624 | 2.142 | 2.540 |
| Block 5, LN2 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 |
| Block 6, LN1 | 3.448 | 1.724 | 4.782 | 0.883 | 0.066 |
| Block 6, LN2 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Block 7, LN1 | 0.000 | 0.000 | 0.078 | 0.000 | 0.000 |
| Block 7, LN2 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Block 8, LN1 | 0.017 | 0.109 | 0.050 | 0.009 | 0.000 |
| Block 8, LN2 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Block 9, LN1 | 0.077 | 1.520 | -- | 0.042 | 0.002 |
| Block 9, LN2 | 0.000 | 0.000 | -- | 0.000 | 0.000 |
| Block 10, LN1 | 14.751 | 3.105 | -- | 2.994 | 1.449 |
| Block 10, LN2 | 59.292 | 35.044 | -- | 10.887 | 24.659 |
| Block 11, LN1 | 19.836 | 0.319 | -- | 5.847 | 1.984 |
| Block 11, LN2 | 1.034 | 3.179 | -- | 0.481 | 0.023 |
Response to Reviewer bnjp (1/2)
Dear Reviewer,
Thank you for the valuable feedback. We appreciate the time you have spent to help us improve our paper, especially pointing out the equivalence between tangential-only EM and EM with an adaptive centered-logit L2 penalty. The suggested fixed or adaptive centered-logit norm penalty baselines achieve strong accuracy and calibration improvements, and have been included in the manuscript as a loss-based alternative to CAREM+FTL. We have updated the manuscript with new experimental results and will now address the issues you have raised:
W1: Orthogonal projection derivation
We agree and have simplified Section 4.1 accordingly.
W2 and Q2: Adaptive logit-norm regularization
Thank you for pointing out this connection. We established it analytically in Eq. 4 and confirmed numerically that the tangential gradient is equivalent to EM with an adaptive centered-logit L2 penalty when the coefficient is detached during differentiation.
We evaluated fixed L2 regularization across all TTA baselines (c=0.01) and swept fixed and adaptive coefficients in Appendix G. The adaptive coefficient was defined as c=β⋅varp(z)/∣z∣22 with β∈[0,1]. This formulation retains part of the radial gradient instead of removing it completely. We have also included the sweep's results in Table 1 for fixed c and Table 2 for adaptive c.
Table 1: Fixed centered-logit L2 regularization sweep results for Tent on standard ImageNet-C. Entries are mean ± standard deviation across four seeds. Results may differ from the main tables due to fresh runs with different random states.
Table 2: Adaptive centered-logit L2 regularization sweep results for Tent on standard ImageNet-C. Entries are mean ± standard deviation across four seeds.
Fixed regularization achieves strong accuracy and calibration, competitive with CAREM+FTL on ViT-Base and slightly worse on ResNet-50-GN. However, performance is sensitive to the choice of c and its optimum varies across backbones, as seen in Table 1. Table 2 shows that adaptive regularization performs well on ViT-Base but transfers less consistently to ResNet-50-GN.
These results demonstrate that appropriately controlling logit norm growth can substantially improve both accuracy and calibration. L2 regularization does not require an additional backward pass, as CAREM does, and is therefore more efficient. However, CAREM has the advantage that it is hyperparameter-free and works well for both ViT-Base and ResNet-50-GN without requiring tuning.