- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on test-time adaptation when unlabeled examples arrive one at a time, including shuffled and class-blocked streams. It proposes OriTTA, which precomputes a geometric reference frame from the frozen source classifier and one synthetic prototype per class. During adaptation, OriTTA updates only normalization parameters, applies angular and radial losses to samples judged reliable, maintains a class-indexed memory bank, and resets to source weights when some existing method collapse.
Critical strengths
-
The geometric motivation is clear and partly validated by ablations.
-
The collapse detector is a potentially reusable contribution.
Critical weaknesses
-
The experiment results are mixed, and the “best in all sixteen settings” claim is not yet convincing comparing with recent SOTA like Sateen.
-
The method still depends on predicted labels, so the self-reinforcement risk is reduced but not eliminated. If a sample is confidently misclassified, OriTTA will adapt it toward the wrong ideal point.
-
The paper emphasizes that OriTTA uses no per-setting tuning. However, the method contains many global parameters, which will make tuning difficult.
-
The evaluation is too narrow to support the " real-world deployment" motivation. The experiments are conducted on CIFAR-C and ImageNet-C. This is a standard and useful benchmark, but it is synthetic corruption, not natural distribution shift.
-
The paper says that OriTTA updates “normalization affine parameters,” but it is not clear exactly which parameters are updated in each backbone.
Questions for authors
-
Exactly which normalization parameters are updated for each backbone?
-
How were the global constants chosen? a grid search will be very costly.
-
Why are directly relevant baselines such as NOTE, DELTA, RoTTA omitted? Please either include them or justify their exclusion.
-
What are the practical costs of the method? Please report anchor-generation time, memory-bank storage, etc.
6 comments
Response to Reviewer 3u29 (2/3)
W3. The paper emphasizes that OriTTA uses no per-setting tuning. However, the method contains many global parameters, which will make tuning difficult.
Response: Thank you for raising this point. The constants of OriTTA are not meant to be tuned. They were set once on ImageNet-C-ResNet50-GN and are applied unchanged to every dataset and backbone, and a new dataset or checkpoint therefore requires no tuning. As a test on data never used during development, we newly ran OriTTA on ImageNet-Sketch with every constant unchanged, and the results are presented in the following table.
<Table R3.> ImageNet-Sketch-ResNet50-GN, shuffled, batch size 1, ten seeds. The last column counts the seeds on which OriTTA is more accurate than the method of that row on the same stream.| Accuracy | Gain over source | Seeds OriTTA wins | |
|---|---|---|---|
| Source | 29.19 | ||
| TENT | 22.78±0.43 | −6.41 | 10/10 |
| EATA | 30.26±0.62 | +1.07 | 10/10 |
| SAR | 31.65±0.18 | +2.46 | 10/10 |
| DeYO | 35.44±0.41 | +6.25 | 10/10 |
| ReCAP | 37.55±0.38 | +8.36 | 10/10 |
| SaTeen | 37.76±0.45 | +8.57 | 10/10 |
| OriTTA | 38.56±0.11 | +9.37 |
OriTTA is the most accurate method on every seed. Additionally, the defaults lie on the flat side of , the trust ratio, and in Table R1. Widening the cone, lowering the trust ratio, or changing moves the mean accuracy by at most 0.40 points, and the accuracy drops only when the cone is narrowed or the trust ratio is raised.
In contrast, the published settings of the recent baselines do not transfer from ImageNet-C to CIFAR-100-C. In the episodic shuffled protocol, DeYO, ReCAP, and SaTeen fall far below the source model at 53.25%, while OriTTA reaches 62.16% with the same constants, as shown in Table 3 in the revised version. Since test-time adaptation provides no labeled validation data for a new search, we consider shared constants to be an advantage in practice.
W4. The evaluation is too narrow to support the " real-world deployment" motivation. The experiments are conducted on CIFAR-C and ImageNet-C. This is a standard and useful benchmark, but it is synthetic corruption, not natural distribution shift.
Response: We appreciate your suggestion. Following your suggestion, we newly evaluated OriTTA on ImageNet-Sketch, a natural distribution shift, with every constant unchanged. As presented in Table R3, OriTTA improves the source model from 29.19% to 38.56% and is the most accurate method on every seed. We added this experiment as Table 5 in Section 4.2 of the revised version.
W5 and Q1. The paper says that OriTTA updates “normalization affine parameters,” but it is not clear exactly which parameters are updated in each backbone. Exactly which normalization parameters are updated for each backbone?
Response: Thank you for the question. OriTTA updates only the per-channel scale and shift of the normalization layers, , and keeps every other weight frozen, including the classifier. The normalization statistics are not updated either, since GroupNorm and LayerNorm compute them from each input and BatchNorm keeps those of the source model. The updated parameters of each backbone are listed in the following table.
<Table R4.> Parameters updated by OriTTA. Total counts every parameter of the backbone including the classifier.| Backbone | Updated parameters | Updated | Total | Ratio |
|---|---|---|---|---|
| ImageNet-C-ResNet50-GN | scale and shift of all 53 GroupNorm layers | 53,120 | 25,557,032 | 0.208% |
| ImageNet-C-ViT-B/16 | scale and shift of the 24 LayerNorm layers in the 12 blocks, without the final LayerNorm | 36,864 | 86,567,656 | 0.043% |
| CIFAR-10-C-WRN-28-10 | scale and shift of all 25 BatchNorm layers | 17,952 | 36,479,194 | 0.049% |
| CIFAR-100-C-WRN-40-2 | scale and shift of all 37 BatchNorm layers | 5,408 | 2,255,156 | 0.240% |
On ViT-B/16, the final LayerNorm before the classifier is kept frozen. This is stated in Appendix A.1, and we added Table R4 as Table 11 in Appendix A.1 of the revised version.
Response to Reviewer 3u29 (3/3)
Q2. How were the global constants chosen? a grid search will be very costly.
Response: Thank you for the good question. We did not run a grid search, and no constant was tuned per setting. During development, we tried a few values for each constant on ImageNet-C-ResNet50-GN, fixed them, and used them unchanged in all sixteen settings and on ImageNet-Sketch. Quantities whose scale differs across backbones, such as , , , , and , are not chosen but computed from the source model by fixed rules (Table 13 in the revised version), and only the remaining dimensionless constants were set on ImageNet-C-ResNet50-GN.
Since the constants were not searched exhaustively, they are not necessarily optimal. In Table R1, several variations, such as a wider cone or a doubled learning rate, are above the defaults. We did not tune them further because test-time adaptation provides no labeled validation data, and a method that needs a search for each dataset could not be deployed in this setting.
Table R1 also shows that a costly search is not needed. sets the opening of the cone, and the accuracy drops when the cone is narrowed and changes little when it is widened. The trust ratio sets the fraction of cone samples that enter the losses, and the accuracy changes little when it is lowered, while fog collapses when it is raised to 0.4. weights the radial loss and has little effect. The learning rate is the only backbone-specific input, with one base rate per normalization type, and it follows the scaling rule for Adam (Malladi et al., 2022) at every batch size, including batch size 1.
Q3. Why are directly relevant baselines such as NOTE, DELTA, RoTTA omitted? Please either include them or justify their exclusion.
Response: We appreciate your suggestion. Following your suggestion, we newly ran RoTTA with its official code on ImageNet-C-ResNet50-GN over the ten seeds of the main tables, and the results are presented in the following table.
<Table R5.> Mean accuracy on ImageNet-C-ResNet50-GN, shuffled, ten seeds. The other baselines are in Table 1 in the revised version.| ImageNet-C-ResNet50-GN, shuffled | episodic | continual |
|---|---|---|
| Source | 31.44 | 31.44 |
| RoTTA | 32.24 | 28.01 |
| OriTTA | 49.59 | 40.81 |
RoTTA stays near the source model in the episodic protocol and falls below it in the continual one. Together with CoTTA, which falls to the chance level at batch size 1 in Table 4 in the revised version, this indicates that methods designed for continual adaptation with batches do not carry over to a single-sample stream.
NOTE and DELTA cannot be compared under our setting for the following reasons. NOTE replaces BatchNorm with instance-aware batch normalization and re-trains the source model with it, whereas every method in our comparison adapts the same released checkpoints without re-training. DELTA corrects the BatchNorm statistics at test time, which does not apply to ResNet50-GN and ViT-B/16 because they have no BatchNorm. On the CIFAR backbones, every method keeps the source statistics at batch size 1, and the re-weighting of DELTA gives a weight of one to the single sample. Therefore, DELTA reduces to the loss it is combined with, such as the entropy loss of TENT, which is in our tables. We added RoTTA with its per-corruption results as Table 16 in Appendix A.10 of the revised version, together with this discussion, and summarized it in Section 4.2.
Q4. What are the practical costs of the method? Please report anchor-generation time, memory-bank storage, etc.
Response: We appreciate your suggestion. Following your suggestion, we measured the costs of OriTTA outside the stream, which consist of a one-off anchor synthesis and two stored images per class. The results are presented in the following table.
<Table R6.> Costs outside the adaptation time.| ImageNet-C-ResNet50-GN | ImageNet-C-ViT-B/16 | CIFAR-10-C-WRN-28-10 | CIFAR-100-C-WRN-40-2 | |
|---|---|---|---|---|
| Anchor synthesis, once per backbone | 12.2 min | 28.8 min | 15 s | 42 s |
| Peak GPU memory during synthesis | 4.3 GiB | 4.1 GiB | 0.4 GiB | 0.3 GiB |
| Bank and anchors, 32-bit | 1.20 GB | 1.20 GB | 0.25 MB | 2.5 MB |
| Bank alone in 8 bits, lossless | 150 MB | 150 MB | 31 kB | 0.31 MB |
The memory bank holds one input image per class, and its size does not grow with the stream, since each slot is overwritten by the latest test input admitted to that class. It stores inputs rather than features because the alignment loss and the detector pass them through the current model. The anchors are kept as well, to re-seed the bank after a reset. For the adaptation time in Table 17 in the revised version in Appendix A.11, OriTTA takes about 26 ms per image on ImageNet-C-ResNet50-GN at batch size 1, compared with 13 ms for TENT and 58 ms for SAR. We added Table R6 as Table 18 in Appendix A.11 of the revised version.
We hope that our response has addressed your concerns. Thank you very much!
Response to Reviewer 3u29 (1/3)
We thank you for the insightful comments and suggestions. We have addressed each of your questions below.
Response: Thank you for the comment. We believe this concern comes from our wording. The claim in the abstract and the contributions is that OriTTA achieves the highest mean accuracy over the fifteen corruptions in all sixteen settings. However, the Conclusion described it as the most accurate method in every setting, which can be read as a claim about each corruption. We corrected this wording in the Conclusion (Section 5) of the revised version.
On single corruptions, as you pointed out, OriTTA is not always first. However, the gaps are far from symmetric. On ImageNet-C-ResNet50-GN in the episodic shuffled protocol, OriTTA trails SaTeen on eight corruptions by 1.2 points on average and by at most 2.3 points, while it leads on the other seven by 10.7 points on average and by up to 39.7 points. These leads include defocus and glass blur, where SaTeen is stable across seeds and OriTTA is still ahead by 7.7 and 5.7 points, and fog, where SaTeen collapses in 62% of one hundred runs while OriTTA never does, as shown in Table 2 in the revised version. This asymmetry is why OriTTA is ahead in the mean accuracy. On ImageNet-Sketch, a natural shift which we newly added, OriTTA is also ahead of SaTeen on every seed, as presented in Table R3 in our response to W3.
Response: Thank you for the insightful comment. As you pointed out, the risk is reduced rather than eliminated, since a confidently misclassified feature that enters the reliable region can be pulled toward the wrong ideal point. In practice, however, we did not observe such errors reinforcing themselves. If they did, they would appear as abrupt failures on some seeds. Over one hundred seeds on fog, DeYO, ReCAP, and SaTeen collapse in 66%, 33%, and 62% of the runs, while OriTTA never collapses and its standard deviation stays at 0.2 to 0.3 points, as shown in Table 2 in the revised version. The same pattern holds at batch size 64, as shown in Table 33 in the revised version.
Two components keep the remaining risk in check. First, the reliable region admits only the fifth of the cone samples of each class that lie closest to the ideal point. To verify its role, we changed this trust ratio together with the other constants, and the results are presented in the following table.
<Table R1.> Effect of each constant on ImageNet-C-ResNet50-GN, episodic, shuffled. The default row is the ten-seed mean of Table 1 in the revised version, and each variation is a single run with seed 2021 in which one constant is changed. The defaults are $\kappa_0 = 0.11$, trust ratio 0.2, and $w_{\rm rad} = 2$.When the trust ratio is raised to 0.4, the region admits samples farther from the ideal points and fog collapses, while lowering it to 0.1 changes the accuracy little. We added Table R1 with the per-corruption results as Table 15 in Appendix A.4 of the revised version.
Second, the alignment loss pulls one stored input of every class toward its fixed ideal point, which counters the drift of the predicted labels toward a few classes. To verify this, we removed the loss on CIFAR-10-C, and the results are presented in the following table.
<Table R2.> OriTTA with and without $\mathcal{L}_{\rm align}$ on CIFAR-10-C-WRN-28-10, episodic, shuffled, seed 2021. The share is taken over the last 500 predictions, where a balanced stream gives 10%.Without the loss, the predictions stay concentrated on a few classes until the end of each corruption, and the detector has to reset the model three times. We added Table R2 as Table 25 in Appendix D.3 of the revised version.