acceptodds
Under review as a conference paper at ICLR 2027

Forecasting When Test-Time Adaptation Will Fail

Abstract

Test-time adaptation (TTA) updates a pre-trained model on unlabeled test data. Many TTA methods recompute batch-normalization (BN) statistics on every test batch. On skewed batches such as pathology slides which are processed one at a time, this can sharply reduce accuracy. We show, in controlled experiments that keep the whole stream class-balanced, that skewed batches alone cause this failure, that recomputing the BN statistics without any gradient step reproduces it, and that the degradation comes from normalizing a batch with statistics dominated by its own class. We turn these findings into a gate that forecasts, before each TTA update, whether adapting on the current batch will help, based on the batch composition predicted by the model without adaptation and its response to the batch's statistics. The forecast is fitted once on held-out labeled source data and needs no target labels. Across CIFAR-10-C, CIFAR-100-C and two pathology datasets, the gate prevents the collapse of every BN-based method we evaluate on class- and slide-ordered streams, retains most of the benefit on shuffled streams, and improves methods designed for skewed streams on pathology. (Code will be made public upon acceptance)

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time adaptation, Batch Normalization, Distribution Shift, Computational Pathology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_M3MhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a gate that forecasts, before each test-time adaptation (TTA) update, whether adapting on the current batch will help or hurt. It uses label-free signals from a frozen source model—the predicted batch composition and the model’s response to the batch’s BN statistics—combined with a linear rule fitted once on held-out labeled source data, so no target labels are needed. It validates the gate on four datasets, wrapping several BN-based and skewed-stream TTA methods under different stream orderings.

Critical strengths

1. The diagnostic experiments are very well designed. The controlled composition sweep keeps the whole stream class-balanced and varies only batch composition. The statistics-swap analysis then pinpoints the key factor: normalizing a batch with BN statistics dominated by its own class. This is a clean and useful finding.

2. The method is well motivated and simple. The entropy-response signal ΔH directly matches the failure identified in the diagnosis. The design is not over-engineered: a linear forecast, nonnegative weights, a zero decision threshold, and one calibration set per dataset. The method and the analysis fit together well.

3. The experimental scope is broad, and the pathology results have practical meaning. The gate is tested with five BN-based methods and five methods designed for skewed streams, across multiple stream orders. On Camelyon17 slide order, it prevents the collapse of BN-based methods and makes gated RoTTA the most accurate method. These results are relevant to real slide-by-slide processing.

Critical weaknesses

1. The calibration set is a practical limitation. It can be taken from source training data, but in some domains—such as rare diseases in medical imaging—labeled data may be very scarce. The authors already show results from 5,000 down to 1,000 calibration images; it would be helpful to push this further and show the failure curve on smaller calibration sets.

2. The extra computation is not fully quantified. The paper states that the gate adds two forward passes per batch. In practice, it is unclear how much extra wall-clock time and compute this costs, especially relative to the TTA update itself and to source inference. A clear overhead measurement would help readers judge deployment cost.

Questions for authors

1. On calibration size: Could the authors report the gate’s performance with much smaller calibration sets, for example 10, 25, 50, 100, and 200 images per class, especially on the pathology datasets? A failure curve would clarify when the gate stops being reliable and would make the practical limitation easier to assess.

2. On computational overhead: Could the authors quantify the extra cost of the two forward passes per batch? Useful numbers would include wall-clock latency, FLOPs, peak memory, and throughput overhead, reported separately for source inference, the TTA update, and the gate. It would also help to state whether the extra frozen model copy is included in the memory measurement.

3 comments
AuthorsOpenReview ↗

Response to Reviewer M3Mh (1/2)

We thank the reviewer for the careful reading of our paper. We agree with both weaknesses: the calibration set is a practical limitation, and the submission did not quantify the cost of the gate. We therefore ran both experiments the reviewer asked for, and they are added to the revised PDF (App. F.9–F.10, Tables A24 and A27, Figure A5).

Q1. Smaller calibration sets. On CIFAR-10-C, 25 labeled images per class give a gate within 0.9 points of the 500 we use. On Camelyon17, 200 patches per class give it within 0.2, while below that it is no longer reliable. For the rare-disease case the reviewer raises, the gate does not need labels.

CIFAR-10-C, labeled images per classClass orderPersistentRecoveryTemporalShuffled
500 (paper), gated BN-Adapt59.8062.3967.5563.9179.26
20059.8162.3767.6064.0079.37
5059.7862.0167.7464.2279.74
2559.7561.5567.8064.3579.87
native BN-Adapt22.6924.7545.6954.5179.94
source model59.5362.4459.5359.5359.53

Gated TENT follows the same pattern (within 0.8 points of its 500-per-class gate from 25 per class up: 82 % on shuffled streams and 60 % on class order at 25, 50 and 200 per class). A few images suffice because every calibration image is seen under 26 appearance conditions, in 25 streams each. At 10 per class no gate can be fitted, because 100 images do not fill two batches of 64. We did not run 100 per class. The gates at 50 and 200 per class differ by at most 0.4 points. On CIFAR-100-C, 10 per class stays within 1.5 points (Table A26a). One fit goes the other way: on the CIFAR-100 model trained on 30,000 images, the 200-per-class gate gives up the gain on shuffled streams (31.7 % against 47.4 % native, source 28.3 %), where 25–100 per class keep it (Table A26b).

On Camelyon17 (K = 2), the pathology dataset on which we ran the size curve (on NCT we report the pseudo-label gate, within 0.5 points of the labeled one, App. F.8):

Camelyon17, labeled patches per classSlide orderWithin-slide shuffledShuffled
paper (full pool), gated BN-Adapt78.781.891.3
1,00079.081.891.4
40078.981.891.3
20078.881.891.3
10081.389.193.4
6465.079.493.4
native BN-Adapt63.779.493.4
source model72.972.972.9

From 200 patches per class the gate matches the one we report. The 100-per-class row is higher because at this budget the pool is too small to fill two batches of the long-tail calibration streams, so the fit reduces to ΔH alone, which is the stronger gate for BN-Adapt when K = 2 (App. F.10). The full gate matters where ΔH alone fails, which is on the methods built for skewed batches. Under class-ordered streams on CIFAR-10-C, the rule ΔH ≤ 0 leaves RoTTA, UnMix-TNS and TRIBE 12–17 points below their calibrated gates (Table A22). Gated RoTTA on Camelyon17 confirms this as it drops to 71.9% on slide order, below the source model's 72.9%, when we fit it on only 100 patches per class (Fig. A5).

At 64 patches per class, the smallest budget at which a calibration stream can be formed at K = 2, the full gate can no longer be fitted and stays near native (65.0%). The ΔH-only gate from the same 128 patches reaches 82.7% on slide order, its best at any budget. We therefore recommend ΔH alone at K = 2 (App. F.10, Figure A5). At K = 1,000, one ImageNetV2 image per class (1,000 labeled images) gives a gate within 0.6 points of the one we report on every ImageNet-C stream type (App. F.10).

The rare-disease case. Here the labels are not required. When we calibrate on the frozen model's own pseudo-labels, our gate stays within 1.5 points of the labeled gate on Camelyon17 for every method but TRIBE (App. F.8). With no source data at all, the rule ΔH ≤ 0 lifts TENT from 56.0% to 76.3% on slide-ordered Camelyon17 (Table A22).

AuthorsOpenReview ↗

Response to Reviewer M3Mh (2/2)

Q2. Overhead. The gate adds two frozen forward passes per batch and no backward pass. Under skewed streams it saves time for methods with expensive updates, because declined batches skip the update.

Cost of the gate (one A100-PCIE-40GB, B = 64)
per batch, CIFAR-10-C TENTsource inference 53 ms, TTA update 43 ms, gate 108 ms
extra compute per batchtwo frozen forward passes, no backward: 2.0× the source forward in FLOPs
wall-clock, gated / native, BN-Adapt and TENT1.2–2.2× on shuffled streams; 0.8–2.2× on class- or slide-ordered streams, where declined batches skip the update
wall-clock, gated / native, methods with expensive updates (SAR-BN, RoTTA, TRIBE)0.6–1.5× on shuffled streams, 0.06–0.95× on skewed streams
peak GPU memoryunchanged within 3 MiB where the gate adapts (the frozen copy is counted); lower where it declines, e.g. TENT on class-ordered CIFAR-10-C about 3,000 → 2,200 MiB
offline calibration, once per model0.6–4 GPU-h on CIFAR, 3–12 GPU-h on pathology; one BN-Adapt fit serves TENT, SAR-BN, EATA and DeYO

We added the new Table A24 (App. F.9) which reports the timings for all ten methods on CIFAR-10-C and for the main methods on CIFAR-100-C, Camelyon17 and NCT, all timed on the same GPU and settings.

Beyond the two questions, the revision adds three results: the full gate on ImageNet-C (Table A4), a comparison with ATC and AutoEval under our protocol (Table A20), and the gates applied unchanged to a second hospital and to tumor prevalences down to 2% (Table A14).

We are grateful for the reviewer's suggestions, which have made the revised paper clearer. If any point leaves a question open, we would be glad to answer it before the discussion closes. We would be grateful if the reviewer would consider raising the rating in the light of the new results.

We thank the reviewer again for their constructive review. We have updated the paper with further results requested by the other reviewers: a comparison with Mutlu et al. and MORPHEUS under our protocol, and a comparison of the calibrated gate with the label-free rule on ImageNet-C. We would be glad to answer any remaining questions the reviewer might have before the discussion closes.

Reviewer_HHHROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work shows failures of batch-normalization-based test-time adaptation under locally skewed class compositions. It uses controlled composition sweeps and statistics-swap experiments to study this failure, then proposes a source-calibrated linear predictor of adaptation gain to decide whether to run an adaptation method or serve the frozen source model.

Critical strengths

The paper diagnoses BN-based adaptation failures and proposes a source-calibrated gate predicting adaptation gain. The controlled interventions, existing ablations, and some pathology results are useful.

Critical weaknesses
  1. The novelty of the forecasting gate remains insufficiently established. AutoEval (§§3.1–3.3), MORPHEUS (§2), and Mutlu et al. (§3.4) provide related precedents, although their targets and calibration access differ. Appendix F.4 and Table A18 provide useful controls, but a prior-feature comparison under matched calibration and decision protocols would better establish the incremental contribution.

  2. The full gate’s effectiveness at ImageNet scale remains unverified. Appendix E.3 reports ΔH AUROC of 0.65–0.75 but explicitly states that the full gate was not evaluated on ImageNet-C. This limits generalization claims without invalidating the evaluated datasets. A bounded full-gate evaluation, or a clear explanation of the limiting factor, is needed.

  3. The source-data access and configuration-selection protocols need clarification. Labeled-source calibration is explicitly disclosed (§2, §4, Appendix F.1), making the gate unavailable from a checkpoint alone. This does not itself imply an unfair comparison, and Table A18 provides useful calibration-free controls. The authors should specify each method’s source-data access and how its configuration was selected.

  4. The severe DeYO degradation is insufficiently explained. On shuffled Camelyon17, DeYO achieves 53.17%, versus 93.48% for BN-Adapt and 72.91% for source (Table 3). Appendix D uses Adam at 1e-3, unlike DeYO’s original SGD-based protocol (Appendix E.3). This difference does not establish unfairness, since the original protocol does not cover this setting, but warrants a selection rationale and filtering/update diagnostics.

  5. SAR-BN’s contribution beyond BN-Adapt remains unclear in this setting. Their six Camelyon17 accuracies match in Table 3, and Table A4 separately notes matching slide-order metrics. This does not establish an implementation error; filtering, resets, or negligible updates could explain it. Update magnitudes and update/reset counts would clarify the observed behavior.

Questions for authors
  1. What evidence establishes the proposed signals’ incremental value over prior forecasting representations under matched calibration?

  2. Why was the full gate not evaluated on ImageNet-C, where ΔH achieves only 0.65–0.75 AUROC?

  3. Did baselines receive comparable source-data access and tuning budgets? If not, how is that difference accounted for?

  4. Does DeYO’s 53.17% shuffled Camelyon17 accuracy persist under a source-validated configuration? What justifies replacing SGD with Adam?

  5. Does SAR-BN perform sustained parameter adaptation on Camelyon17? Please explain its matching BN-Adapt accuracies using update and reset counts.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HHHR (1/2)

We thank the reviewer for their thorough and constructive feedback. We have tried to address all of their points: each of the points has a new table or experiment behind it. We have uploaded a revised PDF with changes and additions based on the review.

Q1. "What evidence establishes the proposed signals’ incremental value over prior forecasting representations under matched calibration?" We agree that we did not establish this well enough in our original submission, so we ran the comparison the reviewer mentions. We compare with ATC and AutoEval, whose code is released, and with Mutlu et al. and MORPHEUS, which we implemented from their papers. Each method's signal replaces ours and is fitted on our calibration streams: ATC and AutoEval through a linear forecast of the per-batch gain, Mutlu et al. and MORPHEUS through their own predictors. All on BN-Adapt and the five CIFAR-10-C stream types of Table 3 (App. F.7, Table A20):

Signal deciding whether to adaptAccuracyPer-batch AUROC
never adapt (source model)60.1–
always adapt (native BN-Adapt)45.5–
ATC's accuracy estimate, maximum-confidence score60.20.69
ATC's accuracy estimate, negative-entropy score60.10.69
AutoEval's Fréchet distance between source and batch features62.10.78
Mutlu et al.61.50.77
MORPHEUS63.00.83
the gate (ΔH, D_q, RG)66.60.94

The Fréchet distance of AutoEval returns to the source model on the class-ordered and persistent streams, but gives up most of the gain on shuffled streams (66.7 % against 79.3 % for our method). Mutlu et al. and MORPHEUS fall below the source model on class-ordered streams (53.4 % and 56.0 % against 59.5 %).

You are right that the targets differ, and Sec. 2 now mentions this. ATC and AutoEval estimate how accurate a fixed model is, not if adapting can actually help. MORPHEUS predicts each method's accuracy after adapting and makes one decision for a whole domain (one corruption type, thousands of images). It does not predict this per batch. Thus, it cannot react to a single skewed batch inside a balanced stream. Mutlu et al. are the closest to our gate: they also decide per batch whether to adapt. Both differ from the gate in two ways. First, their features describe how the frozen model embeds the batch, and neither applies the batch's own statistics. In our work, ΔH does exactly that: it is the operation Sec. 3 identifies as the cause of the failure, and class-ordered streams are where the two methods lose to the source model. Second, in their papers both predictors are trained on labeled shifted data from the benchmark itself: they adapt on some of its shifts, measure the outcome, and test on the held-out shifts. Our gate is fitted on source data only, and its label-free rule ΔH ≤ 0 needs no calibration at all (Table A22).

Q2. "Why was the full gate not evaluated on ImageNet-C, where ΔH achieves only 0.65–0.75 AUROC?" To save computation, we initially stuck with our four datasets. We have now added ImageNet-C as well (Sec. 5.1, App. D.3, Table A4, Figure A2). BN-Adapt and TENT drop from the 18.2 % source model to 3–6 % on class-ordered and single-class-window streams. With the gate both are back at the source model's performance, and on shuffled streams they keep about 90 % of their gain. The gate's AUROC under skew is 0.70–0.75, inside the range you quote for ΔH alone. That is enough here, because only 12–20 % of the skewed batches gain from adapting. SAR-BN, EATA and DeYO with their published ImageNet settings all collapse to 0.9–5.1% and come back to 17.8–18.0% with the gate added. Pseudo-labels instead of ImageNetV2 labels give the same gate under skew, and one labeled image per class stays within 0.6 points (Sec. 5.4, App. F.10).

Q3. "Did baselines receive comparable source-data access and tuning budgets? If not, how is that difference accounted for?" Tuning: no method was tuned, on source or on target streams. Each runs the settings listed in App. C (its published defaults, with one shared optimizer for TENT, EATA and DeYO, see Q4), and each native and gated pair runs the same settings.

Source data: of the methods in Tables 3–4, only EATA (2,000 held-out source images for its Fisher regularizer, as in the published code) and Hybrid-TTN (labeled source-training images to score its channel sensitivities) use source data beyond the checkpoint; the others use none. Our calibrated gate uses a labeled or pseudo-labeled held-out set: 25 images per class are enough on CIFAR-10-C and 200 per class on Camelyon17 (App. F.10; the full curves are in our reply to M3Mh, Q1). Our label-free rule ΔH ≤ 0 uses no source data at all (Table A22). These two settings account for the difference: with pseudo-labels the gate uses no labels, and the rule is compared with the calibrated gates in Table A22.

AuthorsOpenReview ↗

Response to Reviewer HHHR (2/2)

Q4. "Does DeYO’s 53.17% shuffled Camelyon17 accuracy persist under a source-validated configuration? What justifies replacing SGD with Adam?" Adam at 1e-3 was the shared optimizer of TENT, EATA and DeYO on CIFAR and NCT. The reviewer is right that on Camelyon17, where TENT follows Zhang et al. (2022) with SGD, DeYO should have used SGD as well. We have now run DeYO with its published optimizer (SGD, learning rate 2.5e-4, momentum 0.9, its ImageNet-C setting; DeYO was not evaluated on Camelyon17) and added it to the paper (Table A15). To the first question the answer is yes, in every configuration we tried (App. F.4, Table A15). We did not select a configuration on source data, because every variant, including the published one, stays below the source model. The "19 orders" in the table are all hospital-2 streams of Tables 4, A13 and A14 (shuffled, slide order, within-slide shuffled, ten random slide orders, three tumor prevalences and three persistent compositions); those columns are means over the 19.

DeYO on Camelyon17, hospital 2shuffled19 ordersgated, 19 orders
4×4 patch shuffle, Adam (the paper's setting, rerun; the patch shuffle is random, hence not 53.17)60.552.177.0
2×2, 8×8 or pixel shuffle51.5–57.949.6–52.977.0
published SGD protocol55.350.877.0
source model (the 19 orders have a different class mix)72.977.0

Neither the optimizer nor the augmentation is the cause: every variant stays at 50–53 % over the 19 orders. What they share is DeYO's sample selection: the entropy filter admits 99 % of each batch, the PLPD filter keeps about a third, and DeYO updates on every batch with no reset. TENT reaches 93.3 % on the same shuffled stream (Table 4).

Q5. "Does SAR-BN perform sustained parameter adaptation on Camelyon17? Please explain its matching BN-Adapt accuracies using update and reset counts." No, it does not. SAR updates on every batch on Camelyon17, but it also resets after every batch, so what is left is BN-Adapt. We counted this over the 19 hospital-2 orders (App. F.4, Table A16). SAR’s reliability filter keeps 49 % of the samples in each batch and SAR takes one gradient step on every batch. In addition to this, SAR has a safety reset: when the moving average entropy falls below 0.2, it removes its updates and goes back to the source weights. On Camelyon17 this reset fires on 100 % of the batches, because with only two classes, the kept samples are so confident that their average entropy is under the reset threshold of 0.2. We report counts and not update magnitudes, because every update is undone by the reset before the next batch.

We hope the matched comparison (Q1) and the ImageNet-C evaluation (Q2) settle the two points the review marks as central. Furthermore, we hope that the source-data comparison (Q3) and the two audits (Q4, Q5) answer the rest of the reviewer's concerns. If any point leaves a question open, we would be glad to answer it before the discussion closes. Finally, we would be grateful if the reviewer would reconsider the rating in the light of the new evidence.

AuthorsOpenReview ↗

Update to Q1

We have additionally implemented Mutlu et al. and MORPHEUS from their papers and added these to the results table in Q1. Both methods were run with the same calibration streams and decision protocol. They reach 61.5 % / 0.77 and 63.0 % / 0.83, against 66.6 % / 0.94 for the gate.

Reviewer_9bywOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the impact of class-imbalanced test batches on BatchNorm-based test-time adaptation, showing that unreliable BN statistics alone can cause substantial performance degradation. Based on these observations, the authors propose a source-calibrated forecasting gate that predicts whether adaptation will be beneficial for each incoming batch.

Critical strengths

(1) The paper addresses a practically important challenge of BN-based TTA under class-imbalanced test batches.

(2) The controlled experiments provide valuable insights into how unreliable BN statistics lead to adaptation failure.

Critical weaknesses

(1) Limited methodological novelty and significance. The adverse effects of class imbalance on BN statistics and TTA performance have been extensively investigated in prior studies, including TRIBE[1], TEMA[3], UnMix-TNS[3], and related work. Although the controlled experiments provide additional insights, the proposed approach mainly combines three statistical measurements through a simple linear forecasting model. The methodological novelty and substantial advances over existing approaches remain insufficiently demonstrated.

(2) Limited applicability and insufficient justification of the calibration strategy. A broadly applicable TTA method should ideally be model-agnostic and operate without requiring access to source data. However, the proposed approach relies on additional labeled source data for supervised calibration and BN-specific statistics to make adaptation decisions. These requirements substantially restrict its applicability, particularly in practical model-only TTA scenarios or architectures without BatchNorm layers. Moreover, the forecasting model is calibrated on synthetically shifted and artificially constructed test streams, raising concerns about its generalization to unseen target distributions. Although some generalization experiments are provided, the necessity of these additional assumptions and their trade-offs against practical applicability remain insufficiently justified.

(3) Limited evidence for the added value of the forecasting mechanism. As shown in Table 2, substantial improvements under severe class imbalance largely result from skipping adaptation and reverting to source-model predictions, rather than achieving better target-domain adaptation. While avoiding harmful adaptation is valuable, the simple label-free entropy-based rule in Table A18 already achieves comparable performance in several settings. Moreover, the proposed gate can degrade the performance of robust methods such as TRIBE[1], and the complete approach is not evaluated on ImageNet-C. These results leave the additional benefits and general applicability of the supervised forecasting mechanism insufficiently established.

(4) Presentation and clarity. The manuscript could benefit from clearer organization and more explicit definitions of experimental settings and notation. For example, the detailed statistics-swap configurations in Table 1 and the stream constructions in Table 2 are primarily explained in the appendix, while several symbols in Figure 2 are introduced only in the subsequent text. These presentation issues make the methodology and experimental comparisons unnecessarily difficult to follow.

[1] Towards Real-World Test-Time Adaptation: Tri-net Self-Training with Balanced Normalization

[2] Unraveling Batch Normalization for Realistic Test-Time Adaptation

[3] Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9byw (1/3)

We thank the reviewer for their careful review and for stating that the controlled experiments provide additional insights. We have uploaded a revised PDF with changes and additions based on the review. Table and section numbers are those of the revised PDF.

W1. "extensively investigated in prior studies" TRIBE, TEMA, UnMix-TNS and others showed that skewed batches damage BN statistics, and Sec. 2 now credits this. However, each method measured the performance degradation in a protocol that moves a second variable together with the batch composition. We isolate the batch composition without moving any of these variables at the same time. The table compares each paper's main protocol (✓ held fixed while the batch composition varies, ✗ moves with it):

ProtocolStream class priorBatch orderBatch size
TRIBE, DELTA (global and local imbalance)✗✗✓
NOTE, RoTTA, UnMix-TNS, SAR (temporally correlated or class-sorted streams)✓✗✓
TEMA (small i.i.d. batches)✓✓✗
Hybrid-TTN (one batch adapted offline; the batch is the whole target set)✗no stream✓
Sec. 3 (rotated Dirichlet, d_S = 0)✓✓✓

In Sec. 3 we state three things the cited papers do not:

FindingWhat prior work showedWhat Sec. 3 adds
1. Batch composition alone makes adaptation harmfulHybrid-TTN shows that recomputing BN statistics on one class-imbalanced test batch can lower accuracy, even without corruption.The same on full corrupted streams that stay class-balanced: TENT goes from +23.9 to −32.7 points over nine skew levels at B = 200, and the loss at the most skewed level is similar at B = 16 and 64 (Table 1, Table A3).
2. The loss is in the normalization, not in the updateDELTA, SAR, TRIBE and UnMix-TNS name biased batch statistics as a cause and replace the normalization, and Hybrid-TTN shows on one batch that the statistics alone cause a drop. In these protocols the update is not separated from the statistics: DELTA also blames the update, and SAR and DeYO protect it by filtering samples.The statistics explain all of the loss. BN-Adapt, with no gradient step, loses as much as TENT (within 0.1 points at the most skewed level), and the methods that protect the update (SAR-BN, EATA, DeYO) lose the same 32.6 (Table A2). Protecting the update therefore cannot help: the decision has to come before the batch is normalized, which is what the gate does.
3. The loss comes from statistics dominated by the batch's own class, mostly their varianceTEMA attributes the errors of small batches to their low class diversity. Hybrid-TTN explains the failure through the class mean that normalization subtracts.Neither explanation is enough. Statistics from a batch dominated by another class are just as skewed, yet keep 83 % of the gain, so low diversity alone does not cause the loss. A balanced variance alone recovers 80 % of the gap, so most of the loss sits in the variance, not the mean (Table 2).

The closest work is Vianna et al. (Hybrid-TTN). They show the loss when a single batch, adapted offline, has an imbalanced label distribution, with and without covariate shift, and explain it through the class mean that normalization subtracts. In their setting the batch is the whole target set, so the batch composition and the target class prior are one variable. We show the loss on full streams that stay class-balanced, with the stream prior, batch order and batch size held fixed, and the swap of Sec. 3.3 goes beyond their explanation: equally skewed statistics from another class keep 83 % of the gain, and most of the loss sits in the variance.

W1. "a simple linear forecasting model" Under the same configuration, we tried the signals of the two prior estimators with released code, ATC and AutoEval. They reach 60.1–62.1% accuracy and AUROC 0.69–0.78. Our gate reaches 66.6% accuracy and 0.94 AUROC (App. F.7, Table A20, our reply to reviewer HHHR, Q1). We use a simple linear forecaster because larger models add very little. To illustrate, a logistic and a gradient-boosted forecaster on the same three measurements gain at most 0.4 points over it on CIFAR-10-C, CIFAR-100-C and NCT, and at most 2.0 on Camelyon17 (Sec. 5.4, App. F.8).

AuthorsOpenReview ↗

Response to Reviewer 9byw (2/3)

W2. "architectures without BatchNorm layers" We agree with the reviewer, therefore the abstract now scopes the gate to BN-based methods, and Sec. 6 states that both the failure and the probe are specific to BatchNorm architectures only. The methods the reviewer cites (TRIBE, TEMA, UnMix-TNS) act on BatchNorm statistics as well, as do the other methods in Tables 3–4.

W2. "relies on additional labeled source data" We calibrate because which batches harm depends on the method and the dataset: ΔH carries the CIFAR-10-C gate, D_q the CIFAR-100-C gate and RG the Camelyon17 gate, and RoTTA and TRIBE gain on skewed batches on which BN-Adapt collapses, so no fixed threshold fits all of them. Calibration learns this once, before deployment, on the only data where the gain of adapting can be measured: labeled source images, arranged into shuffled and skewed streams so that the fit sees batches where adapting helps and batches where it hurts. No target data enters it, and it adds 7–13 points over the label-free rule for the methods built for skewed batches (Table A22).

Our highest-performance configuration does indeed require a (limited) amount of source data. EATA (2,000 source images), TTN and Hybrid-TTN (source training data for their channel sensitivities), and LSA and DART (labeled source data, to train an adapter or a refinement module before deployment) need it too. In Sec. 2 we now mention this and in Sec. 6 we highlight the trade-off.

The revision also measures how much calibration data is actually needed: CIFAR-10-C and Camelyon17 only require 25 labeled images per class and 200 patches per class respectively. These are almost equal to the paper's gate where we used more data (within 0.9 points; App. F.10, Table A27, Figure A5). The frozen model's own pseudo-labels give one within 0.5 points of the labeled gate on CIFAR-10-C and NCT and within 1.5 on Camelyon17, for every method but TRIBE (Table 3, App. F.8), and the same gate under skew on ImageNet-C (Table A4). A held-out set is not needed either: calibrated on the source-training images, gated BN-Adapt reaches 66.7 % on CIFAR-10-C against 66.6 % with the held-out set (App. F.8).

Finally, we also propose a fully unsupervised gate, the rule ΔH ≤ 0, which requires no source data at all. It avoids the collapse of every BN-based method and sits 1.1–1.4 points under the calibrated gate averaged over the five CIFAR-10-C stream types, 4 points on shuffled streams (Table 3, Table A22).

W2. "artificially constructed test streams" We have 3 experiments on data the calibration never saw, where for the rebuttal we added the tumor prevalence experiment as an additional check. Points 2 and 3 were already included in the appendix.

  1. We applied the Camelyon17 gates unchanged to a second hospital absent from calibration, and to streams with different tumor prevalences (2, 5, 10 %) (table below, App. F.4, Table A14). (Newly added)
  2. We refitted the gates without the class-ordered and long-tail calibration streams: they stay within 0.15 points of the full gates on all datasets (App. F.8).
  3. We evaluated the CIFAR-10 gates on the 6 corruption types that were not simulated in the calibration. The gated BN-based methods kept 99.6% of their gain on shuffled streams (App. F.8).
Applied unchanged toSourceBN-AdaptTENTEATA
a second hospital, slide order82.563.9 → 84.057.5 → 84.964.1 → 84.5
tumor prevalence 2 %94.138.2 → 93.544.6 → 94.043.5 → 93.9

One composition goes the other way: when a 30 % tumor share is held for the whole stream, native BN-Adapt beats its gate (91.6 against 84.4, Table A14).

AuthorsOpenReview ↗

Response to Reviewer 9byw (3/3)

W3. "skipping adaptation and reverting to source-model predictions" It is indeed, and Sec. 3 shows why this is the right action. The statistics explain all of the loss, so a skewed batch is already lost when it is normalized, and no safer update can recover it: SAR-BN, EATA and DeYO, which protect the update, lose the same 32.6 points as BN-Adapt (Table A2). We compared the gate with a labeled per-batch oracle, and it does the same: on class-ordered CIFAR-10-C the oracle ends 1.2–1.3 points above the source model, the gated methods 0.3 (Table A17), and on class-ordered NCT, no method in Table 4 ends above the source model. Our proposal is to decide when adaptation is needed. Where there is something to gain, the gate does so: gated BN-Adapt, TENT, SAR-BN and EATA all end 2–6 points above the source model on slide-ordered Camelyon17, where gated RoTTA is the most accurate method (Table 4), every gated BN-based method ends above native and source on the temporal and recovery CIFAR-10-C streams (Table 3). We further agree that the gate costs TRIBE on CIFAR-10-C (−1.5 on average), and the calibration streams already flag this in advance with no target data (Sec. 5.4, Table A21). Thus, for these settings it can be decided before deployment that the gate does not need to be applied.

W3. "already achieves comparable performance in several settings" You are correct that for the BN-based methods the label-free rule comes close. For the methods built for skewed batches it does not: the rule does not stop RoTTA, UnMix-TNS, TRIBE and Hybrid-TTN from falling below their native versions, 7–13 points under the calibrated gate, and 12–17 under class order for RoTTA, UnMix-TNS and TRIBE (RoTTA 73.0 against 60.5, TRIBE 77.9 against 60.7; Table A22). On ImageNet-C, BN-Adapt and TENT fall to 3–6 % under class order and one-class windows against an 18 % source model. Our gate returns them to 18 % and keeps about 90 % of their gain on shuffled streams (App. D.3, Table A4, Figure A2).

We would also like to highlight our pathology results, which are not discussed in the review. In this medical setting the gate shows a lot of strength: on slide-ordered Camelyon17 it returns BN-Adapt, TENT and EATA from 56–64 % to 75–79 %, above the source model's (72.9). Furthermore, it makes gated RoTTA the most accurate method. On class-ordered NCT it lifts TRIBE from 43 to 95 % (Table 4, Sec. 5.2–5.3).

W4. Presentation. We have made the following changes based on the reviewer's feedback to improve the presentation of the paper. We have added the following: equations for each swap of Table 2 (Sec. 3.3), a new Table 1 with the skew levels, the stream constructions and lengths above Table 3, Figure 2's caption defines ΔH, D_q and RG, a notation line follows the gate equation, and the appendix now has a contents page.

We hope that the comparison of protocols (W1) clarifies what is new in the diagnosis, and that the comparison with prior signals (W1), the second hospital (W2) and the ImageNet-C evaluation (W3) address your concerns on significance and applicability. If any of the points above leaves a question open, we would be glad to answer it or run more experiments before the rebuttal deadline. We would be grateful if you would reconsider the rating in the light of the above.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2)

We would like to start by thanking all the reviewers for their careful and constructive reviews. We are pleased to see the reviewers find the results useful and recognise the aim of our paper: "controlled experiments provide valuable insights into how unreliable BN statistics lead to adaptation failure" (9byw), "controlled interventions, existing ablations, and some pathology results [that] are useful" (HHHR), a method that is "well motivated and simple" and "not over-engineered" (M3Mh). During the rebuttal we ran nine new experiments, covering every point the reviewers raised, and added them to the revised PDF (changes in blue).

Our paper in a nutshell. A single batch can push a test-time adaptation method substantially below the source model. For example, this is the setting of pathology, where slides are processed one at a time. We predict this drop per batch and before the update, based on three label-free measurements computed with the frozen source model. The gate is calibrated once on source data, or used uncalibrated with the rule ΔH ≤ 0. On slide-ordered Camelyon17, BN-Adapt, TENT and EATA fall to 56–64 %, below the 72.9 % source model. With our gate they reach 75–79 %, and gated RoTTA becomes the most accurate method at 85.5 % (76.3 % native). On class-ordered NCT the gate lifts TRIBE from 43 % to 95 %. The same gate prevents the collapse on CIFAR-10-C, CIFAR-100-C and ImageNet-C, and keeps about 97 % of the gain on shuffled CIFAR-10-C streams and about 90 % on ImageNet-C (Tables 3–4, A4, A12).

Is the diagnosis new, given TRIBE, TEMA and UnMix-TNS? (9byw W1) Yes. Previous work showed that skewed batches damage BN statistics, and Sec. 2 credits this. But each paper measured the degradation in a protocol that moves a second variable together with the batch composition: the stream class prior (TRIBE, DELTA), the batch order (NOTE, RoTTA, UnMix-TNS, SAR) or the batch size (TEMA). None holds all three fixed. In Sec. 3 we hold all three fixed and vary the batch composition alone, and we replace the normalization statistics directly to see which part of them carries the loss. This gives three results that the cited papers do not state:

  1. a single batch composition alone is enough to degrade performance, on streams that are class-balanced.
  2. the normalization statistics explain all of the loss: BN-Adapt, with no gradient step, loses as much as TENT, and the methods that protect the update by filtering samples (SAR-BN, EATA, DeYO) lose the same 32.6 points, so protecting the update cannot help
  3. it comes from statistics dominated by the batch's own class, mostly through the variance. Equally skewed statistics from another class keep 83 % of the gain, so low class diversity alone (TEMA's explanation) is not the cause

Prior work names biased statistics as a cause, but does not separate them from the update. Fact 2 is why our gate decides before the batch is normalized, instead of making the update safer. We added the skew levels as a new Table 1, and wrote a row-by-row comparison with the related work in our reply to 9byw, W1.

Is the gate new, given AutoEval, MORPHEUS and Mutlu et al.? (9byw W1, HHHR Q1) Yes. The gate decides per batch and before an update whether a TTA method should be run or not. Otherwise the source model is served. TRIBE, TEMA and UnMix-TNS correct the statistics and always adapt. ATC and AutoEval estimate how accurate a fixed model is, not whether adapting helps. MORPHEUS predicts each method's accuracy after adapting and decides once for a whole domain, so it cannot react to a single skewed batch. Mutlu et al. are the closest: they also decide per batch. The gate differs from both in two ways. First, their features describe how the frozen model embeds the batch, and neither applies the batch's own statistics. ΔH does exactly that: it is the operation that Sec. 3 identifies as the cause of the failure. Second, in their papers both predictors are trained on labeled shifted data from the benchmark itself. Our gate is fitted on source data only, and its label-free rule ΔH ≤ 0 needs no calibration at all.

We also ran the comparison HHHR asks for. We put these methods inside our protocol, on the same calibration streams. They reach 60.1–63.0% accuracy and AUROC 0.69–0.83, while our gate reaches 66.6% and 0.94 AUROC. On class-ordered streams Mutlu et al. and MORPHEUS fall below the source model (53.4% and 56.0% against 59.5%), while the gate stays at it (59.8%) (new Table A20, table in our reply to HHHR, Q1).

Does it scale to ImageNet-C? (9byw W3, HHHR Q2) Yes: we ran the full gate on ImageNet-C for all five BN-based methods. Under skewed batches BN-Adapt and TENT fall to 3–6 %, and our gate returns them to the source model accuracy and keeps about 90 % of their gain on shuffled streams (new Table A4 and Figure A2).

AuthorsOpenReview ↗

General Response (2/2)

Does the gate need labeled source data? (9byw W2, HHHR Q3, M3Mh Q1) Much less than we used, and no data is required for our unsupervised rule ΔH ≤ 0. We calibrate because which measurement signals harm differs between methods and datasets, and source data is the only place where the gain of adapting can be measured before deployment. We ran two new calibrations. We calibrated on the source model's own predictions instead of the ground truth source labels, and the gate stays very close to the labeled one. We then reduced the calibration set: 25 images per class on CIFAR-10-C and 200 patches per class on Camelyon17 already give the gate we report (new Figure A5; calibration curves in our reply to M3Mh, Q1). Finally, our unsupervised rule is not a fallback: for the BN-based methods it is within 1–2 points of the calibrated gate on CIFAR-10-C and on par with it on Camelyon17 (Table A22). Calibration matters most for the methods designed for skewed batches (RoTTA, UnMix-TNS, TRIBE, Hybrid-TTN), where it adds 7–13 points.

Does a gate calibrated on constructed streams transfer? (9byw W2) Yes: we applied the gates unchanged to a second unseen hospital (slide order: BN-Adapt 63.9 → 84.0, source 82.5) and to tumor prevalences of 10, 5 and 2 %, and they hold there, as they do on six corruption types never seen in calibration (new Table A14; table in our reply to 9byw, W2).

What else did we run? (HHHR Q4–Q5, M3Mh Q2) We reran DeYO on Camelyon17 under four augmentations and under its published SGD protocol, counted SAR-BN's update and reset events, and timed the gate against every method on one A100 (new Tables A15, A16 and A24).

Changes to the paper (blue in the PDF):

  • Abstract: the gate scoped to BN-based methods (9byw W2).
  • Sec. 2: positioning against the named methods, and which prior methods also use source data (9byw W1–W2, HHHR Q1, Q3).
  • Sec. 3–4: the BN normalization and the gate written as equations, each swap setting written as the statistics it substitutes, a new Table 1 for the skew levels, a notation line, and ΔH, D_q and RG defined in Fig. 2's caption (9byw W4).
  • Sec. 5: stream constructions above Table 3; ImageNet-C; pseudo-labeled and label-free rows in Table 3; in Sec. 5.4 a calibration-time rule for when a method needs the gate, and the calibration-size result (9byw W2–W4, HHHR Q2–Q3, M3Mh Q1).
  • Appendix: the new tables and figures named above, a comparison with a labeled per-batch oracle (Table A17), the DeYO and SAR-BN audits HHHR asked for (Tables A15–A16), and a contents page.
  • Numbering: the new Table 1 makes the submission's Tables 1–3 the revision's Tables 2–4, and the submission's Table A18 (the label-free rule) is the revision's Table A22; our replies use the revision's numbers.
AuthorsOpenReview ↗

Response to ImageNet questions

We thank the reviewer for the follow-up questions. We have added the following to the updated PDF: the comparison with the rule is in the new Table A28 (App. F.10), and the ImageNet calibration setup is in App. E.1 and Table A5.

1. "On ImageNet-C, how much does the full gate, calibrated using 10,000 ImageNetV2 images, improve over the source-data-free rule ΔH ≤ 0?" Table A28 shows the accuracy and the gate-on rate of both for each of the five methods and three stream types. On the two skewed stream types the gate does not improve on the label-free rule. Both return every method to the source model (18.2 %) and differ by at most 0.05 points. On shuffled streams the label-free rule adapts on only 27 % of the batches and gives up most of the gain of adapting, while the calibrated gate adapts on 87–95 % of them and keeps it. On these streams our calibrated gate is 7.3 to 20.5 points above the label-free rule: it keeps 89–95 % of the gain of adapting, whereas the label-free rule keeps 26–35 % of the gain.

2. "Could you specify the ImageNet perturbation families and levels, calibration-stream construction, and sampling procedure?" The reviewer is right that App. E.1 and Table A5 only described CIFAR and pathology, and we have added ImageNet to both. We use the same setup as for CIFAR. We use the same perturbation families and levels and the same 25 calibration streams per condition (Table A5), with B = 64. The only adjustment we make is for the image size, where we scale the blur to the 224-pixel images. Every stream is drawn from all 10,000 ImageNetV2 images, and the smaller budgets are class-stratified random subsets of them.

3. "clarify whether 0.6 is a maximum or average gap, and report variability across sampled calibration subsets." The 0.6 was the maximum for the one subset we had drawn, but we have now drawn four more random one-image-per-class subsets and repeated the calibration and all ImageNet-C streams for each, for BN-Adapt and TENT (Table A28, last two columns). Over these five subsets the standard deviation is 0.07 points on the skewed streams and 0.09 (BN-Adapt) and 0.23 (TENT) on shuffled streams. We also ran the one-image-per-class gate for SAR-BN, EATA and DeYO, with one subset each. Under skew the one-per-class gate is at most 0.4 points below the ten-per-class gate for BN-Adapt, TENT, SAR-BN, EATA and DeYO. On shuffled streams it is within 1.0 point of it, from 0.8 above (BN-Adapt) to 1.0 below (DeYO). We have edited the sentence in Sec. 5.4 and App. F.10. With one image per class the gate relies mostly on ΔH, with a fitted threshold instead of the threshold being zero. On shuffled streams this takes BN-Adapt from 22.8 % (the label-free rule) to 30.8 % and TENT from 24.6 % to 40.7 %.

We hope this answers the remaining points on ImageNet-C, and that our earlier answers to the other questions have addressed the reviewer's other concerns. We thank the reviewer for all questions and remarks, which have made the paper more precise. If anything is still unclear, we would be glad to answer it before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.