- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a gate that forecasts, before each test-time adaptation (TTA) update, whether adapting on the current batch will help or hurt. It uses label-free signals from a frozen source model—the predicted batch composition and the model’s response to the batch’s BN statistics—combined with a linear rule fitted once on held-out labeled source data, so no target labels are needed. It validates the gate on four datasets, wrapping several BN-based and skewed-stream TTA methods under different stream orderings.
Critical strengths
1. The diagnostic experiments are very well designed. The controlled composition sweep keeps the whole stream class-balanced and varies only batch composition. The statistics-swap analysis then pinpoints the key factor: normalizing a batch with BN statistics dominated by its own class. This is a clean and useful finding.
2. The method is well motivated and simple. The entropy-response signal ΔH directly matches the failure identified in the diagnosis. The design is not over-engineered: a linear forecast, nonnegative weights, a zero decision threshold, and one calibration set per dataset. The method and the analysis fit together well.
3. The experimental scope is broad, and the pathology results have practical meaning. The gate is tested with five BN-based methods and five methods designed for skewed streams, across multiple stream orders. On Camelyon17 slide order, it prevents the collapse of BN-based methods and makes gated RoTTA the most accurate method. These results are relevant to real slide-by-slide processing.
Critical weaknesses
1. The calibration set is a practical limitation. It can be taken from source training data, but in some domains—such as rare diseases in medical imaging—labeled data may be very scarce. The authors already show results from 5,000 down to 1,000 calibration images; it would be helpful to push this further and show the failure curve on smaller calibration sets.
2. The extra computation is not fully quantified. The paper states that the gate adds two forward passes per batch. In practice, it is unclear how much extra wall-clock time and compute this costs, especially relative to the TTA update itself and to source inference. A clear overhead measurement would help readers judge deployment cost.
Questions for authors
1. On calibration size: Could the authors report the gate’s performance with much smaller calibration sets, for example 10, 25, 50, 100, and 200 images per class, especially on the pathology datasets? A failure curve would clarify when the gate stops being reliable and would make the practical limitation easier to assess.
2. On computational overhead: Could the authors quantify the extra cost of the two forward passes per batch? Useful numbers would include wall-clock latency, FLOPs, peak memory, and throughput overhead, reported separately for source inference, the TTA update, and the gate. It would also help to state whether the extra frozen model copy is included in the memory measurement.
3 comments
Response to Reviewer M3Mh (2/2)
Q2. Overhead. The gate adds two frozen forward passes per batch and no backward pass. Under skewed streams it saves time for methods with expensive updates, because declined batches skip the update.
| Cost of the gate (one A100-PCIE-40GB, B = 64) | |
|---|---|
| per batch, CIFAR-10-C TENT | source inference 53 ms, TTA update 43 ms, gate 108 ms |
| extra compute per batch | two frozen forward passes, no backward: 2.0× the source forward in FLOPs |
| wall-clock, gated / native, BN-Adapt and TENT | 1.2–2.2× on shuffled streams; 0.8–2.2× on class- or slide-ordered streams, where declined batches skip the update |
| wall-clock, gated / native, methods with expensive updates (SAR-BN, RoTTA, TRIBE) | 0.6–1.5× on shuffled streams, 0.06–0.95× on skewed streams |
| peak GPU memory | unchanged within 3 MiB where the gate adapts (the frozen copy is counted); lower where it declines, e.g. TENT on class-ordered CIFAR-10-C about 3,000 → 2,200 MiB |
| offline calibration, once per model | 0.6–4 GPU-h on CIFAR, 3–12 GPU-h on pathology; one BN-Adapt fit serves TENT, SAR-BN, EATA and DeYO |
We added the new Table A24 (App. F.9) which reports the timings for all ten methods on CIFAR-10-C and for the main methods on CIFAR-100-C, Camelyon17 and NCT, all timed on the same GPU and settings.
Beyond the two questions, the revision adds three results: the full gate on ImageNet-C (Table A4), a comparison with ATC and AutoEval under our protocol (Table A20), and the gates applied unchanged to a second hospital and to tumor prevalences down to 2% (Table A14).
We are grateful for the reviewer's suggestions, which have made the revised paper clearer. If any point leaves a question open, we would be glad to answer it before the discussion closes. We would be grateful if the reviewer would consider raising the rating in the light of the new results.
We thank the reviewer again for their constructive review. We have updated the paper with further results requested by the other reviewers: a comparison with Mutlu et al. and MORPHEUS under our protocol, and a comparison of the calibrated gate with the label-free rule on ImageNet-C. We would be glad to answer any remaining questions the reviewer might have before the discussion closes.
Response to Reviewer M3Mh (1/2)
We thank the reviewer for the careful reading of our paper. We agree with both weaknesses: the calibration set is a practical limitation, and the submission did not quantify the cost of the gate. We therefore ran both experiments the reviewer asked for, and they are added to the revised PDF (App. F.9–F.10, Tables A24 and A27, Figure A5).
Q1. Smaller calibration sets. On CIFAR-10-C, 25 labeled images per class give a gate within 0.9 points of the 500 we use. On Camelyon17, 200 patches per class give it within 0.2, while below that it is no longer reliable. For the rare-disease case the reviewer raises, the gate does not need labels.
Gated TENT follows the same pattern (within 0.8 points of its 500-per-class gate from 25 per class up: 82 % on shuffled streams and 60 % on class order at 25, 50 and 200 per class). A few images suffice because every calibration image is seen under 26 appearance conditions, in 25 streams each. At 10 per class no gate can be fitted, because 100 images do not fill two batches of 64. We did not run 100 per class. The gates at 50 and 200 per class differ by at most 0.4 points. On CIFAR-100-C, 10 per class stays within 1.5 points (Table A26a). One fit goes the other way: on the CIFAR-100 model trained on 30,000 images, the 200-per-class gate gives up the gain on shuffled streams (31.7 % against 47.4 % native, source 28.3 %), where 25–100 per class keep it (Table A26b).
On Camelyon17 (K = 2), the pathology dataset on which we ran the size curve (on NCT we report the pseudo-label gate, within 0.5 points of the labeled one, App. F.8):
From 200 patches per class the gate matches the one we report. The 100-per-class row is higher because at this budget the pool is too small to fill two batches of the long-tail calibration streams, so the fit reduces to ΔH alone, which is the stronger gate for BN-Adapt when K = 2 (App. F.10). The full gate matters where ΔH alone fails, which is on the methods built for skewed batches. Under class-ordered streams on CIFAR-10-C, the rule ΔH ≤ 0 leaves RoTTA, UnMix-TNS and TRIBE 12–17 points below their calibrated gates (Table A22). Gated RoTTA on Camelyon17 confirms this as it drops to 71.9% on slide order, below the source model's 72.9%, when we fit it on only 100 patches per class (Fig. A5).
At 64 patches per class, the smallest budget at which a calibration stream can be formed at K = 2, the full gate can no longer be fitted and stays near native (65.0%). The ΔH-only gate from the same 128 patches reaches 82.7% on slide order, its best at any budget. We therefore recommend ΔH alone at K = 2 (App. F.10, Figure A5). At K = 1,000, one ImageNetV2 image per class (1,000 labeled images) gives a gate within 0.6 points of the one we report on every ImageNet-C stream type (App. F.10).
The rare-disease case. Here the labels are not required. When we calibrate on the frozen model's own pseudo-labels, our gate stays within 1.5 points of the labeled gate on Camelyon17 for every method but TRIBE (App. F.8). With no source data at all, the rule ΔH ≤ 0 lifts TENT from 56.0% to 76.3% on slide-ordered Camelyon17 (Table A22).