- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper analyses and benchmarks various methods of Test-Time Adaptation. The main idea of the paper, is to benchmark these methods under oracle access. Even in this unrealistic setting, they find that most datasets see no positive improvement from Test-Time Adaptation.
Critical strengths
- The resulting limitation of TTAs is very interesting.
- Ablations validate the findings under different conditions.
- Visualizing the results over different attributes in Fig 1 and Fig 2 increases trust into the paper.
Critical weaknesses
The biggest weakness I see is in the interpretation of the results. I don't see why having access to Oracle-level information wouldn't improve the results, and you don't provide any satisfactory reason for this.
My best explanation for this is a limitation of the datasets (if the dataset does not contain any shift, why should the oracle improve performance). This would substantially reduce the value of your paper: Instead of finding that TTA is often useless, the conclusion would be that TTA is only useful under a significant enough distribution shift. This seems obvious and is partially supported by Fig. 1: The situations with a very significant distribution shift (top right corner) are easily found by both oracle and practical algorithms and the interesting part of your paper only appears with less significant differences.
Could you please clarify if you have a better explanation?
Questions for authors
See my main Weakness. If you can refute my concern, I will recommend either a weak or a clear acceptance.
Could you maybe create a table/figure comparing the contamination rate of various datasets (during testing) against Oracle minus Practical performance? Depending on the result, this could validate my hypothesis or refute it directly.
I would also like to know why the authors rely on AUCPR instead of more modern TSAD methods like VUS-PR or Affiliation-F.
3 comments
Q2. Test Anomaly Prevalence vs. Oracle–Practical Performance
The requested comparison reveals substantial variation in Oracle–Practical detection gains across datasets with similar test anomaly rates. We compare the test anomaly ratios reported in Appendix Table A.1 with the paired Oracle-minus-Practical AUPRC differences across all nine benchmarks.
| Dataset | Test anomaly ratio (%) | Oracle − Practical AUPRC (pp) |
|---|---|---|
| Daphnet | 7.93 | −7.13 |
| GECCO | 1.41 | +15.07 |
| Genesis | 0.41 | +0.02 |
| MSL | 10.48 | +0.00 |
| PSM | 27.76 | +33.91 |
| PUMP | 9.40 | +40.39 |
| SMAP | 12.83 | +0.01 |
| SMD | 4.16 | +1.08 |
| SWAT | 12.14 | +8.93 |
For example, PUMP and SMAP have broadly comparable anomaly ratios (9.40% vs. 12.83%) but markedly different Oracle–Practical gaps (+40.39 vs. +0.01 pp). GECCO also shows a substantial Oracle advantage (+15.07 pp) despite its low anomaly ratio (1.41%). The dataset-level Spearman correlation is ρ = 0.117. Test anomaly prevalence does not directly measure normal-distribution shift.
Q3. Robustness Across Evaluation Metrics
The coexistence of oracle-only-gain (OOG) and neither-improves (NI) outcomes is not specific to AUPRC. We use unadjusted AUPRC as the primary metric to evaluate point-wise anomaly ranking without threshold selection or point adjustment. To assess metric robustness, we additionally evaluate VUS-PR and Affiliation-F on the same 180 configurations.
| Metric | BI | OOG | NI | POG |
|---|---|---|---|---|
| AUPRC (primary) | 27 | 63 | 90 | 0 |
| VUS-PR | 35 | 55 | 90 | 0 |
| Affiliation-F (P95) | 34 | 32 | 114 | 0 |
| Affiliation-F (P99) | 41 | 36 | 92 | 11 |
Under VUS-PR, all 90 original NI cases persist, with 172/180 (95.6%) outcome classifications unchanged. For Affiliation-F, P95 and P99 refer to Frozen source-validation thresholds applied identically to all three arms. Both OOG and NI remain substantial across the alternative metrics, although individual classifications depend on the metric and, for Affiliation-F, the threshold.
I appreciate the authors efforts, and my reservations are partially resolved. I have adjusted my score.
R3-W1/Q1. Why Can Oracle Normal-Only Adaptation Still Fail?
Oracle normal-only selection does not guarantee better detection because normal-sample updates can also make anomalies easier to reconstruct. The update objective reduces normal reconstruction error but does not constrain how anomalous scores change relative to normal scores. We demonstrate this mechanism in a controlled Linear-AE experiment and show a corresponding local effect in an actual nonlinear AE.
1. Controlled explanation under a shift that harms detection
We use a two-dimensional rank-one Linear-AE. Rotating test-time normals by 30°, while keeping the source model fixed, reduces Frozen AUPRC from 1.0000 to 0.9356, demonstrating that the shift harms detection.
Favorable and Harmful conditions share the same shifted normals, source model, normal update samples, and 128-step angular-SGD trajectory. Only anomaly geometry differs; their Frozen anomaly-score distributions are identical, and anomalies never enter updates. Across five seeds, normal update-set loss decreases from 0.2599 to 0.0103 in both conditions. Yet on independent held-out sets (512 normals and 512 anomalies per condition), adapted AUPRC reaches 1.0000 (Favorable) versus 0.3410 (Harmful). In the Harmful condition, mean anomaly error falls from 0.4939 to 0.0014, below the adapted normal mean (0.0104).
The contrast arises because updating the reconstruction subspace toward shifted normals can simultaneously make certain anomalies easier to reconstruct. Depending on anomaly geometry, successful normal fitting can therefore worsen normal–anomaly score separation, even when the distribution shift has already degraded Frozen detection.
2. Supporting evidence from a real AE update
We examined 20 predetermined checkpoints per trajectory across 60 Oracle trajectories. As an illustrative example, Daphnet window 726 (zero-indexed; seed 42, AdamW 1e−4) contains 12 benchmark-labeled normal points. Scoring the same fixed future normal and anomalous samples immediately before and after this update isolates its local effect.
Current normal-window loss decreases by 0.0045, while mean future anomaly scores decrease by 0.0093 and mean future normal scores increase by 0.00081. The mean anomaly-minus-normal gap narrows by 0.0101, and the correctly ordered pair fraction declines from 0.4651 to 0.4634. This illustrates an actual local objective–ranking conflict, not a complete explanation of Daphnet's online degradation.
3. Interpretation of Fig. 1 and remaining scope
Fig. 1 plots Oracle−Frozen and Practical−Frozen AUPRC changes, not distribution-shift severity; its upper-right region therefore cannot establish that larger shifts yield larger gains. Some of the 90 Neither-Improves (NI) settings may indeed have little exploitable adaptation demand, and our experiments do not explain every NI case. Nevertheless, they establish a concrete reason Oracle adaptation can fail beyond insufficient shift: normal-only selection does not ensure that parameter updates preserve anomaly ranking.