acceptodds
Withdrawn from ICLR 2027

Failure-Aware Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adaptation (TTA) mitigates distribution shifts by adapting pre-trained models to unlabeled target domains during deployment. While existing methods successfully boost target-domain accuracy via entropy minimization or self-training, they optimize for predictive performance while largely ignoring the reliability of the model's confidence estimates. Consequently, under severe domain shifts, these adaptation mechanisms break down the capacity for failure detection, causing models to become highly overconfident in their misclassifications. To overcome this limitation, we introduce Masked Augmentation for Reliable Self-adaptation (MARS), a plug-and-play framework for failure-aware TTA designed to simultaneously maximize adaptation accuracy and failure detection reliability. MARS leverages masked augmentations of input samples to serve two complementary objectives: (1) generating masking-ratio-conditioned soft labels to calibrate confidence rankings, and (2) measuring prediction consistency between original and masked samples to yield a robust uncertainty signal. Crucially, this dual-mechanism integrates seamlessly with existing TTA baselines without architectural modifications, supported by rigorous theoretical analysis. Across standard, continual, and mixed-domain setups on diverse benchmarks, MARS substantially improves failure detection metrics while matching or exceeding the target-domain accuracy of state-of-the-art methods. Our code will be made publicly available.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation, Misclassification Detection, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BR7jOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Failure-Aware Test-Time Adaptation introduces MARS, a plug-and-play framework that aims to improve both target-domain accuracy and the reliability of failure detection during adaptation. MARS generates masked views of each input and applies a masking-conditioned ranking loss that encourages confidence to fall as progressively more image content is removed. It further measures prediction consistency across these views and fuses this signal with predictive entropy into a reliability score. The authors evaluate MARS on corruption benchmarks and in standard, continual, and mixed-domain adaptation settings, reporting improved failure-detection metrics while generally maintaining or improving classification accuracy over several TTA baselines.

Critical strengths
  1. The paper addresses an important gap in test-time adaptation: detecting when adaptation makes a prediction unreliable. MARS targets both target-domain accuracy and failure detection explicitly, rather than treating accuracy gains as sufficient evidence of successful adaptation. This matters for practical deployment, where a confidently wrong adapted model can be more harmful than one that recognizes its own uncertainty and defers.

  2. MARS is a simple, broadly evaluated addition to existing adaptation methods, with evidence that reliability gains need not come at the cost of accuracy. Its masked-view ranking loss and consistency score can be added to multiple TTA baselines without altering their architectures. Across corruption benchmarks and in standard, continual, and mixed-domain settings, the paper reports better failure-detection metrics while generally maintaining or improving classification accuracy. For instance, on standard CIFAR-10-C with EATA, adding MARS raises AUROC from 84.79 to 86.03 and lowers FPR@95 from 67.00 to 63.13, while accuracy improves from 82.06 to 82.55. This breadth, together with the matched comparisons, supports the method's practical relevance.

Critical weaknesses
  1. The proposed reliability score has a potential blind spot for confident but unstable predictions. MARS defines its score as r(x)=−H(p)(1−c(x))r(x)=-H(p)(1-c(x)), with higher values indicating greater reliability. Because the product approaches zero—and therefore reads as highly reliable—whenever either entropy is low or masked-view inconsistency is low, a highly confident prediction can receive a strong reliability score even when its masked predictions are unstable. The paper reports that such cases are rare in its evaluation, but this does not establish how the score behaves under the severe or unfamiliar shifts that motivate the method. This matters because failures of exactly this kind are what the method is meant to catch; missing them would undercut the paper's central claim.

  2. The added computation and memory cost may limit practical deployment. In the paper's ablation on 150,000 CIFAR-10-C images, EATA requires 75.4 seconds and 2,268 MB, whereas EATA+MARS with the default K=8K=8 requires 791.6 seconds and 21,624.7 MB—roughly 10.5× the runtime and 9.5× the memory. Even at K=2K=2, EATA+MARS requires 264.5 seconds and 10,362.9 MB. These costs weigh heavily against a method presented as plug-and-play test-time adaptation, especially where inference latency or hardware capacity is constrained.

Questions for authors
  1. Clarification — Reliability-score behavior. The reliability score is r(x)=−H(p)(1−c(x))r(x)=-H(p)(1-c(x)), with higher scores indicating greater reliability. Since the score approaches zero when entropy is low—even if masked-view consistency is poor—how does MARS avoid rating confident-but-unstable predictions as reliable? Could you report failure-detection performance on that subset specifically?

  2. Clarification — Scope of the theoretical guarantee. Proposition 2 establishes the entropy-ordering result at the global minimizer of Lrank\mathcal{L}_{\text{rank}}, but the implemented method optimizes a joint objective that includes the base TTA loss. The paper also reports substantial residual monotonicity violations. Which guarantees, if any, hold for the jointly optimized method, and how do the remaining violations affect failure detection?

  3. Suggestion for future work — Online deployment. A future evaluation could report per-batch latency and peak memory in a realistic streaming setting, and test whether a smaller or adaptive number of masked views retains the reliability gains at a lower cost. This would help establish the method's feasibility for latency- or memory-constrained deployment.

  4. Clarification — Novelty relative to REM. REM already uses masked views at different difficulty levels and combines entropy ranking with masked-view consistency. MARS likewise generates masked views at varying ratios, applies a masking-conditioned ranking loss, and derives a consistency signal from predictions across views. Given this overlap in the core masking-based design, what does MARS contribute methodologically beyond using cross-view consistency as a failure-detection score? Please clarify how its independently sampled block masks and ranking objective differ materially from REM's mask chaining and its ranking/consistency objectives, and provide a controlled comparison or ablation that isolates the source of MARS's gains. This distinction is central to assessing the paper's novelty.

Reviewer_rPxDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies failure detection in TTA. Existing TTA methods often improve target-domain accuracy by minimizing prediction entropy or using pseudo-labels, but these objectives can also make incorrect predictions increasingly confident. As a result, confidence scores become less useful for identifying prediction failures after adaptation. The authors propose a plug-and-play approach (MARS) that uses masked versions of test inputs to address this issue. It introduces two complementary mechanisms: a confidence-ranking loss that assigns less confident soft targets to inputs with higher masking ratios, and a consistency-based score that measures how much predictions change between the original and masked inputs. The latter is combined with a conventional confidence measure to estimate prediction reliability. MARS is evaluated with several existing TTA methods under standard, continual, and mixed-domain adaptation settings. Experiments on corruption benchmarks and an extension to vision-language models show consistent improvements in failure detection metrics, while classification accuracy is generally maintained or modestly improved.

Critical strengths
  1. The proposed method is simple and can be integrated into existing TTA approaches.

  2. The ablation studies are thorough and can provide useful evidence for the design choices.

Critical weaknesses
  1. The ranking targets rely on a rather strong assumption about how masking affects the input. Equation (3) constructs the target distribution entirely from the masking ratio and the pseudo-label predicted on the original image. It therefore assumes that the expected confidence of a masked image can be specified by a fixed function of the fraction of removed patches, regardless of which patches are removed or how much task-relevant information remains. This can be problematic for images in which the discriminative information is concentrated in a small region: masking a few critical patches may destroy the evidence for the predicted class, whereas masking a much larger background region may leave the prediction essentially unchanged. The prescribed targets do not account for this difference. Consequently, the ranking loss may impose an uncertainty level that does not reflect the actual information content of a particular masked view.

  2. The theoretical analysis does not fully support the paper's central claim. Proposition 1 establishes a relationship between masking ratio and the true posterior entropy under assumptions about the spatial distribution of label-relevant information. Proposition 2, however, only shows that the entropy of the prescribed soft target increases with the masking ratio at the global minimum of the ranking loss. This does not establish that the adapted model will exhibit the desired calibration behavior when the ranking loss is jointly optimized with the original TTA objective. The empirical results also leave a substantial gap: Table 11 reports monotonicity violation rates of 50.34–54.00% for EATA+MARS, meaning that the intended confidence ordering still fails for roughly half of the evaluated samples. Although the authors acknowledge this limitation in the appendix, the theoretical results provide more of a motivation for the objective than a convincing explanation of its effectiveness.

  3. The proposed consistency score is not a clean measure of prediction stability. Equation (6) defines consistency as the dot product between the original and masked prediction distributions. Unlike a normalized similarity measure, this quantity also depends on how concentrated the distributions are: even identical, diffuse distributions produce a low dot product. Consequently, the score conflates disagreement between views with predictive uncertainty. This issue is compounded by Equation (7), which multiplies (1-c(x)) by negative entropy, making the two signals less independent than the paper's explanation suggests. The authors discuss the resulting edge cases in Appendix, but the reported frequency analysis does not resolve the underlying interpretation problem. Table 8 also shows that the consistency score alone performs substantially worse than the EATA baseline, while the full method works much better. This raises the question of how much the consistency component contributes as an independent failure signal, as opposed to helping only through its particular combination with entropy.

  4. The computational cost is substantial relative to the gains, and the improvements are not uniformly large. MARS requires multiple additional masked forward passes. In Table 6, evaluating 150,000 corrupted images with EATA takes 75.4 seconds and 2.27 GB of memory, whereas EATA+MARS with eight masked views takes 791.6 seconds and 21.6 GB. Even with only two masked views, the runtime increases to 264.5 seconds and memory usage to 10.4 GB. This is a considerable overhead for a method motivated partly by reliable deployment-time decisions. The benefits also vary across settings. For example, in the mixed-domain CIFAR-10-C experiment, adding MARS to EATA slightly worsens AURC and reduces accuracy from 68.20% to 67.15%, despite improving AUROC and FPR@95.

Questions for authors

See weakness

Reviewer_9AYiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether models can identify their own mistakes during test-time adaptation. It proposes MARS, which uses masked images and soft pseudo-labels to discourage overconfidence as more of the input is hidden. It also combines prediction entropy with agreement between original and masked views to score reliability. Experiments across several datasets and adaptation settings report improved failure detection while largely maintaining classification accuracy.

Critical strengths
  1. Experiments cover multiple datasets, adaptation settings, and baselines, showing broadly consistent improvements in failure detection.
Critical weaknesses
  1. EATA-C (TPAMI 2025) already studies overconfidence in TTA and proposes methods to improve confidence calibration. The paper neither compares against it nor explains the main differences, making it difficult to assess what MARS adds to existing work.

  2. The theoretical assumptions do not guarantee that more masking strictly removes more class information. For example, several patches may contain redundant information, so removing some of them may leave the class fully identifiable. Proposition 2 shows how the handcrafted soft targets behave, but does not establish that the model becomes better at detecting its own mistakes.

  3. The reliability score does not enforce the stated requirement that predictions must be both confident and stable. With r(x)=−H(p)(1−c(x))r(x)=-H(p)(1-c(x)), a highly confident prediction receives a score close to maximum reliability even when it is unstable under masking. The appendix acknowledges this issue, but reporting that such cases are rare in one setting does not establish how much they matter elsewhere.

  4. The additional computational cost is substantial. In Table 6, the default configuration takes about 10.5 times as long as EATA and uses about 9.5 times the reported memory. Since TTA runs during deployment, comparisons under the same time or computational budget are needed to assess whether MARS remains a better choice.

  5. The ablations do not fully explain where the gains come from. In Table 7, using only the new score and using the full method yield nearly identical FPR@95 values of 63.14 and 63.13. Although the full method also improves accuracy and AURC, a comparison using the same masked views with fixed label smoothing would help isolate the benefit of conditioning confidence on the masking ratio. Variability across runs should also be reported, particularly for the smaller gains.

  6. Releasing the implementation would improve reproducibility and make the method easier to evaluate and adopt. While not essential for acceptance, it would strengthen the overall impression of the work.

Questions for authors

See the above weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.