acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Test-Time Adaptation under Attack in Vision–Language Models

Abstract

Evaluating test-time adaptation under attack requires specifying the targeted predictor, the images on which errors occur, and the cost of adaptation. We compare six adaptation methods and zero-shot classification across three vision-language backbones and fifteen datasets, using common weights, starting prompts, and input images within each task. The 1,890 evaluations cover clean inputs and five cached attacks. Their per-image predictions reveal distinctions that aggregate accuracy alone misses. MAC has higher mean accuracy than R-TPT over four attacks (52.96% versus 50.86%), but a lower fraction correct under every attack (31.63% versus 37.41%); this ordering reverses within 15 of the 45 tasks. R-TPT also takes 6.33 times as long on matched attacked inputs in an 18-task local comparison. DBD leads the cached suite, yet its above-clean gains depend strongly on the attack. A separate 100-image DTD study queries all seven predictors directly. DBD's 75.56% cached accuracy falls to 34.38% under adaptive search, but a control choosing between clean and cached inputs already yields 38.94%. Thus most of that gap removes a cached-input benefit; the remaining drop under the full attack is smaller. A matched ten-image check at five times the query budget measures further sensitivity, with substantial uncertainty. These results link method selection to attack coverage, error overlap, clean-correct retention, and measured cost, while keeping shared-input performance distinct from the bounded adaptive case study.

Primary area:
datasets and benchmarks
Keywords:
Adversarial Robustness, Vision-Language Models, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_juxxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks six test-time adaptation/defense methods and a zero-shot baseline across three vision-language backbones, fifteen datasets, and five cached attacks. It analyzes average robustness, cross-attack error overlap, clean-correct retention, and computational cost. The results reveal ranking differences between average and joint robustness, while a limited adaptive study shows that DBD’s gains on cached attacks are substantially reduced under defense-aware attacks.

Critical strengths

1、Large-scale paired evaluation. • Methods are evaluated with shared models, prompts, and attacked inputs, enabling reliable per-image and cross-attack comparisons. 2、Comprehensive and transparent analysis. • The paper evaluates robustness, clean performance, error overlap, and computational cost while clearly acknowledging its experimental limitations.

Critical weaknesses

• Limited adaptive evaluation. Most results measure transfer attacks against the undefended backbone. The adaptive study covers only one backbone, one dataset, and one attack method. • Uncertain comparison stability. Several methods use modified configurations, while the main evaluation uses only one seed and includes quantized attacks that may exceed the stated perturbation bounds.

Questions for authors

• Do the main method rankings persist under adaptive attacks on additional datasets and backbones? • What are the specific differences between this paper and 'Adversaries Fight Back: Revisiting Test-Time Defenders of Vision-Language Models under Adaptive Attacks'? Could you at least include its CLIP-DDT or an equally strong adaptive attack as a comparison? • Are MeanRA and WCRA stable across multiple method and attack seeds? • Can the authors verify that all decoded adversarial examples satisfy the stated perturbation bounds?

9 comments
AuthorsOpenReview ↗

Response to Reviewer juxx (1/6): Adaptive Evaluation and Controls

Thank you for the detailed questions on adaptive evaluation, seeds, input bounds, and common settings.

The added studies do not support a single ranking across all tested adaptive settings. On CLIP-B/16 DTD, Combined accuracy is 28.63% for R-TPT and 27.88% for DBD: the 0.75-point R-TPT-minus-DBD gap has a 95% paired image-bootstrap CI of [-9.50,11.13], leaving their ordering unresolved. On OxfordPets, DBD exceeds R-TPT by 34.13 [21.00,47.50] points. On DataComp, R-TPT shows no demonstrated additional reduction from Initial to Combined. Thus the cached-suite ordering does not carry over uniformly to these full-predictor attacks.

We expanded the original single-dataset adaptive case study with a six-dataset study of all seven original predictors and an independently implemented SPSA/Square comparison on two model/pretraining configurations, CLIP-B/16 and DataComp ViT-B/16. The independent comparison queries zero-shot, R-TPT, MAC and DBD on DTD/OxfordPets: 50 images per dataset for CLIP-B/16 (400 image–method pairs, 800 searches), and twenty of the same ordered IDs for DataComp (160 pairs, 320 searches). Both use the full 47/37-class vocabularies without correctness filtering; the twenty-image cohort does not cover every class. Section 6 and Appendix N give both optimizers, paired uncertainty and the matched twenty-ID B16 subset.

SPSA-style finite differences and Square-style search each use 2000 complete defended-prediction calls, with EOT=4 and two restarts inside that cap. Probes, initialization and candidate evaluations are charged; EOT/restarts are not multiplied into the cap again. Combined selects the output with the lower saved search-bank margin: 4000 search calls and zero extra selection calls. Sixteen disjoint held-out method draws score the conditions, with sixteen calls per unique selected image; aliases reuse scores and actual counts are retained. PGD warm-start generation and separate oracle gates are outside these search counts. Both algorithms start from identical clean, cached PGD and one feasible random stripe input. Initial selects among this same three-input pool using search scores. Neither Initial nor Combined uses held-out correctness for selection. Every prediction resets adaptation and receives only the current input and common context; labels remain outside prediction. Selected inputs obey exact four-intensity-level bounds against retained canonical uint8 anchors. Independent output checks verified all 560 pairs, full class scores, RNG banks, shared initial inputs, selection choices and call accounting.

For B16 DBD on DTD, cached accuracy is 77.88%, Initial 37.63% and Combined 27.88%; Initial-minus-Combined is 9.75 [3.25,18.00] points. On Pets the reduction is 13.75 [5.50,23.63] points. Initialization accounts for much of the cached-to-Combined difference, with further loss beyond that control. MAC loses 12.25 [6.38,19.13] and 35.00 [25.50,44.88] points. These contrasts describe the complete search-and-selection procedure, not isolated causal optimizer effects.

DataComp R-TPT has 35.31/39.06% Combined accuracy on DTD/Pets versus 33.13/39.38% Initial. MAC falls from 26.25/69.06% Initial to 15.00/23.44% Combined; DBD falls from 38.75/74.06% to 30.00/69.06%. Unchanged or increased held-out accuracy does not establish search convergence. The 10000 paired image-bootstrap replicates, seed zero, preserve all draws, methods and conditions of an image. Intervals condition on recorded images, searches and RNG banks, not independent complete attack repetitions. Matched twenty-ID comparisons retain differences in processing, anchors, same-model PGD inputs and RNG realizations; they do not isolate architecture or pretraining causally.

Appendix I reports the separate six-dataset, seven-predictor B16 study (50 images/dataset, 4000 search calls), verified from 2100 existing records. DBD obtains 82.06% cached, 51.98% adaptive and 60.19% clean accuracy, with equal dataset weights. MAC-minus-R-TPT changes from 7.78 [3.86,11.67] points cached to -1.55 [-5.32,2.17] adaptive; the latter includes zero. Focus IDs overlap other studies, including the original DTD cohort, and are not pooled as independent image replications.

Revision locations: Section 6 (Table 3); Appendix I (Tables 19-20); Appendix N (Tables 36-38 and Figure 10).

AuthorsOpenReview ↗

Response to Reviewer juxx (2/6): Budget/EOT Diagnostic and CLIP-DDT

Adversaries Fight Back studies attacks on test-time CLIP defenses through CLIP-DDT. Our question is shared-input method selection across finite attack conditions: how error overlap, clean retention and matched cost affect the comparison. We have broadened adaptive coverage, but have not performed a direct CLIP-DDT comparison or established equivalent attack strength. The requested reference is https://openreview.net/forum?id=hZVSKjtaS5; the separately published CLIP-MAD study is cited independently and is not treated as the same implementation.

The added six-dataset CLIP-B/16 study and independent SPSA/Square comparison on CLIP-B/16 and DataComp query the complete adapted predictor. Independent optimizer families, reductions beyond the shared Initial control, disjoint held-out scoring and retained bounds provide broader empirical adaptive evidence. This is neither a new attack-method contribution nor a claim to outperform CLIP-DDT. DataComp R-TPT shows no demonstrated further reduction from Initial to Combined, making the finite-budget limitation explicit. Neither CLIP-DDT nor CLIP-MAD strength equivalence has been established.

Appendix Q reports the higher-budget/EOT diagnostic: five development images per dataset from the existing DataComp DTD/OxfordPets panel, R-TPT and DBD, and 8000 complete predictor calls per SPSA/Square optimizer at EOT=4 or EOT=16. Two restarts and EOT remain inside each cap, giving 16000 search calls for Combined. Sixteen fresh common held-out draws rescore both the historical 2000-call/EOT4 outputs and each new condition, so comparisons use the same scoring bank. Combined selects by search margins, not held-out correctness; EOT settings are reported separately, without a held-out or cross-EOT winner selection.

At EOT=4, DBD's matched baseline-to-new Combined accuracy changes from 100.00 to 52.50% on DTD and from 77.50 to 56.25% on OxfordPets: reductions of 47.50 and 21.25 percentage points. R-TPT changes from 80.00 to 80.00% and from 60.00 to 58.75%, respectively. At EOT=16, new accuracies are 100.00/77.50% for DBD and 80.00/60.00% for R-TPT, equal to their rescored baselines: no further reduction in these reported cells. Higher EOT leaves fewer proposals under a fixed call cap and is not automatically stronger.

All 40 image-method-configuration rows and 80 searches are complete and independently checked. Appendix Q gives the detailed scoring/gate call accounting and retained completed cost; historical searches are reused. These are descriptive, post-hoc development diagnostics on five images per dataset, not an independent confirmation cohort, evidence of convergence or an equivalent-strength test. The direct CLIP-DDT comparison remains open.

Revision locations: Section 2 (CLIP-DDT/CLIP-MAD); Section 6 and Appendix N (adaptive scope); Appendix Q (Table 44, budget/EOT diagnostic).

AuthorsOpenReview ↗

Response to Reviewer juxx (3/6): Attack-Seed Study

Reviewer juxx — seeds (J3)

The enlarged attack-repeat study reproduces the R-TPT-versus-MAC WCRA ordering on both confirmation cohorts at all five attack seeds. This is consistent with the separate full-manifest defense-repeat study, where R-TPT exceeds MAC in WCRA at all three defense seeds. The two studies vary different sources of randomness and are reported separately.

Appendix P reports five freshly generated offline attack suites on two fixed, class-balanced 200-image cohorts from DTD and OxfordPets, retaining the full 47/37-class vocabularies. The selection rule, IDs, exclusions and protocol were frozen before these confirmation runs. The cohorts exclude the previous focused, development and adaptive-confirmation IDs; selection did not use correctness or method outcomes. These are new focused cohorts, not test images unseen by earlier full-manifest benchmark evaluations.

CLIP-B/16, R-TPT/MAC/DBD, starting prompts, FP32, batch size one, input order and method settings remain fixed. Each suite contains PGD10, APGD-CE30, APGD-T30 and Square5000 at L∞L_{\infty} 4/2554/255. Attack seeds s=0,1,2,3,4s=0,1,2,3,4 vary while an independent per-image defender bank r0 remains fixed; its key excludes the attack and s. All 40 fresh caches, 120 attacked evaluations and six matched clean controls are complete: 25200 predictions. Original-ID, input-byte, label, full-score, model/source and decoded-bound checks pass. Exact runtime and replay checks are enforced within each dataset panel.

The table gives mean ±\pm sample SD across the five attack suites, in percent; SDs are percentage points, not confidence intervals.

DatasetMethodMeanRAMinRAWCRA
DTDR-TPT35.50±0.5435.50 \pm 0.5429.80±0.5729.80 \pm 0.5727.90±0.6527.90 \pm 0.65
DTDMAC32.93±0.7432.93 \pm 0.7424.00±1.1724.00 \pm 1.1716.60±2.1016.60 \pm 2.10
DTDDBD52.88±0.4252.88 \pm 0.4242.40±0.6542.40 \pm 0.6539.80±0.7639.80 \pm 0.76
OxfordPetsR-TPT59.33±0.8259.33 \pm 0.8241.60±1.0841.60 \pm 1.0837.20±2.1737.20 \pm 2.17
OxfordPetsMAC63.80±1.0663.80 \pm 1.0639.10±4.5239.10 \pm 4.5227.30±3.6227.30 \pm 3.62
OxfordPetsDBD90.18±0.3890.18 \pm 0.3885.20±0.7685.20 \pm 0.7682.20±1.1582.20 \pm 1.15

R-TPT exceeds MAC in WCRA in each of the five suites on each dataset. The suite-mean paired R-TPT-minus-MAC differences are +11.30 percentage points on DTD and +9.90 points on OxfordPets. Their conditional 95% paired image-bootstrap intervals are [6.50,16.20] and [4.10,15.70] points, respectively. The 10000 replicates resample original image IDs jointly across methods, attacks, all five suites and clean controls. These intervals condition on the selected cohorts and recorded suites/defender banks; they do not measure population seed variance, imply significance at every individual seed, or include a multiple-comparison correction.

The stable comparison is metric-specific. MAC exceeds R-TPT in OxfordPets MeanRA at all five attack seeds, reproducing the distinction between average and per-image intersection-based selection. DBD leads MeanRA and WCRA on both cohorts at every recorded attack seed. OxfordPets MinRA has a narrower exception: MAC exceeds R-TPT at s3 (46.50 versus 40.50%), while the other four suites favor R-TPT. We therefore do not assert universal ranking stability across metrics. WCRA intersects the four attacks within each s; candidates are never pooled across suite seeds.

Revision locations: Appendix P (Tables 41-43 and Figure 11); Appendix M (Table 28, separate defense-seed study).

Reviewer_jqJZOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper benchmarks six test-time adaptation methods and zero-shot classification across three vision–language backbones and fifteen datasets. It compares average attacked accuracy, error overlap across attacks, preservation of correct clean predictions, and inference cost.

The main finding is useful: MAC has higher average attacked accuracy than R-TPT, but R-TPT correctly classifies more images under all four cached attacks. The paper also includes a full-method adaptive attack study and controls that help explain DBD’s large cached-to-adaptive accuracy gap.

I believe that the methods chosen by the authors are representative within the field, including methods in input correction (DBD ICLR-2026), in prompt tuning (R-TPT CVPR-2025), and in lighweight counterattacks (TTC CVPR-2025). And the experiments conducted are valuable for future researchers.

However, my main concern is whether the benchmark adequately represents the current field enough. It includes recent methods in input correction (DBD ICLR-2026; MAC CVPR-2026), but misses important 2026 methods in prompt tuning (such as A-TPT ICML-2026) and lightweight counterattacks (such as DOC AAAI-2026-Oral). This limits how confidently its findings can guide current method selection in ICLR 2027.

I will give a 3 that I will update to a 4. I hope to update it after seeing a good improvement.

Critical strengths

1. The per-image analysis adds useful information.

MAC outperforms R-TPT in MeanRA, 52.96% versus 50.86%, but falls behind in WCRA, 31.63% versus 37.41% (Table 2, p. 4). Figures 2–3 explain this difference through error overlap. The detailed tables support the reported 15 task-level ranking reversals.

2. The attack settings are clearly distinguished.

The paper separates cached attacks against the frozen backbone from attacks that query the complete adaptation method (Eq. 1, p. 3). This helps avoid overstating what the main benchmark establishes.

3. The controls make DBD’s results easier to interpret.

DBD achieves 75.56% on cached inputs, 40.00% on clean inputs, and 34.38% under adaptive search. The clean/cached selection control already reduces accuracy to 38.94% (p. 9). This shows why the large cached-to-adaptive gap should not be attributed entirely to additional search.

4. The benchmark includes recent correction-based defenses.

The comparison covers recent multi-view input-correction approaches MAC (2026) and DBD (2026). This makes the benchmark’s coverage of these approaches more up to date.

Critical weaknesses

1. The prompt-tuning comparison misses recent methods.

Prompt-tuning methods are represented only by pre-2026 approaches. Appendix B states that A-TPT (ICML 2026) is “outside the common grid” (p. 14), but does not adequately justify its exclusion. This matters because the paper compares adaptation mechanisms and robustness–cost trade-offs. In particular, A-TPT (ICML 2026) combines additional attention computation for multi-view augmentation, making it directly relevant to this comparison. A-TPT performs test-time prompt tuning using the current test image and a standard pretrained CLIP model. It therefore appears to meet the benchmark’s stated information constraints and should be included in robustness–cost trade-offs.

2. The advance over the closest existing benchmark is unclear.

TTA-VLM already studies unified VLM adaptation, fifteen datasets, multiple backbones, and transferred adversarial attacks. Could the authors explain more directly what new knowledge its error-overlap analysis and controls provide beyond that benchmark.

3. Cache issues reduce reproducibility.

Some historical attack settings are incomplete, and decoded ℓ2ℓ_{2} examples can exceed the nominal radius (p.15). Shared-input comparisons remain meaningful, but strict robustness claims and independent reconstruction require these issues to be resolved.

Questions for authors

See critical weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jqJZ (1/3): A-TPT and DOC

Thank you for the suggestions on method coverage, related work, and cache provenance.

Q1. A-TPT and vanilla DOC are now evaluated, with matched processing cost, in a nine-predictor comparison on six full CLIP-B/16 test manifests: DTD, OxfordPets, Caltech101, Flowers102, FGVCAircraft and UCF101. The comparison uses Clean and four fixed cached L∞L_{\infty} attacks. Within each dataset, all predictors share weights, configured starting prompts, class order, precision and input bytes. Appendix M reports Clean/MeanRA/MinRA/WCRA/WCRA+C, individual attacks and dataset-specific costs.

The additions affect method selection: A-TPT has higher observed WCRA than R-TPT on five of six datasets, with DTD the exception; DBD has the highest cached WCRA among the nine configurations on all six. On Pets, A-TPT's 43.31% WCRA exceeds R-TPT's 38.02%, at 705.44 versus 306.44 ms mean attacked processing time. On DTD, A-TPT's 25.00% WCRA is slightly below R-TPT's 25.41%. A-TPT obtains 32.71/63.03% MeanRA on DTD/OxfordPets; DOC obtains 21.84/40.94% MeanRA and 3.66/3.87% WCRA.

On the four added full manifests, A-TPT/R-TPT WCRA is 71.97/68.68% on Caltech101, 39.83/33.29% on Flowers102, 8.91/7.74% on FGVCAircraft and 35.47/32.43% on UCF101. The corresponding A-TPT-minus-R-TPT gaps are +3.29 [2.23,4.34], +6.54 [5.40,7.71], +1.17 [0.54,1.80] and +3.04 [2.25,3.86] percentage points (95% paired image-bootstrap intervals). With equal weights across all six datasets, its MeanRA/WCRA gains are +1.15 [0.86,1.44]/+3.15 [2.76,3.54] points, with a clean change of -1.32 [-1.68,-0.98] points. These are robustness–cost and clean–attacked tradeoffs within the included configurations, not a universal ranking.

DOC uses the released vanilla branch without optional adversarially fine-tuned weights. Its Pets clean accuracy is 63.59% versus zero-shot's 87.63%: a -24.04-point difference, 95% image-bootstrap CI [-25.59,-22.51]. It corrects 77 zero-shot errors and introduces 959 errors on zero-shot-correct images. Two retained development-image reference checks match its correction exactly and final scores within 7.63×10−67.63\times10^{-6} through the shared encoder interface, with unchanged weights. They identify no discrepancy on those inputs but do not explain the full-dataset loss or prove full native-runner equivalence. We report the loss without tuning test results. A-TPT computes attention from the current image and shared pretrained model; attention alone is not a reason to exclude it. Both methods are cited in the revised related work. Reference-function checks and a separate bounded 32-prediction pretrained GPU check support the mechanisms, prompt/input pairing, resets and frozen encoders; they are not the source of the full-manifest performance estimates or proof of full native-runner outcome equivalence.

The retained DTD/OxfordPets manifests contain 1692/3669 images and 90 seed-zero runs; the four added full manifests contribute 180 evaluations. The complete six-dataset comparison has 270 seed-zero evaluations and 783225 predictions. The three-defense-seed study remains restricted to R-TPT/MAC/DBD on DTD/OxfordPets, reusing 30 seed-zero controls and adding 60 evaluations. Appendix M therefore contains 330 distinct evaluations and 944055 predictions. The four added datasets are not three-defense-seed studies; this six-dataset CLIP-B/16 ranking does not extend to the original 45 tasks.

The 10000 paired image-bootstrap resamples (seed zero) pair original image IDs across methods and conditions, stratifying within fixed datasets for the macro; datasets are not resampled. Intervals condition on recorded inputs, attack seed and defense seed, without multiplicity correction; they do not estimate between-seed or adaptive uncertainty.

Timing includes preprocessing, transfers, adaptation, prediction and CPU score copying, with CUDA synchronization before and after each prediction. It excludes model setup, encoded-image decoding, storage, metric computation and two warmup batches; peak allocated memory includes warmup. All costs use the same RTX 5090, FP32, batch one and four Torch CPU threads. The original DTD/Pets examples and Figure 9 retain that two-dataset scope; Appendix M reports the added dataset-specific costs separately.

Revision locations: Section 2; Appendix M (Tables 25-35 and Figure 9).

AuthorsOpenReview ↗

Response to Reviewer jqJZ (2/3): Relation to TTA-VLM

Q2. The added knowledge concerns which robustness–cost decisions a single attacked-accuracy average leaves unresolved. Our paired analysis shows that mean attacked accuracy and the per-image intersection can favor different configurations, identifies the shared and exclusive errors that produce the difference, and connects that decision to matched-cohort processing cost and clean-correct retention. The clean/cached candidate-selection control also separates the interpretation of initialization from the subsequent adaptive search, rather than attributing the entire cached-to-adaptive gap to that search.

Section 2 explicitly acknowledges TTA-VLM as the closest benchmark. It already standardizes adaptation across 15 datasets and multiple model families, evaluates eight episodic and seven online methods, and considers robustness, calibration, OOD detection and stability as well as accuracy and runtime/memory. Its adversarial evaluation is substantive: Section 5 uses transferred PGD generated against CLIP, with 100 steps and radius 4/2554/255 for CLIP-ViT, and seven steps and 1/255 for CLIP-ResNet. Table 9 reports episodic robustness on eleven datasets with CLIP-ViT-B/16. We therefore do not characterize TTA-VLM as a clean-only benchmark or claim fifteen datasets, multiple backbones, common settings, transferred attacks or efficiency measurement as our novelty.

Our focus is a more detailed adversarial method-selection question across multiple recorded attacks: how intersection accuracy, error overlap, exclusive failures, clean retention and measured cost change a paired comparison, followed by full-predictor adaptive studies and explicit starting-candidate controls. This empirical emphasis complements TTA-VLM's broader TTA/trustworthiness evaluation. WCRA and defense-aware evaluation themselves are not claimed as new inventions.

Revision locations: Section 2 (relation to TTA-VLM); Sections 3-6 and Appendices L/N (method selection and adaptive controls).

AuthorsOpenReview ↗

Response to Reviewer jqJZ (3/3): Cache Provenance and Bounds

Q3. The decoded audit is complete for all 225 primary caches and 3,481,275 unique image-condition records, including the eight-cache pilot. For the 180 L∞L_{\infty} caches (2,785,020 images), zero decoded examples exceed 4/255+10−64/255+10^{-6} against the reconstructed clean references. The main MeanRA and WCRA use only the four L∞L_{\infty} attacks; the separate nominal-L2L_2 column does not enter those metrics. Appendix C distinguishes this completed numerical audit from the remaining historical-provenance limitations.

The audit checks encoded-image hashes, sample IDs, labels, geometry and complete per-cache coverage. All records were readable and verified for these reconstruction-relative checks. Norms use float64 differences from explicitly reconstructed float32 processed clean references; tolerances were fixed before counting exceedances: 1×10−61\times10^{-6} for L∞L_{\infty} and 1×10−51\times10^{-5} for L2L_2. As specified in the original protocol, the attack domain is the spatially processed 224×224224\times224 tensor in [0,1]. L∞L_{\infty} norms range from 0 to 0.0156. Strict raw L∞L_{\infty} exceedances remain explicit: 2,124,808 inputs are above 4/2554/255, with maximum excess 1.168718525×10−71.168718525\times10^{-7}, approximately 0.000745% of the nominal radius and below the fixed tolerance. This is a reconstruction-relative tolerance check, not proof of identity with the unretained historical continuous clean anchors or strict zero-tolerance compliance.

For all 45 nominal-L2L_2 caches (696,255 images), 593,335 examples (85.2181%) exceed 1.0+10−51.0+10^{-5}. Norms range from 0.5292 to 1.1124; the maximum excess is 0.1124, or 11.2489% of radius one. This is not a negligible floating-point tolerance. The L2L_2 column remains a separate nominal-pre-encoding-radius, quantized-input comparison and is not evidence of strict decoded robustness at radius 1.0. These failures are disclosed rather than repaired: no input was projected, re-encoded or regenerated. The same retained input bytes keep the paired comparison well-defined on those inputs, and this audit changes no recorded predictions or scores. We have not measured whether projection or regeneration at strict decoded radius one would preserve their accuracies or rankings; no such invariance or statistical non-effect is claimed. Nor do the reconstructed references establish the historical cause of the radius discrepancy. The full audit replaces the earlier 16-image diagnostic as the suite-frequency measurement; it does not restore historical generator versions or anchors.

Revision locations: Section 3; Appendix C (Table 9 and the provenance discussion).

Reviewer_jqJZOpenReview ↗

Final Comment

This work makes a comprehensive evaluation of the existing test-time adversarial defense methods.

  1. This work is carried out with extensive attack settings and computational efficiency, which has exceeded the workload in the original reports of most defense methods. I am very sure that these assessments provide a very valuable reference for follow-up research in this field.

  2. The defense methods of the evaluation are very comprehensive. Test-time adversarial defense methods are mainly divided into three categories: multi-view-based geometric correction and prompt tuning, and single-view purification. This work covers the latest work in these three directions.

Based on the two points, I strongly recommend that ICLR2027 accept this work. I will update the score to 4. It’s a good work.

Reviewer_MgVyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper systematically evaluates seven test-time adaptation (TTA) methods for vision–language models across different backbones, datasets, and adversarial attacks, comparing their robustness and computational costs. By analyzing mean attacked accuracy, per-image worst-case accuracy, error overlap, and clean-correct retention, the study reveals that different evaluation metrics can lead to different method rankings. Furthermore, adaptive attacks targeting the full adapted predictors are used to investigate performance differences between cached-attack and adaptive-attack evaluations.

Critical strengths

The paper provides an extensive experimental evaluation across multiple methods, backbones, datasets, and attack settings, supported by relatively thorough analyses that offer useful insights into the differences in robustness and computational costs among methods.

Critical weaknesses

The main weakness is the lack of clear, actionable guidance on selecting appropriate evaluation metrics under different attack settings. Although the paper demonstrates that different metrics can lead to conflicting method rankings, it does not establish when or why a particular metric should be preferred, limiting the practical value of its findings for future research.

Questions for authors

When different evaluation metrics lead to conflicting method rankings, as observed for MAC and R-TPT, what criteria should researchers use to determine which metric is more appropriate for a given evaluation scenario? Does the analysis provide any actionable guidance beyond recommending that multiple metrics be reported together?

4 comments
AuthorsOpenReview ↗

Metric Selection

Thank you for asking for clearer guidance on choosing a primary metric.

Our actionable recommendation is to specify the input distribution, the attacker's permitted selection rule and the resource constraints before comparing methods. These assumptions determine the primary metric; the metric is not selected after observing which method wins. Section 3 and Appendix L now make the decision rule explicit:

  1. For an image-independent mixture of attack algorithms with specified probabilities, use weighted attacked accuracy. MeanRA is its equal-weight special case.
  2. If the attacker chooses one algorithm for an entire task, use MinRA, the accuracy of the worst algorithm in that task.
  3. If the attacker can choose a recorded candidate separately for each image, use WCRA, the intersection of correctness across those candidates.
  4. If the unperturbed input is also an available choice, use WCRA+C to include clean correctness in that intersection.
  5. To ask how many of a method's clean-correct images survive the suite, use retention alongside clean accuracy; its conditioning set differs between methods.

The mixture probabilities are stated deployment or evaluation assumptions, specified independently of which method they favor. The rule does not estimate them from the reported rankings. When those probabilities or the attacker's selection ability are unspecified, the corresponding method choice remains conditional on an explicitly stated scenario. Processing cost should be assessed under that same scenario. We then compare accuracy among configurations satisfying the stated time/memory constraints; a constraint on mean time uses a different measurement from a percentile or maximum-latency constraint.

The matched 18-task example demonstrates the resulting decision. MAC/R-TPT obtain 55.63/54.88% MeanRA and 34.39/43.15% WCRA at 53.45/338.32 ms mean attacked processing time. Uniform-mixture accuracy favors MAC for this pair, whereas per-image candidate choice favors R-TPT, at 6.33 times the mean processing time. This example changes the choice between these two configurations according to the stated objective and available compute. It does not establish significance of their MeanRA difference or make either the overall winner: DBD remains the cached-suite leader in the complete comparison. Mean latency is not a per-image or percentile latency bound.

The resulting guidance goes beyond reporting several metrics together: choose the evaluation question first, use its corresponding objective and apply the resource constraint before interpreting the ranking. The cached intersection remains a finite-suite, recorded-randomness summary. It is not a certificate over the perturbation ball or a substitute for evaluating an attack that queries the complete adapted predictor; adaptive accuracy answers that separate question.

Revision locations: Sections 3 and 5; Appendix L (Table 24) and Appendix J (Table 21).

Thank you for the clarification. The response partially addresses my concern by providing more explicit guidance on metric selection. However, these rules seem to primarily explain the meanings of existing metrics and their correspondence to predefined threat models. Their applicability also depends on prior knowledge of the attacker's strategy or attack distribution. It remains unclear whether the extensive empirical analysis itself provides new, actionable insights for future evaluation practices beyond these definitions. Therefore, my concern about the paper's broader methodological contribution remains.

Thank you for clarifying the remaining concern. The empirical analysis supports concrete evaluation decisions beyond matching metrics to threat models. In particular, it identifies conclusions that do not require an estimate of attack probabilities, and failures that even the hardest individual attack misses.

1. Selecting the hardest individual attack still misses a consistent difference in failure overlap.

Across the primary 45-task grid, MAC has higher task-averaged MinRA than R-TPT (41.17% vs. 40.14%), but lower WCRA (31.63% vs. 37.41%). The MinRA-to-WCRA gaps, computed before rounding, are 9.54 and 2.72 percentage points. MAC's gap is larger in all 45 tasks and all ten additional attack-seed suites: five seeds on each of two 200-image cohorts, with the defender seed fixed. Thus, this difference is not explained solely by averaging over easier attacks; the attacks defeat different image sets to different extents for these defenses.

Attack coverage is also defense-dependent: PGD-10 attains minimum accuracy for R-TPT in 45/45 tasks, Square for DBD in 45/45, and APGD-T for MAC in 42/45.

Evaluation consequence: retain aligned per-image outcomes and check attack coverage on each defended predictor. Even inspecting every attack column and then retaining only the hardest attack misses these additional failures. The 45/45 result concerns the gap, not a ranking reversal in every task. (Section 4; Appendix D, Table 12; Appendix P, Table 43.)

2. Unknown attack probabilities do not prevent every useful method comparison.

DBD has higher observed accuracy than MAC in all 180 task-attack cells of the primary four-attack cached suite. Within each evaluated task, any common image-independent weighting of those four attacks therefore preserves that accuracy ordering. Processing cost remains a separate consideration.

The matched six-dataset CLIP-B/16 extension at defense seed zero provides an accuracy-cost example: DBD has higher observed accuracy and lower measured mean prediction time than A-TPT in all 30 dataset-condition cells, including clean inputs and four cached attacks. Any common image-independent mixture over these measured conditions preserves the observed accuracy-mean-latency dominance.

Evaluation consequence: distinguish componentwise dominance from genuine trade-offs before requiring attack-frequency estimates. This can exclude a dominated configuration for the measured accuracy-mean-latency objectives within this fixed suite; it does not identify a generally inferior method. The comparisons concern observed condition-level results, not imagewise dominance or simultaneous population-level significance. (Appendices D and M; condition-level records in the analysis supplement.)

3. A large cached advantage can largely disappear under selection alone, before further iterative search.

On the DTD cohorts, a control selecting among the shared clean, cached-PGD and feasible stripe inputs reduces DBD's point-estimate advantage over R-TPT from 42.50 to 3.25 points on CLIP-B/16 (50 images), and from 55.63 to 5.63 on DataComp (20 matching images). These are approximately 92% and 90% reductions in the respective observed inter-method advantages, without additional iterative search. DBD retains substantial observed advantages on OxfordPets, so the finding is domain-specific.

Evaluation consequence: report cached, selection-only and full-search performance separately, using the same starting-candidate pool and held-out scoring independent of candidate selection. This reveals how much of an apparent advantage survives candidate selection before measuring the additional reduction under full search, rather than attributing the entire cached-to-final gap to optimization. (Section 6, Table 3; Appendix N.)

These findings specify what future evaluations should do differently: preserve failure-set information, test for componentwise dominance before assuming a distribution-dependent trade-off, and include candidate-selection controls when interpreting adaptive results. They require no estimate of deployment attack frequencies within their stated scope. Choosing among non-dominated configurations for deployment remains conditional on the intended scenario; our claims remain limited to the evaluated configurations, recorded inputs and search budgets.

AuthorsOpenReview ↗

Follow-up on our additional clarification

Dear Reviewer MgVy,

Thank you again for your time and thoughtful feedback. We would like to briefly follow up on our latest response regarding the paper’s empirical contribution and practical takeaways for future evaluations.

We would appreciate your thoughts on these clarifications and would be happy to address any remaining questions or concerns during the discussion.

Best regards, The authors of Submission #43229

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.