- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper benchmarks six test-time adaptation/defense methods and a zero-shot baseline across three vision-language backbones, fifteen datasets, and five cached attacks. It analyzes average robustness, cross-attack error overlap, clean-correct retention, and computational cost. The results reveal ranking differences between average and joint robustness, while a limited adaptive study shows that DBD’s gains on cached attacks are substantially reduced under defense-aware attacks.
Critical strengths
1、Large-scale paired evaluation. • Methods are evaluated with shared models, prompts, and attacked inputs, enabling reliable per-image and cross-attack comparisons. 2、Comprehensive and transparent analysis. • The paper evaluates robustness, clean performance, error overlap, and computational cost while clearly acknowledging its experimental limitations.
Critical weaknesses
• Limited adaptive evaluation. Most results measure transfer attacks against the undefended backbone. The adaptive study covers only one backbone, one dataset, and one attack method. • Uncertain comparison stability. Several methods use modified configurations, while the main evaluation uses only one seed and includes quantized attacks that may exceed the stated perturbation bounds.
Questions for authors
• Do the main method rankings persist under adaptive attacks on additional datasets and backbones? • What are the specific differences between this paper and 'Adversaries Fight Back: Revisiting Test-Time Defenders of Vision-Language Models under Adaptive Attacks'? Could you at least include its CLIP-DDT or an equally strong adaptive attack as a comparison? • Are MeanRA and WCRA stable across multiple method and attack seeds? • Can the authors verify that all decoded adversarial examples satisfy the stated perturbation bounds?
9 comments
Response to Reviewer juxx (2/6): Budget/EOT Diagnostic and CLIP-DDT
Adversaries Fight Back studies attacks on test-time CLIP defenses through CLIP-DDT. Our question is shared-input method selection across finite attack conditions: how error overlap, clean retention and matched cost affect the comparison. We have broadened adaptive coverage, but have not performed a direct CLIP-DDT comparison or established equivalent attack strength. The requested reference is https://openreview.net/forum?id=hZVSKjtaS5; the separately published CLIP-MAD study is cited independently and is not treated as the same implementation.
The added six-dataset CLIP-B/16 study and independent SPSA/Square comparison on CLIP-B/16 and DataComp query the complete adapted predictor. Independent optimizer families, reductions beyond the shared Initial control, disjoint held-out scoring and retained bounds provide broader empirical adaptive evidence. This is neither a new attack-method contribution nor a claim to outperform CLIP-DDT. DataComp R-TPT shows no demonstrated further reduction from Initial to Combined, making the finite-budget limitation explicit. Neither CLIP-DDT nor CLIP-MAD strength equivalence has been established.
Appendix Q reports the higher-budget/EOT diagnostic: five development images per dataset from the existing DataComp DTD/OxfordPets panel, R-TPT and DBD, and 8000 complete predictor calls per SPSA/Square optimizer at EOT=4 or EOT=16. Two restarts and EOT remain inside each cap, giving 16000 search calls for Combined. Sixteen fresh common held-out draws rescore both the historical 2000-call/EOT4 outputs and each new condition, so comparisons use the same scoring bank. Combined selects by search margins, not held-out correctness; EOT settings are reported separately, without a held-out or cross-EOT winner selection.
At EOT=4, DBD's matched baseline-to-new Combined accuracy changes from 100.00 to 52.50% on DTD and from 77.50 to 56.25% on OxfordPets: reductions of 47.50 and 21.25 percentage points. R-TPT changes from 80.00 to 80.00% and from 60.00 to 58.75%, respectively. At EOT=16, new accuracies are 100.00/77.50% for DBD and 80.00/60.00% for R-TPT, equal to their rescored baselines: no further reduction in these reported cells. Higher EOT leaves fewer proposals under a fixed call cap and is not automatically stronger.
All 40 image-method-configuration rows and 80 searches are complete and independently checked. Appendix Q gives the detailed scoring/gate call accounting and retained completed cost; historical searches are reused. These are descriptive, post-hoc development diagnostics on five images per dataset, not an independent confirmation cohort, evidence of convergence or an equivalent-strength test. The direct CLIP-DDT comparison remains open.
Revision locations: Section 2 (CLIP-DDT/CLIP-MAD); Section 6 and Appendix N (adaptive scope); Appendix Q (Table 44, budget/EOT diagnostic).
Response to Reviewer juxx (3/6): Attack-Seed Study
Reviewer juxx — seeds (J3)
The enlarged attack-repeat study reproduces the R-TPT-versus-MAC WCRA ordering on both confirmation cohorts at all five attack seeds. This is consistent with the separate full-manifest defense-repeat study, where R-TPT exceeds MAC in WCRA at all three defense seeds. The two studies vary different sources of randomness and are reported separately.
Appendix P reports five freshly generated offline attack suites on two fixed, class-balanced 200-image cohorts from DTD and OxfordPets, retaining the full 47/37-class vocabularies. The selection rule, IDs, exclusions and protocol were frozen before these confirmation runs. The cohorts exclude the previous focused, development and adaptive-confirmation IDs; selection did not use correctness or method outcomes. These are new focused cohorts, not test images unseen by earlier full-manifest benchmark evaluations.
CLIP-B/16, R-TPT/MAC/DBD, starting prompts, FP32, batch size one, input order and method settings remain fixed. Each suite contains PGD10, APGD-CE30, APGD-T30 and Square5000 at . Attack seeds vary while an independent per-image defender bank r0 remains fixed; its key excludes the attack and s. All 40 fresh caches, 120 attacked evaluations and six matched clean controls are complete: 25200 predictions. Original-ID, input-byte, label, full-score, model/source and decoded-bound checks pass. Exact runtime and replay checks are enforced within each dataset panel.
The table gives mean sample SD across the five attack suites, in percent; SDs are percentage points, not confidence intervals.
| Dataset | Method | MeanRA | MinRA | WCRA |
|---|---|---|---|---|
| DTD | R-TPT | |||
| DTD | MAC | |||
| DTD | DBD | |||
| OxfordPets | R-TPT | |||
| OxfordPets | MAC | |||
| OxfordPets | DBD |
R-TPT exceeds MAC in WCRA in each of the five suites on each dataset. The suite-mean paired R-TPT-minus-MAC differences are +11.30 percentage points on DTD and +9.90 points on OxfordPets. Their conditional 95% paired image-bootstrap intervals are [6.50,16.20] and [4.10,15.70] points, respectively. The 10000 replicates resample original image IDs jointly across methods, attacks, all five suites and clean controls. These intervals condition on the selected cohorts and recorded suites/defender banks; they do not measure population seed variance, imply significance at every individual seed, or include a multiple-comparison correction.
The stable comparison is metric-specific. MAC exceeds R-TPT in OxfordPets MeanRA at all five attack seeds, reproducing the distinction between average and per-image intersection-based selection. DBD leads MeanRA and WCRA on both cohorts at every recorded attack seed. OxfordPets MinRA has a narrower exception: MAC exceeds R-TPT at s3 (46.50 versus 40.50%), while the other four suites favor R-TPT. We therefore do not assert universal ranking stability across metrics. WCRA intersects the four attacks within each s; candidates are never pooled across suite seeds.
Revision locations: Appendix P (Tables 41-43 and Figure 11); Appendix M (Table 28, separate defense-seed study).
Response to Reviewer juxx (1/6): Adaptive Evaluation and Controls
Thank you for the detailed questions on adaptive evaluation, seeds, input bounds, and common settings.
The added studies do not support a single ranking across all tested adaptive settings. On CLIP-B/16 DTD, Combined accuracy is 28.63% for R-TPT and 27.88% for DBD: the 0.75-point R-TPT-minus-DBD gap has a 95% paired image-bootstrap CI of [-9.50,11.13], leaving their ordering unresolved. On OxfordPets, DBD exceeds R-TPT by 34.13 [21.00,47.50] points. On DataComp, R-TPT shows no demonstrated additional reduction from Initial to Combined. Thus the cached-suite ordering does not carry over uniformly to these full-predictor attacks.
We expanded the original single-dataset adaptive case study with a six-dataset study of all seven original predictors and an independently implemented SPSA/Square comparison on two model/pretraining configurations, CLIP-B/16 and DataComp ViT-B/16. The independent comparison queries zero-shot, R-TPT, MAC and DBD on DTD/OxfordPets: 50 images per dataset for CLIP-B/16 (400 image–method pairs, 800 searches), and twenty of the same ordered IDs for DataComp (160 pairs, 320 searches). Both use the full 47/37-class vocabularies without correctness filtering; the twenty-image cohort does not cover every class. Section 6 and Appendix N give both optimizers, paired uncertainty and the matched twenty-ID B16 subset.
SPSA-style finite differences and Square-style search each use 2000 complete defended-prediction calls, with EOT=4 and two restarts inside that cap. Probes, initialization and candidate evaluations are charged; EOT/restarts are not multiplied into the cap again. Combined selects the output with the lower saved search-bank margin: 4000 search calls and zero extra selection calls. Sixteen disjoint held-out method draws score the conditions, with sixteen calls per unique selected image; aliases reuse scores and actual counts are retained. PGD warm-start generation and separate oracle gates are outside these search counts. Both algorithms start from identical clean, cached PGD and one feasible random stripe input. Initial selects among this same three-input pool using search scores. Neither Initial nor Combined uses held-out correctness for selection. Every prediction resets adaptation and receives only the current input and common context; labels remain outside prediction. Selected inputs obey exact four-intensity-level bounds against retained canonical uint8 anchors. Independent output checks verified all 560 pairs, full class scores, RNG banks, shared initial inputs, selection choices and call accounting.
For B16 DBD on DTD, cached accuracy is 77.88%, Initial 37.63% and Combined 27.88%; Initial-minus-Combined is 9.75 [3.25,18.00] points. On Pets the reduction is 13.75 [5.50,23.63] points. Initialization accounts for much of the cached-to-Combined difference, with further loss beyond that control. MAC loses 12.25 [6.38,19.13] and 35.00 [25.50,44.88] points. These contrasts describe the complete search-and-selection procedure, not isolated causal optimizer effects.
DataComp R-TPT has 35.31/39.06% Combined accuracy on DTD/Pets versus 33.13/39.38% Initial. MAC falls from 26.25/69.06% Initial to 15.00/23.44% Combined; DBD falls from 38.75/74.06% to 30.00/69.06%. Unchanged or increased held-out accuracy does not establish search convergence. The 10000 paired image-bootstrap replicates, seed zero, preserve all draws, methods and conditions of an image. Intervals condition on recorded images, searches and RNG banks, not independent complete attack repetitions. Matched twenty-ID comparisons retain differences in processing, anchors, same-model PGD inputs and RNG realizations; they do not isolate architecture or pretraining causally.
Appendix I reports the separate six-dataset, seven-predictor B16 study (50 images/dataset, 4000 search calls), verified from 2100 existing records. DBD obtains 82.06% cached, 51.98% adaptive and 60.19% clean accuracy, with equal dataset weights. MAC-minus-R-TPT changes from 7.78 [3.86,11.67] points cached to -1.55 [-5.32,2.17] adaptive; the latter includes zero. Focus IDs overlap other studies, including the original DTD cohort, and are not pooled as independent image replications.
Revision locations: Section 6 (Table 3); Appendix I (Tables 19-20); Appendix N (Tables 36-38 and Figure 10).