- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Text-embedding Perturbation Attack (TPA) for generating transferable natural adversarial examples with diffusion models. TPA temporarily perturbs text conditioning during early denoising, progressively restores the original conditioning, and applies surrogate-classifier guidance along the sampling trajectory. Experiments on ImageNet show improved black-box transfer across multiple surrogate–victim pairs. Matched ablations evaluate the contribution of conditioning perturbations, while additional analyses examine image quality, computational cost, and human recognition.
Critical strengths
Consistent transfer gains supported by controlled ablations. TPA improves black-box attack success over the evaluated baselines across all reported surrogate–victim pairs, including adversarially trained classifiers. More importantly, the matched ablation retains the same guidance settings and class/latent seeds, providing meaningful evidence that conditioning perturbations contribute to transferability beyond the other components of the attack.
Practical efficiency and transparent evaluation. The method requires no additional training and uses fewer sampling steps without retries. The paper evaluates all final outputs rather than filtering successful or visually appealing samples. It also clearly acknowledges image-quality trade-offs, differences in baseline budgets, and the limited interpretation of the human study and distributional metrics.
Critical weaknesses
Semantic validity is not established sufficiently broadly to fully interpret attack success. In reference-free generation, disagreement with the requested class may reflect either a classifier failure or a generated image that no longer clearly belongs to that class. The human study provides encouraging recognition results, but covers a limited class subset and one surrogate setting, without a matched comparison against the baselines. Learned quality metrics and similarity to ImageNet-A do not resolve this distinction. Consequently, the reported transfer gains cannot yet be fully separated from possible changes in class fidelity or semantic ambiguity.
The specific contribution of early conditioning exploration remains partly unresolved. The conditioning perturbation and restoration rule is adopted from CADS, so the main contribution is its integration into adversarial generation. The matched ablations demonstrate usefulness, but do not establish whether the benefit specifically depends on early text-conditioning exploration or could be obtained through other forms of sampling diversity. This distinction matters because adding the perturbations also reduces image-quality scores. Comparisons at similar levels of class fidelity and image quality would strengthen the explanation of why the method transfers better.
Questions for authors
- Can you provide a matched human evaluation of TPA and the main baselines, reporting attack success on images independently judged to retain the requested class?
- How does early text-conditioning perturbation compare with later conditioning perturbation or other simple diversity controls under matched guidance and comparable image quality?
- How stable are the transfer improvements across additional generation seeds and a classifier panel held out from method development? Uncertainty estimates would help assess the smaller gains over NatADiff.
4 comments
Q1. Matched human evaluation and attack success on class-valid images
The evaluation design and plans for reporting the results are provided in the “Weakness. Semantic validity and matched human evaluation” response above.
Q2. Perturbation timing and simple diversity controls
The reviewer requested comparisons of early text-conditioning perturbation with later conditioning perturbation and simple diversity controls under matched guidance and comparable image quality.
To examine the effect of perturbation timing, we are conducting a comparison across all 1,000 ImageNet classes. We shift the window in which text-conditioning perturbations are applied while keeping source classes, latent seeds, and adversarial guidance settings fixed, and evaluate attack success alongside image-quality metrics. We will provide the results once generation and evaluation are complete.
Comparisons against early CFG reduction, prompt dropout, and latent stochasticity, together with their interpretation, are presented in the conditioning-exploration response above.
Q3. Stability across additional generation seeds and held-out classifiers
The reviewer requested performance evaluation across additional generation seeds and verification that transfer improvements persist on a classifier panel held out from method development, together with uncertainty estimates to assess the stability of the performance differences relative to NatADiff.
We conducted additional evaluations on VMamba-T, DenseNet-201, and DeiT III-Base. VMamba-T was not used during method development. We also expanded the set of evaluated models within the CNN and Transformer families by including DenseNet-201 and DeiT III-Base, respectively.
| Generation surrogate | Method | VMamba-T ASR (%) ↑ | DenseNet-201 ASR (%) ↑ | DeiT III-Base ASR (%) ↑ |
|---|---|---|---|---|
| RN50 | TPA | 68.70 | 75.30 | 61.85 |
| RN50 | NatADiff | 64.30 | 70.75 | 58.90 |
| RN50 | AdvClass | 34.65 | 38.65 | 33.55 |
| Inception-v3 | TPA | 65.40 | 71.20 | 57.15 |
| Inception-v3 | NatADiff | 55.60 | 62.95 | 51.45 |
| Inception-v3 | AdvClass | 33.20 | 37.40 | 32.40 |
| ViT-H/14 | TPA | 59.85 | 62.00 | 60.70 |
| ViT-H/14 | NatADiff | 55.75 | 59.35 | 56.20 |
| ViT-H/14 | AdvClass | 31.55 | 33.40 | 32.90 |
TPA achieves the highest ASR across all combinations of the three surrogates and three additional victim classifiers.
Experiments with additional generation seeds are ongoing. Once generation and evaluation, including NatADiff, are complete, we will report variability across seeds and uncertainty estimates for the TPA–NatADiff differences. We will also include these results in the revised manuscript’s appendix.
Thank you for the additional results. I maintain my score.
We thank the reviewer for their response and patience. We now provide the additional results discussed in our previous response.
Q1. Attack success on images recognized by humans as belonging to the requested class
We now report the results of the additional human study discussed in our previous response. Using the 148-class set specified by NatADiff as a reference, we evaluated 300 images generated by TPA, NatADiff, and AdvClass using RN50 and ViT-H/14 as surrogates. Images generated with the same source classes and latent seeds were presented without disclosing the generation method, and we aggregated the completed survey responses.
For each participant, we computed classifier-wise ASR on the images they recognized as belonging to the requested class, then averaged the results across participants.
| Surrogate | Method | RN-50 | Inc-v3 | RN-152 | AdvRes | AdvInc | ViT-H | Max-ViT | Swin-B | ViT-B | Average ASR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RN-50 | AdvClass | 100.00† | 32.75 | 23.77 | 28.12 | 30.11 | 27.73 | 17.81 | 28.00 | 23.64 | 34.66 |
| RN-50 | NatADiff | 97.40† | 49.38 | 56.46 | 62.57 | 61.05 | 40.70 | 55.78 | 54.57 | 49.27 | 58.58 |
| RN-50 | TPA | 87.28† | 50.92 | 62.61 | 46.89 | 54.89 | 31.83 | 45.92 | 47.39 | 48.87 | 52.96 |
| ViT-H/14 | AdvClass | 26.25 | 23.00 | 21.00 | 18.62 | 25.38 | 97.62† | 16.23 | 17.04 | 17.43 | 29.17 |
| ViT-H/14 | NatADiff | 42.64 | 60.36 | 33.84 | 46.79 | 53.10 | 98.84† | 45.83 | 40.21 | 51.12 | 52.52 |
| ViT-H/14 | TPA | 55.18 | 48.01 | 51.93 | 52.47 | 51.92 | 78.13† | 52.87 | 47.12 | 53.85 | 54.61 |
TPA achieves higher Average ASR than AdvClass with both surrogates, even on images that participants recognized as belonging to the requested class. Between TPA and NatADiff, NatADiff achieves higher Average ASR with RN50, whereas TPA achieves higher Average ASR with ViT-H/14.
We will add the detailed evaluation procedures and results to the appendix of the revised manuscript.
Q2. Timing of text-conditioning perturbations
We now report the perturbation-timing comparison results discussed in our previous response. Using an RN50 surrogate and all 1,000 ImageNet classes, we generated 2,000 images per condition. We compared Early, Late, and Full-window while keeping the source classes, latent seeds, and adversarial guidance settings fixed.
Early applies perturbations at steps 0–19, and Late at steps 30–49. Full-window applies perturbations over steps 0–49, with its schedule adjusted to match the cumulative noise variance.
| Metric | Early (TPA) | Late | Full-window |
|---|---|---|---|
| Surrogate ASR (%) ↑ | 96.80 | 90.65 | 93.45 |
| Average ASR (%) ↑ | 70.11 | 61.06 | 61.91 |
| TReS ↑ | 80.35 | 79.76 | 82.00 |
| BRISQUE ↓ | 17.51 | 15.98 | 15.80 |
| FID-VAL ↓ | 32.62 | 30.57 | 31.29 |
| FID-A ↓ | 67.73 | 76.75 | 79.08 |
Early achieves the highest Average ASR and the lowest FID-A among the schedules compared. Although Full-window performs better on TReS and BRISQUE, Early achieves a higher attack success rate. These results provide additional empirical support for TPA's choice of perturbing text conditioning during early denoising and subsequently restoring the original conditioning.
We have added the detailed perturbation schedules and complete image-quality metrics to the appendix of the revised manuscript.
Q3. Stability across additional generation seeds and uncertainty estimates
We now report the additional generation-seed results discussed in our previous response. Using an RN50 surrogate and all 1,000 ImageNet classes, we added two seed slots to the original two, evaluating TPA and NatADiff on 4,000 images each.
| Seed slot | TPA Average ASR (%) ↑ | NatADiff Average ASR (%) ↑ | TPA − NatADiff (pp) [95% CI] |
|---|---|---|---|
| 1 | 68.74 | 65.37 | +3.38 [+1.39, +5.44] |
| 2 | 69.66 | 65.86 | +3.80 [+1.64, +5.99] |
| 3 | 69.93 | 64.54 | +5.39 [+3.34, +7.47] |
| 4 | 70.28 | 63.61 | +6.67 [+4.59, +8.73] |
| Overall | 69.65 | 64.84 | +4.81 [+3.47, +6.14] |
TPA achieves higher Average ASR than NatADiff in all four seed slots. The overall improvement is 4.81 percentage points, with a 95% confidence interval of [+3.47, +6.14] percentage points computed using a class-level paired bootstrap.
We have added the per-seed results, uncertainty estimates, classifier-wise ASR, and image-quality metrics to the appendix of the revised manuscript.
We thank the reviewer for the thoughtful review. We particularly appreciate your recognition of the consistent transfer gains supported by our matched ablations, as well as TPA’s practical efficiency and the transparency of our evaluation.
Weakness. Semantic validity and matched human evaluation
The reviewer raised the need to expand the class subset and surrogate settings and include matched human evaluation against the baselines to distinguish classifier failures from reduced class fidelity or semantic ambiguity.
Regarding the class subset, our original human study used the same subset specified by NatADiff because some fine-grained ImageNet classes can be difficult for general participants to distinguish.
Accordingly, we are conducting an additional human study that retains this subset, adds ViT-H/14 alongside RN50 as a surrogate, and directly compares TPA with NatADiff and AdvClass.
Under matched source classes and latent seeds, participants evaluate images from all three methods without knowing the generation method. We will compare attack transferability on images that participants recognize as belonging to the requested class. This evaluation will assess whether transfer gains persist on human-recognized images, independently of learned quality metrics or similarity to ImageNet-A.
Collecting responses from human participants requires additional time. We will provide the results once data collection and analysis are complete.
Weakness. Early conditioning exploration and diversity controls
The reviewer requested additional comparisons to distinguish whether the benefits of conditioning perturbations depend on early text-conditioning exploration or can also be obtained through other forms of sampling diversity, considering both class fidelity and image quality.
Our central view is that broadening exploration during generation through early conditioning adjustments can benefit attack transferability. We use CADS to implement this form of exploration in adversarial generation. To examine this explanation, we compared TPA with TPA w/o text perturbation, early CFG reduction, prompt dropout, and latent stochasticity.
All methods use the same RN50 surrogate, 1,000 ImageNet classes, and two generation-seed slots per class. The table reports Average ASR alongside visual-quality metrics. Average ASR is the mean ASR across all nine classifiers, including the surrogate.
TPA, early CFG reduction, and prompt dropout all improve Average ASR over TPA w/o text perturbation. Notably, early CFG reduction and prompt dropout achieve higher Average ASR than TPA while also performing better on TReS and BRISQUE. These results indicate that the gains are not unique to CADS-style embedding noise and support the broader view that temporarily adjusting early conditioning can benefit cross-model transfer.
The tested latent-stochasticity control does not reproduce these improvements. A plausible explanation is the different intervention location. Latent stochasticity changes the stochastic DDIM transition while keeping text embeddings and CFG fixed, so subsequent denoising continues to use the original conditioning. In contrast, TPA, early CFG reduction, and prompt dropout modify the conditioning signal or its influence on denoising, directly changing the intermediate predictions used by adversarial guidance. The observed pattern is consistent with this distinction and supports our choice to induce exploration at the conditioning level.
We will incorporate these results and the corresponding discussion into the revised manuscript. We report image quality alongside attack success in this comparison; our additional evaluation of class fidelity is described in the semantic-validity response above.