acceptodds
Under review as a conference paper at ICLR 2027

TPA: Early Text-Embedding Perturbations for Natural Adversarial Diffusion

Abstract

Natural adversarial examples show that classifiers can fail at test time even on images that are clearly recognizable to humans, not just on images modified with small input perturbations. A key challenge is to make adversarial examples work across different models without reducing image quality. Text conditioning influences which visual variations of a class the diffusion model explores. Keeping the text condition fixed may restrict adversarial search to only a subset of these variations. We propose Text-embedding Perturbation Attack (TPA), which temporarily perturbs text embeddings during early denoising and progressively restores the original conditions while guiding sampling with classifier gradients. This exploration changes both the denoising trajectory and the intermediate images on which adversarial guidance acts. On ImageNet-1K, TPA achieves higher black-box attack success rates than existing natural adversarial generation methods across all evaluated surrogate–victim pairs, while outperforming NatADiff on most learned image-quality metrics. It also achieves lower FID to ImageNet-A than both baselines. Matched ablations show consistent transfer gains from early conditioning perturbations across the evaluated black-box classifiers, including adversarially trained models. These results establish early text-conditioning exploration as an effective strategy for generating transferable natural adversarial examples and probing test-time vulnerabilities across classifiers.

Primary area:
interpretability and explainable AI
Keywords:
Adversarial Attack, Natural Adversarial Sample, Diffusion Model, Text Conditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8RbrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Text-embedding Perturbation Attack (TPA) for generating transferable natural adversarial examples with diffusion models. TPA temporarily perturbs text conditioning during early denoising, progressively restores the original conditioning, and applies surrogate-classifier guidance along the sampling trajectory. Experiments on ImageNet show improved black-box transfer across multiple surrogate–victim pairs. Matched ablations evaluate the contribution of conditioning perturbations, while additional analyses examine image quality, computational cost, and human recognition.

Critical strengths

Consistent transfer gains supported by controlled ablations. TPA improves black-box attack success over the evaluated baselines across all reported surrogate–victim pairs, including adversarially trained classifiers. More importantly, the matched ablation retains the same guidance settings and class/latent seeds, providing meaningful evidence that conditioning perturbations contribute to transferability beyond the other components of the attack.

Practical efficiency and transparent evaluation. The method requires no additional training and uses fewer sampling steps without retries. The paper evaluates all final outputs rather than filtering successful or visually appealing samples. It also clearly acknowledges image-quality trade-offs, differences in baseline budgets, and the limited interpretation of the human study and distributional metrics.

Critical weaknesses

Semantic validity is not established sufficiently broadly to fully interpret attack success. In reference-free generation, disagreement with the requested class may reflect either a classifier failure or a generated image that no longer clearly belongs to that class. The human study provides encouraging recognition results, but covers a limited class subset and one surrogate setting, without a matched comparison against the baselines. Learned quality metrics and similarity to ImageNet-A do not resolve this distinction. Consequently, the reported transfer gains cannot yet be fully separated from possible changes in class fidelity or semantic ambiguity.

The specific contribution of early conditioning exploration remains partly unresolved. The conditioning perturbation and restoration rule is adopted from CADS, so the main contribution is its integration into adversarial generation. The matched ablations demonstrate usefulness, but do not establish whether the benefit specifically depends on early text-conditioning exploration or could be obtained through other forms of sampling diversity. This distinction matters because adding the perturbations also reduces image-quality scores. Comparisons at similar levels of class fidelity and image quality would strengthen the explanation of why the method transfers better.

Questions for authors
  1. Can you provide a matched human evaluation of TPA and the main baselines, reporting attack success on images independently judged to retain the requested class?
  2. How does early text-conditioning perturbation compare with later conditioning perturbation or other simple diversity controls under matched guidance and comparable image quality?
  3. How stable are the transfer improvements across additional generation seeds and a classifier panel held out from method development? Uncertainty estimates would help assess the smaller gains over NatADiff.
4 comments

We thank the reviewer for the thoughtful review. We particularly appreciate your recognition of the consistent transfer gains supported by our matched ablations, as well as TPA’s practical efficiency and the transparency of our evaluation.

Weakness. Semantic validity and matched human evaluation

The reviewer raised the need to expand the class subset and surrogate settings and include matched human evaluation against the baselines to distinguish classifier failures from reduced class fidelity or semantic ambiguity.

Regarding the class subset, our original human study used the same subset specified by NatADiff because some fine-grained ImageNet classes can be difficult for general participants to distinguish.

Accordingly, we are conducting an additional human study that retains this subset, adds ViT-H/14 alongside RN50 as a surrogate, and directly compares TPA with NatADiff and AdvClass.

Under matched source classes and latent seeds, participants evaluate images from all three methods without knowing the generation method. We will compare attack transferability on images that participants recognize as belonging to the requested class. This evaluation will assess whether transfer gains persist on human-recognized images, independently of learned quality metrics or similarity to ImageNet-A.

Collecting responses from human participants requires additional time. We will provide the results once data collection and analysis are complete.

Weakness. Early conditioning exploration and diversity controls

The reviewer requested additional comparisons to distinguish whether the benefits of conditioning perturbations depend on early text-conditioning exploration or can also be obtained through other forms of sampling diversity, considering both class fidelity and image quality.

Our central view is that broadening exploration during generation through early conditioning adjustments can benefit attack transferability. We use CADS to implement this form of exploration in adversarial generation. To examine this explanation, we compared TPA with TPA w/o text perturbation, early CFG reduction, prompt dropout, and latent stochasticity.

All methods use the same RN50 surrogate, 1,000 ImageNet classes, and two generation-seed slots per class. The table reports Average ASR alongside visual-quality metrics. Average ASR is the mean ASR across all nine classifiers, including the surrogate.

MethodAverage ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
TPA w/o text perturbation59.9482.1015.7131.0279.29
TPA70.1180.3517.5132.6267.73
Early CFG reduction71.0582.1617.3531.9462.88
Prompt dropout72.1781.8717.2533.1060.86
Latent stochasticity58.4481.0917.0831.7184.81

TPA, early CFG reduction, and prompt dropout all improve Average ASR over TPA w/o text perturbation. Notably, early CFG reduction and prompt dropout achieve higher Average ASR than TPA while also performing better on TReS and BRISQUE. These results indicate that the gains are not unique to CADS-style embedding noise and support the broader view that temporarily adjusting early conditioning can benefit cross-model transfer.

The tested latent-stochasticity control does not reproduce these improvements. A plausible explanation is the different intervention location. Latent stochasticity changes the stochastic DDIM transition while keeping text embeddings and CFG fixed, so subsequent denoising continues to use the original conditioning. In contrast, TPA, early CFG reduction, and prompt dropout modify the conditioning signal or its influence on denoising, directly changing the intermediate predictions used by adversarial guidance. The observed pattern is consistent with this distinction and supports our choice to induce exploration at the conditioning level.

We will incorporate these results and the corresponding discussion into the revised manuscript. We report image quality alongside attack success in this comparison; our additional evaluation of class fidelity is described in the semantic-validity response above.

Q1. Matched human evaluation and attack success on class-valid images

The evaluation design and plans for reporting the results are provided in the “Weakness. Semantic validity and matched human evaluation” response above.

Q2. Perturbation timing and simple diversity controls

The reviewer requested comparisons of early text-conditioning perturbation with later conditioning perturbation and simple diversity controls under matched guidance and comparable image quality.

To examine the effect of perturbation timing, we are conducting a comparison across all 1,000 ImageNet classes. We shift the window in which text-conditioning perturbations are applied while keeping source classes, latent seeds, and adversarial guidance settings fixed, and evaluate attack success alongside image-quality metrics. We will provide the results once generation and evaluation are complete.

Comparisons against early CFG reduction, prompt dropout, and latent stochasticity, together with their interpretation, are presented in the conditioning-exploration response above.

Q3. Stability across additional generation seeds and held-out classifiers

The reviewer requested performance evaluation across additional generation seeds and verification that transfer improvements persist on a classifier panel held out from method development, together with uncertainty estimates to assess the stability of the performance differences relative to NatADiff.

We conducted additional evaluations on VMamba-T, DenseNet-201, and DeiT III-Base. VMamba-T was not used during method development. We also expanded the set of evaluated models within the CNN and Transformer families by including DenseNet-201 and DeiT III-Base, respectively.

Generation surrogateMethodVMamba-T ASR (%) ↑DenseNet-201 ASR (%) ↑DeiT III-Base ASR (%) ↑
RN50TPA68.7075.3061.85
RN50NatADiff64.3070.7558.90
RN50AdvClass34.6538.6533.55
Inception-v3TPA65.4071.2057.15
Inception-v3NatADiff55.6062.9551.45
Inception-v3AdvClass33.2037.4032.40
ViT-H/14TPA59.8562.0060.70
ViT-H/14NatADiff55.7559.3556.20
ViT-H/14AdvClass31.5533.4032.90

TPA achieves the highest ASR across all combinations of the three surrogates and three additional victim classifiers.

Experiments with additional generation seeds are ongoing. Once generation and evaluation, including NatADiff, are complete, we will report variability across seeds and uncertainty estimates for the TPA–NatADiff differences. We will also include these results in the revised manuscript’s appendix.

Thank you for the additional results. I maintain my score.

We thank the reviewer for their response and patience. We now provide the additional results discussed in our previous response.

Q1. Attack success on images recognized by humans as belonging to the requested class

We now report the results of the additional human study discussed in our previous response. Using the 148-class set specified by NatADiff as a reference, we evaluated 300 images generated by TPA, NatADiff, and AdvClass using RN50 and ViT-H/14 as surrogates. Images generated with the same source classes and latent seeds were presented without disclosing the generation method, and we aggregated the completed survey responses.

For each participant, we computed classifier-wise ASR on the images they recognized as belonging to the requested class, then averaged the results across participants.

SurrogateMethodRN-50Inc-v3RN-152AdvResAdvIncViT-HMax-ViTSwin-BViT-BAverage ASR
RN-50AdvClass100.00†32.7523.7728.1230.1127.7317.8128.0023.6434.66
RN-50NatADiff97.40†49.3856.4662.5761.0540.7055.7854.5749.2758.58
RN-50TPA87.28†50.9262.6146.8954.8931.8345.9247.3948.8752.96
ViT-H/14AdvClass26.2523.0021.0018.6225.3897.62†16.2317.0417.4329.17
ViT-H/14NatADiff42.6460.3633.8446.7953.1098.84†45.8340.2151.1252.52
ViT-H/14TPA55.1848.0151.9352.4751.9278.13†52.8747.1253.8554.61

TPA achieves higher Average ASR than AdvClass with both surrogates, even on images that participants recognized as belonging to the requested class. Between TPA and NatADiff, NatADiff achieves higher Average ASR with RN50, whereas TPA achieves higher Average ASR with ViT-H/14.

We will add the detailed evaluation procedures and results to the appendix of the revised manuscript.

Q2. Timing of text-conditioning perturbations

We now report the perturbation-timing comparison results discussed in our previous response. Using an RN50 surrogate and all 1,000 ImageNet classes, we generated 2,000 images per condition. We compared Early, Late, and Full-window while keeping the source classes, latent seeds, and adversarial guidance settings fixed.

Early applies perturbations at steps 0–19, and Late at steps 30–49. Full-window applies perturbations over steps 0–49, with its schedule adjusted to match the cumulative noise variance.

MetricEarly (TPA)LateFull-window
Surrogate ASR (%) ↑96.8090.6593.45
Average ASR (%) ↑70.1161.0661.91
TReS ↑80.3579.7682.00
BRISQUE ↓17.5115.9815.80
FID-VAL ↓32.6230.5731.29
FID-A ↓67.7376.7579.08

Early achieves the highest Average ASR and the lowest FID-A among the schedules compared. Although Full-window performs better on TReS and BRISQUE, Early achieves a higher attack success rate. These results provide additional empirical support for TPA's choice of perturbing text conditioning during early denoising and subsequently restoring the original conditioning.

We have added the detailed perturbation schedules and complete image-quality metrics to the appendix of the revised manuscript.

Q3. Stability across additional generation seeds and uncertainty estimates

We now report the additional generation-seed results discussed in our previous response. Using an RN50 surrogate and all 1,000 ImageNet classes, we added two seed slots to the original two, evaluating TPA and NatADiff on 4,000 images each.

Seed slotTPA Average ASR (%) ↑NatADiff Average ASR (%) ↑TPA − NatADiff (pp) [95% CI]
168.7465.37+3.38 [+1.39, +5.44]
269.6665.86+3.80 [+1.64, +5.99]
369.9364.54+5.39 [+3.34, +7.47]
470.2863.61+6.67 [+4.59, +8.73]
Overall69.6564.84+4.81 [+3.47, +6.14]

TPA achieves higher Average ASR than NatADiff in all four seed slots. The overall improvement is 4.81 percentage points, with a 95% confidence interval of [+3.47, +6.14] percentage points computed using a class-level paired bootstrap.

We have added the per-seed results, uncertainty estimates, classifier-wise ASR, and image-quality metrics to the appendix of the revised manuscript.

Reviewer_VfaSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TPA, a training-free approach for generating transferable natural adversarial examples using diffusion models. TPA perturbs text embeddings during early denoising process and progressively restores the original conditioning, while applying classifier-guided updates to the latent trajectory to induce misclassification. The results evaluated on ImageNet-1K show TPA's black-box ASR and visual quality are generally superior to those of AdvClass and NatADiff.

Critical strengths
  1. Novel and simple approach. The paper introduces early text-embedding perturbations into diffusion-based adversarial generation, providing a training-free way to improve cross-model transferability without modifying the generator or surrogate model parameters.
Critical weaknesses
  1. Unclear mechanism. The underlying mechanism of early text-conditioning perturbations remains insufficiently understood. Although the ablations demonstrate improved transferability, the paper provides limited insight into why perturbing text embeddings during early denoising benefits cross-model transfer beyond the hypothesis that it changes the sampling trajectory.

  2. Insufficient justification of the two-stage objective. Appendix E.2 compares targeted-only and untargeted-only objectives, but the results do not establish the superiority of the proposed two-stage schedule. Moreover, the target-selection times differ between the variants, confounding the comparison. More controlled ablations are needed to justify this design choice.

  3. Limited experimental scope. The evaluation focuses on ImageNet-1K and Stable Diffusion 1.5, with only two competing methods for reference-free natural adversarial generation. Evaluations on additional datasets, generative backbones, or vision-language models would strengthen the evidence for the method's generalizability.

  4. The organization and structure of some sections are not clearly presented. For example, the Related Work section would benefit from clearer organization and a more explicit comparison with prior work.

Questions for authors
  1. The proposed mechanism remains insufficiently disentangled. Text-conditioning perturbations affect both the sampling trajectory and the intermediate images used to compute adversarial gradients. Can the authors clarify whether the transfer gains primarily arise from broader sampling exploration or from changes in the adversarial guidance directions? The current ablation does not distinguish these effects.

  2. In Appendix E.2, the targeted-only variant selects its target earlier than TPA, confounding the comparison. Can the authors provide a controlled comparison that isolates the effect of the two-stage objective? Given that targeted-only achieves higher ASR, what specific benefit justifies the proposed schedule?

  3. The choice of early text-conditioning perturbations lacks sufficient mechanistic justification. Is the transfer gain specific to perturbing the conditioning during early denoising, or would similar gains arise from perturbations applied at later steps or throughout sampling? A comparison of different perturbation schedules would help establish why the proposed timing is effective.

  4. Why are both conditional and unconditional text embeddings perturbed independently? Could the authors isolate their respective contributions to transferability and clarify whether the observed gains stem from increased semantic exploration or changes in the effective classifier-free guidance direction?

6 comments

We thank the reviewer for the detailed comments and suggestions for improving our paper. We particularly appreciate your recognition of TPA’s novelty and simplicity, as well as its training-free approach to improving cross-model transferability without modifying the generator or surrogate model parameters.

W1. Mechanism of early conditioning perturbations

The reviewer highlighted the need to explain why transferability improves beyond the hypothesis that conditioning perturbations change the sampling trajectory. To obtain evidence beyond the original ablation, we examined whether simpler early conditioning interventions also improve performance and whether latent stochasticity produces similar gains while keeping text embeddings and CFG fixed.

Using TPA w/o text perturbation as the reference, we compared TPA, early CFG reduction, prompt dropout, and latent stochasticity with the same RN50 surrogate, 1,000 ImageNet classes, and two generation-seed slots per class. The adversarial guidance settings and early intervention window were held fixed.

MethodAverage ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
TPA w/o text perturbation59.9482.1015.7131.0279.29
TPA70.1180.3517.5132.6267.73
Early CFG reduction71.0582.1617.3531.9462.88
Prompt dropout72.1781.8717.2533.1060.86
Latent stochasticity58.4481.0917.0831.7184.81

All three conditioning interventions improve Average ASR over TPA w/o text perturbation. TPA perturbs text embeddings using the CADS rule, CFG reduction weakens the conditioning signal through the guidance weight, and prompt dropout temporarily removes positive conditioning as an inference-time control using the conditional/unconditional formulation of CFG. In contrast, the tested latent-stochasticity control changes the stochastic DDIM transition while preserving the original text embeddings and CFG, and does not produce similar improvements.

Our hypothesis is that fixed early conditioning repeatedly steers generation toward a restricted set of image configurations, limiting the candidates available to adversarial guidance. Temporarily relaxing this conditioning may provide access to alternative configurations, some of which expose failure modes shared by multiple classifiers. This offers a possible explanation for why an attack guided by a single surrogate can transfer to other models.

This motivation is consistent with findings in standard diffusion generation. Wang et al. report that reducing guidance early and increasing it later can improve generation quality and diversity. Kynkäänniemi et al. show that guidance at high noise levels can harm sample and distribution quality, motivating its restriction to a suitable interval. These studies provide independent motivation for treating early conditioning as a design variable; our experiments examine its effect on cross-model transfer in adversarial generation.

The main comparisons show that TPA achieves competitive cross-model transferability relative to AdvClass and NatADiff through this simple early conditioning intervention (Table 1). We believe this empirical finding provides useful insight into the design of transferable generative adversarial attacks.

We will provide per-classifier results, visual-quality comparisons, and further discussion in the appendix.

W2. Justification of the two-stage objective

The reviewer raised concerns about the justification of the two-stage objective and the different target-selection times in the original comparison.

The two-stage schedule offers quality advantages despite slightly lower Average ASR than targeted-only guidance. Using the same RN50 outputs as Table 12—two images per class across all 1,000 classes—the two-stage schedule achieves higher mean scores on five of the seven learned NR-IQA metrics: HyperIQA, DBCNN, ARNIQA, MUSIQ-KonIQ, and TReS. FID-VAL also decreases from 39.01 to 32.18, while Average ASR is 69.20% for two-stage and 70.42% for targeted-only. These quality advantages provide a concrete empirical motivation for adopting the two-stage schedule in natural adversarial generation.

MetricTwo-stage (TPA; target at step 40)Targeted-only (target at step 15)
Average ASR (%) ↑69.2070.42
HyperIQA ↑0.63300.6300
DBCNN ↑0.62950.6281
ARNIQA ↑0.70870.7058
MUSIQ-AVA ↑5.27805.2927
NIMA-AVA ↑5.29895.2998
MUSIQ-KonIQ ↑70.4169.67
TReS ↑79.9479.42
BRISQUE ↓17.3317.04
FID-VAL ↓32.1839.01
FID-A ↓67.5360.16

The original schedules were chosen to match the attack interval and the number of guidance updates. Both methods perform 35 updates over steps 15–49, so this comparison evaluates their complete policies over the same attack interval.

To address the concern about target-selection timing, we also provide a 1,000-class comparison in which both methods select their target at step 40. Two-stage applies untargeted guidance over steps 15–39, followed by targeted guidance over steps 40–49. Late targeted-only applies guidance only over steps 40–49. Two-stage achieves 69.20% Average ASR, compared with 65.59% for late targeted-only, an improvement of 3.61%p. This shows that retaining the early untargeted phase benefits attack success when target selection is delayed until step 40.

MetricTwo-stage (TPA)Late targeted-only
Average ASR (%) ↑69.2065.59
TReS ↑79.9480.05
BRISQUE ↓17.3317.38
FID-VAL ↓32.1830.96
FID-A ↓67.5368.94

We will expand Appendix E.2 to include all seven learned NR-IQA metrics alongside BRISQUE and FID results, and clarify the differences between these two comparisons and the empirical rationale for the selected schedule.

W3. Experimental scope and additional dataset evaluation

The reviewer noted that the competing methods and evaluation settings are limited, and suggested additional datasets, generative backbones, or vision-language models to strengthen the evidence for generalizability. We are extending the evaluation to Oxford-IIIT Pet and explain the additional evaluation and baseline-selection criteria below.

  1. Additional evaluation on Oxford-IIIT Pet

To assess attack transferability and image quality beyond ImageNet, we are conducting additional experiments on Oxford-IIIT Pet. We compare TPA, NatADiff, and AdvClass using RN50, RN152, and ViT as surrogates. We will provide the results once generation and evaluation are complete for all methods.

  1. Scope and selection of competing methods

Our study focuses on reference-free adversarial generation: generating images without a reference image, aiming to preserve the requested class semantics while inducing misclassification, and evaluating cross-model transfer. We selected AdvClass, the adversarial classifier-guidance baseline based on Dai et al. and evaluated by Collins et al., and NatADiff, a recent leading method with strong reported cross-model transferability. We will revise the Related Work section to make the distinctions between our approach and other generative adversarial attacks explicit.

  1. Additional generative backbones and vision-language models

To compare attack methods under a common generative backbone, we evaluated TPA and both baselines using Stable Diffusion 1.5, which is based on latent diffusion models. Our evaluation also focuses on transfer between image classifiers that predict labels from a fixed class set.

We recognize that evaluation on additional generative backbones and vision-language models would provide broader evidence of generalizability. We are examining adaptation to additional generative backbones and suitable VLM evaluation settings. If these experiments can be completed within the rebuttal period, we will provide their results in a follow-up response.

W4. Organization and clarity of presentation

Thank you for highlighting the need to improve the organization and structure of the manuscript. After reviewing the presentation, we agree that some sections would benefit from clearer organization.

We will revise the Related Work section to improve its structure and make the comparisons with prior work more explicit. We will also review the remaining sections for similar readability issues and incorporate these improvements into the revised manuscript.

Q1. Sampling exploration and adversarial guidance directions

Our hypothesis is that broader sampling exploration contributes to the transfer gains. The supporting comparisons and our explanation are provided in W1. These results support the conditioning-exploration explanation, although they do not independently isolate its contribution from changes in adversarial guidance directions.

Q2. Controlled comparison of the two-stage objective

The comparisons addressing target-selection timing and the quality benefits motivating the two-stage schedule are provided in W2.

Q3. Comparison of perturbation schedules

Thank you for suggesting this comparison. To examine the effect of perturbation timing, we are conducting experiments across all 1,000 ImageNet classes. We shift the window in which text-conditioning perturbations are applied while keeping source classes, latent seeds, and adversarial guidance settings fixed, and evaluate attack success alongside image-quality metrics.

We will provide the results once generation and evaluation are complete.

Q4. Contributions of the conditional and null branches

We thank the reviewer for the in-depth question about the mechanisms underlying these performance gains.

As explained in our response to W1, our study focuses on improving cross-model transfer through interventions in early text conditioning, with CADS serving as the mechanism for implementing this intervention. Our evaluation examines the overall effect of this conditioning intervention on transferability. Disentangling the contributions of the conditional and null branches would require a more fine-grained investigation of how the adopted CADS mechanism operates within adversarial generation.

The current comparisons do not isolate the relative contributions of semantic exploration and changes in effective CFG. Changes in conditioning affect both the sampling trajectory and the intermediate predictions used to compute adversarial gradients, so these effects may be interconnected.

We hope our findings provide a useful starting point for further investigations into the mechanisms highlighted by the reviewer.

Reviewer_QFFgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TPA, a training-free method for generating transferable natural adversarial examples from a class prompt and random noise using a frozen text-to-image diffusion model. Its core idea is to treat text conditioning as an exploration variable: during early denoising, TPA injects Gaussian noise into the source and null text embeddings and progressively restores the original embeddings using the CADS perturbation/rescaling rule. This paper also applies a two-stage process to generate adversarial examples: an initial untargeted phase and a fixed-target phase selected from the current trajectory. The method is evaluated on ImageNet-1K with three surrogates (RN-50, Inception-v3, ViT-H/14), a nine-classifier victim panel including adversarially trained and transformer models, and comparisons against NatADiff and AdvClass. The experimental results show that TPA is competitive with baseline methods in achieving higher transfer ASR while keeping better naturalness.

Critical strengths
  1. The paper frames text conditioning as shaping both the denoising trajectory and the intermediate image on which adversarial gradients are computed. The hypothesis that transient early conditioning perturbations can improve access to class-valid failure regions shared by surrogate and victim models is intuitive and complementary to prior work on mixed-class conditioning, target selection, and prompt optimization.

  2. This paper makes comprehensive experiments covering all 1,000 ImageNet classes, three architecturally different surrogates, and a nine-classifier victim panel including adversarially trained models. The evaluation metrics, including black-box transfer, NR-IQA metrics, and FID-VAL/FID-A, make the empirical section persuasive.

  3. The ablation in Section 5.3 is comprehensive; it matches guidance onset, objective, gradient normalization, guidance strength, number of guided steps, class/latent seeds, and output count, offering support for the claim that the perturbation itself contributes to transfer ASR.

Critical weaknesses
  1. The core intervention, CADS-style embedding noise with annealing and moment rescaling, is adopted directly from Sadat et al. (2024). The adversarial guidance, multi-view surrogate evaluation, and two-stage targeted/untargeted objective are also close to existing generative attack designs. The contribution is therefore primarily an empirical combination and demonstration rather than a new attack principle. The paper would be stronger if it clarified what is conceptually new beyond “apply CADS during adversarial diffusion sampling” and why this specific combination is necessary.

  2. The paper repeatedly states that early conditioning perturbations broaden the explored class-valid region or improve access to shared failure modes, but the evidence is indirect. The ablation shows that adding embedding noise improves ASR, not that the mechanism is increased exploration of class-valid configurations. There is no trajectory-level analysis, condition-alignment measurement, feature-space coverage analysis, or comparison against simpler controls such as early CFG reduction, prompt dropout, or latent stochasticity. The mechanistic claim remains a plausible hypothesis rather than a demonstrated explanation.

  3. TPA improves transfer ASR, but the matched ablation shows that this comes with lower scores on all seven learned NR-IQA metrics and worse FID-VAL/BRISQUE. Relative to AdvClass, TPA is not uniformly better on quality metrics: AdvClass has higher NIMA-AVA and lower BRISQUE, which means that the quality–transfer trade-off is understated, and the quality story is mixed. In addition, FID-A is computed against a 200-class ImageNet-A reference while the generated set covers 1,000 classes, so the lower FID-A should not be interpreted too strongly as evidence of better class validity or naturalness.

Questions for authors
  1. How were the surrogate-specific guidance strengths, onset S=15, and switch K=40 selected? Were any of the nine evaluation classifiers used during tuning or model selection? Please provide a tuning protocol and a sensitivity analysis for S, K, and late guidance strength.

  2. Can you provide an ASR-versus-NFE or ASR-versus-wall-clock comparison against NatADiff and AdvClass under matched sampling budgets? The current comparison uses different retry policies and DDIM budgets, so it is difficult to attribute the gains solely to the method.

  3. What direct evidence supports the claim that early conditioning perturbations broaden the explored class-valid region? Can you measure condition alignment, trajectory divergence, feature-space coverage, or class-valid ASR conditioned on human recognition? A comparison against early CFG reduction, prompt dropout, or latent noise controls would help isolate the effect of text-embedding perturbation specifically.

  4. Why does FID-A improve while FID-VAL and most NR-IQA metrics worsen? Since FID-A uses a different class mixture and a 200-class reference, can you provide a class-balanced FID-A or additional class-validity analysis to support the interpretation that TPA generates more natural or class-recognizable failures?

5 comments

We thank the reviewer for the careful review and constructive feedback. We appreciate your recognition of the motivation for using text conditioning to shape adversarial trajectories, the breadth of our evaluation, and the matched ablations supporting the transfer benefit of text-embedding perturbations.

W1. Conceptual contribution and rationale for the combination

The reviewer asked us to clarify the conceptual contribution beyond applying CADS and the rationale for the proposed combination. TPA aims to improve cross-model transfer by encouraging adversarial search to explore more diverse image configurations corresponding to the requested class.

Our hypothesis is that fixed early conditioning repeatedly steers generation toward a restricted set of image configurations, limiting the candidates available to adversarial guidance. Temporarily relaxing this conditioning may provide access to alternative configurations, some of which expose failure modes shared by multiple classifiers. This offers a possible explanation for why an attack guided by a single surrogate can transfer to other models.

To implement this strategy, we use CADS as a tool for early text-embedding perturbation.

Controlled ablations support the transfer gains from this intervention across three surrogates (Section 5.3 and Appendix E.1). In the main comparisons, TPA achieves higher transfer ASR than AdvClass and NatADiff. Its recorded generation time is also shorter under each method’s respective experimental settings (Appendix B.5). Together, these results support the practical value of the proposed design in combining high transferability with efficient generation.

W2. Simple controls supporting early conditioning intervention

The reviewer highlighted the need for additional evidence and comparisons with simpler controls to assess the proposed conditioning-exploration explanation. We compared TPA with TPA w/o text perturbation, early CFG reduction, prompt dropout, and latent stochasticity using the same RN50 surrogate, 1,000 ImageNet classes, and two generation-seed slots per class. The attack settings and early intervention window were held fixed. The table reports Average ASR alongside visual-quality metrics.

MethodAverage ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
TPA w/o text perturbation59.9482.1015.7131.0279.29
TPA70.1180.3517.5132.6267.73
Early CFG reduction71.0582.1617.3531.9462.88
Prompt dropout72.1781.8717.2533.1060.86
Latent stochasticity58.4481.0917.0831.7184.81

TPA, early CFG reduction, and prompt dropout all improve Average ASR over TPA w/o text perturbation. Their consistent gains support our central view that temporarily adjusting early conditioning can benefit cross-model transfer. The tested latent-stochasticity control does not reproduce these improvements.

A plausible explanation is the different intervention location. Latent stochasticity changes the stochastic DDIM transition while keeping text embeddings and CFG fixed, so subsequent denoising continues to use the original conditioning. In contrast, TPA, CFG reduction, and prompt dropout modify the text embeddings or the conditioning weight, changing how the class condition guides early generation. The observed pattern supports our choice to intervene at the early conditioning level.

The reviewer’s suggestion prompted us to examine the latent-stochasticity control and further consider why interventions at different locations produce different outcomes. We appreciate the opportunity to deepen this analysis.

We will incorporate the detailed experimental results and discussion into the revised manuscript.

W3. Quality–transfer trade-off and FID-A evaluation

The reviewer noted that the quality–transfer trade-off is understated and that FID-A is computed against a 200-class ImageNet-A reference while the generated set covers 1,000 classes. To address these concerns, we conducted two additional comparisons: quality metrics at comparable Average ASR and class-balanced FID-A.

  1. Quality metrics at comparable Average ASR

We acknowledge that the original comparison showed higher attack success for TPA but better scores on some quality metrics for AdvClass, making their relative merits difficult to assess because of the quality–transfer trade-off. To assess whether TPA can generate more natural images at comparable attack success rates, we conducted an ASR-matched quality comparison.

To match TPA’s Average ASR with the RN50 surrogate, we adjusted only AdvClass’s adversarial guidance strength gsg_s, keeping its other generation settings unchanged.

MetricTPAASR-matched AdvClass
Average ASR ↑69.27%69.82%
HyperIQA ↑0.63400.3640
DBCNN ↑0.63200.4120
ARNIQA ↑0.70900.5630
MUSIQ-AVA ↑5.2684.361
NIMA-AVA ↑5.3034.450
MUSIQ-KonIQ ↑70.45754.547
TReS ↑80.06545.990
FID-VAL ↓31.03177.555
FID-A ↓66.60773.569
BRISQUE ↓17.93330.999

With an Average ASR difference of only 0.55%p, TPA performs better on all reported quality and distribution-distance metrics. These results provide additional evidence on the quality–transfer trade-off and strengthen the manuscript’s discussion of TPA’s ability to generate natural images while achieving high transfer ASR.

  1. Class-balanced FID-A comparison

For consistency with the comparison against NatADiff, our original evaluation used FID-A computed from the full 1,000-class generated set. In response to the reviewer’s concern, we additionally evaluated class-balanced FID-A to control for the difference in class composition between the generated set and the 200-class ImageNet-A reference.

MethodMean class-balanced FID-A across three reference samples ↓
TPA153.48
NatADiff161.33
TPA w/o text perturbation166.26
AdvClass169.02

TPA retains the lowest FID-A after controlling for class composition. Under this additional control, TPA’s generated-image distribution remains closer to the ImageNet-A reference distribution than those of the comparison methods, strengthening the original FID-A analysis. We will include the results of both additional experiments in the revised manuscript’s appendix.

Q1. Parameter selection and sensitivity

The reviewer requested a hyperparameter-selection protocol and sensitivity analysis. Following NatADiff’s classifier-guidance interval of 0<t<7000<t<700, we fixed the guidance onset at S=15S=15 across all surrogates (Appendix J, Table 7). Motivated by late targeted guidance in prior work, we used K=40K=40 as the reference switch setting and assessed KK and the late guidance strength gg by considering both surrogate ASR and image quality.

Below, we report parameter selection and sensitivity results for RN50 as a representative CNN surrogate.

Under the RN50 schedule S=15,K=40S=15,K=40, we progressively increased the late guidance strength to g=50,100,150g=50,100,150. The corresponding surrogate ASRs were 89.50%, 96.50%, and 99.00%. Among the tested strengths, g=100g=100 was the smallest value that achieved surrogate ASR comparable to the baselines, and we therefore selected it.

For the sensitivity analysis, we used 100 ImageNet classes and two shared seed slots per class, yielding 200 images per setting. Starting from (S,K,g)=(15,40,100)(S,K,g)=(15,40,100), we varied one parameter at a time while keeping the remaining generation and evaluation settings fixed.

Late guidance strength with S=15,K=40S=15,K=40 fixed:

Late guidance strength ggSurrogate ASR (%) ↑Average ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
5089.5062.3980.6218.54130.42196.69
100 (selected)96.5069.9480.6418.02135.02194.70
15099.0074.6180.4717.75136.97192.50

For onset and switch sensitivity, we fixed g=100g=100 and varied either SS or KK relative to the reference schedule S=15,K=40S=15,K=40.

Setting (g=100g=100 fixed)Surrogate ASR (%) ↑Average ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
S=15,K=40S=15,K=4096.5069.9480.6418.02135.02194.70
S=10,K=40S=10,K=4098.0070.8981.0318.40134.90194.21
S=20,K=40S=20,K=4096.5069.6780.6018.20133.41194.04
S=15,K=35S=15,K=3599.0077.8380.2517.51140.47192.34
S=15,K=45S=15,K=4589.5060.3980.4118.62128.46195.85

Average ASR remained within a narrow range of 69.67–70.89% when varying the onset. The switch time and late guidance strength had larger effects: starting targeted guidance earlier or increasing its strength raised ASR. These results characterize performance changes around the selected configuration.

We will provide detailed parameter-selection and sensitivity results for the Transformer surrogate in the appendix.

Q2. Comparison with DDIM50 and no retries

The reviewer pointed out the differences in sampling budgets and retry policies across methods.

In the main comparison, we retained AdvClass’s and NatADiff’s existing sampling procedures, using a 200-step DDIM schedule and adaptive retries rather than imposing reduced sampling budgets that could disadvantage these baselines. We agree that matching the sampling schedule and retry policy provides a more controlled comparison. We therefore conducted an additional experiment using a common 50-step DDIM schedule and exactly one generation attempt per output, without retries.

Using RN50 as the surrogate, we generated 2,000 images per method across all 1,000 ImageNet classes with two shared seed slots per class and evaluated every output. The initial guidance strengths were 500 for AdvClass and 50 for NatADiff.

MethodActual U-Net calls/image ↓Recorded time (s/image) ↓Average ASR (%) ↑TReS ↑BRISQUE ↓FID-VAL ↓FID-A ↓
AdvClass (g=500g=500)504.0946.0279.4716.0229.2972.27
NatADiff (g=50g=50)11024.7767.1375.0016.1831.9777.42
TPA5010.3870.1180.3517.5132.6267.73

TPA achieves the highest Average ASR, exceeding NatADiff by 2.97%p and AdvClass by 24.09%p. Compared with NatADiff, TPA combines higher attack success with approximately 58% shorter recorded generation time: 10.38 versus 24.77 seconds per image. NatADiff retains its time-travel procedure and therefore requires 110 actual U-Net evaluations, whereas TPA uses 50. AdvClass is faster in this setting but has substantially lower Average ASR.

Thus, after matching the DDIM schedule and retry policy, TPA retains its advantages over NatADiff in both attack success and recorded generation time. TPA also achieves the highest TReS and lowest FID-A in this comparison; the remaining quality metrics are reported alongside ASR. We will provide detailed ASR and quality results in the appendix.

Q3. Evidence for class-valid exploration and human recognition

The reviewer requested direct evidence that early conditioning perturbations broaden the explored class-valid region, along with comparisons against simpler diversity controls.

Comparisons against early CFG reduction, prompt dropout, and latent stochasticity, together with their interpretation, are presented in W2. These results provide evidence that early conditioning intervention benefits attack transferability.

To assess whether the generated images retain the requested class, we are conducting an additional human study. We will compare attack transferability across methods on images that participants recognize as belonging to the requested class.

Collecting responses from human participants requires additional time. We will provide the results once data collection and analysis are complete.

Q4. FID-A improvements and changes in other quality metrics

The reviewer asked why FID-A improves while FID-VAL and most NR-IQA metrics worsen, and requested class-balanced FID-A or additional class-validity analysis.

ImageNet-A contains natural adversarial examples—real-world, unmodified images that classifiers misclassify. The lower FID-A therefore indicates that TPA’s generated-image distribution is closer to this reference distribution of natural adversarial examples.

In contrast, visual features that favor attack success may be introduced or amplified during generation, increasing the distributional distance from ImageNet validation or reducing the quality assessed by NR-IQA metrics.

To provide additional evidence regarding naturalness, we report class-balanced FID-A in W3 and will provide the human-study results described in Q3 once the evaluation is complete.

Comments on the paper

1 comment

We present TPA, a training-free method for reference-free natural adversarial generation through early text-conditioning perturbations and progressive restoration. Reviewer VfaS highlighted the novelty and simplicity of the approach, while QFFg and 8Rbr commented positively on the matched ablations. All three reviews noted the reported improvements in cross-model transferability.

During the rebuttal, we conducted additional experiments and revised the manuscript to address the reviewers' concerns. Changes and additions are highlighted in blue in the revised manuscript.

  • Section 5.3 and Appendix H (Rev. QFFg, VfaS, 8Rbr): TPA, early CFG reduction, and prompt dropout improve transfer over the fixed-conditioning variant under matched attack settings, whereas the tested latent-stochasticity control does not. These results support our choice to intervene in early text conditioning.
  • Appendix I (Rev. VfaS, 8Rbr): With adversarial guidance held fixed, early perturbations achieve the highest Average ASR among the early, late, and full-window schedules.
  • Appendix F (Rev. QFFg, 8Rbr): We added a blinded, matched human evaluation across all three methods and two surrogates, measuring ASR on images recognized as the requested class.
  • Section 5.3 (Rev. QFFg): With a common 50-step DDIM schedule and no retries, TPA achieves higher Average ASR than both baselines.
  • Appendices K and L (Rev. 8Rbr): TPA outperforms both baselines on all additional surrogate–victim pairs and exceeds NatADiff across all four RN50 generation-seed slots, supporting the stability of the transfer gains.
  • Appendix N (Rev. VfaS): On Oxford-IIIT Pet, TPA achieves higher Average ASR than AdvClass with all three surrogates and NatADiff with two of three. It also achieves the highest scores on four learned image-quality metrics in all three settings.
  • Appendices J and M (Rev. QFFg): TPA retains the lowest class-balanced FID-A among the compared methods. At comparable Average ASR, it also scores better than AdvClass on all reported quality metrics.
  • Appendix E.2 (Rev. VfaS): We added image-quality comparisons and an ablation with matched target-selection timing to clarify the empirical rationale for the two-stage objective.
  • Appendix G (Rev. QFFg): We clarified how hyperparameters were selected and added sensitivity analyses for RN50 and ViT-H/14.
  • Section 2 (Rev. VfaS): We reorganized Related Work and clarified TPA's relationship to CADS and prior generative attacks.
  • Additional feature-space analysis (Rev. QFFg, VfaS): DINOv2 analysis shows increased feature diversity alongside improved Average ASR under early conditioning interventions, consistent with our exploration-based interpretation.

We believe these additional experiments and revisions provide substantive responses to the reviewers' concerns. TPA combines strong cross-model transfer with efficient generation, without additional model training. Our findings highlight early text-conditioning intervention as a promising direction for developing efficient and transferable natural adversarial attacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.