acceptodds
Under review as a conference paper at ICLR 2027

Let the Carrier Carry the Attack: Preserving the Subject in Adversarial Image Generation

12 views

Abstract

Strong unrestricted adversarial attacks can distort the primary object of an image, hereafter referred to as the *subject*. To preserve subject integrity without compromising attack magnitude, we introduce the **carrier**: a secondary visual element that provides an auxiliary region to facilitate the attack under global classifier guidance. We demonstrate three key findings: 1. A carrier mitigates subject distortion by absorbing a larger share of globally normalized attack updates. 2. A carrier improves cross-model transferability, governed by the strength of target-related features that balance semantic separation and transfer performance. 3. Successful targeted attacks retain the personalized subject as the primary content perceived by humans while successfully misleading the classifier. Our results demonstrate that a visually secondary carrier offers an auxiliary spatial pathway for adversarial changes, enabling strong and transferable attacks while improving subject preservation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attack, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XuNoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses subject fidelity in unrestricted targeted adversarial generation by introducing a carrier: a visually secondary object placed in the background, outside the primary subject, to provide an additional spatial pathway for attack-related modifications. Built on FLUX.1-Kontext-dev with subject-specific LoRA adaptation and SAM3-based localisation, the framework applies globally RMS-normalised classifier guidance. Three implementations—CRA, CIRA, and JIA—are evaluated under three carrier conditions across 600 source–target pairs. Reported results include white-box Top-1 success rates of up to 99.67%, black-box Top-5 success rates of up to 55.07%, and DINOv3 similarity scores of 0.9799–0.9960. The paper also presents a blinded human study covering 5,400 images, reporting a result of 99.96%, and a carrier-whitening intervention under which 52.8% of initially successful attacks lose their target Top-1 prediction.

Critical strengths
  1. An informative intervention-based analysis of the carrier’s contribution. On 1,163 localisable CRA composites, whitening only the unoccluded carrier region while leaving the primary subject unchanged reduces the target logit in 98.3% of samples. The removed area correlates strongly with the logit reduction (Spearman’s ρ = 0.774, p < 0.001), and 52.8% of initially successful attacks lose their target Top-1 prediction. These results provide concrete evidence that the carrier region contributes substantially to the target prediction, beyond what aggregate attack success rates alone can establish.
  2. Substantial experimental scale and internally consistent reporting. The evaluation comprises 600 source–target pairs, 1,800 carrier-controlled cases, 5,400 attacked images, and three annotations per image. My recomputation confirms all verifiable numerical relationships, supporting the internal consistency of the reported results.
Critical weaknesses
  1. The Target Carrier condition introduces a fundamental ambiguity in the adversarial interpretation. Under this condition, an instance of the target class is explicitly inserted into the image. Consequently, some of the reported attack success and cross-model transferability may reflect correct recognition of a genuinely present target object rather than classifier deception. For example, if an image contains both a dog and a castle, predicting “castle” is not necessarily an error. Conditional ASR excludes cases in which the target class is already recognised before the attack, but it does not by itself distinguish adversarial misclassification from correct recognition of the constructed carrier. The whitening results establish the carrier’s importance to the prediction, but do not resolve this distinction: removing a legitimate target-class cue could produce the same effect.
  2. The human study measures primary-subject recognisability rather than attack unobtrusiveness. The prompt template in Appendix B.2 explicitly requests a carrier “in the distant background, slightly blurred by shallow depth of field,” encouraging a small, distant, and defocused object. Participants are then asked to select the “most relevant option describing the primary subject,” with only the subject label corresponding to a salient foreground instance. The per-annotator rates of 99.74%–99.93% suggest that this is a near-ceiling recognition task. The study therefore supports the narrower conclusion that the intended primary subject remains recognisable, but does not establish that the carrier is unobtrusive or that the resulting image appears unedited.
Questions for authors
  1. When the carrier explicitly depicts the target class, what evidence distinguishes classifier deception from correct recognition of an inserted object? In particular, the finding that carrier whitening causes 52.8% of initially successful attacks to fail appears compatible with either explanation. Could the authors collect human judgments of whether the target label validly describes an object present in the image, alongside judgments of which class a classifier would be expected to predict?
1 comment
AuthorsOpenReview ↗

Response to Reviewer: Clarifications on Whitening, Carrier Optimization, and Visual Naturalness

Thank you for your careful review and for recognizing our intervention analysis and experimental consistency. Your comments helped us clarify the interpretation of the carrier experiments and extend our evaluation of visual naturalness.

Weakness 1 & Question 1: Target-class content and the interpretation of whitening

Thank you for raising this distinction. Our setting allows a visible secondary carrier while requiring the personalized subject to remain the primary image content. We agree that, for Target Carriers, target-label membership and original post-attack whitening alone do not establish that the target label is semantically invalid.

However, our original whitening experiment also includes Non-Target Carriers, which do not belong to the target class (Appendix J.3). Removing these carriers causes 51.0% of initially successful attacks to lose the target Top-1 prediction, comparable to 54.6% for Target Carriers. For example, when the target is ambulance and the carrier is an ordinary minivan, the intervention removes a minivan rather than an actual ambulance. Thus, the observed effect is not limited to removing a genuinely present target-class object.

Motivated by your concern, we added pre-attack whitening followed by clean-carrier restoration (Appendix J.4). We whiten the carrier region before optimization and restore its original clean pixels afterward. The final image therefore contains the clean carrier without retaining the changes produced inside that region during the attack. Nevertheless, standard-strength attack success remains below that of the original carrier attacks. Under strong guidance, ASR remains high at 97.96%, but subject DINO is 0.8160, compared with 0.9345 in the original Target-Carrier condition. In other words, restoring the carrier afterward does not recover the subject-preservation benefit obtained when its original content is available during optimization. These results support the carrier’s role during optimization and are consistent with our normalized update-allocation mechanism.

Weakness 2: Subject recognition and visual naturalness

Thank you for pointing out this difference. Our setting does not require the carrier to be hidden or imperceptible. It should remain visible but visually secondary, with the personalized subject remaining the primary image content. Our original human study evaluates this objective, rather than whether the image appears unedited.

Even when an observer knows the original image and notices the added carrier, a semantically plausible background object may still appear more natural than conspicuous distortions. In our NatADiff adaptation, target-related features may emerge on the subject. Our carrier instead provides a designated secondary region for such changes, reducing their impact on the subject. For the strong No-Carrier attacks baseline, it may also produce conspicuous unnatural noise and patterns that seem unnatural for the human observer.

Following your suggestion, we completed a comparative human study to assess how natural the outputs look to observers. Annotators judged outputs from all three carrier conditions to look less unnatural or artificially manipulated than the strong No-Carrier baseline. Together with the original subject-recognition study, this suggests that the carrier can remain visible and visually secondary while the personalized subject stays dominant and the overall image looks more natural. Please see Appendix L.2 for the study setup and full results.


Thank you again for helping us clarify these distinctions and strengthen our evaluation.

Reviewer_RQawOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a carrier-based approach to targeted adversarial image generation. By adding a secondary object to the background, the method aims to reduce distortion of the original subject while improving attack transferability. The authors provide theoretical analyses of these effects and evaluate three attack strategies using different carrier types. Experiments show improved subject preservation and cross-model transferability, particularly when the carrier is closely related to the target class.

Critical strengths

(1) Introducing a carrier to preserve the primary subject during adversarial image generation is an interesting approach.

(2) The proposed methods achieve substantial improvements in black-box transferability while maintaining high subject-preservation scores.

(3) The paper evaluates multiple carrier types and attack strategies, supported by ablation studies and human evaluation.

Critical weaknesses

(1) I appreciate the interesting idea of introducing a carrier, but I have concerns about the practical relevance of the proposed attack setting. While unrestricted adversarial attacks allow substantial visual changes, the freedom to introduce entirely new objects, particularly Hybrid Carriers and Target Carriers, raises questions about what constitutes a genuine adversarial misclassification. For example, if the target class is “ambulance” and an ambulance is directly inserted into the image, predicting “ambulance” may be a reasonable classification rather than an adversarial failure. Although the authors emphasize preserving the original subject, I do not find this sufficient to address my concern. I therefore question whether the proposed setting adequately distinguishes adversarial effects from legitimate changes in image semantics, and whether it reflects a meaningful real-world threat model.

(2) I find the claimed contribution regarding cross-model transferability somewhat unconvincing. It seems intuitive that adding objects with stronger target-related characteristics would make classifiers more likely to predict the target class. I therefore question whether this observation represents a meaningful research contribution or simply reflects the increased presence of target-class semantic information in the image.

(3) The definitions of Non-Target, Hybrid, and Target Carriers seem somewhat ambiguous. For example, in Figure 1, where the target class is “ambulance,” the Non-Target Carrier is a large van that already shares substantial visual characteristics with an ambulance. It is therefore unclear why this object is classified as a Non-Target Carrier rather than a Hybrid Carrier.

(4) The paper does not sufficiently explain why introducing a new carrier is preferable to directly modifying the existing background [1]. In fact, I would argue that modifying the existing background could be more stealthy and offer greater flexibility for adversarial optimization, without introducing additional objects that may alter the image semantics.

[1] Objectcompose: Evaluating resilience of vision-based models on object-to-background compositional changes, ACCV 2024.

Questions for authors

(1) The authors should further justify the practical relevance of this attack setting and clarify why predicting the target class should be considered a genuine adversarial misclassification when a Target Carrier is explicitly inserted into the image, even if the original subject remains dominant.

(2) The authors should further clarify the novelty and significance of this claimed contribution.

(3) The authors should provide clear and rigorous definitions of Non-Target, Hybrid, and Target Carriers, as the distinction between these categories is crucial to the paper's claims and experimental conclusions.

(4) The authors should further justify the necessity and advantages of introducing a new carrier over directly modifying the existing background, particularly in terms of stealthiness, optimization flexibility, and preservation of the original image semantics.

2 comments
AuthorsOpenReview ↗

Response to Reviewer: Clarifications on Attack Validity, Novelty, Carrier Definitions, and Background-Only Optimization

Thank you for your thoughtful review and for recognizing the carrier idea and its subject-preservation benefits. Your comments helped us clarify the setting and definitions, and motivated additional experiments. We address each weakness and its corresponding question below.

1. Practical relevance and genuine adversarial misclassification (W1/Q1).

Our attack objective is to use adversarial classifier guidance to make classification rely more on carrier region while preserving the original subject’s appearance and visual dominance. The corresponding threat setting is an attacker controlling secondary background content in an image presented to a classifier, while retaining the personalized subject as the primary content. We study whether this secondary content can redirect model decisions away from that subject.

We recognize that inserting an ambulance and obtaining an “ambulance” prediction alone does not establish adversarial misclassification. We therefore distinguish construction from optimization: Our white-box conditional ASR remains high after excluding samples whose clean Top-5 predictions already contain the target, showing that object insertion alone does not account for attack success (Appendix A.5). In our ablation experiments, restoring the attacked carrier region to its clean counterpart reduces target predictions despite retaining the carrier object, supporting the contribution of adversarial changes within that region. Our human evaluations show that the original subject remains visually dominant and that the carrier-based outputs appear less visibly manipulated than the strong No-Carrier baseline in the configurations tested (Appendices L.1 and L.2).

2. Novelty and significance of transferability (W2/Q2).

We evaluate whether the optimization-induced prediction shift transfers to other models without further optimization. Our black-box conditional ASR shows that target predictions newly induced by white-box optimization transfer to other models without further optimization, even after excluding their pre-existing target predictions. Thus, the transfer results cannot be explained solely by carrier construction (Appendix G.1, Table 6).

We agree that stronger target-related features can facilitate transfer. Our contribution is to investigate how a spatially separate carrier allows these benefits to coexist with personalized-subject preservation, rather than presenting that intuition alone as the novelty. We examine this across carrier conditions and attack routes, alongside our update-allocation analysis and matched strong-attack preservation results.

Target-related features also support transfer in NatADiff baseline, but their location is not explicitly controlled, potentially altering the primary subject and causing the unnatural looking of image. Our carrier provides a secondary region for such changes.Our NatADiff comparison illustrates how a separate carrier region helps preserve the subject’s appearance (Appendix K.3). The additional human evaluation complements this comparison by showing that the carrier-based outputs also appear less visibly manipulated than the strong No-Carrier baseline (Appendix L.2).


The remaining responses are provided in the follow-up comment below.

3. Rigorous definitions and the minivan example (W3/Q3).

Thank you again for identifying this ambiguity. The distinction concerns carrier class identity and explicitly introduced target-specific attributes, rather than the absence of visual similarity (Appendix A.2). Non-Target Carriers belong to a different class from the target but may share general shape or structure. Hybrid Carriers retain a non-target identity while incorporating target-specific attributes; Target Carriers directly instantiate the target class.

For “ambulance,” an ordinary minivan is Non-Target; a minivan with ambulance-like red-and-white stripes is Hybrid; an ambulance is Target. Shared vehicle structure alone does not make a minivan Hybrid. Following your suggestion, we clarified these definitions in the Introduction. Hybrid denotes the requested construction condition; its attributes are not consistently realized by CIRA/JIA, as disclosed in Appendix B.5.

4. Necessity and advantages over background-only modification (W4/Q4).

Following your suggestion, we added a targeted ObjectCompose adaptation (Appendix K.6). ObjectCompose originally uses an untargeted objective; we replace it with targeted cross-entropy. Targeted attacks may require a larger attack budget to succeed than untargeted attacks.

The carrier is itself part of the background and provides a recognizable secondary object whose features can be exploited by classifier guidance, offering an alternative to modifying the primary subject. Background-only optimization offers spatial freedom while protecting the foreground, but our adaptation shows that this freedom does not necessarily produce effective or visually natural targeted attack images: it preserves the subject yet introduces extensive high-frequency, noise-like background artifacts, with limited transferability. Stronger guidance intensifies distortion without improving attack success.

Applying ObjectCompose globally would remove the background-only constraint for ObjectCompose that protects the subject. In contrast, our carrier provides a secondary region for adversarial changes under global guidance, reducing subject distortion without explicitly excluding the subject from optimization.


Thank you again for helping us clarify these distinctions and strengthen the comparison.

Reviewer_Uxt5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies unrestricted targeted adversarial generation with an emphasis on preserving a specific foreground subject. It uses FLUX with subject-specific LoRA conditioning, introduces a visually secondary carrier outside the subject, and applies classifier guidance during reconstruction or inpainting. Three carrier conditions and three construction/attack routes are evaluated. The experiments report target-class prediction rates, cross-model transfer, subject-preservation metrics, and human judgments of the primary subject. The approach is generative rather than an attack constrained to a small pixel-space Lp ball.

Critical strengths
  1. Preserving a particular subject under strong targeted guidance is a worthwhile objective. Table 2 reports 100% white-box target-Top-1 rates for No-Carrier, Target-CRA, and Target-CIRA, with DINO similarities of 0.7928, 0.9345, and 0.9809. These results support a preservation benefit under the evaluated settings, subject to the label-validity concern below.

  2. I appreciate the clean-prediction conditional ASR checks and the human study covering 5,400 carrier outputs. The 5,398 majority-vote selections of the original subject provide substantial evidence that it remains perceptually dominant. These are useful controls, although they do not fully resolve whether the target label is incorrect.

Critical weaknesses

The strongest transfer results have a central label-validity ambiguity. Target Carrier deliberately places an instance of the target category in the image, while black-box success is primarily target membership in Top-5. If a recognizable castle is present behind a teapot, predicting castle may reflect recognition of real image content. I do not consider Top-5 target membership alone sufficient evidence of a successful targeted adversarial attack in this setting. Humans choosing the teapot as the primary subject does not establish that castle is an invalid image label. Likewise, excluding clean-model target hits shows a change in model prediction, not that the new prediction is semantically wrong.

Questions for authors

Can you compare against a representative compatible spatial/background generative attack, such as an appropriate adaptation of ObjectCompose, and a construction-matched background control without the proposed carrier? Please align subject conditioning, source/reference images where applicable, guidance budgets, and construction-selection effort, and disclose necessary differences. Showing source -> constructed clean -> attacked images, with preservation measured across both stages and detection-failure rates retained, would clarify Figure 1 and establish whether the benefit comes specifically from the carrier.

I am currently leaning toward rejection because the strongest target carrier results are difficult to interpret as adversarial errors, and the available baselines do not sufficiently establish the carrier's added value. I would reconsider and raise my score if the authors resolve the labeling issue with convincing evidence and demonstrate a meaningful advantage under appropriately controlled comparisons.

BTW, In Figure 1, why are the images labeled "Clean" in the third row different from those in the first row? Does the third row already include the carrier after background inpainting, before adversarial guidance is applied?

2 comments
AuthorsOpenReview ↗

Response to Reviewer: Clarifications and Additional Experiments on Carrier Transferability and Subject Preservation

Thank you for your thoughtful review. Your suggestions led us to add a targeted ObjectCompose comparison, a carrier-free background construction control, and an evaluation of subject preservation across the construction and attack stages. We also added examples showing all three images and clarified the two “Clean” rows in Figure 1.

Weakness — Interpretation of Target-Carrier transferability

We agree that “castle” can be a valid label for a visible background object even when the teapot remains the primary subject. Our attack objective is whether classifier guidance shifts predictions toward a designated target associated with a visually secondary carrier while preserving the personalized subject’s appearance and visual dominance. We measure the target prediction on the guided classifier and evaluate whether it transfers to other classifiers without further optimization. Appendix G reports how often the target enters each black-box model’s Top-5 after guidance among images where it was absent from that model’s clean Top-5. For Target Carriers, this measures a shift toward a visible secondary object. Its label can still be semantically valid.

NatADiff also uses target-related features to obtain strong transferability. In our adaptation, those features can emerge on the primary subject and alter its appearance (Appendix K.3). Our carrier provides a spatially separate, visually secondary region for these changes. The existing human evaluation shows that the original subject remains visually dominant (Appendix L). Our follow-up human study shows that annotators found the carrier-based outputs less visibly manipulated than the strong No-Carrier baseline. Together with the primary-subject recognition study, this suggests that the carrier-based outputs preserve the subject’s visual dominance while maintaining a more natural appearance in the evaluated settings. The study protocol and full results are provided in Appendix L.2 (Table 17).

Question 1 — ObjectCompose comparison

Following your suggestion, we added a targeted ObjectCompose adaptation in Appendix K.6. Both methods use the same personalized source images, target classes, evaluation classifiers, and targeted optimization setup. ObjectCompose restricts updates to the background through an explicit mask, while our method applies guidance globally across the entire image, including the primary subject.

ObjectCompose’s mask preserves the foreground, but the attacked images exhibit dense, high-frequency, snow-like artifacts across much of the background. Stronger guidance makes these artifacts more pronounced while lowering targeted attack success. Our carrier provides a structured secondary region for attack-related changes, allowing us to preserve the personalized subject even when the entire image remains open to optimization.

Question 2 — Carrier-free background construction control

According to your suggestion, we added a carrier-free background construction control for CRA, CIRA, and JIA in Appendix K.7. Each route retains its original construction procedure, subject-specific LoRA, and guidance schedule. The control replaces carrier prompts with natural background descriptions and omits the carrier-presence gate.

With the same construction procedure, carrier-free CRA and CIRA still reach full white-box success under strong guidance, but the personalized subject degrades substantially.

The carrier improves preservation of the personalized subject under both ordinary and strong guidance, with the largest gains for CRA and CIRA under strong guidance. This benefit is particularly relevant for examples that are harder to attack, where stronger guidance may be needed to induce the target prediction. In these cases, the carrier helps preserve the subject’s appearance even at higher guidance strengths.


The remaining responses are provided in the follow-up comment below.

Question 3 — Subject preservation across construction and attack

Your suggestion helped us identify when the carrier contributes to subject preservation. Figures 13 and 14 show the source, constructed clean, and attacked images for the carrier-free and Target-Carrier examples, respectively. The accompanying tables report subject similarity and detection outcomes at each stage. Detection failures remain in the attack-success denominator, and similarity scores are reported alongside their valid-pair counts.

In both the carrier-free and carrier-based CRA and CIRA experiments, subject similarity remains high after background reconstruction, and SAM3 detects all subjects. Under strong attack guidance, subjects in the carrier-free controls can become distorted enough that SAM3 fails to detect them. The carrier-based results retain higher subject similarity, and SAM3 detects all subjects after the attack. This indicates that the carrier’s contribution to subject preservation arises primarily during the attack stage, after background reconstruction.

Question 4 — The two “Clean” rows in Figure 1

The first row shows the clean source images for the No-Carrier baseline. The third row shows the clean images after background inpainting with a carrier and before adversarial guidance. Each “Attacked” row corresponds to the “Clean” row immediately above it. We revised the Figure 1 caption to make this distinction explicit.


Thank you again for helping us clarify these distinctions and strengthen the comparison.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

Summary of Revisions and Additional Experiments

We sincerely thank all reviewers for their constructive feedback and thoughtful suggestions. Their comments have helped us clarify the scope of our contributions and substantially strengthen the experimental evaluation.

In response to the reviews, we have revised the manuscript and conducted several additional experiments:

  1. ObjectCompose baseline comparison (Appendix K.6): We added a targeted adaptation of ObjectCompose to compare carrier-based global guidance with background-only optimization, examining attack effectiveness, subject preservation, and visual artifacts.

  2. Carrier-free ablation controls (Appendix K.7): We introduced construction-matched carrier-free controls across CRA, CIRA, and JIA. Additional stage-wise evaluations distinguish subject changes during background construction from those caused by adversarial guidance.

  3. Carrier whitening intervention analysis (Appendix J.4): We added pre-attack carrier whitening followed by clean-carrier restoration to further investigate the carrier's role during optimization, beyond its presence in the final image.

  4. Human evaluation (Appendix L.2): We conducted an additional comparative human study assessing perceived visual naturalness, complementing our original primary-subject recognition evaluation.

We also clarified the definitions of the three carrier conditions, the interpretation of target predictions in the presence of Target Carriers, and the construction stages illustrated in Figure 1.

Together, these additions provide further evidence for the carrier's contribution to preserving the personalized subject under adversarial guidance, while clarifying the scope and interpretation of our transferability results.

We greatly appreciate the reviewers' time and consideration and welcome further discussion or clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.