acceptodds
Under review as a conference paper at ICLR 2027

ECLIPSE: Orthogonal Subspace Projection for Triggerless Bias Injection in Diffusion Models

Abstract

The widespread adoption of modern multi-encoder text-to-image (T2I) diffusion models expands the attack surface for stealthy bias injection. This raises an important question in model security: In these complex, high-dimensional architectures, can attackers perform effective, triggerless bias injection while preserving structural fidelity? We introduce ECLIPSE, a framework for triggerless implicit bias injection. Unlike explicit backdoor attacks, implicit interventions avoid predefined triggers but can suffer from feature-level semantic entanglement, which causes conceptual bleeding. ECLIPSE addresses this challenge in two ways: (1) An encoder-specific Single-Stage Orthogonal Subspace Gradient Projection (OSGP) removes components of the accumulated LoRA gradients that align with dominant neutral feature directions before each optimizer update, concentrating optimization in their orthogonal complement. (2) LoRA-factor norm regularization limits excessive adapter changes while tuning a single target encoder, helping preserve non-target visual details without requiring a memory-intensive teacher model. Evaluations across the CLIP and T5-XXL text encoders of FLUX.1-schnell show that ECLIPSE achieves high manipulation success rates on fine-grained tasks, such as altering facial micro-expressions, as measured by a VLM judge calibrated against human annotations. It reduces interference between targeted semantics and non-target visual content while largely preserving the evaluated out-of-domain concepts, although limited changes remain. These findings highlight a potential security risk in the open-source generative supply chain and motivate architecture-aware defense mechanisms.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Text-to-Image Diffusion Models, Generative Model Security, Triggerless Bias Injection, Model Supply-Chain Attacks, Low-Rank Adaptation, Orthogonal Subspace Gradient Projection.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BRbpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework(ECLIPSE) for triggerless implicit bias injection. It inserts LoRA adapters into MLP blocks of a text encoder, constructs a neutral-subspace projector from final-layer representations, and projects accumulated LoRA gradients before each AdamW step. Experiments cover CLIP and T5-XXL, multiple generation architectures, and two attention-based detectors.

Critical strengths

The problem is timely and security-relevant and the paper presents a concrete, runnable engineering pipeline and evaluates it across two text encoders and several generative architectures.

Critical weaknesses
  1. The VLM judge was "calibrated against human annotations" is not supported by any reported calibration results. Since the core ASR numbers depend on this judge, the absence of calibration evidence leaves the central evaluation unverified.

  2. The matched alarm rates establish that ECLIPSE does not increase detector alarms at the evaluated operating points. However, without demonstrating that the reproduced detectors detect a known trigger-based attack in the same setup, it remains unclear how strongly these results support detector evasion

Questions for authors
  1. What exactly does the "Original" row in Table 1 represent? Is it a clean–clean control, or a baseline reference? If it is a clean–clean control, how does the 28% non-Same rate compare with the 0.05% corruption rate reported for the clean–clean control in Table 4? Please explain the discrepancy in these two baselines and how the 28% noise rate in Table 1 affects the interpretation of the reported ASR.

  2. Can the authors provide a positive control showing that the reproduced SET and DAA detectors successfully identify a known trigger-based attack under the same evaluation conditions?

  3. The paper repeatedly mentions human calibration. What are the specific methods and results of this human calibration?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BRbp

We sincerely thank the reviewer for these precise questions, which target the three evaluations we care most about. We address them in order.

Q1. The 28% non-Same rate in the "Original" row of Table 1 vs. the 0.05% corruption rate in Table 4

The "Original" row is a clean–clean control: two images generated from the unmodified model are compared by the VLM judge, which must output one of three relative-emotion labels. The 28% non-Same (18.2% more-positive + 9.8% more-negative) is therefore the judge's intrinsic classification noise on near-identical pairs — its noise floor. Table 4's corruption rate (0.30% in the revision; 0.05% in the submitted version), by contrast, comes from a fidelity check, not an emotion judge: each generated image is scored as Preserved / Corrupted / Unclear against a reference via CLIP similarity (the Unclear column happens to be zero). It measures whether an image is degraded, not which bias it expresses. The two numbers quantify different things — attack strength (emotion) versus image fidelity — and are not comparable.

This floor does not bias our conclusions: it is additive and direction-agnostic, so the attack effect is the excess ASR above the baseline. Subtracting the 28% floor yields net effects of +51.7pp (negative bias) and +70.8pp (positive bias), both highly significant. In the revision we will add a footnote to Table 1 clarifying that the "Original" row is a clean–clean judge control and report the net ASR explicitly.

Q2. A positive control for the SET and DAA detectors

We reproduced both detectors and evaluated them on a known trigger-based attack, Rickrolling (Struppek et al., ICCV 2023), whose non-Latin character trigger redirects generation to a hidden target, under the same evaluation setup:

DetectorTarget attackTPRClean FPR
SETRickrolling100% (500/500)8.2%
DAARickrolling100% (200/200)—

DAA reports detection on trigger prompts per its original protocol; SET additionally reports the clean false-positive rate. Both detectors reliably identify the known backdoor with a high true-positive rate (and, for SET, a low clean false-positive rate), confirming the reproductions are functional rather than degenerate. Since ECLIPSE keeps detector alarm rates at the matched clean baseline, its evasion is genuine stealth rather than an artifact of broken detectors.

Q3. The specific methods and results of human calibration

We collected judgments from 9 annotators over 20 paired images under a unified three-way protocol (more-positive / more-negative / unchanged), with the judge prompt calibrated toward facial micro-expressions (raising the clean-image specificity from 72% to 88%). Inter-annotator agreement is 71.0% pairwise with Fleiss' κ = 0.349, indicating reliable annotations. The judge attains 57.9% agreement with the majority vote (Cohen's κ = 0.176). The gap reflects a criterion difference: the judge attends to micro-expressions whereas humans judge overall appearance, making the judge more conservative and more likely to label subtle changes as "unchanged." Consequently, the absolute ASR is a conservative lower bound, while relative comparisons (e.g., with vs. without OSGP) are unaffected. The full protocol and agreement statistics will be added to Sec. 4.4 ,Table 5(Human calibration of the judge.)

I thank the authors for their detailed rebuttal and additional experiments. The newly added ablation studies and the positive control results for the SET/DAA detectors indeed address some of my previous concerns.

However, I still have significant reservations regarding the reliability of the core evaluation. The high ASR reported in the paper, as well as the relative comparisons between methods, rely heavily on the VLM judge. The newly added human validation covers only 20 image pairs with 9 annotators, which remains limited in scale. Moreover, the chance-corrected agreement between the VLM and the human majority vote is low (Cohen’s κ = 0.176). These results are insufficient to support the authors' claims that the "absolute ASR is a conservative lower bound" and that "relative comparisons are unaffected."

I also agree with orTc's concern that a sufficient distinction must be made between target semantic manipulation and non-target content preservation. The current evidence is insufficient to determine to what extent the outputs judged as successful by the VLM actually achieve the intended emotional shift while preserving non-target content. Therefore, I cannot establish sufficient confidence in the reported ASR and the core conclusions it supports.

While I acknowledge the value of the authors' additional work and the importance of the research question, I maintain my original score.

Reviewer_orTcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ECLIPSE, a triggerless bias-injection attack that modifies text encoders in diffusion models through LoRA adapters. It combines OSGP with LoRA-factor norm regularization to steer generated content toward attacker-specified attributes while limiting unintended visual changes.

Critical strengths

The paper investigates an important model-supply-chain security problem and demonstrates that modifying a single text encoder can induce targeted semantic biases without explicit input triggers. The evaluation covers CLIP and T5-XXL, compares ECLIPSE with existing attacks, and includes ablation studies, cross-architecture experiments, and evaluations against attention-based backdoor detectors.

Critical weaknesses
  1. The motivation for OSGP relies on the assumption that dominant principal directions extracted from neutral-prompt representations correspond to features that should be preserved. However, PCA identifies directions of high variance rather than semantically protected attributes, and these directions may also encode identity, demographics, or other meaningful variations. This assumption is particularly important because OSGP removes only one direction from T5-XXL's 4,096-dimensional representation space, yet Table 2 shows that the positive-bias ASR increases from 22.10% to 80.60%. Without comparisons against random-direction projection or alternative projector ranks, it remains unclear whether this improvement arises from the specific semantic properties of the selected neutral subspace or from gradient filtering more generally. (Sec. 3.2, Page 4; Table 2, Page 8)
  2. The paper relies on Qwen3-VL-30B-A3B-Instruct to evaluate attack success by classifying relative emotional changes between paired generated images. Although the abstract states that the VLM judge is calibrated against human annotations, the paper does not report the corresponding annotation protocol, sample size, inter-annotator agreement, or quantitative agreement between human and VLM judgments. Since the main results report ASRs as high as 94.10%, the absence of a documented validation procedure makes it difficult to determine how reliably these scores reflect the intended emotional changes rather than the judge's own classification behavior. (Abstract, Page 1; Sec. 4.1–4.2, Pages 6–7)
  3. The paper attributes reduced semantic interference to the combination of OSGP and LoRA-factor norm regularization, but their contributions to content preservation are not fully disentangled. Table 2 evaluates OSGP mainly through negative, positive, and unchanged generation rates, demonstrating improvements in target-direction steering rather than structural preservation. Figure 4 examines how the regularization weight affects ASR, CLIP-I, and SSIM, but does not provide corresponding preservation comparisons with and without OSGP. Consequently, the current ablations do not directly establish whether OSGP independently reduces non-target visual changes or primarily improves attack effectiveness. (Sec. 4.3, Page 8; Table 2, Page 8; Figure 4, Page 9)
  4. The generalization experiments do not fully establish how ECLIPSE behaves under diverse or conflicting prompt semantics. The main evaluation uses 1,000 portrait prompts with explicit emotion-related expressions filtered out, while Figure 3 extends the attack to occupational gender bias and Appendix C provides qualitative examples involving urban weather and illumination. Although these experiments demonstrate effectiveness beyond facial expressions, they do not evaluate whether the injected bias can override explicitly specified opposing attributes. In addition, the out-of-domain preservation experiment uses only the CLIP-targeted negative-bias checkpoint on 3,777 nonhuman prompts, leaving preservation behavior under other attack configurations insufficiently examined. (Sec. 4.1, Page 6; Figure 3, Page 8; Appendix B.2, Page 16; Appendix C, Page 17)
Questions for authors
  1. Table 2 shows that removing only one principal direction from T5-XXL's 4,096-dimensional representation space substantially improves attack success. Why does such a low-rank projection produce such a large effect? Could the authors compare the selected neutral PCA directions with random orthogonal directions, alternative principal components, and different projector ranks to clarify whether the improvement is specific to the proposed neutral-subspace construction? (Sec. 3.2, Page 4; Table 2, Page 8)
  2. How was the VLM judge calibrated against human annotations? Could the authors provide the human-evaluation protocol and report quantitative agreement between human judgments and VLM predictions, particularly for subtle emotional changes that may be difficult to classify consistently? (Abstract, Page 1; Sec. 4.1, Pages 6–7)
  3. Table 2 demonstrates the contribution of OSGP to attack success, while Figure 4 mainly evaluates the effect of norm regularization on visual preservation. Could the authors report CLIP-I, SSIM, and attribute-level preservation measurements with and without OSGP under otherwise matched training conditions? This would clarify whether OSGP independently reduces semantic interference rather than only improving target-direction steering. (Sec. 4.3, Pages 8–9)
  4. How does ECLIPSE behave when the original prompt explicitly specifies an attribute that conflicts with the attack target, such as an angry facial expression under positive-bias injection? Could the authors evaluate such conflicting prompts and extend the out-of-domain preservation analysis to additional encoder and bias configurations? (Sec. 4.1, Page 6; Appendix B.2, Page 16)
1 comment
AuthorsOpenReview ↗

Response to Reviewer orTc

We sincerely thank the reviewer for these thorough questions, which target the core assumptions behind OSGP and the reliability of our evaluation. We address each in turn.

Q1. Why a low-rank projection produces such a large effect, and its specificity

We must first correct a typo in Sec. 3.2: the text stated k=1k=1, but the actual operating point is k=256k=256 for T5-XXL (and k=3k=3 for CLIP). We apologize for the confusion and will fix it in the revision. The reviewer's concern about a "single" direction thus stems from this error; we in fact project out 256 neutral principal directions (6.25% of T5-XXL's 4,096 dimensions).

On the substance, the revision reports two ablations. Table 3 sweeps the rank kk from 0 (no projection) to 1024: ASR rises sharply from 30.3% at k=0k=0 to 69.9% at k=256k=256 and then saturates (71.2% / 72.1% at k=512/1024k=512/1024), with fidelity degrading monotonically, confirming k=256k=256 as the operating point. Table 4 tests whether the effect is specific to the top-kk neutral directions by replacing them with bottom-kk or random directions: top-kk reaches 69.9% ASR with only 3.1% opposite-direction drift, versus 47.0% (bottom-kk) and 44.7% (random). The improvement is therefore specific to removing the neutral principal subspace, not to projecting out an arbitrary set of directions.

Q2. VLM judge calibration

The full protocol and statistics are now reported in Table 5 (Sec. 4.4). In brief, we calibrated the Qwen3-VL judge prompt toward facial micro-expressions, raising clean-image specificity from 72% to 88%; we then validated against 9 annotators over 20 paired images. Inter-annotator agreement is 71.0% pairwise (Fleiss' κ\kappa = 0.349), indicating reliable labels. The judge agrees with the majority vote on 57.9% (Cohen's κ\kappa = 0.176). The gap reflects a criterion difference — the judge attends to micro-expressions while humans judge overall appearance — making the judge more conservative, so the reported absolute ASR is a conservative lower bound, while relative comparisons (e.g., with vs. without OSGP) are unaffected.

Q3. Disentangling OSGP from the norm regularizer

We agree the two contributions were not fully disentangled, and have added a full with/without-OSGP comparison under identical norm regularization in Table 2 (Sec. 4.3). On CLIP, OSGP yields a Pareto improvement: ASR rises from 55.7% to 75.3% while CLIP-I increases from 0.941 to 0.946 — OSGP simultaneously strengthens the attack and preserves content, which the norm regularizer alone cannot do. On T5, the without-OSGP variant shows slightly higher fidelity only because its attack is far weaker (30.3% vs. 69.9% ASR); a barely-changed image trivially preserves structure. OSGP therefore independently improves directional steering and content preservation.

Q4. Conflicting prompts and out-of-domain generalization

(a) Conflicting prompts. We evaluated prompts that explicitly specify an attribute opposing the injected bias. Because the injected shift is a prompt-agnostic, bounded global offset, a strong explicit attribute is only partially — not fully — overridden:

PromptNegativePositiveSame
Agree (negative bias)96.9%0.1%3.0%
Conflict — "angry" (positive bias)68.4%27.3%4.3%
Agree (positive bias)3.8%62.5%33.7%
Conflict — "smiling" (negative bias)16.7%10.4%72.9%

The target-direction rate drops from 96.9% (negative agree) / 62.5% (positive agree) to 27.3% (positive bias vs. "angry") / 16.7% (negative bias vs. "smiling"). In the two conflict settings, the opposing direction appears in 68.4% ("angry") and 10.4% ("smiling", with a further 72.9% judged Same, i.e., not steered toward the injected bias). ECLIPSE thus exerts a subtle, bounded, distribution-level shift rather than an unconditional override — precisely the threat model we intended.

(b) Out-of-domain preservation. We extended the OOD analysis from the single CLIP negative-bias checkpoint to all four {T5,CLIP}×{negative,positive}\{\text{T5}, \text{CLIP}\} \times \{\text{negative}, \text{positive}\} configurations on the 3,777 non-human prompts. The Preserved rate stays at 93.4%–94.5% (vs. 99.70% for the clean model) with CLIP image similarity 0.943–0.961 across all four settings, showing that OOD preservation is a general property of ECLIPSE rather than an artifact of one checkpoint.

Reviewer_uRKwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ECLIPSE, a framework for triggerless bias injection in text-to-image diffusion models. It trains LoRA adapters in either CLIP or T5-XXL by combining alignment to a shared bias target, norm regularization, and gradient projection away from dominant neutral feature directions. Experiments on FLUX.1-schnell and other architectures show effective attribute manipulation while largely preserving non-target content. Further evaluations assess out-of-domain preservation and reveal limited discrimination by two attention-based detectors at the tested thresholds.

Critical strengths

1.A well-motivated security problem. The paper highlights how modifying a single text encoder can introduce systematic biases without explicit triggers, exposing a vulnerability in the T2I model supply chain.

2.A lightweight, text-only approach. Training updates only LoRA parameters in one encoder, using neutral prompts and a text-derived bias target while keeping the remaining model frozen.

3.Strong main results. ECLIPSE improves attack success over both baselines across all four FLUX settings. Experiments on additional architectures and component ablations provide further empirical support.

Critical weaknesses

1.The shared alignment target lacks a prompt-specific semantic anchor. The target (e_{tgt}=2f_{bias}-f_{neu}) is computed from a fixed prompt pair and shared across all training inputs. Aligning every prompt with the same target encourages convergence toward a common representation, without explicitly preserving prompt-specific semantics. The content degradation observed without norm regularization is consistent with this concern.

2.OSGP’s core design motivation is not directly validated. OSGP removes gradient components aligned with dominant neutral feature directions to reduce interference with non-target semantics. However, its ablation reports only attack-related outcomes, without comparing content preservation with and without OSGP. Improved attack success does not establish that the projection protects neutral semantics or reduces semantic entanglement.

3.Several design choices remain underexplained. The paper provides limited explanation or sensitivity analysis for the target prompt pair, adapter placement, and number of removed principal components. These choices may materially affect the attack–preservation trade-off.

Questions for authors

1.What motivates aligning all prompts with a shared absolute target? How would it compare with a prompt-specific target hi0+2(fbias−fneu)h_i^0+2(f_{bias}-f_{neu}), where hi0h_i^0 is the clean pooled representation, at comparable attack success rates?

2.Why should principal directions estimated from final-layer features provide appropriate constraints for gradients across different MLP blocks? Does this interpretation remain valid after AdamW rescales the projected gradients? Please also report preservation metrics for the OSGP ablation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer uRKw

We sincerely thank the reviewer for the careful reading and these precise, constructive questions, which target the two design choices we care most about. We address Q1 and Q2 in turn.

Q1 — Shared absolute target vs. prompt-specific target

The shared target follows directly from our threat model: ECLIPSE must ship a static adapter that injects bias into arbitrary unseen prompts (trained on 1.8K CelebA prompts, evaluated on 1K held-out FairFace and 3.8K out-of-domain COCO prompts), so the effect must be a prompt-agnostic, distribution-level shift. A shared absolute target encodes exactly this, whereas a prompt-specific target optimizes only the prompts it is defined on and cannot generalize the shift. Realizability points the same way: a rank-limited adapter inside an MLP block implements a near-constant offset in the residual stream, not an input-dependent mapping.

We trained the proposed prompt-specific target etgti=ecleani+(fbias−fneu)e_{\mathrm{tgt}}^i = e_{\mathrm{clean}}^i + (f_{\mathrm{bias}}-f_{\mathrm{neu}}) under identical settings on T5-XXL:

BiasTargetNegPosSameFID↓Pick↑CLIP-I↑SSIM↑
NegativeShared (ours)69.9%3.1%27.0%18.8422.720.9380.792
NegativePrompt-specific57.7%25.5%16.8%15.8922.940.9570.822
PositiveShared (ours)11.4%80.6%8.0%15.7723.130.9600.831
PositivePrompt-specific11.0%70.2%18.8%12.8823.070.9720.869
Original (clean)—18.2%9.8%72.0%—20.731.0001.000

Two observations follow. First, the shared target is markedly stronger: +12.2 pp (negative) and +10.4 pp (positive) attack success. Second — and more telling — the prompt-specific variant is less targeted: under negative bias it drifts 25.5% of samples to the opposite (positive) attribute, versus 3.1% for the shared target, because ecleanie_{\mathrm{clean}}^i carries prompt-specific content that dilutes the bias direction. The prompt-specific variant does improve fidelity (CLIP-I +0.019/+0.012, SSIM +0.030/+0.038), but the shared target already preserves content well (CLIP-I 0.94–0.96, SSIM 0.79–0.83) while delivering a stronger, cleaner attack.

Q2 — Cross-layer reuse, AdamW rescaling, and OSGP preservation metrics

(a) We do not assume a layer-invariant PCA basis. The residual stream has fixed width, so every adapted block writes on the same axes and the injected offset is a bounded, near-constant low-rank shift; the dominant neutral direction from final-layer features is thus representative across depth. Table 4 (Direction of the projected subspace) confirms this: excising the top-kk directions yields 69.9% negative ASR with only 3.1% opposite drift, whereas bottom-kk and random directions degrade to 47.0% and 44.7% — OSGP's benefit comes from removing the specific neutral subspace, not an arbitrary one.

(b) The reviewer is right that orthogonality is not preserved verbatim: OSGP applies P=I−UkUk⊤P = I - U_k U_k^\top while AdamW applies a diagonal preconditioner, and the two do not commute. We have revised the wording from guarantee to suppression and quantify the residual. Measuring ρ=∥UU⊤ΔW∥F/∥ΔW∥F\rho = \|U U^\top \Delta W\|_F / \|\Delta W\|_F, ρ\rho drops from 41.7% to 18.6% at k=256k{=}256 (60.2% →\to 25.9% at k=1024k{=}1024; full rank sweep in Table 3), a $\sim$2.2–2.3$\times$ reduction throughout training. The leakage comes solely from the preconditioner; a post-optimizer projection in the revision will remove it at no inference cost.

(c) Table 2 (Ablation of OSGP for directional steering) reports CLIP-I and SSIM with and without OSGP. On CLIP, OSGP is a Pareto improvement — ASR 55.7% →\to 75.3%, CLIP-I 0.941 →\to 0.946, SSIM 0.782 →\to 0.801. On T5-XXL, the no-OSGP variant is only marginally more faithful (CLIP-I 0.961 vs. 0.938) because it attacks far more weakly (30.3% vs. 69.9%). OSGP improves directionality rather than trading fidelity for attack success.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.