acceptodds
Under review as a conference paper at ICLR 2027

IDOff: Stealthy Adversarial Attack on Tuning-Free Personalization

11 views

Abstract

Tuning-free personalization enables high-fidelity identity reproduction from only a few reference portraits, raising concerns about unauthorized identity synthesis. Existing attacks can disrupt identity reproduction at the cost of degraded facial quality and prompt adherence, leaving visible signs of intervention. To address this limitation, we introduce IDOff, a stealthy adversarial attack framework that disrupts identity reproduction while preserving generation quality and prompt consistency. Our key insight is that identity carriers, the intermediate representations conveying reference identity to the generator, serve as an exploitable surface for such stealthy attacks. To this end, IDOff optimizes bounded reference-image perturbations to redirect these carriers toward identity-agnostic anchors derived from generic-person text embeddings or a population prototype. Extensive evaluations demonstrate that IDOff achieves substantial identity disruption while maintaining generation quality comparable to the no-attack baseline. An ablation study shows that the anchors help preserve generation quality, while transfer experiments demonstrate identity disruption on five unseen personalization models. These findings reveal a vulnerability in tuning-free personalization: its identity carriers can be exploited for stealthy adversarial manipulation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Stealthy adversarial attacks, Tuning-free personalization, Diffusion models, Facial identity protection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bu87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IDOff, a privacy-protection method for tuning-free personalization, designed to achieve both effective identity protection and high-quality image generation. Unlike personalization methods such as DreamBooth or LoRA, this approach focuses on models that can extract and replicate a person’s identity from reference images without any fine-tuning.

Critical strengths
  1. The paper considers the inherent trade-off between identity protection and generation quality, a key challenge in existing privacy-protection methods. This is a highly relevant and practical research objective.

  2. The introduction of "Identity-Agnostic Anchors" provides a clear optimization objective for perturbing identity features, rather than simply pushing them away from the original representation.

Critical weaknesses
  1. The paper characterizes existing methods, such as Anti-DreamBooth and SimAC, as "adversarial attacks." However, the primary intent of these techniques is to safeguard user identity privacy, effectively making them proactive privacy-defense mechanisms. While it is technically accurate to describe them as "adversarial attacks" against generative models, the paper should more clearly distinguish between the technical perspective of attacking a generative model and the user-centric perspective of privacy protection. Furthermore, while the paper’s title highlights "Stealthy Adversarial Attack," the actual research objective is to protect user identity rather than to carry out malicious attacks. This phrasing may lead to misunderstandings regarding the paper’s research focus and its threat model.
  2. This work focuses specifically on optimizing "Identity Carriers" within the context of tuning-free personalization. However, fine-tuning-based personalization methods learn subject identities by updating model parameters. It remains unclear whether the perturbations generated by IDOff can effectively prevent such models from learning and reproducing the original identity.
  3. In real-world applications, images uploaded or shared by users are often subject to operations such as JPEG compression, scaling, or cropping. Furthermore, before being used for personalization, such images might be processed by an adversary using techniques like "diffusion-based purification." These operations could weaken or even completely eliminate the adversarial perturbations. Given that practical identity privacy protection must account for various image-processing workflows, robustness experiments are essential to demonstrate the method's real-world utility.
  4. Although the paper conducts transfer experiments across five personalization models not used during the optimization process, it reports only Identity Similarity, omitting results for generation quality and prompt consistency.
  5. IDOff requires joint gradient optimization across multiple personalized generative models, involving multiple iterations to produce protected images. In contrast, IDProtector can generate protective perturbations in a single forward pass using a pre-trained perturbation generator. It remains unclear whether the gains in generation quality provided by IDOff are sufficient to offset the additional computational costs.
  6. The experiments lacks independent ablation studies for each core component. For instance, the specific performance contributions of different types of anchors, Residual Regularization, and gradient coordination strategies remain unquantified. Furthermore, the paper lacks sensitivity analysis for key hyperparameters.
  7. The paper employs four text prompts during the optimization phase, but it is not explicitly stated whether these prompts are strictly distinct from those used in the final evaluation. If the same or highly similar text conditions are used for both optimization and testing, the reported results may not accurately reflect the method’s ability to generalize to unseen prompts. In real-world applications, unauthorized users could employ arbitrary text prompts to generate portraits in various poses, settings, or styles.
Questions for authors
  1. Can IDOff effectively prevent the reconstruction of the original identity even when the protected image undergoes JPEG compression, scaling, cropping, or purification processes based on diffusion models?
  2. Does the paper ensure that the identities and text prompts used for perturbation optimization are strictly separated from those used in the final evaluation?
7 comments
AuthorsOpenReview ↗

Privacy framing, fine-tuning scope, and full per-metric robustness to JPEG, downsampling, and cropping

Thank you for highlighting how the title and threat model read from the photograph owner's perspective.

Why the revised setup changed. Rather than patching issues piecemeal, we updated our protocol once and re-ran all evaluations uniformly (explaining numerical differences from September):

  • Prompt decoupling: Fixed gender phrases (PhotoMaker/InstantID) and the disjoint DigiFace prototype (IP-Adapter) with zero scene text, ensuring strict prompt separation.
  • Shared affine augmentation (std = 0.003): Improves resilience to routine preprocessing/geometric transforms without overfitting to test-time JPEG or cropping.
  • Official 8/255 baselines: September's iterative baselines (9/255 500-step PGD; old IDGuardian) lacked official code. To avoid confounding budget/step mismatches or reimplementation artifacts with prior work, we removed unofficial PGD and evaluate strictly official releases at 8/255: released IDProtector encoder and official IDGuardian (plus Clean-224 isolating IDGuardian's mandatory 224x224 downscaling).
  • Full per-metric reporting: We replaced TOPIQ with whole-image MUSIQ and DSL-FIQA, and report ArcFace and AdaFace individually throughout.

1. Privacy objective and terminology (W1).

We agree and will revise the title, abstract, and Section 1 to state the photograph owner's proactive privacy goal explicitly: adding bounded perturbations before sharing photos so unauthorized tuning-free models fail to reproduce their identity while still generating plausible, prompt-aligned portraits. We will distinguish the technical mechanism (adversarial optimization) from the defensive threat model, and describe Anti-DreamBooth, SimAC, IDProtector, and IDGuardian as proactive privacy defenses.

2. Scope relative to fine-tuning (W2).

IDOff acts on inference-time identity carriers, whereas DreamBooth and LoRA update model parameters. We will state this boundary clearly and note that fine-tuning protection remains a separate open question.

3. Full per-metric robustness to JPEG compression, downsampling, and cropping (W3 / Q1).

Direct finding: Identity protection survives all tested transforms across all three main generators on both ArcFace and AdaFace, while generated quality stays close to or above equally processed untreated controls.

Protected and untreated references underwent identical processing before generation: JPEG Q85, bicubic downsampling to 75% of original width and height (178x218 -> 134x164), and a seeded random crop retaining 80-100% of each dimension independently, followed by bicubic resizing to the original size (identical boxes across both arms). Cells give paired IDOff / equally processed untreated values (200 scheduled pairs, 50 owners; 199-200 valid pairs everywhere except InstantID Crop with 170 ID pairs across 45 owners and 196 quality pairs across 49 owners):

GeneratorProcessingArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-T
PhotoMaker v1Native-0.0490/0.3980-0.0125/0.38005.1835/5.37910.8163/0.798975.09/74.780.2939/0.3030
PhotoMaker v1JPEG Q850.1470/0.39630.1436/0.37945.1361/5.40110.7753/0.796173.81/74.690.3063/0.3030
PhotoMaker v1Bicubic 75%-0.0455/0.3960-0.0130/0.37835.1015/5.30290.8057/0.791274.59/74.450.2988/0.3022
PhotoMaker v1Crop 80-100%0.0838/0.39090.0863/0.37095.0732/5.30800.7797/0.792173.88/74.380.3065/0.3010
InstantIDNative-0.4652/0.6995-0.3449/0.64744.7688/4.95740.8015/0.745268.79/62.820.2875/0.2853
InstantIDJPEG Q85-0.3147/0.6980-0.2435/0.64704.7664/4.98720.8172/0.745569.51/62.900.2901/0.2852
InstantIDBicubic 75%-0.3961/0.6984-0.3042/0.64834.7510/4.96620.8071/0.740568.69/62.360.2876/0.2847
InstantIDCrop 80-100%-0.4627/0.6997-0.3504/0.65514.8461/5.04850.8016/0.755368.00/63.430.2818/0.2805
IP-Adapter Plus FaceNative0.0594/0.32510.0556/0.30304.8012/4.49150.7563/0.655173.27/71.660.2953/0.2801
IP-Adapter Plus FaceJPEG Q850.1339/0.33360.1285/0.31024.2746/4.35910.6597/0.665269.58/71.490.2883/0.2810
IP-Adapter Plus FaceBicubic 75%0.0802/0.31200.0650/0.28964.8215/4.40970.6827/0.635172.21/71.170.2903/0.2784
IP-Adapter Plus FaceCrop 80-100%0.1018/0.32960.0965/0.30384.3666/4.44720.6675/0.638769.72/71.930.2862/0.2821

IDOff keeps both ArcFace and AdaFace substantially below equally processed untreated controls across all 12 pairs (JPEG Q85 weakens protection most), while quality and CLIP-T stay close to or above untreated controls. During optimization we adapted IDProtector's affine augmentation (Song et al., Sec. 4.5; six shared Gaussian offsets, std = 0.003).

AuthorsOpenReview ↗

Full per-metric robustness to diffusion-based purification (DiffPure, GrIDPure, Noisy Upscaling)

5. Full per-metric transfer across five unseen generators (W4).

Direct finding: Identity suppression and prompt consistency transfer reliably to all five unseen generators, while facial and whole-image quality vary by architecture and metric.

Reusing the same saved protected photographs without adaptation, the table below reports every individual metric--ArcFace, AdaFace, FaceScore, DSL-FIQA, whole-image MUSIQ, CLIP-T, and SCRFD FDFR--across all five unseen generators and all four methods (50 owners x 4 scenes = 200 scheduled generations per method; lower ArcFace, AdaFace, FDFR better; higher quality/CLIP-T better):

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
ConsistentIDUntreated0.54270.49864.24830.854576.040.27900.025
ConsistentIDIDProtector0.29540.26093.51940.841774.760.27200.050
ConsistentIDIDGuardian0.34500.30533.61330.847075.530.27800.050
ConsistentIDIDOff-0.0555-0.02843.75020.835674.580.27770.050
FastComposerUntreated0.29750.27214.38650.659273.750.25930.000
FastComposerIDProtector0.16320.14803.84520.628971.390.25740.030
FastComposerIDGuardian0.17450.15824.17580.615371.900.25890.010
FastComposerIDOff0.13290.12514.13560.638572.060.26110.010
PhotoMaker v2Untreated0.51210.47994.96780.860474.360.29690.115
PhotoMaker v2IDProtector0.28180.24464.76550.855274.720.29510.120
PhotoMaker v2IDGuardian0.37880.34194.81800.831073.120.30210.115
PhotoMaker v2IDOff-0.0652-0.03014.79230.866975.270.30820.120
UniPortraitUntreated0.59070.55885.47350.812976.360.30860.035
UniPortraitIDProtector0.38070.34805.37200.814175.950.31000.055
UniPortraitIDGuardian0.46270.42595.48410.802676.020.31490.060
UniPortraitIDOff-0.0702-0.03485.40190.825476.450.31360.047
PuLIDUntreated0.62380.57645.19370.777971.020.31550.110
PuLIDIDProtector0.27000.23684.94860.753168.700.31060.095
PuLIDIDGuardian0.46180.41255.14310.748769.760.31680.115
PuLIDIDOff-0.3529-0.27925.31900.765469.790.31910.115

Paired 95% confidence intervals (IDOff minus untreated) and valid pair counts (out of 200):

Unseen generatorArcFace diff CIAdaFace diff CIFaceScore diff CIDSL-FIQA diff CIMUSIQ diff CICLIP-T diff CIValid (ID/Face/Img)
ConsistentID[-0.6275, -0.5645][-0.5501, -0.4988][-0.6766, -0.3218][-0.0302, -0.0094][-1.95, -0.97][-0.0053, 0.0027]189/200/200
FastComposer[-0.1804, -0.1472][-0.1611, -0.1315][-0.3240, -0.1869][-0.0313, -0.0121][-2.65, -0.94][-0.0006, 0.0041]198/198/200
PhotoMaker v2[-0.6081, -0.5458][-0.5354, -0.4841][-0.3345, -0.0273][-0.0004, 0.0138][0.42, 1.41][0.0082, 0.0147]176/200/200
UniPortrait[-0.6921, -0.6215][-0.6212, -0.5593][-0.1542, 0.0151][0.0050, 0.0201][-0.18, 0.36][0.0024, 0.0079]188/199/199
PuLID[-1.0172, -0.9325][-0.8920, -0.8151][0.0209, 0.2261][-0.0177, -0.0072][-1.58, -0.86][0.0018, 0.0054]174/200/200

IDOff achieves strictly lower ArcFace and AdaFace similarity than untreated references and both official baselines on all five unseen generators (all 10 identity CIs strictly negative). CLIP-T is preserved or significantly improved on all five models; FaceScore significantly rises on PuLID, matches untreated on UniPortrait, and declines on ConsistentID, FastComposer, and PhotoMaker v2. Against official baselines at the matched 8/255 budget (removing our earlier unofficial PGD row and retaining IDGuardian's 224x224 input pipeline), IDOff achieves higher mean FaceScore than released IDProtector on all five generators (strictly positive CIs on ConsistentID, FastComposer, and PuLID) and strictly higher CLIP-T on all five. Against official IDGuardian, IDOff achieves strictly higher FaceScore on PuLID (CI [0.0731, 0.2795]), higher mean FaceScore on ConsistentID (CI [-0.0320, 0.3144]), statistically indistinguishable FaceScore on FastComposer, PhotoMaker v2, and UniPortrait (all CIs include zero), and strictly higher DSL-FIQA on four of five generators.

AuthorsOpenReview ↗

Full per-metric transfer evaluation and paired 95% CIs across five unseen generators

5. Full per-metric transfer across five unseen generators (W4).

Reusing the same saved protected photographs without adaptation, the table below reports every individual metric--ArcFace, AdaFace, FaceScore, DSL-FIQA, whole-image MUSIQ, CLIP-T, and SCRFD FDFR--across all five unseen generators and all four methods (50 owners x 4 scenes = 200 scheduled generations per method; lower ArcFace, AdaFace, FDFR better; higher FaceScore, DSL-FIQA, MUSIQ, CLIP-T better):

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
ConsistentIDUntreated0.54270.49864.24830.854576.040.27900.025
ConsistentIDIDProtector0.29540.26093.51940.841774.760.27200.050
ConsistentIDIDGuardian0.34500.30533.61330.847075.530.27800.050
ConsistentIDIDOff-0.0555-0.02843.75020.835674.580.27770.050
FastComposerUntreated0.29750.27214.38650.659273.750.25930.000
FastComposerIDProtector0.16320.14803.84520.628971.390.25740.030
FastComposerIDGuardian0.17450.15824.17580.615371.900.25890.010
FastComposerIDOff0.13290.12514.13560.638572.060.26110.010
PhotoMaker v2Untreated0.51210.47994.96780.860474.360.29690.115
PhotoMaker v2IDProtector0.28180.24464.76550.855274.720.29510.120
PhotoMaker v2IDGuardian0.37880.34194.81800.831073.120.30210.115
PhotoMaker v2IDOff-0.0652-0.03014.79230.866975.270.30820.120
UniPortraitUntreated0.59070.55885.47350.812976.360.30860.035
UniPortraitIDProtector0.38070.34805.37200.814175.950.31000.055
UniPortraitIDGuardian0.46270.42595.48410.802676.020.31490.060
UniPortraitIDOff-0.0702-0.03485.40190.825476.450.31360.047
PuLIDUntreated0.62380.57645.19370.777971.020.31550.110
PuLIDIDProtector0.27000.23684.94860.753168.700.31060.095
PuLIDIDGuardian0.46180.41255.14310.748769.760.31680.115
PuLIDIDOff-0.3529-0.27925.31900.765469.790.31910.115

Paired 95% confidence intervals (IDOff minus untreated) and valid pair counts (out of 200):

Unseen generatorArcFace diff CIAdaFace diff CIFaceScore diff CIDSL-FIQA diff CIMUSIQ diff CICLIP-T diff CIValid (ID/Face/Img)
ConsistentID[-0.6275, -0.5645][-0.5501, -0.4988][-0.6766, -0.3218][-0.0302, -0.0094][-1.95, -0.97][-0.0053, 0.0027]189/200/200
FastComposer[-0.1804, -0.1472][-0.1611, -0.1315][-0.3240, -0.1869][-0.0313, -0.0121][-2.65, -0.94][-0.0006, 0.0041]198/198/200
PhotoMaker v2[-0.6081, -0.5458][-0.5354, -0.4841][-0.3345, -0.0273][-0.0004, 0.0138][0.42, 1.41][0.0082, 0.0147]176/200/200
UniPortrait[-0.6921, -0.6215][-0.6212, -0.5593][-0.1542, 0.0151][0.0050, 0.0201][-0.18, 0.36][0.0024, 0.0079]188/199/199
PuLID[-1.0172, -0.9325][-0.8920, -0.8151][0.0209, 0.2261][-0.0177, -0.0072][-1.58, -0.86][0.0018, 0.0054]174/200/200

IDOff achieves strictly lower ArcFace and AdaFace similarity than untreated references and both official baselines on all five unseen generators (all 10 identity-difference CIs strictly negative). CLIP-T is preserved or improved on all five models; FaceScore rises on PuLID, matches untreated on UniPortrait, and declines on ConsistentID, FastComposer, and PhotoMaker v2. Against baselines at the matched 8/255 budget (removing our earlier unofficial PGD row, and retaining IDGuardian's 224x224 input pipeline), IDOff outperforms released IDProtector on FaceScore across all five generators (and MUSIQ on four of five), outperforms IDGuardian on FaceScore on ConsistentID and PuLID, and outperforms IDGuardian on DSL-FIQA on four of five.

Reviewer_rnDQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IDOff, an adversarial perturbation method designed to disrupt identity reproduction in tuning-free image personalization while preserving the visual quality and prompt consistency of generated images. The proposed approach combines source-identity repulsion with model-specific identity-agnostic anchors, intended to guide perturbed identity representations toward generic-person conditions. The perturbations are jointly optimized against three personalization models (PhotoMaker v1, InstantID, and IP-Adapter Plus Face) using gradient coordination and adaptive weighting.

Critical strengths
  1. Interesting problem formulation and promising identity–quality trade-off. The paper goes beyond merely disrupting identity reproduction by explicitly aiming to preserve generation quality and prompt consistency. This is a meaningful objective for identity-protection methods. The results on three personalization models, supported by the human evaluation and anchor ablation, provide encouraging evidence that identity disruption does not necessarily require severe visual degradation.

  2. Multi-model optimization and promising transferability. The method jointly targets three different personalization architectures and demonstrates identity-disruption transfer to five unseen models without additional optimization. This is an important result suggesting that the proposed perturbations are not restricted to the exact architectures used during optimization.

Critical weaknesses
  1. Insufficient evidence for the central claim of preserving generation quality and prompt consistency.

A central contribution of the paper is its ability to disrupt identity reproduction without degrading generated images. However, the automatic quality metrics (TOPIQ and FaceScore) are evaluated only on detected facial regions, while CLIP-T measures text–image alignment rather than general perceptual quality. The differences in CLIP-T between competing methods are often small, yet confidence intervals are not reported. Furthermore, the qualitative examples in Figure 3 do not always make the claimed distortions introduced by competing methods apparent.

Although the human evaluation partially addresses these concerns, its protocol is insufficiently documented, and IDGuardian, one of the main baselines, is excluded without explanation. Importantly, the transfer experiments on five unseen models report only identity similarity, leaving it unclear whether the claimed preservation of generation quality and prompt consistency also transfers.

These limitations are consequential because preserving generation quality, rather than simply reducing identity similarity, is the primary motivation and claimed advantage of the proposed method.

  1. Incomplete validation of the optimization strategy and practical robustness.

The proposed approach combines several components, including joint optimization over three models, PCGrad, and adaptive loss weighting. However, the experiments do not establish whether optimizing against all three models is necessary, or whether the proposed dynamic weighting provides an advantage over simple uniform weighting. The existing ablation removes all anchor-alignment terms simultaneously but does not isolate these optimization choices.

Additionally, the robustness of the resulting perturbations to common image transformations, such as JPEG compression and resizing, is not evaluated. This is particularly relevant for the intended identity-protection application, where reference images may undergo such transformations before being used for personalization.

Consequently, it remains difficult to determine how much of the reported improvement can be attributed to the proposed optimization strategy and how reliably the protection would work in practical settings.

Questions for authors
  1. Preservation of generation quality on unseen models.

Section 4.5 demonstrates identity-disruption transfer to five unseen personalization models, but only reports identity similarity. Could the authors also provide quality and prompt-consistency metrics (e.g., TOPIQ, FaceScore, and CLIP-T) for these models? This is important for establishing whether the main claimed advantage of IDOff, rather than identity disruption alone, transfers to unseen architectures.

  1. Robustness to common image transformations.

How robust are the optimized perturbations to standard image transformations, particularly JPEG compression and resizing? Even a limited evaluation with a few common transformation settings would help assess the practical applicability of the proposed protection mechanism.

  1. Necessity of joint multi-model optimization.

The proposed method jointly optimizes perturbations against PhotoMaker, InstantID, and IP-Adapter Plus Face. How important is optimizing against all three models simultaneously? Could the authors provide a single-model versus multi-model optimization ablation, reporting both identity similarity and generation quality on the evaluation models? This would help establish the contribution of the joint optimization strategy.

  1. Contribution of adaptive weighting.

The method introduces adaptive loss-based gradient weighting (Eq. 7), but its contribution is not isolated experimentally. Could the authors compare the full method against PCGrad with uniform model weights, keeping all other optimization settings unchanged? This would clarify whether the adaptive weighting strategy provides a meaningful improvement over a simpler alternative.

  1. Reliability and interpretation of the automatic evaluation.

The CLIP-T differences between methods are often very small, and the paper mentions bootstrap confidence intervals without reporting them. Could the authors provide paired 95% confidence intervals for the main quality and prompt-consistency comparisons? Additional image-level quality assessments would also help substantiate the claims about overall generation quality.

Furthermore, the paper states that unavailable generations and recognition scores are excluded. Could the authors clarify whether filtering is performed separately for each metric and report the number of valid and excluded samples for each model–method combination?

  1. Human evaluation protocol and missing baseline.

The human evaluation reports 287 completed questionnaires, but important details are missing. How many unique annotators participated? How many independent ratings were collected per generated image? Was the presentation order randomized, and what was the inter-annotator agreement? Could the authors provide uncertainty estimates for the reported human scores?

Additionally, why was IDGuardian excluded from the human evaluation despite being one of the main competing methods in Table 1? Its inclusion, or a justification for its exclusion, would help establish the completeness of the perceptual comparison.

6 comments
AuthorsOpenReview ↗

Full main-table metrics, whole-image MUSIQ, paired 95% CIs, and revising Figure 3

Thank you for pointing out that face-crop metrics alone cannot establish whole-image quality and for requesting the paired confidence intervals.

Why the revised setup changed. Rather than patching issues piecemeal, we updated four elements together and re-ran all experiments under one unified protocol (explaining numerical differences from September):

  • Prompt decoupling: Fixed gender phrases (PhotoMaker/InstantID) and the disjoint DigiFace prototype (IP-Adapter) with zero scene text, eliminating optimization-evaluation prompt overlap.
  • Shared affine augmentation (std = 0.003): Improves resilience to routine preprocessing/geometric transforms without overfitting to test-time JPEG or cropping.
  • Official 8/255 baselines: September's iterative baselines (9/255 500-step PGD; old IDGuardian) lacked official code. To avoid confounding budget/step mismatches or reimplementation artifacts with prior work, we removed unofficial PGD and evaluate strictly official releases at 8/255 (IDProtector encoder and official IDGuardian, plus Clean-224 isolating IDGuardian's mandatory 224x224 downscaling).
  • Metric & audit upgrades: We replaced TOPIQ with whole-image MUSIQ and DSL-FIQA, report ArcFace and AdaFace individually, and withdrew the pre-revision human survey because it rated earlier pre-decoupling outputs.

W1 / Q5. Full per-metric main table (three target generators).

On 467 unaltered images from the earlier survey, TOPIQ's Spearman correlation with human ratings was -0.084 (PhotoMaker, n = 174), -0.004 (IP-Adapter, n = 152), and 0.409 (InstantID, n = 141), motivating its removal. Lower ArcFace, AdaFace, FDFR better; higher quality/CLIP-T better:

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
PhotoMaker v1Untreated0.39800.38005.37910.798974.780.30300.000
PhotoMaker v1IDProtector0.15710.13214.90160.775773.990.29900.000
PhotoMaker v1Clean-2240.35680.33205.43000.796674.710.30250.000
PhotoMaker v1IDGuardian0.24570.22575.29470.775673.790.30660.000
PhotoMaker v1IDOff-0.0490-0.01255.18350.816375.090.29390.000
InstantIDUntreated0.69950.64744.95740.745162.820.28530.000
InstantIDIDProtector0.23580.20794.27380.672556.030.27960.025
InstantIDClean-2240.69230.64164.90300.733161.350.28690.000
InstantIDIDGuardian0.56170.51234.77870.697257.990.28830.000
InstantIDIDOff-0.4652-0.34494.76880.801568.790.28750.000
IP-Adapter Plus FaceUntreated0.32500.30324.49150.655171.660.28010.000
IP-Adapter Plus FaceIDProtector0.14670.13164.45610.703270.250.28410.000
IP-Adapter Plus FaceClean-2240.32090.29764.44450.685171.950.28130.000
IP-Adapter Plus FaceIDGuardian0.03170.03464.18980.739771.270.28360.010
IP-Adapter Plus FaceIDOff0.05940.05564.80120.756373.270.29530.005

Paired 95% CIs for (IDOff minus Untreated) (20,000-resample identity bootstrap; 200 pairs, 50 owners; InstantID DSL-FIQA = 199 pairs):

GeneratorFaceScore diff CIDSL-FIQA diff CIMUSIQ diff CICLIP-T diff CI
PhotoMaker v1[-0.3991, 0.0018][0.0079, 0.0271][-0.1691, 0.8034][-0.0156, -0.0029]
InstantID[-0.3831, -0.0016][0.0347, 0.0788][4.3611, 7.6791][-0.0022, 0.0066]
IP-Adapter Plus Face[0.1322, 0.4757][0.0786, 0.1247][0.8464, 2.4035][0.0110, 0.0192]

IDOff significantly raises DSL-FIQA on all three models and MUSIQ on InstantID (+5.96) and IP-Adapter (+1.60), while InstantID FaceScore (-0.1886) and PhotoMaker CLIP-T (-0.0091) dip slightly.

W1. Revising Figure 3. We will replace Figure 3 with matched-seed outputs from these revised official-baseline runs and list per-image scores. At 8/255, IDOff beats released IDProtector with strictly favorable CIs on ArcFace, AdaFace, FaceScore, DSL-FIQA, and MUSIQ across all three models (e.g., InstantID MUSIQ CI [10.70, 14.98]). Against official IDGuardian on IP-Adapter, AdaFace CI includes zero ([-0.0025, 0.0431]), while IDOff achieves significantly higher FaceScore (CI [0.3827, 0.8436]), MUSIQ (CI [1.19, 2.85]), and CLIP-T (CI [0.0061, 0.0172]).

AuthorsOpenReview ↗

Full per-metric evaluation and paired 95% CIs on five unseen transfer generators

W1 / Q1. Full identity, quality, consistency, and coverage transfer on five unseen generators.

Direct finding: Identity suppression and prompt consistency transfer reliably across all five unseen generators, while facial and whole-image quality vary by architecture and metric.

Reusing the same saved protected photos without adaptation, we report ArcFace, AdaFace, FaceScore, DSL-FIQA, whole-image MUSIQ, CLIP-T, and SCRFD FDFR (50 owners x 4 scenes = 200 scheduled generations/method; lower ArcFace, AdaFace, FDFR better; higher quality/CLIP-T better):

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
ConsistentIDUntreated0.54270.49864.24830.854576.040.27900.025
ConsistentIDIDProtector0.29540.26093.51940.841774.760.27200.050
ConsistentIDIDGuardian0.34500.30533.61330.847075.530.27800.050
ConsistentIDIDOff-0.0555-0.02843.75020.835674.580.27770.050
FastComposerUntreated0.29750.27214.38650.659273.750.25930.000
FastComposerIDProtector0.16320.14803.84520.628971.390.25740.030
FastComposerIDGuardian0.17450.15824.17580.615371.900.25890.010
FastComposerIDOff0.13290.12514.13560.638572.060.26110.010
PhotoMaker v2Untreated0.51210.47994.96780.860474.360.29690.115
PhotoMaker v2IDProtector0.28180.24464.76550.855274.720.29510.120
PhotoMaker v2IDGuardian0.37880.34194.81800.831073.120.30210.115
PhotoMaker v2IDOff-0.0652-0.03014.79230.866975.270.30820.120
UniPortraitUntreated0.59070.55885.47350.812976.360.30860.035
UniPortraitIDProtector0.38070.34805.37200.814175.950.31000.055
UniPortraitIDGuardian0.46270.42595.48410.802676.020.31490.060
UniPortraitIDOff-0.0702-0.03485.40190.825476.450.31360.047
PuLIDUntreated0.62380.57645.19370.777971.020.31550.110
PuLIDIDProtector0.27000.23684.94860.753168.700.31060.095
PuLIDIDGuardian0.46180.41255.14310.748769.760.31680.115
PuLIDIDOff-0.3529-0.27925.31900.765469.790.31910.115

Paired 95% CIs (IDOff minus untreated) and valid pair counts (out of 200):

Unseen generatorArcFace diff CIAdaFace diff CIFaceScore diff CIDSL-FIQA diff CIMUSIQ diff CICLIP-T diff CIValid (ID/Face/Img)
ConsistentID[-0.6275, -0.5645][-0.5501, -0.4988][-0.6766, -0.3218][-0.0302, -0.0094][-1.95, -0.97][-0.0053, 0.0027]189/200/200
FastComposer[-0.1804, -0.1472][-0.1611, -0.1315][-0.3240, -0.1869][-0.0313, -0.0121][-2.65, -0.94][-0.0006, 0.0041]198/198/200
PhotoMaker v2[-0.6081, -0.5458][-0.5354, -0.4841][-0.3345, -0.0273][-0.0004, 0.0138][0.42, 1.41][0.0082, 0.0147]176/200/200
UniPortrait[-0.6921, -0.6215][-0.6212, -0.5593][-0.1542, 0.0151][0.0050, 0.0201][-0.18, 0.36][0.0024, 0.0079]188/199/199
PuLID[-1.0172, -0.9325][-0.8920, -0.8151][0.0209, 0.2261][-0.0177, -0.0072][-1.58, -0.86][0.0018, 0.0054]174/200/200

Key takeaways for the revision:

  • Universal identity suppression: On every unseen generator, IDOff reduces both ArcFace and AdaFace well below untreated inputs and both official baselines (all 10 identity CIs strictly negative).
  • Reliable prompt consistency: CLIP-T significantly improves on PhotoMaker v2, UniPortrait, and PuLID, and matches untreated on ConsistentID and FastComposer (CIs include zero).
  • Architecture-dependent quality: FaceScore significantly rises on PuLID (+0.1253), matches untreated on UniPortrait, and drops on ConsistentID, FastComposer, and PhotoMaker v2; MUSIQ significantly rises on PhotoMaker v2 (+0.91) and matches untreated on UniPortrait, while declining on ConsistentID, FastComposer, and PuLID.
  • Baselines: Against released IDProtector, IDOff achieves higher mean FaceScore on all five models (strictly positive CIs on ConsistentID, FastComposer, PuLID) and strictly higher CLIP-T on all five. Against official IDGuardian, IDOff achieves strictly higher FaceScore on PuLID (CI [0.0731, 0.2795]), higher mean FaceScore on ConsistentID (CI [-0.0320, 0.3144]), statistically comparable FaceScore on the other three (all CIs include zero), and strictly higher DSL-FIQA on four of five.
AuthorsOpenReview ↗

Full per-metric robustness to JPEG compression, downsampling, cropping, and purification

W2 / Q2. Complete per-metric robustness to routine image transformations.

Direct finding: Identity suppression survives JPEG compression, 75% downsampling, and 80-100% random cropping across all three main generators on both ArcFace and AdaFace, while generated quality remains close to or above equally processed untreated controls.

Protected and untreated references underwent identical processing before generation: JPEG Q85, bicubic downsampling to 75% of original width and height (178x218 -> 134x164), and a seeded random crop retaining 80-100% of each dimension independently, followed by bicubic resizing to the original size (identical crop boxes across both arms). The table reports every individual metric as paired IDOff / equally processed untreated (200 pairs across 50 owners in all cells, except native IP-Adapter ArcFace/AdaFace = 199, InstantID DSL-FIQA = 199 for Native/JPEG/Bicubic 75%, and InstantID Crop with 170 ID pairs across 45 owners and 196 quality pairs across 49 owners):

GeneratorProcessingArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-T
PhotoMaker v1Native-0.0490/0.3980-0.0125/0.38005.1835/5.37910.8163/0.798975.09/74.780.2939/0.3030
PhotoMaker v1JPEG Q850.1470/0.39630.1436/0.37945.1361/5.40110.7753/0.796173.81/74.690.3063/0.3030
PhotoMaker v1Bicubic 75%-0.0455/0.3960-0.0130/0.37835.1015/5.30290.8057/0.791274.59/74.450.2988/0.3022
PhotoMaker v1Crop 80-100%0.0838/0.39090.0863/0.37095.0732/5.30800.7797/0.792173.88/74.380.3065/0.3010
InstantIDNative-0.4652/0.6995-0.3449/0.64744.7688/4.95740.8015/0.745268.79/62.820.2875/0.2853
InstantIDJPEG Q85-0.3147/0.6980-0.2435/0.64704.7664/4.98720.8172/0.745569.51/62.900.2901/0.2852
InstantIDBicubic 75%-0.3961/0.6984-0.3042/0.64834.7510/4.96620.8071/0.740568.69/62.360.2876/0.2847
InstantIDCrop 80-100%-0.4627/0.6997-0.3504/0.65514.8461/5.04850.8016/0.755368.00/63.430.2818/0.2805
IP-Adapter Plus FaceNative0.0594/0.32510.0556/0.30304.8012/4.49150.7563/0.655173.27/71.660.2953/0.2801
IP-Adapter Plus FaceJPEG Q850.1339/0.33360.1285/0.31024.2746/4.35910.6597/0.665269.58/71.490.2883/0.2810
IP-Adapter Plus FaceBicubic 75%0.0802/0.31200.0650/0.28964.8215/4.40970.6827/0.635172.21/71.170.2903/0.2784
IP-Adapter Plus FaceCrop 80-100%0.1018/0.32960.0965/0.30384.3666/4.44720.6675/0.638769.72/71.930.2862/0.2821

Key observations:

  • Protection persists across all 12 generator-condition pairs: On both ArcFace and AdaFace, IDOff stays substantially below equally processed untreated inputs. JPEG Q85 causes the largest attenuation (e.g., PhotoMaker v1 ArcFace rises from -0.0490 to 0.1470 vs. 0.3963 untreated; InstantID ArcFace rises from -0.4652 to -0.3147 vs. 0.6980 untreated).
  • Generated quality stays well-behaved: Across all three models, whole-image MUSIQ, FaceScore, DSL-FIQA, and CLIP-T stay close to or above equally processed untreated controls (e.g., InstantID maintains MUSIQ 68.00-69.51 vs. 62.36-63.43 untreated and DSL-FIQA 0.8016-0.8172 vs. 0.7405-0.7553 untreated across all three transforms).

Affine augmentation and diffusion purification. During protection, we adapted IDProtector's affine augmentation (Song et al., Sec. 4.5), sampling six Gaussian offsets per reference (std = 0.003) shared across all three branches (applied to InstantID's alignment matrix and image coordinates for PhotoMaker/IP-Adapter). We also evaluated DiffPure, GrIDPure, and Noisy Upscaling across all three main generators: while stronger purifiers partially attenuate protection, IDOff keeps post-purification ArcFace and AdaFace similarity below native untreated baselines in all nine purifier-generator pairs (full 9-row per-metric table provided in our reply to Reviewer bu87 and the diffusion purification supplement).

Reviewer_i2WhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates adversarial perturbations for tuning-free portrait personalization, aiming to reduce identity similarity between generated images and the reference person while preserving generation quality and prompt consistency. The authors propose IDOff, which applies bounded perturbations to reference images by combining source-identity repulsion with identity-agnostic anchor alignment. For the three models, anchors are constructed using contextual person-related text features, attention responses, and population prototypes, respectively, and the perturbations are jointly optimized. Experimental results show that IDOff reduces identity similarity while largely preserving output quality on the models included in optimization. It also demonstrates the ability to disrupt identity reproduction on models excluded from optimization.

Critical strengths

1.The paper distinguishes failures in identity reproduction from degradation in generation quality, noting that natural, prompt-consistent outputs may still fail to faithfully preserve the reference identity. This issue is relevant to understanding the reliability of personalized generation models and studying facial identity privacy protection. 2.The method combines identity repulsion with anchor constraints to weaken source-identity features while guiding the model to continue generating natural portraits, thereby reducing the impact of identity disruption on generation quality. Rather than assuming that different models share a common representation space, the authors design corresponding alignment mechanisms for text fusion, attention responses, and image tokens. 3.The paper does not rely solely on identity similarity, but also incorporates multiple quality metrics, prompt-consistency metrics, and blinded human evaluation. This provides strong support for the interpretation that the anchors primarily help preserve generation quality. 4.IDOff reduces identity similarity on all five models excluded from optimization, indicating that the perturbations exhibit a degree of cross-model transferability.

Critical weaknesses

1.The current results only show that identity disruption can be maintained after removing the anchors while retaining the other components. They do not isolate the contribution of the repulsion term itself. In particular, the effects of residual regularization and gradient coordination across models cannot be ruled out. I recommend adding a variant that uses only the repulsion term, followed by variants that separately add residual regularization, gradient coordination, and anchors. 2.The paper primarily aims to demonstrate identity disruption while preserving output quality. However, Table 4 reports only identity similarity on models excluded from optimization, without reporting quality, prompt consistency, face detection failure rates, or human evaluation results. Although these results support the transferability of identity disruption, they are insufficient to establish that output-level stealthiness also transfers. I recommend conducting a complete evaluation on the transfer models. 3.The paper explicitly states that the evaluation identities were used during development and that the perturbed images were saved as lossless PNG files. Therefore, the current results do not fully address two practical questions: whether the method design and hyperparameters generalize to an independent set of identities, and whether the protection remains effective after uploading, compression, or resizing. I recommend fixing the method design and hyperparameters before evaluating on an independent identity set, and testing JPEG compression at multiple levels, resizing, and moderate cropping. Identity disruption, generation quality, and input photograph quality should all be reported after these transformations. 4.On IP-Adapter, IDOff achieves better generation quality but weaker identity disruption than IDProtector-PGD; the two methods also use different perturbation budgets. It therefore remains unclear whether the quality advantage comes from the method design or from a weaker attack. I recommend using the same perturbation budget, varying the attack strength of each method, and plotting trade-off curves between identity disruption and generation quality. The key comparison is whether IDOff still maintains better generation quality at comparable levels of identity disruption.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Revision overview and component ablations (repulsion, anchors, and IP-Adapter-specific regularizer)

Thank you for the constructive review and for suggesting a repulsion-only ablation to cleanly separate component roles.

Why the revised setup changed. Rather than patching individual concerns across incompatible runs, we updated our protocol once and re-ran all evaluations uniformly (which explains numerical differences from the September draft):

  • Prompt decoupling: Optimization now uses fixed generic gender phrases (PhotoMaker/InstantID) and the disjoint DigiFace prototype (IP-Adapter) with zero scene text, eliminating optimization-evaluation prompt overlap.
  • Shared affine augmentation (std = 0.003): Added to improve resilience to routine preprocessing and geometric transforms without overfitting to test-time JPEG or cropping.
  • Official matched-budget (8/255) baselines: In September, both iterative baselines (IDProtector-PGD at 9/255, 500 steps; old IDGuardian) were unofficial reimplementations without authors' code. To avoid confounding budget/step mismatches or reimplementation artifacts with prior work, we removed unofficial PGD and evaluate exclusively official author releases at 8/255: the released IDProtector encoder and official IDGuardian (plus Clean-224 to isolate IDGuardian's mandatory 224x224 resizing).
  • Full per-metric reporting: We replaced TOPIQ (inconsistent correlation with human ratings) with whole-image MUSIQ and complementary facial metric DSL-FIQA, and report ArcFace and AdaFace individually across all main, transfer, robustness, and purification tables.

W1. Isolating source-identity repulsion.

Direct finding: Source-identity repulsion is the primary driver of identity suppression. Removing repulsion while keeping the anchors, residual regularizer, and joint solver raises ISM (average signed ArcFace/AdaFace cosine similarity; lower is stronger protection) on every generator (200, 200, and 199 matched pairs across 50 owners):

GeneratorWithout repulsion (ISM)Full IDOff (ISM)
PhotoMaker v10.0702-0.0307
InstantID0.1785-0.4050
IP-Adapter Plus Face0.20780.0575

W1. Role of anchors and the IP-Adapter-specific residual regularizer.

Direct finding: Anchoring and residual regularization recover facial quality and prompt consistency while keeping identity similarity low. Crucially, the residual regularizer in Eq. (6) is exclusively designed for IP-Adapter Plus Face (set to zero on PhotoMaker v1 and InstantID, which use contextual class anchors rather than multi-token image prompts). Only IP-Adapter encodes identity as a 16-token sequence where the population anchor aligns only the token mean, leaving internal centered-token geometry unconstrained. Demonstrating its quality gain specifically on IP-Adapter directly validates its architectural purpose.

Holding the joint solver fixed, we tested (i) removing all anchors and the regularizer (repulsion only) and (ii) removing only the regularizer (ablation / full IDOff on IP-Adapter; 190 and 198 matched pairs for ISM, 200 for FaceScore and CLIP-T):

Ablation (IP-Adapter Plus Face)ISM (lower better)FaceScore (higher better)CLIP-T (higher better)
Remove anchors + regularizer (repulsion only)-0.0373 / 0.05744.2732 / 4.80120.2373 / 0.2953
Remove regularizer only-0.0422 / 0.05784.4127 / 4.80120.2780 / 0.2953

Repulsion alone drives ISM lower than full IDOff but severely degrades FaceScore (4.2732 vs. 4.8012) and CLIP-T (0.2373 vs. 0.2953). Retaining the anchor while removing only the regularizer recovers part of that loss (4.4127, 0.2780), confirming that both terms are needed on IP-Adapter.

Gradient coordination scope. All comparisons above fix gradient normalization, PCGrad, and adaptive branch weighting. We treat these as standard multi-task solver choices for optimizing the joint objective rather than claimed algorithmic contributions. In the revision, we will explicitly label repulsion and anchoring in the framework figure, state clearly that the residual regularizer is IP-Adapter-specific, and move solver update details to the implementation appendix.

AuthorsOpenReview ↗

Full per-metric evaluation on five transfer generators (ArcFace, AdaFace, quality, consistency, FDFR)

W2. Complete evaluation on the five unseen transfer generators.

Direct finding: Identity suppression and prompt consistency (CLIP-T) transfer reliably across all five unseen generators with untreated-level detection rates (FDFR); facial and whole-image quality vary by architecture and metric.

Reusing the same protected photos without adaptation, we report ArcFace, AdaFace, FaceScore, DSL-FIQA, whole-image MUSIQ, CLIP-T, and SCRFD FDFR (50 owners x 4 scenes = 200 scheduled generations/method; lower ArcFace, AdaFace, FDFR better; higher quality/CLIP-T better):

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
ConsistentIDUntreated0.54270.49864.24830.854576.040.27900.025
ConsistentIDIDProtector0.29540.26093.51940.841774.760.27200.050
ConsistentIDIDGuardian0.34500.30533.61330.847075.530.27800.050
ConsistentIDIDOff-0.0555-0.02843.75020.835674.580.27770.050
FastComposerUntreated0.29750.27214.38650.659273.750.25930.000
FastComposerIDProtector0.16320.14803.84520.628971.390.25740.030
FastComposerIDGuardian0.17450.15824.17580.615371.900.25890.010
FastComposerIDOff0.13290.12514.13560.638572.060.26110.010
PhotoMaker v2Untreated0.51210.47994.96780.860474.360.29690.115
PhotoMaker v2IDProtector0.28180.24464.76550.855274.720.29510.120
PhotoMaker v2IDGuardian0.37880.34194.81800.831073.120.30210.115
PhotoMaker v2IDOff-0.0652-0.03014.79230.866975.270.30820.120
UniPortraitUntreated0.59070.55885.47350.812976.360.30860.035
UniPortraitIDProtector0.38070.34805.37200.814175.950.31000.055
UniPortraitIDGuardian0.46270.42595.48410.802676.020.31490.060
UniPortraitIDOff-0.0702-0.03485.40190.825476.450.31360.047
PuLIDUntreated0.62380.57645.19370.777971.020.31550.110
PuLIDIDProtector0.27000.23684.94860.753168.700.31060.095
PuLIDIDGuardian0.46180.41255.14310.748769.760.31680.115
PuLIDIDOff-0.3529-0.27925.31900.765469.790.31910.115

Paired 95% CIs (IDOff minus untreated) and valid pair counts (out of 200):

Unseen generatorArcFace diff CIAdaFace diff CIFaceScore diff CIDSL-FIQA diff CIMUSIQ diff CICLIP-T diff CIValid (ID/Face/Img)
ConsistentID[-0.6275, -0.5645][-0.5501, -0.4988][-0.6766, -0.3218][-0.0302, -0.0094][-1.95, -0.97][-0.0053, 0.0027]189/200/200
FastComposer[-0.1804, -0.1472][-0.1611, -0.1315][-0.3240, -0.1869][-0.0313, -0.0121][-2.65, -0.94][-0.0006, 0.0041]198/198/200
PhotoMaker v2[-0.6081, -0.5458][-0.5354, -0.4841][-0.3345, -0.0273][-0.0004, 0.0138][0.42, 1.41][0.0082, 0.0147]176/200/200
UniPortrait[-0.6921, -0.6215][-0.6212, -0.5593][-0.1542, 0.0151][0.0050, 0.0201][-0.18, 0.36][0.0024, 0.0079]188/199/199
PuLID[-1.0172, -0.9325][-0.8920, -0.8151][0.0209, 0.2261][-0.0177, -0.0072][-1.58, -0.86][0.0018, 0.0054]174/200/200
  • Baselines: IDOff achieves strictly lower ArcFace and AdaFace than untreated inputs and both official baselines on all five generators (all 10 CIs strictly negative). Against released IDProtector, IDOff achieves higher mean FaceScore on all five models (strictly positive CIs on ConsistentID, FastComposer, PuLID) and strictly higher CLIP-T on all five. Against official IDGuardian, IDOff achieves strictly higher FaceScore on PuLID (CI [0.0731, 0.2795]), higher mean FaceScore on ConsistentID (CI [-0.0320, 0.3144]), statistically comparable FaceScore on the other three (all CIs include zero), and strictly higher DSL-FIQA on four of five.
  • Detection coverage: SCRFD missed 10/200, 2/200, 24/200, 9/199, and 23/200 IDOff portraits (table order), closely matching untreated counts (5/200, 0/200, 23/200, 7/199, 22/200)--confirming low similarity is not caused by face-detection collapse.
  • Human evaluation: Because the September survey rated earlier pre-decoupling outputs prior to official IDGuardian integration, we will withdraw that outdated table and present these revised transfer results as comprehensive automatic evaluations.
AuthorsOpenReview ↗

Generalization scope and full per-metric robustness to image processing and diffusion purification

W3. Identity, prompt, and hyperparameter generalization scope.

Direct finding: IDOff learns no model weights or identity prototypes from the 50 evaluated owners, is now strictly decoupled from downstream scene prompts, and applies one fixed global configuration across all 50 owners.

Specifically: (1) all generator and encoder weights stay frozen, the IP-Adapter population prototype is calibrated on 100 synthetic DigiFace identities completely disjoint from the 50 CelebA owners, and PhotoMaker/InstantID use one fixed generic gender phrase; (2) downstream scene prompts enter strictly at generation time after protected photos are saved; and (3) all 50 owners share one fixed configuration (8/255 budget, 1/255 step, two 200-step restarts, Gaussian affine std = 0.003) with no per-owner tuning. Because these 50 CelebA owners served as our development cohort when fixing that global configuration, evaluation on an additional held-out owner cohort remains open and will be noted explicitly as a limitation.

W3. Full per-metric results after JPEG compression, downsampling, and cropping.

Direct finding: Identity protection persists across all tested transformations and generators on both ArcFace and AdaFace, while generated quality and prompt alignment remain close to or above equally processed untreated controls.

Protected and untreated references underwent identical processing before generation: JPEG Q85, bicubic downsampling to 75% of original width and height (178x218 -> 134x164), and a seeded random crop retaining 80-100% of each dimension independently, followed by bicubic resizing to the original size (identical crop boxes across both arms). Cells report paired IDOff / equally processed untreated values (200 scheduled pairs across 50 owners; 199-200 valid pairs in all cells except InstantID Crop with 170 valid ID pairs across 45 owners and 196 quality pairs across 49 owners):

GeneratorProcessingArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-T
PhotoMaker v1Native-0.0490/0.3980-0.0125/0.38005.1835/5.37910.8163/0.798975.09/74.780.2939/0.3030
PhotoMaker v1JPEG Q850.1470/0.39630.1436/0.37945.1361/5.40110.7753/0.796173.81/74.690.3063/0.3030
PhotoMaker v1Bicubic 75%-0.0455/0.3960-0.0130/0.37835.1015/5.30290.8057/0.791274.59/74.450.2988/0.3022
PhotoMaker v1Crop 80-100%0.0838/0.39090.0863/0.37095.0732/5.30800.7797/0.792173.88/74.380.3065/0.3010
InstantIDNative-0.4652/0.6995-0.3449/0.64744.7688/4.95740.8015/0.745268.79/62.820.2875/0.2853
InstantIDJPEG Q85-0.3147/0.6980-0.2435/0.64704.7664/4.98720.8172/0.745569.51/62.900.2901/0.2852
InstantIDBicubic 75%-0.3961/0.6984-0.3042/0.64834.7510/4.96620.8071/0.740568.69/62.360.2876/0.2847
InstantIDCrop 80-100%-0.4627/0.6997-0.3504/0.65514.8461/5.04850.8016/0.755368.00/63.430.2818/0.2805
IP-Adapter Plus FaceNative0.0594/0.32510.0556/0.30304.8012/4.49150.7563/0.655173.27/71.660.2953/0.2801
IP-Adapter Plus FaceJPEG Q850.1339/0.33360.1285/0.31024.2746/4.35910.6597/0.665269.58/71.490.2883/0.2810
IP-Adapter Plus FaceBicubic 75%0.0802/0.31200.0650/0.28964.8215/4.40970.6827/0.635172.21/71.170.2903/0.2784
IP-Adapter Plus FaceCrop 80-100%0.1018/0.32960.0965/0.30384.3666/4.44720.6675/0.638769.72/71.930.2862/0.2821

JPEG Q85 causes the strongest attenuation among the tested transforms, yet both ArcFace and AdaFace stay well below equally processed untreated controls on all three generators. On the reference inputs themselves, we strictly enforce the 8/255 L-infinity pixel bound; perceptual quality metrics evaluate the generated portraits.

Diffusion purification overview. We also evaluated DiffPure, GrIDPure, and Noisy Upscaling on all 200 IDOff reference photos across all three main generators (full 9-row per-metric table provided in our reply to Reviewer bu87 and the diffusion purification supplement). While stronger purifiers partially attenuate protection, IDOff keeps post-purification ArcFace and AdaFace similarity below native untreated levels in all nine purifier-generator pairs (e.g., on InstantID, where native untreated ArcFace/AdaFace is 0.6995/0.6474, post-purification values are 0.0143/0.0075 under DiffPure, -0.0684/-0.0429 under GrIDPure, and 0.3509/0.3156 under Noisy Upscaling).

AuthorsOpenReview ↗

Equal-budget comparison on IP-Adapter and all three main generators with paired 95% CIs

W4. Equal-budget comparison on IP-Adapter Plus Face (and why IDProtector-PGD was removed).

Direct finding: At a matched 8/255 budget with official author code, IDOff strictly dominates the released IDProtector encoder on every metric, and achieves markedly higher FaceScore, MUSIQ, and CLIP-T than official IDGuardian at statistically comparable AdaFace similarity.

In September, IDProtector-PGD (9/255, 500 steps) was our own reimplementation without official PGD code, confounding budget, step count, and reimplementation fidelity. For a clean, reproducible 8/255 comparison, we removed that unofficial row and evaluate strictly official baselines: the released IDProtector encoder (feed-forward) and official IDGuardian (iterative, retaining its 224x224 preprocessing alongside a resize-only control, Clean-224).

1. Full per-metric comparison across all three main generators (lower ArcFace, AdaFace, FDFR better; higher quality/CLIP-T better):

GeneratorMethodArcFaceAdaFaceFaceScoreDSL-FIQAMUSIQCLIP-TFDFR
PhotoMaker v1Untreated0.39800.38005.37910.798974.780.30300.000
PhotoMaker v1IDProtector0.15710.13214.90160.775773.990.29900.000
PhotoMaker v1Clean-2240.35680.33205.43000.796674.710.30250.000
PhotoMaker v1IDGuardian0.24570.22575.29470.775673.790.30660.000
PhotoMaker v1IDOff-0.0490-0.01255.18350.816375.090.29390.000
InstantIDUntreated0.69950.64744.95740.745162.820.28530.000
InstantIDIDProtector0.23580.20794.27380.672556.030.27960.025
InstantIDClean-2240.69230.64164.90300.733161.350.28690.000
InstantIDIDGuardian0.56170.51234.77870.697257.990.28830.000
InstantIDIDOff-0.4652-0.34494.76880.801568.790.28750.000
IP-Adapter Plus FaceUntreated0.32500.30324.49150.655171.660.28010.000
IP-Adapter Plus FaceIDProtector0.14670.13164.45610.703270.250.28410.000
IP-Adapter Plus FaceClean-2240.32090.29764.44450.685171.950.28130.000
IP-Adapter Plus FaceIDGuardian0.03170.03464.18980.739771.270.28360.010
IP-Adapter Plus FaceIDOff0.05940.05564.80120.756373.270.29530.005

Resizing alone (Clean-224) barely shifts IP-Adapter similarity (ArcFace 0.3209 vs. 0.3250 raw; AdaFace 0.2976 vs. 0.3032 raw).

2. Paired IP-Adapter Plus Face comparisons at 8/255 (95% CIs for IDOff minus baseline):

GeneratorMetricIDOff / IDProtector95% CI (Diff)Pairs
IP-Adapter Plus FaceArcFace0.0594 / 0.1469[-0.1117, -0.0643]199
IP-Adapter Plus FaceAdaFace0.0556 / 0.1318[-0.0980, -0.0558]199
IP-Adapter Plus FaceFaceScore4.8012 / 4.4561[0.1291, 0.5620]200
IP-Adapter Plus FaceDSL-FIQA0.7563 / 0.7032[0.0377, 0.0689]200
IP-Adapter Plus FaceMUSIQ73.27 / 70.25[2.29, 3.77]200
IP-Adapter Plus FaceCLIP-T0.2953 / 0.2841[0.0058, 0.0164]200

As a fixed single-pass feed-forward encoder, IDProtector cannot sweep attack strength continuously at inference. At 8/255, IDOff strictly dominates it on every metric (and likewise dominates on ArcFace, AdaFace, FaceScore, DSL-FIQA, and MUSIQ across PhotoMaker v1 and InstantID).

GeneratorMetricIDOff / IDGuardian95% CI (Diff)Pairs
IP-Adapter Plus FaceArcFace0.0595 / 0.0317[0.0006, 0.0546]197
IP-Adapter Plus FaceAdaFace0.0558 / 0.0345[-0.0025, 0.0431]197
IP-Adapter Plus FaceFaceScore4.8012 / 4.1898[0.3827, 0.8436]200
IP-Adapter Plus FaceDSL-FIQA0.7563 / 0.7397[-0.0001, 0.0330]200
IP-Adapter Plus FaceMUSIQ73.27 / 71.27[1.19, 2.85]200
IP-Adapter Plus FaceCLIP-T0.2953 / 0.2836[0.0061, 0.0172]200

On IP-Adapter, IDGuardian has slightly lower ArcFace (CI [0.0006, 0.0546]) and statistically indistinguishable AdaFace (CI [-0.0025, 0.0431]), while IDOff achieves markedly higher FaceScore (+0.6114), MUSIQ (+1.99), and CLIP-T (+0.0117). On PhotoMaker v1 and InstantID, IDOff outperforms IDGuardian by large margins on ArcFace, AdaFace, DSL-FIQA, and MUSIQ. Continuous multi-strength curves across iterative methods remain untested and will be noted as a limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.