IDOff: Stealthy Adversarial Attack on Tuning-Free Personalization

Tuning-free personalization enables high-fidelity identity reproduction from only a few reference portraits, raising concerns about unauthorized identity synthesis. Existing attacks can disrupt identity reproduction at the cost of degraded facial quality and prompt adherence, leaving visible signs of intervention. To address this limitation, we introduce IDOff, a stealthy adversarial attack framework that disrupts identity reproduction while preserving generation quality and prompt consistency. Our key insight is that identity carriers, the intermediate representations conveying reference identity to the generator, serve as an exploitable surface for such stealthy attacks. To this end, IDOff optimizes bounded reference-image perturbations to redirect these carriers toward identity-agnostic anchors derived from generic-person text embeddings or a population prototype. Extensive evaluations demonstrate that IDOff achieves substantial identity disruption while maintaining generation quality comparable to the no-attack baseline. An ablation study shows that the anchors help preserve generation quality, while transfer experiments demonstrate identity disruption on five unseen personalization models. These findings reveal a vulnerability in tuning-free personalization: its identity carriers can be exploited for stealthy adversarial manipulation.