ResMark: Late-Stage Residual Insertion for Fidelity-Preserving Watermarking in Diffusion Denoisers

Text-to-image diffusion models need reliable watermarking for provenance and attribution. Denoiser-side watermarking couples the watermark to the iterative denoising process. A first-order sensitivity analysis suggests that early perturbations are transported through more subsequent solver transitions. As a result, the watermarked output can differ from its watermark-free counterpart under matched prompts and noise—a phenomenon we call paired-output drift. We formulate denoiser-side watermarking as late-stage residual insertion. Low-Noise Training (LNT) trains the watermark LoRA on low-noise timesteps. At inference, late-step activation enables it only over the final solver steps through a soft ramp. Because the first active step begins from a watermark-free trajectory, we propose a Teacher-Relative Insertion Loss (TRI), which trains the LoRA-enabled prediction to match the frozen base model prediction plus the desired watermark residual on the same watermark-free latent. Across SD1.5 U-Net and PixArt transformer denoisers, ResMark substantially improves paired-output fidelity over full-trajectory denoiser watermarking while retaining strong message recovery and robustness to conventional distortions and AI-regeneration attacks.