- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes localized concept erasure through masked preference learning on minimally edited safe–unsafe image pairs. Residual image features injected through a frozen IP-Adapter provide target-related visual cues, and a BPO objective replaces DPO. Visual erasure and ESD-style semantic cleanup update separate self-attention and cross-attention LoRAs. Experiments primarily evaluate nudity erasure on SD v1.4, with additional concepts and SDXL/FLUX extensions.
Critical strengths
-
Meaningful combination of localized supervision and visual cues. In Table 4, removing localized triplets or residual injection increases UnlearnDiffAtk ASR from 6.34% to 33.80% or 24.65%, supporting their practical value.
-
Strong reported erasure–preservation trade-off on SD v1.4. Table 1 reduces I2P detector-positive outputs from 337 to 1 while reporting FID 13.12 and LPIPS 23.67. Table 8 further supports suppression across four image-conditioning interfaces.
Critical weaknesses
- The gradient explanation does not fully describe the implemented objective. Although the unclipped loss has a non-vanishing margin derivative, Algorithm 1 uses hard clipping, which gives zero derivatives outside the clipping interval. Clipping frequency is unreported.
- The target-specificity of the residual cue is insufficiently justified**
Eq. (4) define . However, feature subtraction does not ensure that isolates target-specific information: the residual may also encode replacement content, editing artifacts, or changes in non-target attributes.
Questions for authors
- Please specify hyperparameters . What fraction of training samples is clipped, and does this support the persistent-gradient explanation?
- What evidence supports the claim that the residual cue in Eq. (4) isolates target-specific visual information?
We thank Reviewer p9tq for recognizing our localized supervision, residual visual cues, and erasure–preservation trade-off. Below, we will address each of your concerns in detail.
Q1: Hyperparameters, clipping frequency, and the persistent-gradient explanation
Hyperparameters. For SD v1.4, β scales the preference margin, τ bounds it before exponentiation, λbg weights background preservation, and γ controls target-word cleanup strength:
Clipping frequency. As you noted, hard clipping m~=clip(m,−τ,τ) gives zero preference gradient outside the clipping interval. We measured the fraction outside either bound in a separate 1,200-step visual-only BPO run with residual cues, with clipping enabled throughout:
Substantial late-stage clipping does not support persistent gradients throughout training. Within the unclipped interval, DPO and SBA-BPO multiply the negative margin gradient by
gD(m)=1+em1,gB(m)=4e−2m+em.
DPO's multiplier decays as the positive margin grows. SBA-BPO's remains positive and increases for m>ln(2)/3; it is not monotonically increasing over all positive margins. Clipping stabilizes this mechanism; it does not certify complete erasure of clipped samples.
We will document the settings and clipping statistics and specify the unclipped regime.
Q2: Evidence that residual cues are more target-focused
Our claim is stronger target focus relative to full-image features, rather than complete semantic isolation. Paired subtraction is designed to reduce shared-scene information.
(1) Quantitative target-focus diagnostic. Inspired by WinCLIP's local similarity mapping [1] and RISE's deletion-based evaluation [2], we test target–background separation and the response drop after target removal. Across 16 reference pairs and 20 disjoint Pikachu test pairs, the encoder, crops, and masks are fixed. Responses are cosine similarities; the gap compares target and background means, AUROC ranks target pixels, and removal drop weights matched-crop response decreases by target coverage.
Residual cues improve all three indicators, including a 47.7% larger target–background gap. This supports stronger target focus in the tested encoder space, rather than complete semantic isolation.
See Figure 2 in the revised main text and Appendix A.3.3 for visualizations and quantitative diagnostics.
(2) Why the cue need not directly penalize replacement content. As in Eq. (6), the residual cue conditions only the dispreferred unsafe branch, while the preferred safe branch is evaluated with a null cue on the safe latent. This safe branch anchors the replacement image. This asymmetry addresses the concern but does not guarantee preservation of every replacement attribute.
(3) Direct conditioning ablation and training benefit. With BPO fixed, we compared full-image and residual cues in 1,200-step visual-only training without target-word cleanup. IP-Adapter detections use 135 sources; benign LPIPS uses 64 matched COCO inputs:
In the full framework, removing residual injection also raises UnlearnDiffAtk ASR from 6.34% to 24.65%. These results support the target focus, robustness, and preservation benefits of residual cues; we will replace the purity claim accordingly.
References
[1] J. Jeong et al. WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation. CVPR, 2023.
[2] V. Petsiuk, A. Das, and K. Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. BMVC, 2018.