acceptodds
Under review as a conference paper at ICLR 2027

BEM: Learning Bregman Divergences for Concept Erasure in Flow Matching Models

Abstract

Concept erasure removes harmful, private, or copyrighted concepts from text-to-image models while preserving their other abilities. Leading models such as FLUX and Stable Diffusion 3 are trained with flow matching, and erasure methods for them fine-tune the model toward a guided teacher's velocities under a Euclidean loss. Yet the squared distance is only one member of a larger family: Bregman divergences are exactly the losses under which flow matching recovers the correct velocity field, and each member shapes differently what is erased and what is preserved. We introduce BEM (Bregman erasure matching), which is, to our knowledge, the first to learn this divergence through bi-level optimization, so that the fine-tuned model forgets the concept on unseen prompts while its unrelated generations stay unchanged. In the experiments we target the suppression of nudity, violence, real identities, and copyrighted characters. Under the evaluation protocol of GEM, BEM removes more of the concept than GEM on most settings, cutting the unsafe rate on bloody gore from to and the FID on nudity from to , while retention holds on some concepts and degrades on others.

Primary area:
generative models
Keywords:
concept erasure, machine unlearning, flow matching, Bregman divergence, bi-level optimization, text-to-image generation, AI safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rVD3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This proposes a new method called BEM to erasure concepts from modern flow-matching models. Prior work (like GEM) use velocity-matching objectives and this paper raises the point now that this silently assumes a Squared Distance measure for the divergence to the wanted "erased" behavior, which penalizes every deviation from the ideal velocity direction equally. The paper reveals that this choice does not have to be that way: any so-called "Bregmann diverence" can be used, leading to more complex penalization constraint, for example, one that is derived from the preserved utility of the model. The authors propose to learn this divergence measure in a bi-level optimization by parameterizing it as a strictly convex function (apparently a property that Bregman divergences need to fulfill according to its definition). They evaluate BEM within the evaluation framework of GEM by essentially plugging in their proposed, regularized BEM objective.

Critical strengths
  • Inspiring perspective and clear derivation of the motivating thoughts.
  • Novel idea to use a learned objective for concept erasure instead of relying on a static divergence measure or fixed regularization.
Critical weaknesses
  • Absence of ablations to substantiate the design choices. Even though I know that this method seems to be quite compute-heavy, I would have expected (1) some variations in the design of the potential network φ\varphi, (2) sweeps of the weight ρ\rho, (3) fixed-potential variants, (4) no regularization, and (5) alternative LerasureL_{erasure} choices (ESD loss, GEM loss, ...) accompanied by the BEM bi-level Bregman-potential regularization.
  • The paper never fully discusses the around 50x longer runtime of BEM compared to GEM; and what that means for practical deployment.
Questions for authors

Future Work

  • Is it possible to choose a different LretentionL_{retention} criterion that, e.g., also includes an entire set of retention concepts, or erase multiple concepts simulatenously?
  • Can you provide some ablations to support the design of your method? Especially, ablations like (3-5) from the ones mentioned under critical weaknesses would help to better judge the effectiveness of the approach.
Reviewer_qJNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BEM, a concept erasure algorithm for flow-matching models. Building on GEM’s training procedure, BEM replaces GEM’s contrastive loss with a learnable Bregman divergence and introduces a bi-level optimization framework that jointly optimizes the model and the loss function. Experimental results show that BEM achieves better concept erasure performance than GEM in most evaluation settings, but it can reduce retention of non-target concepts in some cases and incurs significantly higher training costs.

Critical strengths
  1. An interesting and mathematically motivated perspective. The paper distinguishes errors in concept-related versus orthogonal directions and argues that they affect erasure and retention differently, incorporating this distinction into the loss design. By optimizing the error metric itself, the method offers a useful perspective on concept erasure.

  2. Clear implementation and reproducibility details. The paper clearly describes the approximations used to translate the theoretical formulation into a practical algorithm and provides detailed descriptions of the network architecture, experimental settings, and hyperparameters. These details facilitate assessment of the method and support reproducibility.

Critical weaknesses
  1. Insufficient motivation. Direction-dependent error weighting is mathematically intuitive, but the paper lacks concrete examples, analysis, or diagnostic experiments showing the limitations of existing metrics and the benefits of the proposed design. The need for a learnable Bregman divergence and the associated bi-level optimization therefore remains insufficiently justified. The authors should strengthen the motivation with concrete examples or diagnostic evidence.

  2. Lack of an overall framework diagram . The method is explained largely through text and equations, with no diagram illustrating the overall framework or the bi-level optimization pipeline. Adding a framework diagram that clarifies module interactions, information flow, and the bi-level optimization process would substantially improve readability.

Questions for authors

1.Reliability of single-step inner updates. The implicit hypergradient is derived under an inner stationarity condition, whereas the implementation performs only one inner update per outer step. Does the warm start keep the model sufficiently close to a stationary point throughout training? Could you provide stationarity residuals or comparisons across different numbers of inner updates to validate this approximation? 2. Baseline comparisons and contribution attribution. Could you supplement the results cited from GEM with direct comparisons against additional methods under a consistent evaluation protocol? Since BEM replaces GEM’s hinge objective with a regression objective and introduces a learned divergence, could you include ablations using fixed MSE, fixed directional weighting, and simplified potentials to isolate these contributions? 3. Performance–cost trade-off. BEM substantially increases training time and reduces non-target retention in some settings. How does it compare with GEM under matched computational budgets? Could you report erasure and retention across different budgets to clarify the benefits of the additional computation and the settings where it is worthwhile? 4. Positioning relative to prior work. Could you discuss relevant work on loss learning, metric learning, and learned Bregman divergences? In particular, how does BEM differ from existing divergence-based concept unlearning methods in its formulation, theoretical basis, and optimization mechanism?

Reviewer_gDF6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies concept erasure in flow-matching text-to-image models (FLUX, Stable Diffusion 3). The authors observe that existing erasure methods that regress toward a teacher-guided target velocity all use the squared Euclidean distance, which treats errors along the concept direction and orthogonal directions equally. Since Bregman divergences are exactly the losses under which flow matching recovers the correct velocity field (Theorem 2.1), the squared distance is only one member of a valid family. The authors propose BEM, which keeps GEM's unlearning procedure (LoRA adapter, frozen teacher, noise window, optimizer) unchanged but replaces GEM's hinge loss with a Bregman divergence generated by a learned strictly convex potential, and learns that potential via bi-level optimization: the inner problem erases under the current divergence, and the outer problem minimizes a meta-loss combining an erasure criterion (projection of student displacement onto the concept–anchor separation) and a retention criterion (squared deviation on the anchor prompt). The potential is parameterized as a partially input-convex network with separate curvature along the concept direction, and the hypergradient uses the implicit function theorem with conjugate gradient and a Gauss-Newton approximation. Experiments erase nudity, bloody gore, real identities, and copyrighted characters on FLUX and SD3.5, comparing against GEM under the same evaluation protocol.

Critical strengths
  1. The core observation is clean and well-grounded: erasure by velocity matching inherits the valid flow-matching loss family, so the divergence is a free parameter that prior work fixes by convention. Theorem 2.1 is used appropriately to justify the design space.
  2. The method is carefully controlled: only the loss is swapped while GEM's unlearning procedure and evaluation protocol are held fixed, making the comparison attributable to the loss rather than to confounders.
  3. The potential family is well-designed—strict convexity, contains the squared distance, cheap Hessian—and the bi-level formulation with an implicit-function hypergradient is technically sound, with forward/backward Jacobian-vector products avoiding second-order graphs.
  4. Results are strong on several benchmarks: gore unsafe rate drops from 53.9% to 14.4%, nudity FID from 8.64 to 6.64, and rights-protected erasure improves markedly on both FLUX and SD3.5.
  5. The paper is candid about retention degradation (Einstein/Mandela, SD3.5 Stitch) and offers a plausible attribution, which is good scientific practice.
Critical weaknesses
  1. Retention is inconsistent: gains on some concepts, significant losses on others (Einstein retention 72.7% vs. 93.3%; SD3.5 average retention 92.0% vs. 97.0%). The paper attributes this to "complete erasure," but this is post-hoc and not quantitatively disentangled from genuine collateral damage.
  2. Computational cost is severe: a meta-step takes ~290 s vs. 4 s for an inner step; rights-protected erasure takes ~80 min and SD3.5 Stitch takes 438 min vs. 10 min for GEM. This undermines practical deployment and is under-discussed as a limitation.
  3. The bi-level scheme uses one inner step between meta-updates and evaluates the hypergradient at an approximate inner solution; the theoretical justification (implicit function theorem at θ*(ψ)) does not strictly hold, and the gap is not analyzed.
  4. Evaluation relies on automated detectors (NudeNet, Q16, Qwen2.5-VL) and 100 generations per concept; several reported differences are within noise, and the classifier swap from Gemini to Qwen complicates direct comparison with GEM's reported numbers.
  5. The meta-loss design (ρ, erasure/retention criteria, saliency weighting) involves several hand-chosen components whose sensitivity is not thoroughly ablated in the main text.
Questions for authors
  1. Can you disentangle "complete erasure" from collateral damage on Einstein/Mandela? Would a retention-aware meta-loss or a larger retention set fix the drop?
  2. What is the sensitivity to ρ, the number of inner steps between meta-updates, and CG iterations? How far is the approximate inner solution from θ*(ψ)?
  3. Given the 70–400× runtime increase, is the gain over GEM worth the cost? Can the learned potential be amortized or transferred across concepts?
  4. How much do results depend on the detector swap (Gemini → Qwen2.5-VL)? Can you report GEM under the same detector for all benchmarks?
  5. Does the learned potential generalize to unseen concepts, or is it re-learned per concept? What does the learned curvature along the concept direction look like?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.