- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents a new concept erasure method for flow matching models called TRACE, which is supposed to be "trajectory-aware". They postulate that concept erasure can induce conflicting velocities paths in a flow model, which degrades the model utility. It's main novelty seems to lie in a pre-unlearning stage that optimizes the anchor trajectory to reduce the utility-degrading side-effects of the unlearning itself. They evaluate TRACE on many concept-unlearning scenarios and measure erasure efficacy with CLIP-based metrics and utility additionally with FID. TRACE appears to outperform prior work on these metrics, but the appendix comes with extensive additional formal proofs, which are unfortunately not really connected to the main paper and raise some doubts on their true relevance to the main contribution of this work.
Critical strengths
- Intuitively sensible motivation and overall interesting deep dive into potential pitfalls of concept erasure in these models.
Critical weaknesses
- Uninformative metrics based on CLIP. CLIP is known to be not a very good signal on its own. Evaluation needs more appropriate measures as established concept classifiers like NudeNet or an additional VLM-based evaluation, especially for identity or nudity unlearning. More qualitative comparisons are needed too. The provided qualitative comparison in Figure 3 does not really reflect a superiority of TRACE over the baselines EA or GEM.
- Unfortunately, the paper does appear over-formalized (around 23 proofs) which makes it difficult to grasp the essential contribution. Explained pseuocode or a documented code supplement would have helped to better understand the actual method and how it differs to existing methods.
Questions for authors
Clarifications
- In the paper, it is claimed that achieving effective erasure while preserving utility is particularly challenging for flow matching models (line 52), but you do not provide any empirical evidence for it. Can you provide convincing quantitative support for your claim? This would require analyzing the erasure/utility trade-off for fixed methods across model types.
- Which prompt benchmarks or prompt templates were used for the evaluation? Can you evaluate on established unsafe prompt benchmarks like T2I-RiskyPrompts, I2P, etc.?
- Can you explain how Propositions A.16 to A.22 relate to the main paper? In particular, the terms Bayes floor and the first-order scale of mixture ambiguity were puzzling to me.
- You are writing in the Appendix that the experiments were performed at a resolution of pixels. Is that also the case for the baselines? What hardware was used for these experiments? How many GPUs at once? I am asking because the runtime reported for some of these methods make the impression that the experiments were conducted on a smaller resolution.
Future Work
- More informative metrics are needed. Anchor-win is an odd choice, because what we primarily care about is whether the erased target concept is truly erased, not whether the surrogate anchor is faithfully generated instead. CLIP is not a very fine-grained signal and needs to be complemented by more qualitative results, proper VLM-based evaluation, and whenever possible by specialized scenario-specific classifiers. Can you extend your main experiments by additional metrics to better compare to related work and give the readers an idea of how much harmful/unwanted generations remain?
- Can you please provide a direct methodological comparison (perhaps with pseudocode) between TRACE and, for example, ESD or GEM?