- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-concept safety guidance for text-to-image generation, motivated by the observation that existing inference-time safeguards are typically designed for a single undesired concept and may interfere when several harmful or copyrighted concepts appear together. The paper introduces CHC-Bench, which constructs prompts containing two to four co-occurring targets, and proposes QPS, a quadratic-programming aggregation rule that combines concept-specific safety corrections. Experiments on Stable Diffusion 1.5 and FLUX.1-dev show that QPS generally suppresses more co-occurring targets than top-1 selection while preserving substantially more image fidelity than direct summation.
Critical strengths
The paper identifies a concrete weakness in dominant-category selection: when several undesirable concepts are simultaneously present, choosing only one correction may leave the others effectively untreated at that sampling step. QPS provides a direct geometric response by treating every active correction as a constraint and finding the smallest update that preserves their directional effects jointly. The two-direction analysis is particularly intuitive: the method smoothly interpolates between averaging compatible directions and requiring increasingly large corrections as directions conflict, while explicitly identifying the infeasible opposite-direction case.
Critical weaknesses
QPS guarantees that the aggregated correction preserves the scalar projection of each standalone guidance direction, but it is unclear how strongly this geometric property should be interpreted as a safety guarantee. In particular, the guarantee is defined with respect to the supplied correction rather than actual concept suppression, so it does not ensure that the retained direction remains semantically effective throughout sampling. This gap becomes concrete for SLD: the QP constraints are enforced before the momentum update, while the paper explicitly notes that they do not guarantee retention after momentum is applied. As a result, the theoretical guarantee applies to an intermediate correction rather than necessarily to the update ultimately delivered to the sampler. The paper would benefit from clarifying what safety property directional retention is intended to guarantee in the full sampling pipeline, and how this property relates to actual downstream concept suppression.
The treatment of infeasible QPS cases makes the reported safety performance somewhat difficult to interpret. The theoretical analysis explicitly shows that the QP can become infeasible when guidance directions strongly conflict, including the limiting case of opposite corrections. However, the evaluation states that images rejected by the QPS solver are excluded from the harmful-rate computation.
Questions for authors
how should the directional-retention constraints be interpreted as a safety guarantee when the standalone guidance direction itself may be imperfect and when downstream sampler operations such as momentum can alter the applied direction?
5 comments
Thank you for the response. The explanation about aggregation-level guarantee is reasonable. The link between retention and actual concept suppression is acknowledged but still not shown. How often is the constraint satisfied while the concept is still not suppressed?
On infeasibility, the response does not talk near-infeasible cases under strongly conflicting directions, where the correction can become very large. How often does this happen, and how does it affect image quality? I will keep my score for now. Results on these two points would help.
Dear authors, To be specific about my remaining concerns and to increase my confidence in your work, I extend my previous comment on your rebuttal with two questions above. I am glad to raise my evaluation if they are resolved.
We thank the reviewer for their thoughtful comments and valuable insights for improving our paper.
We interpret directional retention as a guarantee about aggregation, not about the construction of safety guidance nor the ultimate safety. That a single-concept correction di may fail to remove its concept is common, and QPS does not attempt to resolve this.
With that said, naively extending guidance to multiple concepts adds a second source of failure. Combining category-specific corrections can cause harmful conflicts, and the aggregation rule itself can weaken or drop a correction that would have worked alone. QPS addresses this failure by ensuring that aggregation does not weaken or drop any active correction along its own direction.
However, this does not ensure that a correction that works alone still works in the aggregate, since the other components of the update and the resulting trajectory differ from standalone guidance. Likewise, minimizing the correction norm is a geometric objective and does not necessarily lead to the best preservation of benign image content. We therefore evaluate final concept suppression and image quality empirically in this paper.
To address the reviewer's concern about downstream operations, we examine whether this geometric guarantee is preserved through the sampling pipeline.
Settings applied before aggregation (e.g., SLD strength or DVE thresholds) determine the supplied corrections di to which the per-step constraints apply. After aggregation, our pipelines apply an affine map T(δ)=cδ+r at each step, with a scalar c (e.g., CFG scale, sampler coefficient) and an offset r independent of δ (e.g., SLD's momentum buffer). Then (T(δ)−T(di))⊤(cdi)=c2(di⊤δ−∥di∥22)≥0, so from the same state the delivered update moves along each cdi, the direction in which di alone would move it, at least as far as di alone would.
This compares the delivered update with that of di alone under the same momentum buffer; the absolute projection after momentum is not guaranteed. We therefore checked the retention constraint on the correction that the sampler actually receives after momentum (SD1.5, SLD), and it still held for 99.5% of step–concept pairs under QPS.
However, operations such as dynamic thresholding and guidance rescaling (disabled in our pipelines) fall outside this guarantee and may produce updates that no longer satisfy the directional-retention constraints.