acceptodds
Under review as a conference paper at ICLR 2027

Aggregating Safety Guidance for Co-occurring Concepts via Quadratic Programming

Abstract

Despite advances in safety guidance for text-to-image generation, suppressing multiple co-occurring undesired concepts remains challenging. Prior work has shown that naively aggregating category-specific safety guidances can cause directional attenuation, motivating selecting the most aligned guidance with the current generative state. Although such heuristic can mitigate interference among concurrent safety signals, it remains unclear whether it can reliably suppress all targets when multiple undesired concepts co-occur within a single image. To investigate this question, we introduce CHC-Bench, a benchmark for evaluating the simultaneous suppression of restricted concepts of different kinds. We construct a prompt-generation and verification pipeline that retains prompts whose generated images contain all specified targets, enabling evaluation of their joint suppression. We further propose QPS, which formulates multi-guidance aggregation as a quadratic program. QPS finds the minimum-norm correction to the base generative update while jointly preserving each active guidance source's standalone directional effect. Experiments on CHC-Bench demonstrate the importance of jointly considering multiple guidance signals for suppressing co-occurring targets. Evaluations on conventional safety benchmarks further demonstrate a balanced safety–utility tradeoff, indicating that joint aggregation remains useful in single target settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Guidance, Multi-Concept Erasure Benchmark, Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4VnDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-concept safety guidance for text-to-image generation, motivated by the observation that existing inference-time safeguards are typically designed for a single undesired concept and may interfere when several harmful or copyrighted concepts appear together. The paper introduces CHC-Bench, which constructs prompts containing two to four co-occurring targets, and proposes QPS, a quadratic-programming aggregation rule that combines concept-specific safety corrections. Experiments on Stable Diffusion 1.5 and FLUX.1-dev show that QPS generally suppresses more co-occurring targets than top-1 selection while preserving substantially more image fidelity than direct summation.

Critical strengths

The paper identifies a concrete weakness in dominant-category selection: when several undesirable concepts are simultaneously present, choosing only one correction may leave the others effectively untreated at that sampling step. QPS provides a direct geometric response by treating every active correction as a constraint and finding the smallest update that preserves their directional effects jointly. The two-direction analysis is particularly intuitive: the method smoothly interpolates between averaging compatible directions and requiring increasingly large corrections as directions conflict, while explicitly identifying the infeasible opposite-direction case.

Critical weaknesses

QPS guarantees that the aggregated correction preserves the scalar projection of each standalone guidance direction, but it is unclear how strongly this geometric property should be interpreted as a safety guarantee. In particular, the guarantee is defined with respect to the supplied correction rather than actual concept suppression, so it does not ensure that the retained direction remains semantically effective throughout sampling. This gap becomes concrete for SLD: the QP constraints are enforced before the momentum update, while the paper explicitly notes that they do not guarantee retention after momentum is applied. As a result, the theoretical guarantee applies to an intermediate correction rather than necessarily to the update ultimately delivered to the sampler. The paper would benefit from clarifying what safety property directional retention is intended to guarantee in the full sampling pipeline, and how this property relates to actual downstream concept suppression.

The treatment of infeasible QPS cases makes the reported safety performance somewhat difficult to interpret. The theoretical analysis explicitly shows that the QP can become infeasible when guidance directions strongly conflict, including the limiting case of opposite corrections. However, the evaluation states that images rejected by the QPS solver are excluded from the harmful-rate computation.

Questions for authors

how should the directional-retention constraints be interpreted as a safety guarantee when the standalone guidance direction itself may be imperfect and when downstream sampler operations such as momentum can alter the applied direction?

5 comments

We thank the reviewer for their thoughtful comments and valuable insights for improving our paper.

Question: How should the directional-retention constraints be interpreted as a safety guarantee when the standalone guidance direction itself may be imperfect and when downstream sampler operations such as momentum can alter the applied direction?

We interpret directional retention as a guarantee about aggregation, not about the construction of safety guidance nor the ultimate safety. That a single-concept correction did_i may fail to remove its concept is common, and QPS does not attempt to resolve this.

With that said, naively extending guidance to multiple concepts adds a second source of failure. Combining category-specific corrections can cause harmful conflicts, and the aggregation rule itself can weaken or drop a correction that would have worked alone. QPS addresses this failure by ensuring that aggregation does not weaken or drop any active correction along its own direction.

However, this does not ensure that a correction that works alone still works in the aggregate, since the other components of the update and the resulting trajectory differ from standalone guidance. Likewise, minimizing the correction norm is a geometric objective and does not necessarily lead to the best preservation of benign image content. We therefore evaluate final concept suppression and image quality empirically in this paper.


To address the reviewer's concern about downstream operations, we examine whether this geometric guarantee is preserved through the sampling pipeline.

Settings applied before aggregation (e.g., SLD strength or DVE thresholds) determine the supplied corrections did_i to which the per-step constraints apply. After aggregation, our pipelines apply an affine map T(δ)=c δ+rT(\delta)=c\,\delta+r at each step, with a scalar cc (e.g., CFG scale, sampler coefficient) and an offset rr independent of δ\delta (e.g., SLD's momentum buffer). Then (T(δ)−T(di))⊤(c di)=c2(di⊤δ−∥di∥22)≥0(T(\delta)-T(d_i))^\top(c\,d_i)=c^2(d_i^\top\delta-\|d_i\|_2^2)\ge0, so from the same state the delivered update moves along each c dic\,d_i, the direction in which did_i alone would move it, at least as far as did_i alone would.

This compares the delivered update with that of did_i alone under the same momentum buffer; the absolute projection after momentum is not guaranteed. We therefore checked the retention constraint on the correction that the sampler actually receives after momentum (SD1.5, SLD), and it still held for 99.5% of step–concept pairs under QPS.

However, operations such as dynamic thresholding and guidance rescaling (disabled in our pipelines) fall outside this guarantee and may produce updates that no longer satisfy the directional-retention constraints.


Weakness: The treatment of infeasible QPS cases.

  • We agree that this requires clarification. During our experiments, we encountered 2 failures which resulted from cycling in the active-set solver, despite the underlying problems being feasible. After noticing the failure, we adopted support enumeration solver instead, and conducted experiments using it. Both cases were resolved and are included in the reported results. Also, the OSQP solver we included (stated in Appendix B.2) has not produced any failures.
  • We will retain the exclusion rule in the evaluation protocol, and report how many images the solvers rejected in each experiment to prevent confusion.

Thank you for the response. The explanation about aggregation-level guarantee is reasonable. The link between retention and actual concept suppression is acknowledged but still not shown. How often is the constraint satisfied while the concept is still not suppressed?

On infeasibility, the response does not talk near-infeasible cases under strongly conflicting directions, where the correction can become very large. How often does this happen, and how does it affect image quality? I will keep my score for now. Results on these two points would help.

Dear authors, To be specific about my remaining concerns and to increase my confidence in your work, I extend my previous comment on your rebuttal with two questions above. I am glad to raise my evaluation if they are resolved.

Reviewer_diM1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

This paper investigates the challenge of suppressing multiple co-occurring undesired concepts in text-to-image generation. The authors introduce CHC-Bench to evaluate joint concept suppression and propose QPS, a quadratic programming approach that aggregates multiple safety guidance signals while preserving their individual directional effects. Experiments demonstrate improvements over existing averaging method, with a trade-off between suppression effectiveness and generation quality.

Critical strengths
  1. The paper addresses a practically relevant problem of jointly suppressing multiple undesired concepts within a single generated image.
  2. The proposed method is straightforward and intuitive.
  3. The experiments cover multiple safety guidance methods and different generative backbones, demonstrating the applicability of QPS across different settings.
Critical weaknesses
  1. The paper proposes QPS to jointly aggregate multiple safety guidance signals at each denoising step, but lacks an in-depth analysis of why this is necessary. Existing work (CASG) allows different harmful concepts to be selected across denoising steps, potentially addressing multi-concept suppression over the entire generation process. The paper does not sufficiently investigate why such step-wise selection remains inadequate or how QPS addresses this limitation throughout the denoising process.
  2. The paper introduces CHC-Bench to evaluate the suppression of co-occurring undesired concepts, but its design does not fully align with the central motivation of multi-harmful-concept suppression. Specifically, the necessity of including CHAR+HARM is not sufficiently justified, as practical safety scenarios often involve multiple harmful concepts (e.g., sexual content and violence). In the more relevant HARM+HARM setting, QPS underperforms Sum (43.3% vs. 60.3%), and utility metrics are not separately reported for HARM+HARM.
  3. The contributions of this paper appear to be relatively incremental. The challenges of multi-concept suppression and safety guidance conflicts have been explored in prior work. QPS primarily formulates guidance aggregation as a quadratic programming problem, while CHC-Bench builds upon the existing CoME-Bench construction pipeline.
Questions for authors
  1. Some subsets of I2P also involve multiple harmful concepts. Could the authors provide a separate evaluation of QPS on these subsets to further assess its effectiveness in multi-harmful-concept suppression?
  2. Have the authors evaluated the sensitivity of QPS to different hyperparameter settings, such as SLD safety strengths (e.g., Medium and Strong) and CFG guidance scales?
5 comments

We thank the reviewer for their thoughtful comments and valuable insights for improving our paper.

Q1: Could the authors provide a separate evaluation of QPS on these subsets to further assess its effectiveness in multi-harmful-concept suppression?

  • We report the multi-category subset (389 prompts) of the I2P used in the paper. QPS retains the lowest ASR and LG among the non-summation rules for every constructor.

Multi-category subset (903) on SD1.5, ASR / LG (%).

Constructorvanillaconcattop-1meansumQPS
SLD41.1 / 17.78.4 / 2.78.4 / 3.011.3 / 5.90.6 / 0.04.9 / 2.3
CCFG41.1 / 17.722.7 / 10.922.6 / 8.726.1 / 11.61.4 / 1.116.1 / 8.0
CRG41.1 / 17.721.9 / 10.912.3 / 3.726.0 / 10.13.4 / 0.66.5 / 3.2

Q2. Have the authors evaluated the sensitivity of QPS to different hyperparameter settings, such as SLD safety strengths (e.g., Medium and Strong) and CFG guidance scales?

  • QPS had lower ASR and LG than top-1 at every SLD strength (Medium, Strong, Max) and every CFG scale (5.0, 7.5, 10.0).
  1. We reran SD1.5 SLD at its Medium and Strong presets on UnsafeDiffusion (UD, 904 prompts) and COCO-1K.
SLD strengthRuleUD ASR ↓UD LG ↓CLIP ↑CLIP-I ↑FID ↓
–vanilla51.526.326.691.0000.00
Mediumtop-129.415.026.030.90534.0
MediumQPS19.78.625.560.87040.7
Strongtop-119.17.925.230.84743.7
StrongQPS12.45.624.090.80251.2
Maxtop-19.44.124.410.79753.3
MaxQPS3.71.922.690.74464.1
  1. We reran SD1.5 SLD-Max at CFG 5.0 and 10.0 in addition to 7.5 (the paper's setting) on UD and COCO-1K. COCO CLIP-I and FID are measured against vanilla generations at the same CFG.
CFGRuleUD ASR ↓UD LG ↓CLIP ↑CLIP-I ↑FID ↓
5.0vanilla47.626.026.381.0000.00
5.0top-17.93.923.600.78455.4
5.0QPS2.31.322.050.73166.2
7.5vanilla51.526.326.691.0000.00
7.5top-19.44.124.410.79753.3
7.5QPS3.71.922.690.74464.1
10.0vanilla54.525.426.771.0000.00
10.0top-19.35.424.450.79752.6
10.0QPS4.82.422.950.74663.1
  • We explain why the retention constraints themselves hold for any value of these hyperparameters in our response to Reviewer 4VnD (Question 1).

Q3. Why CHAR + HARM row is necessary?

  • A copyrighted character is not harmful in itself, but it is an undesired concept in the sense the paper addresses: we target co-occurring undesired concepts, including copyright-infringing generations (Sec. 1), and CHC-Bench is accordingly a co-occurring harm and copyright benchmark. CHAR+HARM evaluates situations in which a model owner's policy restricts both the reproduction of specified copyrighted characters and harmful content. For a prompt combining these targets, removing only the character identity or only the harmful attribute leaves one requirement unmet.

Weakness 1. Lack of in-depth analysis why CASG is not sufficient for removing multiple concepts

  • Step-wise selection lets top-1 address different concepts over the trajectory, but it leaves the others uncorrected when multiple concepts are active. Each concept is therefore corrected on fewer steps, and this deficit grows with the number of co-active concepts. When the selected correction conflicts with a skipped one, it can even partly counteract the skipped concept's correction.
  • QPS removes both limitations at every guided step: no concept is skipped or attenuated along its own direction, whichever other concepts are active. This is consistent with Table 2, where the advantage of QPS over top-1 on CHAR+HARM grows with KK.

Weakness 2. In HARM + HARM setting, QPS underperforms SUM

SUM removes more HARM+HARM targets mainly because its correction is larger: rescaling either rule to the other's norm closes most of the gap, and the extra magnitude costs content, a cost that grows as more concepts are targeted.

  • In the revision, all CHC-Bench rows are regenerated and evaluated, dropping scored column in paper's Table 7 (Appendix D). The table below reports removal and preservation on the 300 HARM+HARM prompts.
  • We also added VIEScore2 [1] for measuring benign utility preservation. SC (semantic consistency) rates how well the image follows the prompt. We score it against the cleaned prompt, so it measures how much of the requested benign content remains. PQ (perceptual quality) rates freedom from artifacts and distortions. All ∧ SC>0 is the share of rows in which every target is judged removed and the image still keeps some of the prompt content (SC > 0).
RuleTarget removal (%)All (%)CLIP ↑CLIP-I ↑FID ↓SC ↑PQ ↑All ∧ SC>0 (%)
top-160.540.723.250.524253.15.765.0440.3
mean56.535.023.600.568235.56.015.1635.0
sum93.088.318.060.518320.12.272.5034.0
QPS66.346.022.990.537254.05.425.0343.3

  • Our hypothesis is that most of SUM's higher removal comes from a larger correction. When the corrections do not conflict, the sum already satisfies every retention constraint. QPS moves exactly as far along each binding did_i as did_i alone (di⊤δ=∥di∥2d_i^\top\delta = \|d_i\|^2), while SUM goes further by its overlap with the other corrections (di⊤δ=∥di∥2+∑j≠idi⊤djd_i^\top\delta = \|d_i\|^2 + \sum_{j\ne i} d_i^\top d_j).
  • To separate magnitude from direction, we rescaled each rule's correction at every step to the norm of the other rule's correction for the same proposals.
RuleTarget removal (%)All (%)CLIP ↑CLIP-I ↑FID ↓
sum93.088.318.060.518320.1
QPS66.346.022.990.537254.0
QPS scaled to sum93.589.318.140.502313.5
sum scaled to QPS70.752.323.380.551248.8
  • Rescaling resulted each strategy showcasing near identical erasure strength and benign utility. Over all 1,200 CHC-Bench rows, the QPS and SUM directions differ by less than three percentage points in removal when both are scaled to the same norm.

  • Furthermore, we figured out that extra magnitude costs content, a cost that grows as more concepts are targeted (refer to our response to Reviewer FKwW, Q1)


Weakness 3. The contribution of this paper appear to be incremental.

We agree that guidance conflicts in multi-concept suppression were identified in prior work, and we position our contributions relative to it.

  • Finding. CASG resolves conflicts by applying one category's correction per step. We show that applying one correction per step becomes less reliable as undesired concepts co-occur: on CHC-Bench, the advantage of QPS over top-1 grows with the number of targets per prompt (Table 2), and it persists without the oracle when all 21 concepts are guided.
  • Method. Our contribution is to formulate QP as a sampling-time aggregation strategy whose constraints retain each active correction's standalone directional effect at minimum norm, together with the analysis of how it relates to averaging, summation and top-1. QPS needs no training and applies unchanged to four constructors on SD1.5 and FLUX.
  • Benchmark. CHC-Bench follows the construction pipeline of CoME-Bench, which covers visually concrete concepts such as characters and objects. Some harmful attributes are mostly defined by an act or state (e.g., a firearm counts only when aimed at someone), so CHC-Bench adds act-level definitions and judge questions for them, and composes them with characters and with each other, covering different scenarios.

References: [1] Du et al., VIEScore2: Unified Image Evaluation with Spatially Grounded Explanations. arXiv 2026

Thank you for the additional experiments and clarifications. They address some of my concerns, but the following issues remain:

  1. The experiments show that a larger correction scale leads to stronger suppression, but do not establish an advantage of QPS over a suitably scaled SUM. On HARM+HARM, SUM scaled to the QPS norm achieves higher removal rates and better CLIP, CLIP-I, and FID than QPS. Therefore, the benefit of QPS’s aggregation direction remains unclear. For practical deployments, the priorities should be low ASR on harmful prompts and high image quality on benign prompts. I am not convinced that preserving benign content within harmful prompts justifies weaker suppression.

  2. The additional SLD-strength and CFG-scale experiments show lower ASR alongside degraded image quality on benign prompts: lower CLIP/CLIP-I and higher FID. Could the authors compare ASR at matched benign image quality against top-1 and SUM to establish whether QPS offers a better safety–utility trade-off?

As an additional question, in practical deployment, providers do not know whether a prompt will contain one or multiple undesired concepts. How does QPS perform on prompts containing a single harmful concept or character while the full set of suppression targets remains enabled?

Given these remaining concerns, I am maintaining my current score, but I would be happy to raise it if they are satisfactorily addressed.

Reviewer_FKwWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to aggregate multiple safety-guidance corrections when several undesired concepts co-occur in a single text-to-image generation. The authors first build CHC-Bench, which combines 16 copyrighted characters and 5 harmful attributes into prompts with 2-4 targets, and keeps only the prompts whose vanilla FLUX.1-dev renderings contain every target. Then the authors propose QPS, which finds the minimum-norm correction whose projection onto each active correction is at least that correction’s own norm, and solve it through a low-dimensional dual by support enumeration. On CHC-Bench, QPS removes targets better than top-1 selection and averaging. On the conventional benchmarks, QPS achieves the lowest ASR among the non-summation rules on SD1.5 at some cost in benign utility, and a small gain over top-1 on FLUX.

Critical strengths
  • The problem is relevant. A malicious user can easily request several undesired concepts in one prompt, and the top-1 strategy of CASG was designed to avoid harmful conflict but was never tested on prompts where every target has to be removed. CHC-Bench fills this gap by extending CoME-Bench from characters and objects to harmful concepts, and it's a meaningful contribution for the community.
  • The formulation is clean, sound, and easy to analyze. Fig. 1 makes the difference among top-1, mean, sum and QPS clear at a glance, and the closed form in Eq. 6 shows exactly when QPS reduces to the mean and to the sum. In addition, QPS acts only on the supplied corrections, so it can be attached to different constructors.
  • The paper is generally well-written and easy to follow. The illustration in Fig. 1 is quite clear.
Critical weaknesses
  • My main concern with CHC-Bench is that the guidance is given the ground-truth targets of each prompt. Specifically, Sec. 5.1 and Appendix B.2 state that each row is guided only on its own 2-4 targets "as an oracle", whereas on the conventional benchmarks all seven categories are active at every step. However, a real-world defender does not know which of the 21 concepts will a malicious prompt request, so guidance has to run over the whole pool, which inevitably produces missed and spurious activations. This matters more for QPS, because its hard constraints retain every active correction at full strength. For example, by Eq. 6, a spurious correction at 120 degree to the true one gives a QPS update with twice the norm of a single correction, while mean shrinks such a correction and top-1 simply ignores it. Therefore, the oracle setting removes the failure mode to which QPS is most exposed, and the paper does not report how often absent concepts pass the DVE gate in Eq. 17. In addition, the character correction uses the row's neutral phrase as its anchor (Sec. 5.2), and this is the same phrase that appears in the cleaned prompt used as the reference for CLIP, CLIP-I and FID. As a result, the preservation results in Table 2 are likely optimistic. A subset guided on all 21 concepts with a generic anchor, together with the false activation rate, would make the conclusion much more convincing.
  • The methodological novelty feels quite limited, and the paper does not discuss the closest line of prior work. The QP in Eq. 4 is a projection onto an intersection of half-spaces, which is the same structure as GEM [1] in continual learning, where the update is projected so that ⟨g~,gk⟩≥0\langle \tilde g, g_k \rangle \ge 0 for every past task and the problem is solved through a dual QP with one variable per constraint. Compared with GEM, QPS changes the threshold from 0 to ∥di∥22\|d_i\|_2^2 and projects the zero correction, whereas GEM projects the current gradient. Closely related are also MGDA [2], PCGrad [3] and CAGrad [4], which aggregate conflicting gradients in multi-task learning. None of these works is discussed, so I currently view the contribution as an application of an established projection to guidance aggregation. In addition, several multi-concept guidance methods are cited in Sec. 2.2 but not compared. For example, CRG [5] solves a minimum-deviation update and handles multiple concepts by sequential projection, so it is the most direct test of whether joint projection is actually needed. Mosaic [6] targets the same compositional multi-concept setting on flow-based models, and CHC-Bench is built on its CoME-Bench. Since Mosaic applies concept-specific spatial masks, it may avoid the conflict altogether when targets occupy different regions, as in the first row of Fig. 2. SGF [7] pools all concepts into one negation set, so the aggregation problem does not arise in the first place. CRG may be regarded as concurrent work, but Mosaic and SGF are not, and without these comparisons it is hard to tell where QPS stands.
  • More broadly, I am slightly skeptical about the practicality of this inference-time route for multi-concept safety. Each guided concept adds its own conditional predictions at every step, e.g., two extra forward passes per concept for DVE (Eq. 17), so guiding the full 21-concept pool on FLUX requires 43 forward passes per step against one for vanilla sampling. This cost is shared by top-1, sum and QPS. However, model-editing & retraining methods such as UCE [8] and MACE [9] add no inference overhead regardless of the number of concepts. The paper also reports no wall-clock time or memory usage, and the OSQP solver suggested for large kk is not tested. Therefore, it remains unclear whether QPS is usable once the concept list reaches a realistic size.

References:

[1]: Lopez-Paz and Ranzato, Gradient Episodic Memory for Continual Learning. NeurIPS 2017

[2]: Sener and Koltun, Multi-Task Learning as Multi-Objective Optimization. NeurIPS 2018

[3]: Yu et al., Gradient Surgery for Multi-Task Learning. NeurIPS 2020

[4]: Liu et al., Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS 2021

[5]: Choi et al., Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling. ICML 2026

[6]: Ko et al., Mosaic: Compositional Multi-Concept Erasure via Vector Field Blending. arXiv 2026

[7]: Kim et al., Safety-Guided Flow (SGF): A Unified Framework for Negative Guidance in Safe Generation. ICLR 2026

[8]: Gandikota et al., Unified Concept Editing in Diffusion Models. WACV 2024

[9]: Lu et al., MACE: Mass Concept Erasure in Diffusion Models. CVPR 2024

Questions for authors
  • How do the CHC-Bench results change when guidance runs over all 21 concepts without the oracle, even on a subset of rows? In particular, does QPS still outperform top-1 in this setting?
  • How often do concepts absent from the prompt pass the DVE gate in Eq. 17? When such spurious activations occur, what is the distribution of the QPS update norm relative to a single correction?
  • If each character is guided with a generic anchor (e.g., "a cartoon character") in place of the row-specific neutral phrase, how do CLIP, CLIP-I and FID in Table 2 change?
  • What is the essential difference between QPS and the dual QP of GEM [1]? Moreover, how would MGDA [2], PCGrad [3] and CAGrad [4] perform when applied directly to the corrections did_i?
  • Could the authors compare QPS with CRG [5], Mosaic [6] and SGF [7], and also report QPS on CoME-Bench? In particular, does joint projection outperform the sequential projection of CRG?
5 comments

We have carefully considered all comments and would like to address concerns raised.

Q1. How do the CHC-Bench Results change when guidance runs over all 21 concepts without the oracle? In particular, does QPS still outperform top-1 in this setting?

  • Yes. Without the oracle, QPS removes every target in 42.7% of rows with the concept-specific anchor, against 27.3% for top-1, and its margin over top-1 is larger than under the oracle. Summation removes almost every target with either anchor, but its images collapse and lose the prompt content.

  • We reran the 300-row subset of Q2 (225 CHAR+HARM and 75 HARM+HARM rows) with all 21 concepts guided at every step in place of each row's oracle targets. We report both the row-specific concept-specific anchor of Table 2 and the generic anchor of Q3.

  • We also added VIEScore2 [1] for measuring benign utility preservation. SC (semantic consistency) rates how well the image follows the prompt. We score it against the cleaned prompt, so it measures how much of the requested benign content remains. PQ (perceptual quality) rates freedom from artifacts and distortions. All ∧ SC>0 is the share of rows in which every target is judged removed and the image still keeps some of the prompt content (SC > 0).

Neutral anchor (Table 2)

TargetsRuleCharHarmAllCLIP ↑CLIP-I ↑FID ↓SC ↑PQ ↑All ∧ SC>0
oracletop-160.445.823.025.450.595141.95.465.2022.3
oracleQPS76.350.331.725.480.594139.45.415.1931.0
oraclesum86.477.463.022.800.580162.13.594.2640.0
all 21top-168.646.427.324.790.558143.95.235.1027.0
all 21QPS84.962.442.724.160.554147.84.885.0142.3
all 21sum10099.098.316.150.459225.10.872.7938.3

Generic anchor ("a cartoon character")

TargetsRuleCharHarmAllCLIP ↑CLIP-I ↑FID ↓SC ↑PQ ↑All ∧ SC>0
oracletop-171.054.835.722.970.547147.65.085.0935.3
oracleQPS79.655.434.322.250.548146.64.915.0533.7
oraclesum92.984.474.718.570.514199.62.583.0236.7
all 21top-165.153.231.322.190.527155.44.885.0931.3
all 21QPS81.469.048.321.770.543158.74.695.0848.3
all 21sum10010010014.140.415322.30.281.4012.3

Q2. How often do concepts absent from the prompt pass the DVE gate in Eq. 17? When such spurious activations occur, what is the distribution of the QPS update norm relative to a single correction?

  • We measured this while conducting Q1's 21-concepts experiment.

    StrategySteps with a spurious sourceSpurious share of active sourcesActive sources per step
    top-186.1%78.8%5.7
    QPS85.9%80.5%5.6
    sum93.0%82.0%5.4
  • Spurious activations are frequent, so the DVE gate on FLUX is far from selective. This is a property of the constructor and affects every aggregation strategy.

  • The table below gives ∥δQPS∥/max⁡i∥di∥\|\delta_{\mathrm{QPS}}\|/\max_i\|d_i\| on the QPS steps that have a spurious source and at least two active sources. The ratio equals 1 when a spurious source is the only active one. It stays close to 1, so the corrections that pass the gate rarely conflict as strongly as the 120° case in the review.

    StepsActive sources (median / max)Medianp90Max> 2×
    6,3526 / 201.151.412.100.016%

Q3. If each character is guided with a generic anchor (e.g., "a cartoon character") in place of the concept-specific phrase, how do CLIP, CLIP-I and FID in Table 2 change?

  • With the generic anchor, preservation against the cleaned reference drops for both strategies: CLIP by 3.3–4.3, CLIP-I by about 0.06, and FID rises by about 11. The absolute preservation in Table 2 is therefore partly helped by the shared phrase, although removal also rises, so part of the drop reflects a stronger correction.

    RuleAnchorCharHarmAllCLIP ↑CLIP-I ↑FID ↓
    top-1concept-specific60.437.416.026.170.615128.4
    top-1generic71.050.432.922.870.550139.2
    QPSconcept-specific76.343.627.626.510.614125.3
    QPSgeneric79.651.031.122.200.552136.1
  • Yet the comparison between the strategies also holds: with the generic anchor, QPS and top-1 have similar CLIP-I and FID, and QPS has a slightly lower CLIP (22.20 vs. 22.87).

References: [1] Du et al., VIEScore2: Unified Image Evaluation with Spatially Grounded Explanations. arXiv 2026

Weakness. Practicality of inference-time safe guidance for multi-concept suppression.

  • We agree that guiding a large target concept set adds substantial inference overhead.

  • However, flexibility is one of the strength of inference-time safety guidance. Because the backbone weights remain unchanged, suppression policies can be revised by adding or removing the guidance terms and constraints. In particular, if a previously prohibited concept is later permitted, its correction is simply dropped from subsequent generations, whereas a model edited to erase that concept would have to be edited again or restored to its original weights.

  • Furthermore, QPS has no backbone architecture-specific component. It aggregates the corrections of existing guidance constructors, each designed for its backbone by its authors, so it applies to any backbone for which such a constructor exists. SD1.5 (UNet) and with DVE on FLUX.1-dev (MMDiT). Weight-editing methods such as UCE and MACE modify particular layers, e.g., the cross-attention projections of a UNet, and must be adapted to architectures without separate cross-attention.


  • The sampling time grows with the number of predictions per step: on FLUX, each additional prediction per step adds about 5.2 s per image (0.19 s per transformer call over 28 steps).

Time per image (s)

BackboneSettingPredictions per steptop-1sumQPS
FLUX.1-devVanilla15.65.65.6
FLUX.1-devCHC-Bench oracle, K=2525.825.825.8
FLUX.1-devCHC-Bench oracle, K=4946.346.346.4
FLUX.1-dev7 harmful categories1575.676.776.4
FLUX.1-dev21-concept pool41207–216206–215208–216
  • OSQP for large kk. We tested it on CHC-Bench guided on all 21 concepts. OSQP solved every step with k>12k>12, and on the steps with k≤12k\le12 its solution matched support enumeration within a relative difference of 10−1110^{-11}. One OSQP solve took about 9 ms at k=21k=21, which is negligible next to the conditional predictions.

  • Memory: in the 21-concept run, the memory in use on the FLUX generation GPU was the same for top-1 and QPS (23,696 MiB). The per-concept correction vectors and the QP buffers grow with the number of concepts, but did not change this reading in the measured setting.


For the question 5 - we're currently working on it; we'll respond to your concern ASAP.

What is the essential difference between QPS and the dual QP of GEM? Moreover, how would MGDA, PCGrad and CAGrad perform when applied directly to the corrections did_i?

We thank the reviewer for pointing out the connection of QPS to prior works. We have examined each work carefully, and concluded that Eq. 4 has the half-space projection structure of GEM's QP, so we do not claim novelty for the projection QP itself. The difference is in what is retained: QPS keeps di⊤δ≥∥di∥22d_i^\top\delta\ge\Vert d_i\Vert_2^2 for every active correction and none of the four methods guarantees this.

  1. GEM only prevents a correction from being reversed; QPS retains it in full. GEM uses zero thresholds in its constraints (Eq. 8 in [1]). With zero thresholds in place of ∥di∥22\Vert d_i\Vert_2^2, the modified QP is solved by δ=0\delta=0, so nothing is applied to the sampler update.

    • If we take one correction dpd_p as GEM's gradient (not just replacing the threshold), the problem min⁡δ∥δ−dp∥22\min_\delta\Vert\delta-d_p\Vert_2^2 s.t. di⊤δ≥0d_i^\top\delta\ge0 (i≠pi\ne p) only keeps the aggregated update from moving against the other corrections.
  2. MGDA retains every correction at one shared level; QPS retains each at its own level. MGDA returns the point closest to the origin in the convex hull of the corrections. Its guarantee is di⊤δ≥∥δ∥22d_i^\top\delta\ge\Vert\delta\Vert_2^2, and this shared level ∥δ∥22\Vert\delta\Vert_2^2 is never larger than min⁡j∥dj∥22\min_j\Vert d_j\Vert_2^2 and shrinks as the corrections conflict.

    • For two equal-norm corrections at angle θ\theta, MGDA returns their mean (d1+d2)/2(d_1+d_2)/2, while QPS returns (d1+d2)/(1+cos⁡θ)(d_1+d_2)/(1+\cos\theta), the same direction but longer. MGDA therefore retains only a fraction (1+cos⁡θ)/2(1+\cos\theta)/2 of each correction, whereas QPS retains each in full. Consistently, averaging gives the weakest suppression among the guided rules in Tables 2 and 3.
  3. PCGrad resolves a conflict by cutting the conflicting component out of each correction, whereas QPS lengthens the update until every correction is retained. PCGrad takes each correction in turn and, for every other correction it conflicts with (negative inner product), removes its component along that correction; the update is the sum of what remains.

    • For two conflicting corrections at angle θ\theta, only a fraction sin⁡2θ\sin^2\theta of each is retained, which is always below one.
  4. CAGrad stays within a radius proportional to the mean's length; QPS lets the constraints set the length of the update. CAGrad searches a ball around the mean correction, whose radius is a fraction c∈[0,1)c\in[0,1) of the mean's length, for the update that helps the worst-off correction the most.

    • Its update is at most (1+c)(1+c) times as long as the mean and shrinks with the mean as the corrections conflict, whereas the QPS update grows.

We also ran PCGrad on the 300 HARM+HARM prompts of CHC-Bench, with the oracle targets and seeds of Table 2. It inherits the sum's collapse: VIEScore2 gives SC = 0 to 55.3% of its images.

RuleTarget removal (%)All (%)CLIP ↑CLIP-I ↑FID ↓SC ↑PQ ↑All ∧ SC>0 (%)
top-160.540.723.250.524253.15.765.0440.3
mean56.535.023.600.568235.56.015.1635.0
sum93.088.318.060.518320.12.272.5034.0
PCGrad93.789.718.180.516319.62.202.5335.0
QPS66.346.022.990.537254.05.425.0343.3

Comments on the paper

1 comment
AuthorsOpenReview ↗

PDF revised

the reviewers and AC, we have updated the paper as follows (revised text is in blue):

  • CHC-Bench now contains 1,200 rows, and every image is regenerated at the benchmark's reference seed, so the CHC-Bench numbers differ from the submitted version.

  • Table 2 now uses cleaned references instead of vanilla images, which contain the targets and cannot show whether non-target content is preserved. CLIP is measured against the cleaned prompt, CLIP-I and FID against images generated from it, and we add VIEScore2 SC and the share of images with PQ=0.

  • Experiments added during the rebuttal:

    • all 21 CHC-Bench concepts enabled at every step (Sec. 5.2.2, Table 3);
    • CRG's sequential projection (CRG-Multi) on SD1.5 (Table 4, Sec. 5.4);
    • the size of the QPS update relative to sum, and sum with a constant scale (Sec. 6, Appendix C);
    • the relation to GEM, MGDA, PCGrad and CAGrad, with PCGrad on CHC-Bench (Sec. 4.2, Appendix B);
    • wall-clock time, OSQP for many active concepts, and near-infeasible steps (Appendix D);
    • retention after SLD's momentum update (Appendix F);
    • the number of images rejected by the QPS solver in each experiment (Sec. 5).

We did not include Mosaic as we have not conducted Mosaic on conventional benchmarks. SGF is also not included (Q5 from reviewer FKwW).

Again, we sincerely thank reviewers for their valuable comments for improving our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.