acceptodds
Under review as a conference paper at ICLR 2027

Mind the Modality Gap: From Constitutional AI to Contextual Policy-Alignment Through Language Bridging

Abstract

Most generative-model alignment methods encode a fixed notion of harmfulness, whereas deployed systems must often satisfy policies specific to a platform, application, or content domain. We introduce AnyCAI, an end-to-end framework that extends Constitutional AI beyond text through a language bridge. Given a natural-language constitution and seed requests, a generator produces an artifact, a modality-capable critic evaluates that artifact without seeing the generating prompt, and a text-only reviser converts the critique into a corrected request. The resulting artifact-grounded pairs adapt an accessible conditioning encoder while the generative backbone remains frozen. AnyCAI generalizes the target from safety to policy compliance: conventional harmlessness is recovered when the constitution specifies harmful content, while other constitutions can express deployer-specific requirements. We pair Policy Compliance Rate with semantic fidelity on policy-compatible requests and evaluate conventional safety, a selective policy relaxation, a marketplace policy, and policy composition across text-to-image, text-to-video, vision-language, and text-to-music settings. The reusable contribution is the complete constitution-to-data-to-alignment pipeline; its supervision pairs, adapters, and evaluation distributions remain policy-specific.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Constitutional AI, multimodal alignment, policy compliance, generative model safety, synthetic alignment data, representation alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gqKuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnyCAI turns a written policy into training pairs through a loop of generation, visual critique, and prompt revision. These pairs train a small conditioning-encoder adapter. Experiments show improved judge-measured compliance and demonstrate policy editing across several settings.

Critical strengths
  1. Editing a written policy gives a practical way to change model behavior. The rule-relaxation experiment shows that permitted content can be restored while retaining compliance with other rules.

  2. Separate evaluation judges and human audits provide useful checks. The paper reports judge errors openly.

Critical weaknesses
  1. The method depends heavily on the training critic, but its reliability is not adequately tested. The critic decides what needs correction and when a revision is suitable for training. A missed violation can become an accepted target, while a false alarm can cause permitted content to be removed. The paper audits separate evaluation judges, but those checks do not show how accurately the training critic applies the policy or how its errors affect the adapter.

  2. It is unclear how much the visual critique contributes. The method combines feedback on generated images with prompt revision and encoder adaptation, so the final improvement does not tell us which part matters most. Comparisons with a text-only critic or static pairs written from the policy would help show whether inspecting generated images provides better supervision.

  3. The general-purpose framing is stronger than the evidence beyond safety. The non-safety policies show modest gains and leave many violations unresolved. It is also unclear whether training an adapter offers a better trade-off than using the same policy to rewrite prompts or resample outputs at inference time. These comparisons matter because accepting an editable policy is a central part of the claimed contribution.

Questions for authors
  1. Please try to address weaknesses 1 and 2.

  2. On non-safety policies, how does AnyCAI compare with resampling and inference-time rewriting in compliance, fidelity, and cost? Please validate the main comparisons with human judgments.

2 comments
AuthorsOpenReview ↗

Response to Reviewer gqKu

We thank the reviewer for the detailed feedback. Questions shared with the other reviewers are answered in the general response (Tables G1–G3); we reply to each point below.

W1: Reliability of the training critic and how its errors affect the adapter.

  • Existing evidence. In the Marketplace corpus, a prompt-blind visual audit of the selected pairs rejected 1717 of the initial 9696 critic-accepted revisions (17.717.7%). These were replaced by audited reserves. This directly estimates training-critic precision on accepted targets for a hard non-safety policy.
  • New measurement. We re-judged every image of the Base GCR loop (681681 images from 500500 seed requests) with the evaluation judge (InternVL2-8B, per principle, prompt-blind). This judge is a different model from the training critic (GPT-4o).
    • Agreement. The critic's accept/revise decisions agree with the judge on 89.989.9% of images (Cohen's κ=0.73\kappa = 0.73).
    • Training targets are clean. Only the 170170 trajectories that were revised and then accepted become training pairs. The judge flags 11 of their 170170 final images (0.60.6%). On the same trajectories it flags 127127 of the 170170 original images, and 126126 of those are clean after revision.
    • Disagreements are mostly conservative. Of the 182182 images the critic sent back for revision, the judge finds 4949 compliant (26.926.9%). Such an error costs an unnecessary rewrite, not a violating target. The critic accepted at the first iteration 1919 of 329329 images (5.85.8%) that the judge flags. Accepted requests are not used for training, so these are missed training opportunities, not corrupted targets.
  • Effect on the adapter. Since almost every training target is clean, critic errors mostly cost an unnecessary revision or a missed training opportunity rather than reaching the adapter. In the closest controlled comparison (Xkc3 Q4), pairs written from the policy without any critique cost 4.34.3 PCR points on I2P.

W2: How much does the visual critique contribute? We ran the suggested comparisons: a text-only critic, static policy-written pairs and predefined pairs, all with the same adapter, objective, steps and NN (tables in our reply to Xkc3 Q4).

  • On safety: the critique–revision loop matters (+4.3+4.3 on I2P over static policy-written pairs), and a text-only critic is close to the image critic, because the violation is written in the request.
  • On natural requests: the image matters. A text-only critic predicts only 4.34.3%, 25.025.0% and 6.06.0% of the rendered people, promotional-graphics and framing violations of Marketplace, against 88.988.9% of background violations. In a matched ablation on 174174 natural product titles (same requests, NN and recipe; only the critic's input differs), adapters trained on image-critic pairs are more compliant than adapters trained on text-critic pairs on both benchmarks: +4.6+4.6 PCR points [0.8,8.3][0.8, 8.3] on Marketplace-Challenge and +2.8+2.8 [0.7,4.9][0.7, 4.9] on ESCI, at the same fidelity.

W3 / Q2: Non-safety policies: AnyCAI vs rewriting and resampling (compliance, fidelity, cost); modest gains; human validation. Marketplace, same judge and prompts for every method, mean over 33 seeds. Resampling regenerates with a new seed until the evaluation judge finds the image compliant, up to 33 attempts; with the evaluation judge as selector, this is an optimistic upper bound. Bold: best per column.

MethodPCR, ChallengePCR, ESCICLIP-S (ESCI)Added cost per request at deployment
Vanilla37.137.112.512.50.3330.333none
Negative prompt44.644.632.332.30.3280.328none
Resampling (Vanilla, up to 33 attempts)51.251.222.422.40.334\mathbf{0.334}2.72.7 generations and 2.72.7 judge passes on average (ESCI)
LLM rewrite90.8\mathbf{90.8}51.7\mathbf{51.7}0.3190.31911 LLM call
AnyCAI77.177.141.241.20.3230.323none
  • Resampling is weaker and costlier than the adapter. Even with the evaluation judge as selector, three attempts reach 22.422.4% on ESCI, about half of what AnyCAI reaches with a single generation (41.241.2%).
  • Rewriting is more compliant, at a recurring cost and lower fidelity. Its advantage comes almost entirely from the white-background rule, which can be anticipated from the request. On the violations that only the rendered image reveals (people, graphics, framing), AnyCAI has fewer than rewriting on each rule (any of the three: 24.324.3% vs 26.026.0%; reply to Xkc3 Q4), though each difference is within its 9595% CI.
  • Modest absolute compliance. We agree it is not saturated, and we will state this explicitly. Among the methods that add no inference cost, AnyCAI is the most compliant (ESCI 41.241.2% vs 32.332.3% for the negative prompt and 12.512.5% for Vanilla).
  • Human validation. With labels from two blinded annotators, AnyCAI's gain over Vanilla is +25.3+25.3 PCR points on Marketplace and +33.3+33.3 on T2V, in line with the paper's judge (reply to sL4d W2/Q3).
Reviewer_gqKuOpenReview ↗

Post-Rebuttal Assessment

Thank you for your responses.

The critic audit and matched text-only/static-pair comparisons address my first two concerns. AnyCAI does not clearly surpass LLM rewrite but is cheaper. Despite modest non-safety compliance and incomplete human validation, the new evidence addresses my main objections about the critic, visual feedback, and inference-time alternatives.

I increase my rating.

Reviewer_sL4dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ANYCAI, a framework for aligning multimodal generative models with natural-language policies. A modality-aware critic evaluates generated artifacts, while a text-only reviser constructs alignment pairs, using the resulting pairs to train lightweight encoder adapters through LoRA while keeping the generative backbone frozen.

Critical strengths
  1. The paper discusses an important issue of adapting generative models to different deployment-specific policies. Representing policies as editable and composable natural-language constitutions provides a flexible alternative for fixed safety alignment goals.

  2. The experiments cover multiple image-generation policies, text-to-video generation, and VLMs. And get substantial improvement on ViSU-test using 100 alignment pairs.

Critical weaknesses
  1. The paper lacks a controlled comparison with prompt rewriting guided by constitution that does not inspect generated artifacts, making it difficult to isolate the benefits of artifact-grounded GCR. Moreover, the use of policy-specific reviser prompts, data filtering procedures, and alignment configurations raises questions about the claim that adapting to a new policy only requires editing the constitution.

  2. The main conclusions in the paper rely heavily on automated judges, but their reliability varies depending on the policy. For example, Table 7 shows that on a small human-annotated sample, the false-negative rate for the Marketplace single-product rule is as high as 75%. In addition, text-to-video compliance is evaluated using only three sampled frames, which may miss brief violations or cases that requires temporal context. These issues make me wonder whether the reported PCR really reflects actual policy compliance.

Questions for authors
  1. Could the authors compare ANYCAI with prompt rewriting based on the constitution, without artifact-level criticism?

  2. The paper emphasizes adapting to new policies by editing the constitution. Could the authors clarify the end-to-end computational costs of generating alignment pairs and training an adapter for each new policy?

  3. How sensitive are the reported improvements to errors in automated judges? Could the authors provide additional human evaluations, particularly for the Marketplace and text-to-video settings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer sL4d

We thank the reviewer for the detailed feedback. Questions shared with the other reviewers are answered in the general response (Tables G1–G3); we reply to each point below.

W1/Q1: Constitution-guided prompt rewriting without artifact-level critique; policy-specific components. We ran this comparison in two forms:

  • (a) At inference time. The same LLM as our reviser, with the constitution in context, rewrites every request. Full results are in our reply to Xkc3 Q1. AnyCAI is higher on I2P and ViSU-test with higher fidelity. Rewriting is higher on Ring-a-Bell and on Marketplace overall compliance. Rewriting fails to apply the Weapons relaxation and costs n⋅kn \cdot k LLM calls at deployment.
  • (b) As a training-data source. The same adapter trained on policy-written pairs (no images, no critique) or with a text-only critic in the GCR loop. On Base, GCR pairs beat policy-written pairs by 4.34.3 points on I2P and 1.51.5 on ViSU-test; the text-only critic is close to GCR, because safety violations are written in the requests. On natural Marketplace requests, a text-only critic predicts only 44–2525% of the rendered people, graphics and framing violations. With the same 174174 natural requests and NN, adapters trained on image-critic pairs are more compliant than adapters trained on text-critic pairs (Marketplace-Challenge +4.6+4.6 PCR points [0.8,8.3][0.8, 8.3], ESCI +2.8+2.8 [0.7,4.9][0.7, 4.9]). Details are in our reply to Xkc3 Q4.

On policy-specific components, see our reply to Xkc3 Q2. Every policy goes through the same pipeline, with no curated dataset and a run cost of the same order. Weapons-permitted changes only the constitution. For Marketplace we report the four settings adapted to its compositional rules, and we will make the corresponding sentence of the abstract precise.

W2/Q3: Reliance on automated judges (Marketplace FNR, T2V frames); human evaluation.

  • Human evaluation (new). Two annotators, blind to method and prompt, judged items from the main comparisons against the full constitution (one screen per item, all rules): 160160 Marketplace images (8080 natural ESCI requests, Vanilla and AnyCAI, the images judged in the paper) and 100100 T2V videos (5050 prompts, Vanilla and AnyCAI N=307N=307). 4040 items were annotated by both (agreement 7070% on Marketplace, 9090% on T2V); we exclude the 88 on which they disagree, and resolving them either way moves the gaps below by at most 3.53.5 points. PCR (%):
SettingPairsVanilla, humansAnyCAI, humansGain, humansGain, paper's judge (same items)
Marketplace (ESCI)757516.016.041.341.3+25.3+25.3 [13.3,37.3][13.3, 37.3]+25.4+25.4
T2V484866.766.7100.0100.0+33.3+33.3 [20.8,47.9][20.8, 47.9]+31.2+31.2

The gains reported in the paper hold under human judgment. The paper's judge agrees with the human labels on 8989% of the Marketplace images (Cohen's κ=0.72\kappa = 0.72) and 9999% of the videos (κ=0.96\kappa = 0.96).

  • Marketplace FNR on the single-product rule (Table 7). The 7575% was estimated on 44 violating images. On the 4040 images that the annotators judged in violation of this rule, the judge misses 4040%; on the background rule it errs the other way (3030% false positives). Overall the judge is slightly stricter than humans, by about 77 points for both methods, so the gain is unchanged (table above). We will replace Table 7's estimate with this audit.
  • T2V frames. We re-judged all 4,5004{,}500 videos of the main T2V comparison (1,5001{,}500 per method) with all 1616 frames instead of 33, changing nothing else (same judge, rules, prompt and aggregation). Compliance barely moves and the conclusions are unchanged (PCR, %; bold: best per column):
Method33 frames (paper)1616 frames
Vanilla71.171.171.071.0
SafeCLIP-H96.596.596.696.6
AnyCAI (N=307N=307)98.9\mathbf{98.9}98.5\mathbf{98.5}

AnyCAI's gain over Vanilla is +27.9+27.9 points with 33 frames and +27.5+27.5 with 1616 (9595% CIs [24.3,31.5][24.3, 31.5] and [23.9,31.1][23.9, 31.1]); 1111 of the 1,5001{,}500 AnyCAI videos change verdict.

  • Independent judge. We re-judged the same Base images with an independent policy-conditioned judge, PolicyShiftGuard-7B (fine-tuned from Qwen2.5-VL-7B; the paper's judge for these images is InternVL2-8B), which is not used anywhere in our pipeline. Across 1111 methods it gives the same ranking as the paper's judge: Spearman ρ=1.000\rho = 1.000 on I2P, 0.9730.973 on ViSU-test, 0.9700.970 on Ring-a-Bell. It is uniformly stricter, so our conclusions do not depend on the judge.
AuthorsOpenReview ↗

Response to Reviewer sL4d – 2

Q2: End-to-end compute per new policy. Each seed request costs at most 44 generations, each judged by the critic, and 33 reviser calls (the loop makes at most 33 revisions). In the runs that produced the paper's training data:

PolicySeed requestsGenerations and critic callsReviser callsAPI (current GPT-4o prices)Generation time (one GPU, ≈8\approx 8 s per image)
Base500500681681182182≈\approxUSD 44≈1.5\approx 1.5 h
Marketplace9269261,8791{,}879953953≈\approxUSD 1010≈4\approx 4 h

Training the adapter takes under five minutes on one GPU and produces a 1313 MB adapter; at deployment the LoRA is merged into the encoder, with zero additional inference cost. By contrast, inference-time rewriting costs n⋅kn \cdot k LLM calls (median 0.90.9 s, ≈480\approx 480 tokens each) for every request served.

AuthorsOpenReview ↗

Reminder from the Authors

Dear Reviewer,

Thank you again for your detailed review, which has helped us strengthen our paper. We have addressed each of your points individually in our rebuttal, including new results and additional evidence. As the discussion period ends soon, we would be grateful if you could let us know whether our responses resolve your concerns, or whether any points need further clarification. If you find our answers satisfactory, we would kindly ask you to consider updating your score. Thank you for your time and careful consideration. Best regards, The Authors

Reviewer_Xkc3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper frames content moderation for generative models as policy alignment: a deployer writes a natural-language constitution, and given any prompt, the model should not generate what the constitution forbids, while still producing what the prompt asks for when it is permitted. To this end, the authors aim to redirect violating prompts toward compliant versions of themselves. Their method, AnyCAI, builds the training data for this redirection with a generate–critique–revise loop. A VLM critic compares the generated image against the constitution and reports any violations without seeing the input prompt, and a text-only reviser minimally rewrites the prompt based on the critique. Each pair of an original rejected prompt and its critic-accepted revision is then used to fine-tune a text encoder with LoRA, while the generative backbone stays frozen. At inference, the fine-tuned encoder maps a violating prompt to a representation close to that of its compliant revision, so the frozen generator produces a compliant output without any extra model call. Experiments cover four text-to-image policies, plus a safety policy for text-to-video and for a VLM’s vision encoder. The main claim is that AnyCAI lies on the empirical compliance–fidelity frontier in every setting.

Critical strengths
  • This is an interesting and relevant problem. Different deployments need different content policies. Existing steering methods are trained toward one fixed definition of unsafe content, so they cannot relax a rule or enforce rules outside generic safety (e.g. “no logos”) without new data and retraining. Guardrails, in turn, add inference-time cost and leave the generator unchanged.
  • Low cost. Only a small LoRA on the encoder is trained (under five minutes, 13 MB), while the generator stays frozen. The training pairs are produced automatically.
Critical weaknesses
  • The novelty is limited. Every component already exists (the generate–critique–revise loop, encoder redirection, VLM safety judges, LLM prompt rewriting). The only contribution is connecting these pieces together, with no new insight.
  • Poor presentation. The introduction never makes the problem concrete: it gives no example prompt, no expected output, and no measure of success (these only appear in 3.1 and 3.4). The key tension, that blocking forbidden content can damage allowed content, is explained only in 5.
  • Weak motivation. The paper motivates the problem only with hypothetical examples, with no evidence that existing methods actually fail in practice. It also never explains why a trained adapter is needed. The obvious cheap alternative, an LLM that rewrites each prompt using the constitution as context at inference time, needs no training, is just as easy to update, and is never discussed or compared.
Questions for authors
  • A simple way to enforce the policy is to give an LLM the constitution as context and have it rewrite every incoming prompt at inference time, then pass the rewritten prompt to the unmodified generator. This needs no training and is just as easy to update. How does this baseline compare with the trained adapter in compliance, fidelity, and inference cost or latency? What concrete advantage does the adapter offer over it?
  • The abstract says a new policy only requires editing the constitution. However, do the critic and reviser prompts and the read-out also have to change, and is the manual visual audit (as for the Marketplace pairs, Table 5) also needed? Roughly how much human effort did each policy take?
  • Are violating prompts redirected or just degraded? The paper argues that violating prompts are turned into their closest compliant version rather than refused. However, PCR only checks compliance, and CLIP-S is measured only on harmless COCO prompts. For violating prompts under the Base policy, how much of the allowed content is kept? For example, if the prompt is “a gun pointed at a bed with a pink blanket”, is the bed and blanket still drawn?
  • Does training on real failures matter? The introduction argues that pairs built from the model’s actual failures are better than predefined unsafe to safe pairs, such as the paired prompts in ViSU. Can you train the same adapter, with the same objective and the same number of pairs N, on each data source and compare the results? Without this, it is unclear whether the GCR loop contributes anything beyond providing training pairs.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Xkc3

We thank the reviewer for the detailed feedback. Questions shared with the other reviewers are answered in the general response (Tables G1–G3); we reply to each point below.

W1: Novelty. We agree that generation, critique, revision and encoder adaptation each exist in isolation. What does not exist is their combination for multimodal models. Constitutional AI and principle-driven self-alignment operate only on text (Sec. 2): the model critiques and revises its own output, and an image or a video cannot be revised that way. The steering methods for image, video and vision-language models are instead trained toward one fixed definition of unsafe content. Our answer to this modality gap is the language bridge (Sec. 3): a modality-capable critic is the only component that reads the output, and it states in natural language which principles the output violates; the correction stays text-to-text, on the request. To our knowledge, no prior method turns a written constitution into alignment data and an adapter for image, video or vision-language models. The contribution is this complete constitution →\rightarrow artifact-grounded data →\rightarrow encoder-alignment pipeline built on the language bridge, and the experiments in this rebuttal show it yields properties the individual pieces do not:

  1. Artifact-grounded critique finds failures that are not in the text. On natural Marketplace requests, 7575–9696% of the rendered violations of three of the four rules cannot be predicted from the request (see the reply to Q4).
  2. Distilling those corrections into the conditioning encoder gives zero inference overhead.
  3. The language bridge makes the pipeline modality-agnostic. Only the critic depends on the modality, and it judges the artifact without the prompt: the same loop applies unchanged to text-to-video and to a VLM's vision encoder, where there is no prompt to read.
  4. The same pipeline implements policy relaxation (Weapons-permitted). Erasure methods and inference-time rewriting both fail to restore the permitted content.

W2: Presentation. Thank you; we will revise the introduction as follows:

  • Open with a concrete running example: a request, the expected compliant output, and how compliance (PCR) and fidelity are measured.
  • State the compliance–fidelity tension there instead of in Sec. 5.

For example, the Marketplace request "one plain wooden recipe box, centered on a white seamless background … tie a separate red hanging tag to its handle" should produce the box on white without the tag. PCR checks the four marketplace rules; CLIP-S checks that the box is still the box.

W3: Motivation (evidence that existing methods fail; why an adapter).

  • Policy relaxation (Weapons-permitted). The paper shows this in Sec. 4.2.2 and Fig. 3 (third panel). On the prompts shared by all methods in this rebuttal (the set of Table G1), SafeCLIP and SafetyDPO, trained for a fixed notion of unsafe content, reach high compliance (98.598.5% and 97.297.2% PCR) but lose fidelity on the set that contains the permitted requests (CLIP-S 0.3200.320 and 0.3340.334, against 0.3700.370 for Vanilla). The two erasure methods we added behave the same way: RECE reaches 97.297.2% at 0.3240.324, and UCE keeps fidelity (0.3680.368) but stays near Vanilla's compliance (89.489.4% vs 87.387.3%). The Weapons adapters stay on the empirical frontier (92.292.2% PCR at 0.3650.365 with N=25N=25, 95.495.4% at 0.3550.355 with N=99N=99).

  • Flexibility. As the reviewer notes, existing steering methods are trained toward one fixed definition of unsafe content and cannot relax a rule without new data and retraining. In AnyCAI the policy is an input. The same method and pipeline give the Base policy from the Base constitution (96.596.5% PCR at CLIP-S 0.3430.343, N=150N=150) and the relaxed policy from a one-principle overlay (95.495.4% at 0.3550.355, N=99N=99), from 9999 automatically generated pairs and under five minutes of training.

  • Non-safety rules (Marketplace). No fixed safety method expresses these rules. The natural inference-time alternative, a negative prompt, reaches 32.332.3% PCR on natural ESCI titles versus 41.241.2% for AnyCAI.

  • Why an adapter rather than inference-time rewriting. See Q1: the recurring cost, the fidelity loss and the failure to relax a rule.

AuthorsOpenReview ↗

Response to Reviewer Xkc3 – 2

Q1: LLM rewriting baseline: compliance, fidelity, cost; advantage of the adapter. The rewriter is the same LLM as our reviser, with the constitution in context; it rewrites every request, and the rewritten prompt goes to the unmodified generator. Compliance on all three policies is in Table G1: rewriting is higher on Ring-a-Bell, Weapons-permitted and Marketplace, AnyCAI on I2P and ViSU-test. The adapter's advantages are cost, fidelity and policy relaxation.

Cost: rewriting pays at every request, AnyCAI once. Base policy; nn requests served, kk attempts per request.

LLM rewriteAnyCAI
Before deploymentnone681681 generations and critic calls, 182182 reviser calls (at most 44 and 33 per seed request), ≈\approxUSD 44, ≈1.5\approx 1.5 GPU-hours, under 55 minutes of training
At deploymentn⋅kn \cdot k LLM calls (median 0.90.9 s, p95 1.81.8 s, ≈480\approx 480 tokens each)none: the LoRA is merged into the encoder

With k=3k=3 this adds ≈2.7\approx 2.7 s (median) per request, about a third of an SDXL generation, and every request depends on an external LLM.

Fidelity: rewriting is lower on every policy. CLIP-S; bold: the better of rewriting and AnyCAI.

Policy (CLIP-S prompts)VanillaLLM rewriteAnyCAI
Base (COCO)0.3590.3590.3490.3490.356\mathbf{0.356}
Weapons-permitted (COCO + permitted weapons)0.3700.3700.3440.3440.355\mathbf{0.355}
Marketplace (ESCI)0.3330.3330.3190.3190.323\mathbf{0.323}

Rewriting alters 98.898.8% of requests, including benign ones (e.g., it drops "young" from "a young man concentrates during a baseball game").

Policy relaxation: rewriting does not use it. Weapons-permitted; rewriting with the relaxed constitution, seed 11. Bold: best per column.

ModelPCRCLIP-S (COCO + permitted weapons)
AnyCAI, Base policy96.5\mathbf{96.5}0.3430.343
LLM rewrite, Weapons policy96.296.20.3440.344
AnyCAI, Weapons policy (N=99N=99)95.495.40.355\mathbf{0.355}

Given the relaxed constitution, the rewriter behaves like a Base-policy model (same compliance and fidelity) and still removes the weapon from 3636% of the permitted-weapon requests that contain one (some of which also contain a threat). AnyCAI trained on the relaxed constitution has higher fidelity at similar compliance.

Where rewriting's compliance advantage comes from. On Ring-a-Bell, the violation is spelled out in the request (adversarial tokens), and the rewriter simply deletes it. On Marketplace, it comes almost entirely from the white-background rule, which is predictable from the request: the rewriter appends "on a white background" to natural titles and deletes the explicit requests for a coloured background in Marketplace-Challenge. The other three rules (no people, no added promotional graphics, a single centered product) are mostly violated by what the generator adds on its own: a text-only reader predicts only 44–2525% of these violations from the request (Table G2). A rewriter sees only the request, so it cannot target them; AnyCAI's training pairs come from critiques of the rendered images. On the requests where these violations are text-unpredictable, AnyCAI has fewer of them than rewriting on each of the three rules (table in Q4 below; any of the three: 24.324.3% vs 26.026.0%) at zero inference cost, though each difference is within its 9595% CI. With everything else matched, adapters trained on image-critic pairs are more compliant than adapters trained on text-critic pairs (Table G3).

Q2: Does a new policy only require editing the constitution? Human effort. Whatever the constitution says, a new policy goes through the same pipeline: the same code, critic and reviser models, loop and adapter, and no curated dataset. What changes is the input, a constitution and a pool of seed requests, and the cost of a run is of the same order for every policy (Table 5: 500500–926926 seed requests and 9999–307307 retained pairs; for T2I, under five minutes of training). We will make the abstract precise about which settings may also change:

  • Base →\rightarrow Weapons-permitted: only the constitution changes (a use-case overlay). Critic and reviser instructions, loop parameters and training configuration are identical (Sec. 4.2.2).
  • Marketplace: its rules are compositional (background, people, added graphics, framing of a single product), and we adapted four settings of the same pipeline, all reported in the paper:
    • a stricter per-principle critic output schema (App. B);
    • a reviser instruction that asks for span-level edits (App. B);
    • a prompt-blind visual audit of the selected training pairs (Table 5);
    • a read-out selected on a held-out dev split (Table 10).

Human effort per policy is the same in every setting, Marketplace included: writing the constitution, which is the input of the method. No image or training pair is labelled by hand.

AuthorsOpenReview ↗

Response to Reviewer Xkc3 – 3

Q3: Redirected or degraded? Is the permitted content kept? We measured this directly. ViSU-test pairs every unsafe request with a human-written safe counterpart: the same scene without the violating element. For each method we compute CLIP-S between the image generated from the unsafe request and that safe counterpart ("retention"; higher means the permitted content is depicted), and CLIP-S between the image and the unsafe request itself (lower means the image moved away from the violation). Same 3,8603{,}860 requests for every method, seed 11. Bold: highest retention among the methods not trained on ViSU pairs; †\dagger trained on ViSU-train pairs.

MethodRetention ↑\uparrowSimilarity to the unsafe request ↓\downarrow
Vanilla0.2740.2740.3730.373
Negative prompt0.2700.2700.3410.341
ESD0.2680.2680.3600.360
SafetyDPO0.2710.2710.3320.332
UCE0.2760.2760.3720.372
RECE0.2610.2610.3190.319
LLM rewrite0.2840.2840.3350.335
AnyCAI (N=150N=150)0.290\mathbf{0.290}0.3450.345
Safe-CLIP$^\dagger$0.3030.3030.3060.306
Adapter on ViSU's predefined pairs$^\dagger$0.2990.2990.3250.325
  • AnyCAI redirects rather than degrades. Retention rises from 0.2740.274 (Vanilla) to 0.2900.290 (paired difference +0.016+0.016, 9595% CI [0.015,0.017][0.015, 0.017]), while similarity to the unsafe request falls from 0.3730.373 to 0.3450.345: the image moves away from the violation and toward the permitted scene. RECE moves further from the request (0.3190.319) but loses permitted content (retention 0.2610.261).
  • AnyCAI keeps more permitted content than every alternative not trained on ViSU pairs, including inference-time LLM rewriting (paired +0.005+0.005 [0.004,0.007][0.004, 0.007]). All paired intervals exclude zero.
  • Only methods trained on ViSU-train pairs score higher: Safe-CLIP (0.3030.303) and the adapter trained on ViSU's predefined pairs (0.2990.299). Their training targets are written in the same style as the reference captions, and such pairs exist only for this policy.
  • Retention grows with the number of GCR pairs (0.2840.284 at N=25N=25, 0.2900.290 at N=100N=100–150150).

We will add qualitative examples, including the reviewer's "a gun pointed at a bed with a pink blanket".

Q4 : Does training on real failures matter? Same adapter, objective and NN on each data source. Same trainer, objective, steps and N=150N=150 (seed 11). Bold: the best of the data sources available for any policy; ViSU's predefined pairs (†\dagger) exist only for this policy.

Training pairsI2PViSU-testRing-a-BellCLIP-S
GCR (image critic)89.4\mathbf{89.4}95.795.719.619.60.3570.357
LLM-written from the policy, no images85.185.194.294.219.019.00.359\mathbf{0.359}
GCR with a text-only critic87.687.696.0\mathbf{96.0}21.4\mathbf{21.4}0.3570.357
ViSU predefined safe captions$^\dagger$91.291.297.497.428.628.60.3570.357
  • The critique–revision loop matters. GCR pairs beat policy-written pairs by 4.34.3 points on I2P and 1.51.5 on ViSU-test.

  • Predefined pairs. Where a curated paired dataset exists for a policy, it can be evaluated with the same objective, and on Base ViSU's pairs are strongest. Such data exists only for conventional safety: there is no paired dataset for Weapons-permitted or Marketplace, which is the case AnyCAI targets. ViSU-test is also drawn from the same distribution as ViSU's training pairs.

  • The image versus the text. On Base, a text-only critic is close to the image critic, because in safety benchmarks the violation is almost always written in the request. Natural requests differ: on 250250 natural ESCI product titles, a text-only critic predicts 88.988.9% of the white-background violations that Vanilla renders, but only 4.34.3%, 25.025.0% and 6.06.0% of the people, promotional-graphics and framing violations (Table G2 in the general response).

    On the requests where the text-only critic predicts no violation of a rule, which is 245245–247247 of the 250250 for the people and framing rules, the violation rates are (%, 33 seeds; lower is better, bold: lowest):

Rule (text-unpredictable requests)VanillaNegative promptLLM rewriteAnyCAI
No people / body parts18.118.114.614.611.211.28.9\mathbf{8.9}
No promotional graphics5.25.23.03.03.23.21.9\mathbf{1.9}
Single, centered, fully visible product25.725.724.124.120.320.318.9\mathbf{18.9}
Any of the three (226226 requests)36.436.429.929.926.026.024.3\mathbf{24.3}

Matched Marketplace ablation (Table G3 in the general response). With the same 174174 natural titles and NN, image-critic pairs give higher compliance than text-critic pairs: +4.6+4.6 PCR points [0.8,8.3][0.8, 8.3] on Challenge and +2.8+2.8 [0.7,4.9][0.7, 4.9] on ESCI (paired, bootstrap over requests), at the same fidelity. The image critic corrects a people, graphics or framing violation in 6565 of the 174174 pairs, the text-only critic in 3232.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers. All three ask the same two questions, and we ran new experiments to answer them:

  1. Inference-time LLM rewriting with the constitution in context (Xkc3 Q1, sL4d Q1, gqKu Q2). We use the same LLM as our reviser (gpt-4o-mini, pinned snapshot, same temperature), given the constitution and each incoming request, and asked to make the smallest change that makes the output compliant (returning compliant requests unchanged). The rewritten prompt goes to the unmodified generator. We evaluate it on three of the four T2I policies: Base, Weapons-permitted and Marketplace.
  2. Does the artifact-grounded critique matter? (Xkc3 Q4, sL4d W1, gqKu W2). We train the same adapter with the same objective, steps and NN on (i) ViSU's predefined safe captions, (ii) pairs written by the LLM from the policy without generating or inspecting images, and (iii) the GCR loop with a text-only critic that reads the request instead of the image. We also measure how many of the violations the generator actually renders a text-only critic could have predicted.

Table G1: Inference-time LLM rewriting vs AnyCAI. PCR (%) is judged as in the paper, on the same prompts for every method; CLIP-S measures fidelity. Mean over 33 seeds (LLM rewrite on Weapons-permitted: 11 seed). AnyCAI is the paper's adapter for each policy (Base N=150N=150, Weapons-permitted N=99N=99, Marketplace). Bold: the better of rewriting and AnyCAI.

PolicyMetricVanillaLLM rewriteAnyCAI
BasePCR, I2P74.774.784.984.988.3\mathbf{88.3}
BasePCR, ViSU-test77.477.495.495.496.0\mathbf{96.0}
BasePCR, Ring-a-Bell7.97.937.9\mathbf{37.9}20.620.6
BaseCLIP-S (COCO)0.3590.3590.3490.3490.356\mathbf{0.356}
Weapons-permittedPCR87.387.396.2\mathbf{96.2}95.495.4
Weapons-permittedCLIP-S (COCO + permitted weapons)0.3700.3700.3440.3440.355\mathbf{0.355}
MarketplacePCR, Challenge37.137.190.8\mathbf{90.8}77.177.1
MarketplacePCR, ESCI12.512.551.7\mathbf{51.7}41.241.2
MarketplaceCLIP-S (ESCI)0.3330.3330.3190.3190.323\mathbf{0.323}

Table G2: Which violations can be predicted from the request? Marketplace, 250250 natural ESCI product titles, Vanilla generator, 33 seeds. For each rule: how often the rendered image violates it, and what fraction of those violations a text-only critic predicts from the request alone.

Marketplace ruleRendered violations (%)Predicted from the request (%)
White background82.982.988.988.9
No people / body parts18.718.74.34.3
No promotional graphics6.46.425.025.0
Single, centered, fully visible product26.826.86.06.0

Table G3: Image critic vs text-only critic, matched. Marketplace. The same 174174 natural ESCI product titles (disjoint from the test set) are revised by the GCR loop twice, once with the image critic and once with a text-only critic that reads the request; everything else is identical (critic model, reviser, constitution, adapter recipe, NN). Mean over 33 seeds. Bold: the better of the two. Both adapters train only on natural titles, so their Challenge compliance is below the paper's adapter.

Training pairsPCR, ChallengePCR, ESCICLIP-S (ESCI)
GCR with a text-only critic44.644.643.943.90.322\mathbf{0.322}
GCR with the image critic49.2\mathbf{49.2}46.7\mathbf{46.7}0.3210.321

Main findings:

  • Rewriting is a strong but costly baseline (Table G1). On Base, AnyCAI is higher on I2P and ViSU-test, with higher fidelity; rewriting is higher on Ring-a-Bell. On Marketplace, rewriting reaches higher overall compliance, almost entirely on the white-background rule. In all settings, rewriting has lower fidelity and alters 98.898.8% of Base requests, including benign ones.
  • Rewriting does not implement a policy relaxation. Under Weapons-permitted, rewriting keeps the fidelity of a Base-policy model and does not restore the permitted content.
  • Rewriting is a recurring cost. It adds n⋅kn \cdot k LLM calls at deployment (nn requests, kk attempts each). AnyCAI's cost is paid once and adds nothing at inference.
  • The critique–revision loop matters. On Base, with the same adapter, objective and NN, GCR pairs beat pairs written by the same LLM from the policy without generating any image by 4.34.3 PCR points on I2P and 1.51.5 on ViSU-test.
  • Artifact-grounded critique catches failures text cannot predict (Table G2). On natural Marketplace requests, a text-only critic predicts 8989% of the rendered white-background violations, but only 44–2525% of the rendered people, promotional-graphics and framing violations.
  • The image critic produces better training data than a text-only critic (Table G3). With the same 174174 natural product titles, NN and recipe, adapters trained on image-critic pairs are more compliant on both Marketplace benchmarks (paired differences: +4.6+4.6 PCR points on Challenge, 9595% CI [0.8,8.3][0.8, 8.3]; +2.8+2.8 on ESCI, [0.7,4.9][0.7, 4.9]), at the same fidelity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.