acceptodds
Under review as a conference paper at ICLR 2027

Constitutional adapters: Inference-time interventions for misalignment and misuse

Abstract

Training models to act in accordance with an explicitly defined set of principles, or "constitution," has shown promise as a robust and transparent mechanism for AI alignment. However, the generality and flexibility of such methods remain unclear. Here, we show that constitution-consistent behavior can be distilled from synthetic corpora into lightweight objects (low-rank adapters and steering vectors). Despite never seeing a harmful request or jailbreak during training, such objects increase jailbreak defense success and measured alignment—particularly at long context lengths and against multi-turn attacks, where they outperform both prompted and steered baselines. Subtracting control-trained from constitution-trained objects further accentuates these effects, yielding defenses we call "constitutional adapters" (CAs). CAs can be trained on a base model, transferred zero-shot to its post-trained checkpoint, and scaled at inference time to predictably trade off defense for benign compliance. Taken together, these results recommend CAs as a lightweight, portable, and tunable lever for mitigating misalignment and misuse in API deployments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, alignment, misuse, adversarial robustness, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ku5qOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces constitutional adapters (CAs) trained on synthesized corpora with a differential idea to substract from a controlled text group. During inference time, CAs can be applied with a tunable strength to achieve zero-shot transfer for improving the jailbreak defense, especifially for long padding and multi-turn attacks. The experimental results show the proposal effectively defend the attack and is more effective an system prompts and simple refusal steering ,demonstrating a better trade-off between over-refusal and general capability. The authors position CAs as a portable inference-time lever against both misuse and misalignment.

Critical strengths
  • The investigated alignment problem is important and the inference-time lightweight approache might be more practically relevant for API deployments. Framing constitution-derived behavior as a small, transferable adapter is a sensible systems goal.

  • The idea of of training CAs using matched control corpus + task-arithmetic / activation difference, is interesting. The paper shows control-only objects can hurt safety while difference objects help. Combined with long-context padding and multi-turn evaluations, this yields the paper’s most convincing empirical story.

Critical weaknesses
  • Claude-centric pipeline bias is under-examined relative to the generality claimed. Corpus generation, rewriting, filtering, and much of the judging stack sit inside the Claude family (Opus/Sonnet/Haiku). Gains may partly reflect transferring Claude-like stylistic and preference patterns into open models, as Claude was the most aligned models as stated by the author, rather than a portable “constitution” signal. Without stronger cross-judge, cross-generator, or non-Claude constitution controls, it is hard to justify that the pipeline is generalizable beyond this ecosystem.

  • The title claim that CAs jointly mitigate misuse and misalignment is overstated. Jailbreak / attack results are interesting and comparatively well supported. Misalignment evidence (e.g. eval awareness) is narrower, judge-dependent, and does not warrant equating “lower scores on these probes” with robust misalignment mitigation.

Questions for authors
  • It is nice to have cross models results such as Gemma and Qwen; Will the corpus generated by a less aligned models also shows the similar effect to sufficiently defend the attack?
  • Will the datasets be released for future studies?
1 comment

Response to Reviewer ku5q

Thank you for the useful comments, and for noting the importance of our problem setting and the interest of the matched-control idea.

W1 – Claude-centric pipeline. Reviewer wnJa shared this concern. As noted in the general response, we have replaced all evaluation judges with non-Claude models (MiniMax-M3 and GLM-5.3), and the paper's relative conclusions largely hold. Every judged figure and table in the manuscript has been updated accordingly. The control corpus was written by the same Claude pipeline in a matched register, yet objects trained on it weaken static and multi-query defense (Figures 3–4), so Claude's style alone does not produce the effect. As a cross-generator control, a constitution corpus written entirely by a non-Claude model still raises mean defense in every attack family, though less (Q1 below). The new Limitations section also acknowledges that Claude wrote our corpora, based on Claude's constitution.

Revision: Sections 2.4, 3 and 5.1; Appendices D.7 and D.8

W2 – Misalignment results are less convincing. Our misalignment evidence rests on Petri (we removed the agentic evaluation; see k73x W4), where constitution and especially difference objects raise the alignment score substantially (difference ReLU steer +3.4 points on the 10-point scale), whereas the safety and constitution system prompts add only 0.9 and 0.3 points (the latter not significant). These effects change little if at all under the original Claude judge (Appendix D.7), showing judge-independence and robustness.

We agree that automated audits like Petri are not perfect, particularly at frontier scale, given evaluation awareness as well as what are likely to be more severe, out-of-distribution incidents (such as OpenAI-HuggingFace). However, at the 4B scale that is our focus here, we believe that Petri still provides useful signal, so we have kept the title. We are not aware of better methods for probing alignment at this scale, and we believe that in practice they have been useful for bootstrapping more aligned models (reported in the Claude 4 System Card). We have therefore elected to keep the title claim, and we hope that the additional judge results convince this reviewer that such a claim is warranted.

Revision: Sections 3.3, 5 and 5.1; Figure 5; Appendix D.6 (Figure 14)

Q1 – Corpora generated by less aligned models. This is a very interesting question. We suspected that the answer would be yes, given the multi-stage review pipeline (Figure 1) and the fact that during generation, constitutional information resides in the prompt and not simply the weights. However, generating these 40M+ token corpora is an expensive proposition (more than USD 10k), and restarting the whole pipeline from scratch was infeasible in the allotted review period. We therefore attempted a smaller-scale experiment, generating a constitution corpus with Qwen3.8-Max and the identical filtering pipeline, and training on 1M tokens of it.

Constitution LoRAs trained on the Qwen-written corpus still raise Qwen3.5-4B's mean defense in every attack family, though less than LoRAs trained on the same amount of Opus-written text: they raise defense success by 7.0, 9.5 and 8.6 pp on static, multi-query and multi-turn attacks, against 14.7, 38.0 and 23.3 pp for the Opus-written LoRAs, with less over-refusal and no capability difference. Part of the defense therefore does not depend on a Claude model having written the corpus, but the generator affects its size, at least through a pipeline tuned on Opus output.

Revision: Appendix D.8 (Figure 17), with pointers in Sections 2.1 and 5.1 and in Appendix A

Q2 – Will the datasets be released for future studies? Yes. The Reproducibility Statement says that all code and synthetic data on which the results depend have been uploaded to GitHub. The link is not provided yet to maintain blinding but will be included in the final paper upload.

Please let us know whether these responses address your concerns. If they do, we’d be grateful if you would reflect that in your assessment. If not, we’d welcome the chance to respond to anything outstanding before the discussion period closes.

Reviewer_wnJaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Constitutional Adapters (CAs), a lightweight tool to improve LLM alignment. The authors use Claude constitution to generate the training corpus, and use them to generate activation steering and parameter steering adaptors. By additionally subtracting the control part, the authors are able to generate the Difference part. Experiments across multiple models, tasks and scenarios show that CAs improve jailbreak defense and reduce misalignment, especially in long-context and multi-turn settings.

Critical strengths
  • The motivation for this paper is clear and interesting. Improving model safety using only constitution data without access to harmful or jailbreaking data is a promising research direction, and I do think this lightweight approach has practical value.
  • Experiments are comprehensive and well conducted, the authors test multiple recent models across many safety benchmarks, attack methods, and defense methods. For attack methods, they consider single-turn, multi-turn, and multi-query scenarios using multiple datasets. The authors also measure over-refusal and model capabilities to make the conclusion more general.
Critical weaknesses
  • In the experiments, the authors utilize Refusal Steer as a baseline method compared to activation steering methods. However, when it comes to LoRA as the parameter steering part, there is no corresponding baseline for that. This is one of my greatest concerns for this work, and please find more information in the Questions section.
  • As shown in Figure 6, the main approaches proposed by this paper (Difference ReLU steer and Difference LoRA) both perform relatively weakly on tool-using and coding tasks compared to other methods. Unfortunately, nowadays tool-using and coding are the most important capabilities of agents.
Questions for authors
  • As you mentioned in Section 2.1, you have already tried using LoRA adaptors with harmful requests or successful jailbreaks, but they did not perform well in terms of generalizability. Can you also report those results for all LoRA-related experiments? This could also serve as the baseline for the parameter steering approach, as discussed in the Weakness section. I would suggest the authors include those results in the rebuttal phase and revision.
  • Currently, all closed-source models used in this paper are from the Claude family. Opus 5 and Fable 5.1 generate the training corpus, while Haiku 4.5, Sonnet 5, and Opus 5 serve as judges in different tasks. This would potentially induce evaluation bias, which makes it hard to assess the generalizability of the proposed models. Can the authors replace the judges with models from different families like GPT and Gemini?
3 comments

Response to Reviewer wnJa

Thank you for the helpful feedback and the broadly positive review, emphasizing the clarity of the motivation and the comprehensiveness of our evaluations.

W1 and Q1 – LoRA baseline distilled from the distribution of refused jailbreaks. We agree this baseline would be instructive, so we have added it in the revision. On Qwen3.5-4B, we distilled the safety system instruction (SI) and, separately, the constitution SI into LoRAs and ReLU steers, trained to match the prompted model on 4,582 harmful and jailbreak prompts it refused and as many benign prompts it answered. On static attacks, which come from their own training distribution, these baselines roughly match our constitution-corpus objects (the constitution-SI LoRA falls 4.9 pp short). On multi-turn attacks, outside that distribution, the constitution-corpus objects raise multi-turn defense success 2.0× (ReLU steer) and 4.4× (LoRA) as much as the best distilled object of each type, and the difference objects defend more still. These results are now included in an Appendix because we worry that adding four new objects with three seeds each would overly clutter the main figures, but we are happy to do so if this reviewer deems it essential to include those baselines in the main text.

Revision: new appendix on the distilled baselines; more precise description in main text (Section 2.1; Appendix D.9; Figures 18 and 19; Table 5)

W2 – Weak performance on tool use and coding. We agree this is an important limitation. At the default dose on Qwen3.5-4B, difference LoRAs reduce BigCodeBench and BFCL pass rates by ~21 and ~14 pp, and difference ReLU steers by ~8 and ~13 pp; the new Limitations section states this cost. Two considerations bear on its practical significance. First, our intended deployment applies CAs adaptively: a session- or user-level monitor raises α only when abuse is suspected, so benign coding and agentic sessions would run unsteered (α = 0) and incur no cost.

Second, the ReLU steers' tool-use cost does not require constitutional content. On Qwen3.5-4B, 79% (control-corpus ReLU steer) and 85% (constitution ReLU steer) of their new BFCL failures contain no tool call at all and almost none are malformed calls, and the control steer, trained on a matched corpus without constitutional content, stops calling as often and loses more accuracy. This points to the prose-only format both corpora share; BigCodeBench losses, by contrast, are mostly incorrect code. Including example tool calls in a more diverse synthetic corpus may mitigate the cost, as the Limitations now note.

Revision: cost stated in the Limitations (Section 5.1); the claim that differencing reduces the cost narrowed to Qwen3.5-4B (Section 5.2); MMLU and GSM8K statement corrected (Section 3.4; Appendix C); new error-analysis appendix (Appendix D.10; Table 6)

Q2 – Possibility of Claude self-evaluation bias. We agree that this is a practical concern, shared with reviewer ku5q. As described in the general response, we have replaced all evaluation judges with non-Claude models (MiniMax-M3; GLM-5.3 for Petri) and updated every judged figure and table accordingly. Section 2.4 now gives the reason, and notes that our conclusions were preserved under the original Claude judges as well. GPT and Gemini models were not available to us through our API provider during the rebuttal period.

Revision: Section 2.4; Appendix D.7

We’re glad you found the work valuable. If these additions strengthen your assessment of the paper, we would be grateful if you would consider reflecting that in your score.

Thank you for your further clarifications and additional experiments, which address most of my concerns and strengthen the paper. I am happy to raise my score to 4.

Thank you for engaging with our responses and for raising your score. We appreciate it, and your suggestions have made the paper stronger.

Reviewer_k73xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether behavior learned from constitution-based synthetic text can be transferred to a deployed language model through small, adjustable interventions. It evaluates these interventions against jailbreaks and simulated misalignment, alongside their effects on benign responses and task performance. The results show different trade-offs across prompting, activation steering, and adapter-based interventions, with control-corpus subtraction having effects that depend on the evaluation and intervention type.

Critical strengths
  1. Training a separate intervention on a control corpus makes the comparison more informative than simply contrasting constitutional training with an unchanged model. It provides a way to examine whether the observed behavior reflects constitutional content or other consequences of synthetic-text training.
  2. The treatment of nonlinear steering is mathematically explicit. Equation 4 subtracts the outputs of two ReLU functions evaluated on the same residual state, rather than assuming that subtracting their parameters produces an equivalent intervention.
  3. The base-to-instruct transfer setup addresses a practical deployment question. The additional results across four model families also help establish how much the findings depend on the main Qwen checkpoint.
  4. The evaluation includes benign compliance, coding, and tool use rather than treating reduced harmful compliance as sufficient evidence of a successful intervention. This exposes costs that would be missed by reporting knowledge and mathematics scores alone.
Critical weaknesses
  1. The absolute baselines in the defense tables appear to use a different metric from the reported improvements. Table 6 lists 99.2 for unsteered Crescendo and then a defense improvement of 34.3 points for difference LoRA. These cannot both be defense-success percentages. Similarly, Table 5 gives 76.1 for unsteered Best-of-75 and a 71.1-point improvement. The baselines seem to be attack-success rates, but the captions and general reporting convention do not make this explicit. Please clarify the metric and direction of every baseline so that absolute defended performance can be reconstructed unambiguously.
  2. Model selection is asymmetric between the two training conditions. Appendix B uses a constitutional holdout for early stopping both constitutional and control objects, and all six control runs stop early while all six constitutional runs reach 16 epochs. Hyperparameters are also selected on constitutional data. This makes the subtraction depend on two updates selected under different relationships to the validation distribution. Please explain why constitutional validation loss is an appropriate stopping criterion for control training, and how unequal training progress affects the interpretation of subtraction with a fixed coefficient of one.
  3. The geometric evidence does not fully establish that subtraction cancels register while preserving constitutional content. Near-zero cosine similarity with one refusal direction rules out close alignment with that vector, but not a broader refusal-related effect. Moreover, applying the negative control intervention still harms defense in Figure 9, indicating that behavior is not simply reversed by changing the update’s sign. Please clarify how these observations support the proposed decomposition, particularly given the input-dependent ReLU gates and the fact that directional cosine similarity does not describe intervention magnitude.
  4. The agentic leaking result is difficult to separate from reduced ability to execute tool-mediated actions. Control ReLU steering reduces leaking by 19.1–27.8 points in Table 8 despite weakening jailbreak defense, while lowering BFCL performance by 33.4 points in every seed in Table 10. Constitutional ReLU steering nearly eliminates leaking but also loses 28.5–29.5 BFCL points. Please explain how the existing transcripts distinguish a deliberate decision not to leak from malformed calls, failed execution, or broader suppression of tool use. Without that distinction, reduced harmful action is not necessarily evidence of improved alignment.
  5. The claim of never seeing a harmful request during training needs a more precise operational definition. Constitutional advice scenarios are deliberately generated around pressure to violate norms or subvert oversight, using a seed about advancing the AI’s interests. The documented filters address refusals, realism, alignment, and named entities, but do not directly establish that every resulting user request is non-harmful. Please distinguish the absence of jailbreak templates from the absence of harmful requests, and clarify how the stronger claim was checked in the actual training corpus.
Questions for authors

Please refer to weakness.

3 comments

Response to Reviewer k73x

Thank you for the constructive comments.

W1 – Absolute baselines versus reported improvements. The initial submission reported the baseline as attack success rate (ASR), and the improvements as defense improvements; we agree that this was needlessly confusing on our part. Every results table now reports its first row in the same higher-is-better quantity as its cells (defense success rate, DSR = 100 − ASR; benign compliance; alignment score), so first row + cell approximates the value after the intervention. Your two examples now read 5.9% + 69.4 pp ≈ 75% DSR (Crescendo, Table 9, formerly Table 6) and 43.8% + 55.2 pp ≈ 99% (Best-of-75, Table 8, formerly Table 5), under the new judges (general response).

Revision: Tables 7–12 (formerly 4–7 and 9–10); App. D.11; Fig. 5

W2 – Model selection is asymmetric. We chose the constitutional validation loss as the stopping criterion for control training because we specifically wanted to control for shared tone, register and format, rather than teach the control adapters a bunch of (presumably irrelevant) content. However, we agree that this could in principle affect the interpretation of the difference objects, especially given that the coefficients were held fixed. We therefore trained new control objects (and derived new difference objects) using early stopping on the control validation corpus. No control fit stopped early: all six trained to the 16-epoch cap, as the constitutional fits did, so training is now matched and a coefficient of one subtracts a control update of nearly the constitutional update's size. The refit kept the learning rate and warm-up searched on constitution text, and under them held-out control loss kept falling until the end of the schedule. The results are qualitatively identical across the evaluations we re-ran (static, multi-query and multi-turn jailbreaks, over-refusal, knowledge and math); for example, measured against the same unsteered runs, the difference LoRA's multi-turn gain is +58.0 pp before and +57.6 pp after.

Revision: App. D.4 (Fig. 11); App. B; Sec. 2.3

W3 – Geometric evidence for register vs. content decomposition. Thank you for the opportunity to clarify and extend the geometric evidence (App. D.3). We agree that "Near-zero cosine similarity with one refusal direction rules out close alignment with that vector, but not a broader refusal-related effect." The revision therefore tests the control objects against 279 refusal directions spanning every layer but the last and 9 template positions: they sit at chance on the first refusal component (0.94–1.05× a random direction), whereas the constitution objects' writes are enriched 2.8×. Within each layer's refusal span, the control objects are only mildly enriched (1.2–1.35×, similar the constitution objects; Fig. 9). What we believe most directly rules out a broader refusal direction is the negated control. On average, applying θ_K positively or negatively harms static defense and raises benign compliance, compared to the true refusal steer which reverses with its sign (Fig. 9). The control's effect is therefore mostly sign-independent, unlike an anti-refusal direction.

Where we agree most with the reviewer is that the previous draft gave no clear evidence of a shared component beyond the cosine of ReLU vectors at a single layer. The revision measures it at every layer each object type modifies (Fig. 10). θ_C and θ_K write into a shared subspace: 14–19% of each LoRA update's energy lies in the other's write space, against 0.16% for random subspaces and about 0.3% for an unrelated LoRA. In the forward pass their residual-stream displacements align (cosine 0.55–0.72), and subtracting θ_K brings the next-token distribution closer to the unsteered model in all 36 seed × evaluation cells (mean KL 0.46 vs 0.75 nats per token for the ReLU steer, 0.52 vs 0.75 for the LoRA), whereas adding θ_K multiplies θ_C's KL by 2.3–3.3. While we have not measured what this shared component encodes, it depends more on which model wrote the text than on its content (App. D.3), and it is what we mean by "common register".

Revision: App. D.3 (Figs. 9–10); Sec. 3.2; Fig. 1 caption; Secs. 4 and 5

W4 – Agentic misalignment vs. degraded tool use. We agree. Our transcript analysis showed that the agentic leaking reductions were confounded with reduced tool use: malformed calls were negligible, but the constitution and difference objects issued no tool call in 72–85% of transcripts, at similar rates even in scenarios without a goal conflict. We therefore think it is clearer to omit the agentic misalignment evaluations, and we have removed them. Misalignment is now assessed with Petri, and tool use remains measured by BFCL, where the control and constitution ReLU steers' new failures likewise mostly lack a tool call (App. D.10).

Revision: agentic evaluations removed (Secs. 2.4 and 3.3; Fig. 5; App. C, D.1, D.6 and D.11); App. D.10 (Table 6)

W5 – No harmful requests during training. You were correct. As written, this claim was overly broad for the constitution corpus. We have now audited both corpora. No record copies any of our fixed evaluation prompts or static attack templates, but at least 765 constitution records contain a harmful request and at least 804 depict a jailbreak attempt, which the text almost always declines. We have removed the claim: the main text now states only that the corpora copy none of our fixed evaluation prompts or attack templates, and App. A.8 reports the full audit.

Revision: abstract; Secs. 1, 3.1, 4 and 5; AI use statement; App. A.4, A.8 (Table 3) and C

If these revisions address your concerns, we'd be grateful if you'd consider revising your score; otherwise, please tell us what remains and we'll respond quickly.

Thanks for the detailed response. I think W1–W4 are fine with me. However, W5 is the sticking point. The audit shows the constitution corpus contains 765+ harmful requests and 800+ declined jailbreak attempts, and the control corpus has none by design. So θ_C − θ_K may simply be "800 refusal demonstrations minus register," not "constitutional content minus register." Your own 2.8× refusal-span enrichment fits that reading. This undercuts the paper's core framing, makes beating a 128-pair refusal vector unsurprising, and puts the multi-turn gains on exactly the kind of jailbreak the corpus depicts. Deleting the sentence narrows the claim but doesn't test it. I'd need to see θ_C retrained without the flagged records and how much of the gains survive.

Given my aforementioned concerns, I have decided to maintain a rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.