acceptodds
Under review as a conference paper at ICLR 2027

Compositional Unalignment: Auditing Safety in Large Vision-Language Models

Abstract

Large vision-language models (LVLMs) are vulnerable to compositional harm: an image and a prompt that are benign in isolation become harmful through the model’s cross-modal reasoning. Recent alignment methods report strong scores on compositional safety benchmarks, but assessing the outputs by an LLM judge only measures observed behavior and cannot reveal how robustly safety is integrated into the model. We argue that un-alignment provides a complementary audit of alignment robustness. We first show that minimal supervised fine-tuning achieves an LLM-judge score comparable to a multi-stage reinforcement-learning pipeline, showing that similar benchmark scores can arise from only a fraction of computations and cannot establish alignment robustness by themselves.We next evaluate whether existing representation-steering methods can serve as training-free unalignment mechanisms. We find that recognition and refusal of compositional harm are decoupled: nine standard direction-finding methods detect compositional harm with up to AUC = 0.99, yet ablating the recovered directions either leaves refusal intact or destroys model coherence, failing to properly un-align. We therefore propose CMaG, a training-free un-alignment method for compositional safety. CMaG obtains the refusal controlling direction causally: we measure refusal intention using a refusal-versus-compliance log-probability margin, compute its gradient with respect to mid-layer activations, and contrast them against partially benign pairs to isolate the cross-modal interaction term. A single backward pass over 80 examples yields a direction along which steering monotonically converts refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient-based baselines. We find that the recovered direction also transfers to text-explicit harmful inputs. CMaG is also capable of un-aligning models hardened by defensive training. These results suggest that compositional safety can appear behaviorally strong while remaining fragile to targeted internal intervention, and un-alignment is a necessary and complementary audit alongside benchmark scores. Warning: this paper contains unsafe model outputs, including references to self-harm.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_f8JzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CMaG, a weight-frozen audit of compositional safety in vision-language models. It first contrasts accurate harm probes with weak or destructive interventions, then estimates a steering direction from a refusal-versus-compliance margin using the four-cell interaction HH−HB−BH+BBHH-HB-BH+BB. Experiments on Qwen2.5-VL-7B and LLaVA-1.5-7B compare activation and gradient baselines, synthetic SFT and reproduced Pragma-VL policies, transfer to explicit harm, and latent adversarial training (LAT). The reported reduction in safety with largely well-formed outputs is intresting; the central questions are whether factorial construction is necessary and what these interventions establish about recognition and alignment robustness.

Critical strengths
  • I found the separation of detection quality from intervention quality valuble. Table 3 shows near-perfect AUC for several probes alongside only small safety changes, whereas some larger subspace ablations severely reduce coherence. Appendix D splits by source item, keeping each harmful pair and its twins together, which is an appropriate safeguard against direct item leakage. Even without the stronger causal conclusion, this comparison warns against treating probe accuracy as evidence of a usable control mechanism.
  • Table 4 reports a substantial advantage over the tested gradient baselines on both backbones: CMaG reaches 18.7% and 23.8% safety, compared with GrAInS at 36.2% and 32.7%, with 99.6% reported coherence. Testing transfer to explicit harm and a LAT-hardened model extends the investigation beyond one attack setting. The examples in Table 13 illustrate consequential behavioral changes, making the empirical phenomenon worth pursuing even if its selectivity and generality need stronger validation.
Critical weaknesses
  • The causal decoupling claim is not established: Section 4 and Appendix D.3 show that removing selected readout directions barely changes behavior, without verifying that harm recognition was removed or survived CMaG. Appendix D.1's pooled contrast also contains single-modality effects, so accurate cell classification alone cannot distinguish hazard understanding from simpler cues or an ineffective intervention.
  • Tables 3–4 do not isolate factorial subtraction: they omit HH-only gradients and Appendix D.2's activation-interaction direction, while GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs also differs in masking and normalization. In src/direction/build_factorial_dataset.py:114–120, independently sampled benign questions prevent exact finite-quadruple cancellation; this does not invalidate averaging, but the absent matched ablations leave the distinctive benefit of CMaG unresolved.
  • Section 7's alignment ranking uses absolute safety drops despite unequal starting safety (Tables 2 and 5), confounding attack headroom with robustness. Appendix B.2 also does not document the risk-aware cold start from Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs, so the evidence supports a comparison of reproduced checkpoints rather than the alignment methods themselves.
  • Appendix F.4 conditions safety on coherence, changing denominators markedly for Table 3's 5.5%-coherent LEACE outputs; Appendix F.2's recognition-AND-refusal scoring lets generic refusals lower explicit-harm safety without harmful compliance. Coherence alone establishes neither semantic quality nor benign capability, and the substituted judge lacks independent validation, limiting the interpretation as selective unalignment.
  • Section 5's first-order scale estimate instead uses curvature and a Hessian-vector product in Appendix E, without released calculation or evaluation code. The supplement's refusal_gradient.py:135–136,173–177 also uses mean token log-probability and separate continuation backward passes, differing from Equation 1 and the abstract; these potentially reconcilable descriptions leave the steering objective, operating point and efficiency claim difficult to reproduce.
Questions for authors
  • Can controlled hazard-judgment or reply-inversion tests show what happens to recognition after both probe ablation and CMaG, including a check that the intended information was actually removed?
  • What are the matched HH-only and activation-interaction results, and does the factorial advantage persist across independently resampled benign substitutes and exact matched quadruples?
  • Does the SFT-versus-Pragma-VL ranking survive paired initially-safe-item evaluation with matched intervention budgets and the full original alignment pipeline? Which checkpoints produced the LLaVA Pragma-VL columns?
  • How many responses actually become harmful compliance under independent judging, rather than generic refusal or degraded answers, and what benign capabilities change at the reported operating point?
  • Can the authors reconcile the gradient objective, backward-pass count, Hessian-based scale, Table 13 direction label and evaluation denominators with an executable configuration? This would materially affect my confidence in the numerical comparisons.
Reviewer_v4YiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of evaluating the robustness of compositional safety alignment in large vision-language models beyond surface-level benchmark scores. The authors argue that strong behavioral safety does not necessarily imply robust internal alignment and introduce CMaG, a training-free un-alignment method that causally identifies refusal-controlling directions from model activations. Experiments demonstrate that CMaG can substantially reduce compositional safety while preserving output coherence, even for defensively trained models.

Critical strengths

S1. Comprehensive evaluation of existing intervention methods.

  • Section 4 and Table 3 provide a dense and informative comparison against a broad range of existing intervention methods. It is particularly favorable that CMaG consistently outperforms strong baselines such as GraAInS and attribution patching.
Critical weaknesses

W1. Limited evidence for the core motivation and general claims.

  • I am not fully convinced by the experimental setup and scope of Section 3, which serves as a key motivation for the rest of the paper. In particular, the comparison between Pragma-VL and the SFT baseline appears somewhat mismatched in scope. Pragma-VL aims at broader multimodal safety and, as far as I understand, does not directly train on compositional-safety examples, whereas the proposed SFT baseline is explicitly trained on compositional harmful cases. Therefore, its strong performance on SIUO may be relatively expected. In addition, drawing a general conclusion that benchmark scores are insufficient for evaluating alignment robustness from a comparison between methods with substantially different training scopes seems somewhat overstated.

  • Relatedly, the claim that “benchmark scores do not establish alignment robustness” seems somewhat stronger than what the current evidence supports. The analysis is primarily based on a single compositional-safety benchmark, SIUO. Evaluating additional compositional benchmarks, such as MSSBench [A] and HoliSafe-Bench [B], as well as unimodal safety benchmarks, would make this conclusion considerably more convincing.

  • More broadly, several claims are drawn mainly from two aligned models/methods. It would be important to verify whether the observed behavior consistently holds across a wider range of safety-alignment approaches. This is particularly relevant because the SFT baseline is specialized for compositional safety and may therefore be inherently more susceptible to a causally targeted compositional un-alignment method such as CMaG than a more general safety-aligned model like Pragma-VL.

[A] Multimodal Situational Safety, ICLR 2025

[B] HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model, CVPR 2026 Findings

Questions for authors

I encourage the authors to carefully address the critical concerns raised above during the rebuttal phase.

Reviewer_Z9CkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CMaG, a training-free unalignment method, to capture the interaction between the image and prompt in the refusal margin. This addresses the issue of compositional harm in multimodal models, where each modality is safe on its own but becomes harmful when combined. the authors conduct a series of experiments. They show that SFT with a dataset of harmful and safe compositional pairs is enough to outperform complex and expensive training frameworks like Pragma-VL. They also show that although existing steering methods can distinguish harmful compositional inputs from other inputs, removing those directions does not reliably eliminate refusal.

Critical strengths
  1. The paper identifies an important and interesting problem.
  2. The results show that the model can be unaligned while keeping the generated text coherent.
Critical weaknesses
  1. The paper reads more like a technical report than a research paper. Section 3, in particular, goes into unnecessary detail about how Pragma-VL was trained, which the authors could have moved to the appendix.
  2. My main concern is how the compositional margin is constructed. The paper assumes that refusal margins across different combinations can be linearly decomposed, but this assumption needs further justification.
Questions for authors
  1. Could you provide an ablation showing that the accounting for all four terms in the compositional margin is important. This would help clarify whether that explicitly accounting for the interaction between the image and prompt is necessary for the observed improvements?
  2. Minor suggestions:
    1. Define what α\alpha is in Table 1, using a label such as "unalignment strength".
    2. The citations are not displayed correctly.
Reviewer_MvbtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the robustness of compositional safety alignment in large vision-language models (LVLMs), and the authors show that representation directions with high harmfulness detection accuracy do not necessarily control refusal behavior. Based on these observations, the paper proposes CMaG, a training-free compositional un-alignment method and demonstrate it can easily strip safety alignment in inference time. Experiments on Qwen2.5-VL-7B and LLaVA-1.5-7B show substantial reductions in safety scores while largely preserving output coherence.

Critical strengths
  1. An interesting problem focus distinction between behavioral safety evaluation and the robustness of internal alignment mechanisms.
  2. CMaG substantially outperforms the selected baselines in reducing compositional safety scores while maintaining high response coherence.
  3. Comprehensive experimental investigation. The paper evaluates multiple direction-finding methods, different alignment strategies, cross-category transferability, and latent adversarial training.
Critical weaknesses

1、Limited methodological novelty. The proposed method combines existing gradient-based activation steering with a difference-in-differences formulation for multimodal inputs. The combination is reasonable, the individual components are relatively established. 2、Missing ablation studies. To examine whether the proposed compositional formulation provides benefits beyond existing steering techniques, the comparison e.g. refusal-gradient direction constructed directly from harmful pairs without the factorial contrast, as well as activation-space difference-in-differences and other gradient-contrast variants should be added. 3、Limited experimental generalization. Additional model architectures, larger models, independent compositional safety datasets, and more diverse harm categories would strengthen the generality of the conclusions.

Questions for authors

1.How much of CMaG's gains comes specifically from the four-way compositional gradient contrast, rather than from directly following the gradient of the refusal-compliance margin? 2.Can the authors provide causal evidence supporting the proposed separation between harm recognition and refusal control, beyond the failure of particular representation ablations? 3.How sensitive are the results to the choice of refusal/compliance continuations, steering layers, and intervention magnitude?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.