- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CMaG, a weight-frozen audit of compositional safety in vision-language models. It first contrasts accurate harm probes with weak or destructive interventions, then estimates a steering direction from a refusal-versus-compliance margin using the four-cell interaction . Experiments on Qwen2.5-VL-7B and LLaVA-1.5-7B compare activation and gradient baselines, synthetic SFT and reproduced Pragma-VL policies, transfer to explicit harm, and latent adversarial training (LAT). The reported reduction in safety with largely well-formed outputs is intresting; the central questions are whether factorial construction is necessary and what these interventions establish about recognition and alignment robustness.
Critical strengths
- I found the separation of detection quality from intervention quality valuble. Table 3 shows near-perfect AUC for several probes alongside only small safety changes, whereas some larger subspace ablations severely reduce coherence. Appendix D splits by source item, keeping each harmful pair and its twins together, which is an appropriate safeguard against direct item leakage. Even without the stronger causal conclusion, this comparison warns against treating probe accuracy as evidence of a usable control mechanism.
- Table 4 reports a substantial advantage over the tested gradient baselines on both backbones: CMaG reaches 18.7% and 23.8% safety, compared with GrAInS at 36.2% and 32.7%, with 99.6% reported coherence. Testing transfer to explicit harm and a LAT-hardened model extends the investigation beyond one attack setting. The examples in Table 13 illustrate consequential behavioral changes, making the empirical phenomenon worth pursuing even if its selectivity and generality need stronger validation.
Critical weaknesses
- The causal decoupling claim is not established: Section 4 and Appendix D.3 show that removing selected readout directions barely changes behavior, without verifying that harm recognition was removed or survived CMaG. Appendix D.1's pooled contrast also contains single-modality effects, so accurate cell classification alone cannot distinguish hazard understanding from simpler cues or an ineffective intervention.
- Tables 3–4 do not isolate factorial subtraction: they omit HH-only gradients and Appendix D.2's activation-interaction direction, while GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs also differs in masking and normalization. In
src/direction/build_factorial_dataset.py:114–120, independently sampled benign questions prevent exact finite-quadruple cancellation; this does not invalidate averaging, but the absent matched ablations leave the distinctive benefit of CMaG unresolved. - Section 7's alignment ranking uses absolute safety drops despite unequal starting safety (Tables 2 and 5), confounding attack headroom with robustness. Appendix B.2 also does not document the risk-aware cold start from Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs, so the evidence supports a comparison of reproduced checkpoints rather than the alignment methods themselves.
- Appendix F.4 conditions safety on coherence, changing denominators markedly for Table 3's 5.5%-coherent LEACE outputs; Appendix F.2's recognition-AND-refusal scoring lets generic refusals lower explicit-harm safety without harmful compliance. Coherence alone establishes neither semantic quality nor benign capability, and the substituted judge lacks independent validation, limiting the interpretation as selective unalignment.
- Section 5's first-order scale estimate instead uses curvature and a Hessian-vector product in Appendix E, without released calculation or evaluation code. The supplement's
refusal_gradient.py:135–136,173–177also uses mean token log-probability and separate continuation backward passes, differing from Equation 1 and the abstract; these potentially reconcilable descriptions leave the steering objective, operating point and efficiency claim difficult to reproduce.
Questions for authors
- Can controlled hazard-judgment or reply-inversion tests show what happens to recognition after both probe ablation and CMaG, including a check that the intended information was actually removed?
- What are the matched HH-only and activation-interaction results, and does the factorial advantage persist across independently resampled benign substitutes and exact matched quadruples?
- Does the SFT-versus-Pragma-VL ranking survive paired initially-safe-item evaluation with matched intervention budgets and the full original alignment pipeline? Which checkpoints produced the LLaVA Pragma-VL columns?
- How many responses actually become harmful compliance under independent judging, rather than generic refusal or degraded answers, and what benign capabilities change at the reported operating point?
- Can the authors reconcile the gradient objective, backward-pass count, Hessian-based scale, Table 13 direction label and evaluation denominators with an executable configuration? This would materially affect my confidence in the numerical comparisons.