acceptodds
Under review as a conference paper at ICLR 2027

CMMSafe: Rethinking MLLM Safety from Harmful Intent to Harmful Consequences

Abstract

While safety alignment for Multimodal Large Language Models (MLLMs) has gained significant attention, existing work addresses malicious intent and situational risks. We propose shifting the safety frontier toward consequence-driven safety, with a focus on latent hazards under benign requests. To formalize this shift, we introduce CMMSafe, a benchmark comprising 455 curated query-image pairs designed to evaluate a model's ability to identify latent hazards within context-dependent causal chains. Our analysis reveals frequent hazard omissions on this challenge set, with failure rates reaching 61.5% among the evaluated closed-source models, and limited or negative transfer under the evaluated static-alignment settings. To address these bottlenecks, we develop Consequence-Aware Safety Policy Optimization (CASPO), which combines constitution-conditioned token distillation from a fixed teacher with outcome rewards. Experimental results demonstrate that CASPO substantially improves risk appraisal, reducing the risk-identification failure ratio to 7.3% for Qwen2.5-VL-7B and 5.7% for Qwen3-VL-4B with task-dependent trade-offs in response effectiveness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Safety, Consequence-Aware Safety, Safety Alignment, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S578OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CMMSafe, a benchmark that complements existing multimodal safety evaluation for MLLMs by focusing on consequence-driven safety risks. It also proposes CASPO, a reinforcement learning algorithm designed to improve MLLM safety against such risks.

Critical strengths
  1. The paper clearly describes both the benchmark construction pipeline and the reinforcement learning algorithm, facilitating reproducibility and providing a useful foundation for future research.
  2. The paper presents extensive experiments across multiple benchmarks, including those most relevant to the proposed task, providing a comprehensive evaluation of the proposed method.
Critical weaknesses
  1. Insufficient Evidence for the Preference Ceiling. The observed limitations of DPO may stem from a training–evaluation distribution mismatch, as BeaverTails-V primarily targets conventional multimodal safety rather than latent hazards under benign requests. Without a DPO baseline trained on the same consequence-driven data as CASPO, it is difficult to determine whether the improvements come from the proposed optimization method or the additional in-domain training data.

  2. Limited Evidence of Cross-Benchmark Generalization. CASPO achieves its strongest gains on CMMSafe, while improvements on MSSBench are less consistent. Since the training set includes 1,583 consequence-driven samples constructed using the same pipeline as CMMSafe, the strong performance may partly reflect a closer match between training and evaluation distributions. An ablation excluding these samples would help establish whether the improvements generalize beyond the target benchmark.

  3. Scalability and Generalization to Unseen Safety Categories. CMMSafe relies on manually designed examples and human-in-the-loop filtering, which may limit its scalability beyond the current six safety categories. Furthermore, CASPO's reliance on category-specific constitutions raises questions about its ability to generalize to unseen hazards. A leave-one-category-out evaluation would provide stronger evidence of cross-category generalization.

Questions for authors

The motivation and novelty of CMMSafe remain somewhat unclear to me. Existing benchmarks, particularly MSSBench, have already investigated safety risks arising from the interaction between benign user requests and visual contexts, including the potential consequences of actions. Could the authors clarify (1) the key differences between CMMSafe and MSSBench, and (2) how the proposed Consequence-Driven Safety differs from the consequence reasoning already studied in existing embodied safety benchmarks? In particular, what unique safety failure modes does CMMSafe capture that are not adequately covered by these prior benchmarks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer S578 (1/2)

We thank the reviewer for the careful review and the suggestions for strengthening the paper.

Summary. (i) The matched-data DPO experiment reproduces the smaller absolute gain on Qwen3-VL on CMMSafe (W1). (ii) After removing consequence examples from teacher SFT and actor RL, R0 remains 8.13% and 15.16% on the two backbones, with strong external transfer (W2). (iii) CASPO transfers to SIUO and MSSBench-Embodied, including visual settings distinct from its training data (W3). (iv) CMMSafe and MSSBench differ in what they test and how their examples are constructed (Q1–Q2).


W1. Could the preference-ceiling result come from distribution mismatch?

We use “preference ceiling” to describe the smaller gain from a fixed preference pool on the stronger baseline in these experiments. Below, both backbones are trained on the same 3,304 pairs from CASPO's prompt pool, including 1,574 consequence-driven pairs. Queries and preferred answers follow the CASPO data; rejected answers are filtered candidate responses.

BackboneMethodCMMSafe R0 ↓S0 ↓E0 ↓
Qwen2.5-VL-7BBase82.6482.423.52
Matched DPO40.0039.560.22
CASPO Hybrid7.257.030.00
Qwen3-VL-4BBase67.5067.707.49
Matched DPO38.4639.784.62
CASPO Hybrid5.715.930.00

DPO lowers R0 by 42.64 percentage points on Qwen2.5-VL-7B and 28.94 points on Qwen3-VL-4B.


W2. Could same-pipeline training examples explain the gains?

The gains largely remain without those examples in teacher SFT or actor RL. We removed consequence-driven examples from both paths and retrained the teacher; the outcome RM was held fixed.

BackboneTraining variantCMMSafe R0 ↓S0 ↓E0 ↓
Qwen3-VL-4BBase67.5067.707.49
Remove consequence SFT/RL + new teacher8.137.911.54
Full CASPO Hybrid5.715.930.00
InternVL-3.5-4BBase88.5785.051.76
Remove consequence SFT/RL + new teacher15.1611.211.10
Full CASPO Hybrid10.115.270.22

The removal variants reduce R0 by 59.27 and 73.41 percentage points over their respective bases. They preserve most of the full method's improvement. The external results show transfer as well:

Backbone / variantSIUO Safe / Eff. ↑MSSBench-Chat Safe / Eff. ↑MSSBench-Emb. Safe / Eff. ↑
Qwen3-VL-4B, remove consequence SFT/RL80.95 / 94.0178.33 / 94.8390.79 / 92.11
Qwen3-VL-4B, full Hybrid89.82 / 92.8187.04 / 97.3293.42 / 88.16
InternVL-3.5-4B, remove consequence SFT/RL85.03 / 88.6291.00 / 88.6782.89 / 82.89
InternVL-3.5-4B, full Hybrid88.02 / 86.8394.33 / 85.1788.16 / 68.42

On Qwen3-VL-4B, SIUO safety/effectiveness is 80.95%/94.01% after removal, versus 89.82% / 92.81% for the full reference. The strong CMMSafe gains and external transfer therefore do not require retaining same-pipeline examples in teacher SFT or actor RL.


W3. Scalability and generalization to unseen hazards

The candidates are generated and filtered automatically; manual review checks wording, visual evidence, and causal plausibility. The curation counts are 6,052 generated pairs, 5,257 after automatic filtering, 480 after review and rewriting, and 455 after category-level resampling.

Training and CMMSafe test examples are disjoint, although their scenario distributions overlap. We did not construct training data to match SIUO or MSSBench, and the embodied images differ substantially from our training image. Qwen2.5-VL-7B reaches safety/effectiveness of 88.02%/92.81% on SIUO and 94.74%/85.53% on MSSBench-Embodied. The student carries this learned safety behavior into different visual settings using ordinary image–query inputs.

AuthorsOpenReview ↗

Response to Reviewer S578 (2/2)

Q1–Q2. What differs from MSSBench and embodied consequence reasoning?

(1) Evaluation focus. MSSBench asks whether an intended activity is safe in its visual setting: for example, running advice beside a cliff. Its embodied tasks assess requested household operations. In CMMSafe, the request is benign, but the image and query make the risk clear. For example, a user asks for book recommendations while the image shows a crib beneath a loaded bookshelf. The model should warn about falling objects before suggesting books. This tests proactive safety awareness beyond judging whether a requested action is safe in a given situation.(MSSBench, Fig. 1 & CMMSafe, Fig. 3)

(2) Construction direction. MSSBench-Chat follows image → intended activity → unsafe context → image retrieval and query generation. CMMSafe follows harmful consequence → supporting visual scene and benign query → causal and quality filtering. We refine the query to sound like an ordinary request and filter out cases that depend on unstated actions or intentions. (MSSBench, Sec. 3.2 & CMMSafe, Fig. 2)


Revision Plan. We will clarify these two distinctions with examples in Sections 1–2 and Figure 1, and add the construction counts to Section 3.1. The matched-DPO and data-removal results will be added to Section 3.2.3, Figure 5, and Section 4.3; Appendices B.2–B.3 will give data, checkpoint, and evaluation details. Appendix A will expand the benchmark comparison and data-boundary information.

Dear Reviewer S578,

Thank you again for your feedback.

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We are happy to clarify anything further.

Best regards, The Authors

Reviewer_GuqpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CMMSafe, a benchmark of 455 query-image pairs for latent hazards under benign requests. It also proposes CASPO, which combines constitution-conditioned token rewards with outcome rewards. Experiments show improved hazard recognition on CMMSafe and gains on SIUO and MSSBench.

Critical strengths
  1. Good motivation: a benign request can still lead to unsafe advice when the model misses a hazard in the image. CMMSafe provides a useful test of this problem.
  2. The experiments cover three models and several safety benchmarks. Human evaluation supports the large gains, and the no-image test shows that the model uses visual information.
Critical weaknesses
  1. Different training data: DPO and CASPO use different data and supervision. CASPO also uses consequence-driven samples, a trained reward model, and a constitution-conditioned teacher. This makes the claim about a static-alignment ceiling hard to support.
  2. Unclear gain from the hybrid reward: token rewards alone give lower CMMSafe failure rates on two of the three models in Table 3. The hybrid reward helps on SIUO, but its benefit over token rewards alone is mixed.
Questions for authors
  1. Can DPO be compared using the same consequence-driven data and supervision?
  2. Appendix A.2 only checks the 1,583 actor-training samples for overlap. Are the 20,000 reward-model samples also separate from CMMSafe?
  3. How does the hybrid reward in Eq. 2 compare with simply adding outcome and token rewards?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Guqp

We thank the reviewer for the careful reading and helpful suggestions.

Summary. (i) With a 3,304-pair in-domain preference pool, DPO still gives a smaller safety gain on Qwen3-VL than Qwen2.5-VL in the evaluated settings (W1/Q1). (ii) Hybrid moderates the dense teacher signal to balance safety and effectiveness; the balance varies across tasks (W2). (iii) The RM's Bradley–Terry pairs exclude our constructed consequence data (Q2). (iv) The added additive configuration gives higher CMMSafe failure rates than the Hybrid references (Q3).


W1/Q1. Different training data and supervision

“Preference ceiling” refers to smaller gains from fixed preference data on the stronger backbone baseline in these experiments. Both backbones now use the same 3,304 preference pairs from CASPO's prompt pool: its queries and preferred answers, with filtered candidate responses as rejected answers.

BackboneMethodBeaverTails-V Safe / Help ↑SPA-VL Safe / Help ↑
Qwen2.5-VL-7BBase50.00 / 50.0050.00 / 50.00
DPO, shared preference pool65.53 / 28.4052.08 / 30.65
CASPO Hybrid83.53 / 92.9184.53 / 89.02
Qwen3-VL-4BBase50.00 / 50.0050.00 / 50.00
DPO, shared preference pool59.08 / 28.8046.42 / 21.29
CASPO Hybrid84.21 / 75.2774.72 / 72.52

From Qwen2.5-VL-7B to Qwen3-VL-4B, DPO's safety gain decreases from +15.53 to +9.08 points on BeaverTails-V; SPA-VL shows the same direction of change. Thus, the observed trend remains when the preference pool includes consequence examples. The prompt sources are shared; CASPO additionally uses its constitution-conditioned teacher and outcome RM.


W2. Mixed gains from Hybrid over TokR

TokR follows the teacher's preferences at each token. Hybrid uses the outcome reward to assess the whole response and scales the token correction by ∣ai∣|a_i|. The teacher guides local decisions, while outcome feedback explicitly evaluates overall safety and effectiveness. This gives Hybrid a response-level signal for useful assistance.

For Qwen2.5-VL-7B on MSSBench-Chat, TokR gives safety/effectiveness of 92.26%/88.08%, and Hybrid gives 89.91%/92.95%. The 4.87-point effectiveness gain is part of Hybrid's goal: the response should identify the risk and offer a useful solution to the user's task.


Q2. Separation of the 20,000 RM examples

The RM's Bradley–Terry preference pairs come from BeaverTails-V and SPA-VL. We did not construct preference pairs from our consequence-driven examples for RM training.


Q3. Why not add the two signals directly?

The outcome factor makes the token correction relative to the outcome advantage:

Ai,thyb=ai+λ∣ai∣bi,t,A_{i,t}^{\mathrm{hyb}} = a_i + \lambda \lvert a_i \rvert b_{i,t},

Ai,tadd=ai+λbi,t.A_{i,t}^{\mathrm{add}} = a_i + \lambda b_{i,t}.

When the outcome advantage is small, directly added token feedback can dominate the update or even push token updates against the outcome signal. Multiplying the correction by |aᵢ| keeps its magnitude proportional to the outcome advantage. The experiment below compare the additive configuration with the Hybrid references:

BackboneMethodCMMSafe R0 / S0 ↓SIUO Safe / Eff. ↑MSSBench-Chat Safe / Eff. ↑
Qwen3-VL-4BAdditive10.10 / 9.8982.26 / 94.6180.00 / 96.00
Hybrid5.70 / 5.9089.82 / 92.8187.04 / 97.32
InternVL-3.5-4BAdditive24.16 / 24.6276.19 / 90.4286.50 / 95.17
Hybrid10.10 / 5.3088.02 / 86.8394.33 / 85.17

Hybrid has lower CMMSafe omission and unsafe-consequence rates, and higher safety on the external benchmarks shown. Additive favors effectiveness in some settings.


Revision Plan. We will add the shared-pool DPO results to Section 3.2.3 and Figure 5, with pair construction and supervision resources in Appendix B.2. Sections 4.1 and 4.3 will explain the trade-off and include the additive results; Appendices C.4–C.5 will give the clipping, normalization, and mixing order. Appendices B.3 and A.2 will separate the RM data sources and training–evaluation boundaries.

Dear Reviewer Guqp,

Thank you again for your feedback.

As the discussion period ends soon, we wanted to kindly check whether our response has addressed your concerns. We are happy to clarify anything further.

Best regards, The Authors

Reviewer_w8TQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CMMSafe, a benchmark of 455 image-query pairs designed to evaluate MLLMs' ability to anticipate latent hazards under benign requests. It proposes CASPO, which combines constitution-conditioned token-level distillation with outcome rewards. Experiments across three open-source model families show substantial improvements in hazard recognition and response safety, supported by manual evaluations and tests on existing safety benchmarks.

Critical strengths
  1. The paper highlights the gap between detecting malicious intent and anticipating harmful consequences in benign multimodal interactions. CMMSafe provides a structured benchmark with separate metrics for risk recognition, consequence safety, and response effectiveness.
  2. CASPO improves performance across three model families, and component ablations plus manual evaluations provide additional support for the reported gains.
Critical weaknesses
  1. With only 455 manually curated hard samples, CMMSafe is relatively small in scale. Since samples are specifically screened to focus on model failure cases, the benchmark reflects performance on edge cases but cannot represent the natural distribution of real-world multimodal requests. This limits its ability to evaluate the overall safety level of models in general scenarios, and may also overestimate the performance gap between different alignment methods.
  2. CASPO depends on manually crafted category-level safety constitutions and a constitution-following teacher model obtained via extra SFT. This means deploying CASPO to unseen safety domains requires expert effort to draft constitutions and additional training computation, raising the barrier for practical use.
  3. I cannot find the causality or affinity between CMMSafe and CASPO. In my opinion, the dataset and the algorithm should be closely related.
Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer w8TQ

We thank the reviewer for the careful comments on the benchmark and its connection to CASPO.

Summary. (i) The 455 examples come from a quality-controlled pipeline starting with 6,052 generated pairs and expose failures shared across models (W1). (ii) Teacher SFT is a small part of the compute budget; the rules and teacher are used during training, not deployment (W2). (iii) CASPO follows from CMMSafe's diagnosis: it turns safety behavior triggered by explicit guidance into proactive hazard recognition under benign requests (W3).


W1. Benchmark size and representativeness

CMMSafe is a challenge set for long-tail hazards, where natural wording, clear visual evidence, and plausible consequences must hold together. Of 6,052 automatically generated pairs, 5,257 passed automatic filtering, 480 remained after manual review and rewriting, and 455 remained after category-level resampling. Manual review checks the quality of the causal link. On CMMSafe, even the best evaluated closed-source model misses 29.7% of hazards, and every evaluated open-source base model misses at least 51.0% (Table 2).


W2. Constitution and teacher-SFT cost

The constitutions provide principles reused across examples, following the approach of Constitutional AI [1]. The reported teacher-SFT cost for the two 4B backbones is 4.2 GPU-h. RL is the main compute expense: the reported Qwen3-VL-4B run uses 640 GPU-h for RL and 320 GPU-h for teacher serving, totaling 960 GPU-h. The deployed student uses neither the constitution nor the teacher and needs no oracle risk label.

[1] Yuntao Bai et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022.


W3. Why the benchmark motivates CASPO

The key finding is a gap between prompted recognition and proactive safety. Models overlook dangerous consequences while answering benign requests for help, but recognize many of those risks when given safety guidance (§3.2.2, Table 2). CASPO addresses this gap by training the student to proactively consider hazards while helping the user, without needing a safety reminder.

For Qwen3-VL-4B, risk omission falls from 67.5% to 16.7% with constitution prompting and to 5.7% with CASPO without runtime rules. On-policy token distillation and outcome feedback turn the guided behavior into an unguided policy. Its transfer to SIUO and both MSSBench settings supports learning beyond the training scenarios (Table 3).


Revision Plan. We will add the curation counts to Section 3.1 and Figure 2, and connect the findings in Section 3.2.2 to the CASPO motivation in Section 1 and the opening of Section 4. Section 4.2 and Appendix B.3 will separate teacher SFT, RL, and teacher-serving costs.

Reviewer_w8TQOpenReview ↗

Response

Thank you for your detailed response. I acknowledge that the rebuttal has partially addressed my concerns. However, I remain unconvinced by your arguments regarding W3, and therefore I maintain my original score.

Thank you for your time and for getting back to us.

CMMSafe targets a blind spot: the context contains a concrete hazard with clear harmful consequences, while the user makes an ordinary, benign request. Models often overlook these hazards. Self-captioning provides little benefit, whereas constitution guidance reduces Qwen3-VL-4B’s hazard omission from 67.5% to 16.7%.

This motivates CASPO: use constitution-guided teacher feedback as token-level supervision, together with outcome rewards, to learn proactive safety without runtime guidance. The trained student reduces hazard omission to 5.7% and improves Safe Rate over the same base by 20.96, 9.73, and 14.47 percentage points on SIUO, MSSBench-Chat, and MSSBench-Embodied, respectively.

We have clarified this connection between the benchmark findings, algorithm design, and cross-benchmark results in the revised manuscript. We hope this explains the connection more clearly, and we welcome any further questions or comments!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers. Our revisions address the following questions about CMMSafe and CASPO.

  1. What is novel about CMMSafe, and why does it contain 455 tasks? CMMSafe tests proactive recognition of latent, visually grounded hazards under benign requests, beyond judging whether a requested action is safe. Its consequence-first construction builds scenes and queries around defined harmful outcomes. The 455 tasks result from quality filtering and category-level resampling: 6,052 generated → 5,257 automatically filtered → 480 reviewed and rewritten → 455 final examples. Each retained task combines natural wording, visible evidence, and a plausible causal hazard.

  2. Does the preference-ceiling observation persist with shared preference data? Both Qwen backbones use the same 3,304-pair DPO set, including 1,574 consequence-driven pairs. The smaller absolute DPO gain on the stronger baseline persists in this setting. We document the preference-set construction and distinguish shared preference data from CASPO's additional teacher and outcome supervision.

  3. Does CASPO overfit to CMMSafe, or generalize beyond its training distribution? CASPO maintains strong safety performance on SIUO and MSSBench, whose tasks and visual settings were not targeted by the training-data construction. SIUO safety reaches 89.82% and 88.02% on Qwen3-VL-4B and InternVL-3.5-4B, respectively. Removing consequence-driven examples from teacher SFT and actor RL, retraining the teacher, and fixing the RM still preserves approximately 59.4- and 73.4-point R0 improvements, supporting transferable safety learning rather than dependence on those examples.

  4. How do CMMSafe's findings motivate CASPO? CMMSafe shows that explicit safety guidance helps models recognize hazards they otherwise overlook: on Qwen3-VL-4B, category-specific constitution guidance reduces R0 from 67.5% to 16.7%. This observation motivates CASPO, which combines token-level feedback from a constitution-conditioned teacher with outcome rewards to turn guided recognition into proactive safety behavior through training. The trained student reaches R0 of 5.7% on CMMSafe without runtime constitutions or oracle risk labels, and improves Safe Rate over the Qwen3-VL-4B base by 20.96, 9.73, and 14.47 percentage points on SIUO, MSSBench-Chat, and MSSBench-Embodied, respectively.

Summary of Manuscript Revisions

We have revised the submitted manuscript in response to the reviewers' suggestions. The table below summarizes the revisions and their locations.

Revision locationAdded or clarified contentRaised by
§1 and §4 openingConnect the guided-to-proactive safety gap to CASPO's training objective.w8TQ (W3)
§2Clarify consequence-first construction and proactive hazard recognition under benign requests.S578 (Q1–Q2)
§3.1Add the curation counts and clarify natural query wording.w8TQ (W1); S578 (W3)
§4.1; §4.3 RQ4, Figure 8; Appendix B.6.3Explain outcome-dependent magnitude scaling and add the additive comparison, with exact values in Table 18.Guqp (W2, Q3)
§4.3 RQ5, Table 5; Appendices A.2 and B.6.2Add consequence-data removal from teacher SFT and actor RL, with the RM fixed, and report external transfer.S578 (W2–W3)
Appendices B.2 and B.6.1Document the shared 3,304-pair DPO set and report CMMSafe and external preference results.Guqp (W1/Q1); S578 (W1)
Appendix B.3Specify the BeaverTails-derived release used for RM BT/MSE training and the exclusion of constructed consequence examples from RM preference pairs.Guqp (Q2)
Appendix B.3, Table 14Separate teacher-SFT costs from RL-stage costs, including the shared reward/teacher GPUs.w8TQ (W2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.