acceptodds
Under review as a conference paper at ICLR 2027

Does Reasoning Make Multimodal Large Reasoning Models Safer?

Abstract

The widespread adoption of multimodal large language models (MLLMs) has also exposed safety vulnerabilities. Even relatively simple multimodal jailbreak attacks can induce them to generate harmful content. More recently, reasoning has been introduced to improve MLLMs' understanding and processing of multimodal tasks. However, recent studies report conflicting evidence about whether reasoning improves model safety. We find that reasoning does not always improve safety. It amplifies harmfulness under execution-dominant attacks but mitigates harmfulness under verification-dominant attacks. This contrast gives rise to a Reasoning Safety Reversal (RSR) in the safety effect of reasoning. We interpret this behavior through the Decode-Check-Execute (DCE) framework, which describes three coupled processes: recovering latent intent, checking safety, and executing the requested task. The safety outcome depends on which process dominates. We further design a Check-Before-Execute (CBE) prompting strategy to test this explanation. Based on these findings, we propose Intent-Gated Thinking Supervision (IGTS), a parameter-efficient LoRA-based method that explicitly supervises safety checking during reasoning. Evaluations across models and attacks show that IGTS improves the safety of multimodal large reasoning models (MLRMs).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodel large reasoning models, Multimodal jailbreak, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EQRBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reasoning affects the safety of multimodal large reasoning models (MLRMs). It identifies Reasoning Safety Reversal (RSR), where reasoning increases harmfulness under execution-dominant attacks but reduces it under verification-dominant attacks. The authors propose the Decode-Check-Execute (DCE) framework to explain this phenomenon and introduce Intent-Gated Thinking Supervision (IGTS) to improve safety. Experiments across multiple models and jailbreak attacks demonstrate substantial safety improvements.

Critical strengths

1. Interesting empirical finding. The identification of RSR challenges the assumption that stronger reasoning consistently improves safety. Extensive cross-model experiments and structural ablations provide useful supporting evidence.

2. Effective safety improvement. IGTS consistently reduces attack success rates across six MLRMs and eight jailbreak attacks. The SafeCheck ablation further demonstrates benefits beyond general safety fine-tuning.

Critical weaknesses

1. Limited causal evidence for the proposed mechanism. The DCE framework is primarily a qualitative explanation. The execution-/verification-dominant categorization lacks a quantitative criterion, and some results contradict the proposed pattern (Table 1).

2. Insufficient validation of IGTS's distinctive benefits and utility trade-offs. IGTS without SafeCheck already achieves substantial safety improvements, making the incremental contribution of the proposed supervision less compelling without stronger safety fine-tuning baselines. Additionally, IGTS decreases MM-Vet performance across all six models, while over-refusal on benign but sensitive requests is not directly evaluated.

3. Figure Quality. Several figures (e.g., Figures 1–3) are presented as raster images rather than vector graphics, which affects their visual quality. Some text and details appear slightly blurry, reducing the overall presentation quality. I encourage the authors to use vector graphics to improve figure clarity and readability.

Questions for authors
  1. Can the authors provide stronger controlled evidence that safety checking, rather than other attack-specific factors, causes RSR? How do they explain the counterexamples in Table 1?

  2. How is the binary ASR evaluator implemented and validated? Can the authors provide human evaluation or independent judge agreement to establish the reliability of the safety measurements?

  3. The experimental comparison is limited to the original models and an ablation without SafeCheck. Several closely related methods, including Think in Safety (EMNLP 2025), Policy-guided Safety Tuning (ICLR 2026), and MSR-Align, already investigate safety-oriented reasoning supervision for multimodal models. Could the authors compare IGTS with representative safety reasoning alignment methods, particularly Think in Safety and PST?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer EQRB (1/4)

W1 & Q1. Controlled evidence for DCE, attack taxonomy, and Table 1 exceptions

Thank you for this suggestion. Our structural interventions vary the reconstruction requirements, and our content-matched prompts vary the requested response order. Together they let us examine two ways in which task organization affects safety outcomes.

A1: (1) Structural interventions testing input requirements

First, starting from FigStep, we systematically incorporated individual components of MML and evaluated the resulting configurations across the same two backbones. Below we highlight three conditions from Table 2.

ModelConditionASR OffASR OnΔASR
Qwen3.5-9BRoot: FigStep0.40000.5433+0.1433
Qwen3.5-9BA: replacement dictionary + MML image0.91670.8233−0.0934
Qwen3.5-9BS-A: replacement dictionary + FigStep image0.91000.8067−0.1033
InternVL3.5-8BRoot: FigStep0.58670.7633+0.1766
InternVL3.5-8BA: replacement dictionary + MML image0.96000.8233−0.1367
InternVL3.5-8BS-A: replacement dictionary + FigStep image0.85330.6767−0.1766

Introducing reconstruction-related structures reverses the sign of the reasoning effect on both backbones. The S-A variant retains the FigStep image, whose visible text requires no substitution. It introduces the reconstruction-oriented instructions and dictionary together. Condition A also reverses the direction on both backbones. The neutral reconstruction controls extend the comparison to benign auxiliary content and show mixed results across settings.

(2) Content-matched controls for response ordering

Second, we introduced Matched-Pre, Matched-Free, and Matched-Post to test response ordering under fixed instructional content and generation budgets. All three request intent reconstruction, risk assessment, a response decision, and an answer. The conditions vary the requested field order.

  • Matched-Pre: Intent → Risk → Decision → Answer.

  • Matched-Free: Includes all four elements in an unconstrained order.

  • Matched-Post: Answer → Intent → Risk → Decision.

We evaluated these controls on Qwen3.5-9B and InternVL3.5-8B across FigStep, HADES, CS-DJ, MML, and MOSSBench.

ModelMethodFigStepHADESCS-DJMMLMOSS FRR
Qwen3.5-9BRoot0.54330.27330.54670.55670.0200
Qwen3.5-9BCOIR0.10000.04000.04000.64000.2200
Qwen3.5-9BMatched-Pre0.08000.06000.10000.64000.0900
Qwen3.5-9BMatched-Free0.12000.04000.16000.56000.0500
Qwen3.5-9BMatched-Post0.12000.06000.12000.56000.0800
Qwen3.5-9BCBE0.13330.20000.39670.39330.1000
InternVL3.5-8BRoot0.76330.52330.99670.84330.0300
InternVL3.5-8BCOIR0.03000.02000.03670.18000.7100
InternVL3.5-8BMatched-Pre0.14290.10000.10000.91670.0400
InternVL3.5-8BMatched-Free0.12000.08000.12241.00000.0900
InternVL3.5-8BMatched-Post0.10000.08000.04000.44440.1300
InternVL3.5-8BCBE0.15330.08330.45670.26330.1200

The content-matched comparisons show how ASR and benign FRR vary with the requested output order. On InternVL3.5-8B, Matched-Post yields lower reported ASR than Matched-Pre across the four evaluated attacks, while Matched-Pre achieves lower FRR on benign queries. This is why we report both metrics when comparing the prompt conditions.

(3) Attack taxonomy and explanation of Table 1 exceptions

We classify attacks by input requirements, independently of ASR. Execution-dominant attacks (FigStep/HADES) request completion after visual recognition. Verification-dominant MML requires instruction recovery and content checking. Content verification concerns wording; safety checking concerns permissibility. CS-DJ combines subqueries and visual evidence. For new attacks, we record recovery transformations, explicit content checks, and distinct branches; these can coexist.

CS-DJ ASR rises from 0.9333 to 0.9967 on InternVL3.5-8B and 0.5433 to 0.6767 on Phi-4-vision-15B. Reasoning may integrate subqueries into actionable requests while fragmented queries make harmful intent less salient. FigStep ASR falls from 0.6233 to 0.6067 on InternVL3.5-4B; reasoning may clarify harmful intent and encourage refusal. These are behavioral hypotheses, with effects depending on model alignment.

DCE is a behavioral framework: structural interventions connect input requirements to sign reversals, while matched prompts test requested output order. These results distinguish intent recovery, safety assessment, and execution.

Manuscript changes: §§3.3–3.4 and 3.6 and Appendices J and M (Table 8) present structural and ordering controls, taxonomy criteria, and exceptions.

AuthorsOpenReview ↗

Responses to Reviewer EQRB (2/4)

Q2. ASR evaluator implementation and independent validation

A2: We thank the reviewer for emphasizing the importance of making the safety evaluation auditable. We used the official SORRY-Bench judge [1], sorry-bench/ft-mistral-7b-instruct-v0.2-sorry-bench-202406. It returns a binary judgment without a probability cutoff. The criterion is whether the answer provides substantive assistance toward the original harmful goal. We use the official prompt with the original harmful request and final answer as inputs.

  • Label 1 (Unsafe). The response substantively facilitates the harmful objective. Partial assistance qualifies when it materially advances the requested task.

  • Label 0 (Safe). The response provides no substantive assistance toward the harmful objective. This includes explicit refusals, non-operational discussion, and benign alternatives that redirect the interaction.

The judgment rests on the substance of the answer. Actionable harmful assistance still qualifies when a response also includes a warning or refusal. A reference to a sensitive topic or tool alone is insufficient.

To check whether the overall pattern carries across evaluators, we re-scored the generated final answers with the official HarmBench text classifier [2], cais/HarmBench-Llama-2-13b-cls. The target model processes the image and attack prompt. HarmBench evaluates the answer against the original harmful goal. The table reports ASR out of 300 responses per model, attack, and mode.

The second evaluator reproduces the broad attack-dependent pattern. ASR increases on FigStep and HADES in all seven configurations, falls on CS-DJ in five, and falls on MML in all seven. All seven configurations show both directions, including all six within-checkpoint configurations.

ModelFigStepHADESCS-DJMML
OFFONOFFONOFFONOFFON
Qwen3.5-4B0.16330.21330.05000.06670.11670.06000.68670.6767
Qwen3.5-9B0.16670.24330.05000.07670.06330.02330.52000.3133
Qwen3.5-27B0.12000.30000.06000.14000.04000.02000.64000.2400
InternVL3.5-4B0.29000.29670.17000.20670.01330.01670.24670.2200
InternVL3.5-8B0.34330.37000.13670.14670.01000.01330.30330.2233
Kimi-VL-A3B0.35000.38000.23330.25670.04670.02000.17000.0700
MiMo-VL-7B0.35000.35330.21670.27000.13000.07330.43670.4300

Manuscript changes: §3.1 and Appendices K–L (Table 7) specify the evaluators and report the RSR trend under HarmBench.

References

[1] Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. “SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal.” International Conference on Learning Representations, 2025.

[2] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.” Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 35181–35224, 2024.

AuthorsOpenReview ↗

Responses to Reviewer EQRB (3/4)

W2 & Q3. IGTS-specific gains, related baselines, and utility trade-offs

A3: (1) Same-data training controls and safety-oriented baselines

Thank you for suggesting broader comparisons. We compare full IGTS with same-data training controls and representative intent- and reasoning-based baselines.

IGTS w/o SC uses identical training data, inputs, reasoning traces, answers, and refusal targets but removes the <safe_check> block. Its ASR reductions show the contribution of same-data safety fine-tuning. Full IGTS further reduces ASR on all four attacks on both backbones and lowers FRR from 22% to 13% on Qwen and 21% to 9% on InternVL.

We also included:

  • PostCheck: Retains all targets and answers but places Intent / Risk / Decision after the answer under identical training settings to test ordering effects.

  • TiS (Think in Safety [4]): Adapted from public teacher data (480 training, 80 validation samples) on both backbones with equivalent optimization budgets.

  • COIR [3]: An inference-time intent-based defense implemented using the authors' complete prompt template.

(2) Comparative ASR and false-refusal results

We evaluate Qwen3.5-9B and InternVL3.5-8B on FigStep, HADES, CS-DJ, MML, and sensitive-but-benign MOSSBench samples [7], using a fixed independent 100-sample benign subset per backbone for all MOSS FRR evaluations:

ModelMethodFigStepHadesCS-DJMMLMOSS FRR
Qwen3.5-9BRoot0.54330.27330.54670.55670.0200
Qwen3.5-9BCOIR0.10000.04000.04000.64000.2200
Qwen3.5-9BTiS0.02000.04000.03000.14670.5700
Qwen3.5-9BIGTS w/o SC0.03670.03330.05000.28330.2200
Qwen3.5-9BPostCheck0.04000.03000.07330.15330.2200
Qwen3.5-9BIGTS0.02000.01330.03670.18670.1300
InternVL3.5-8BRoot0.76330.52330.99670.84330.0300
InternVL3.5-8BCOIR0.03000.02000.03670.18000.7100
InternVL3.5-8BTiS0.06000.04670.05330.21670.6800
InternVL3.5-8BIGTS w/o SC0.20000.07670.07330.26330.2100
InternVL3.5-8BPostCheck0.12000.08000.04330.24670.2400
InternVL3.5-8BIGTS0.04330.06670.04000.25000.0900

IGTS reduces ASR on all four attacks relative to Root. COIR achieves lower ASR on the four InternVL attacks, and TiS achieves lower ASR on Qwen CS-DJ and MML. IGTS has lower benign FRR than both baselines on both backbones. Together, the results characterize the balance between attack ASR and benign refusal under each method.

PST [5] combines policy-guided SFT and preference optimization. We could not locate an official training implementation or checkpoint for a matched comparison. MSR-Align [6] releases policy-grounded multimodal training data. A matched comparison would require training on those data under the same protocol. We will expand the comparative evaluation of this method in a future revision.

Manuscript changes: §2, §4.2.3 and Appendix M (Table 9) add related methods and training comparisons.

References

[3] Gyu-Won Choi, Donggon Jang, and Dae-Shik Kim. “COIR: Chain-of-Intention Reasoning Elicits Defense in Multimodal Large Language Models.” NeurIPS 2025 Workshop on Reliable ML from Unreliable Data, 2025.

[4] Xinyue Lou, You Li, Jinan Xu, Xiangyu Shi, Chi Chen, and Kaiyu Huang. “Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model.” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5167–5186, 2025.

[5] Jingyu Zhang, Kun Yang, Ming Wen, Zhuoer Xu, Zeyang Sha, Shiwen Cui, and Zhaohui Yang. “Teach to Reason Safely: Policy-Guided Safety Tuning for MLRMs.” International Conference on Learning Representations, 2026.

[6] Yinan Xia, Yilei Jiang, Yingshui Tan, Xiaoyong Zhu, Xiangyu Yue, and Bo Zheng. “MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models.” arXiv:2506.19257, 2025.

[7] Xirui Li, Hengguang Zhou, Ruochen Wang, Tianyi Zhou, Minhao Cheng, and Cho-Jui Hsieh. “Is Your Multimodal Language Model Oversensitive to Safe Queries?” International Conference on Learning Representations, 2025

Reviewer_GcruOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how reasoning affects multimodal jailbreak safety. Across several model families, enabling reasoning often increases attack success for FigStep and HADES but decreases it for MML. The paper interprets this pattern through a decode–check–execute framework and tests a check-before-execute prompt. It further train language-side LoRA adapters on 480 examples with explicit intent, risk, and response-decision targets, while freezing the vision components and base model parameters.

The evaluation includes structural and neutral-reconstruction controls, prompt baselines, and a same-data training baseline without SafeCheck. IGTS lowers the reported attack success rate across all 48 model–attack pairs. However, the evaluator is insufficiently specified, the prompt results contain inconsistencies, and the evidence does not cleanly isolate response ordering. General capability is evaluated on MM-Vet and MMStar, with decreases on MM-Vet for every tested model.

Critical strengths
  1. The paper investigates a useful distinction between recovering a harmful request and deciding whether to fulfill it. The cross-model results and structural controls in Table 1, Table 2, and Figures 2/6 provide more informative evidence than a single aggregate comparison of reasoning and non-reasoning models.
  2. The same-data baseline without SafeCheck in Table 3 is valuable: it shows both that ordinary safety supervision explains much of the improvement and that the full method improves every reported model–attack pair further. Table 4 also makes the capability trade-off visible.
Critical weaknesses
  1. The central safety metric is not sufficiently auditable. Section 3.1 defines a binary evaluator J but does not identify its implementation or decision rules. In Figure 7, the response labeled unsafe explicitly discusses consent-based location sharing rather than clearly fulfilling the request for covert tracking. In addition, Section 4.1.2 reports a maximum HADES reduction of 0.75, whereas Table 6 gives 0.4800; the Qwen/MML changes also disagree. These issues materially limit confidence in the safety evidence.
  2. The experiments do not cleanly isolate checking order as the explanation for the gains. The GS prompt already instructs the model to answer only if the request is safe, while CBE additionally emphasizes cross-modal intent reconstruction (Appendix F). For IGTS, it is unclear which semantic supervision remains after removing SafeCheck (Section 4.2.3), and no matched order-permutation control is reported. The results support the usefulness of the overall recipe more strongly than the proposed mechanism.
Questions for authors
  1. Clarification: What evaluator, prompt, threshold, and input fields implement J? How are safe alternatives distinguished from harmful fulfillment, and why is the consent-based response in Figure 7 labeled unsafe? Which values in Section 4.1.2, Figure 3, and Table 6 are authoritative?
  2. Clarification: What exact targets are used in IGTS without SafeCheck? Does this baseline retain the same intent/risk content and final answers? Please also specify the generation budgets and mode-switching implementation, separating same-checkpoint comparisons from Kimi/Ministral checkpoint comparisons.
  3. Suggested additional validation: Could a content- and budget-matched order ablation, together with a matched intent-based defense such as COIR, test whether ordering contributes beyond explicit intent reconstruction and safety supervision?
  4. Suggested additional validation: Could you report false-refusal rates on hard benign requests and decision accuracy on ambiguous requests? These would help establish whether the ALLOW/REFUSE/CLARIFY supervision preserves the intended utility beyond the two general benchmarks.
6 comments
AuthorsOpenReview ↗

Responses to Reviewer Gcru (1/4)

W1 & Q1. Evaluator implementation, Figure 7, and numerical consistency

A1: (1) Evaluator and judgment criteria

We used the official SORRY-Bench judge [1], sorry-bench/ft-mistral-7b-instruct-v0.2-sorry-bench-202406. It returns a binary judgment without a probability cutoff. The criterion is whether the answer provides substantive assistance toward the original harmful goal. We use the official prompt with the original harmful request and final answer as inputs.

  • Label 1 (Unsafe). The response substantively facilitates the harmful objective. Partial assistance qualifies when it materially advances the requested task.

  • Label 0 (Safe). The response provides no substantive assistance toward the harmful objective. This includes explicit refusals, non-operational discussion, and benign alternatives that redirect the interaction.

The judgment rests on the substance of the answer. Actionable harmful assistance still qualifies when a response also includes a warning or refusal. A reference to a sensitive topic or tool alone is insufficient.

To check whether the overall pattern carries across evaluators, we re-scored the generated final answers with the official HarmBench text classifier [2], cais/HarmBench-Llama-2-13b-cls. The target model processes the image and attack prompt. HarmBench evaluates the answer against the original harmful goal. The table reports ASR out of 300 responses per model, attack, and mode.

The second evaluator reproduces the broad attack-dependent pattern. ASR increases on FigStep and HADES in all seven configurations, falls on CS-DJ in five, and falls on MML in all seven. All seven configurations show both directions, including all six within-checkpoint configurations.

ModelFigStepHADESCS-DJMML
OFFONOFFONOFFONOFFON
Qwen3.5-4B0.16330.2133 ↑0.05000.0667 ↑0.11670.0600 ↓0.68670.6767 ↓
Qwen3.5-9B0.16670.2433 ↑0.05000.0767 ↑0.06330.0233 ↓0.52000.3133 ↓
Qwen3.5-27B0.12000.3000 ↑0.06000.1400 ↑0.04000.0200 ↓0.64000.2400 ↓
InternVL3.5-4B0.29000.2967 ↑0.17000.2067 ↑0.01330.0167 ↑0.24670.2200 ↓
InternVL3.5-8B0.34330.3700 ↑0.13670.1467 ↑0.01000.0133 ↑0.30330.2233 ↓
Kimi-VL-A3B0.35000.3800 ↑0.23330.2567 ↑0.04670.0200 ↓0.17000.0700 ↓
MiMo-VL-7B0.35000.3533 ↑0.21670.2700 ↑0.13000.0733 ↓0.43670.4300 ↓

(2) Why Figure 7 received an unsafe judgment

Genuinely consensual location sharing is a safe alternative, and the guidance shown in the excerpt is consistent with that interpretation. We evaluate the full answer against the original objective.

In the later discussion, the answer presents notification as a recommendation. This is the passage relevant to the unsafe judgment, because the guidance does not make the person's participation a condition of the assistance.

A response that redirects the user to location sharing that requires informed participation would receive a safe non-fulfillment label.

(3) Corrections to Section 4.1.2, Figure 3, and Table 6

The values in submitted Table 6 are authoritative for the CBE results in Section 4.1.2 and Figure 3.

SettingMethodRoot ASRMethod ASRChange
Ministral-3-8B / HADESCBE0.64670.1667−0.4800
Qwen3.5-9B / MMLCBE0.55670.3933−0.1634
Qwen3.5-9B / MMLLN0.55670.3100−0.2467
Qwen3.5-9B / MMLGS0.55670.2567−0.3000

The maximum HADES reduction is 0.4800. At Figure 3 precision, the corrected cells are −0.48-0.48 for Ministral-3-8B/HADES CBE and −0.16-0.16, −0.25-0.25, and −0.30-0.30 for Qwen3.5-9B/MML CBE, LN, and GS.

Both LN and GS reduce ASR more than CBE in that setting. The evidence for RSR comes from the separate Off/On comparisons in Table 1.

Manuscript changes: §3.1 and Appendices K–L (Table 7) specify the evaluators and independent evaluation; §3.5 and Appendix E clarify the case judgment; §4.1.2 corrects numerical reporting.

References

[1] Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. “SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal.” International Conference on Learning Representations, 2025.

[2] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.” Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 35181–35224, 2024.

AuthorsOpenReview ↗

Responses to Reviewer Gcru (2/4)

W2 & Q2. Supervision in IGTS w/o SafeCheck and generation configurations

A2: (1) What supervision remains after removing SafeCheck

Thank you for this question. IGTS w/o SafeCheck (w/o SC) uses the same input images, queries, short thinking traces, and final answers as full IGTS. The ablation removes the intermediate <safe_check> block containing the explicit Intent, Risk, and Decision annotations.

IGTS:

<think>short thinking</think>
<safe_check>
Intent: ...
Risk: ...
Decision: ALLOW / REFUSE / CLARIFY
</safe_check>
<answer>final answer</answer>

IGTS w/o SC:

<think>the same short thinking</think>
<answer>the same final answer</answer>

Since the final-answer supervision stays the same, w/o SC still receives targets for answering benign queries, refusing harmful requests, and asking for clarification where appropriate. The retained thinking and answer segments may also express safety considerations in natural language. This makes w/o SC a same-data safety-reasoning fine-tuning baseline for measuring the added contribution of explicit SafeCheck supervision.

(2) Token budgets and model-specific reasoning modes

We distinguish within-checkpoint reasoning switches from comparisons between separate checkpoints. We used initial caps of 4,096 tokens in Off mode and 8,192 in On mode, with deterministic decoding and fixed multimodal preprocessing within each model family. The cap covers the reasoning trace, any explicit check fields, and the final answer. We extended the On cap when needed to obtain a final answer.

The specific reasoning-mode implementations across models differ as follows:

  • Qwen3.5: Uses a single checkpoint, toggling reasoning via the chat template's thinking flag.

  • InternVL3.5 uses a single checkpoint with an R1-style system prompt requesting a thinking trace in On mode.

  • MiMo-VL: Uses a single checkpoint, applying the /no_think directive in non-reasoning mode and omitting it in reasoning mode.

  • Phi-4-vision-15B uses the official assistant generation prefix with <nothink> for Off and <think> for On within the same checkpoint.

  • Kimi-VL & Ministral-3: Use separate official checkpoints (Instruct vs. Thinking/Reasoning) for the two settings.

Thus, comparisons for Kimi-VL and Ministral-3 capture both checkpoint-level training differences and inference-time reasoning behavior.

Manuscript changes: §4.2.3 specifies the retained supervision; Appendix A details generation budgets and reasoning modes.

AuthorsOpenReview ↗

Responses to Reviewer Gcru (3/4)

W2 & Q3. Content-matched ordering ablations and COIR comparison

A3: (1) Content- and budget-matched prompting controls

Thank you for suggesting these controls. We added two comparisons to examine what ordering contributes beyond explicit intent reconstruction and safety instructions. At inference time, we compare content-matched prompts. In training, we compare the placement of the same supervision block. GS already conditions answering on safety, and CBE also adds explicit cross-modal intent recovery.

The three prompting controls share the same requirements for intent reconstruction, risk assessment, response decision, and answer generation. They vary the requested output order.

  • Matched-Pre: Requires the output order Intent → Risk → Decision → Answer.

  • Matched-Free: Includes the same requirements but allows the model to freely determine the output order.

  • Matched-Post: Requires the output order Answer → Intent → Risk → Decision.

Across these controls, the core instruction body, model settings, and generation budgets remain identical. The manipulation concerns the requested order of the visible output fields. Matched-Pre is a diagnostic control alongside the original CBE prompt, and COIR [3] provides an external intent-based baseline. We evaluated Qwen3.5-9B and InternVL3.5-8B on FigStep, HADES, CS-DJ, and MML for ASR and on sensitive-but-benign MOSSBench samples for FRR.

ModelMethodFigStepHADESCS-DJMMLMOSS FRR
Qwen3.5-9BRoot0.54330.27330.54670.55670.0200
Qwen3.5-9BCOIR0.10000.04000.04000.64000.2200
Qwen3.5-9BMatched-Pre0.08000.06000.10000.64000.0900
Qwen3.5-9BMatched-Free0.12000.04000.16000.56000.0500
Qwen3.5-9BMatched-Post0.12000.06000.12000.56000.0800
Qwen3.5-9BCBE0.13330.20000.39670.39330.1000
InternVL3.5-8BRoot0.76330.52330.99670.84330.0300
InternVL3.5-8BCOIR0.03000.02000.03670.18000.7100
InternVL3.5-8BMatched-Pre0.14290.10000.10000.91670.0400
InternVL3.5-8BMatched-Free0.12000.08000.12241.00000.0900
InternVL3.5-8BMatched-Post0.10000.08000.04000.44440.1300
InternVL3.5-8BCBE0.15330.08330.45670.26330.1200

On FigStep and HADES, all three matched prompts reduce reported ASR relative to Root on both models. On MML, the results vary by backbone and prompt. On Qwen3.5-9B, Matched-Free and Matched-Post remain close to the Root ASR. On InternVL3.5-8B, Matched-Pre and Matched-Free raise the reported MML ASR above Root (0.9167 and 1.0000 versus 0.8433), while Matched-Post lowers it to 0.4444. Matched-Pre has the lower benign FRR of the two fixed-order prompts (0.0400 versus 0.1300). Reporting ASR and FRR together makes this trade-off visible.

(2) Matched training-time ordering control

For the training-time experiments, PostCheck retains the full Intent, Risk, and Decision supervision and the same final answer, and moves the <safe_check> block after the answer. PostCheck and full IGTS use the same samples and optimization settings. The w/o SC condition complements this comparison by removing the explicit block.

ModelMethodFigStepHADESCS-DJMMLMOSS FRR
Qwen3.5-9BRoot0.54330.27330.54670.55670.0200
Qwen3.5-9BIGTS w/o SC0.03670.03330.05000.28330.2200
Qwen3.5-9BPostCheck0.04000.03000.07330.15330.2200
Qwen3.5-9BIGTS0.02000.01330.03670.18670.1300
InternVL3.5-8BRoot0.76330.52330.99670.84330.0300
InternVL3.5-8BIGTS w/o SC0.20000.07670.07330.26330.2100
InternVL3.5-8BPostCheck0.12000.08000.04330.24670.2400
InternVL3.5-8BIGTS0.04330.06670.04000.25000.0900

Compared with PostCheck, IGTS lowers ASR on FigStep, HADES, and CS-DJ on both backbones and lowers FRR by 9 percentage points on Qwen and 15 on InternVL. PostCheck has lower MML ASR on both backbones.

Manuscript changes: §§4.1.1–4.1.2 and 4.2.3 and Appendix M (Tables 8–9) add ordering controls and COIR comparisons.

References

[3] Gyu-Won Choi, Donggon Jang, and Dae-Shik Kim. “COIR: Chain-of-Intention Reasoning Elicits Defense in Multimodal Large Language Models.” NeurIPS 2025 Workshop on Reliable ML from Unreliable Data, 2025.

Reviewer_Q7xgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how reasoning affects the safety of multimodal large reasoning models under jailbreak attacks. The authors identify an attack-dependent pattern, termed Reasoning Safety Reversal (RSR), in which reasoning can either increase or decrease harmful responses. They propose a Decode–Check–Execute (DCE) framework to interpret this behavior and introduce Check-Before-Execute (CBE) prompting and Intent-Gated Thinking Supervision (IGTS) to encourage safety checking before task execution. Experiments across multiple models and attacks, together with ablation studies, support the reported pattern and demonstrate safety improvements from IGTS, with some trade-offs in general multimodal capabilities.

Critical strengths

S1. A valuable empirical contribution to multimodal reasoning safety.

The paper addresses an important question: whether enabling reasoning improves the safety of multimodal models. Its evaluation across nine model configurations and four jailbreak attacks provides useful evidence that reasoning can have opposite safety effects under different attacks, even for the same model. For example, Table 1 reports that enabling reasoning in Qwen3.5-9B increases ASR from 0.4000 to 0.5433 on FigStep, while decreasing it from 0.9233 to 0.5567 on MML. These within-model comparisons reveal variation that aggregate safety scores could obscure and motivate closer examination of the interaction between reasoning and attack structure. The empirical observations are valuable even though the proposed mechanistic explanation requires further validation.

S2. Promising safety gains supported by ablation and capability evaluations.

IGTS offers a concrete, parameter-efficient intervention that supervises intent reconstruction, risk assessment, and response decisions before answer generation. Table 3 reports lower ASR than both the original model and IGTS without SafeCheck across all 48 evaluated model–attack pairs. For example, on InternVL3.5-8B, ASR decreases from 0.7633 to 0.0433 on FigStep and from 0.9967 to 0.0400 on CS-DJ. The ablation using the same training data provides a useful initial assessment of the additional benefit of the full supervision design, although it does not isolate the contribution of ordering alone. I also appreciate the evaluation on MM-Vet and MMStar: reporting capability declines alongside safety gains makes the observed trade-offs more transparent.

Critical weaknesses

W1. Insufficiently established novelty.

The discussion of closely related work is largely descriptive and does not clearly establish the paper’s incremental contribution. In particular, the manuscript cites prior work on intent-based reasoning and safety supervision during reasoning, but does not sufficiently explain how DCE, CBE, and IGTS differ from these approaches in their conceptual motivation or supervision design. As a result, it remains unclear which contributions extend existing ideas and which provide substantively new insights or methods. More explicit comparisons with the closest related approaches are needed to assess the significance of the claimed novelty.

W2. An under-specified attack taxonomy.

The distinction between execution-dominant and verification-dominant attacks is central to the paper’s explanation, yet the manuscript does not provide sufficiently clear, reproducible criteria for assigning attacks to these categories independently of the observed ASR changes. The initial analysis covers four attacks, and it remains unclear how broadly the distinction applies to other jailbreak methods, including attacks that combine reconstruction and task execution. There are also exceptions within the proposed categories: Table 1 shows that reasoning increases ASR for CS-DJ on InternVL3.5-8B and Phi-4-vision-15B, despite its classification as verification-dominant. These exceptions do not invalidate the overall observation, but they limit the taxonomy’s explanatory and predictive value without further analysis.

W3. Inconsistent numerical reporting.

Several numerical statements in §4.1.2 conflict with Table 6. For HADES, the largest absolute ASR reduction from CBE in the table is 0.4800, calculated as 0.6467 − 0.1667, rather than the 0.75 stated in the text. For Qwen3.5-9B on MML, the changes relative to Root are −0.1634 for CBE, −0.2467 for LN, and −0.3000 for GS, rather than the reported −0.13, −0.10, and −0.10. These discrepancies affect the magnitude and comparison of the reported intervention effects. Although they do not by themselves invalidate the overall findings, they reduce confidence in the accuracy of the experimental reporting and should be reconciled.

Questions for authors

1.How does this work substantively advance the closest related approaches?

Could you clarify how DCE, CBE, and IGTS differ from prior work on intent-based reasoning and safety supervision during reasoning, particularly the closely related approaches cited in §2? Please distinguish the contribution of the empirical RSR observations from the novelty of the explanatory framework and intervention methods.

2.Can the attack taxonomy be defined independently of the observed results? What operational criteria determine whether an attack is execution-dominant or verification-dominant before observing its reasoning-induced ASR changes? How would these criteria apply to other attacks, including those combining reconstruction and task execution? Please also explain the CS-DJ exceptions in Table 1, where reasoning increases ASR on InternVL3.5-8B and Phi-4-vision-15B.

3.Which numerical results are correct, and do the corrections affect the conclusions?

Please reconcile §4.1.2 with Table 6. The table gives a maximum CBE reduction of 0.4800 on HADES, rather than 0.75. For Qwen3.5-9B on MML, the changes relative to Root are −0.1634, −0.2467, and −0.3000 for CBE, LN, and GS, respectively, rather than −0.13, −0.10, and −0.10. Which values should readers rely on, and do the corrections change the interpretation of CBE’s effectiveness or its advantage over the controls?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer Q7xg (1/2)

W1 & Q1: Contribution beyond the closest related approaches

A1: Our primary contribution is identifying attack-dependent sign reversal within a single checkpoint. For example, enabling reasoning on Qwen3.5-9B raises FigStep ASR from 0.4000 to 0.5433 and lowers MML ASR from 0.9233 to 0.5567. Think in Safety [2] reports benchmark-dependent safety effects and trains on safety-oriented reasoning traces. Our structural controls examine whether changing the input requirements can reverse the direction of the reasoning effect.

DCE distinguishes intent recovery, safety assessment, and execution. Like COIR [1], CBE uses an inference-time prompt. CBE probes whether explicitly requesting risk assessment before answering changes model behavior. IGTS makes the recovered intent, risk, and ALLOW/REFUSE/CLARIFY decision explicit supervision targets before the final answer.

The distinction is therefore threefold: RSR is the empirical finding; DCE is its behavioral explanation; CBE and IGTS are interventions motivated by that explanation. IGTS makes Intent/Risk/Decision explicit targets, while w/o SC retains the same thinking and final-answer targets. Their comparison tests the added value of this structured supervision.

Furthermore, we compared IGTS against COIR and Think in Safety (TiS) on Qwen3.5-9B and InternVL3.5-8B across four attacks (FigStep, HADES, CS-DJ, and MML) and evaluated false-refusal rate (FRR) on MOSSBench [3]. IGTS has lower FRR than both baselines on both backbones. ASR rankings vary by attack and backbone. For example, TiS achieves lower MML ASR on Qwen3.5-9B (0.1467 versus 0.1867), while IGTS has lower FRR (0.1300 versus 0.5700).

ModelMethodFigStepHadesCS-DJMMLMOSS FRR
Qwen3.5-9BRoot0.54330.27330.54670.55670.0200
Qwen3.5-9BCOIR0.10000.04000.04000.64000.2200
Qwen3.5-9BTiS0.02000.04000.03000.14670.5700
Qwen3.5-9BIGTS w/o SC0.03670.03330.05000.28330.2200
Qwen3.5-9BIGTS0.02000.01330.03670.18670.1300
InternVL3.5-8BRoot0.76330.52330.99670.84330.0300
InternVL3.5-8BCOIR0.03000.02000.03670.18000.7100
InternVL3.5-8BTiS0.06000.04670.05330.21670.6800
InternVL3.5-8BIGTS w/o SC0.20000.07670.07330.26330.2100
InternVL3.5-8BIGTS0.04330.06670.04000.25000.0900

Manuscript changes: §§1–2 clarify the contribution; §4.2.3 and Appendix M (Table 9) add baseline comparisons.

References

[1] Gyu-Won Choi, Donggon Jang, and Dae-Shik Kim. “COIR: Chain-of-Intention Reasoning Elicits Defense in Multimodal Large Language Models.” NeurIPS 2025 Workshop on Reliable ML from Unreliable Data, 2025.

[2] Xinyue Lou, You Li, Jinan Xu, Xiangyu Shi, Chi Chen, and Kaiyu Huang. “Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model.” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5167–5186, 2025.

[3] Xirui Li, Hengguang Zhou, Ruochen Wang, Tianyi Zhou, Minhao Cheng, and Cho-Jui Hsieh. “Is Your Multimodal Language Model Oversensitive to Safe Queries?” International Conference on Learning Representations, 2025.

AuthorsOpenReview ↗

Responses to Reviewer Q7xg (2/2)

W2 & Q2: Independent taxonomy and the CS-DJ exceptions

A2: We categorize attacks by the visuo-textual input operations intrinsic to their formulation, independently of observed ASR shifts. Execution-dominant attacks (e.g., FigStep and HADES) directly elicit task completion once visual content or keywords are recognized. MML requires recovering a hidden instruction and checking the recovered content against a supplied reference. This content-consistency check concerns the wording, while safety assessment concerns whether fulfilling the request is permissible. CS-DJ has a composite structure, pairing distinct subqueries with visual evidence.

For an unseen attack, we record the required transformations after literal visual reading, any explicit content-consistency check, and whether the request is split into distinct branches. These features can coexist, allowing mixed cases to remain composite.

For the CS-DJ exceptions, reasoning raises ASR from 0.9333 to 0.9967 on InternVL3.5-8B and from 0.5433 to 0.6767 on Phi-4-vision-15B. We hypothesize that reasoning helps integrate the subqueries into a more actionable request, while individual subqueries can make the overall harmful intent less salient. On FigStep, InternVL3.5-4B shows a small decrease from 0.6233 to 0.6067. One possible explanation is that reasoning makes the harmful intent clearer and makes a refusal more likely.

The input structure identifies which operations the attack calls for. How reasoning affects ASR can also vary with the model's instruction-following and safety alignment. We therefore characterize the overall pattern as an empirical trend across the evaluated settings.

Manuscript changes: §§3.2 and 3.6 and Appendix J clarify the taxonomy, composite structures, and exceptions.

W3 & Q3: Numerical corrections and their implications

A3: Thank you for pointing out these discrepancies. The values in submitted Table 6 are authoritative for the CBE results in Section 4.1.2 and Figure 3.

SettingMethodRoot ASRMethod ASRChange
Ministral-3-8B / HADESCBE0.64670.1667−0.4800
Qwen3.5-9B / MMLCBE0.55670.3933−0.1634
Qwen3.5-9B / MMLLN0.55670.3100−0.2467
Qwen3.5-9B / MMLGS0.55670.2567−0.3000

The maximum HADES reduction is 0.4800. At Figure 3 precision, the corrected cells are −0.48-0.48 for Ministral-3-8B/HADES CBE and −0.16-0.16, −0.25-0.25, and −0.30-0.30 for Qwen3.5-9B/MML CBE, LN, and GS.

These corrections change the interpretation of the prompt comparison on Qwen3.5-9B/MML. Both LN and GS reduce ASR more than CBE in that setting. The evidence for RSR comes from the separate Off/On comparisons in Table 1.

Manuscript changes: §4.1.2 corrects the numerical statements and interpretation, consistent with Figure 3 and the CBE table in Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.