acceptodds
Withdrawn from ICLR 2027

StageBreak: Diagnosing Stage-Selective Safety Failures in Multimodal Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Jailbreak evaluation usually reduces multimodal safety to a single endpoint: whether an unsafe response is produced. That endpoint conflates three different cases. The model may fail to recognize risk, lose an early refusal preference, or cross the safety boundary later in decoding. We introduce StageBreak a diagnostic framework that assigns the earliest valid operational failure among risk recognition, refusal preference, and safe generation while preserving the required upstream criteria. A stage attribution is accepted only for clean-eligible examples whose risky task semantics remain invariant and whose harmful endpoint, prerequisite-preservation, and audit gates agree on the same sample. Low-rank repair, reverse transplantation, cross-stage controls, and cross-attack transfer then measure the interventional relevance of the attributed representation. The resulting attribution distinguishes semantic corruption from recognition, refusal-preference, and generation-time failures. We evaluate whether these attributed failures persist across tasks, external benchmarks, and held-out models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal jailbreak, safety alignment, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dNaKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StageBreak, a framework for diagnosing jailbreak failures in MLLMs. The authors decompose the model's safety process into four stages: URDG. Based on prerequisite constraints, the framework attributes successful jailbreak attacks to the earliest operational failure stage.

Methodologically, the authors combine hidden-state probing, refusal likelihood margins, constrained adversarial optimization, and low-rank activation interventions to enable stage-targeted attacks and failure attribution. The authors further employ targeted repair, reverse transplantation, and cross-stage controls to validate the attribution results, and evaluate their stability across multiple models, attack settings, and external benchmarks.

Critical strengths

S1. Clear motivation and well-organized presentation.

The paper addresses a meaningful limitation of conventional jailbreak evaluation, which primarily relies on Attack Success Rate (ASR) without investigating the underlying causes of safety failures. Moving beyond binary attack success toward fine-grained failure diagnosis is a worthwhile research direction. The paper is generally well structured, and the proposed pipeline is presented in a relatively clear and systematic manner.

S2. Attempts to validate attribution through independent auditing and interventions.

The authors separate attack optimization from stage auditing, reducing the risk of using identical measurement instruments for both attack construction and evaluation. Furthermore, targeted repair, reverse transplantation, and cross-stage controls are introduced to investigate the relevance of the attributed representations rather than relying solely on probe-based classification.

S3. Broad experimental coverage.

The experiments cover multiple multimodal models, attack methods, and external benchmarks, along with various ablation studies and cross-model transfer analyses. These experiments provide a substantial empirical basis for examining the applicability of the proposed framework.

Critical weaknesses

W1. Limited methodological novelty and insufficient justification of the framework's complexity.

The proposed framework primarily combines existing techniques, including hidden-state probing, likelihood-based refusal measurement, constrained optimization, and SVD-based activation intervention. Moreover, the R/D/G decomposition is conceptually similar to prior structured safety-decision frameworks.[1] The paper does not sufficiently clarify what fundamentally new methodological contribution emerges from this combination. In particular, it remains unclear whether stage-specific attribution and intervention offer meaningful advantages over simpler, stage-agnostic alternatives.

W2. Insufficient validation of stage attribution and limited scientific insights.

The R/D/G stages are operationally defined through heterogeneous measurements and manually specified logical gates, without sufficient evidence that they correspond to distinct causal safety mechanisms. Although the authors provide intervention experiments and cross-model analyses, these results primarily demonstrate consistency with the proposed attribution criteria rather than independently validating the underlying stage decomposition. Furthermore, several experimental findings follow naturally from the predefined constraints and evaluation metrics, offering limited new insights into the mechanisms of multimodal jailbreak failures.

[1]. Gu, T., Huang, K., Li, L., Luo, R., Huang, S., Wang, Z., ... & Wang, Y. (2026). From sparse decisions to dense reasoning: A multi-attribute trajectory paradigm for multimodal moderation. arXiv preprint a

Questions for authors

Q1. Necessity of stage-specific decomposition.

Can the authors compare StageBreak against a simpler stage-agnostic baseline, such as a unified safety direction or a pooled clean–attack activation subspace? Would the proposed stage-specific interventions still provide significant benefits under matched experimental settings?

Q2. Validity and scientific insights of stage attribution.

What evidence supports the claim that R/D/G represent distinct causal failure mechanisms rather than different operational measurements of a shared safety-related representation? Beyond validating the predefined attribution criteria, what new insights into multimodal jailbreak failures does the framework reveal?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dNaK

We thank the reviewer for recognizing StageBreak's motivation, independent auditing, and broad experimental evaluation. We particularly appreciate the questions concerning methodological novelty, the necessity of stage decomposition, and the scientific insights supported by our framework.

1. Methodological contribution and necessity of stage decomposition (W1, Q1).

StageBreak's central contribution is prerequisite-preserving failure attribution: identifying which safety criterion becomes invalid while its upstream prerequisites and the original risk semantics remain intact. This goes beyond detecting safety-related signals or combining multiple safety measurements.

Equations (6)–(9) formalize sample-level eligibility, independent stage auditing, and constrained stage-targeted optimization. Importantly, separating attack steering from auditing prevents the optimization objective from directly determining its own evaluation outcome.

We appreciate the connection to Gu et al. (2026). Their UniMod framework develops structured reasoning trajectories and multi-attribute supervision to improve multimodal moderation. StageBreak addresses a different question: when an existing MLLM produces a harmful response under attack, which safety criterion failed, and can the corresponding representation be identified through independent auditing and intervention? Thus, the distinction lies in the research objective, attribution requirements, and interventional validation, rather than simply the number of modeled stages.

To directly test whether stage-specific decomposition provides benefits beyond a unified safety representation, we conducted an additional free-decoding evaluation on matched attacked samples using rank-2 interventions.

For R/D/G, strict ASR without intervention is 0.718/0.744/0.701. Unified rank-2 repair reduces these values to 0.523/0.551/0.536, while stage-specific repair further reduces them to 0.359/0.387/0.418, yielding additional absolute reductions of 0.164/0.164/0.118.

These results provide direct behavioral evidence that stage-aware intervention can outperform a pooled, stage-agnostic subspace under matched intervention capacity. Crucially, these improvements are measured from freely decoded responses, rather than being determined by the attribution rules.

2. Attribution validity and scientific insights (W2, Q2).

StageBreak identifies functionally distinguishable failure types and examines their representation-level relevance through interventions that provide evidence beyond the predefined audit criteria.

Three findings are particularly informative.

First, matched-stage repair achieves recovery rates of 0.720/0.690/0.660 for R/D/G, whereas applying a mismatched stage basis to the corresponding failure group yields target recovery no greater than 0.120 (Table 3). This demonstrates intervention selectivity beyond the stage labels themselves.

Second, intervention effects exhibit a structured asymmetry: R-targeted repair can also influence downstream D/G behavior, while D- and G-targeted interventions have much weaker effects on already violated upstream criteria. This reveals stage-dependent functional relationships without requiring a strictly sequential internal causal architecture.

Third, cross-model transfer distinguishes harmful endpoint transfer from failure-identity retention (Figure 3). Transferred Strict ASR is 0.460/0.481/0.444 for R/D/G, while Stage Match among harmful transferred successes reaches 0.828/0.838/0.799. These results reveal that successful transfer and preservation of the diagnosed failure identity capture different aspects of model vulnerability.

Importantly, the attribution gates determine which failures are validly assigned, but do not prescribe the observed intervention outcomes or cross-model transfer behavior. Together with the additional free-decoding comparison, these findings provide evidence of stage-dependent behavioral sensitivity and structured relationships among safety failures, rather than merely reproducing predefined classification rules.

We hope these clarifications and additional behavioral evidence better convey StageBreak's methodological contribution and scientific insights. We would be grateful if the reviewer could reconsider the assessment in light of these results.

Reviewer_QLA8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StageBreak, a diagnostic framework for identifying stage-specific safety failures in multimodal large language models (MLLMs). Instead of evaluating jailbreaks solely through attack success rates, StageBreak distinguishes three failure stages: risk recognition (R), refusal preference (D), and safe generation (G). The framework uses prerequisite-preserving constraints and independent auditing to attribute failures, followed by low-rank activation repair and reverse transplantation to validate stage-associated representations.

Critical strengths

1. Meaningful and fine-grained safety failure diagnosis. The paper moves beyond conventional attack success rate evaluation by distinguishing failures in risk recognition, refusal preference, and safe generation. Its prerequisite-preserving attribution mechanism helps avoid confusing genuine safety failures with task understanding or semantic corruption.

2. Comprehensive experimental validation with intervention-based analysis. The paper evaluates StageBreak across multiple models, attack families, and external benchmarks. More importantly, low-rank activation repair, reverse transplantation, and cross-stage controls provide evidence that the identified stage-associated representations have meaningful intervention effects, strengthening the credibility of the proposed diagnostic framework.

Critical weaknesses

1. Limited Demonstration of Practical Utility.​ Although the paper motivates stage-level diagnosis as a means to identify which safety mechanisms should be strengthened, its practical utility remains insufficiently demonstrated. The intervention experiments provide evidence of stage-associated representations, but do not establish whether stage-level diagnosis leads to more effective or efficient safety improvements compared to stage-agnostic approaches.

2. Unclear connection between representation-level repair and actual behavioral improvement. The intervention experiments employ teacher-forced replay, but it is unclear whether the "ASR after" values in Table 3 are obtained through fresh free decoding after intervention. The paper also lacks sufficient details on clean–adversarial trajectory alignment and how activation patches are applied during generation. Recovering audit scores along fixed trajectories does not necessarily demonstrate improved safety in actual model outputs. Furthermore, Table 3 reports a single aggregated no-intervention ASR, while repair results are evaluated on different stage-specific subsets, without corresponding stage-wise baselines or sample sizes. These issues make it difficult to assess the actual behavioral effectiveness and stage specificity of the interventions.

Questions for authors

1.Can the authors demonstrate how StageBreak's stage-level diagnosis benefits downstream safety improvement, such as targeted safety fine-tuning or defense?

2. Validation of Stage Attribution. Does the human evaluation validate final harmfulness, risk-semantic preservation, or the complete R/D/G stage assignments? Please clarify the evaluation sample sizes and report stage-wise misclassification patterns and unattributed sample rates. In particular, how reliably does the D-stage prefix likelihood margin correlate with actual refusal behavior under free decoding on adversarial inputs?

3. Intervention Protocol and Behavioral Effectiveness. How are the "ASR after" values in Table 3 obtained from teacher-forced interventions? Are complete responses regenerated through free decoding after applying the interventions? How are clean and adversarial trajectories aligned, and at which positions are activation patches applied? Please provide stage-specific ASR before and after intervention, along with sample sizes and uncertainty estimates, to clarify whether the improvements reflect actual behavioral changes rather than changes in audit scores.

4. Consistency Between Figure 4 and Table 7. Figure 4 marks ((H,W)=(14,6)) as the fixed setting used in Table 7. However, based on the color scale, the corresponding values for VQA and Image Dialogue appear to be approximately 0.70 and 0.75, respectively, whereas the stage-specific SASR values in Table 7 range from 0.525–0.611 and 0.548–0.624. Do these results use different metrics, evaluation subsets, denominators, or aggregation procedures? Additionally, how exactly do (H) and (W) enter the stage attribution procedure defined in Eq. (8)? Clarifying these points is important for interpreting the sensitivity analysis.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QLA8

We thank the reviewer for recognizing the value of fine-grained safety diagnosis, prerequisite-preserving attribution, and intervention-based validation. The questions help us clarify how StageBreak's diagnostic evidence translates into behavioral improvements.

1. Practical utility and behavioral improvement (W1, W2, Q1).

To directly assess the benefit of stage-level diagnosis, we conducted an additional free-decoding evaluation comparing stage-specific repair with a unified safety subspace, using matched attacked samples and rank-2 interventions.

Under free decoding, strict ASR for R/D/G is 0.718/0.744/0.701 without intervention, 0.523/0.551/0.536 with unified rank-2 repair, and 0.359/0.387/0.418 with stage-specific repair.

While the unified subspace improves safety across all three groups, stage-specific repair achieves further absolute ASR reductions of 0.164/0.164/0.118 for R/D/G. These results provide direct behavioral evidence that stage-aware intervention can outperform a shared, stage-agnostic basis under matched intervention capacity.

2. Intervention protocol and actual generation (Q3).

Equations (11)–(12) define interventions using paired clean–adversarial activations at pre-specified layer–token positions WsW_s. Teacher-forced replay provides aligned activation references for the original Table 3 intervention protocol.

The additional experiment instead evaluates strict harmfulness on complete responses generated through free decoding under intervention, rather than relying on audit-score changes along fixed trajectories. Its stage-wise no-intervention baselines and matched intervention comparisons are therefore distinct from the original Table 3 analysis. We will clarify the activation-alignment and patching procedures, evaluation sample sizes, and uncertainty estimates in the supplementary protocol.

3. Attribution validity and human evaluation (Q2).

Human adjudication evaluates the reliability of behavioral and semantic judgments, while R/D/G identities are assigned through the independent prerequisite-aware audit in Eq. (8). Accordingly, human agreement is not treated as direct accuracy of complete stage assignments.

Appendix Table 6 reports 0.909 macro human agreement, while Table 10 reports a 0.884 correlation between the D-stage refusal margin and free-decoding refusal behavior, remaining 0.821 under held-out attack families.

Table 2 further reports Stage Match values of 0.904/0.917/0.895 for R/D/G. The corresponding non-match fractions (0.096/0.083/0.105) include both alternative stage identities and unattributed harmful outcomes, rather than constituting misclassification rates.

Clean eligibility is determined before attack, while post-attack semantic or task-understanding violations invalidate stage attribution. Inconsistent or incomplete stage evidence can further leave harmful responses unattributed. We will clarify this accounting and the human-evaluation sample sizes.

4. Figure 4 and Table 7 consistency (Q4).

We appreciate the reviewer's careful observation. Table 7 gives task-averaged Strict ASRs of 0.717/0.760 for VQA/Image Dialogue, compared with conditional SASRs of 0.563/0.594. The apparent Figure 4 values are closer to the former. We will reconcile the plotting data, metric definition, and aggregation denominator, and correct the corresponding presentation.

Here, HH specifies the recent decoded context supplied to the temporal auditor, while WW specifies the consecutive audit positions used to construct stage evidence. They affect the audit inputs underlying Eq. (8), rather than introducing additional attribution gates. We will clarify this connection and distinguish audit-dependent stage attribution from endpoint-level attack success.

We hope the additional behavioral evidence and these clarifications address the reviewer's concerns regarding practical utility and experimental validity. We would be grateful if the reviewer could reconsider the assessment in light of these results.

Thank you for the rebuttal and the additional experiments. However, some of my concerns remain unresolved. Therefore, I will assign an overall score of 2.

Reviewer_CCsHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StageBreak, a diagnostic framework models multimodal safety as a prerequisite-ordered sequence: task understanding (UU), risk recognition (RR), refusal preference (DD), and safe generation (GG) and decomposes jailbreak failures into RR, DD, and GG. To decouple attack optimization from evaluation, the framework uses disjoint probes, refusal and compliance prefix banks, and risk-semantic verifiers for stage-targeted perturbation search versus post-attack stage auditing. The authors further apply low-rank interventions on hidden states to test whether each attributed stage corresponds to a distinct activation subspace. Experiments across seven open-weight MLLMs on StageAudit, USB-Hard, and VLSU evaluate stage-valid attack success, cross-model transfer, and stage-specific intervention recovery.

Critical strengths

The core motivation is well-founded: multimodal safety evaluation should not be reduced to a single overall attack success rate, as identical harmful outputs can stem from fundamentally different internal failure modes. Decoupling the optimization instruments from the audit instruments prevents overfitting the attack to the evaluation metric, and the ablation results confirm that sharing probes or removing independent auditing substantially degrades stage match. The low-rank SVD interventions provide clear interventional evidence for stage specificity, as repairing activations with the matched stage basis restores safety far more effectively than using off-diagonal stage bases.

Critical weaknesses

The prerequisite-ordered taxonomy seems to be heuristic and assumes a strictly feed-forward trajectory, overlooking recursive or non-monotonic dynamics where risk recognition only emerges mid-decoding or oscillates with self-correction. Consequently, a sizable fraction of successful jailbreaks remain unattributed, and reporting the continuous mean change in task understanding obscures the actual binary gate rejection rate between stage-matched harmful responses and stage-valid successes. All experiments are limited to 7B–8B dense models, and the rank-2 subspace repair relies on teacher-forced replay with the paired clean activation of the same prompt, while also using extra proposal-validation queries beyond the control budget. Given that recovery drops substantially under population references or across attack families, do the fixed refusal prefixes and static rank-2 linear subspace assumption still hold on stronger reasoning VLMs or dynamically routed MoE models?

Questions for authors
  1. How can the prerequisite-ordered formulation handle non-monotonic trajectories where risk recognition or self-correction occurs mid-generation, and what are the main causes behind the unattributed harmful responses?
  2. Do the fixed prefix refusal metric and the static rank-2 linear subspace assumption still hold when scaled to stronger models such as Qwen3.5-9B/27B or MoE architectures where perturbations alter expert routing?
2 comments
AuthorsOpenReview ↗

Response to Reviewer CCsH

We thank the reviewer for recognizing StageBreak's motivation, independent auditing, and intervention-based validation. We appreciate the thoughtful questions, which help clarify the interpretation of stage attribution and the scope of our intervention findings.

1. Prerequisite ordering and non-monotonic behavior.

The U/R/D/G ordering specifies an operational attribution protocol, rather than a strictly feed-forward model of internal safety computation. It identifies the earliest violated audited prerequisite, not necessarily the chronological onset of an internal failure.

StageBreak explicitly examines decoding-time behavior. Equations (4)–(5) distinguish transient safety-margin fluctuations from sustained late violations by requiring an initially safe prefix followed by m=3m=3 consecutive unsafe-side steps. More complex trajectories involving renewed risk recognition or self-correction can remain outside this operational classification, preserving the distinction between supported and ambiguous attributions.

Importantly, semantic or task-understanding violations invalidate stage eligibility, whereas probe–behavior disagreement and incomplete stage evidence can leave otherwise harmful responses unattributed. The continuous U-retention score measures preservation strength, while the binary U✓U^\checkmark gate in Eq. (7) determines sample-level validity. We will make this distinction and the gate-wise accounting clearer.

2. Intervention specificity and behavioral effectiveness.

Rank-2 intervention provides a controlled, matched-capacity setting for examining stage-associated representations. Paired clean activations isolate within-sample intervention effects, while population-reference and cross-attack controls assess portability.

Complementing the teacher-forced intervention analysis in Table 3, our additional free-decoding evaluation measures strict ASR on matched attacked samples. For R/D/G, no-intervention ASR is 0.718/0.744/0.701. Unified rank-2 repair reduces these values to 0.523/0.551/0.536, whereas stage-specific repair further reduces them to 0.359/0.387/0.418. Thus, stage-specific intervention provides additional absolute ASR reductions of 0.164/0.164/0.118 over the unified subspace, supporting distinguishable stage-associated behavioral effects.

Regarding computational accounting, Table 2 reports 522/560/624 total queries for StageBreak-R/D/G versus 512 for the same-optimizer controls. The common search-cycle cap and additional proposal-validation queries are distinct components of this accounting; we will clarify this distinction.

3. Stronger VLMs and MoE architectures.

The D-stage metric uses 16 semantically matched refusal/compliance prefix pairs per audit bank, with separate optimization and auditing banks. Its operating point is calibrated per model against free-decoding refusal behavior and paraphrase variants, rather than using an architecture-independent threshold.

The present mechanism analysis establishes stage-associated effects on the evaluated dense VLMs. Extending this analysis to stronger reasoning models and dynamically routed MoE architectures would involve architecture-specific calibration and potentially expert-conditioned interventions.

Together, these findings support StageBreak as a prerequisite-aware diagnostic framework with independently audited attribution and complementary representation-level and behavioral intervention evidence.

AuthorsOpenReview ↗

Gentle Reminder of Rebuttal

Dear Reviewer CCsH,

Thank you again for your valuable comments and suggestions. We would sincerely appreciate it if you could take another look at our rebuttal and revised manuscript and let us know whether our clarifications have addressed your concerns. We would be very happy to discuss any remaining questions or suggestions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.