acceptodds
Under review as a conference paper at ICLR 2027

Sparse Routing Fragments Safety: Diagnosing and Aligning Multimodal MoEs

Abstract

Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling multimodal models via sparse, conditional computation. Yet, dynamically routing tokens across specialized experts introduces fundamental challenges for cross-modal safety alignment. In this work, we demonstrate that multimodal MoEs exhibit pronounced vulnerabilities when transferring textual safety policies to semantically identical visual inputs. While text and visual prompts activate substantially divergent pathways, controlled route transplantation during inference fails to restore visual safety, showing that runtime execution paths alone cannot account for the breakdown. Examining training dynamics reveals Safety Write Fragmentation, where equivalent safety supervision disperses across largely disjoint parameters and exhibits conflicting update directions, thereby limiting cross modal accessibility of learned safety policies. To resolve this fragmentation, our analysis indicates that writing updates to shared parameter substrates restores cross modal transfer, while policy divergence concentrates primarily at generation onset. Building on these diagnostic findings, we introduce SOT (Safety Onset Transport), which equips models with lightweight shared parameters and steers visual safety policies starting from the generation onset. Across three MoE architectures, SOT mitigates cross modal blind spots, lifting multimodal safety from 54.1% to 79.4% on DeepSeek and 82.7% to 88.9% on Qwen3, while keeping core utility largely intact, matching base performance at 80.0% and 86.0% MMBench circular accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Safety, Mixture-of-Experts, Cross-Modal Safety Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_tJKzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Multimodal MoEs show a larger text-safe/image-unsafe gap than a comparable dense model, and inference-time route transplantation does not fix it. The authors attribute this to “Safety-Write Fragmentation”: text and image safety gradients land on largely disjoint experts and point in poorly aligned directions. They propose SOT, a shared low-rank residual trained with text/visual margin losses and first-token text-to-vision distillation, which improves MM-Safe on three MoE VLMs (e.g., 54.1→79.4% on DeepSeek-VL2-Small).

Critical strengths
  1. A careful diagnostic chain (routing → gradient geometry → generation onset), with an informative negative result on route transplantation.
  2. Sound evaluation: three seeds, Repair/Regression metrics, OOD testing, and judge validation.
Critical weaknesses
  1. The diagnosis does not connect to the method. All baselines already use the same shared adapter, so SOT’s gains come from its losses, not from the “shared substrate” finding. SOT is not MoE-specific and is never tested on a dense model. The MoE-vs-dense claim rests on a single model pair.

  2. The trade-offs are understated. On DeepSeek, over-refusal rises sharply: MOSSBench 1.3→18.3, the highest of all methods. On Qwen3, MM-PT beats SOT on MM-Safe and Repair. The largest gains come from in-distribution FigStep, and prefilling/adaptive attacks are not evaluated.

Questions for authors
  1. How does SOT perform with routed-only adapters, i.e., does the shared substrate actually matter?

  2. Does SOT give a smaller gain on dense Qwen3-VL-32B?

  3. How much SOT remains under prefilling attacks (a forced non-refusal first token) and under adaptive attacks targeting SOT?

Reviewer_WAG8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-modal safety degradation in MOE models. The authors attribute this failure to Safety-Write Fragmentation, where textual and visual inputs induce safety updates on different parameter subsets, and the gradient directions aren't aligned enough. Based on these insights, they propose Safety Onset Transport, combining a shared safety substrate, margin-based alignment, and onset distillation to improve cross-modal safety transfer. Experiments on three multimodal MoE architectures demonstrate improvements in safety performance.

Critical strengths
  1. The paper's analysis of cross-modal safety failures is insightful. It specifically examines the reasons why cross-modal safety transfer is hindered by combining routing transplantation, gradient analysis, and generation-onset interventions.

  2. SOT achieves notable safety improvements across three multimodal MoE architectures, particularly on DeepSeek-VL2-Small.

Critical weaknesses
  1. The correlation in Figure 5 is partly expected from the gradient-based virtual update, it doesn't directly demonstrate how Safety-Write Fragmentation relates to actual cross-modal safety failures. This weakens the evidence supporting the proposed mechanism. More direct evidence linking the fragmentation metrics to cross-modal safety failures in generated responses is needed.

  2. SOT significantly increased over-refusal on DeepSeek-VL2-Small (going from 1.3% to 18.3%). Plus, its improvements vary across different architectures, and does not outperform MM-PT on Qwen3-VL-30B-A3B. These results raise concerns about whether its advantages are stable and how much it costs to boost safety.

Questions for authors
  1. Could the authors show whether lower directional safety-write alignment is associated with a higher rate of text-safe/image-unsafe failures during free generation?

  2. Table 4 only reports the ablation of the two safety transport components without evaluating the contribution of the shared safety substrate. Could the authors provide this ablation under the same settings? This would help clarify whether the shared substrate is necessary for the reported improvements.

Reviewer_DQE2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies Mixture-of-Experts (MoE) vision-language models that refuse a harmful request given as text but answer it when it is given as an image. It reports that text and image versions use different experts (Route Divergence 0.239; Figure 2), and that forcing the image input onto the text input's experts at inference does not restore safety (Section 2.1; Figure 3). It then names a parameter-level view, "Safety-Write Fragmentation": safety gradients from text and image fall on different experts and point in different directions (Section 2.2). The proposed method, SOT, freezes the model and router, adds a low-rank residual branch to every MoE block, and trains it with a text safety-margin loss, an image-to-text margin loss, and a KL loss that matches the image input's first-token distribution to the text input's (Section 3; Eqs. 2-8). On DeepSeek-VL2-Small, Qwen3-VL-30B-A3B and Aria-Chat, the authors report:

  • on the base model, the experts that a safety update would change most overlap less between the text and image versions of a request (0.47 on DeepSeek, 0.43 on Qwen3) than between an image and the same image with a different layout (0.78, 0.77) (Figure 4a); where they do overlap, the two updates point in different directions (cosine 0.204 and 0.249; Figure 5);
  • after one text-only update of norm 0.1, the share of the text safety gain that carries over to images is 0.153 for routed experts and 0.368 for shared parameters on DeepSeek (Table 1);
  • average multimodal safety rises from 54.1 to 79.4 on DeepSeek, 82.7 to 88.9 on Qwen3, and 59.0 to 63.2 on Aria (Table 2);
  • MMBench Circular accuracy stays near Base (80.2 to 80.0 on DeepSeek, 86.0 to 86.0 on Qwen3), while over-refusal rises, most on DeepSeek (Table 3; Table 14).
Critical strengths
  • The problem is real and measured on released models. The text-safe, image-unsafe cases come from public MoE vision-language models on public benchmarks, not from a constructed setting. For example, DeepSeek-VL2-Small has 42 of 100 such cases on FigStep and 116 of 345 on SafeBench (Appendix D.1, Table 9).
  • Several parts of the evaluation are careful. Results are averaged over three seeds (Table 2; Tables 11-13), both Repair (fixed blind spots) and Regression (newly broken safe cases) are reported, MM-SafetyBench is held out entirely as an out-of-distribution test (Appendix D.2, Table 10), and the judge agrees with human labels on 95.3% of 250 samples (Appendix E, Table 15).
  • The gain on DeepSeek is clear. SOT raises average multimodal safety from 54.1 to 79.4, against 71.3 for the best baseline (MM-PT), with the highest Repair on all four benchmarks and the lowest Regression (3.8) (Table 2).
Critical weaknesses

1. The problem is known, and the method is simple.

  • Vision-language models refusing a request as text but answering it as an image is already known; the paper itself cites prior work on it (Section 1).
  • SOT is a small LoRA-style adapter added to each MoE layer, trained so the image input's first answer token matches the text input's (Eqs. 2, 7).

2. The diagnosis does not show a cause beyond "text and image use different experts."

  • Route transplantation only measures safety. Forcing the image input onto the text input's experts (Figure 3) is judged only by the safety margin. There is no check of general performance, so a forced route may simply break the model's computation. The test does not rule routing out.
  • The gradient overlap restates routing. Experts that an input is not routed to get no gradient. So lower overlap between text and image (0.47 vs. 0.78; Figure 4a) is expected from different routing. It is not evidence that training causes the gap.
  • Real safety training is ignored. The analysis assumes safety is learned from text and then transferred to images, using one hypothetical text-only gradient step (Table 1). Modern VLMs are also safety-trained on images, and the paper does not discuss how the base models were aligned.
  • The claim that MoE makes the gap worse rests on one model pair. The only dense-vs-MoE comparison is Qwen3-VL-32B vs. Qwen3-VL-30B-A3B (1.9% vs. 10.7% text-safe, image-unsafe; Figure 1). These two models may differ in more than MoE, so one pair cannot show that MoE is the cause. More dense-vs-MoE pairs would be needed.

3. The natural baseline is missing.

  • All baselines train the same added adapter as SOT, on only the first 4 tokens of the safe answer (Appendix B.3). None is standard LoRA fine-tuning of the model's own layers on image inputs with full safe answers.
  • Even against these baselines, SOT clearly wins only on DeepSeek. On Qwen3, MM-PT is higher (89.8 vs. 88.9); on Aria, SOT and MM-SFT are within seed noise (63.2 vs. 62.6) (Table 2).

4. The utility claim is overstated.

  • MMBench barely changes for any method (Table 14), so it cannot show that SOT preserves utility better than the baselines.
  • On DeepSeek, SOT refuses harmless requests more than any other method (MOSSBench 1.3 to 18.3; XSTest-safe 8.0 to 17.6; Table 14).

5. The text-vs-image pairs are unclear for most benchmarks. In HarmBench and BeaverTails-V, the harmful request is already in the text and the image only adds context. The paper does not say how the text-only versions of these benchmarks and SafeBench were built (Appendix B.2), so it is unclear that each pair is the same request in two forms.

Overall assessment The paper studies a real problem on released models and public benchmarks, and parts of the evaluation are careful: three seeds, Repair and Regression, a held-out out-of-distribution set, and a validated judge. The gain on DeepSeek is clear.

My main concern is that these do not support the paper's central claims. The text-safe, image-unsafe gap is already known. The diagnosis does not separate Safety-Write Fragmentation from the effect of different routing, and its evidence for shared parameters is a single reverted gradient step. The method is a small adapter with standard losses, and its advantage is clear on one of three models against baselines that share its adapter and short supervision window, at the cost of the highest over-refusal on that model. What would change my assessment is a diagnosis that controls for routing (Question 1) and a comparison with standard LoRA fine-tuning on the model's own layers with full safe responses (Question 3).

Questions for authors
  1. What evidence shows a cause beyond text and image using different experts? For example, does the forced routing in Figure 3 keep general performance intact, and do the overlap and cosine gaps remain for text-image pairs whose routes are similar?
  2. How were the three base models safety-aligned, and did that include image safety data? If image safety training is already standard, what does SOT offer over simply adding more image safety data to that training?
  3. How does SOT compare with standard LoRA fine-tuning of the model's own layers on image inputs with full safe answers, using the same data?
  4. How were the text-only versions of HarmBench, SafeBench and BeaverTails-V requests built?
  5. On DeepSeek, how much of SOT's safety gain comes from refusing more often in general, given its higher over-refusal?
  6. SOT aligns only the first answer token. Does it hold up against prefilling attacks that write the first tokens for the model?
Reviewer_oSaEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The aurhtor reveals Safety-Write Fragmentation that limits cross modal accessibility of learned safety policies. The analysis in this work indicates that writing updates to shared parameter substrates restores cross modal transfer, while policy divergence concentrates primarily at generation onset. Therefore, the author introduces SOT (Safety Onset Transport), which equips models with lightweight shared parameters and steers visual safety policies starting from the generation onset. Effective results are made across multiple different MoE architectures.

Critical strengths

1、A meaningful investigation and problem definition. A good research view from sparse routing and parameter-update geometry, which differs from the conventional methods about input and output behavior. 2、The design of the framework is clear and well motivated. Informative perspectives are made from the routing divergence, structural and directional safety-gradient methods. 3、Reasonably comprehensive experiments. Covering MoE architectures, multiple benchmarks, several baselines, and the experiments shows that the substantial improvements are made, e.g., on deepseek model.

Critical weaknesses
  1. The causal relationship between sparse routing and cross-modal safety failures is not sufficiently established. 2、The SOT consists of shared low-rank residual parameters, preference-margin optimization, and KL-based knowledge distillation—are individually well-established techniques. The combination is reasonably motivated. It would be better that more anslysis on fundamentally necessity components in the method. 3、The generalization of across different architectures. The improvements across different architectures varies significantly. More anslysis and the phenomenon studies should be added.
Questions for authors

1、Why does the improvements show large differences between the DeepSeek series models and Qwen, Aria? 2、How does the proposed method compare with the conventional shared-adapter distillation with KL regularization ? 3、Can the authors provide more results on different examples, layers, and models?

Reviewer_ZXrGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why multimodal Mixture-of-Experts models may refuse harmful text prompts but comply with semantically equivalent visual prompts. The paper argues that routing divergence alone does not explain this failure and identifies Safety-Write Fragmentation, where text and visual safety supervision updates different parameters in conflicting directions. The authors propose Safety Onset Transport (SOT), which adds a lightweight shared safety pathway and transfers the text-aligned policy to visual inputs at the beginning of generation. Across three multimodal MoE architectures, SOT improves multimodal safety, including gains from 54.1% to 79.4% on DeepSeek and from 82.7% to 88.9% on Qwen3, while largely preserving benchmark capability. However, the improvements introduce measurable over-refusal and benign-helpfulness trade-offs, particularly on DeepSeek.

Critical strengths

The diagnosis is well structured. The paper separates routing, gradient overlap, gradient direction, update transfer, and actual generation safety. It does not assume that different routes automatically prove different safety learning (Sections 2 and B.5). The analysis goes beyond correlation. The matched-versus-shuffled routing test and norm-matched virtual updates are more convincing than activation plots alone. Table 1 also reports both visual gains and retention ratios, which helps clarify the limits of the shared-substrate result. The comparisons are fairly broad and transparent. The paper includes SFT, DPO, text-only, and multimodal baselines, reports seed-level results, and measures both safety and benign over-refusal. The large DeepSeek over-refusal cost is clearly reported rather than hidden. SOT provides a clear design idea. It makes the safety residual available across expert routes and focuses cross-modal supervision at generation onset. On DeepSeek, MM-Safe rises from 56.6% with the margin alone to 75.2% with onset distillation and 79.4% when both are used (Table 4).

Critical weaknesses

W1, The causal claim is too broad. The dense-versus-MoE comparison does not isolate sparsity, and the route-transplant experiment only tests one intervention. The paper should narrow its claims and test routing across the full generation sequence. The gradient evidence needs stronger checks. W2, The analysis depends on a top 10% threshold, and the reported cosine values indicate weak positive alignment rather than clearly conflicting gradients. Threshold sweeps, raw distributions, and matched-gain controls would strengthen the argument. W3, The ablations do not separate architecture from objective. It remains unclear whether the gains come from shared parameters, onset distillation, or their combination. A factorial ablation across adapter locations and loss terms is needed. W4, First-token safety may not be durable. The onset effect is compelling, but the paper does not test unsafe prefixes, later-token recovery, multi-turn interactions, or adaptive attacks. Full-response robustness should be evaluated. W5, The practical claims should be narrower. Safety remains limited on some benchmarks, over-refusal is substantial on DeepSeek, and computational costs are not fully reported. The paper should include safety versus helpfulness trade-offs, runtime measurements, and broader attack settings.

Questions for authors

1.What precisely is transplanted in Figure 3, and at which prompt and generation positions? Can the authors show that matched and shuffled interventions are comparably valid, with per-example paired tests and intervention-fidelity checks? 2. How sensitive are the gradient results to the 10% threshold? Please include raw distributions, controls, and the missing Aria analysis. 3. Does SOT remain safe when the first token is bypassed? Please test unsafe prefixes, recovery later in generation, and adaptive attacks. 4. What is the practical operating point? Please report human validation, safety-helpfulness trade-offs, judgment coverage, and computational overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.