acceptodds
Desk rejected at ICLR 2027

Steering Does Not Imply Localization: Multilingual Refusal Control in Mixture-of-Experts Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Intervening on a small set of experts in a mixture-of-experts (MoE) model can alter refusal behavior, but behavioral control alone does not show that refusal during unmodified inference depends on those experts. This distinction is especially important across languages: an intervention that works across the evaluated languages need not reveal a shared refusal mechanism. We construct an evaluation set by aligning requests from 11 benchmark collections across 12 languages and study Qwen3 and Gemma-4, two open-weight MoE models with manipulable sparse routing. Using multilingual routing contrasts, we select model-specific expert sets and fix them before evaluation. On 1,977 held-out English sources, forcing these experts increases observed refusal by about seven percentage points in both models and exceeds layer-matched controls. The original routers also select these experts during unmodified inference, yet removing them changes refusal much less than forcing does, with different levels of removal sensitivity in the two models. For corresponding requests in 11 non-English languages, forcing the selected experts likewise outperforms matched controls. The intervention also increases refusal of benign requests and should therefore not be interpreted as a net safety benefit. These results identify a multilingual interface for controlling refusal, but do not establish that the selected experts form a localized mechanism on which refusal depends.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Mixture-of-Experts, Multilingual Refusal, Causal Interventions, LLM Safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ynqyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies safety controls (“refusal) and the impact of individual experts. It builds upon previous work that identifies experts important to refusal and shows that force-activating such experts changes behavior (increasing refusal, sometimes too much) while removing such experts has minimal impact. They show this is similar in the multilingual context, even if removal actually has more impact.

Critical strengths
  1. Motivation behind studying this problem (interpretability of safety, especially in multilingual context)
  2. Builds a large dataset that could be useful if released. With this dataset, the paper presents large-scale empirical results on multilingual “refusal” on safety benchmarks
Critical weaknesses
  1. Paper is impossible to parse; writing is brutally jargonized and wording is consistently poor. Examples from just the abstract: “behavioral control” “unmodified inference” “need not reveal a shared refusal mechanism” “manipulable sparse routing” “routing contrasts” “layer-matched controls” “forcing” “removal sensitivity” “refusal-control interface” “localized mechanism”.
  2. No code or data released, so first contribution of “an evaluation set” is non-existent. L150-152 claims that humans reviewed 3,300 translations three-times over and there are zero details about how such language-experts were sourced or compensated. This gives no confidence that any of this happened.
  3. No novel findings about MoEs: in sparse, redundant MoEs expert-removal rarely does much. Given MoE redundancy, expert removal and expert force-activation clearly have different impact (expert sufficiency vs. necessity). Many prior papers have revealed that collecting routing statistics capture something behaviorally meaningful (like those cited L38-39).
  4. Multilingual angle did not present anything notable
Questions for authors
  1. What is “layer-matched” controls ? Why would you apply steering at different layers across languages in the first place ?
  2. Did you end up using Jailbreak and RedTeam data ? I’d recommend ignoring if you do not know where/how they were created.
Reviewer_MqxTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multilingual refusal control in mixture-of-experts (MoE) models. The authors construct an evaluation set by aligning requests from 11 benchmark collections across 12 languages, focusing on Qwen3 and Gemma-4. They first select expert sets and fix them before evaluation. They find that forcing these experts increases observed refusal, whereas removing them has a much smaller effect. The authors further evaluate forcing in 11 non-English languages, where the selected experts likewise achieve higher refusal rates than layer-matched controls.

Critical strengths

S1 The intersection of MoE, safety, and multilingualism is a potentially important research topic.

Critical weaknesses

W1. The paper is not well organized. For example, in Section 4, it is difficult to distinguish which metrics and experimental procedures are newly introduced by the authors and which are adopted from existing work.

W2. The results appear to differ from those of SAFEx (Lai et al., 2025, "Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identification"), which shows that masking safety-critical experts makes the model more vulnerable. The authors might want to explain the reasons for this apparent discrepancy.

W3. I do not see that the paper's claimed contribution of examining steering and removal together provides significant enough new insight beyond existing research. Nor do I see how incorporating multilingual data advances this insight. Although the paper motivates the combination by arguing that cross-lingual steering does not necessarily imply a shared refusal mechanism, the experiments do not directly investigate whether refusal mechanisms are shared across languages in the main contents.

Questions for authors

Q1. In Section 4, which metrics and experimental methods are original, and which are adopted from existing research? Reusing established metrics is fine, but if a metric or method is newly proposed, the authors should provide additional justification for why it is needed, given that safety and MoE are already crowded research areas.

Q2. Why do the results appear to differ from those reported in SAFEx?

Q3. What new insights does examining steering and removal together provide compared to existing research?

Q4. What additional insights does the multilingual dataset contribute beyond existing research? For example, how do the findings differ from those of Steering MoE LLMs via Expert (De)Activation (Fayyaz et al., 2026), which also investigates cross-lingual transfer of expert steering?

Q5. Line 215: What does KK denote?

Q6. What are the definitions of "prompt-derived" and "response-derived"?

Q7. Please provide a precise definition of "dose" as used in Section 5.4 and Figure 4.

Q8. What does "localization" mean in the context of this paper, and how do the authors justify the claim made in the title?

Minor: It may not be appropriate to use behavioral judges developed by the same organizations as the evaluated models (Qwen3.7-Max and Qwen3, as well as Gemini 2.5 Flash and Gemma-4), especially since Qwen3.7-Max also serves as the tie-breaker. Judges from independent developers, such as Kimi, GLM, Claude Opus, or OpenAI's GPT models, might be preferable.

Reviewer_gm53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a 12-language refusal dataset and picks a small set of experts in two MoE models (Qwen3 and Gemma-4) based on their routing when the model refuses. Forcing the router to pick these experts raises refusal by about 7 points and beats random experts from the same layers in all 12 languages. Removing the same experts barely changes refusal, so the authors argue that experts that can steer refusal are not necessarily the ones the model relies on.

Critical strengths

(1) The experimental design is rigorous. Expert sets are frozen before evaluation, and all forcing results are compared against layer-matched controls with source-level bootstrap intervals. (2) The paper clearly separates controlling refusal from depending on it. Reporting forcing, natural use, and removal as distinct questions avoids reading a working intervention as evidence of a mechanism. (3) The aligned 12-language dataset with a human translation audit is a useful resource. It allows controlled cross-lingual comparisons that most prior refusal studies lack.

Critical weaknesses

(1) The main finding compares two operators that the paper itself says are not comparable. Forcing changes routing on every token, while removal only matters on the 32–38% of tokens where the experts were already chosen (Table 19), and the next-ranked expert simply takes their place. So a small removal effect may come from the design rather than from redundancy. Without a positive control showing that this removal test can detect a localized mechanism when one exists, the “control–dependence gap” is hard to interpret.

(2) The experiment that would actually answer the question is missing. If other experts compensate, removing more of them step by step (for example 10, 50, 100) should eventually reduce refusal. There is a dose curve for forcing but none for removal, and multilingual removal is only tested on 330 previously excluded sources. As a result, the paper ends at “we cannot tell,” which limits what readers learn.

(3) I am not sure forcing reveals a refusal interface rather than a general push toward refusing. Benign refusal rises more than harmful refusal in both models (+15.09 vs. +4.25 for Gemma-4, Table 8). The gains across categories also track how much room was left above the 87% baseline (r = .87), and even the matched control doubles benign refusals in Gemma-4 (24 to 50, Table 9).

(4) Some choices weaken the comparisons. K was set to 10 for Qwen3 and 3 for Gemma-4 during exploratory development, yet the abstract highlights a difference in removal sensitivity between the two models, which could simply come from removing three times fewer experts. The response-derived policy does not work for Gemma-4 (−0.66, Table 14), but its full main-frame results are not reported anywhere.

(5) The paper misses two closely related works on the gap between intervention and localization [1, 2]. [1] show that where a fact is localized does not predict where editing works, and [2] show that patching can act through pathways the model does not normally use, which is the concern the replay check in Table 19 seems to address. Discussing these would help position the contribution, which currently reads as applying a known idea to MoE experts.

References [1] Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. NeurIPS 2023. [2] Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.

Questions for authors

(1) What happens to refusal if you remove more experts step by step, for example the top 10, 50, and 100 by your discovery score? This would show whether the small removal effect comes from other experts taking over or from these experts simply not mattering much.

(2) Have you compared forcing and removal under similar route-change exposure? Removal only affects the 32–38% of tokens where the experts were already selected, so I would like to know whether the gap remains when forcing is limited to a similar share of tokens.

(3) Since Qwen3 removes 10 experts and Gemma-4 only 3, can the difference in removal sensitivity between the two models be separated from the choice of K?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.