- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how single-step expert-routing interventions in sparse Mixture-of-Experts based autoregressive large language models affect subsequent generated tokens and measured safety compliance. To disentangle decoding pathways, the paper proposes a four-cell decoder audit that separates the effect of the immediate next token generated and the retained KV Cache for the experiments. Experiments show that state-only routing transfers do change the subsequent generation, but they produce little change in the complete-answer compliance. The paper also identifies isolated cases where evaluating response prefixes versus complete answers can lead to different conclusions about intervention effects.
Critical strengths
The paper investigates a relevant question about whether changes in expert routing affect safety behavior through a systematic four-cell audit. The experiments include several useful controls, such as different-request donors, a dense-model residual comparator, and final-layer controls.
Critical weaknesses
The novelty of the proposed approach is limited, as separating the generated token from the KV Cache at a single decoding step into a four-cell audit is a relatively straightforward decomposition, and it is unclear about what new insights it provides specifically about MoE safety. The empirical evaluations is very limited, e.g. it contains only 17 HarmBench misinformation requests and 32 TruthfulQA questions in the held-out cohort. Also, as Qwen3-MoE and OLMoE only comply with 0/21 and 2/33 requests natively, the setup cannot meaningfully test safety repair. TruthfulQA evaluation relies on automated LLM graders without human evaluation conducted to validate the results. On writing, the paper is difficult to follow, with excessive procedural details, repetitive qualifications, and unclear organization that obscure the main contributions, probably due to use of LLM in writing.
Questions for authors
- Can the authors provide a more comprehensive evaluation, for example, with a larger set of base prompts, more cases where models initially comply with harmful requests, and human validation for the TruthfulQA grading? Currently, having only 2/33 OLMoE and 0/21 Qwen3-MoE native positive comparisons provides little insight into safety repair.
- Can the authors clarify what new insights the proposed four-cell audit provides specifically about MoE safety, beyond the existing activation-patching approaches?