acceptodds
Under review as a conference paper at ICLR 2027

Routing-Guided Single-Expert Interventions Measured Against a Model’s Own Execution Variability

Abstract

How large is the effect of changing one expert at one token in a sparse mixture-of-experts model, and how precisely can it be measured? We answer both questions by comparing the model with itself: five mathematically equivalent BF16 executions of the same tokens establish how much each measured quantity varies with execution, and an end-to-end FP64 reference measures the intervention at higher arithmetic precision. We apply this approach to Qwen3-30B-A3B solving linear systems. A linear detector on executed gate weights retrieves 94% of annotated computation events within a 5% review budget on each held-out trajectory. Its ranking is highly stable as expert membership changes across executions. A fixed rule on the same weights nominates the expert to remove; in same-state comparisons, the average drop in a frozen numerical probe's position score grows with the removed expert's natural weight. Attenuating the most frequently nominated expert improves local digit evidence while reducing that position's share of the probe's readout. These opposing contributions retain their average directions across executions. At the model's answer, median high-precision effect magnitudes are 0.02–0.04 nat against a median natural execution range of 0.50 nat. Single BF16 pairs report several-fold larger median magnitudes and carry little directional information. Raising only the precision of the final readout preserves most of that spread, which therefore forms before the readout. Small gaps between competing router or output scores identify the choices most likely to change. Routing guides where to look and which expert to test, interventions reveal what changes, and execution comparisons make numerical precision part of experimental design.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.