acceptodds
Under review as a conference paper at ICLR 2027

Direct Spectral Actuation: Prediction-Preserving Expert Differentiation in Mixture-of-Experts

Abstract

Trained mixture-of-experts (MoE) models may lack task-relevant expert differences, leaving routers with little useful gradient signal. Direct Spectral Actuation (DSA) adds protected expert-output residuals after training without changing the anchor prediction. Under a fixed function-space budget, DSA selects the write that maximizes added router-gradient power under the original task loss. This yields a leading singular mode at a symmetric anchor and a metric-aware trust-region solution at a general trained anchor. DSA freezes the residual adapter and updates only the reader; native experts and the pretrained router remain fixed. Controlled tasks and CIFAR-10-C test protection and the subsequent read, while an OpenML-CC18 replay tests selective intervention. On one frozen native top-8 OLMoE checkpoint, DSA reduces held-out NLL relative to the anchor by 12.4–25.8% on SST-2 and 40.7% on average on RTE. It improves on matched reader-only adaptation in all six evaluations and yields lower task-mean NLL than a matched final-layer RoMA-objective baseline on both tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Expert Specialization, Prediction-Preserving Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EHzcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Direct Spectral Actuation (DSA), a method for improving mixture-of-experts (MoE) models after training by introducing expert-output differences while preserving the model’s original predictions. DSA formulates differentiation as a constrained optimization problem and leverages spectral analysis to select perturbations that maximize the useful gradient signal available to the router.

Critical strengths

Improving Mixture-of-experts' routing gradient is highly relevant and important to tackle load balancing and collapse problems in MoE models.

Critical weaknesses

Overall, the paper suffers from incoherent writing and is extremely poor written. Some significant writing issues:

Abstract. The abstract does not clearly state what the paper is trying to do or how it is improving mixture-of-experts models. There is no clear problem statement or justifiable drawbacks of current methods, making it difficult to follow what the paper is trying to achieve. Moreover, the abstract includes terms such as "anchor prediction", "expert-output residuals" without any explanation.

Introduction. The introduction does not present any relevant works in mixture-of-experts or how the proposed method is positioned against existing literature. This makes it difficult to access what is actually the contribution of the paper.

Method.

  • In 3.1, the authors mentioned "the fused loss" but it was not defined anywhere prior to the section. There are also very informal statements such as "the router is also locally blind" or "Readable opportunity".
  • The theoretical results also do not have clear theoretical implications or any high level explanation. Moreover, the authors introduce notations that are not properly defined. For example, Hom(Tg,RD)Hom(T_g, R^D) is not defined in Theorem 2.

Experiments.

  • The experiment section is also unorganized without a clear message.
  • The results are also limited to one model and two tasks.
Questions for authors

N/A.

2 comments

Thank you for pointing out where the paper was difficult to follow. Your comments prompted us to rethink how we explain the motivation, method and experiments. We have put considerable thought and care into this work, and your feedback has helped us see how to present the ideas and supporting evidence more clearly.

The question we have been working on is how to change a trained model's response to rerouting while preserving its current predictions. We believe this perspective is worth studying and sharing, and want to present the construction and evidence in a form that others can examine, question and build on. We have revised the manuscript with this aim in mind, and describe the main changes below.

1 Motivation and problem statement

The abstract now describes the problem through the router's task-loss learning signal. The introduction makes it concrete with two identical experts, each with output ff and weight one half. Reweighting these outputs leaves the mixture unchanged, so the task loss provides no routing gradient through this mixture. We add opposite residuals δ\delta and −δ-\delta, which cancel at equal weights. Increasing the first weight by ε\varepsilon and decreasing the second by the same amount changes the mixture to f+2εδf + 2\varepsilon\delta.

The example shows how the current prediction can stay fixed while the response to rerouting changes. The write introduces these protected differences, and the read subsequently uses them through the original task loss. Exact prediction protection applies before the read; the routing update can then change the prediction.

2 Relationship to existing methods

Section 2 now places DSA alongside expert-specialization methods, function-preserving growth and upcycling, and protected optimization. We explain the relationship through what the intervention preserves. For example, AlphaEdit projects parameter edits into the null space of protected knowledge keys. DSA constrains expert-output residuals to cancel under the frozen mixture weights while changing the response available to later rerouting.

3 Method and theory

Section 3 introduces expert outputs, routing weights, the mixture state and the task loss before the decomposition. We explain its implication directly: preserving the current mixture still leaves freedom to change its response to reweighting. Section 4 constructs residuals with this property. The read holds the residuals fixed and updates a small routing correction using the original task loss.

Section 5 distinguishes the initial-gradient criterion used to propose a write from NLL after the executed read, which is used to assess it. Revised DSA includes first-order selection (FOS) as a candidate and reference, and retains it unless another candidate lowers post-read NLL on every designated development input–label pair. Section 6 then explains how an independent audit informs deployment. This separates preservation of the starting prediction, improvement on the development cover and assessment before deployment.

4 Experimental questions and evidence

We now state the purpose of each comparison in Section 7. Figure 4a holds the routing displacement fixed with and without a protected write. On 129 HellaSwag calibration examples, writing adds 0.013612 nats of mean NLL reduction, isolating its contribution to the same read.

Table 2 reports held-out SST-2 and RTE endpoints before and after calibration, and states that DSA retains FOS on all three SST-2 fits and the shared RTE fit. The shared write-and-read endpoints improve pooled NLL over frozen OLMoE in both scoring conditions. The controlled construction in Section 5.4 separately shows when another write can improve on FOS under the same budget and first accepted step.

Figure 4b tests source transfer and reports mean NLL increases in both transfer directions, despite small within-source gains. Appendix B.6 reports the retrospective OpenML deployment replay. We label calibration and development comparisons separately from held-out task evaluation, and specify that the native interventions use the final mixture layer of one frozen OLMoE checkpoint.

We would be very grateful if you could take another look at the revised manuscript. We hope it now gives a clearer account of the idea and the evidence behind it, and would value your reassessment. The opportunity to share and discuss this work with the community means a great deal to us.

I appreciate the authors' rebuttal and their efforts to address the raised concerns. Nevertheless, I believe the original submission still requires substantial revisions to improve the clarity and coherence of its technical presentation. A more thorough revision would strengthen the overall quality of the paper, and I recommend another round of peer review before it can be considered further.

Reviewer_TLggOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

A moe model has a team of specialists and a dispatcher(router) that decides how much to trust each one. If the specialists all give the same answer, the dispatcher has nothing to learn, because reshuffling trust changes nothing. DSA edits the specialists such that their weighted average answer, which is what the model outputs, stays exactly the same, but they now disagree in a direction that is useful for the task. For example, one leans positive sentiment and another negative. The main thing is results dont follow the presnted methodology.

Critical strengths
  • A principled, exactly prediction-preserving post-hoc intervention: Thm. 2/3 give a clean characterisation of the expert-output freedom hidden by the routed mixture, with exact protection verified numerically (drift around 10−1510^{-15}) and an explicit abstain/restore path. This matters because safe, verifiable post-training edits to deployed MoEs are a real need.
  • Exemplary transparency and reproducibility: SHA-256-bound artifacts, deterministic table builders, explicit disclosure of retrospective analyses, and reported negative arms ("Always DSA" -19.17%; the PERFT tie). This matters because it makes the paper's limitations auditable.
Critical weaknesses
  • wrong author list :On the Representation Collapse of Sparse Mixture of Experts ( Furu Wei, looks like hallucination), Towards understanding mixture of experts in deep learning (Yi Deng)
  • The experiments do not support the claimed mechanism: Random protected writes with about 1% of DSA's added-gradient power (0.011 vs 1.013, Table 8) recover about 87% of DSA's mean NLL gain and beat it on SST-2. A reader-only baseline without any write (CE-only, Table 13) matches or exceeds DSA's accuracy in 3/6 partitions. PERFT+read is statistically tied (Δ=0.0033\Delta=0.0033, CI [−0.0138,0.0204][-0.0138,0.0204]). The "RoMA-objective" baseline in the abstract is numerically identical to CE-only. The spectral selection, which is the core contribution, is therefore not shown to matter for task outcomes.
  • The headline gains are largely calibration, at a very small scale: On RTE every method has NLL>ln⁡2\mathrm{NLL}>\ln2 and accuracy is 45--56%, so the "40.7% NLL reduction" is partial recalibration of a chance-level classifier, and no temperature or bias baseline is given. The native evidence is one OLMoE checkpoint, only its final layer, 768 coefficients, two binary tasks, and a reused RTE evaluation cache. The other "MoEs" are dense models whose logits are copied into collapsed mixtures. Significance for real expert specialisation is therefore unestablished.
Questions for authors
  • Expert Choice routing Outdated NeurIPS 2022, cited as arXiv.
  • Methods used but not cited:Armijo line search, trust-region subproblem, generalised Gauss–Newton metric, temperature scaling
  • missing work : AlphaEdit (ICLR 2025) projects edits into the null space of preserved knowledge, which is conceptually the same, MoE specialisation at scale: GShard, ST-MoE, Mixtral, and DeepSeekMoE (fine-grained and shared experts that explicitly target redundancy and collapse). Expert-specialised fine-tuning, ESFT (Wang et al., EMNLP 2024), and MoE-PEFT methods such as MixLoRA are also relevant.
  • Does a temperature scale, or a two-parameter logit bias, fit on the same 512 calibration rows, match DSA's NLL on SST-2, RTE, and CIFAR-10-C?
  • Report DSA against the best of the eight random-safe seeds, not only their mean, at the same budget.
  • Give DSA the same 48 AdamW steps used for CE-only.
  • State which OpenML number is prospective. The appendix says the +1.12% rule was derived after the counterfactuals; the pre-evaluation gate acts on 7 of 45 tasks.
  • Thm. 6, is the definition of the operator norm. Eq. 14 is the triangle inequality, not a substantive theorem
1 comment

Thank you for pointing out the gap between initial gradient power and the loss after reading. Your comparison with random writes prompted us to change how DSA selects a direction.

1. Initial gradient and finite read

The protected residuals cancel at the original routing weights, so the model starts from exactly the same prediction. They change the initial routing gradient from rr to r+Bar+Ba, giving the router a way to use the new differences between experts. Figure 3 shows that two writes with the same initial gradient can reach different losses after the same accepted step.

DSA uses first-order selection (FOS) as its reference and the maximum-gradient write as a proposal. Under the same write budget and Armijo rule, it replaces FOS only when the executed read lowers NLL on every covered development pair (§5.3). DSA retained FOS on the original OLMoE fits. In the two-channel example, reallocating the same write budget gives a minimum NLL gain of 0.00170 nats over FOS across the covered pairs at the same accepted step (§5.4).

2. Protected writing and random controls

Figure 4a applies the same routing displacement to frozen OLMoE with and without a write. Starting from identical predictions, protected writing adds 0.013612 nats of mean NLL reduction on 129 HellaSwag calibration examples. On held-out SST-2, DSA also lowers NLL relative to reader-only at the same step size in all three partitions. The best of eight random protected writes, selected by fit NLL at the same budget, scores 0.3560 versus 0.3596 for DSA (Table 5).

In the submitted RTE comparison, seed 4606 had the lowest fit NLL among the eight random writes.

RTE partitionSubmitted gradient criterion, NLLBest random write by fit NLL
10.86080.8935
20.88510.9604
30.77600.8324

3. Calibration and accuracy

We fit each endpoint's logit scale and intercept on the same 512 examples (Table 2). RTE shares one fit across its three partitions.

TaskCalibrated frozen OLMoE, NLL (correct)Calibrated DSA, NLL (correct)
SST-20.3159 (2659/3072)0.2779 (2731/3072)
RTE0.6944 (523/1024)0.6856 (559/1024)

On RTE, temperature-only and bias-only calibration of the frozen model give NLLs of 0.6968 and 0.7908. In the controlled CIFAR-10-C mixture, a scale and top-class offset fitted on 512 images give NLLs of 1.1192 for the frozen model and 1.0954 averaged over five protected writes on the same 60 corruption and severity combinations. The native results cover the final mixture layer of one OLMoE checkpoint.

4. Matched 48-step training

On the original 512-example fits, both arms start with zero routing correction, take 48 AdamW steps and use the same final Armijo rule. DSA holds the write fixed during reader training. The table pools the calibrated results.

TaskCE-only, NLL (correct)DSA, NLL (correct)
SST-20.2741 (2737/3072)0.2776 (2731/3072)
RTE0.6910 (536/1024)0.6856 (558/1024)

On RTE, the NLL reduction is 0.00544 nats (paired 95% CI [0.0008, 0.0102]). The two arms are close at step 48, and most of the gain appears after the final read. On SST-2, DSA lowers raw NLL and CE-only has lower calibrated NLL. We removed the RoMA comparison from the abstract because its scores match CE-only to the reported precision. The submitted RTE comparison with PERFT + read has a paired interval spanning zero.

5. Transfer and deployment

The small NLL gains on the HellaSwag groups used for selection disappear when we switch sources. This motivates a separate check before deployment.

  • OpenML gate set before evaluation. It selected 7 of 45 tasks. Those seven had a 4.21% mean relative NLL reduction.
  • OpenML paired audit. We designed this rule after seeing the counterfactual results. Across all 45 tasks, it gives a 1.12% mean relative NLL reduction (95% CI [0.59%, 1.77%]). NLL rises on 0/45 tasks with the audit and 7/45 under unconditional deployment (Table 10).

6. Theory and related work

  • We agree that the norm identity and triangle inequality are standard. The revised Theorem 6 gives the minimum write cost for a prescribed gain after the read in the stated group-shift family with a common first accepted step.
  • Section 2 now discusses AlphaEdit, GShard, ST-MoE, Mixtral, DeepSeekMoE, ESFT and MixLoRA. AlphaEdit protects knowledge keys through a null-space projection. Our projection makes residuals cancel at the original mixture weights so they can affect the prediction when routing changes.
  • The NeurIPS proceedings list Furu Wei in Chi et al. and Yihe Deng in Chen et al.. We cite the proceedings version of Expert Choice and add references for Armijo search, trust regions, generalized Gauss–Newton and calibration.
Reviewer_kk8FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DSA, a method that adds useful differences between MoE expert outputs while keeping the initial model prediction unchanged. Under a fixed budget for these changes, DSA chooses a direction that strengthens the router gradient from the task loss. It then freezes the added residuals and updates only the routing adjustment parameters. Theoretical analysis, controlled experiments, and tests on OLMoE support prediction preservation before the routing update and task improvements afterward.

Critical strengths

The method has a clear motivation and theoretical basis. Creating expert differences while keeping the initial prediction unchanged offers a useful way to improve router learning. The paper gives clear conditions for prediction preservation and direction selection, and explains the scope of its guarantees.

The control experiments support the method's basic value. The experiments include reader-only updates, random protected directions, and no-read controls, which help separate the roles of different parts of the method. On OLMoE, DSA achieves lower test NLL than reader-only updates in the SST-2 and RTE evaluations.

Critical weaknesses

The final benefit of the direction selection method needs further testing. Compared with the simpler first-order-safe method, the paper mainly reports gains in gradient strength and response direction alignment. It does not directly compare their final NLL and accuracy under the same update budget.

Questions for authors

With the same residual budget, number of reader updates, and line search rules, does DSA consistently achieve lower test NLL and higher accuracy than the first-order-safe method? Please provide results for each run and estimates of uncertainty.

2 comments

Thank you for your encouraging comments on the prediction-preserving construction and theoretical analysis, and for asking us to compare final NLL and accuracy directly. In the revised Direct Spectral Actuation (DSA) procedure, the first-order selection (FOS) write is both a candidate and the reference for comparison. We retain it whenever the search finds no alternative that meets the development criterion for post-read NLL improvement.

1 Final NLL and accuracy under matched conditions

On the original OLMoE splits, DSA retained FOS on all three SST-2 fits and the shared RTE fit. The two procedures therefore reached the same final NLL and accuracy in all six evaluation partitions.

Both methods used the same prediction-protection constraint, Frobenius write radius of 0.25, and one task-loss routing update per evaluated endpoint. Their paired Armijo searches used the same step-size grid, read radius and acceptance rule. Each SST-2 run used a separate 512-example fit; the three RTE evaluation partitions shared one 512-example fit. The table reports uncalibrated evaluation scores and the first accepted step.

Evaluation partitionFirst accepted stepNLL (nats), DSA and FOSCorrect / total, DSA and FOS
SST-2 (1)1/80.365567867 / 1,024
SST-2 (2)1/40.342109903 / 1,024
SST-2 (3)1/80.371240860 / 1,024
RTE (1)1/40.854891176 / 342
RTE (2)1/40.879323168 / 341
RTE (3)1/40.774366188 / 341

Because DSA retained the FOS endpoints, the paired 95% bootstrap intervals are [0, 0] nats for NLL and [0, 0] percentage points for accuracy at the reported precision, using 10,000 paired resamples per partition. These intervals describe evaluation-sample uncertainty conditional on the fitted endpoints; they do not measure variability across new fits.

2 Why the revised procedure can retain FOS

We distinguish proposing a write from deciding whether to keep it. The initial-gradient criterion supplies candidate directions, but a larger initial gradient alone does not determine NLL after a finite routing update. Revised DSA therefore compares the actual read outcomes. A candidate replaces FOS only if it lowers NLL on every designated development input–label pair in the development cover; otherwise, we keep FOS (Section 5.3). Selection uses development data, with held-out performance evaluated separately.

3 What the comparisons establish

FOS and DSA share the prediction-preserving write construction and the task-loss routing read. Their comparison assesses the additional role of write selection. Table 2 shows that the shared endpoints improve held-out NLL over frozen OLMoE before and after logit calibration. These are outcomes of the shared write-and-read procedure; Figure 4a separately isolates the contribution of writing with the routing displacement held fixed.

The controlled two-channel construction shows how further write selection can add a benefit. Reallocating the same write budget achieves a minimum NLL gain of approximately 0.00170 nats over FOS across the covered pairs, with the same first accepted read step (Section 5.4; Appendix A.16). This establishes a setting in which the finite-step criterion can improve on the linear reference. On the original SST-2 and RTE fits, the search instead retained FOS, as reported above.

Your question helped us make this distinction clearer. We hope the direct endpoint comparison and the controlled construction explain both the observed task outcomes and the additional role of finite-step write selection.

Thank you for adding the direct comparison with FOS under the same write budget and routing update protocol. This addresses my question about final performance. The revised DSA retained FOS on all reported SST-2 and RTE fits, yielding identical held-out NLL and accuracy. The controlled example shows that finite-step write selection can improve on FOS in that setting. Its benefit on real tasks remains an open question.

I continue to find the prediction-preserving construction and its theoretical analysis valuable. My overall rating remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.