- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Direct Spectral Actuation (DSA), a method for improving mixture-of-experts (MoE) models after training by introducing expert-output differences while preserving the model’s original predictions. DSA formulates differentiation as a constrained optimization problem and leverages spectral analysis to select perturbations that maximize the useful gradient signal available to the router.
Critical strengths
Improving Mixture-of-experts' routing gradient is highly relevant and important to tackle load balancing and collapse problems in MoE models.
Critical weaknesses
Overall, the paper suffers from incoherent writing and is extremely poor written. Some significant writing issues:
Abstract. The abstract does not clearly state what the paper is trying to do or how it is improving mixture-of-experts models. There is no clear problem statement or justifiable drawbacks of current methods, making it difficult to follow what the paper is trying to achieve. Moreover, the abstract includes terms such as "anchor prediction", "expert-output residuals" without any explanation.
Introduction. The introduction does not present any relevant works in mixture-of-experts or how the proposed method is positioned against existing literature. This makes it difficult to access what is actually the contribution of the paper.
Method.
- In 3.1, the authors mentioned "the fused loss" but it was not defined anywhere prior to the section. There are also very informal statements such as "the router is also locally blind" or "Readable opportunity".
- The theoretical results also do not have clear theoretical implications or any high level explanation. Moreover, the authors introduce notations that are not properly defined. For example, is not defined in Theorem 2.
Experiments.
- The experiment section is also unorganized without a clear message.
- The results are also limited to one model and two tasks.
Questions for authors
N/A.
2 comments
I appreciate the authors' rebuttal and their efforts to address the raised concerns. Nevertheless, I believe the original submission still requires substantial revisions to improve the clarity and coherence of its technical presentation. A more thorough revision would strengthen the overall quality of the paper, and I recommend another round of peer review before it can be considered further.
Thank you for pointing out where the paper was difficult to follow. Your comments prompted us to rethink how we explain the motivation, method and experiments. We have put considerable thought and care into this work, and your feedback has helped us see how to present the ideas and supporting evidence more clearly.
The question we have been working on is how to change a trained model's response to rerouting while preserving its current predictions. We believe this perspective is worth studying and sharing, and want to present the construction and evidence in a form that others can examine, question and build on. We have revised the manuscript with this aim in mind, and describe the main changes below.
1 Motivation and problem statement
The abstract now describes the problem through the router's task-loss learning signal. The introduction makes it concrete with two identical experts, each with output f and weight one half. Reweighting these outputs leaves the mixture unchanged, so the task loss provides no routing gradient through this mixture. We add opposite residuals δ and −δ, which cancel at equal weights. Increasing the first weight by ε and decreasing the second by the same amount changes the mixture to f+2εδ.
The example shows how the current prediction can stay fixed while the response to rerouting changes. The write introduces these protected differences, and the read subsequently uses them through the original task loss. Exact prediction protection applies before the read; the routing update can then change the prediction.
2 Relationship to existing methods
Section 2 now places DSA alongside expert-specialization methods, function-preserving growth and upcycling, and protected optimization. We explain the relationship through what the intervention preserves. For example, AlphaEdit projects parameter edits into the null space of protected knowledge keys. DSA constrains expert-output residuals to cancel under the frozen mixture weights while changing the response available to later rerouting.
3 Method and theory
Section 3 introduces expert outputs, routing weights, the mixture state and the task loss before the decomposition. We explain its implication directly: preserving the current mixture still leaves freedom to change its response to reweighting. Section 4 constructs residuals with this property. The read holds the residuals fixed and updates a small routing correction using the original task loss.
Section 5 distinguishes the initial-gradient criterion used to propose a write from NLL after the executed read, which is used to assess it. Revised DSA includes first-order selection (FOS) as a candidate and reference, and retains it unless another candidate lowers post-read NLL on every designated development input–label pair. Section 6 then explains how an independent audit informs deployment. This separates preservation of the starting prediction, improvement on the development cover and assessment before deployment.
4 Experimental questions and evidence
We now state the purpose of each comparison in Section 7. Figure 4a holds the routing displacement fixed with and without a protected write. On 129 HellaSwag calibration examples, writing adds 0.013612 nats of mean NLL reduction, isolating its contribution to the same read.
Table 2 reports held-out SST-2 and RTE endpoints before and after calibration, and states that DSA retains FOS on all three SST-2 fits and the shared RTE fit. The shared write-and-read endpoints improve pooled NLL over frozen OLMoE in both scoring conditions. The controlled construction in Section 5.4 separately shows when another write can improve on FOS under the same budget and first accepted step.
Figure 4b tests source transfer and reports mean NLL increases in both transfer directions, despite small within-source gains. Appendix B.6 reports the retrospective OpenML deployment replay. We label calibration and development comparisons separately from held-out task evaluation, and specify that the native interventions use the final mixture layer of one frozen OLMoE checkpoint.
We would be very grateful if you could take another look at the revised manuscript. We hope it now gives a clearer account of the idea and the evidence behind it, and would value your reassessment. The opportunity to share and discuss this work with the community means a great deal to us.