- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of knowledge editing for MoE models. The authors propose a method MoTE, which extends MEMIT-style closed-form editing to MoE layers and makes it faster than the previous method MoEEdit. The method mainly includes three parts. It leverages the Woodbury identity to reduce the size of the matrix inversion, solves the update only once at the last layer and spreads it to earlier layers, and uses a Tucker decomposition of the expert weights. Experiments show that MoTE is 2–3 times faster than MoEEdit, with slightly lower editing quality.
Critical strengths
-
The paper studies an important and practical problem in efficient knowledge editing for MoE models, which are widely used in modern LLMs.
-
The method is efficient and faster than previous baselines.
-
The experiments are comprehensive including multiple settings and baselines.
Critical weaknesses
-
My first concern is about the Tucker decomposition. Table 7 studies the role of Tucker decomposition. When combined with Table 2, we can see that Tucker without whitening (63.70% Utility on Qwen3-30B) is even worse than no Tucker (82.72%). Thus, the gain seems to come from whitening rather than the tensor structure. Feel free to correct me if I miss any number or details.
-
For the layer-spreading schedule, the update is solved at the last layer and spread to earlier layers. Why this works needs better justification, since each layer has different keys and its own Tucker factors. The authors give some evidence in Table 3, but it compares the updates after projecting them to only 3 dimensions, which may hide large differences. The author can directly calculate the cosine similarity in the original space and see the gap. This is important because in Table 2, this schedule gives most of the speedup (1251.4s to 553.8s on Qwen3-30B), while Woodbury only adds a small gain.
-
MoTE seems to be quite sensitive to the ridge coefficients as shown in Table 9 compared to the baseline methods.
Questions for authors
See weaknesses.
We are grateful to the Reviewer for their feedback on our work!
All changes in the revised PDF are marked in blue. Two of them affect several answers below.
W1: Thanks for the question! We would like to clarify that we utilize Tucker structure to retain the majority of the performance that loses with our multi layer update rule formulated in Eq 5. The whitening, as described in [1], is a separate procedure that minimizes damage dealt to LLMs by the Tucker decomposition itself, it can not be used without Tucker decomposition.
W2: Thank you for this point. We used the 3D projection in Table 3 because the first three principal components of the joint set of updates hold most of their energy (89 to 96% across the settings we tried), so the projection looked like a fair summary. We agree that this does not rule out large differences outside these components, so we ran a direct comparison in the original space. For every edited layer l of Qwen3-30B-A3B and every input x we computed the change of the MoE block output, Δyl(x)=∑jgj(x)ΔWjkj(x), that is, the outputs of the edited down projections weighted by the router gates, without the residual stream. Keys and gates are taken from the unedited model, so all methods are compared on exactly the same inputs. We used one batch of 50 COUNTERFACT edits (the last subject token of each edit prompt) and all tokens of their 500 neighborhood prompts, and compared MoEEdit, MoTE and MoTE w/o Tucker & Woodbury, each with its own hyperparameters. On the edit prompts, the total change summed over the edited layers points in the same direction for MoEEdit and MoTE: the mean cosine is 0.83 (0.84 between MoEEdit and MoTE w/o Tucker & Woodbury). At the last layer MoTE and its variant without the Tucker structure agree in direction (cosine 0.999), and the change of MoTE is about 2.2 times larger. On the neighborhood prompts the change made by MoTE is about three times larger than that of MoEEdit (1.01 against 0.32 per token, summed over layers), which is consistent with its lower Specificity on this model. So the two methods arrive at a very similar total edit of the residual stream. We see this as a question about the mechanism rather than about whether the method works: MoTE stays close to MoEEdit on the editing metrics while being up to 3x faster end to end. A precise account of how the solution obtained at the last layer is shared across the edited layers is a natural direction for future work, and we will add this analysis to the appendix of the final version. We also agree with your reading of Table 2: most of the speedup comes from collecting statistics and solving once, and the Woodbury identity adds a smaller part on top. Applying a Woodbury corollary is crucial for our method, because instead with Tucker structured updates we would need to invert a matrix (re⋅rin)2, which is 98.304×98.304 in case of Qwen3-30B-A3B, which results OOM on any modern GPU. Reducing factors is not a solution to this problem, as we stated in Appendix C, line 796.
W3: This is correct. We moved the table to the main text and recomputed it for both methods on the same code (Section 5.3, paragraph "Sensitivity to \lambda" and the new λ table, in blue). Both methods need a backbone-specific λ, but MoTE is more sensitive, mainly on one side. When λ is larger than the scale of the T\times T kernel, the update shrinks to almost nothing and the metrics approach those of the unedited model. On Qwen3.6-35B-A3B Efficacy falls from 92.4 at λ=10−2 to 8.8 at λ=1 while Specificity stays at 87.2, and on Qwen3-30B-A3B Utility goes from 86.43 at λ=10−1 to 49.54 at λ=1. The failure is therefore a missing edit and not a damaged backbone. MoEEdit has a wider plateau: on Qwen3-30B-A3B it stays within one point between λ=10−1 and λ=1 (88.34 and 89.26). On Qwen3.6-35B-A3B MoEEdit stays within 0.3 points between λ=10−2 and λ=1 (80.09 to 80.35), while MoTE drops from 79.99 to 35.89. The sensitivity is added to Limitations (in blue). On GPT-OSS-20B both methods are best at λ=1, and MoTE loses more when $\lambda $decreases: 18.2 Utility points at λ=10−1 against 7.4 for MoEEdit.
We believe the revision addresses Reviewer’s concerns in full. If we have addressed all questions satisfactorily, we would kindly ask the Reviewer to reconsider their evaluation or continue the discussion.