acceptodds
Under review as a conference paper at ICLR 2027

Efficient Closed-Form Knowledge Editing for Mixture-of-Experts LLMs via Tucker-Structured Updates

Abstract

Knowledge editing (KE) offers a cheap substitute for repeatedly fine-tuning LLMs. Most KE methods are built for dense feed-forward layers, while modern LLMs increasingly use Mixture-of-Experts (MoE) architectures which cost less memory and run faster at inference. As a result, a growing family of models has no editing tools designed for them. We introduce a MEMIT-like framework for knowledge editing in MoE-based LLMs. The method builds on the tensor structure of MoE layers to state the editing objective directly at the level of individual experts and uses the Woodbury matrix identity so that the full stacked matrix of expert weights never has to be formed or inverted. The final update reduces to inverting fixed low-rank matrices and needs no extra backward passes. In experiments, our approach stays within a small margin of strong baselines on the main KE metrics while speeding up the editing procedure by up to x. The gain comes from the batched MEMIT-style formulation together with the low-dimensional inversions made possible by the Woodbury identity. Closed-form, parameter-modifying KE therefore extends beyond dense layers at low cost and can be applied to modern sparse LLM architectures at scale.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge editing, Mixture of Experts, Tucker Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PqgXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of knowledge editing for MoE models. The authors propose a method MoTE, which extends MEMIT-style closed-form editing to MoE layers and makes it faster than the previous method MoEEdit. The method mainly includes three parts. It leverages the Woodbury identity to reduce the size of the matrix inversion, solves the update only once at the last layer and spreads it to earlier layers, and uses a Tucker decomposition of the expert weights. Experiments show that MoTE is 2–3 times faster than MoEEdit, with slightly lower editing quality.

Critical strengths
  1. The paper studies an important and practical problem in efficient knowledge editing for MoE models, which are widely used in modern LLMs.

  2. The method is efficient and faster than previous baselines.

  3. The experiments are comprehensive including multiple settings and baselines.

Critical weaknesses
  1. My first concern is about the Tucker decomposition. Table 7 studies the role of Tucker decomposition. When combined with Table 2, we can see that Tucker without whitening (63.70% Utility on Qwen3-30B) is even worse than no Tucker (82.72%). Thus, the gain seems to come from whitening rather than the tensor structure. Feel free to correct me if I miss any number or details.

  2. For the layer-spreading schedule, the update is solved at the last layer and spread to earlier layers. Why this works needs better justification, since each layer has different keys and its own Tucker factors. The authors give some evidence in Table 3, but it compares the updates after projecting them to only 3 dimensions, which may hide large differences. The author can directly calculate the cosine similarity in the original space and see the gap. This is important because in Table 2, this schedule gives most of the speedup (1251.4s to 553.8s on Qwen3-30B), while Woodbury only adds a small gain.

  3. MoTE seems to be quite sensitive to the ridge coefficients λ\lambda as shown in Table 9 compared to the baseline methods.

Questions for authors

See weaknesses.

1 comment

We are grateful to the Reviewer for their feedback on our work!

All changes in the revised PDF are marked in blue. Two of them affect several answers below.

W1: Thanks for the question! We would like to clarify that we utilize Tucker structure to retain the majority of the performance that loses with our multi layer update rule formulated in Eq 5. The whitening, as described in [1], is a separate procedure that minimizes damage dealt to LLMs by the Tucker decomposition itself, it can not be used without Tucker decomposition.

W2: Thank you for this point. We used the 3D projection in Table 3 because the first three principal components of the joint set of updates hold most of their energy (89 to 96% across the settings we tried), so the projection looked like a fair summary. We agree that this does not rule out large differences outside these components, so we ran a direct comparison in the original space. For every edited layer l of Qwen3-30B-A3B and every input x we computed the change of the MoE block output, Δyl(x)=∑jgj(x)ΔWjkj(x)\Delta y_l(x) = \sum_j g_j(x) \Delta W_j k_j(x), that is, the outputs of the edited down projections weighted by the router gates, without the residual stream. Keys and gates are taken from the unedited model, so all methods are compared on exactly the same inputs. We used one batch of 50 COUNTERFACT edits (the last subject token of each edit prompt) and all tokens of their 500 neighborhood prompts, and compared MoEEdit, MoTE and MoTE w/o Tucker & Woodbury, each with its own hyperparameters. On the edit prompts, the total change summed over the edited layers points in the same direction for MoEEdit and MoTE: the mean cosine is 0.83 (0.84 between MoEEdit and MoTE w/o Tucker & Woodbury). At the last layer MoTE and its variant without the Tucker structure agree in direction (cosine 0.999), and the change of MoTE is about 2.2 times larger. On the neighborhood prompts the change made by MoTE is about three times larger than that of MoEEdit (1.01 against 0.32 per token, summed over layers), which is consistent with its lower Specificity on this model. So the two methods arrive at a very similar total edit of the residual stream. We see this as a question about the mechanism rather than about whether the method works: MoTE stays close to MoEEdit on the editing metrics while being up to 3x faster end to end. A precise account of how the solution obtained at the last layer is shared across the edited layers is a natural direction for future work, and we will add this analysis to the appendix of the final version. We also agree with your reading of Table 2: most of the speedup comes from collecting statistics and solving once, and the Woodbury identity adds a smaller part on top. Applying a Woodbury corollary is crucial for our method, because instead with Tucker structured updates we would need to invert a matrix (re⋅rin)2(r_e \cdot r_{in})^2, which is 98.304×98.30498.304 \times 98.304 in case of Qwen3-30B-A3B, which results OOM on any modern GPU. Reducing factors is not a solution to this problem, as we stated in Appendix C, line 796.

W3: This is correct. We moved the table to the main text and recomputed it for both methods on the same code (Section 5.3, paragraph "Sensitivity to \lambda" and the new λ\lambda table, in blue). Both methods need a backbone-specific λ\lambda, but MoTE is more sensitive, mainly on one side. When λ\lambda is larger than the scale of the T\times T kernel, the update shrinks to almost nothing and the metrics approach those of the unedited model. On Qwen3.6-35B-A3B Efficacy falls from 92.4 at λ=10−2\lambda=10^{-2} to 8.8 at λ=1\lambda=1 while Specificity stays at 87.2, and on Qwen3-30B-A3B Utility goes from 86.43 at λ=10−1\lambda=10^{-1} to 49.54 at λ=1\lambda=1. The failure is therefore a missing edit and not a damaged backbone. MoEEdit has a wider plateau: on Qwen3-30B-A3B it stays within one point between λ=10−1\lambda=10^{-1} and λ=1\lambda=1 (88.34 and 89.26). On Qwen3.6-35B-A3B MoEEdit stays within 0.3 points between λ=10−2\lambda=10^{-2} and λ=1\lambda=1 (80.09 to 80.35), while MoTE drops from 79.99 to 35.89. The sensitivity is added to Limitations (in blue). On GPT-OSS-20B both methods are best at λ=1\lambda=1, and MoTE loses more when $\lambda $decreases: 18.2 Utility points at λ=10−1\lambda=10^{-1} against 7.4 for MoEEdit.

We believe the revision addresses Reviewer’s concerns in full. If we have addressed all questions satisfactorily, we would kindly ask the Reviewer to reconsider their evaluation or continue the discussion.

Reviewer_NQu3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MoTE for efficient batched knowledge editing in MoE language models. It builds on existing editing objectives and per-expert preservation projectors, uses the Woodbury identity to reduce matrix inversion costs, and computes a Tucker core update at the last selected layer before reconstructing updates through each layer’s own factors. Experiments on three MoE models and two editing benchmarks demonstrate a trade-off between shorter editing time, some loss of editing quality, and increased peak memory.

Critical strengths

The paper addresses the computational bottleneck of layer-wise and expert-wise editing and demonstrates end-to-end speedups of 1.92–3.05×, including target-vector computation and preprocessing. Component ablations and long-horizon editing experiments provide useful evidence of its practical value.

Critical weaknesses

The comparisons do not establish whether MoTE offers a better quality–time trade-off than MoEEdit with fewer solver iterations under matched runtime budgets. Could the authors vary the number of BCD iterations in MoEEdit and report editing quality against end-to-end runtime across computational budgets? The current results establish that MoTE is faster than the selected baseline configurations, but do not determine whether its advantage persists at matched runtime or comparable editing quality.

Questions for authors

Could the authors include a control that preserves the corresponding whitening and regularization effects without using the full Tucker-based cross-layer construction? The existing whitening ablations establish its importance, but do not sufficiently distinguish the benefits of the resulting update constraints from those of the tensor structure and core-transfer mechanism.

2 comments

We are grateful to Reviewer for their assessment of our work!

All changes in the revised PDF are marked in blue. Two of them affect several answers below. While preparing the response we found an error in how the shared expert of Qwen3.6-35B-A3B was handled. The model scales the shared expert output by an input-dependent gate σ(wsh⊤x)\sigma(w_{\mathrm{sh}}^\top x), and our implementation used a constant gate of 1 for both MoTE and MoEEdit. We fixed this for both methods and reran Qwen3.6. The corrected description is in Section 5.1 and in Appendix C, paragraph "Shared expert" (both in blue). The Qwen3.6 rows of Tables 1, 2, 4 and 6 are recomputed on the corrected code (in blue). The other two backbones have no shared expert and are not affected. The speedup is now stated consistently as "up to 3$\times$", with the range 1.92–3.05$\times$ from Table 2 (Contributions, in blue).

W1: Thanks for a brilliant suggestion on the improvement of our work! We ran this experiment. For MoEEdit we varied the number of BCD passes from 1 to the default and measured Utility and end-to-end time on COUNTERFACT .

ModelMoEEdit, 1 pass2 passes3 passes4 passes (default)MoTE
Qwen3-30B-A3B, Utility89.3288.7588.9088.9285.25
Qwen3-30B-A3B, time, s1078.51136.31194.01251.4410.3
GPT-OSS-20B, Utility65.4969.1569.5169.7361.76
GPT-OSS-20B, time, s320.1338.7356.5373.6195.0
Qwen3.6-35B-A3B, Utility81.6780.2080.4680.3579.99
Qwen3.6-35B-A3B, time, s834.2932.81030.41128.1473.5
Two things follow. First, MoEEdit quality barely depends on the number of passes on Qwen3-30B-A3B, so fewer passes are a fair and cheaper configuration of the baseline, and we now report it. Second, fewer passes do not bring MoEEdit to the runtime of MoTE. The BCD iterations are not the bottleneck. Most of MoEEdit's time goes into collecting activations layer by layer for every batch, and this cost does not depend on the number of passes. With one pass MoEEdit is still about 1.64–2.63$\times$ slower than MoTE end-to-end. So the picture is a trade-off and we present it as one. The full table is in the new Appendix D.7 (in blue). There we also state that the times for 1 to 3 passes are estimated from the profile of the default run, and that for GPT-OSS-20B this series uses a different v* setting than Table 1, shared by all runs in the row, which is why its Utility values are lower.

Q1: We would answer in a way that we understand Reviewer's concern, so we would be happy to continue discussion if something is wrong. Our experiments with Tucker decomposition showed that decomposition itself leads to model degradation. In Table 5, our experiments has shown that without whitening the model turns into random predictor. In a TD-MoE article [1] one can found that whitening procedure can prevent model corruption. In detail, we refer to Section 3.3 of [1]. There the authors follow [2] and state that input activations can be highly correlated and raw Tucker decomposition can be not sufficient. This also motivated previous works [3] and [4] to use the same procedure to decorrelate input or output modes making decompositions more stable. Before editing we collect covariance Σin=1NX⊤X\Sigma_{in}=\frac{1}{N}X^{\top}X from general text dataset e.g. Wikitext, where X∈RN×dinX\in \mathbb{R}^{N\times d_{in}} is a matrix of input activations, and Σout=1N(∇L)Y⊤(∇YL)\Sigma_{out}=\frac{1}{N} (\nabla \mathcal{L})_Y^{\top}(\nabla_Y \mathcal{L}). We would like to emphasize that statistics source is separate from the editing prompts set. In fact, whitening is a separate procedure that does not relate to the knowledge editing itself, but we wanted to highlight its crucial role in Tucker decomposition of LLMs. Our results in Table 5 strongly correlate with a Table 1 in [1], which shows that input whitening shows better ability to save model performance after Tucker decomposition and further inference.

If the explanation is clear enough, due to the fact that other concerns have been addressed earlier, we would kindly ask to reconsider the score.

[1] Xu, Yuebin, et al. "Td-moe: Tensor decomposition for moe models." International Conference on Learning Representations (ICLR). 2026.

[2] Yuan, Zhihang, et al. "Asvd: Activation-aware singular value decomposition for compressing large language models." arXiv preprint arXiv:2312.05821 (2023).

[3] Wang, Xin, et al. "Svd-llm: Truncation-aware singular value decomposition for large language model compression." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Li, Wei, et al. "Structured mixture-of-experts LLMs compression via singular value decomposition." (2025).

My concern remains:

On Qwen3-30B-A3B, MoTE reduces editing time to 410.3 seconds compared with an estimated 1,078.5 seconds for a single pass of MoEEdit. Utility, the overall editing quality metric, falls from 89.32 to 85.25, a decrease of 4.07 points. These results establish a speed advantage for MoTE, but the time savings at comparable editing quality remain unclear.

The authors explain the role of whitening and show that removing it substantially degrades performance. My remaining question is what Tucker decomposition and the reuse of a core update across layers contribute beyond a simpler update rule that retains whitening and imposes comparable constraints on parameter updates. Ablations that isolate the contribution of each component would address this question.

Reviewer_BA1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper works on knowledge editing of Mixture of Expert (MoE) models. The authors propose a MEMIT-inspired closed-form knowledge editing method (MoTE). MoTE leverages the Woodbury identity to reduce the computational cost and uses Tucker decomposition to capture the structural relationships among experts. To further improve efficiency, MoTE computes the update at the target layer and distributes it across preceding layers using a MEMIT-style update. Experiments on MoTE demonstrate relatively competitive editing performance and higher efficiency compared to existing methods.

Critical strengths
  • The problem studied in this problem is important and interesting.

  • The proposed method is mathematically grounded in general. The Woodbury (push-through) reduction can motivate future works.

Critical weaknesses
  • My first main concern about the paper is about the proposed “Spreading updates” step. From Eq. 5 and Eq. 6, this could potentially cause routing instability. In MEMIT, the updates are spread among multiple layers, where each layer needs to re-forward the model with the lower edited layers and then update its parameters accordingly. The proposed method solves for the update only once at the last edited layer L and directly reuses this solution which is scaled by 1/(L−l+1)1/(L-l+1) in the lower layers. However, this will lead to routing instability. If the parameters in layer L−1L−1 are updated, the input to layer LL changes, so the routing in layer LL could be perturbed and the expert selection can be flipped. Since the update at layer L is solved using the gates and expert keys computed before the lower layers are edited, it could end up flipping the actual routing after these edits. The results in Table 6 also confirm that the proposed method has less stable routing when compared with MoEEdit.

  • My second main concern is that the the effectiveness of the proposed Tucker-structured editor is not well established. In Table 2, the performance improvement with Tucker looks more like a trade-off between specificity and generalization, which is a common trade-off in knowledge editing. So, it is hard to be convinced the Tucker structure fixes the discussed issues in Sec. 4.2, which the MEMIT-style editing paradigm treats experts independently. The improvement by the Tucker structure also seems relatively marginal on GPT-OSS.

  • As a minor concern, the writing quality can be improved. For example, the claimed performance gain are not consistent in the paper. The “up to 3×” is used in the abstract, “up to 4×” in the contributions and “up to n times” in Sec. 4.3. Table 2 reports 1.92–3.05×.

Questions for authors
  • Why is the solution GG obtained at layer LL “layer-agnostic”?
  • Eq. 9 solves only at layer LL and reuses the resulting solution in lower layer l scaled by 1/(L−l+1)1/(L-l+1) and by that layer’s independently computed HOSVD factors. Can you clarify and explain this design?
1 comment

We thank Reviewer for their effort in reviewing our work!

All changes in the revised PDF are marked in blue. Two of them affect several answers below. While preparing the response we found an error in how the shared expert of Qwen3.6-35B-A3B was handled. The model scales the shared expert output by an input-dependent gate σ(wsh⊤x)\sigma(w_{\mathrm{sh}}^\top x), and our implementation used a constant gate of 1 for both MoTE and MoEEdit. We fixed this for both methods and reran Qwen3.6. The corrected description is in Section 5.1 and in Appendix C, paragraph "Shared expert" (both in blue). The Qwen3.6 rows of Tables 1, 2, 4 and 6 are recomputed on the corrected code (in blue). The other two backbones have no shared expert and are not affected. Other results The speedup is now stated consistently as "up to 3$\times$", with the range 1.92–3.05$\times$ from Table 2 (Contributions, in blue).

W1: We agree with the description of the mechanism. The solve at layer L uses gates and keys computed before the lower layers are written, so it is an approximation, and its cost shows up as the Utility gap to MoEEdit in Table 1. We checked directly whether the routing at the edited layers flips. On one batch of 50 COUNTERFACT edits on Qwen3-30B-A3B we compared the top-8 expert sets at the subject token before and after the edit. For MoTE the Jaccard overlap is 1.00 / 0.996 / 0.987 / 0.982 / 0.943 on layers 3 to 7. For MoEEdit, which re-forwards the model after every layer, it is 1.00 / 0.70 / 0.67 / 0.59 / 0.57. So on the edited prompts the routing at the edited layers moves less under MoTE than under MoEEdit. This measurement is added to Appendix D.1 (last sentence, in blue). Table 6 measures something different: routing similarity in the downstream layers 11 to 40. There MoTE is indeed lower than MoEEdit, most visibly on Qwen3.6-35B-A3B after the shared-expert fix (33.7 to 47.1 against 76.8 to 85.0), and we do not want to dismiss this. On this model the drop depends on where the target vectors are optimized: with v* at layer 5, as in MoEEdit, routing similarity is 66 to 80 at a COUNTERFACT Utility of 75.0 (new row of Table 6, in blue). Our position is that routing similarity is an indirect signal, and the direct question is whether the model still works. On MMLU (Table 4) MoTE loses 0.40 / 0.64 / 0.78 points and MoEEdit loses 0.51 / 1.15 / 0.45. This agrees with reports that moderate routing changes in trained MoE models leave quality almost unchanged, for example on Qwen3-30B-A3B itself in GSPO (Zheng et al., 2025, Section 5.3) and across four MoE models in Skliar et al. (TMLR 2025).

W2: Thanks for the note! We would like to clarify that the main goal of our method is to make editing of MoE models cheaper in terms of time and compute, that is why we suggest multilayer spread in Eq 5, which obviously leads to performance degradation. Other components of our method are intended to restore lost scores. Comparing “MoTE” and “MoTE w/o Tucker” rows in Table 2 demonstrates that usage of Tucker Structure does not bring only the generalization-specifity tradeoff. It restores most of the generalization score which gets lost because of our multilayer spread in Eq. 5 (which turns into Eq. 9 in case of Tucker) and is crucial for designing a method comparable to MoEEdit.

W3: Thanks for such an important note! We revised our writing, and now in all mentioned places one can find “up to 3$\times$”, which is consistent with a Table 2.

Q1: The wording was too strong and we removed it. What we meant is narrow: all edited layers use the same multilinear ranks, so the core has the same shape on every layer and can be written to another layer through that layer's own factors. This is now what the text says (Section 4.3, end of the paragraph after Eq. 7, in blue). We do not claim that the core solved at layer L is the optimal core for a lower layer.

Q2: Reviewer’s reading is correct. We collect statistics and solve once, at the last edited layer, and write the scaled core into each lower layer through that layer's own HOSVD factors and projectors (Section 4.3, "Spreading updates", in blue). The motivation is cost. In MoEEdit the per-layer statistics collection is the largest part of the runtime, and Eq. 9 leaves only one such collection. The abstract now names this as the main source of the speedup (in blue). We have no theoretical argument that the choice is optimal for the lower layers, and we added this to Limitations (last sentence, in blue).

In addition we would to provide a results of ablation suggested by Reviewer NQu3, which compares our method with MoEEdit with reduced number of BCD steps, one can find it in a revised pdf, Appendix, D, Table 13. It shows that fewer BCD passes barely change the quality of MoEEdit and do not bring its runtime close to MoTE: with one pass MoEEdit is still 1.64–2.63$\times$ slower end-to-end.

We would kindly ask Reviewer to reconsider their score or continue discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.