acceptodds
Withdrawn from ICLR 2027

Dynamic Mixture of Adaptive Experts for Lifelong Multimodal Knowledge Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Updating Multimodal Large Language Models (MLLMs) without catastrophic forgetting remains a fundamental challenge. Existing lifelong knowledge editing methods face two critical limitations: interference from previously learned knowledge, and knowledge capacity dilemma arising from the difficulty of predefining appropriate parameters. We propose Dynamic Mixture of Adaptive Experts (MoAE), a novel framework that addresses these challenges through dynamic expert expansion and adaptive knowledge capacity. Each expert comprises a shared LoRA and multiple specific LoRAs that expand progressively based on expert utilization frequency. A hierarchical routing mechanism is proposed to select relevant experts and their corresponding specific LoRAs for each input. Extensive experiments on ComprehendEdit and VLKEB benchmarks across MiniGPT-4, LLaVA-1.5, and Qwen2.5-VL demonstrate that MoAE achieves superior overall performance on reliability, generality, and locality in lifelong editing scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jhwgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies lifelong multimodal knowledge editing and proposes MoAE, an extension of MoE-based editing (ELDER) with dynamic expert expansion. Specifically, (1) a new expert is added when the distance between an input representation and existing routers exceeds a threshold ϵ\epsilon; (2) each expert consists of a shared LoRA and multiple specific LoRAs, where a new specific LoRA is added every m=5m=5 samples processed by that expert; and (3) a replay-based margin loss is used to prevent router drift, combined with an external image-text retrieval step (DINOv2 + reranker) for inference gating. The method is evaluated on ComprehendEdit and VLKEB.

Critical strengths
  1. The motivation is practical. Since pre-defining a fixed number of experts and LoRA ranks is difficult for streaming edits of unknown length (Fig. 1a), expanding experts by representation distance and adding specific LoRAs by usage count is a sensible design.
  2. The paper identifies the "router forgetting" issue when adding new routers online and addresses it by replaying boundary representations with a margin loss (Sec. 3.2.1), which is shown to be effective in Table 3.
Critical weaknesses
  1. Unfair inference retrieval setup inflates Locality and masks Generality drops (Sec. 3.2.3). The memory bank MM stores pre-extracted DINOv2 and text-reranker features for all edited samples along with p=20%p=20\% of unedited locality samples from the training set. At test time, if the nearest neighbor in MM is a locality sample, the model skips the edited modules entirely and returns the original model f0f_0. This trivially gives 100.00 T-L in Table 1 and makes the baseline comparison unfair. Meanwhile, on Text Generality (T-G), where the edited modules are actually used, MoAE drops noticeably behind ELDER (e.g., ComprehendEdit T=100T=100: 66.00 vs. 78.26; VLKEB T=400T=400: 28.42 vs. 46.25). Its higher AVG score (Eq. 4) is mainly driven by averaging in the saturated locality metrics.
  2. Mismatch between hl(xt)h^l(x_t) and h0(xt)h^0(x_t) across layers, plus copy-paste errors in Table 1. Sec. 3.2 triggers expansion at layer ll using layer-ll features hl(xt)h^l(x_t) and routers RlR^l, but Sec. 3.2.1 and Eq. (3) state that MM only stores a single representation h0(xt)h^0(x_t) per sample to compute router weights s(h0(xt),Rl)s(h^0(x_t), R^l) across all injected layers (22–26). Since feature spaces change across layers, it is unclear how h0(xt)h^0(x_t) can be directly compared with deeper routers RlR^l during routing and replay. Also, in Table 1, all six metrics for Merging (T=100T=100) and SD-LoRA (T=400T=400) are identical across ComprehendEdit and VLKEB, which looks like a copy-and-paste error.
Questions for authors
  1. What are the T-L, M-L, and KPI scores of MoAE if you remove the external DINOv2 + reranker retrieval step and the 20% stored locality samples in Sec. 3.2.3, and use only the original deferral mechanism from ELDER?

  2. Could you clarify the discrepancy between hl(xt)h^l(x_t) and h0(xt)h^0(x_t)? In which feature space do the routers RlR^l (for layers 22–26) reside, and how is router replay (Eq. 2) computed for deeper layers if only h0(xt)h^0(x_t) is stored?

  3. Why are the results for Merging (T=100T=100) and SD-LoRA (T=400T=400) in Table 1 identical across ComprehendEdit and VLKEB?

Reviewer_RNztOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MoAE for lifelong multimodal knowledge editing. It targets the problem that a fixed parameter budget either causes interference between edits or wastes parameters. MoAE adds a new expert when an input's representation is farther than a threshold from all existing routers, a rule the authors describe as a simplified DPMM. Each expert is split into a shared LoRA and several specific LoRAs, and a new specific LoRA is added for every 5 samples the expert processes. Router replay with a margin loss keeps routing stable for earlier edits. At inference, a retrieval gate built on DINOv2 and a BGE reranker decides whether an input should be edited at all. Experiments on ComprehendEdit and VLKEB with three MLLM backbones report the best average score in most settings.

Critical strengths
  1. The motivation is clear. Figure 1(a) shows the capacity–efficiency trade-off that a fixed parameter budget creates in sequential editing.

  2. The evaluation is reasonably broad: two benchmarks, three backbones, sequences of up to 1000 edits, and a module-level ablation.

  3. Router replay directly addresses "router forgetting", a real side effect of expanding the expert pool, and the ablation shows that it matters.

Critical weaknesses
  1. Unfair locality gains. The headline improvements on M-L and KPI come almost entirely from the retrieval gate, which uses training-set locality samples and two external encoders (DINOv2, BGE) unavailable to any baseline. The core editing components are not responsible for the main gains.

  2. Overclaimed results and unreliable tables. Generality often trails baselines (e.g., 66.00 vs. 78.26 for ELDER on ComprehendEdit T=100). Table 1 contains identical rows for Merging and SD-LoRA across datasets, and ELDER++ underperforms ELDER despite more parameters — both casting doubt on baseline reproducibility.

  3. Limited novelty and missing overhead analysis. The DPMM framing reduces to standard distance-threshold clustering, and the remaining components follow LEMOE, DualPrompt, GRACE, and ELDER. The paper also never reports memory or latency, leaving its efficiency claims unverified.

Questions for authors

Same as weakness.

Reviewer_sFvMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoAE for lifelong multimodal knowledge editing through dynamic expert expansion and adaptive knowledge capacity. A new expert is added when the input representation exceeds a distance threshold from all existing router embeddings. Each expert contains a shared LoRA and multiple specific LoRAs. Representation replay with a margin loss helps maintain historical expert assignments. During inference, hierarchical routing selects experts and specific LoRAs, while image-text feature retrieval determines whether to apply editing. Comparative experiments support the method's effectiveness.

Critical strengths
  1. The paper addresses the knowledge capacity dilemma caused by under-provisioning and over-provisioning capacity, while the hierarchical expert framework separates shared and sample-specific knowledge through a shared LoRA and multiple specific LoRAs.
  2. The paper identifies router forgetting caused by expert expansion and introduces representation replay to mitigate it.
  3. MoAE achieves the highest AVG across the four settings in Table 1, providing evidence of its effectiveness.
Critical weaknesses
  1. The method shares substantial similarities with ELDER and LEMoE. Its technical novelty appears limited, and a distinct algorithmic insight is not clearly established.
  2. LiveEdit is cited but omitted from the experimental comparisons. Its use of low-rank experts and routing for lifelong multimodal knowledge editing makes it a directly relevant baseline.
  3. Table 3 does not highlight the best and second-best results using bold and underlining, making comparisons harder to follow.
Questions for authors
  1. In Table 3, MoAE improves KPI but reduces T-G compared with wo Gating. Could the authors explain this trade-off?
  2. The method expands specific LoRAs and stores historical representations, image-text features, and indices. How do storage requirements and inference latency grow with the number of edits? Please provide an efficiency evaluation.
  3. In Table 1, Merging at T=100 and SD-LoRA at T=400 have identical results across all metrics on ComprehendEdit and VLKEB. Could the authors explain these identical values?
Reviewer_HbGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Mixture of Adaptive Experts, a framework for lifelong multimodal knowledge editing. The method dynamically expands experts based on representation similarity and adjusts the capacity of individual experts by introducing shared and specific LoRA modules. It further incorporates representation replay to mitigate router forgetting and a hierarchical routing mechanism with feature-based retrieval to select the relevant editing parameters. The authors evaluate MoAE on ComprehendEdit and VLKEB using MiniGPT-4, LLaVA-1.5, and Qwen2.5-VL, reporting improvements in editing reliability, locality, and overall performance over several existing baselines.

Critical strengths
  1. We think the most contribution is the paper provides the inspiration that the two-level expansion strategy is a useful design for lifelong knowledge editing. By separating expert-level expansion from fine-grained LoRA capacity allocation, the framework attempts to accommodate uneven expert utilization while reducing interference between edited knowledge.
  2. The empirical experimental results and ablation results proved that the sequential editing is practically promising.
Critical weaknesses

My core concern is that the paper is an incremental novelty compared to existing works, because we can only see that you group and combine several established techniques rather than introduce a substantially new learning mechanism. Dynamic expert expansion and representation-based module selection have been investigated in previous work such as LEMoE and SEMA. ELDER already employs a mixture of LoRA modules with routing and deferral mechanisms for lifelong model editing. Similarly, shared and specialized adaptation modules, as well as representation replay, are established ideas in continual learning.

In a addition, the experimental design makes it difficult to determine how much of the reported improvement originates from adaptive expert expansion.

Questions for authors

1.Could you provide a more explicit comparison with ELDER, SEMA, and LEMoE?

2.How much improvement comes independently from dynamic expert expansion, specific-LoRA expansion, and their combination under matched parameter and computation budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.