acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Routing Mismatch in Multimodal Knowledge Editing

Abstract

Multimodal knowledge editing aims to update specific knowledge in multimodal large language models (MLLMs) without full retraining. We identify that in methods requiring editing parameter selection, the gap between teacher-forcing training and autoregressive generation is mainly caused by routing mismatch: optimal parameters selected in training are not reliably activated during generation. To bridge this gap, we propose a generation-aligned optimization framework that formulates knowledge editing as a sequential decision process. We simulate generation scenarios to allow the model to perform parameter routing and optimization on prefixes it may encounter during generation. To prevent gradient corruption caused by erroneous prefixes, we further propose a Generation-Aligned Prefix (GAP) loss that selectively propagates gradients only along valid tokens. Experiments on VLKEB and ComprehendEdit demonstrate that our method significantly improves generation accuracy under both single-edit and lifelong-edit settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal language models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Ccg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the performance discrepancy between Teacher Forcing (TF) training and Autoregressive Generation (AG) in parameter-routing-based Multimodal Knowledge Editing. The authors identify "routing mismatch"—where optimal parameters selected during training are not reliably activated during generation—as a primary cause of generation failures. To mitigate this, they propose a "Generation-Aligned Optimization" framework featuring a Generation-Aligned Prefix (GAP) loss and Semantic-Anchor Gating. The GAP loss truncates gradient propagation after the first incorrectly generated token to prevent knowledge corruption, while Semantic-Anchor Gating uses frozen multimodal features to stabilize routing decisions. The method is evaluated on VLKEB and ComprehendEdit benchmarks using multiple MLLM backbones.

Critical strengths
  1. The paper provides a solid empirical diagnosis of why dynamic inference methods (like ELDER or MELO) fail during autoregressive generation. Isolating the "routing mismatch" from general exposure bias offers a useful perspective on the fragility of dynamic parameter selection in editing.

  2. The experiments are extensive, evaluating across multiple recent MLLM backbones (Qwen2.5-VL, LLaVA-OneVision) and comparing against a wide array of both static and dynamic inference editing methods.

Critical weaknesses
  1. The paper heavily frames the Generation-Aligned Optimization and GAP loss as a "sequential decision-making process" inspired by Reinforcement Learning (RL) (Section 3.3). However, a closer inspection of Equation (4) reveals that the GAP loss is essentially a simple Truncated Cross-Entropy Loss—it merely masks out the loss computation after the first incorrectly predicted token. Dressing up a basic sequence-masking heuristic as a novel RL framework is a severe overstatement and detracts from the paper's academic rigor. Furthermore, training on model-generated prefixes is a highly standard technique (e.g., Scheduled Sampling) to mitigate exposure bias, limiting the theoretical novelty of this approach.

  2. The proposed Semantic-Anchor Gating resolves the issue of hidden-state drift by using a frozen CLIP model to compute cosine similarities for activation decisions. This is a highly standard practice utilized by numerous prior memory-based or retrieval-based editing methods (e.g., SERAC). Presenting this as a core methodological contribution lacks conceptual innovation.

  3. The design philosophy of the GAP loss—penalizing the model the moment it deviates from the exact literal string of the ground truth—raises significant concerns regarding exact-match overfitting. While this strategy artificially inflates exact-match metrics like Rel(G) and T-G(G), it forces the model into rote memorization of specific answer templates. The paper completely fails to investigate whether this strict token-matching enforcement degrades the model's open-ended generation capabilities, semantic flexibility, or general multimodal QA performance.

  4. The authors attempt to strictly distinguish "routing mismatch" from "exposure bias." However, in a dynamic routing network where routing depends on the generated prefix, routing mismatch is fundamentally a direct symptom and downstream consequence of exposure bias. The conceptual decoupling feels somewhat forced and lacks theoretical depth.

Questions for authors

see weakness

Reviewer_MnwTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the mismatch between training-time and generation-time routing in multimodal knowledge editing. It proposes a Generation-Aligned Prefix (GAP) loss and a gating mechanism based on frozen representations. Experiments on two datasets show improvements in generation accuracy, and additional experiments evaluate GAP when applied to existing editing methods.

Critical strengths
  1. The controlled interventions in Figure 1 distinguish routing changes from generated-prefix errors and identify a specific failure mode in the evaluated editor. This gives a more concrete explanation of the training-generation gap.
  2. The gains in generation accuracy are substantial, particularly on VLKEB. Table 4 also shows improvements when GAP is added to ELDER and BalancEdit, supporting its usefulness beyond the proposed architecture.
Critical weaknesses
  1. The optimization interpretation needs clarification. Section 3.3 motivates the method using a sampled-action REINFORCE objective, while the introduction describes reference-token supervision at the first error. The connection between these updates is not explained. GAP appears to be masked supervised cross-entropy, but the paper should clarify whether it is a policy-gradient estimator or a supervised surrogate. Equation (4) also needs explicit conventions for a first-token error under zero-based indexing and for an entirely correct sequence.
  2. The ablations do not fully separate prefix-consistent routing from prefix truncation. Figure 2(b) compares against training on generated prefixes, while the CE term in Equation (5) uses full-answer routing. Neither supplies a control using token-wise routing on ground-truth prefixes without truncation. Table 2 does isolate the gate within the proposed architecture; a shared-gate comparison with an existing editor would further clarify the source of the locality advantage.
Questions for authors
  1. The introduction states that GAP supervises the reference token at the first error. How does this update follow from Equation (3), and how are the boundary cases in Equation (4) implemented?
  2. Under a matched budget, how does GAP compare with token-wise routing on ground-truth prefixes without truncation, or routing fixed from the image-question prompt during both training and generation?
  3. Do the generation and locality advantages over ELDER or BalancEdit remain when they use the same semantic gate and locality training data?
Reviewer_7k3HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies routing mismatch as a major contributor to the gap between teacher-forcing training and autoregressive generation in multimodal knowledge editing. It proposes a generation-aligned optimization framework incorporating step-wise dynamic routing, Generation-Aligned Prefix (GAP) loss, and Semantic-Anchor Gating. Experiments on VLKEB and ComprehendEdit demonstrate substantial improvements in generation accuracy, particularly under lifelong editing settings.

Critical strengths
  1. The paper provides convincing empirical evidence of routing mismatch by separately controlling prefix conditioning and expert assignment.
  2. The proposed method substantially improves generation accuracy, especially in lifelong editing scenarios on VLKEB.
  3. Applying GAP loss to ELDER and BalancEdit also yields notable improvements, while the ablation studies generally support the effectiveness of the proposed components.
Critical weaknesses
  1. GAP loss is closely related to existing prefix-based training strategies, and its connection to REINFORCE-style optimization is not sufficiently justified. Additionally, Eq. (4) leaves certain boundary cases undefined, such as when the first token is incorrect or all tokens are correct.
  2. Although Figure 2(b) compares GAP with vanilla autoregressive loss and Table 2 evaluates different components, a more systematic comparison of fixed versus dynamic routing and truncated versus untruncated supervision would help clarify their individual contributions.
  3. On VLKEB with T = 1000, Rel(G) and T-G(G) remain at 36.3% and 20.2%, respectively. These results suggest that generation reliability and textual generalization remain challenging despite substantial improvements over existing baselines.
  4. The manuscript specifies only one random seed, and the referenced Appendix A is absent from the provided version. Moreover, Table 3 shows that incorporating GAP increases training time from 0.80 to 1.20 seconds per sample, which warrants further clarification of the computational efficiency claims.
Questions for authors

follow the weakness

Reviewer_VdjaOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors first show that knowledge editing methods that select parameters dynamically do worse during generation because of routing mismatch. During training, the router selects experts after seeing the full answer. However, during generation, it only has the prefix so far, so it can select different experts, which might not have learned the edit.

To fix this, they propose a new GAP (Generation-Aligned Prefix) loss, which supervises only the tokens where the generated prefix matches the reference, up to and including the first wrong token, and ignores everything after that. Thus experts are selected from the prefix during training, and only for right prefix.

Another improvement the authors propose is Semantic-Anchor Gating, which decides whether an input should use the editing module, as not every input during inference needs editing. For this, they compare the input’s image and text embeddings from frozen CLIP with the stored embeddings of edit and locality (unrelated questions) samples, and the module is turned on only if the nearest sample is an edit sample.

Their approach beats other methods on average of reliability, generality and locality. The ablation study shows GAP improves generation accuracy and the gate improve locality. Finally, they show that adding GAP to two other dynamic-routing methods, ELDER and BalancEdit, also improves their generation accuracy.

Critical strengths
  • The problem and solution are well motivated. They show the problem nicely where they change prefix and routing separately, vary how many experts match, and measures how often generation reuses the trained experts. The GAP loss idea is simple - stop supervising once the output deviates from the reference, while correcting the first mistake.
  • They show GAP can improve generation accuracy in other dynamic-routing methods (ELDER and BalancEdit)
  • Strong results. Rel(G) is the best in every setting of Table 1
Critical weaknesses
  • After training with GAP, how often does generation reuse the trained experts? That would show GAP leds to aligned routing.
  • How can Rel be lower than Rel(G)? e.g. GAP only Rel(G) is 79.5 vs 71.4 for Rel in Table 2.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.