acceptodds
Under review as a conference paper at ICLR 2027

Reinforcing Large Vision Language Models for Multi-Hop Reasoning in Lifelong Knowledge Editing

Abstract

Lifelong multimodal knowledge editing aims to correct outdated or incorrect knowledge in large vision language models (LVLMs) without retraining. Recent studies have significantly improved editing quality in terms of locality, generality, and reliability. However, the ability of post-edit models to perform multi-hop question answering over sequentially injected facts remains critically overlooked. This challenge stems from the confinement of existing supervision signals to the edited fact itself, with no dedicated mechanism for coherent multi-hop reasoning chains. To bridge this gap, we propose **MoE-GRPO**, a reinforcement learning framework built on GRPO for lifelong multimodal knowledge editing over MoE-based editors. Using two-stage singular value perturbation via full SVD decomposition and carefully designed outcome-level rewards, MoE-GRPO provides direct supervision over multi-hop reasoning, bypassing the need to explicitly model intermediate parameter disruption. Extensive experiments on BLIP2-OPT, LLaVA-v1.5, and MiniGPT4 demonstrate that MoE-GRPO achieves substantial relative gains in multi-hop reasoning, while accepting a small, deliberate trade-off on single-hop editing metrics (Reliability, Locality, Generality) that is intrinsic to the multi-hop reasoning objective. Ablation studies reveal practical trade-offs between editing quality and reasoning ability induced by key design choices. Furthermore, MoE-GRPO generalizes consistently across different MoE-based editors, yielding robust reasoning improvements.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge editing, multi-hop reasoning, group-relative policy optimization, large vision language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fxr2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MoE-GRPO extends LiveEdit with reinforcement learning to improve multi-hop reasoning after lifelong multimodal knowledge editing. It updates the expert generator and router while freezing the backbone, using singular-value perturbation, a max-based objective, asymmetric gradient clipping, and a composite reward.

Critical strengths

MoE-GRPO targets the gap between retaining edited facts and using them in multi-hop reasoning, with improvements on BLIP2 and LLaVA. The study evaluates multiple backbones and ablates several rewards, clipping, gradient, and perturbation choices. Applying the method to LEMoE provides limited evidence that it can extend beyond LiveEdit.

Critical weaknesses
  1. Eq. (8) requires a likelihood for the joint action of expert matrices and router outputs, but the paper specifies stochasticity only over expert singular values. It defines neither the induced action density nor the resulting importance ratio and reference KL. The main text decomposes the current matrices, whereas Appendix K fixes the singular vectors to the pretrained basis. If old and new policies use different bases, their distributions may have different support, leaving the objective undefined.

  2. In Eq. (15), the negative sign means that maximizing the reward discourages positive residual similarity and can favor concentrating capacity in fewer experts, contrary to the claimed compatibility and co-activation. Eq. (16) compares vectors whose lengths grow with the expert pool without defining alignment or padding. Both rewards use parameter norms rather than routing probabilities, so they do not directly measure routing consistency. The main objective also contains no multi-hop reward, making the claim of direct multi-hop supervision inaccurate.

  3. The evaluation definitions are inconsistent. Eq. (31) compares a rephrased input with the unedited model's original output, which conflicts with generalization to the new fact. Eq. (32) defines an indicator metric, while Figures 4 and 7 use unexplained partial credit. The metric and decoding procedure are therefore unclear.

  4. The claimed small trade-off is not supported at 1,000 BLIP2 edits, where reliability and text generalization each drop by about 20 points relative to LiveEdit. MiniGPT4 gains are small, and the 100-edit LEMoE result is nearly unchanged. Some reported gains also use different baselines or edit counts from those stated: 30.48% is relative to LEMoE rather than LiveEdit, while 2.17% comes from another edit-count comparison.

  5. The max/min objectives are not tuned separately, and the paper does not report matching the Gaussian perturbation energy. Freezing the backbone does not establish gradient underestimation, and Eq. (11) appears to apply the noise scale twice in the later stage.

  6. The AI-use statement limits generative AI to language editing, whereas Appendix D.1 uses GLM-5.1 to synthesize auxiliary QA data. This disclosure should be corrected.

Questions for authors
  1. How is the joint action density defined, and how are the importance ratio and reference KL computed? Do old and new policies use the same singular basis and support?

  2. Why does the negative similarity in Eq. (15) encourage compatible residuals? How are the different-length vectors in Eq. (16) aligned, and why do parameter norms measure routing consistency?

  3. Is Eq. (31) incorrect? What metric and decoding protocol produce the tables and the partial scores in Figures 4 and 7? Which baselines and edit counts define the reported gains?

  4. After tuning learning rate and KL separately, does max still outperform min? Does SVD perturbation remain better when Gaussian perturbation energy is matched? At comparable editing quality and cost, does MoE-GRPO outperform further training of LiveEdit?

Reviewer_bXCAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MoE-GRPO, a reinforcement-learning framework for improving multi-hop reasoning after sequential knowledge edits in vision-language models. It builds on LiveEdit’s small editing experts while keeping the underlying model frozen. Experiments on BLIP2, LLaVA, and MiniGPT4 show improved outcome on multi-hop reasoning.

Critical strengths
  • The paper addresses a meaningful gap between updating knowledge and reasoning with that knowledge, as a model may correctly recall a newly edited fact while still answering related questions using its old knowledge.
  • The paper conducts extensive experiments across a wide range of settings.
Critical weaknesses
  • The practical significance of the paper’s focus is not well motivated, given that agents can easily update their knowledge through tools like search through their harnesses.
  • The proposed pipeline is of great complexity and it’s unclear whether all the proposed complexity is necessary to achieve a good performance.
Questions for authors

N/A

Reviewer_GMMMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds reinforcement learning and structured exploration to a LiveEdit-style editor to improve multi-hop reasoning after sequential knowledge edits. The task is important because successfully updating individual facts does not automatically ensure that a model uses those updates correctly in relation-chain reasoning.

Critical strengths
  1. The multi-hop task is genuine and valuable. VLKEB includes visual relation-chain reasoning rather than simply relabeling ordinary generalization, and experiments cover different backbones and edit-sequence lengths.

  2. Multi-hop gains are clear, with reward, noise, and min/max ablations. Comparisons with LiveEdit, LEMoE, and RLEdit help position the contribution. These results and existing ablations should be acknowledged rather than described as absent.

Critical weaknesses
  1. The core training definition remains insufficiently specified. Actions involve expert parameters and routing, with randomness from singular-value sampling. Computable joint-action probabilities, current/old policy ratios, KL terms, and gradient paths are needed, along with the parameters updated when adapting LEMoE. This is a core reproducibility gap, not proof that the RL method is invalid.

  2. The trade-off between multi-hop gains and single-hop editing quality is substantial. Some BLIP2 single-hop metrics decline by approximately 20 percentage points, which is not a minor cost. Retuned baselines with similar single-hop quality and training budgets are needed to establish whether the multi-hop advantage persists. This severe loss should not be generalized to every backbone.

Genuine multi-hop improvements deserve credit, but core reproducibility details and the benefit-cost analysis remain insufficient. The earlier erroneous objection to Equation (16) is withdrawn and does not inform the rating; the current judgment rests on the training-definition and performance-trade-off concerns.

Questions for authors

Can you provide action-probability and parameter-update definitions sufficient to implement the training procedure, and report multi-hop/single-hop trade-off curves, particularly comparisons at similar single-hop performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.