acceptodds
Under review as a conference paper at ICLR 2027

EAGPO: Edit-Anchored Generalization Policy Optimization for Knowledge Editing

13 views

Abstract

Knowledge editing aims to precisely update factual knowledge in large language models. However, existing methods struggle to generalize the editing knowledge to semantic variants, limiting their practical application. This fragility often arises because these methods optimize solely for a single target expression, neglecting the broader semantic neighborhood that real-world queries inhabit. To bridge this gap, we propose EAGPO, a novel framework that enhances knowledge editing generalization to semantic variants through Edit-Anchored Generalization Policy Optimization. It first employs a standard knowledge editing method to precisely integrate the new knowledge. Then, it performs reinforcement learning training via two core components: an Anchor-Generalization Training (AGT) mechanism that optimizes over structured query clusters, and a Policy Gradient Alignment Advantage (PGAA) module that stabilizes updates through geometric gradient alignment. Theoretically, EAGPO strengthens knowledge protection, provides a tighter generalization error bound, and reduces policy gradient variance. Empirical results on the KnowEdit benchmark show that EAGPO significantly outperforms strong baselines across three types generalization tasks, improving generalization performance by 25%–35% over the original knowledge editing methods, while achieving a 25% improvement in computational efficiency over GRPO training. Our code is available at https://anonymous.4open.science/r/Edit-GRPO-119DEAGPO https://anonymous.4open.science/r/Edit-GRPO-119D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Editing, Knowledge Generalization, Reinforcement Learning, Gradient Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iVdkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

EAGPO adds a reinforcement-learning stage after an initial knowledge edit to improve generalization beyond the original edit query. Anchor-Generalization Training groups the edit query with subject-alias, reversed-relation, and one-hop queries, using one sampled response per query. The advantage baseline is computed from the generalization queries rather than the full cluster. PGAA then adds a cosine-similarity term between each generalization response’s score gradient and the anchor gradient, measured at a selected layer.

Critical strengths
  1. The main comparison shows improvements beyond simply adding post-edit training.

  2. The paper includes a useful alignment ablation and evaluations beyond the main benchmark.

Critical weaknesses
  1. The generalization guarantee in Theorem 4.2 is not established by the proof. Appendix B.2 defines uniform stability over all neighboring datasets, but its key step assumes that the replaced sample is a “typical conflict sample” and uses the conflict fraction as its probability of occurrence. That is an average-case argument, not the required worst-case bound. If the anchor is replaced, its gradient also changes the alignment coefficients of the other samples; the proof does not bound this shared dependence. The subsequent comparison has another problem: separate upper bounds on the two parameter divergences do not imply the claimed inequality between the divergences themselves. Likewise, a smaller upper bound on generalization error does not establish a strict ordering of the actual errors. These gaps prevent the theorem from supporting the paper’s claim of provably better generalization.

  2. The variance guarantee in Theorem 4.5 relies on an unjustified comparison of estimator means and second moments. Appendix B.3 states that the baselines are independent of action sampling for a given query. However, for a generalization query, the peer mean includes that query’s own sampled reward. It is therefore not an action-independent baseline. The added term λcos⁡(gi,g1)gi\lambda \cos(g_i, g_1) g_i generally has nonzero expectation as well, so comparable estimator means cannot simply be assumed. The proof then neglects aggregate off-diagonal terms described as O(1/G)O(1/G), without showing that they are smaller than the retained variance improvement. Under its stated λ=O(1/G)\lambda = O(1/G) scaling, the retained conflict contribution in Equation 30 can be only O(1/G3)O(1/G^3). An uncontrolled O(1/G)O(1/G) remainder cannot be discarded when establishing its sign. Figure 2 may show lower measured variance during training, but it does not resolve these analytical issues.

Questions for authors
  1. Theoretical claims: Can you provide corrected statements and proofs for Theorems 4.2 and 4.5 that address worst-case neighboring datasets, the dependence introduced by the anchor and peer baseline, and the estimator mean and covariance terms? Please also distinguish guarantees for the simplified update from claims about the normalized, clipped training objective.
  2. Held-out evaluation: Appendix A.1 mentions original train/test splits, while Appendix C constructs training clusters from the anchor and its portability queries. What exactly is held out: anchors, query variants within an anchor, relation templates, or some combination? Please give the training and evaluation counts and explain whether test-anchor portability queries ever enter RL training.
  3. Implementation: Is the descent sign in Algorithm 1 a typographical error? How is Equation 4 implemented when the generalization rewards have zero or near-zero standard deviation? Please specify the stabilizer, any skipped batches, and which parameters contribute to the alignment calculation.
  4. Empirical interpretation: How is the policy-gradient variance in Figure 2 estimated, and what do the ± values in Tables 1 and 7 represent? For Table 3, are previous-session facts reevaluated after later sessions, or does each row report current-session performance? These definitions affect the evidence for optimization stability and resistance to forgetting.
Reviewer_mj4eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EAGPO, a framework that combines knowledge editing with reinforcement learning to improve generalization to semantic variants. It introduces two core components: Anchor-Generalization Training and Policy Gradient Alignment Advantage. The paper provides theoretical analyses of reliability, generalization, and gradient variance, and reports improved generalization on KnowEdit with lower training cost than GRPO.

Critical strengths

The paper presents an extensive evaluation across multiple model backbones and knowledge editing methods, supported by detailed ablation studies and additional analyses of sequential editing, multi-hop reasoning, and computational efficiency.

Critical weaknesses

In Appendix B.2, Equation (23) and Equation (24) provide separate upper bounds on δTstd\delta_T^{\mathrm{std}} and δTedit\delta_T^{\mathrm{edit}}, but these bounds do not imply the multiplicative relationship asserted immediately afterward. Since the subsequent stability and generalization arguments rely on this unjustified step, the current proof does not establish the generalization guarantee claimed in Theorem 4.2.

Questions for authors

See Weaknesses

Reviewer_KT2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EAGPO framework, for improving knowledge-editing generalization to semantic variants.

The method proposes a two-stage pipeline where the first stage applys a standard editor (ROME, MEMIT, etc.) on an anchor query, and the second stage runs a GRPO-style RL stage over an Anchor–Generalization Training (AGT) cluster (anchor + paraphrase/variant queries.

The advantages in GRPO are peer-mean baseline that excludes the high-reward anchor, an extra term PGAA representing a detached cosine between generalization and anchor score gradients is added.

Empirical results on KnowEdit with Qwen3-8B / Llama3-8B as backbones show that EAGPO improves aspects such as Reliability / Subject-Replace over baseline methods including initial editing, SFT, and post-edit GRPO (EtCon). Ablation stduy provide some interesting findings regarding sequential WikiBigEdit sessions, theory on reliability, generalization stability, and gradient variance.

Critical strengths
  • The tackled problem is of importance: previous single-experission KE often fails under rephrasing and light compositional probes; the targeted generalization on sementic variants is important for broader applications of KE methods.

  • The proposed method EAGPO is generally intuitive and effective. The peer-mean baseline correctly targets a real GRPO artifact and the additional term further enforces generalization.

Critical weaknesses
  • The performance gain is overclaimed, as the claimed 25%–35% generalization lift is largely vs brittle Initial Editing, not vs the paper’s own strongest baselines. The gain shrinks rapidly when comparing with strong baselines like SFT.

  • The core novelty is under-justified, and theory does not adequately support the algorithm like the added term in GRPO.

    (1) Adding lambda * cos(g_j, g_1) into detached A_j is a strong heuristic that depends on which layer’s gradients are aligned; ablations show removing it largely returns performance toward GRPO, but that does not establish a principled mechanism. (2) Efficiency claims vs GRPO are also partly confounded by AGT’s one rollout per query rather than a matched compute comparison.

Questions for authors
  • what's the effect of lambda * cos term beyond filtering conflicted samples? an interesting baseline would be masking/downweighting samples with cos < 0. in this way the effect can be better explained
  • A matched compute baseline should be provided, e.g., regarding number of rollouts. And the performance gain in the abstract need to be adjusted accordingly

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.