acceptodds
Withdrawn from ICLR 2027

PACE: Pre-Write Exposure in Continual Knowledge Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual knowledge editing has a blind spot: editors decide where and how to write a new fact but typically do not estimate which prior mappings that writing may expose. We formulate this problem as pre-write exposure and study sparse edit-conditioned feed-forward-network support as a prospective exposure signal. This leads to PACE, a bounded-state editor that extracts one activation path before each write and reuses it to coordinate localization, risk-aware replay, sparse solving, and preservation. Across GPT-J, LLaMA-3, and routed Qwen3, support overlap predicts interference more strongly than prompt semantics, while controlled interventions that change overlap under closely matched realized write strength produce corresponding changes in prior-edit damage. Under a fixed replay budget, better exposure ranking is strongly associated with lower forgetting, and the same ranking signal transfers to other editors. At reliability matched to UltraEdit, PACE improves locality by – points and reduces forgetting by – points on GPT-J and LLaMA-3. These results suggest a broader principle for continual editing: model which prior mappings a write may expose before deciding how to apply it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual knowledge editing, pre-write exposure, model editing, lifelong knowledge editing, interference control, large language models, sparse editing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ghGfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PACE estimates interference before a knowledge edit using overlap between sparse FFN activation supports. The same support state guides localization, replay, sparse least-squares updates, and preservation. Experiments across dense and routed models examine interference prediction and retention under matched reliability. The theoretical guarantees concern fixed, represented FFN mappings.

Critical strengths

The paper addresses an important problem and tests its risk signal through several useful controls. Matching realized write strength in the support intervention helps rule out weaker edits as the explanation for reduced interference. Transfer to other editors further supports the signal's usefulness.

Table 1 reports locality gains of 1.9-2.0 percentage points and reductions in forgetting of 1.2-1.7 points relative to UltraEdit at matched reliability. Development-based operating-point selection and five matched stream orders make these improvements worth taking seriously.

Critical weaknesses
  • The ranking path and the write support are not the same object. Jaccard uses the full active-layer path, but only boundary layers are materialized; Appendix A.6 explicitly gives no guarantee for interior layers. I could not find a boundary-restricted ranking ablation, so it is unclear whether the signal predicts the write or just representational similarity.
  • The intervention matches target gain and update norm but not the exposed-key energy ∥κi,t∥\|\kappa_{i,t}\|. Proposition 1 says damage depends on both overlap and key energy, so forced-high support may simply include higher-energy coordinates. The dose-response could be a key-energy effect.
  • The intervention's outcome is margin loss, not answer correctness. A prior edit can lose margin and still be answered correctly. The causal claim should be tied to flip rate or accuracy.
  • The guarantee stops at eviction, and the cache holds 64 edits while the paper reports full-history For. Please separate cache-resident and evicted retention; otherwise the "lifelong" gain may be mostly sparsity.
  • The comparative advantage is still thin. MIR already predicts which old examples will be damaged, and NSE/NMKE already do activation-based sparse editing; PACE's bounded implementation and shared state are distinct, but the comparisons do not isolate their benefit. Against LightEdit, PACE is 85.6 vs 92.5 Gen at 10K ZsRE with only a 0.2 For. edge, and at 50K LightEdit leads all four quality metrics.
  • Rel. and For. need consistent definitions: 93.4 + 7.2 = 100.6 is impossible if they share the same population, and Table 77's 116 observations exceed the 64-edit cache. These affect the matched-reliability claim and the drift analysis.
  • The method has 13 core hyperparameters: 5 path, 4 replay, 3 solver, and 1 preservation. Table 18 tests 11 of them (10 core plus deditd_{edit}), leaving τcoord\tau_{coord}, λ\lambda, and γr\gamma_r without a reported sweep. That is a lot of interacting knobs to tune, and the paper does not show that the defaults are robust enough to make the full design easy to reproduce.
Questions for authors
  1. Which results isolate the benefit beyond MIR-style replay and NSE/NMKE-style sparse editing? Could comparable experiments report both retention and selection cost?

  2. Does Eq. B1 define For. in Tables 1 and 5, and when and over which queries is Rel. measured? If Rel. is final accuracy on the same population, its sum with For. cannot exceed 100, whereas GPT-J UltraEdit gives 93.4 + 7.2 = 100.6. Please clarify.

  3. Table 77 contains 116 observations despite a replay capacity of 64 edits. Are these edit-layer observations, and do the statistical analyses account for observations sharing an edit?

  4. Under which editing and serving budgets does PACE offer a favorable quality-cost trade-off against LightEdit? Do the 50K deficit and 100K crossover in Table 40 persist across matched stream orders?

  5. How were defaults other than the matched margin selected, and were tuning budgets comparable? In particular, τcoord\tau_{coord}, λ\lambda, and γr\gamma_r are not in Table 18; please justify their fixed values or report their sensitivity. Does the Jaccard-versus-learned-ranker advantage persist across matched stream orders?

  6. Please provide the source files or plotting code for Figures 1 and 3, and clarify whether generative image tools were used. The AI use statement on p. 9 mentions language editing and presentation feedback but not figure generation; please reconcile the two.

  7. Please compare boundary-restricted Jaccard with full-path Jaccard while keeping the write fixed, and report ∥κi,t∥\|\kappa_{i,t}\| for each intervention arm. If key energy co-varies with overlap, the intervention does not isolate overlap.

  8. Please report answer-level correctness or flip rate for the prior edit in each intervention arm, not only margin loss.

  9. Please report For. by cache membership and edit age, and measure path/calibration drift on the dense backbones on a non-repeated stream.

  10. Please report full reliability–forgetting Pareto curves, and compare path-overlap with semantic, lexical, gradient, and learned rankers inside the cross-editor wrapper, not only against random.

Reviewer_P25tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

PACE proposes a continual knowledge editing framework that extracts one prewrite activation path and reuses it for localization, replay ranking, sparse solving, and preservation. The central claim is that prewrite exposure, estimated by path overlap with retained edits, is a useful signal for controlling cross edit interference. The paper supports this with predictive controls, support intervention, replay ranking controls, cross editor transfer, and matched reliability experiments.

Critical strengths

The mechanism controls include prospective prediction, forced overlap intervention, fixed budget ranking, and state decoupling. These controls are stronger than a single leaderboard comparison.

Critical weaknesses

The paper is difficult to follow and poorly organized. The core method feels like a patchwork of existing techniques, and the writing reads as if it was generated by an AI without sufficient human editing.

In addition, the main figure is not placed in the main text, and the formulas and terminology are not grounded in the standard model editing literature but instead feel invented or repurposed. The main experiments include very few baseline comparisons, and the theoretical proofs in the appendix do not support the core theoretical claims.

Questions for authors

Can the authors add more relevant baselines directly to the main experiments in the main text?

Reviewer_dcyZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to make a sequence of factual edits to a model w/o affecting previous edits. It proposes PACE, which uses activation overlap to identify earlier edits that may be affected and change how new updates are done. The authors test their method across several models and benchmarks.

Critical strengths

None

Critical weaknesses

The paper is written in such a way that it makes it almost impossible for me to understand and follow. This does not make me confident at all in evaluating the claims and experiments in the paper, and, for me, is reason enough to justify my low score. I tried very hard, and multiple times, to understand the paper, but just cannot do so. The writing and flow is incomprehensible to a human, and I suspect the paper is written wholly with AI. I encourage the authors to write the paper in a clear, comprehensible manner so that readers can understand it.

Questions for authors

I encourage the authors to write the paper in a clear, comprehensible manner so that readers can understand it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.