- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PACE estimates interference before a knowledge edit using overlap between sparse FFN activation supports. The same support state guides localization, replay, sparse least-squares updates, and preservation. Experiments across dense and routed models examine interference prediction and retention under matched reliability. The theoretical guarantees concern fixed, represented FFN mappings.
Critical strengths
The paper addresses an important problem and tests its risk signal through several useful controls. Matching realized write strength in the support intervention helps rule out weaker edits as the explanation for reduced interference. Transfer to other editors further supports the signal's usefulness.
Table 1 reports locality gains of 1.9-2.0 percentage points and reductions in forgetting of 1.2-1.7 points relative to UltraEdit at matched reliability. Development-based operating-point selection and five matched stream orders make these improvements worth taking seriously.
Critical weaknesses
- The ranking path and the write support are not the same object. Jaccard uses the full active-layer path, but only boundary layers are materialized; Appendix A.6 explicitly gives no guarantee for interior layers. I could not find a boundary-restricted ranking ablation, so it is unclear whether the signal predicts the write or just representational similarity.
- The intervention matches target gain and update norm but not the exposed-key energy . Proposition 1 says damage depends on both overlap and key energy, so forced-high support may simply include higher-energy coordinates. The dose-response could be a key-energy effect.
- The intervention's outcome is margin loss, not answer correctness. A prior edit can lose margin and still be answered correctly. The causal claim should be tied to flip rate or accuracy.
- The guarantee stops at eviction, and the cache holds 64 edits while the paper reports full-history For. Please separate cache-resident and evicted retention; otherwise the "lifelong" gain may be mostly sparsity.
- The comparative advantage is still thin. MIR already predicts which old examples will be damaged, and NSE/NMKE already do activation-based sparse editing; PACE's bounded implementation and shared state are distinct, but the comparisons do not isolate their benefit. Against LightEdit, PACE is 85.6 vs 92.5 Gen at 10K ZsRE with only a 0.2 For. edge, and at 50K LightEdit leads all four quality metrics.
- Rel. and For. need consistent definitions: 93.4 + 7.2 = 100.6 is impossible if they share the same population, and Table 77's 116 observations exceed the 64-edit cache. These affect the matched-reliability claim and the drift analysis.
- The method has 13 core hyperparameters: 5 path, 4 replay, 3 solver, and 1 preservation. Table 18 tests 11 of them (10 core plus ), leaving , , and without a reported sweep. That is a lot of interacting knobs to tune, and the paper does not show that the defaults are robust enough to make the full design easy to reproduce.
Questions for authors
-
Which results isolate the benefit beyond MIR-style replay and NSE/NMKE-style sparse editing? Could comparable experiments report both retention and selection cost?
-
Does Eq. B1 define For. in Tables 1 and 5, and when and over which queries is Rel. measured? If Rel. is final accuracy on the same population, its sum with For. cannot exceed 100, whereas GPT-J UltraEdit gives 93.4 + 7.2 = 100.6. Please clarify.
-
Table 77 contains 116 observations despite a replay capacity of 64 edits. Are these edit-layer observations, and do the statistical analyses account for observations sharing an edit?
-
Under which editing and serving budgets does PACE offer a favorable quality-cost trade-off against LightEdit? Do the 50K deficit and 100K crossover in Table 40 persist across matched stream orders?
-
How were defaults other than the matched margin selected, and were tuning budgets comparable? In particular, , , and are not in Table 18; please justify their fixed values or report their sensitivity. Does the Jaccard-versus-learned-ranker advantage persist across matched stream orders?
-
Please provide the source files or plotting code for Figures 1 and 3, and clarify whether generative image tools were used. The AI use statement on p. 9 mentions language editing and presentation feedback but not figure generation; please reconcile the two.
-
Please compare boundary-restricted Jaccard with full-path Jaccard while keeping the write fixed, and report for each intervention arm. If key energy co-varies with overlap, the intervention does not isolate overlap.
-
Please report answer-level correctness or flip rate for the prior edit in each intervention arm, not only margin loss.
-
Please report For. by cache membership and edit age, and measure path/calibration drift on the dense backbones on a non-repeated stream.
-
Please report full reliability–forgetting Pareto curves, and compare path-overlap with semantic, lexical, gradient, and learned rankers inside the cross-editor wrapper, not only against random.