- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CFPP, a preservation-covariance modification for locate-then-edit methods. It attenuates preservation constraints along edit-relevant directions through whitening and adaptive soft projection, followed by trace normalization. CFPP integrates with existing editors and is evaluated across three models and three knowledge-editing benchmarks.
Critical strengths
The paper identifies a concrete geometric conflict between editing and preservation and proposes a reusable modification without additional trainable parameters. Experiments cover multiple editors, editing scales, and sequential settings, with consistent improvements in aggregate scores over the corresponding base editors.
Critical weaknesses
-
The theoretical analysis does not establish behavioral preservation. Appendix A.2 shows that projection preserves orthogonal key components, but this does not establish preservation of the original key–value mappings or bound the resulting preservation error, such as (|\Delta K_p|_F^2). Trace normalization also provides no such guarantee. The connection between the geometric argument and actual knowledge preservation therefore remains incomplete.
-
Evaluation of general capabilities and multi-hop reasoning is insufficient. The paper lacks general-capability benchmarks before and after editing; the generation metrics in Appendix D.6 provide limited evidence in this regard. Wiki-recent includes portability, but its coverage of multi-hop reasoning versus alias-based queries is unclear. A focused general-capability evaluation and an explicit multi-hop benchmark would better establish the practical value of the method.
-
The comparative results require clearer interpretation. AlphaEdit underperforms MEMIT in several settings, warranting clarification of baseline configurations and evaluation protocols. Although CFPP improves Avg. in all 18 paired comparisons in Table 3, specificity decreases throughout, and some gains are marginal, such as 73.55→73.62 for EAMET on Mistral/ZsRE.
Questions for authors
The same as weaknesses.
3 comments
Dear Reviewer,
Thank you again for your time and thoughtful reviews. Could you please share whether the authors’ response has addressed your main concerns, and whether any questions remain? A brief follow-up would be very helpful as I prepare the meta-review.
Please also ensure that your review reflects your current assessment in light of the authors’ response and the discussion.
Many thanks for your continued engagement.
Best regards,
Area Chair
Thank you for the clarification, and apologies for my delayed response. I appreciate the new perspective this work offers. However, the models evaluated in the experiments date from roughly two years ago, and it is unclear whether the findings still hold for current state-of-the-art models. My concerns therefore remain, and I will keep my original score.
We thank the reviewer for the constructive feedback. We have added experiments and clarified the scope of our preservation claims.
Answer1: We agree that the projection argument in Appendix A.2 leaves open how the subsequent weight update affects the original key–value mappings. We added Appendix A.3 to quantify this effect. For fixed original preservation keys (K_p) at a single edited layer, the squared change in their output values is (|\Delta K_p|_F^2). Equation (23) gives a direction-wise upper bound on this error in terms of the edit residual, the projection gates, and the trace calibration factor. It also follows from the update expression that a fixed key’s output is unchanged if its whitened representation is orthogonal to the full edit subspace.
With the keys, residual, and gates fixed, trace calibration strengthens the remaining preservation penalties and tightens the corresponding terms in the bound. Fully gated directions retain zero penalty. Compared with the variant without calibration, enabling calibration improves specificity by 0.59 and 0.45 percentage points on CounterFact and ZsRE, respectively, and locality by 0.90 percentage points on Wiki-recent. The bound characterizes local output drift for fixed keys at one layer. We assess preservation at the model level using specificity and locality.
Answer2: To address the concern about general capabilities, we added a focused evaluation in Table 14, comparing the unedited LLaMA3-8B model with MEMIT and MEMIT-CFPP after CounterFact editing across six benchmarks. Relative to MEMIT, CFPP performs better on MRPC, SST-2, and RTE, while scoring modestly lower on MMLU, CoLA, and NLI. Overall, the added evaluation shows that CFPP retains competitive general-task performance and improves upon MEMIT on several benchmarks.
We also added a direct multi-hop evaluation on the POPULAR split of RippleEdits (Table 8). Compared with MEMIT, CFPP improves logical generalization, both compositionality criteria, and subject aliasing, indicating stronger multi-hop and compositional editing. At the same time, its preservation and relation-specificity scores remain competitive, with only small decreases relative to MEMIT. Wiki-recent portability evaluates transfer to reasoning variants and alias-based prompts; RippleEdits provides a complementary, explicit assessment of multi-hop and compositional effects. These results show that CFPP strengthens multi-hop editing while maintaining competitive preservation and specificity.
Answer3: Table 3 reports means and standard deviations over three random seeds. The reviewer’s observation that ‘AlphaEdit underperforms MEMIT in several settings’ reflects our Table 3 setup, which evaluates massive editing rather than sequential editing. Under this setup, AlphaEdit performs below MEMIT in several settings, consistent with the large-scale editing results reported in EAMET: ROBUST MASSIVE MODEL EDITING VIA EMBEDDING ALIGNMENT OPTIMIZATION, whose experimental protocol is similar to ours.
Regarding the specificity decreases across the Table 3 pairings, we added a λ sweep comparing the efficacy–specificity frontiers of MEMIT and MEMIT-CFPP on LLaMA3-8B with 10,000 CounterFact edits. At the closest matched-specificity points, efficacy is nearly identical while generalization improves. At nearby points with lower but still comparable specificity, CFPP improves both efficacy and generalization. Across much of the tested range, CFPP improves efficacy and generalization but does not reach the base method’s highest specificity. At the smaller scale of 1,000 edits, the specificity cost is mitigated, and CFPP’s specificity can even slightly exceed that of the base method.
The blue text in the newly uploaded manuscript highlights the additional content we have added.