- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the sequential editing task. The authors seek to find, among the infinitely many feasible solutions that achieve the target edit, a parameter modification that is as good as possible. To this end, they propose Canon Edit. The method first employs a linear approximation in the parameter space to delineate the feasible region satisfying the editing objective, which serves as a constraint. It then constructs a second-order approximation of a cost function to measure the degree of disruption to previously stored knowledge. A correction mechanism is further introduced to refine the solution when the second-order approximation is inaccurate. This formulation transforms the final optimization task into a strictly convex optimization problem that admits a closed-form optimal update. Intuitively, the overall framework resembles a trust-region optimization process in which editing success is treated as a constraint while locality (preservation of unrelated knowledge) is treated as the optimization objective. The authors demonstrate through experiments that the proposed method indeed achieves strong performance.
Critical strengths
- The idea introduced in this paper is very interesting. The constrained optimization perspective of knowledge editing in this paper provides an elegant framework that is valuable for future work to reference and build upon.
- The experimental evaluation is thorough, extending the testing range to 10,000 sequential edits. At this scale, the proposed method indeed demonstrates strong and consistent performance.
Critical weaknesses
- Canon Edit directly formulates the minimization of disruption to unrelated knowledge as the sole optimization objective and solves for a unique solution under the editing constraint. However, real-world editing tasks involve additional important desiderata beyond faithfulness and locality; for example, improving the portability on the edited knowledge. If such objectives conflict with the current constraint, it is unclear how they can be reconciled. These additional objectives do not appear to lend themselves to simple second-order approximations and are thus difficult to integrate into the proposed framework. Moreover, the current closed-form solution architecture seemingly provides no mechanism to adjust the trade-off among different objectives.
- The core of Canon Edit is built upon numerous approximation steps. Given the highly complex and nonlinear parameter space of LLMs, the accumulated approximation error is inevitably substantial. This makes the method heavily dependent on the correction mechanism. The effectiveness and robustness of the overall approach may therefore be sensitive to how well the correction procedure compensates for these approximation errors.
Questions for authors
- The main experimental results in Table 1 appear somewhat unusual. For instance, why are the online efficacy and generalization metrics (Rel. & Gen.) for AlphaEdit so low (e.g., 25.17% Rel.), even performing below one-shot methods like ROME and MEMIT? Meanwhile, the Locality scores for these methods are all very high, which collectively gives the impression that the edits were not successfully applied in the first place. I would appreciate it if the authors could further clarify whether the baseline implementations and hyperparameter selection processes differ from their standard/official implementations, or explain other possible reasons that may account for this.
I would be happy to increase my score if the authors clarify my concerns.
4 comments
3. Baseline Fairness and the Unexpected AlphaEdit Results (Q1)
Thank you for carefully examining the AlphaEdit results in Table 1. Table 1 reports 25.17% online efficacy for AlphaEdit on CounterFact-plus/OneHop with GPT-J, motivating a closer examination of target construction and evaluation settings.
Table 1 evaluates the CounterFact-plus/OneHop request stream under the reported controlled sequential-editing protocol, including a fixed GPT-J editing layer and a teacher-forced target-versus-original likelihood criterion. AlphaEdit obtains 25.17% online efficacy, while AlphaEdit with the Forward Replay target achieves 88.32% on that same stream.
The substantial efficacy increase with the Forward Replay target highlights the importance of target construction under the controlled sequential protocol. The comparison specifies the editing layer, evaluation criterion, and request stream to support reproducibility.
We traced the higher AlphaEdit+FR efficacy in Table 15 (93.56%) to the distinct CounterFact/MCF request stream. Both tables now identify their exact streams, accounting for the different reported values. The 25.17% AlphaEdit result is measured under the specified OneHop protocol.
Our shared-pool experiment additionally compares selection rules using identical parent weights, verified candidates, and evaluation conditions. CanonEdit achieves the lowest held-out disturbance (0.2504) across all four selectors, with strictly positive paired 95% confidence intervals against each alternative.
These matched-condition results establish the independent contribution of canonical representative selection and complement the sequential baseline comparisons.
We sincerely appreciate the reviewer's constructive feedback and encouragement. The theoretical clarifications and additional experiments further demonstrate how CanonEdit combines constrained optimization, informative local protection geometry, and efficient nonlinear verification to identify low-interference successful updates. We hope these additional results satisfactorily address your concerns and provide sufficient grounds for reconsidering your initial rating.
Follow-up on Rebuttal and Further Discussion
Dear Reviewer wxQY,
Thank you again for your encouraging feedback and helpful suggestions. In our rebuttal, we clarified how CanonEdit can support multiple editing objectives, added experiments on nonlinear refinement, and explained the AlphaEdit results and evaluation settings.
As the ICLR discussion period is approaching its end, we would sincerely appreciate your thoughts on whether these responses address your concerns. We would be happy to discuss any remaining questions.
Thank you very much for your time and consideration.
CanonEdit (#5375): Request for Response from Reviewer wxQY
Dear Reviewer wxQY,
Thank you again for your encouraging comments and for expressing your willingness to reconsider your assessment.
As the discussion period is nearing its end, we sincerely hope to hear your thoughts on our rebuttal, particularly our clarifications of the AlphaEdit baseline results and the additional experiments on nonlinear refinement.
We would greatly appreciate it if you could review these responses and let us know whether your main concerns have been addressed.
Thank you very much for your time and consideration.
Response to Reviewer wxQY
We thank the reviewer for recognizing the value of CanonEdit's constrained optimization perspective and its consistent performance over 10,000 sequential edits. Your comments raise important questions about objective flexibility, approximation reliability, and baseline evaluation. We address these concerns below.
Thank you for highlighting portability and the need to balance multiple editing objectives. CanonEdit provides a natural mechanism for incorporating these requirements by separating behavioral constraints from the protected-cost objective.
Additional differentiable requirements, such as portability across related queries, can be expressed as target-margin constraints:
minz21z⊤Bzs.t.Aeditz≥bedit,Aportz≥bport.
When the local feasible set is nonempty, the resulting convex quadratic program retains the same optimization structure. Nonlinear verification subsequently evaluates the realized editing and portability behavior.
For multiple protection objectives, CanonEdit also admits a weighted local metric:
B=∑iλiGi+αM,λi≥0,
where each Gi represents a locally quadratic protection criterion. The weights λi control their relative importance, while the target margins γ and protected-disturbance budget ϵ regulate feasibility and acceptance.
These extensions clarify how CanonEdit accommodates additional behavioral requirements and explicit objective trade-offs. They preserve the local convex formulation when the corresponding constraints and cost approximations satisfy its assumptions. The reported experiments establish the original faithfulness–locality setting, providing the foundation for these broader formulations.
We appreciate your concern regarding approximation errors in nonlinear LLMs. CanonEdit addresses this through a combination of local geometric estimation, full-model verification, and targeted re-linearization. We provide additional empirical evidence for each component.
First, the original GPT-J mechanism analysis (Fig. 3(c)) reports a 0.74 log-scale correlation (95% bootstrap CI: [0.65, 0.82]) between predicted protected energy and measured disturbance. Our new shared-pool experiment further evaluates generalization of the protected-cost ranking: CanonEdit achieves a held-out static MAE of 0.2504, compared with 0.4105, 0.3493, and 0.3482 for max-margin, first-valid, and random-valid selection.
Second, nonlinear verification directly checks candidate feasibility in the original model. In the 10K CounterFact–GPT-J experiment, 93.90% of requests follow the closed-form path, 6.13% trigger re-linearization, and the average number of full candidate evaluations is 1.40. These measurements characterize how frequently the method requires nonlinear correction.
To further examine robustness, we conducted a controlled GPT-2-XL CounterFact experiment (N=600), varying the maximum number of refinement steps K:
Increasing the refinement budget from K=0 to K=4 improves online target success by 3.84 percentage points and reduces failed writes from 65 to 44. Meanwhile, K=2 achieves the lowest protected drift, illustrating the interaction between correction strength, interference, and computational cost.
Together, these results establish that the local protected-cost approximation provides informative candidate rankings, while nonlinear refinement offers a measurable and controllable mechanism for recovering target feasibility. The limited re-linearization frequency in the original 10K experiment further supports the practical efficiency of this design.