- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the target-construction stage of locate-then-edit editing. The authors argue that, because of LayerNorm/RMSNorm, the gradient with respect to the target hidden state is nearly orthogonal to . Gradient descent therefore mostly rotates , and its norm grows as a side effect (median –). Since the weight update scales with , this extra norm hurts locality. The proposed Spherical Editing (SE) reparameterizes the target so that it stays on the sphere of radius . Built on FE, SE keeps efficacy unchanged and consistently improves KL/top-k locality with 10k edits on zsRE and CounterFact across three LLMs. It also helps MEMIT and AlphaEdit in sequential editing.
Critical strengths
- Simple and consistent. SE adds no parameters and only changes how the target is built. It improves locality in all six batch settings and also works on MEMIT and in sequential editing.
- Clear, checkable mechanism. The gradient statistics, the norm growth and the radial scan fit together well, and the story is easy to verify.
Critical weaknesses
- The key control is missing. Table 1 is almost fully explained by a shorter . With purely tangential growth, the expected reductions are about 25% / 41% / 28%, against the reported 24% / 41% / 27%. Existing knobs (the term, the clamp , early stopping) also shorten the residual. Without comparing at matched or matched efficacy, I cannot tell whether SE is a better trade-off or just stronger regularization.
- The theory is stated more strongly than it holds. Theorem 2 assumes plain GD on cross-entropy, but MEMIT-style optimization uses Adam with KL and weight-decay terms. "Magnitude contributes little" is also overstated: in Figures 7–8, the NLL at is several times higher than at , and success collapses below .
Questions for authors
- How does SE compare with the host re-tuned (stronger , smaller clamp, or early stopping) to the same mean or the same Success?
- Which optimizer and loss terms are used for target optimization, and how does Theorem 2 apply under them?
3 comments
Response to W2 & Q2 (Final Rescaling, Target NLL, and Adam Optimization)
Thank you for the valuable suggestion.
Weakness 2 (Figures 7–8). SE optimizes the target direction at the original radius throughout. Figures 7–8 examine a different operation: one-shot rescaling of an already optimized FE target, holding its direction fixed. Residual paths retain radial sensitivity (Remark, lines 193–197), so this post-hoc operation can raise target NLL. SE keeps optimization and delivery at the same radius. We clarify this distinction in revised Appendix A.3.
We compare these two ways of imposing the original radius using the optimization prompts of the first 300 edit requests in each setting. SE yields lower mean target NLL than final rescaling in all four reported model–dataset settings:
| Model and dataset | FE: final rescaling NLL ↓ | FE + SE NLL ↓ |
|---|---|---|
| GPT2-XL zsRE | 0.021 | 0.013 |
| GPT2-XL CounterFact | 0.017 | 0.014 |
| Qwen CounterFact | 0.074 | 0.025 |
| GLM CounterFact | 0.007 | 0.006 |
For the main-table configurations, SE also improves , Top-1 and Top-10 over final rescaling across all six settings.
Weakness 2 & Question 2 (Optimizer and loss). Theorem 2 describes tangential gradient descent. Following MEMIT and FE, we use Adam with . For GPT2-XL, the learning rate is 0.5, with at most 34 updates and loss < 0.05 stopping. The loss is NLL + 0.0625 KL + , with raw residual clamped to . FE uses ; SE uses the spherical target in Eq. 14. FE uses ; SE uses the setting and / 0.1 on zsRE / CounterFact. There is no optimizer weight decay, gradient clipping or learning-rate schedule (Appendix A.1).
Adam uses momentum and coordinatewise scaling. For FE's actual post-clamp target step , the norm change obeys
Theorem 2 is the tangential-GD case of this identity. Figure 4 shows norm growth under the actual Adam optimization; SE prevents this growth by keeping throughout.
I thank the authors for the new results and clarifications. Based on the method’s contribution I maintain my original assessment.
Response to W1 & Q1 (Matched-Clamp Comparisons and Re-tuned FE Controls)
Thank you very much for the detailed and constructive review and suggestions.
Tables 2–3 contain updated SE results and, following MEMIT, Gen. and Overall Score. Baselines use the tuned configurations from the official FE repository.
Weakness 1 & Question 1 (Matched and re-tuned FE). Theorem 3 explains the single-edit benefit of a shorter residual; the controls test whether ordinary FE tuning achieves the same effectiveness–locality combination.
(1) Matched clamp. GPT2-XL zsRE, with all other settings fixed (w=0.5, lr 0.5):
These are supplementary runs, separate from the main-table results below. SE improves efficacy and locality with comparable or higher Gen.
(2) Re-tuned FE. All controls modify FE on the same 10,000 GPT2-XL zsRE requests. Defaults: lr 0.5, residual penalty 0.5, radius penalty 0, clamp 0.75, lambda 15,000, at most 34 updates, loss < 0.05 stopping. SE uses the clamp setting labeled 0.65.
Here lr sets the target-optimization learning rate; step caps its updates. Residual penalty weights target displacement, ∥m−h∥/∥h∥2; radius penalty weights (∥m∥/∥h∥−1)2. Clamp bounds displacement relative to the original norm. Lambda weights preservation of non-target keys in the weight solve (default 15,000); update multiplies the final weight update by the listed factor. The stronger residual penalty (2.0) matches SE's Success but not its locality. Controls that reach comparable locality sacrifice editing efficacy.
(3) GPT2-XL CounterFact controls:
SE achieves better locality than these penalties; clamps and preservation weights that reach comparable locality reduce Success/Accuracy. With MEMIT's settings unchanged, SE also improves all six metrics in Table 6.