acceptodds
Under review as a conference paper at ICLR 2027

Revisiting the Dynamics of Target Optimization in Large Language Model Editing

Abstract

Knowledge editing has emerged as a promising approach for efficiently updating knowledge in large language models (LLMs). These methods first optimize a target hidden state that steers the model toward a new output, and then edit the model parameters to map the original hidden state to this target. While most existing methods focus on the second stage of parameter editing to preserve other knowledge, we find that the geometry of upstream target hidden-state optimization is also critical to editing locality. For LLMs equipped with normalization layers such as LayerNorm or RMSNorm, we find that changes in the direction of the target hidden state are essential for injecting new knowledge, whereas the magnitude contributes little. However, we also find that the norm of the target hidden state continuously accumulates during gradient descent optimization. We further show that this unnecessary norm growth increases interference with non-target knowledge, thereby degrading editing locality. Building on this insight, we propose Spherical Editing (SE), a simple yet effective method that constrains the target hidden state to remain on the sphere with the same radius as the original state throughout optimization. By eliminating unnecessary radial growth while retaining essential directional changes, SE reduces interference with other knowledge without sacrificing editing success. Extensive experiments on three LLMs, two benchmarks, and 10,000 edits demonstrate that SE consistently improves editing locality while maintaining editing effectiveness. Code is available at https://anonymous.4open.science/r/se-review-artifact-5869/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model editing, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iXraOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the target-construction stage of locate-then-edit editing. The authors argue that, because of LayerNorm/RMSNorm, the gradient with respect to the target hidden state mm is nearly orthogonal to mm. Gradient descent therefore mostly rotates mm, and its norm grows as a side effect (median ∥m∥/∥h∥=1.21\|m\|/\|h\| = 1.21–1.411.41). Since the weight update scales with ∥m−h∥\|m-h\|, this extra norm hurts locality. The proposed Spherical Editing (SE) reparameterizes the target so that it stays on the sphere of radius ∥h∥\|h\|. Built on FE, SE keeps efficacy unchanged and consistently improves KL/top-k locality with 10k edits on zsRE and CounterFact across three LLMs. It also helps MEMIT and AlphaEdit in sequential editing.

Critical strengths
  • Simple and consistent. SE adds no parameters and only changes how the target is built. It improves locality in all six batch settings and also works on MEMIT and in sequential editing.
  • Clear, checkable mechanism. The gradient statistics, the norm growth and the radial scan fit together well, and the story is easy to verify.
Critical weaknesses
  • The key control is missing. Table 1 is almost fully explained by a shorter ∥m−h∥\|m-h\|. With purely tangential growth, the expected reductions are about 25% / 41% / 28%, against the reported 24% / 41% / 27%. Existing knobs (the ℓ2\ell_2 term, the clamp c=0.75c=0.75, early stopping) also shorten the residual. Without comparing at matched ∥m−h∥\|m-h\| or matched efficacy, I cannot tell whether SE is a better trade-off or just stronger regularization.
  • The theory is stated more strongly than it holds. Theorem 2 assumes plain GD on cross-entropy, but MEMIT-style optimization uses Adam with KL and weight-decay terms. "Magnitude contributes little" is also overstated: in Figures 7–8, the NLL at ∥h∥\|h\| is several times higher than at ∥m∥\|m\|, and success collapses below ∥h∥\|h\|.
Questions for authors
  1. How does SE compare with the host re-tuned (stronger ℓ2\ell_2, smaller clamp, or early stopping) to the same mean ∥m−h∥\|m-h\| or the same Success?
  2. Which optimizer and loss terms are used for target optimization, and how does Theorem 2 apply under them?
3 comments
AuthorsOpenReview ↗

Response to W1 & Q1 (Matched-Clamp Comparisons and Re-tuned FE Controls)

Thank you very much for the detailed and constructive review and suggestions.

Tables 2–3 contain updated SE results and, following MEMIT, Gen. and Overall Score. Baselines use the tuned configurations from the official FE repository.

Weakness 1 & Question 1 (Matched and re-tuned FE). Theorem 3 explains the single-edit benefit of a shorter residual; the controls test whether ordinary FE tuning achieves the same effectiveness–locality combination.

(1) Matched clamp. GPT2-XL zsRE, with all other settings fixed (w=0.5w=0.5, lr 0.5):

ClampMethodSuccess ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑
0.55FE72.6252.2563.351.00874.86
FE + SE73.9854.0364.310.86277.73
0.60FE74.1554.5465.071.21169.85
FE + SE75.1755.7365.830.97275.01
0.70FE75.5056.1266.781.45764.80
FE + SE76.5457.3367.511.27166.17
0.75FE76.0056.5967.541.58960.92
FE + SE76.6857.3867.461.36363.17

These are supplementary runs, separate from the main-table results below. SE improves efficacy and locality with comparable or higher Gen.

(2) Re-tuned FE. All controls modify FE on the same 10,000 GPT2-XL zsRE requests. Defaults: lr 0.5, residual penalty 0.5, radius penalty 0, clamp 0.75, lambda 15,000, at most 34 updates, loss < 0.05 stopping. SE uses the clamp setting labeled 0.65.

Method and settingSuccess ↑Accuracy ↑DKL ↓Top-1 ↑Top-10 ↑
lr 0.5 (default)76.0056.621.58360.8632.74
lr 0.2575.5256.341.63260.1632.35
lr 0.12575.3355.701.77356.9330.41
residual penalty 0.175.6055.931.60260.6532.37
residual penalty 0.2575.8156.011.59960.5732.41
residual penalty 1.075.9856.151.56461.3333.04
residual penalty 2.076.0156.271.54762.0933.71
radius penalty 2.575.5455.941.36364.2736.81
radius penalty 2574.9954.771.37160.4135.96
radius penalty 25075.8356.181.47657.7934.72
clamp 0.5572.6252.251.00874.8644.82
clamp 0.6574.9855.231.34166.9737.51
clamp 0.7075.5056.121.45764.8035.31
step 446.4418.950.39396.0659.98
step 862.8039.161.06677.6538.40
step 1067.8145.841.25971.7135.83
step 1574.1854.131.56662.3732.75
step 2075.6655.951.58361.6432.79
final rescaling75.5155.731.17768.9438.81
lambda 20,50075.1255.791.55562.2033.41
lambda 60,00064.5143.471.39063.2837.46
update ×0.9574.3854.411.36467.2036.47
update ×0.971.9451.301.13772.7140.83
update ×0.875970.3349.121.03575.1142.98
FE + SE76.0256.731.14269.5739.29

Here lr sets the target-optimization learning rate; step caps its updates. Residual penalty weights target displacement, ∥m−h∥/∥h∥2\lVert m-h\rVert/\lVert h\rVert^2; radius penalty weights (∥m∥/∥h∥−1)2(\lVert m\rVert/\lVert h\rVert-1)^2. Clamp bounds displacement relative to the original norm. Lambda weights preservation of non-target keys in the weight solve (default 15,000); update multiplies the final weight update by the listed factor. The stronger residual penalty (2.0) matches SE's Success but not its locality. Controls that reach comparable locality sacrifice editing efficacy.

(3) GPT2-XL CounterFact controls:

Method and settingSuccess ↑Accuracy ↑DKL ↓Top-1 ↑Top-10 ↑
FE93.3075.171.32453.2848.42
FE: residual penalty 1.093.1174.701.32653.2048.52
FE: residual penalty 2.092.8474.771.33353.1048.55
FE: clamp 0.7092.5173.901.25154.3849.80
FE: clamp 0.6090.2870.611.09856.7952.66
FE: clamp 0.5588.9968.361.01058.3454.30
FE: λ=30,000\lambda=30{,}00089.1564.831.23554.2050.68
FE: λ=60,000\lambda=60{,}00079.6747.381.11056.2153.12
FE + SE93.3175.271.20955.3150.23

SE achieves better locality than these penalties; clamps and preservation weights that reach comparable locality reduce Success/Accuracy. With MEMIT's settings unchanged, SE also improves all six metrics in Table 6.

AuthorsOpenReview ↗

Response to W2 & Q2 (Final Rescaling, Target NLL, and Adam Optimization)

Thank you for the valuable suggestion.

Weakness 2 (Figures 7–8). SE optimizes the target direction at the original radius throughout. Figures 7–8 examine a different operation: one-shot rescaling of an already optimized FE target, holding its direction fixed. Residual paths retain radial sensitivity (Remark, lines 193–197), so this post-hoc operation can raise target NLL. SE keeps optimization and delivery at the same radius. We clarify this distinction in revised Appendix A.3.

We compare these two ways of imposing the original radius using the optimization prompts of the first 300 edit requests in each setting. SE yields lower mean target NLL than final rescaling in all four reported model–dataset settings:

Model and datasetFE: final rescaling NLL ↓FE + SE NLL ↓
GPT2-XL zsRE0.0210.013
GPT2-XL CounterFact0.0170.014
Qwen CounterFact0.0740.025
GLM CounterFact0.0070.006

For the main-table configurations, SE also improves DKLD_{KL}, Top-1 and Top-10 over final rescaling across all six settings.

Weakness 2 & Question 2 (Optimizer and loss). Theorem 2 describes tangential gradient descent. Following MEMIT and FE, we use Adam with β=(0.9,0.999)\beta=(0.9,0.999). For GPT2-XL, the learning rate is 0.5, with at most 34 updates and loss < 0.05 stopping. The loss is NLL + 0.0625 KL + w∥r∥/∥h∥2w\lVert r\rVert/\lVert h\rVert^2, with raw residual rr clamped to ∥r∥≤c∥h∥\lVert r\rVert\le c\lVert h\rVert. FE uses m=h+rm=h+r; SE uses the spherical target mrotm_{\mathrm{rot}} in Eq. 14. FE uses (w,c)=(0.5,0.75)(w,c)=(0.5,0.75); SE uses the c=0.65c=0.65 setting and w=0.5w=0.5 / 0.1 on zsRE / CounterFact. There is no optimizer weight decay, gradient clipping or learning-rate schedule (Appendix A.1).

Adam uses momentum and coordinatewise scaling. For FE's actual post-clamp target step ut=mt+1−mtu_t=m_{t+1}-m_t, the norm change obeys

∥mt+1∥2−∥mt∥2=2⟨mt,ut⟩+∥ut∥2.\lVert m_{t+1}\rVert^2-\lVert m_t\rVert^2=2\langle m_t,u_t\rangle+\lVert u_t\rVert^2.

Theorem 2 is the tangential-GD case of this identity. Figure 4 shows norm growth under the actual Adam optimization; SE prevents this growth by keeping ∥mrot∥=∥h∥\lVert m_{\mathrm{rot}}\rVert=\lVert h\rVert throughout.

I thank the authors for the new results and clarifications. Based on the method’s contribution I maintain my original assessment.

Reviewer_4KUTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies target hidden-state optimization in locate-then-edit knowledge editing. It argues that target direction is important for installing an edit, whereas norm growth during optimization is largely unnecessary and can increase interference with unrelated knowledge. The proposed Spherical Editing (SE) uses a differentiable reparameterization to keep the post-residual target on the sphere with the original hidden-state norm. The main experiments integrate SE into FE and evaluate 10,000 simultaneous edits on three models and two datasets, with additional scaling, MEMIT, and sequential-editing experiments.

Critical strengths

Tables 2 and 3 show that SE improves all reported locality metrics over FE in all six model/dataset combinations while maintaining comparable target performance. For example, on Qwen2.5-7B-Instruct/CounterFact, KL divergence decreases from 1.257 to 0.925 while generation accuracy increases slightly from 96.71 to 96.96. The MEMIT experiment in Table 6 and the sequential experiments with FE, MEMIT, and AlphaEdit in Table 4 provide useful evidence that the modification can transfer across editing procedures. This practical consistency is the strongest argument for acceptance.

Critical weaknesses

W1. The theoretical results support a narrower explanation than the manuscript's main claims, particularly for the actual batch-editing setting.

Theorem 1 is correct under scale invariance, but normalization alone does not establish that assumption for the complete downstream network. Appendix A.7, Equation 30 explicitly includes an identity-path contribution to the gradient; only the normalized-branch contribution is guaranteed to be tangential. Table 5 supports approximate orthogonality empirically, but does not establish that residual-path radial contributions are necessarily small in general. Likewise, Theorem 2 assumes exactly tangential gradients and ordinary gradient descent. Equation 35 shows that a small nonzero radial gradient can change the sign of norm growth depending on the step size. The relationship to the actual optimizer and its update direction should therefore be established explicitly.

More consequentially, Theorem 3 concerns a single rank-one edit and hidden-state perturbation. The main experiments instead use 10,000 simultaneous edits and output-distribution locality metrics. In Equation 5, the residual columns interact through the shared linear solve; reducing individual residual norms does not generally guarantee a smaller combined perturbation because cross-edit contributions can cancel or reinforce each other. Nor does a smaller hidden-state perturbation automatically imply a smaller output KL divergence. These distinctions limit the explanatory reach of the theorem, even though the empirical results remain useful.

W2. The experiments do not isolate the benefit of spherical optimization from simpler final rescaling or ordinary update-size control.

The most direct missing comparison is FE followed by a single normalization of the final optimized target to the original hidden-state norm. Table 1 measures representation perturbation after such rescaling, while Appendix A.3 measures activation-steering success on the same prompts used for target optimization, without updating model parameters. Neither reports the end-to-end editing effectiveness and behavioral locality of this simpler alternative alongside SE.

This distinction matters because Equation 14 changes the optimization trajectory and effective gradient scale as well as the final target norm. A comparison with FE using matched residual or parameter-update norms, or appropriately tuned target regularization, is also needed to distinguish a specifically useful spherical constraint from the general effect of making smaller updates. These comparisons should use the same edit sets and optimization budgets and should compare locality at similar edit effectiveness. Without them, the reported results establish that SE improves the chosen FE configuration, but leave the added value of constraining the target during optimization and the source of the improvement unresolved.

Questions for authors
  1. Mechanism control: How do FE, FE with final-only target normalization, and SE compare on end-to-end target accuracy and locality? How do the results change when FE is tuned to match SE's residual or parameter-update norm at comparable edit effectiveness?

  2. Optimization clarification: Which optimizer, learning rate, stopping rule, regularizers, and clipping or clamping operations are used for target construction? Do the actual optimizer updates, rather than only the prediction-loss gradients, exhibit the geometry assumed in Theorem 2?

  3. Theoretical scope: Is Theorem 3 intended only as a single-edit surrogate analysis? If so, please narrow the corresponding claims. If it is intended to explain the batch results more generally, what assumptions or measurements account for cross-edit interactions in Equation 5?

  4. Generalization clarification: Which prompt types are included in the reported target Success and Accuracy? Since Appendix A.3 evaluates the optimization prompts, can the authors provide a separate comparison on held-out paraphrases for FE, final-only normalization, and SE?

These questions concern the interpretation and adequacy of the current evidence and could materially change my assessment.

2 comments
AuthorsOpenReview ↗

Response to W2 & Q1 (Final Rescaling, Matched-Clamp Comparisons, and Update-Size Controls)

Thank you very much for the detailed and constructive review and suggestions.

Tables 2–3 contain updated SE results and, following MEMIT, Gen. and Overall Score. Baselines use the tuned configurations from the official FE repository.

Weakness 2 & Question 1 (Final rescaling and update-size controls).

(1) Final rescaling. Final rescaling is a simplified SE variant. Residual paths retain radial sensitivity (Remark, lines 193–197), so different optimization and delivery radii can raise loss. SE optimizes direction at the original radius throughout. Main settings, 10k edits:

DatasetModelTarget constructionSuccess ↑Accuracy ↑DKL ↓Top-1 ↑Top-10 ↑
zsREGPT2-XLFE76.0056.621.58360.8632.74
FE: final rescaling75.5155.731.17768.9438.81
FE + SE76.0256.731.14269.5739.29
QwenFE91.9476.794.18415.8531.62
FE: final rescaling87.3869.064.10016.0234.11
FE + SE93.5180.273.22325.4347.49
GLMFE86.0267.881.18554.1259.39
FE: final rescaling84.5064.631.19254.8459.59
FE + SE86.4367.911.03354.9160.47
CounterFactGPT2-XLFE93.3075.171.32453.2848.42
FE: final rescaling93.7375.441.22754.8749.93
FE + SE93.3175.271.20955.3150.23
QwenFE99.4096.711.25755.5756.15
FE: final rescaling99.0494.331.10958.0558.58
FE + SE99.3696.810.84563.5262.39
GLMFE99.2897.530.94063.9961.44
FE: final rescaling99.1697.260.86565.0763.00
FE + SE99.2797.670.85166.3564.11

Qwen zsRE uses clamp 1.0 for FE/final rescaling and 0.60 for SE.

Supplementary matched settings: GPT2-XL zsRE, 10k edits; both use the clamp setting labeled 0.65, residual penalty 0.5 and lr 0.5:

MethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑Top-10 ↑
FE + final rescaling69.9972.4652.5766.311.01171.5240.4140.93
FE + SE70.7476.0256.7367.211.14269.5739.0139.29

SE has higher efficacy, Gen. and Overall Score; final rescaling has better locality.

(2) Matched clamp. All other settings are fixed (GPT2-XL zsRE; w=0.5w=0.5, lr 0.5):

ClampMethodSuccess ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑
0.55FE72.6252.2563.351.00874.86
FE + SE73.9854.0364.310.86277.73
0.60FE74.1554.5465.071.21169.85
FE + SE75.1755.7365.830.97275.01
0.70FE75.5056.1266.781.45764.80
FE + SE76.5457.3367.511.27166.17
0.75FE76.0056.5967.541.58960.92
FE + SE76.6857.3867.461.36363.17

These are supplementary runs, separate from the main-table results.

(3) Re-tuned FE and update scaling. All controls modify FE on the same 10,000 GPT2-XL zsRE requests. Defaults: lr 0.5, residual penalty 0.5, radius penalty 0, clamp 0.75, lambda 15,000, at most 34 updates, loss < 0.05 stopping. SE uses the clamp setting labeled 0.65.

Method and settingSuccess ↑Accuracy ↑DKL ↓Top-1 ↑Top-10 ↑
lr 0.5 (default)76.0056.621.58360.8632.74
lr 0.2575.5256.341.63260.1632.35
lr 0.12575.3355.701.77356.9330.41
residual penalty 0.175.6055.931.60260.6532.37
residual penalty 0.2575.8156.011.59960.5732.41
residual penalty 1.075.9856.151.56461.3333.04
residual penalty 2.076.0156.271.54762.0933.71
radius penalty 2.575.5455.941.36364.2736.81
radius penalty 2574.9954.771.37160.4135.96
radius penalty 25075.8356.181.47657.7934.72
clamp 0.5572.6252.251.00874.8644.82
clamp 0.6574.9855.231.34166.9737.51
clamp 0.7075.5056.121.45764.8035.31
step 446.4418.950.39396.0659.98
step 862.8039.161.06677.6538.40
step 1067.8145.841.25971.7135.83
step 1574.1854.131.56662.3732.75
step 2075.6655.951.58361.6432.79
final rescaling75.5155.731.17768.9438.81
lambda 20,50075.1255.791.55562.2033.41
lambda 60,00064.5143.471.39063.2837.46
update ×0.9574.3854.411.36467.2036.47
update ×0.971.9451.301.13772.7140.83
update ×0.875970.3349.121.03575.1142.98
FE + SE76.0256.731.14269.5739.29

Here lr sets the target-optimization learning rate; step caps its updates. Residual penalty weights target displacement, ∥m−h∥/∥h∥2\lVert m-h\rVert/\lVert h\rVert^2; radius penalty weights (∥m∥/∥h∥−1)2(\lVert m\rVert/\lVert h\rVert-1)^2. Clamp bounds displacement relative to the original norm. Lambda weights preservation of non-target keys in the weight solve (default 15,000); update multiplies the final weight update by the listed factor. Shrinking FE's update to comparable DKLD_{KL} sacrifices editing efficacy. SE also improves MEMIT with its settings unchanged (Table 6).

AuthorsOpenReview ↗

Response to W1 & Q3, Q2, Q4 (Theoretical Analysis, Adam Optimization, and Paraphrase Generalization)

Thank you for the valuable suggestion.

Weakness 1 & Question 3 (Theory).

(1) Near-orthogonality. The Remark under Theorem 1 (lines 193–197) already distinguishes scale-invariant normalized branches from residual side paths, including Eq. 30’s identity path. Normalized branches suppress radial sensitivity. Table 5 shows near-orthogonality of the full prediction gradient: median radial/tangential ratios are 1.6% / 1.5% / 1.3% on GPT2-XL / Qwen / GLM, with 95% of steps within 2.8° of the tangent space.

(2) Norm growth. Theorem 2 describes tangential gradient descent. For approximate tangency, Eq. 35 gives growth when η∥gt∥2>2mt⊤gt\eta\lVert g_t\rVert^2>2m_t^\top g_t. Figure 4 reports median ∥m∥/∥h∥\lVert m\rVert/\lVert h\rVert of 1.21 / 1.41 / 1.24 under the actual optimization on GPT2-XL / Qwen / GLM.

(3) Batch editing and output locality. Theorem 3 establishes the single-edit hidden-state relation ∥Δk′∥∝∥m−h∥\lVert\Delta k'\rVert\propto\lVert m-h\rVert. Table 1 measures the actual 10,000-edit update of Eq. 5, including all cross-edit interactions: rescaling reduces preservation perturbation by 24–42% across three models and two datasets. In Tables 2–3 and the controls above, SE lowers DKLD_{KL} relative to FE in all six settings, compared with five for final rescaling.

Question 2 (Optimizer). Following MEMIT and FE, we use Adam with β=(0.9,0.999)\beta=(0.9,0.999). For GPT2-XL, the learning rate is 0.5, with at most 34 updates and loss < 0.05 stopping. The loss is NLL + 0.0625 KL + w∥r∥/∥h∥2w\lVert r\rVert/\lVert h\rVert^2, with raw residual rr clamped to ∥r∥≤c∥h∥\lVert r\rVert\le c\lVert h\rVert. FE uses m=h+rm=h+r; SE uses the spherical target mrotm_{\mathrm{rot}} in Eq. 14. FE uses (w,c)=(0.5,0.75)(w,c)=(0.5,0.75); SE uses the c=0.65c=0.65 setting and w=0.5w=0.5 / 0.1 on zsRE / CounterFact. There is no optimizer weight decay, gradient clipping or learning-rate schedule (Appendix A.1).

Adam uses momentum and coordinatewise scaling. For FE's actual post-clamp target step ut=mt+1−mtu_t=m_{t+1}-m_t, the norm change obeys

∥mt+1∥2−∥mt∥2=2⟨mt,ut⟩+∥ut∥2.\lVert m_{t+1}\rVert^2-\lVert m_t\rVert^2=2\langle m_t,u_t\rangle+\lVert u_t\rVert^2.

Figure 4 shows norm growth under the actual Adam optimization; SE prevents this growth by keeping ∥mrot∥=∥h∥\lVert m_{\mathrm{rot}}\rVert=\lVert h\rVert throughout.

Question 4 (Generalization). Success/Accuracy use original edit prompts. On zsRE, Gen. is target-token top-1 accuracy on held-out paraphrases. Overall Score is the harmonic mean of Success, Gen. and Top-1. SE achieves a higher Overall Score while retaining comparable paraphrase generalization. Our matched runs (first comment, item (2)) show improved efficacy and locality with comparable or higher Gen. Selected results from revised Table 2 (10,000 edits):

zsRE

ModelMethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑
GPT2-XLMEMIT55.8466.8246.3361.461.80444.4629.18
AlphaEdit59.7672.4353.8266.471.94746.8428.52
FE67.5676.0056.6267.491.58360.8633.16
FE + SE (ours)70.7476.0256.7367.211.14269.5739.01
Qwen2.5-7B-InstructMEMIT13.9682.2859.0978.964.5955.2616.40
AlphaEdit25.0088.2271.7281.724.85610.3721.88
FE34.9391.9476.7983.894.18415.8528.27
FE + SE (ours)48.3693.5180.2783.183.22325.4343.87
GLM-4-9B-ChatMEMIT62.9085.6767.5873.651.73144.5651.50
AlphaEdit61.1982.7466.2172.491.86243.2150.72
FE68.9786.0267.8874.671.18554.1260.34
FE + SE (ours)69.3686.4367.9174.221.03354.9161.96

For the requested FE / final-rescaling / SE paraphrase comparison, Gen. is 67.49 / 66.01 / 67.21 on GPT2-XL zsRE and 83.89 / 75.25 / 83.18 on Qwen zsRE. With MEMIT's settings unchanged, our supplementary paraphrase evaluation also raises Gen. from 61.46 to 64.26.

CounterFact (revised Table 3, 10,000 edits). Gen. is the fraction of held-out paraphrases preferring the new object to the original.

ModelMethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑
GPT2-XLFE71.3093.3075.1779.451.32453.2849.95
FE + SE (ours)72.3893.3175.2779.051.20955.3151.73
Qwen2.5-7B-InstructFE77.4899.4096.7193.771.25755.5757.36
FE + SE (ours)82.0799.3696.8193.050.84563.5263.61
GLM-4-9B-ChatFE81.9199.2897.5391.540.94063.9962.03
FE + SE (ours)83.0799.2797.6791.150.85166.3564.49
Reviewer_5yHvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the optimization process of the target hidden state mm in locate-then-edit knowledge editing. The authors observe that in large language models containing LayerNorm or RMSNorm, the gradient of the target loss with respect to mm is typically dominated by tangential components, so optimization primarily changes the direction of mm; however, when performing gradient descent in ordinary Euclidean space, tangential updates cause ∥m∥\|m\| to grow passively.

Based on this, the paper proposes Spherical Editing (SE): through reparameterization, the target state is constrained to lie on a sphere with radius equal to the original hidden state norm ∥h∥\|h\|, so that optimization primarily proceeds through direction changes while avoiding unnecessary radial growth. The authors argue that radial growth contributes limitedly to the target knowledge but increases parameter editing residuals, thereby exacerbating interference with non-target knowledge.

The paper conducts experiments on GPT2-XL, Qwen2.5-7B-Instruct, and GLM-4-9B-Chat using the ZSRE and COUNTERFACT datasets, testing up to 10,000 edits. Based on the experimental setup, the primary results should correspond to applying SE to FE, with additional results reported for its combination with MEMIT and AlphaEdit. The experiments show that SE generally maintains target Success/Accuracy on the original editing prompts while reducing the DKL of non-target output distributions and improving Top-k overlap.

Overall, the problem addressed in this paper has a certain degree of novelty, and SE has the potential to be a simple, minimally invasive, and plug-and-play method. However, the current version lacks sufficient justification regarding the applicability conditions of the core theoretical chain, the independent contribution of SE relative to other simple alternatives, and the generalization ability of edited knowledge under equivalent paraphrases.

Critical strengths
  1. The research perspective has a certain degree of novelty, focusing on the relatively under-studied aspect of target state construction

    Existing knowledge editing work primarily focuses on editing locations, parameter update formulas, and locality regularization. This paper further analyzes the direction and magnitude of the target hidden state, pointing out that the optimization process of the target state itself may affect editing locality. This problem formulation has certain value and provides an analytical perspective different from traditional parameter update viewpoints for understanding locate-then-edit methods.

  2. The SE method is simple and demonstrates stable locality improvements across multiple editing settings

    SE only modifies the construction of the target state without introducing additional editing networks or new model parameters. The paper observes across multiple models, datasets, and large-scale and sequential editing settings that SE generally reduces DKL and improves Top-1/5/10 overlap without significantly degrading the editing success rate on original target prompts.

    If these results can be confirmed through more rigorous controlled experiments, SE would have good practical value and applicability across editing frameworks.

Critical weaknesses
  1. The applicability scope of the core theoretical conclusions needs to be tightened, and some derivations remain incomplete. What the paper rigorously proves is that: for scale-invariant functions, or for normalized computation branches, the gradient is orthogonal to the current hidden state under ideal conditions. However, the gradient of a complete Pre-LN network also contains components from the residual/skip path, which are not necessarily orthogonal to mm. Therefore, LayerNorm/RMSNorm alone cannot imply that the complete target loss gradient is necessarily approximately orthogonal; this is currently supported primarily by empirical measurements. Furthermore, Theorem 2 assumes exact orthogonality: mt⊤gt=0.m_t^\top g_t=0. Under actual approximate orthogonality, ∥mt+1∥2−∥mt∥2=−2ηmt⊤gt+η2∥gt∥2,\|m_{t+1}\|^2-\|m_t\|^2=-2\eta m_t^\top g_t+\eta^2\|g_t\|^2, so the norm does not necessarily increase monotonically. The paper has not yet provided error bounds or corresponding step-size conditions for the approximate orthogonality case. Theorem 3 also requires further clarification. The paper derives Eq. 11 from the ridge solution Eq. 5 with finite regularization, but Eq. 11 appears to lack terms related to λ\lambda. The authors need to clarify whether this equation is an exact result, an approximation, or the solution to a different optimization problem. Additionally, the analysis for the single-edit case cannot be directly generalized to rigorous conclusions for large-scale multi-edit scenarios, because different residuals become coupled through matrix inversion in the multi-edit setting. Therefore, the paper currently supports the empirical observation that "norm growth is correlated with parameter perturbation and degraded locality," but it has not yet sufficiently established the general theory that "radial growth universally causes locality degradation in real model behavior."

  2. Missing key controlled experiments that can distinguish the independent contribution of SE. The paper's main results appear to correspond primarily to FE + SE, but the naming SE (ours) in the tables is not sufficiently clear. More importantly, the current experiments have not adequately ruled out the following alternative explanations:

    • Early stopping of plain FE;
    • Post-hoc target rescaling after FE optimization converges;
    • Smaller learning rates or different numbers of optimization steps;
    • Explicit norm regularization;
    • The different optimization trajectories and convergence behaviors induced by SE.

    Since the paper links norm growth to the number of tangential gradient update steps, early stopping is a directly relevant strong baseline. If plain FE can already achieve similar locality with fewer steps, or if rescaling the target state back to ∥h∥\|h\| after optimization reproduces SE's benefits, then the current experiments cannot demonstrate that spherical reparameterization itself is the primary cause.

    It is recommended to compare at least, under matched target Success, Accuracy, or target loss: original FE, FE with different stopping steps, post-hoc rescaling, explicit norm regularization, and FE+SE.

  3. Equivalent paraphrase generalization (GR) is not explicitly reported. The paper primarily reports Success and Accuracy on the original target editing prompts, as well as DKL and Top-k metrics on non-target knowledge, but does not explicitly report the generalization performance of edited knowledge under semantically equivalent paraphrases, syntactic reformulations, or different prompt templates—i.e., Generalization/GR.

    This omission is particularly important for this paper. The core claim is:

    "The direction of the target hidden state carries the primary editing information, while norm growth contributes limitedly to the target knowledge."

    If this claim holds, SE should not only maintain success rates on the original editing prompts but also maintain or improve performance on equivalent paraphrases of the target knowledge. High Success/Accuracy solely on the original rewrite prompts cannot sufficiently demonstrate that the new knowledge has been robustly written into the model. The model may only produce localized responses to specific prompt trigger patterns. In particular, one possibility needs to be excluded: SE improves locality through more conservative parameter updates but simultaneously weakens the generalization of knowledge across different paraphrases.

    Therefore, the current experiments are insufficient to fully support the strong conclusion that "SE improves locality without sacrificing overall editing quality." A more cautious conclusion should be:

    "SE maintains editing success rates on the tested original target prompts and improves the reported non-target output preservation metrics; its impact on equivalent paraphrase generalization has not been fully verified."

    The authors are advised to comprehensively report and compare GR performance metrics.

Questions for authors
  1. Does SE (ours) in Table 2 and Table 3 strictly correspond to FE + SE?​

    If so, it is recommended to rename the table entries to FE + SE. Please specify whether FE and SE use the same editing layers, optimizer, learning rate, optimization steps, stopping criteria, and target loss thresholds.

  2. Please rigorously clarify the relationship between Eq. 11 and Eq. 5.​

    Eq. 5 is the ridge solution with finite λ\lambda, while Eq. 11 appears to lack λ\lambda-dependent terms. Is Eq. 11 an approximation, a limiting case, or the solution to a different hard-constraint problem?

  3. What optimizer is used for the actual target state optimization?​

    Please specify whether SGD, Adam, AdamW, momentum, gradient clipping, or learning rate scheduling are used. Theorem 2 addresses plain gradient descent, whereas Adam's actual update direction is not necessarily equal to the raw gradient; therefore, the correspondence between theory and implementation needs to be clarified.

  4. How is ∥Δkj∥22\|\Delta k_j\|_2^2 in Table 1 aggregated?​

    Are the reported values sums, averages over the preservation set, or regularized terms incorporating λ/u\lambda/u weights? Please provide the complete formula and the size of the preservation set.

  5. What is the precise definition of Top-k overlap?​

    For multi-token outputs, does Top-k compare at the first next-token position, or is it compared position-by-position under teacher forcing?

3 comments
AuthorsOpenReview ↗

Response to W2 & Q1 (Controlled Comparisons and Settings) and W3 (Paraphrase Generalization)

Thank you very much for the detailed and constructive review and suggestions.

Tables 2–3 update SE results and add Gen. and Overall Score following MEMIT. Baselines use the official FE repository's tuned configurations.

Weakness 2 & Question 1 (Controls and settings).

(1) Supplementary matched runs. SE is now labeled FE + SE. Each GPT2-XL zsRE pair shares layers, covariance, Adam, lr, step budget, stopping rule and clamp:

ClampMethodSuccess ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑
0.55FE72.6252.2563.351.00874.86
FE + SE73.9854.0364.310.86277.73
0.60FE74.1554.5465.071.21169.85
FE + SE75.1755.7365.830.97275.01
0.70FE75.5056.1266.781.45764.80
FE + SE76.5457.3367.511.27166.17
0.75FE76.0056.5967.541.58960.92
FE + SE76.6857.3867.461.36363.17

(2) Alternative controls. Same 10,000 GPT2-XL zsRE requests; FE defaults: lr 0.5, residual penalty 0.5, radius penalty 0, clamp 0.75, at most 34 updates, loss < 0.05 stopping. Controls modify FE as stated:

Method and settingSuccess ↑Accuracy ↑DKL ↓Top-1 ↑Top-10 ↑
lr 0.5 (default)76.0056.621.58360.8632.74
lr 0.2575.5256.341.63260.1632.35
lr 0.12575.3355.701.77356.9330.41
residual penalty 0.175.6055.931.60260.6532.37
residual penalty 0.2575.8156.011.59960.5732.41
residual penalty 1.075.9856.151.56461.3333.04
residual penalty 2.076.0156.271.54762.0933.71
radius penalty 2.575.5455.941.36364.2736.81
radius penalty 2574.9954.771.37160.4135.96
radius penalty 25075.8356.181.47657.7934.72
clamp 0.5572.6252.251.00874.8644.82
clamp 0.6574.9855.231.34166.9737.51
clamp 0.7075.5056.121.45764.8035.31
step 862.8039.161.06677.6538.40
step 1067.8145.841.25971.7135.83
step 1574.1854.131.56662.3732.75
step 2075.6655.951.58361.6432.79
final rescaling75.5155.731.17768.9438.81
lambda 20,50075.1255.791.55562.2033.41
lambda 60,00064.5143.471.39063.2837.46
update ×0.9574.3854.411.36467.2036.47
update ×0.971.9451.301.13772.7140.83
update ×0.875970.3349.121.03575.1142.98
FE + SE76.0256.731.14269.5739.29

Residual penalty weights target displacement; radius penalty weights (∥m∥/∥h∥−1)2(\lVert m\rVert/\lVert h\rVert-1)^2. Clamp bounds displacement relative to the original norm. Lambda weights preservation of non-target keys in the weight solve (default 15,000); update multiplies the final weight update by the listed factor.

(3) Final rescaling. This simplified SE variant optimizes at one radius and delivers at another; residual radial sensitivity can raise loss (Remark, lines 193–197). SE optimizes direction at the original radius throughout. Matched supplement: same data, clamp label 0.65, residual penalty 0.5, lr 0.5.

MethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑Top-10 ↑
FE + final rescaling69.9972.4652.5766.311.01171.5240.4140.93
FE + SE70.7476.0256.7367.211.14269.5739.0139.29

SE has higher efficacy, Gen. and Overall Score; final rescaling has better locality.

Weakness 3 (Generalization). Gen. is target-token top-1 accuracy on held-out zsRE paraphrases, and the fraction of CounterFact paraphrases preferring the new object to the original. Overall Score is the harmonic mean of Success, Gen. and Top-1. In (1), SE improves efficacy and locality with comparable or higher Gen. Revised Tables 2–3 (10,000 edits):

zsRE

ModelMethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑
GPT2-XLMEMIT55.8466.8246.3361.461.80444.4629.18
FE67.5676.0056.6267.491.58360.8633.16
FE + SE (ours)70.7476.0256.7367.211.14269.5739.01
Qwen2.5-7B-InstructMEMIT13.9682.2859.0978.964.5955.2616.40
FE34.9391.9476.7983.894.18415.8528.27
FE + SE (ours)48.3693.5180.2783.183.22325.4343.87
GLM-4-9B-ChatMEMIT62.9085.6767.5873.651.73144.5651.50
FE68.9786.0267.8874.671.18554.1260.34
FE + SE (ours)69.3686.4367.9174.221.03354.9161.96

CounterFact

ModelMethodOverall Score ↑Success ↑Accuracy ↑Gen. ↑DKL ↓Top-1 ↑Top-5 ↑
GPT2-XLFE71.3093.3075.1779.451.32453.2849.95
FE + SE (ours)72.3893.3175.2779.051.20955.3151.73
Qwen2.5-7B-InstructFE77.4899.4096.7193.771.25755.5757.36
FE + SE (ours)82.0799.3696.8193.050.84563.5263.61
GLM-4-9B-ChatFE81.9199.2897.5391.540.94063.9962.03
FE + SE (ours)83.0799.2797.6791.150.85166.3564.49
AuthorsOpenReview ↗

Response to W1 & Q2 (Theory), Q3 (Optimizer), Q4 (Table 1 Aggregation), and Q5 (Top-k Definition)

Thank you very much for the valuable suggestion. Weakness 1 & Question 2 (Theory).

(1) Near-orthogonality. The Remark under Theorem 1 (lines 193–197) already distinguishes scale-invariant normalized branches from residual side paths. Normalized branches suppress radial sensitivity. Table 5 shows near-orthogonality of the full prediction gradient: median radial/tangential ratios are 1.6% / 1.5% / 1.3% on GPT2-XL / Qwen / GLM, with 95% of steps within 2.8° of the tangent space.

(2) Norm growth. Theorem 2 describes tangential gradient descent. For approximate tangency, Eq. 35 gives growth when η∥gt∥2>2mt⊤gt\eta\lVert g_t\rVert^2>2m_t^\top g_t. Figure 4 reports median ∥m∥/∥h∥\lVert m\rVert/\lVert h\rVert of 1.21 / 1.41 / 1.24 under the actual optimization on GPT2-XL / Qwen / GLM.

(3) Eq. 11 and Eq. 5. Eq. 11 is the single-edit form of Eq. 5. The revised expression explicitly includes λ\lambda: s=C−1k/(λ+k⊤C−1k)s=C^{-1}k/(\lambda+k^\top C^{-1}k) (revised lines 258–261). Since ss is independent of mm, ∥Δk′∥∝∥m−h∥\lVert\Delta k'\rVert\propto\lVert m-h\rVert and Theorem 3 remain unchanged.

(4) Batch editing and output locality. Theorem 3 establishes the single-edit mechanism. Table 1 evaluates the actual 10,000-edit update, including all cross-edit interactions: rescaling reduces preservation perturbation by 24–42% across all three models and both datasets. Tables 2–3 further demonstrate the gains in end-to-end output locality.

Question 3 (Optimizer). Following MEMIT and FE, we use Adam with β=(0.9,0.999)\beta=(0.9,0.999). For GPT2-XL, the learning rate is 0.5, with at most 34 updates and loss < 0.05 stopping. The loss is NLL + 0.0625 KL + w∥r∥/∥h∥2w\lVert r\rVert/\lVert h\rVert^2, with raw residual rr clamped to ∥r∥≤c∥h∥\lVert r\rVert\le c\lVert h\rVert. FE uses m=h+rm=h+r; SE uses the spherical target mrotm_{\mathrm{rot}} in Eq. 14. FE uses (w,c)=(0.5,0.75)(w,c)=(0.5,0.75); SE uses the c=0.65c=0.65 setting and w=0.5w=0.5 / 0.1 on zsRE / CounterFact. There is no optimizer weight decay, gradient clipping or learning-rate schedule (Appendix A.1).

Adam uses momentum and coordinatewise scaling. For FE's actual post-clamp target step ut=mt+1−mtu_t=m_{t+1}-m_t, the norm change obeys

∥mt+1∥2−∥mt∥2=2⟨mt,ut⟩+∥ut∥2.\lVert m_{t+1}\rVert^2-\lVert m_t\rVert^2=2\langle m_t,u_t\rangle+\lVert u_t\rVert^2.

Figure 4 shows norm growth under the actual Adam optimization; SE prevents this growth by keeping ∥mrot∥=∥h∥\lVert m_{\mathrm{rot}}\rVert=\lVert h\rVert throughout.

SE/FE reach loss < 0.05 on 294/289 zsRE and 270/257 CounterFact requests (300 each); median updates: 15/16 and 16/18.

Question 4 (Table 1 aggregation). Appendix A.10 specifies the sum ∑j=n+1n+u∥Δkj∥2=u tr(ΔCΔ⊤)\sum_{j=n+1}^{n+u}\lVert\Delta k_j\rVert^2=u\,\mathrm{tr}(\Delta C\Delta^\top), without the regularization weight λ\lambda. Here Δ\Delta is the first edited layer's update for all 10,000 edits jointly. The preservation keys come from 100,000 Wikipedia samples: u=u= 44,068,071 / 66,142,941 / 68,593,529 for GPT2-XL / Qwen / GLM.

Question 5 (Top-k definition). Following FE, Top-kk is ∣topk(p)∩topk(p⋆)∣/k|\mathrm{top}_k(p)\cap\mathrm{top}_k(p^\star)|/k, where pp and p⋆p^\star are the next-token distributions before and after editing. It is evaluated at the first next-token position of each locality prompt, without teacher forcing, and averaged over 10,000 zsRE / 100,000 CounterFact prompts (Appendix A.10).

Thanks for the Authors' detailed response. My concerns have been largely addressed, so I'd like to update my score from 2 to 3 (weak acceptance). The reason I don't improve it to 4 is that although I think this work has good practical value and applicability, but the overall performance improvement is relative limited. More important, from the perspective of fundamentally solving the model editing problem, this work functions as a "patch" for internal model editing methods; its potential for long-term, further development appears relatively limited. If my understanding in this regard is off the mark, I welcome further discussion and exchange with the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.