acceptodds
Under review as a conference paper at ICLR 2027

LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing

Abstract

Lifelong knowledge editing aims to efficiently and sequentially update language models over time, as new knowledge becomes available or when the model makes mistakes, while preserving acceptable performance on past knowledge. One unresolved challenge is that most existing methods modify a fixed set of layers for all new knowledge samples, reducing flexibility and increasing catastrophic forgetting. Another is requiring access to previous knowledge and extensive pre-processing to obtain data statistics. To address these challenges, we introduce LOKI, a novel approach that uses dynamic layer selection based on the Hilbert-Schmidt Independence Criterion and projects gradient updates onto the null-space of the model weights, bypassing the requirement for previous knowledge access. We show that LOKI achieves superior performance to existing approaches across a wide variety of experiments, achieving up to a 14% improvement in average accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Knowledge Editing, Continual Learning, Machine Learning, Null-space

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ocUpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current knowledge editing methods edit predefined, fixed model layers, which limits their flexibility. Updating the same layers repeatedly may also cause catastrophic forgetting. To address these challenges, the paper proposes LOKI. LOKI dynamically selects editing layers using the Hilbert–Schmidt Independence Criterion (HSIC) and projects weight updates onto the weight null space, enabling more flexible knowledge editing.

Critical strengths

The paper clearly identifies the limitations of fixed editing layers in current knowledge editing methods and proposes a proxy metric to effectively identify which layers to edit.

The proposed LOKI algorithm achieves strong editing performance, demonstrating its effectiveness.

Critical weaknesses

LOKI requires more optimizer parameters than previous methods. Although it achieves strong editing performance, its increased computational cost departs somewhat from one of the original goals of knowledge editing, which is to modify model behavior at a low computational cost.

LOKI performs SVD on the weight matrices of the selected layers before each edit. Although its editing time is comparable to baseline methods at the 7B scale, the computational cost of SVD will become more substantial as model size increases.

The paper lacks parameter sensitivity experiments, particularly for the hyperparameters newly introduced in HIB.

There are some issues with the presentation, such as an unresolved cross-reference on line 858.

Questions for authors

For a single edit using LOKI, how much time is spent on layer selection, SVD, and gradient updates respectively?

How do changes in the values of the hyperparameters newly introduced in LOKI affect editing performance?

How does LOKI perform on the CounterFact dataset?

On Llama-3, the selected layers are concentrated in layers 22–26. How would LOKI perform if these layers were fixed for editing instead of being selected dynamically?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ocUp (Part 1)

We thank the reviewer for the careful reading and constructive feedback, and for recognizing that LOKI identifies an important limitation of fixed-layer editing and achieves strong performance. Below, we address the weaknesses and questions raised by the reviewer. Unless stated otherwise, all new experiments use the following protocol: Llama-3-8B, sequential single edits, a single NVIDIA H200 GPU.

Weaknesses

W1: Optimizer parameters and computational cost

"LOKI requires more optimizer parameters than previous methods. [...] its increased computational cost departs somewhat from one of the original goals of knowledge editing, which is to modify model behavior at a low computational cost."

We would first like to clarify that LOKI does not use any additional parameters or memory at inference time. After each edit, the model is simply the base model with updated weights in the selected layers: there are no adapters, side memories, codebooks or routing modules. The parameters optimized during an edit and their Adam state exist only while that edit is performed and are freed afterwards, so they do not accumulate over sequential edits.

How we report the cost of an edit. We measure the computational cost of a single edit along three axes:

  • Time: wall-clock seconds per edit, taken from Table 6 of the paper. As in the paper, this excludes data loading, evaluation, and the one-time preprocessing of the methods that rely on past-knowledge covariance statistics (ROME, MEMIT, AlphaEdit and SimIE).
  • Memory: the peak GPU memory during an edit, including the model weights themselves (≈30 GB for Llama-3-8B), averaged over edits.
  • Parameters: since "trainable parameters" means different things for different families of editors, we report three counts. Optimized counts the parameters that receive gradients and optimizer state during an edit. Modified counts the base-model weights whose values change after the edit. Added counts new parameters introduced by the editor (adapters, side memories, codebooks), which remain at inference time. For gradient-based editors (FT, LoRA, WISE, LOKI), the optimized parameters are exactly the modified or added ones. For locate-then-edit methods (ROME, MEMIT, AlphaEdit, SimIE) they are not: gradient descent optimizes only a single 4,096-dim target vector, but a closed-form least-squares update then rewrites full FFN matrices (one layer for ROME, five for MEMIT/AlphaEdit). One Llama-3-8B down-projection has 4,096 × 14,336 = 58.7M parameters.

Results. In terms of memory, LOKI's peak during an edit (35.4 GB) is lower than that of the locate-then-edit methods MEMIT (37.9 GB), AlphaEdit (37.0 GB) and SimIE (36.2 GB), and the whole edit fits comfortably on a single 80 GB GPU. In terms of how much of the model an edit changes, LOKI (176M, three selected layers) sits between ROME (58.7M) and MEMIT/AlphaEdit (293.6M). The larger optimized-parameter count relative to locate-then-edit methods is paid for by the transient Adam state (1.31 GB), which does not translate into higher memory than MEMIT, AlphaEdit or SimIE.

In terms of time, averaged over the five dataset/backbone settings of Table 6, LOKI takes 15.0 s per edit, which is faster than AlphaEdit (16.1 s), ROME (17.3 s), WISE (23.5 s) and SimIE (27.6 s), i.e. every baseline that is competitive in our benchmarks. Moreover, these numbers exclude the one-time preprocessing required by MEMIT and AlphaEdit, which takes ∼25 h on an H200 for Llama-3-8B (Table 7) and must be repeated for every new model or change of edited layers. Figure 1(b) shows the same trade-off visually: LOKI attains the lowest perplexity at comparable per-edit runtime. We therefore believe LOKI is in line with the goal of low-cost knowledge editing, and in fact lowers the total cost compared to the strongest baselines. We will include Table R1 and this discussion in the revised manuscript.

Table R1. Cost of a single edit. Time per edit is the average over the five dataset/backbone settings of Table 6. Parameter counts and peak memory are measured on Llama-3-8B, ZsRE. Peak memory includes the model weights.

MethodOptimized paramsModified paramsAdded paramsTime per edit (s)Peak memory (GB)
FT58.7M (1 layer)58.7M00.6431.30
ROME4K (value vector)58.7M (rank-1 update, 1 layer)017.2932.8
MEMIT4K (target vector)293.6M (5 layers)011.3937.94
AlphaEdit4K (target vector)293.6M (5 layers)016.0836.96
SimIE4K (ROME base)58.7M027.6036.18
GRACE4K per edit (codebook value)04K per edit (codebook grows)6.4130.25
DEFER58.7M (adaptor)058.7M8.0631.52
LoRA3.4M (r=8, q/v, all layers)03.4M10.2430.19
AdaLoRA3.4M03.4M12.6330.24
WISE58.7M (side memory, 1 layer)058.7M23.5532.56
LOKI176M (3 layers)176M014.9635.36
AuthorsOpenReview ↗

Response to Reviewer ocUp (Part 2)

W2: SVD cost at larger model scale

"LOKI performs SVD on the weight matrices of the selected layers before each edit. Although its editing time is comparable to baseline methods at the 7B scale, the computational cost of SVD will become more substantial as model size increases."

We agree with the reviewer that the SVD cost will grow with the size of the edited down-projection matrices. However, we would like to note that it does not grow with the total size of the model, but only with the size of these matrices. Due to layer selection, LOKI computes the SVD only for the mm selected layers per edit, not for all layers, so adding depth to a model does not increase the cost. Furthermore, LOKI decomposes the down-projection Wout∈Rd1×d2W_{\text{out}} \in \mathbb{R}^{d_1 \times d_2} directly, where d1d_1 is the hidden size and d2d_2 the FFN intermediate size. This is cheaper than the d2×d2d_2 \times d_2 covariance decomposition used by AlphaEdit, although LOKI performs it per edit rather than once during preprocessing. At the 7B/8B scale, the SVD is already included in our timing results (Table 6), and we break it down per stage in Q1 below.

To quantify this scaling, we timed LOKI's per-layer null-space SVD (torch.linalg.svd, full matrices, float32) on the real FFN down-projection of one layer of Llama-3-8B, Llama-2-13B and Llama-3-70B (Table R2). From 8B to 70B, the SVD cost grows by 5.8×, which is less than the ≈8.8× growth in parameter count that drives the forward/backward cost of the optimization step (and of every gradient-based baseline). We therefore expect the SVD's share of the edit time to stay roughly constant or decrease at larger scale. For reference, AlphaEdit's one-time projector requires an SVD of a d2×d2d_2 \times d_2 covariance per edited layer (29.5 s at 8B, 236.7 s at 70B), in addition to collecting the covariance statistics themselves.

We will include this scaling analysis in the revised manuscript.

Table R2. Time and memory of LOKI's null-space SVD on a single FFN down-projection.

Modeldown_proj shape (d1×d2d_1 \times d_2)LOKI SVD per layer (s)Per edit, 3 layers (s)Peak mem (GB)
Llama-3-8B4096 × 143361.123.361.77
Llama-2-13B5120 × 138242.016.042.00
Llama-3-70B8192 × 286726.4619.397.11

W3: Hyperparameter sensitivity

"The paper lacks parameter sensitivity experiments, particularly for the hyperparameters newly introduced in HIB."

We would first like to point the reviewer to the existing sensitivity results in the paper. Table 9 sweeps the null-space dimension null_dim, showing that performance is flat over [500, 2000] (0.886–0.890 Avg) and that all values used in Table 8 lie within 0.010 of the optimum. The ablation in Table 5 covers the setting where the HIB regularization coefficients are set to zero (λx=λy=0\lambda_x = \lambda_y = 0 in the objective, i.e. "Reg." removed), as well as replacing HIB layer selection with fixed layers. Q4 below further studies layer selection.

For the hyperparameters newly introduced in HIB, we provide a sensitivity analysis for the ratio λx/λy\lambda_x / \lambda_y and for the bandwidth σ\sigma of the Gaussian kernel (ZsRE, Llama-3-8B, 1000 sequential edits, all other hyperparameters fixed at their Table 8 values). Recall that HIB scores each layer ll as

HIB(Xt,l)=λyHSIC(Z^(l),Z(L))−λxHSIC(Xt,Z^(l))=λy(HSIC(Z^(l),Z(L))−λxλyHSIC(Xt,Z^(l))).\mathrm{HIB}(X_t, l) = \lambda_y \mathrm{HSIC}\big(\hat Z^{(l)}, Z^{(L)}\big) - \lambda_x \mathrm{HSIC}\big(X_t, \hat Z^{(l)}\big) = \lambda_y \Big( \mathrm{HSIC}\big(\hat Z^{(l)}, Z^{(L)}\big) - \tfrac{\lambda_x}{\lambda_y} \mathrm{HSIC}\big(X_t, \hat Z^{(l)}\big) \Big).

Since the common factor λy>0\lambda_y > 0 scales the scores of all layers equally, the top-mm ranking is invariant to jointly rescaling λx,λy\lambda_x, \lambda_y, so for layer selection the ratio λx/λy\lambda_x / \lambda_y is the only quantity that matters. A ratio of 1 and σ=1\sigma = 1 correspond to the default values reported in the paper.

As shown in Tables R3 and R4, across a 256× range of the λ\lambda ratio and a 100× range of σ\sigma, average performance varies by at most 0.4 and 0.7 points, respectively. LOKI is therefore not very sensitive to the choice of these hyperparameters and only experiences minor fluctuations in average performance.

We will include these sweeps in the revised manuscript, next to Table 9.

Table R3. Sensitivity to the ratio λx/λy\lambda_x / \lambda_y (ZsRE, Llama-3-8B, 1000 sequential edits).

λx/λy\lambda_x / \lambda_y1641 (default)1/41/16
Avg.888.887.890.886.888

Table R4. Sensitivity to the HSIC kernel bandwidth σ\sigma (ZsRE, Llama-3-8B, 1000 sequential edits).

HSIC σ\sigma0.10.51 (default)510
Avg.888.889.890.883.883
AuthorsOpenReview ↗

Response to Reviewer ocUp (Part 3)

W4: Presentation

"There are some issues with the presentation, such as an unresolved cross-reference on line 858."

We thank the reviewer for pointing this out. The unresolved cross-reference on line 858 should point to Table 12, and we will fix it in the revised manuscript.

Questions

Q1: Time breakdown of a single edit

"For a single edit using LOKI, how much time is spent on layer selection, SVD, and gradient updates respectively?"

To answer this, we added timers to each of the three stages of a LOKI edit, following the protocol of Table 6 (seconds per edit, averaged over all edits). Selection is HIB scoring, including the computation of the hidden representations; SVD is the computation of the null-space projectors for the selected layers; and Optimization is the gradient updates.

As shown in Table R5, layer selection with HIB is cheap in every setting (≈0.3–0.5 s, 1–5% of the edit), as it requires only a single forward pass, with no causal tracing or per-layer optimization. The remaining time is split between the SVD and the optimization. The SVD time scales with the number of selected layers at roughly 1.2 s per layer: 3 layers on ZsRE (3.58 s), 5 layers for Mistral-7B on SelfCheckGPT (5.98 s), and 7 layers for Llama-3-8B on SelfCheckGPT (8.34 s). On Temporal, where the targets are long, the optimization dominates (83%) and the SVD accounts for only 16% of the edit time.

We will include this breakdown in the revised manuscript.

Table R5. Time breakdown of a single LOKI edit (seconds per edit, share of the total in parentheses).

SettingSelectionSVDOptimizationTotal
Llama-3-8B, ZsRE0.32 (3%)3.58 (37%)5.75 (60%)9.65
Mistral-7B, ZsRE0.34 (5%)3.58 (52%)2.95 (43%)6.88
Llama-3-8B, SelfCheckGPT0.43 (3%)8.34 (53%)6.84 (44%)15.63
Mistral-7B, SelfCheckGPT0.47 (4%)5.98 (55%)4.45 (41%)10.91
GPT-J, Temporal0.30 (1%)3.81 (16%)20.35 (83%)24.47

Q2: Hyperparameter sensitivity

"How do changes in the values of the hyperparameters newly introduced in LOKI affect editing performance?"

We refer the reviewer to our response to W3 above.

Q3: CounterFact

"How does LOKI perform on the CounterFact dataset?"

To answer this, we ran τ ∈ {1K, 2K, 3K} sequential edits on CounterFact with Llama-3-8B, comparing LOKI with AlphaEdit, WISE and FT. We would like to emphasize an important difference in the setup. The baselines use their published Llama-3-8B hyperparameters, and AlphaEdit [1], like MEMIT [2] before it, uses CounterFact as one of its primary evaluation benchmarks. In contrast, for LOKI we use the base ZsRE configuration (Table 8) without any CounterFact-specific tuning.

Despite this, the results in Table R6 favor LOKI as the number of edits grows. AlphaEdit attains the best average score at τ = 1K and 2K, mainly through higher generalization and locality. However, its performance degrades quickly as edits accumulate: its reliability drops from .984 to .914 to .377, and its average collapses to .217 at τ = 3K. WISE retains perfect locality but fails to insert most edits (reliability ≤ .158), while FT loses almost all locality (≤ .025).

In contrast, LOKI is the most stable method across the edit sequence. Its reliability stays at ≈.95 for all τ, its average changes by only 0.4 points from 1K to 3K, and at τ = 3K it attains both the highest reliability and the highest average score. This mirrors the trend we observe on ZsRE (Figure 3), where the gap between LOKI and the baselines grows with the number of edits.

We note that LOKI's locality on CounterFact is lower than on ZsRE and decreases with τ (.200 to .116). We will include these results in the revised manuscript and discuss locality on CounterFact as a limitation and a direction for future work.

Table R6. Sequential editing on CounterFact, Llama-3-8B. Bold is the best average for each τ.

MethodτRelGenLocAvg
FT1K.815.542.025.460
WISE1K.158.0871.000.415
AlphaEdit1K.984.494.304.594
LOKI1K.952.276.200.476
FT2K.809.530.019.453
WISE2K.120.0691.000.397
AlphaEdit2K.914.496.221.544
LOKI2K.949.321.143.471
FT3K.732.446.023.400
WISE3K.105.0621.000.389
AlphaEdit3K.377.214.060.217
LOKI3K.950.349.116.472
Reviewer_fEzUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes LOKI, a lifelong knowledge editing method that selects layers separately for each edit using an HSIC-based score and projects gradient updates onto the selected weight matrices’ null spaces. It aims to reduce interference without storing historical examples or adding memory modules. Experiments on these model backbones and these datasets, including sequences of up to 1k edits, report improved overall performance against several editing baselines.

Critical strengths

The experiments cover Llama-3, Mistral and GPT-J on ZsRE, SelfCheckGPT, and Temporal, with comparisons against AlphaEdit, WISE, and GRACE. LOKI performs well across these different tasks. For example, after 1k edits on ZsRE with Llama, its average score is 0.89, compared with 0.76 for AlphaEdit.

LOKI does not need old training examples or an extra memory module in sec 4. For each edit, it chooses which layers to modify and uses the model weights themselves to restrict the changes. This is useful when historical data are unavailable.

LOKI uses an HSIC-based score to select layers in a single forward pass, rather than always modifying the same layers. The ablation in Table 5 shows that this choice improves performance over fixed-layer editing.

Critical weaknesses
  1. Comparisons with REPAIR [1] and HoReN [2], where implementations are available, would help establish how LOKI performs against other lifelong editing approaches.

[1] REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration [2] HoReN: normalized hopfield retrieval for large-scale sequential model editing

  1. The evaluation stops at 1,000 edits, which provides limited evidence for long-term stability. Longer sequences, such as 5K, 10K, or 50K edits, with retrospective evaluation of earlier edits, would strengthen the lifelong-editing claim.
Questions for authors

The authors should provide additional robustness results to support LOKI’s performance on longer sequential editing tasks.

3 comments
AuthorsOpenReview ↗

Response to reviewer fEzU (Part 1)

We would like to thank the reviewer for recognising the merits of our work and their feedback. Below we would like to address the reviewer's concerns.

Weaknesses

W1: Comparisons with REPAIR [1] and HoReN [2]

We thank the reviewer for their suggestion, and note that we already compare against 11 distinct baselines across three models and three datasets with varying number of edits. We discuss the suggested works below:

REPAIR [1]

REPAIR is a lifelong editing framework built on WISE[4]-style side memories. Edits are written into copies of an FFN layer's weights, kept apart from the main model, and a router decides at inference time whether a query is served by a side memory or by the original weights. REPAIR protects historical knowledge by preserving it in an isolated side memory; in-contrast, one of the main goals of LOKI was to overcome the reliance on additional parameters to move towards a lifelong sustainable editing approach.

HoReN [2]

HoReN is a GRACE[5]-style memory based method which attaches a key–value codebook to one FFN layer. Each edit adds an entry: a normalized key built from the prompt's hidden states, and a trained value. At inference, a normalized Hopfield-style retrieval matches the query against the stored keys. If one matches closely enough, its value is applied; otherwise the model runs unchanged. By design, like GRACE, such codebook-based methods work very well with regards to Loc. and Rel. but struggle with generalization (Gen.). Moreover, HoReN preserves knowledge by using additional memory: edits sit outside the weights and are applied only when a stored key matches. The codebook grows with every edit (one key and value each) and a lookup over all stored keys is needed at inference, which also grows in computation time with the number of edits. Again, using no additional memory is one of our main goals. Thus, LOKI writes each edit into the existing weights, constrained to the current weight's null-space. It adds no parameters, keeps no record of past edits, and leaves inference cost constant.

Furthermore, we would like to note that HoReN is contemporaneous work. To the best of our knowledge, it is a pre-print on arXiv and not published in any peer-reviewed venues yet. We refer to the ICLR Reviewer Guidelines regarding contemporaneous works.

Regardless, we will provide citations and the proper discussions of each work, as well as comparison results in Tables R1 and R2 below using the same experiment setting as Table 1 in the paper and using the available public implementations. We see that LOKI outperforms every method in the Gen. metric. HoReN and its predecessor GRACE achieve slightly higher values of Rel. and Loc. but at the cost of Gen. Considering LOKI uses no additional parameters, and constant memory and computation overhead, we believe these results are a testament to the efficacy of our method.

Table R1. Comparison with REPAIR and HoReN, ZsRE, Llama-3-8B, 100 sequential edits:

MethodAdditional params/memoryRelGenLocAvg
LOKI✗.947.898.920.922
HoReN✓.998.7961.000.931
GRACE✓1.000.391.000.80
REPAIR✓.422.380.580.460
WISE✓.633.6291.000.754

Table R2. Comparison with REPAIR and HoReN, ZsRE, Llama-3-8B, 1000 sequential edits:

MethodAdditional params/memoryRelGenLocAvg
LOKI✗.916.874.879.890
HoReN✓.993.7891.000.927
GRACE✓1.000.371.000.79
REPAIR✓.236.228.655.373
WISE✓.569.559.999.709
AuthorsOpenReview ↗

Response to reviewer fEzU (Part 2)

W2: Longer sequences, such as 5K, 10K, or 50K edits

"Longer sequences, such as 5K, 10K, or 50K edits, with retrospective evaluation of earlier edits, would strengthen the lifelong-editing claim."

We would like to note that we follow the experiment setting of previous lifelong editing works ([3],[4]) for choosing 1000 as the maximum number of reported edits. Furthermore, we provide batch editing (100 edits per batch) results for up to 2000 edits in Tables 10-11. With that said, we provide longer sequence sequential editing results on the ZsRE and Llama settings for 5K and 10K edits in Tables R3 and R4 respectively below. We can see that LOKI still significantly outperforms existing works, some of which such as AlphaEdit collapse beyond 5000 edits. 50K edits sits beyond the time scope of the rebuttal period. We will add the 50K edit results, on top of the below tables to the final version upon acceptance.

Table R3. ZsRE Llama3-8B-Instruct, 5000 sequential edits

methodτ\tauRelGenLocAvgwall time (h)
LOKI5K.853.787.718.7868.3
WISE5K.291.2871.000.5266.9
FT5K.688.635.058.4600.8
AlphaEdit5K.005.005.011.0079.2

Table R4. ZsRE Llama3-8B-Instruct, 10000 sequential edits

methodτ\tauRelGenLocAvgwall time (h)
LOKI10K.833.765.589.72916.4
WISE10K.261.2581.000.50614.1
FT10K.604.564.045.4041.7
AlphaEdit10K.004.004.011.00719.6

Retrospective analysis of earlier edits:

The first 5,000 edits are the same in both runs, so scoring them after 5K and after 10K edits shows how much the extra 5,000 edits erase (paired, ± SE, i.e. the difference in paired edits before and after is aggregated over all edits). The results are shown in Table R5 below. We can see that LOKI degrades the most gracefully out of compared baselines.

methodAvg on first 5K edits, after 5Kafter 10Kchange10K run: Avg on oldest 1Kon newest 1K
LOKI.786.701−.085 ± .003.67.77
FT.460.346−.114 ± .003.31.56
WISE.526.5260.68.48
AlphaEdit.007.005−.002.00.01

Questions:

Q1: The authors should provide additional robustness results to support LOKI’s performance on longer sequential editing tasks.

We refer the reviewer to W2.

References

[1] REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration

[2] HoReN: normalized hopfield retrieval for large-scale sequential model editing

[3] Fang, Junfeng, et al. "Alphaedit: Null-space constrained knowledge editing for language models." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Wang, Peng, et al. "Wise: Rethinking the knowledge memory for lifelong model editing of large language models." Advances in Neural Information Processing Systems 37 (2024): 53764-53797.

[5] Hartvigsen, Tom, et al. "Aging with grace: Lifelong model editing with discrete key-value adaptors." Advances in neural information processing systems 36 (2023): 47934-47959.

This is addressed my concerns . I'll keep my score

Reviewer_C6E9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LOKI, a lifelong knowledge-editing method designed to sequentially update large language models while limiting interference with previous knowledge. For each incoming edit, LOKI computes an HSIC-based information-bottleneck score using input, intermediate FFN, and final-layer representations, and selects the highest-scoring layers for modification. It then applies singular value decomposition to the selected FFN output matrices and projects gradient updates onto a truncated weight null space, aiming to preserve historical behavior without storing past activations or introducing persistent auxiliary parameters. The implementation additionally employs HSIC regularization, KL regularization, and generated prompt prefixes.

The authors evaluate LOKI on ZsRE, SelfCheckGPT, and Temporal using Llama-3-8B-Instruct, Mistral-7B, and GPT-J-6B. Comparisons cover fine-tuning, parameter- and memory-based editors, and locate-then-edit methods. The paper reports strong reliability, generalization, and locality trade-offs, particularly for long ZsRE and SelfCheckGPT editing sequences. Supporting experiments include component ablations, runtime and preprocessing comparisons, null-space analyses, and sequential batch-editing results.

Critical strengths
  1. Practical motivation and design. LOKI addresses an important problem in lifelong knowledge editing: reducing reliance on stored historical activations, persistent auxiliary parameters, and expensive activation-statistic preprocessing. If its preservation mechanism can be properly justified, this design would make sequential editing substantially easier to deploy and scale.

  2. Broad empirical evaluation and promising long-sequence results. The paper evaluates three datasets and three model backbones against a wide range of baselines, and supplements the main results with ablations, runtime analysis, and batch-editing experiments. The strong results on long ZsRE and SelfCheckGPT sequences are particularly encouraging and suggest that the method may have practical value, although the evaluation protocol and statistical robustness require further clarification.

Critical weaknesses
  1. The central theoretical justification does not establish historical-knowledge preservation. Theorem 4.1 states Null⁡(Z0⊤)⊂Null⁡(W)\operatorname{Null}(Z_0^\top)\subset\operatorname{Null}(W), but this inclusion does not imply PZ0=0PZ_0=0 or ΔPZ0=0\Delta PZ_0=0 for updates projected onto the weight null space. Thus, the theorem does not rule out interference with historical features, which is the central motivation for the proposed projection. The Grassmannian-distance results in Table 4 do not resolve this logical gap. This is a fundamental concern about the method’s justification rather than a minor proof issue.

  2. The evidence for lifelong retention is currently ambiguous. Equations (14)-(16) appear to evaluate each edit using fθtf_{\theta_t}, potentially immediately after that edit is inserted. If so, the results measure immediate editing success but do not directly demonstrate that earlier edits survive subsequent updates. This leaves the paper’s main lifelong-editing claim insufficiently validated. The absence of variances over independent edit orders for the longest and most important experiments further limits confidence in the reported improvements.

Questions for authors
  1. Theoretical justification:​ Can the authors derive a condition under which the proposed update provably satisfies ΔPZ0=0\Delta PZ_0=0, or provide a quantitative bound on ∥PZ0∥\|PZ_0\| and the resulting historical-output drift? How should Theorem 4.1 be revised so that its exact claim remains consistent with the approximate initialization assumption W0≈0W_0\approx0?

  2. Evaluation timing:​ In Tables 1-3, is each edit evaluated only immediately after its own insertion using fθtf_{\theta_t}, or are all earlier edits re-evaluated using the final model fθτf_{\theta_\tau}? If the former, can the authors provide final-checkpoint reliability and generalization over all previous edits, as well as retention as a function of edit distance?

  3. Implemented optimization:​ What is the exact update used in the experiments? In particular, how are Adam, gradient projection, optimizer state, and gradient clipping ordered, and how do they preserve the constraint in Equation (8)? Does the implementation enforce a cumulative constraint ∥Δ∥F≤η\|\Delta\|_F\leq\eta, and where do the KL regularization and generated prefixes enter the objective?

3 comments
AuthorsOpenReview ↗

Response to reviewer C6E9 (Part 1)

First, we would like to thank the reviewer for recognizing the practical significance of our work as well as their constructive feedback. Below, we would like to address the weaknesses and questions posed by the reviewer:

Weaknesses:

W1: The central theoretical justification does not establish historical-knowledge preservation

"Theorem 4.1 proves that Null(Z0)⊆Null(W)\mathrm{Null}(Z_0) \subseteq \mathrm{Null}(W), but this inclusion does not imply ΔPZ0=0\Delta PZ_0=0 for updates projected onto the weight null space. ...This is a fundamental concern about the method’s justification rather than a minor proof issue."

We appreciate the reviewer noting there are no issues with the proof correctness under the given assumptions. We would like to point out that using a projection to Null(W)\mathrm{Null}(W) is in itself a well-motivated design: it uses the underutilized subspace within the model weights, without the need for access to or preprocessing of historical data. Theorem 4.1 links our approach to the null space approach of AlphaEdit [2], providing an additional justification for its use. We believe that our method's success despite this discrepancy on the null spaces makes reporting our empirical results to the community even more important.

Furthermore, we agree that the theorem does not guarantee that ΔPZ0=0\Delta P Z_0 = 0. We further address this concern in our response to the reviewer's Q1 below (see Theorem R1).

W2: The evidence for lifelong retention is currently ambiguous

(1). Evaluation Timing:

"Equations (14)-(16) appear to evaluate each edit using fθtf_{\theta_t}, potentially immediately after that edit is inserted."

The fθtf_{\theta_t} in equations (14)-(16) is a notational mistake and we appreciate the reviewer for the keen eye. Rest assured, the code and all the reported numbers are using the sequential lifelong editing scenario: the final model fθτf_{\theta_\tau} is used for evaluation after all edits are made. We provide further edit retention results in the response to the reviewer's Q2 down below. We will fix the notational mistake in the revised manuscript.

(2). Standard Error for 1000 edits

"The absence of variances over independent edit orders for the longest and most important experiments further limits confidence in the reported improvements."

We appreciate the reviewer's concern about standard error bars over random ordering for τ=1000\tau=1000. Table R1 reports averages with variance over three random edit orders for LOKI, which is less than 0.005 points of difference from Table 1. In short, the reported results are not significantly different from the results over multiple orderings and LOKI is not very sensitive to edit order. We will add these results to the paper.

Table R1. Final-model performance with standard error using three random edit orders, 1000 ZsRE edits, Llama-3-8B-Instruct

methodRelGenLocAvg
LOKI.914±.003.872±.004.868±.003.885±.001
AuthorsOpenReview ↗

Response to reviewer C6E9 (Part 2)

Questions

Q1: Theoretical Justification

"Can the authors derive a condition under which the proposed update provably satisfies ΔPZ0=0\Delta P Z_0 = 0, or provide a quantitative bound on ∣ΔPZ0∣|\Delta P Z_0| and the resulting historical-output drift?"

To address the reviewer’s concern, we provide additional theoretical results in the form of an upper bound on the expected drift on historical data, as well as an empirical measurement of LOKI as compared to a random subspace projection.

(1). Bound.

Theorem R1: Let Z0=[z1,...,zn]Z_0 = [z_1,...,z_n] be the historical features and C0=1nZ0Z0⊤C_0=\frac1n Z_0Z_0^\top. If every row of weight update Δ\Delta lies in subspace S\mathcal S, then

1n∥ΔZ0∥F2 ≤ λmax⁡(PSC0PS)∥Δ∥F2,λmax⁡(PSC0PS)=max⁡u∈S, ∥u∥=11n∥Z0⊤u∥2,\frac1n\lVert\Delta Z_0\rVert^2_F\ \le\ \lambda_{\max}(P_{\mathcal S}C_0P_{\mathcal S}) \lVert\Delta\rVert^2_F, \qquad \lambda_{\max}(P_{\mathcal S}C_0P_{\mathcal S})=\max_{u\in\mathcal S,\ \lVert u\rVert=1}\frac1n\lVert Z_0 ^\top u\rVert^2,

where PSP_\mathcal{S} is the orthogonal projection matrix to subspace S\mathcal{S}, and C0=1nZ0Z0⊤C_0=\frac1n Z_0Z_0^\top. The above bound follows a simple proof which we will provide in the revised appendix (it will be Section B). Theorem R1 states that the drift is in fact bounded by the largest eigenvalue of the projected covariance matrix. However, the above bound holds for any arbitrary subspace S\mathcal S rather than the weight null-space. Therefore, in the section below we compare the effectiveness of the weight null-space with a random subspace of the same dimension.

(2). Editing Results Ablation on Subspace Projection.

In order to disentangle the effect of the weight null-space projection, we run LOKI for 1000 sequential edits with three different projection settings: 1. No projection 2. Projection into a uniformly sampled 1000-dim subspace 3. 1000-dim subspace of Null(W)\mathrm{Null}(W) (Canonical LOKI). The results are presented in Table R2. We can see that as expected, the biggest gain is in the Loc. metric. Both the random and LOKI projection have much higher Loc. than no projection, which is in line with the bound in part (1), as a smaller subspace can only lead to smaller eigenvalues. Additionally, we can see a significant improvement for LOKI over the random subspace, motivating the usage of the weight null-space for historical data preservation. Finally, both LOKI and random exhibit gains in Rel. and Gen. as well, which we hypothesize is due to less interference with earlier edits in the sequential editing scenario. We will add the new bound and table to the final version.

Table R2. 1000 sequential ZsRE edits, Llama-3-8B-Instruct, final-model evaluation

Write subspaceRelGenLoc
None (no projection).857.834.479
Random 1000-dim subspace.899.864.840
LOKI (1000-dim subspace of Null(W)\mathrm{Null}(W)).916.874.879

Regarding the approximate initialization assumption, it is a common assumption (see [1], page 397). The new drift bound (Theorem R1) does not require it.

AuthorsOpenReview ↗

Response to reviewer C6E9 (Part 3)

Q2. Evaluation Timing

"In Tables 1–3, is each edit evaluated only immediately after its own insertion using fθtf_{\theta_t}, or are all earlier edits re-evaluated using the final model fθτf_{\theta_\tau}?"

As mentioned in the weaknesses section, the case is the latter.

"Can the authors provide final-checkpoint reliability and generalization over all previous edits, as well as retention as a function of edit distance?"

All metrics in Tables 1–3 are computed on the final model fθτf_{\theta_\tau} after all edits, so they already measure retention of earlier edits. For retention as a function of edit distance, we provide Table R3 which breaks final-model Rel down by insertion position. We can see that, while there is some degradation for previous edits, as is expected in the lifelong scenario, there isn't a huge gap between newer edit performance and older edit retention, meaning LOKI is capable of retaining earlier edits in lifelong editing.

Table R3. Final-model Rel by insertion position, 1000 ZsRE edits, Llama-3-8B-Instruct

Write subspaceEdits 1–250251–500501–750751–1000
LOKI.90.92.91.94

Q3. Implemented Optimization

"What is the exact update used in the experiments? In particular, how are Adam, gradient projection, optimizer state, and gradient clipping ordered, and how do they preserve the constraint in Equation (8)? Does the implementation enforce a cumulative constraint ∣Δ∣F≤η|\Delta|_F \le \eta, and where do the KL regularization and generated prefixes enter the objective?"

The per-step objective is

1∣G∣∑g∈GCE(Ot∣[g,Xt])+β KL(fθ(⋅∣st) ∥ fθt−1(⋅∣st))−∑l∈S∗HIB(Xt,l),\frac{1}{|G|}\sum_{g\in G}\mathrm{CE}\big(O_t\mid[g,X_t]\big)+\beta\,\mathrm{KL}\big(f_\theta(\cdot\mid s_t)\,\|\,f_{\theta_{t-1}}(\cdot\mid s_t)\big)-\sum_{l\in S^*}\mathrm{HIB}(X_t,l),

where GG is the edit prompt with a set of generated prefixes, sts_t is the prompt "{subject} is a", and β\beta is kl_factor (Table 8).

For standard optimizers such as Adam, each LOKI step runs one Adam step (opt.step()). The resulting weight change Δ\Delta is then replaced by ΔP\Delta P and norm-clipped before it is applied; Adam's state is not modified. Because the projection acts on the update actually written to WW, Wt−1Δ⊤=0W_{t-1}\Delta^\top=0 holds exactly at every step, up to floating-point error. We will make sure to elaborate further on these details in the final version.

References

[1] Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. "The elements of statistical learning." (2009): 337-387.

[2] Fang, Junfeng, et al. "Alphaedit: Null-space constrained knowledge editing for language models." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Wang, Peng, et al. "Wise: Rethinking the knowledge memory for lifelong model editing of large language models." Advances in Neural Information Processing Systems 37 (2024): 53764-53797.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.