- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Current knowledge editing methods edit predefined, fixed model layers, which limits their flexibility. Updating the same layers repeatedly may also cause catastrophic forgetting. To address these challenges, the paper proposes LOKI. LOKI dynamically selects editing layers using the Hilbert–Schmidt Independence Criterion (HSIC) and projects weight updates onto the weight null space, enabling more flexible knowledge editing.
Critical strengths
The paper clearly identifies the limitations of fixed editing layers in current knowledge editing methods and proposes a proxy metric to effectively identify which layers to edit.
The proposed LOKI algorithm achieves strong editing performance, demonstrating its effectiveness.
Critical weaknesses
LOKI requires more optimizer parameters than previous methods. Although it achieves strong editing performance, its increased computational cost departs somewhat from one of the original goals of knowledge editing, which is to modify model behavior at a low computational cost.
LOKI performs SVD on the weight matrices of the selected layers before each edit. Although its editing time is comparable to baseline methods at the 7B scale, the computational cost of SVD will become more substantial as model size increases.
The paper lacks parameter sensitivity experiments, particularly for the hyperparameters newly introduced in HIB.
There are some issues with the presentation, such as an unresolved cross-reference on line 858.
Questions for authors
For a single edit using LOKI, how much time is spent on layer selection, SVD, and gradient updates respectively?
How do changes in the values of the hyperparameters newly introduced in LOKI affect editing performance?
How does LOKI perform on the CounterFact dataset?
On Llama-3, the selected layers are concentrated in layers 22–26. How would LOKI perform if these layers were fixed for editing instead of being selected dynamically?
5 comments
Response to Reviewer ocUp (Part 2)
W2: SVD cost at larger model scale
"LOKI performs SVD on the weight matrices of the selected layers before each edit. Although its editing time is comparable to baseline methods at the 7B scale, the computational cost of SVD will become more substantial as model size increases."
We agree with the reviewer that the SVD cost will grow with the size of the edited down-projection matrices. However, we would like to note that it does not grow with the total size of the model, but only with the size of these matrices. Due to layer selection, LOKI computes the SVD only for the selected layers per edit, not for all layers, so adding depth to a model does not increase the cost. Furthermore, LOKI decomposes the down-projection directly, where is the hidden size and the FFN intermediate size. This is cheaper than the covariance decomposition used by AlphaEdit, although LOKI performs it per edit rather than once during preprocessing. At the 7B/8B scale, the SVD is already included in our timing results (Table 6), and we break it down per stage in Q1 below.
To quantify this scaling, we timed LOKI's per-layer null-space SVD (torch.linalg.svd, full matrices, float32) on the real FFN down-projection of one layer of Llama-3-8B, Llama-2-13B and Llama-3-70B (Table R2). From 8B to 70B, the SVD cost grows by 5.8×, which is less than the ≈8.8× growth in parameter count that drives the forward/backward cost of the optimization step (and of every gradient-based baseline). We therefore expect the SVD's share of the edit time to stay roughly constant or decrease at larger scale. For reference, AlphaEdit's one-time projector requires an SVD of a covariance per edited layer (29.5 s at 8B, 236.7 s at 70B), in addition to collecting the covariance statistics themselves.
We will include this scaling analysis in the revised manuscript.
Table R2. Time and memory of LOKI's null-space SVD on a single FFN down-projection.
| Model | down_proj shape () | LOKI SVD per layer (s) | Per edit, 3 layers (s) | Peak mem (GB) |
|---|---|---|---|---|
| Llama-3-8B | 4096 × 14336 | 1.12 | 3.36 | 1.77 |
| Llama-2-13B | 5120 × 13824 | 2.01 | 6.04 | 2.00 |
| Llama-3-70B | 8192 × 28672 | 6.46 | 19.39 | 7.11 |
W3: Hyperparameter sensitivity
"The paper lacks parameter sensitivity experiments, particularly for the hyperparameters newly introduced in HIB."
We would first like to point the reviewer to the existing sensitivity results in the paper. Table 9 sweeps the null-space dimension null_dim, showing that performance is flat over [500, 2000] (0.886–0.890 Avg) and that all values used in Table 8 lie within 0.010 of the optimum. The ablation in Table 5 covers the setting where the HIB regularization coefficients are set to zero ( in the objective, i.e. "Reg." removed), as well as replacing HIB layer selection with fixed layers. Q4 below further studies layer selection.
For the hyperparameters newly introduced in HIB, we provide a sensitivity analysis for the ratio and for the bandwidth of the Gaussian kernel (ZsRE, Llama-3-8B, 1000 sequential edits, all other hyperparameters fixed at their Table 8 values). Recall that HIB scores each layer as
Since the common factor scales the scores of all layers equally, the top- ranking is invariant to jointly rescaling , so for layer selection the ratio is the only quantity that matters. A ratio of 1 and correspond to the default values reported in the paper.
As shown in Tables R3 and R4, across a 256× range of the ratio and a 100× range of , average performance varies by at most 0.4 and 0.7 points, respectively. LOKI is therefore not very sensitive to the choice of these hyperparameters and only experiences minor fluctuations in average performance.
We will include these sweeps in the revised manuscript, next to Table 9.
Table R3. Sensitivity to the ratio (ZsRE, Llama-3-8B, 1000 sequential edits).
| 16 | 4 | 1 (default) | 1/4 | 1/16 | |
|---|---|---|---|---|---|
| Avg | .888 | .887 | .890 | .886 | .888 |
Table R4. Sensitivity to the HSIC kernel bandwidth (ZsRE, Llama-3-8B, 1000 sequential edits).
| HSIC | 0.1 | 0.5 | 1 (default) | 5 | 10 |
|---|---|---|---|---|---|
| Avg | .888 | .889 | .890 | .883 | .883 |
Response to Reviewer ocUp (Part 3)
W4: Presentation
"There are some issues with the presentation, such as an unresolved cross-reference on line 858."
We thank the reviewer for pointing this out. The unresolved cross-reference on line 858 should point to Table 12, and we will fix it in the revised manuscript.
Questions
Q1: Time breakdown of a single edit
"For a single edit using LOKI, how much time is spent on layer selection, SVD, and gradient updates respectively?"
To answer this, we added timers to each of the three stages of a LOKI edit, following the protocol of Table 6 (seconds per edit, averaged over all edits). Selection is HIB scoring, including the computation of the hidden representations; SVD is the computation of the null-space projectors for the selected layers; and Optimization is the gradient updates.
As shown in Table R5, layer selection with HIB is cheap in every setting (≈0.3–0.5 s, 1–5% of the edit), as it requires only a single forward pass, with no causal tracing or per-layer optimization. The remaining time is split between the SVD and the optimization. The SVD time scales with the number of selected layers at roughly 1.2 s per layer: 3 layers on ZsRE (3.58 s), 5 layers for Mistral-7B on SelfCheckGPT (5.98 s), and 7 layers for Llama-3-8B on SelfCheckGPT (8.34 s). On Temporal, where the targets are long, the optimization dominates (83%) and the SVD accounts for only 16% of the edit time.
We will include this breakdown in the revised manuscript.
Table R5. Time breakdown of a single LOKI edit (seconds per edit, share of the total in parentheses).
| Setting | Selection | SVD | Optimization | Total |
|---|---|---|---|---|
| Llama-3-8B, ZsRE | 0.32 (3%) | 3.58 (37%) | 5.75 (60%) | 9.65 |
| Mistral-7B, ZsRE | 0.34 (5%) | 3.58 (52%) | 2.95 (43%) | 6.88 |
| Llama-3-8B, SelfCheckGPT | 0.43 (3%) | 8.34 (53%) | 6.84 (44%) | 15.63 |
| Mistral-7B, SelfCheckGPT | 0.47 (4%) | 5.98 (55%) | 4.45 (41%) | 10.91 |
| GPT-J, Temporal | 0.30 (1%) | 3.81 (16%) | 20.35 (83%) | 24.47 |
Q2: Hyperparameter sensitivity
"How do changes in the values of the hyperparameters newly introduced in LOKI affect editing performance?"
We refer the reviewer to our response to W3 above.
Q3: CounterFact
"How does LOKI perform on the CounterFact dataset?"
To answer this, we ran τ ∈ {1K, 2K, 3K} sequential edits on CounterFact with Llama-3-8B, comparing LOKI with AlphaEdit, WISE and FT. We would like to emphasize an important difference in the setup. The baselines use their published Llama-3-8B hyperparameters, and AlphaEdit [1], like MEMIT [2] before it, uses CounterFact as one of its primary evaluation benchmarks. In contrast, for LOKI we use the base ZsRE configuration (Table 8) without any CounterFact-specific tuning.
Despite this, the results in Table R6 favor LOKI as the number of edits grows. AlphaEdit attains the best average score at τ = 1K and 2K, mainly through higher generalization and locality. However, its performance degrades quickly as edits accumulate: its reliability drops from .984 to .914 to .377, and its average collapses to .217 at τ = 3K. WISE retains perfect locality but fails to insert most edits (reliability ≤ .158), while FT loses almost all locality (≤ .025).
In contrast, LOKI is the most stable method across the edit sequence. Its reliability stays at ≈.95 for all τ, its average changes by only 0.4 points from 1K to 3K, and at τ = 3K it attains both the highest reliability and the highest average score. This mirrors the trend we observe on ZsRE (Figure 3), where the gap between LOKI and the baselines grows with the number of edits.
We note that LOKI's locality on CounterFact is lower than on ZsRE and decreases with τ (.200 to .116). We will include these results in the revised manuscript and discuss locality on CounterFact as a limitation and a direction for future work.
Table R6. Sequential editing on CounterFact, Llama-3-8B. Bold is the best average for each τ.
| Method | τ | Rel | Gen | Loc | Avg |
|---|---|---|---|---|---|
| FT | 1K | .815 | .542 | .025 | .460 |
| WISE | 1K | .158 | .087 | 1.000 | .415 |
| AlphaEdit | 1K | .984 | .494 | .304 | .594 |
| LOKI | 1K | .952 | .276 | .200 | .476 |
| FT | 2K | .809 | .530 | .019 | .453 |
| WISE | 2K | .120 | .069 | 1.000 | .397 |
| AlphaEdit | 2K | .914 | .496 | .221 | .544 |
| LOKI | 2K | .949 | .321 | .143 | .471 |
| FT | 3K | .732 | .446 | .023 | .400 |
| WISE | 3K | .105 | .062 | 1.000 | .389 |
| AlphaEdit | 3K | .377 | .214 | .060 | .217 |
| LOKI | 3K | .950 | .349 | .116 | .472 |
Response to Reviewer ocUp (Part 1)
We thank the reviewer for the careful reading and constructive feedback, and for recognizing that LOKI identifies an important limitation of fixed-layer editing and achieves strong performance. Below, we address the weaknesses and questions raised by the reviewer. Unless stated otherwise, all new experiments use the following protocol: Llama-3-8B, sequential single edits, a single NVIDIA H200 GPU.
Weaknesses
W1: Optimizer parameters and computational cost
We would first like to clarify that LOKI does not use any additional parameters or memory at inference time. After each edit, the model is simply the base model with updated weights in the selected layers: there are no adapters, side memories, codebooks or routing modules. The parameters optimized during an edit and their Adam state exist only while that edit is performed and are freed afterwards, so they do not accumulate over sequential edits.
How we report the cost of an edit. We measure the computational cost of a single edit along three axes:
Results. In terms of memory, LOKI's peak during an edit (35.4 GB) is lower than that of the locate-then-edit methods MEMIT (37.9 GB), AlphaEdit (37.0 GB) and SimIE (36.2 GB), and the whole edit fits comfortably on a single 80 GB GPU. In terms of how much of the model an edit changes, LOKI (176M, three selected layers) sits between ROME (58.7M) and MEMIT/AlphaEdit (293.6M). The larger optimized-parameter count relative to locate-then-edit methods is paid for by the transient Adam state (1.31 GB), which does not translate into higher memory than MEMIT, AlphaEdit or SimIE.
In terms of time, averaged over the five dataset/backbone settings of Table 6, LOKI takes 15.0 s per edit, which is faster than AlphaEdit (16.1 s), ROME (17.3 s), WISE (23.5 s) and SimIE (27.6 s), i.e. every baseline that is competitive in our benchmarks. Moreover, these numbers exclude the one-time preprocessing required by MEMIT and AlphaEdit, which takes ∼25 h on an H200 for Llama-3-8B (Table 7) and must be repeated for every new model or change of edited layers. Figure 1(b) shows the same trade-off visually: LOKI attains the lowest perplexity at comparable per-edit runtime. We therefore believe LOKI is in line with the goal of low-cost knowledge editing, and in fact lowers the total cost compared to the strongest baselines. We will include Table R1 and this discussion in the revised manuscript.
Table R1. Cost of a single edit. Time per edit is the average over the five dataset/backbone settings of Table 6. Parameter counts and peak memory are measured on Llama-3-8B, ZsRE. Peak memory includes the model weights.