acceptodds
Under review as a conference paper at ICLR 2027

Do We Really Need Hypernetworks for Sequential Meta Model Editing?

Abstract

Meta-learning-based model editors commonly employ a learned transformation, often implemented by a hypernetwork, to transform the first-step edit gradient before constructing the final parameter update. We ask how much these learned transformations actually contribute to sequential meta model editing. Across multiple meta editors, language models, and datasets, we find limited additional benefit from the learned transformation: untransformed updates remain highly competitive and can even outperform their transformed counterparts. Further analysis shows that much of the editing capability is already supported by the underlying update formulation. Motivated by these findings, we propose RawEdit, a hypernetwork-free meta editor that directly uses the raw first-step gradient and strengthens the update formulation through a direct residual and history-aware regularization. RawEdit learns only two scalars per edited layer to control edit strength and regularization while explicitly accounting for previously edited directions. Extensive experiments show that RawEdit consistently achieves strong editing performance across models and datasets, outperforming existing editors by up to 20% in efficacy, generality, and locality, while substantially reducing computational cost and remaining stable under long-term sequential editing with up to edits. Moreover, compared with subject-based editors, RawEdit effectively mitigates superficial editing and remains applicable to challenging settings where reliance on subject representations imposes inherent limitations. Code is available at https://anonymous.4open.science/r/RawEdit-760D.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Language Model, Knowledge Editing, Meta Learning Model Editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZXK3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates a key question in sequential meta-learning-based model editing: to what extent do existing editors owe their performance to edit-signal transformations, such as hypernetworks, rather than to the subsequent parameter update rules? To address this question, the authors disentangle edit-signal transformation from parameter update construction. Through transformation removal, update-direction analysis, editing-location experiments, and ridge-regression ablations, they find that untransformed edit signals remain competitive in the evaluated sequential editing settings, while editing-layer selection and ridge-based updates substantially influence generality and locality.

Motivated by these observations, the authors propose RawEdit, a hypernetwork-free editor that directly uses the raw output gradient, scaled by a learned coefficient, as the desired residual. It constructs a closed-form update that combines current-edit fitting, protection of previously edited directions, and weight-perturbation regularization. Only two scalars are learned per edited layer to control edit strength and regularization.

Experiments cover four language models and two datasets, ZSRE and WikiBigEdit, along with evaluations of general capability preservation, multi-hop generalization, same-subject editing, long-form editing, and superficial editing under attack. The results show that RawEdit achieves strong performance in multiple configurations and remains competitive with more complex editors while using substantially fewer learnable parameters.

Critical strengths
  1. The research question is clear and thought-provoking. The paper reexamines whether a core component of existing meta-learning-based editors is necessary. Separating edit-signal transformation from parameter update construction helps identify the contributions of individual design choices, rather than attributing the performance of the entire framework to the hypernetwork. This perspective provides useful guidance for future editor design.

  2. The method is simple, with clearly motivated components. RawEdit’s objective explicitly addresses three requirements: fitting the current edit, protecting historical directions, and controlling update magnitude. The resulting design is easy to understand, and learning only two scalars per layer substantially simplifies the trainable editor.

Critical weaknesses
  1. The methodological novelty is relatively limited. RawEdit’s methodological contribution primarily lies in simplifying and combining existing update formulations. Ridge-based updates and constraints incorporating editing history already appear in the prior methods discussed in the paper, and RawEdit follows RLEdit’s meta-training strategy. While this simplification has practical value, the methodological advance is relatively incremental. The paper could more clearly articulate the distinctive contribution of direct residual fitting and its combination with history-aware regularization, while distinguishing these methodological contributions from the empirical analysis.

  2. The theoretical and mechanistic explanations remain limited. Although the paper provides an optimization objective and a closed-form update, its central conclusions are primarily supported by empirical observations. In particular, it does not systematically explain why learned transformations provide limited benefits in sequential editing but offer clear advantages in independent single-edit settings with MEND-style updates. The authors acknowledge this limitation in the conclusion. Further analysis of the conditions underlying this contrast and its relationship to the update rule would help clarify the scope of the findings and strengthen the paper’s theoretical depth.

Questions for authors
  1. Performance gains involve metric trade-offs, and some claims require clearer support. RawEdit does not outperform the baselines across all settings and metrics. For example, in Table 3, on Qwen3-8B with ZSRE, RawEdit achieves ES/GS/LS scores of 80.9/73.3/67.7, compared with 97.8/89.2/54.4 for AlphaEdit. This represents a trade-off between better locality and lower efficacy and generality, rather than an across-the-board improvement. The claim in Section 6.2 that RawEdit consistently achieves the strongest overall editing performance across models and datasets therefore requires an explicit criterion for assessing overall performance, or should be stated more cautiously.

  2. Evaluation of large-scale joint editing within a single batch remains limited. Although the paper reports sequential editing results with 20,000 and 17,000 accumulated edits, the main experiments use a fixed batch size of 100. The same-subject editing experiments also use batches of 100 edits. The evaluation does not cover substantially larger single batches, such as jointly applying 1,000 or 10,000 edit requests. Comparing performance and resource requirements across different batch sizes would help establish RawEdit’s scalability to large-scale joint editing.

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewers for their valuable and constructive feedback. Our point-by-point responses follow below.

W1. The methodological novelty is relatively limited...

We thank the reviewer for raising this point. We refer the reviewer to our response to W2&Q2 raised by reviewer cafY, where we provide a detailed discussion of the methodological contributions of RawEdit and its distinctions from existing methods.

W2. The theoretical and mechanistic explanations remain limited.

Reviewer cafY also raises this point. We refer the reviewer to our response to W3&Q3 of cafY for a related discussion. There, we hypothesize that, under sequential editing, continuously evolving model states introduce evolving, history-dependent edit signals, making it more challenging to learn non-trivial transformations. To improve learning stability under sequential editing, recent works have introduced various sequential stabilization techniques, including memory backtracking, lifelong normalization, and whitening [1,2,3,4]. Interestingly, we find that all these methods exhibit the transformation degeneration phenomenon identified in our work. In addition to RLEdit, UltraEdit, and StableEdit evaluated in our paper, we have the following additional results for EMSEdit [4], another sequential meta-editor based on hypernetwork transformations.

DatasetModelESGSLSUpdate Cosine
ZSREGPT2-XL62.255.072.899.42
ZSREGPT-J-6B76.070.150.298.91
ZSRELLaMA3.1-8B87.183.226.599.18
ZSREQwen3-8B80.676.051.498.67
WikiBigEditGPT2-XL58.552.650.898.73
WikiBigEditGPT-J-6B69.364.032.699.56
WikiBigEditLLaMA3.1-8B73.067.133.897.84
WikiBigEditQwen3-8B66.860.736.799.31

As shown, although EMSEdit achieves strong ES and GS performance, comparison with the untransformed baseline in Table 1 shows that directly using raw edit signals remains competitive and can even yield better results. More importantly, EMSEdit exhibits high cosine similarity between transformed and untransformed updates, further reinforcing our finding of transformation degeneration in state-of-the-art sequential meta-editors. These observations, together with RawEdit's superior performance (Table 3, Figure 3), make us reconsider how much we actually gain from hypernetwork transformations in sequential meta-editing.

As the reviewer points out, transformation degeneration is currently only an empirical finding. While it is useful and deserves further attention from the research community, we have not yet theoretically characterized its underlying mechanism. Understanding how stabilization techniques for sequential editing relate to the degeneration of learned transformations therefore remains an open question.

This question is particularly challenging because model states and the resulting edit-signal distributions continuously evolve with the editing history, making the learning dynamics of hypernetworks difficult to characterize. StableEdit provides a theoretical explanation of how lifelong normalization, combined with ridge-regularized regression, promotes stable sequential updates through bounded update norms and asymptotic orthogonality. We believe that this theoretical framework may provide a useful starting point for further investigating the relationship between learning stability and transformation degeneration.

[1] Reinforced Lifelong Editing for Language Models, ICML 2025.
[2] UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models, TMLR 2026.
[3] More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing, ICML 2026.
[4] EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing, WWW 2026.

Q1. Performance gains involve metric trade-offs, and some claims require clearer support.

We thank the reviewer for pointing this out. We agree that RawEdit does not outperform AlphaEdit on Qwen3-8B across all metrics, where AlphaEdit achieves higher ES and GS, whereas RawEdit achieves higher LS. This reflects a trade-off among editing efficacy, generality, and locality on Qwen3-8B, rather than an across-the-board improvement.

We have revised the statement in Section 6.2 (page 7) to avoid overstating our results and more accurately characterize RawEdit's performance.

(Line 373)

RawEdit consistently achieves the strongest overall editing performance across models and datasets.

Revised:

RawEdit achieves the highest overall editing performance (measured by ES+GS+LS) on three of the four models (GPT2-XL, GPT-J-6B, and LLaMA3.1-8B), and remains competitive on Qwen3-8B.

AuthorsOpenReview ↗

Author Response (2/2)

Q2. Evaluation of large-scale joint editing within a single batch remains limited.

We thank the reviewer for raising this point. We use a batch size of 100 for two reasons. First, this follows the sequential editing protocol adopted by our baselines, including UltraEdit, StableEdit, AlphaEdit, and SPHERE. Second, this setting allows us to examine long-term sequential editing through a sufficiently large number of consecutive model updates. Specifically, the 20k ZSRE edits and 17k WikiBigEdit edits are processed through 200 and 170 sequential updates, respectively. Using substantially larger batches would reduce the number of model updates, thereby providing fewer opportunities to evaluate the accumulation of editing effects over a long editing sequence.

Nevertheless, following the reviewer's suggestion, we investigate the effect of batch size on RawEdit using 10, 20, 100, 500, 1,000, and 10,000 edits per batch. We conduct these experiments across four model-dataset combinations: LLaMA3.1-8B and Qwen3-8B on ZSRE and WikiBigEdit. Furthermore, using LLaMA3.1-8B on WikiBigEdit (17K edits) as an example, we compare RawEdit with AlphaEdit, RLEdit, StableEdit, and UltraEdit under different batch sizes. In this comparison, we report ES, GS, LS, average time per edit (in seconds), and GPU memory consumption (in GB) during inference and training. All experiments are conducted on a single NVIDIA A100 GPU with 80 GB of memory, with all LLMs loaded in bfloat16 precision. The results are reported in Appendix D.2 of the revised submission (page 25).

As shown in Table 14 (page 26) and Figure 10 (page 24), editing performance generally improves with increasing batch size, but can deteriorate when the batch size becomes excessively large. For example, on LLaMA3.1-8B with WikiBigEdit (Table 14), RawEdit achieves its best ES (86.9) at a batch size of 1,000, compared with 80.6 at a batch size of 10. However, further increasing the batch size to 10,000 leads to a substantial drop in ES/GS (65.2/60.0). Similar trends can be observed across other baselines. This behavior can be explained by two competing factors: excessively small batches lead to frequent model updates and potential over-editing [1], whereas excessively large batches increase interference among edits within the same batch, potentially hindering effective editing [2].

To alleviate the latter issue, we apply multiple consecutive updates to each large batch. Specifically, after computing and applying an update using the current batch, we recompute the update on the same batch using the already updated model [3,4], repeating this process for the specified number of updates before proceeding to the next batch. We evaluate 1, 2, and 3 consecutive updates per batch with a batch size of 10,000. The results are reported below.

MethodNumber of UpdatesESGSLSTime Per EditGPU Memory (inference)GPU Memory (training)
RawEdit165.260.080.60.063434
RawEdit284.474.558.90.123434
RawEdit378.270.752.50.183434

As shown, applying two updates substantially improves RawEdit's ES/GS from 65.2/60.0 to 84.4/74.5, while reducing LS from 80.6 to 58.9. This demonstrates that multiple updates can help mitigate the editing interference associated with large batches, although they also increase the computational cost.

Regarding resource requirements, meta-editors are faster than subject-based editors (e.g., AlphaEdit) because they directly construct weight updates from first-step gradient signals, whereas subject-based editors require iterative optimization to obtain the desired key-value pairs. Smaller batch sizes generally incur higher time per edit, as the weight update formulation needs to be computed and applied more frequently. Larger batches require more GPU memory because the update formulation involves larger matrices. For the training overhead of meta-editors, we refer to Table 11 (page 22) for a detailed comparison. RLEdit requires substantial memory to train the hypernetwork, while StableEdit maintains statistical information for whitening. Like AlphaEdit, RawEdit maintains historical Gram matrices to preserve previously edited knowledge. Importantly, history-aware regularization (HAR) is a plug-and-play component that can also be incorporated into other baselines, as demonstrated by our experiments in Table 13 (page 25).

[1] Model editing harms general abilities of large language models: Regularization to the rescue, EMNLP 2024.
[2] EAMET: Robust Massive Model Editing via Embedding Alignment Optimization, ICLR 2026.
[3] EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing, WWW 2026.
[4] Rethinking residual distribution in locate-then-edit model editing, NeurIPS 2025.

Dear Reviewer ZXK3,

We sincerely appreciate your positive assessment of our work and your recognition of its contributions. In particular, we are grateful that you found our investigation into the necessity of learned gradient transformations in sequential model editing thought-provoking, and recognized the value of disentangling edit-signal transformations from parameter update construction. We also appreciate your recognition of RawEdit's simplicity, clearly motivated components, and potential to provide useful guidance for future editor design.

Thank you also for your constructive questions, which have helped us further clarify and improve the paper. As the reviewer-author discussion period is approaching its deadline, for your convenience, we briefly summarize our responses:

  1. Methodological contributions (W1): We clarified the distinctions between RawEdit and existing methods, particularly MEMIT/AlphaEdit and MALMEN, highlighting the differences in residual construction, historical regularization, token-wise gradient selection, and the removal of hypernetwork transformations. We refer to our detailed discussion in response to W2&Q2 of reviewer cafY.

  2. Theoretical and mechanistic explanations (W2): We further discussed our hypothesis that evolving model states and history-dependent edit signals make non-trivial transformations more challenging to learn during sequential editing. We also provided additional experimental results on EMSEdit, which further support the transformation degeneration phenomenon, and discussed how StableEdit's theoretical analysis may provide a starting point for future investigation.

  3. Performance trade-offs (Q1): We acknowledged the trade-off between efficacy, generality, and locality on Qwen3-8B and revised Section 6.2 to clarify that RawEdit achieves the highest overall performance, measured by ES+GS+LS, on three of the four models, while remaining competitive on Qwen3-8B.

  4. Large-scale batch editing (Q2): Following your suggestion, we evaluated batch sizes ranging from 10 to 10,000 across four model-dataset combinations. We additionally compared RawEdit with AlphaEdit, RLEdit, StableEdit, and UltraEdit under different batch sizes, investigated multiple consecutive updates for batches of 10,000 edits, and reported the corresponding performance and resource requirements in Appendix D.2.

Given the importance of revealing the degeneration phenomenon of hypernetwork transformations in existing sequential meta-editing and its implications for future research, we hope our findings and the proposed RawEdit can provide useful insights to the community. If our clarifications have addressed your concerns and you also believe that our work can provide useful insights for future research on sequential model editing, we would greatly appreciate it if you would consider giving our work a higher rating.

Thank you again for your time and constructive feedback. We look forward to hearing from you.

Reviewer_o7TPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits a fundamental design choice in sequential meta-learning-based model editing: whether a learned transformation, typically implemented as a hypernetwork, is actually necessary for converting the first-step edit signal into effective parameter updates. Through a series of controlled analyses across multiple editors, language models, and datasets, the authors show that untransformed updates can remain highly competitive with their transformed counterparts, suggesting that much of the editing capability may instead come from the underlying update formulation. Motivated by these findings, the paper proposes RawEdit, a hypernetwork-free editor that directly uses the raw gradient signal together with direct residual fitting and history-aware regularization. Experiments on long-term sequential editing, including up to 20,000 edits, show that RawEdit achieves strong efficacy, generality, and locality while substantially reducing editor-side complexity, and remains competitive across several more challenging editing scenarios.

Critical strengths
  1. The research question is important and insightful. While much prior work focuses on designing better hypernetworks for meta-learning-based editing, this paper challenges the more fundamental assumption that such transformations are necessary at all.

  2. The analysis is systematic and experimentally comprehensive. The paper carefully investigates the role of the transformation through performance comparisons, update-geometry analysis, and studies of the underlying update formulation, while also evaluating editing performance across multiple models, datasets, and challenging editing settings.

Critical weaknesses

The main concern is that ES and GS, which are core metrics for evaluating editing effectiveness and generalization, are computed using teacher-forced generation rather than autoregressive generation. Although Appendix A.1 defines ES/GS in terms of sequence-level prediction correctness, inspection of the released code suggests that the actual implementation uses teacher forcing together with token-level accuracy. This exposes the model to ground-truth target prefixes that would not be available at inference time and, as discussed in The Mirage of Model Editing [1], can lead to substantially overestimated editing performance. It would therefore be helpful for the authors to clarify this discrepancy and, ideally, re-evaluate the main results using autoregressive decoding.

[1] The Mirage of Model Editing: Revisiting Evaluation in the Wild. ACL 2025.

Questions for authors

How does RawEdit compare with a properly implemented fine-tuning baseline? The current RegFT baseline performs poorly, whereas Fine-tuning Done Right in Model Editing [2] shows that appropriately implemented fine-tuning can substantially outperform methods such as AlphaEdit, RLEdit, and UltraEdit. Given this paper's own observation that raw gradient signals already contain strong editing information, comparing RawEdit with a stronger fine-tuning baseline would help clarify the actual benefit of its update formulation.

[2] Fine-tuning Done Right in Model Editing. ICLR 2026.

6 comments
AuthorsOpenReview ↗

Author Response (1/2)

Q1

We thank the reviewer for raising this interesting question. As the reviewer rightly noticed, RawEdit looks very much like fine-tuning. RawEdit uses raw edit signals without transformation. If we further remove the ridge and history-aware regularization from RawEdit's update formulation, the resulting update becomes closely related to standard fine-tuning, with only one remaining difference in token-wise gradient selection (see our post Background on Token-wise Gradient Selection for details).

So, how LocFT-BF proposed in Fine-tuning Done Right in Model Editing acutually performs? Upon inspecting their code, we found that LocFT-BF does not follow sequential editing but rather adopts a batch-editing strategy. Specifically, fine-tune.py (Lines 856–874) shows that the training loop repeatedly traverses the entire dataset over multiple epochs, reshuffling and revisiting all previously processed edits. That is, their setting does not distinguish between current and historical edits, as all edits remain accessible and are repeatedly optimized. This is fundamentally different from the sequential editing setting in existing works, such as AlphaEdit, BetaEdit, SPHERE, RLEdit, UltraEdit and StableEdit, where historical edits are no longer accessible.

To align LocFT-BF with the sequential editing setting, we perform only one gradient update per batch before proceeding to the next batch, rather than performing multiple gradient updates within each batch as in RegFT. This allows a direct comparison with RawEdit, which also uses a single gradient update per batch. For LocFT-BF, we search over four learning rates (1e-4, 5e-4, 1e-3, and 5e-3) and evaluate it across four models and two datasets. The results are summarized below.

GPT2-XL

DatasetMethodLRESGSLS
WikiBigEditLocFT-BF1e-416.614.788.1
5e-422.319.845.8
1e-323.720.833.6
5e-311.810.19.7
AlphaEdit-18.013.18.0
RLEdit-55.749.857.5
RawEdit-56.149.672.7
ZSRELocFT-BF1e-420.617.887.9
5e-437.832.277.9
1e-344.137.973.6
5e-338.733.639.4
AlphaEdit-21.017.611.8
RLEdit-56.052.179.9
RawEdit-63.958.550.6

Qwen

DatasetMethodLRESGSLS
WikiBigEditLocFT-BF1e-464.961.639.2
5e-462.058.830.0
1e-358.453.422.9
5e-338.035.912.7
AlphaEdit-81.762.033.7
RLEdit-63.757.634.5
RawEdit-70.661.450.9
ZSRELocFT-BF1e-480.973.850.3
5e-485.578.439.3
1e-385.377.331.1
5e-375.769.914.3
AlphaEdit-97.889.254.4
RLEdit-77.873.154.3
RawEdit-80.973.367.7

GPT-J

DatasetMethodLRESGSLS
WikiBigEditLocFT-BF1e-463.559.332.0
5e-454.751.219.9
1e-336.333.814.5
5e-32.42.40.0
AlphaEdit-67.954.530.7
RLEdit-66.260.935.4
RawEdit-81.770.646.3
ZSRELocFT-BF1e-482.378.333.3
5e-474.171.118.4
1e-345.844.313.5
5e-33.43.50.1
AlphaEdit-71.163.260.0
RLEdit-72.167.456.3
RawEdit-87.277.059.9

LLaMA

DatasetMethodLRESGSLS
WikiBigEditLocFT-BF1e-469.765.445.6
5e-463.559.130.2
1e-361.657.024.2
5e-314.413.94.6
AlphaEdit-47.341.221.9
RLEdit-70.064.233.6
RawEdit-85.175.356.3
ZSRELocFT-BF1e-489.783.847.6
5e-491.384.629.8
1e-389.884.220.1
5e-355.050.75.9
AlphaEdit-86.881.233.3
RLEdit-84.480.532.3
RawEdit-88.781.261.1

As shown, LocFT-BF underperforms RawEdit in most settings, particularly in locality, validating the effectiveness of our update formulation under sequential editing. Surprisingly, LocFT-BF performs well on ZSRE with Qwen and LLaMA, even outperforming RLEdit and AlphaEdit. While this highlights the effectiveness of raw edit signals, it also raises a broader concern about the progress of sequential model editing: at least so far, SOTA model editors often even underperform simple approaches such as LocFT-BF and RawEdit.

AuthorsOpenReview ↗

Author Response (2/2)

W1

We thank the reviewer for pointing out this evaluation issue. We clarify that our ES/GS evaluation protocol, including both the mathematical definitions provided in Appendix A.1 of the original submission and the corresponding code implementations, follows those of existing baselines: RLEdit (Table 2, ZSRE), UltraEdit (Table 2, ZSRE and WikiBigEdit), StableEdit (Table 1, ZSRE and WikiBigEdit), AlphaEdit (Table 1, ZSRE), BetaEdit (Table 1, ZSRE), and SPHERE (Table 1, ZSRE). We appreciate the reviewer's insightful comment, which motivated us to further examine editing efficacy and generality under autoregressive decoding. In this setting, an edit is considered successful only when all tokens in the generated sequence exactly match the target sequence. We denote the resulting sequence-level metrics as ES_seq and GS_seq, respectively.

Following the editing settings in our paper, we conduct 20,000 sequential edits on ZSRE and 17,000 sequential edits on WikiBigEdit. We report the resulting sequence-level performance (ES_seq and GS_seq) and compare it with the original token-level results (ES and GS).

Results on ZSRE with LLaMA3.1-8B.

MethodESES_seqGSGS_seqLS
RegFT41.124.239.413.828.1
AlphaEdit86.871.681.258.433.3
BetaEdit83.568.376.754.551.2
SPHERE81.864.175.951.245.1
RLEdit84.469.280.558.032.3
UltraEdit81.366.275.552.162.7
StableEdit86.168.481.257.651.8
RawEdit88.771.281.258.961.1

Results on WikiBigEdit with LLaMA3.1-8B.

MethodESES_seqGSGS_seqLS
RegFT48.026.143.514.836.9
AlphaEdit47.327.041.214.221.9
BetaEdit68.650.352.821.452.6
SPHERE61.839.549.619.747.7
RLEdit70.053.764.238.933.6
UltraEdit76.154.169.240.240.4
StableEdit77.255.369.741.840.9
RawEdit85.161.275.346.756.3

As shown, the two evaluation protocols yield different efficacy and generality scores across all methods. When moving from token-level to sequence-level evaluation, the average efficacy and generality decrease by 16.3 and 23.4 percentage points on ZSRE, respectively, and by 20.9 and 28.5 percentage points on WikiBigEdit. For RawEdit, efficacy/generality change from 88.7/81.2 to 71.2/58.9 on ZSRE (20k edits), and from 85.1/75.3 to 61.2/46.7 on WikiBigEdit (17k edits). Similar changes are observed across the baseline methods.

Notably, RawEdit remains highly competitive under both evaluation protocols. On WikiBigEdit, it achieves the highest sequence-level efficacy and generality, at 61.2 and 46.7, respectively, while also maintaining the highest locality. These results demonstrate the effectiveness of RawEdit under different evaluation protocols. We also note that these autoregressive sequence-level evaluations do not alter the main claims of our paper: existing sequential meta-editors exhibit transformation degeneration, and directly using untransformed edit signals can achieve highly competitive editing performance, even outperforming transformed updates.

We sincerely appreciate the reviewer's insightful comment, which has helped us strengthen the evaluation in our paper. Accordingly, we have revised Appendix A.1 (page 13) to provide detailed mathematical definitions of both teacher-forced token-level and autoregressive sequence-level efficacy and generality, and added Appendix D.7 (page 28) to present their experimental comparison. Specifically, we re-evaluate all eight methods under both evaluation protocols on LLaMA3.1-8B and GPT-J-6B, using both ZSRE and WikiBigEdit. We hope these additional analyses and results can serve as a useful reference for future studies on sequential model editing.

AuthorsOpenReview ↗

Background on Token-wise Gradient Selection (1/2)

Beyond the ridge-regularization and history-aware regularization terms, the remaining difference between RawEdit and standard fine-tuning concerns how token-wise gradient contributions are selected to construct the weight update. To understand this distinction, it is important to distinguish between the token positions used to construct the editing loss and those whose gradient contributions are retained for the weight update. We explain these two aspects in detail below.

1. Editing Loss Construction

Each edit consists of an input sequence xinx_{\mathrm{in}} and a desired output sequence xoutx_{\mathrm{out}}. They are concatenated to form the complete sequence:

x=[xin;xout].(1)x = [x_{\mathrm{in}};x_{\mathrm{out}}]. \tag{1}

To the best of our knowledge, almost all existing model editors, including fine-tuning [1,2,3], locate-then-edit [4,5,6], and meta-editing methods [7,8,9,10,11], construct the editing loss only over the target-output tokens:

Ledit=−∑t∈Tanslog⁡pθ(xt∣x<t),(2)\mathcal{L}_{\mathrm{edit}} = -\sum_{t\in\mathcal{T}_{\mathrm{ans}}} \log p_{\theta}(x_t\mid x_{<t}), \tag{2}

where Tans\mathcal{T}_{\mathrm{ans}} denotes the token positions corresponding to xoutx_{\mathrm{out}}. Therefore, the construction of the editing loss does not distinguish RawEdit from standard fine-tuning.

2. Gradient contribution selection

By backpropagating the editing loss in Eq. (2), the gradient with respect to an edited linear layer W\mathbf W can be decomposed into token-wise outer products [7]:

∇WLedit=∑t=1Tzo,t′zi,t⊤,(2)\nabla_{\mathbf W}\mathcal L_{\mathrm{edit}} = \sum_{t=1}^{T} \mathbf z'_{o,t}\mathbf z_{i,t}^{\top}, \tag{2}

where zi,t\mathbf z_{i,t} and zo,t\mathbf z_{o,t} denote the input and output representations of the edited linear layer W\mathbf W at token position tt, respectively, zo,t′=∂Ledit/∂zo,t\mathbf z'_{o,t}=\partial\mathcal L_{\mathrm{edit}}/\partial\mathbf z_{o,t}, and TT is the total number of tokens in the concatenated sequence xx.

The distinction between RawEdit and standard fine-tuning arises at this stage. Specifically, let Tans\mathcal{T}_{\mathrm{ans}} denote the answer-token positions. RawEdit retains only the outer products associated with these positions:

GRawEdit=∑t∈Tanszo,t′zi,t⊤(3)\boxed{ \mathbf G_{\mathrm{RawEdit}} = \sum_{t\in\mathcal{T}_{\mathrm{ans}}} \mathbf z'_{o,t}\mathbf z_{i,t}^{\top} } \tag{3}

In contrast, standard fine-tuning retains the contributions from all token positions:

GFT=∑t=1Tzo,t′zi,t⊤(4)\boxed{ \mathbf G_{\mathrm{FT}} = \sum_{t=1}^{T} \mathbf z'_{o,t}\mathbf z_{i,t}^{\top} } \tag{4}

3. Weight Update

For RawEdit, we concatenate the input representations and output gradients of all answer-token positions to construct two matrices [8,9,10,11]:

Zi=[zi,1,…,zi,M],Zo′=[zo,1′,…,zo,M′],\mathbf Z_i = [\mathbf z_{i,1},\ldots,\mathbf z_{i,M}], \qquad \mathbf Z'_o = [\mathbf z'_{o,1},\ldots,\mathbf z'_{o,M}],

where M=∣Tans∣M=|\mathcal{T}_{\mathrm{ans}}| denotes the number of answer-token positions. For multiple edits, we concatenate all of them. We then apply Eq. (11) of our paper (Page 7) to construct the final weight update. Note that although zz is written as a vector in Eq. (11) for simplicity, it can also represent a matrix ZZ containing multiple token representations.

For standard fine-tuning, it directly uses the full gradient for weight updates:

ΔWFT=−ηGFT,\Delta\mathbf W_{\mathrm{FT}} = -\eta\mathbf G_{\mathrm{FT}},

where η\eta is the learning rate.

Selecting gradient contributions only from answer-token positions was first introduced in [8] and subsequently adopted by [9, 10, 11] and our work. We generally find that this strategy helps preserve locality during sequential editing. We report RawEdit's performance using all-token gradient contributions in our next official comment post.

[1] Model Editing by Standard Fine-Tuning, ACL 2024.
[2] Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuning, ICLR 2025.
[3] Fine-Tuning Done Right in Model Editing, ICLR 2026.
[4] Locating and Editing Factual Associations in GPT, NeurIPS 2022.
[5] Mass-Editing Memory in a Transformer, ICLR 2023.
[6] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR 2025.
[7] Fast Model Editing at Scale, ICLR 2022.
[8] Massive Editing for Large Language Models via Meta Learning, ICLR 2024.
[9] Reinforced Lifelong Editing for Language Models, ICML 2025.
[10] UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models, TMLR 2026.
[11] More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing, ICML 2026.

Reviewer_cafYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines the contribution of learned gradient transformations to sequential model editing. Identity replacements, update projections, and layer and regularization analyses suggest that the update formulation accounts for much of the observed performance. Motivated by this finding, RawEdit combines raw output-gradient residuals with history-aware ridge regularization, learning two scalars per edited layer. Evaluation covers four language models, up to 20,000 sequential edits, and additional generalization and editing settings. The appendix also identifies a single-edit setting where MEND's learned transformation remains beneficial.

Critical strengths
  1. The component analysis provides a useful empirical insight. Replacing the transformation, removing its orthogonal contribution, and controlling update magnitude when testing ridge regression offer complementary evidence about where editing performance comes from (§4). This makes the simplification informative beyond introducing another editor. The contrasting MEND results in Appendix B.4 appropriately establish a boundary to the finding.
  2. RawEdit translates this insight into a simple and competitive method. Long editing sequences, direct-residual and history ablations, general-capability evaluations, and comparisons with specialized editors provide substantial validation (§6; Appendix D). The relationships to earlier ridge-based and history-preserving approaches are acknowledged. These findings and the practical simplification support acceptance despite the modest novelty of the individual ingredients.
Critical weaknesses
  1. Table 11 lists RawEdit's 14 learned scalars but omits historical Gram matrices maintained in Algorithm 1. While Appendix D.1 acknowledges different parameter types, the comparison should distinguish learned parameters from total persistent state.
  2. The methodological novelty of RawEdit appears incremental relative to existing editing formulations. Its residual-fitting objective, historical-key preservation, and closed-form update closely resemble the MEMIT/AlphaEdit line, while MALMEN already connects gradient-based signals with least-squares editing. These connections are acknowledged in Section 5 and Appendix A.2. A more explicit comparison would clarify the distinct contribution of RawEdit's residual construction and simplification, while keeping the controlled component analysis central to the paper's contribution.
  3. Minor: the central finding lacks a clear mechanistic explanation. Although Section 7 acknowledges this gap, it remains unclear when learned transformations should provide little additional benefit, limiting the finding's usefulness for designing other editors.
Questions for authors
  1. Clarification: how are RawEdit's historical Gram matrices accounted for in Table 11? Please distinguish learned parameters from the total persistent editor state using the current implementation; no additional benchmark is needed.
  2. Clarification: could you explicitly distinguish RawEdit's residual construction and historical regularization from MEMIT/AlphaEdit and MALMEN, and explain what is newly contributed? For the minor mechanism question, please clarify the conditions under which the Section 7 hypothesis should hold, considering Appendix B.4. A short conceptual discussion using the existing analysis would suffice.
3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank you for your valuable and constructive feedback. We provide our point-by-point responses below.

W1&Q1. Clarification: how are RawEdit's historical Gram matrices accounted for in Table 11? Please distinguish learned parameters from the total persistent editor state using the current implementation; no additional benchmark is needed.

Thank you for pointing this out. We agree that Table 11 does not fully reflect RawEdit's persistent state. The reported 14 scalars refer only to its learnable parameters, excluding the historical Gram matrices used for history-aware regularization (HAR). Inspired by AlphaEdit and BetaEdit, we regard HAR as a general, plug-and-play enhancement rather than a RawEdit-specific component (see Tables 3 and 13 for its effectiveness). Consequently, we did not include the storage overhead of HAR in the RawEdit-specific parameter count.

Nevertheless, we acknowledge that the storage cost of HAR should be explicitly reported. The storage cost depends on the number of edited layers, the input dimension of each edited weight, and the numerical precision. In our experiments, we edit seven MLP down-projection layers. For numerical precision, we follow the default BF16 setting in the StableEdit code repository.

The total storage overheads of HAR over the four models are shown below.

ModelInput dimensionStorage
LLaMA3.1-8B14,3362.68 GiB
Qwen3-8B12,2881.97 GiB
GPT-J-6B16,3843.50 GiB
GPT2-XL-1.5B6,4000.53 GiB

As shown above, the storage cost is primarily determined by the input dimension, growing quadratically with it. Nevertheless, the overhead remains manageable for modern LLMs. For example, Qwen3.6-27B has an input dimension of 17,408, for which the historical Gram matrices across seven layers require only approximately 3.95 GiB.

We have revised Appendix D.1 (page 24) to report the storage overhead of HAR across the four models and clarify its distinction from RawEdit's learnable parameters. We sincerely thank the reviewer for raising this important point.

W3&Q3. For the minor mechanism question, please clarify the conditions under which the Section 7 hypothesis should hold, considering Appendix B.4. A short conceptual discussion using the existing analysis would suffice.

We clarify that the benefits of learned transformations depend on two factors: the editing regime (sequential vs. non-sequential) and the subsequent update formulation. The former is examined in Sections 4.1 and 4.2 and Appendix B.4, while the latter is investigated in Section 4.3, Table 3, and Table 13, covering editing location, ridge regression, historical regularization (HAR), and gradient-induced residual construction (GIR). Among these factors, the mechanism by which the editing regime affects hypernetwork learning remains less understood. We therefore offer a hypothesis based on our existing observations, leaving a systematic investigation for future work.

In sequential editing, each edit modifies the model parameters, causing subsequent edit signals to be extracted from different model states depending on the preceding editing history. Consequently, a transformation beneficial for one model state may not provide the same benefit for another. Learning a non-trivial transformation that consistently improves editing across these varying states may therefore be challenging. Moreover, the raw edit signal is already effective, and the subsequent update formulation provides useful inductive biases for constructing the final parameter update. Thus, when the benefits of non-trivial transformations are inconsistent across model states, retaining the raw edit signal through a conservative, near-identity transformation may be a more robust solution during hypernetwork learning.

In contrast, in MEND's editing setting (Appendix B.4), edit signals are always extracted from the original model, allowing the hypernetwork to focus solely on learning the transformation itself, without considering the ever-changing model state as in sequential editing. Nevertheless, Appendix B.4 also shows that the advantage of learned transformations depends on the update formulation: their benefit is substantial with the original MEND-style raw update but becomes considerably smaller when ridge regression is adopted. Together, these observations suggest that both the editing regime and the subsequent update formulation influence the additional benefits of learned transformations. More broadly, these findings motivate us to rethink current advances in sequential model editing. Our results demonstrate that directly leveraging the raw first-step gradient signal, together with an appropriate update formulation, can achieve highly competitive overall performance against existing meta-learning and subject-based editors.

AuthorsOpenReview ↗

Author Response (2/2)

W2&Q2

Thank you for this question. We clarify the distinctions by discussing the three terms in RawEdit's editing objective (Eq. 10) in turn.

1. The first term

The first term aims to fit the current edit by finding a weight update ΔW\Delta W whose effect on the selected input kk matches the desired residual rr. This follows the same key–value remapping principle as locate-then-edit (LTE) methods such as ROME, MEMIT, and AlphaEdit. However, they differ in the positions where kk and rr are extracted and how the desired residual rr is constructed.

Regarding editing positions, LTE methods typically construct kk and rr at the last subject-token position. In contrast, meta-learning-based editors, including MALMEN [1], RLEdit, UltraEdit, StableEdit, EMSEdit [2], and RawEdit, construct them at answer-token positions. For details about answer-token selection, we refer the reviewer to our response to Reviewer 07TP under Background on Token-wise Gradient Selection.

Regarding residual construction, LTE methods typically iteratively optimize the desired residual at the last subject-token position. In contrast, RawEdit obtains the layer input ziz_i and output zo=Wziz_o=Wz_i through a single forward pass, and then computes z′o=∇zoLeditz'o=\nabla{z_o}\mathcal{L}_{\mathrm{edit}} through a single backward pass to directly construct the desired residual r=ηzo′r=\eta z'_o. This avoids iterative optimization and substantially reduces editing cost (Table 3).

Existing meta-editors instead construct the desired residual as r=η(zi⊤zi)zo′r=\eta(z_i^\top z_i)z'_o (Appendix C.1). This formulation was first introduced in MALMEN and subsequently adopted by RLEdit, UltraEdit, StableEdit, and EMSEdit. Specifically, MALMEN first assumes that the parameter update follows a gradient-form outer product (MEND-style):

ΔWgrad=ηzo′zi⊤.\Delta W_{\mathrm{grad}}=\eta z'_o z_i^\top.

Applying this intermediate update to the original input ziz_i yields the residual:

rGIR=η(zi⊤zi)zo′.r_{\mathrm{GIR}}=\eta(z_i^\top z_i)z'_o.

This residual is then used to construct the final ridge-regression-based update. Table 3 evaluates the effect of using rGIRr_{\mathrm{GIR}}. Overall, this residual does not yield consistent performance improvements across models and datasets. Ideally, we would conduct the same ablation on RLEdit, UltraEdit, and StableEdit to assess its contribution across different editors. However, this is not straightforward because these methods apply the hypernetwork transformation to rGIRr_{GIR}. Consequently, the transformed rGIRr_{GIR} is tightly coupled with the hypernetwork transformation: removing rGIRr_{GIR} would also eliminate the hypernetwork transformation itself, making it difficult to disentangle their respective effects.

MALMEN does not explicitly justify the necessity of this residual construction in its original paper, and our experiments with RawEdit reveal no consistent benefits from it. Therefore, RawEdit removes this design and directly uses the raw output gradient as the desired residual.

2. The second term

The second term, history-aware regularization (HAR), aims to preserve historically edited knowledge. It is inspired by existing state-of-the-art LTE methods for sequential editing, including AlphaEdit, SimIE [3], HSE [4], and BetaEdit. A key distinction in its construction is that these LTE methods build historical Gram matrices using representations at the last subject-token position, whereas RawEdit uses answer-token representations.

More importantly, to the best of our knowledge, RawEdit is the first to incorporate HAR into meta-learning-based editing. Table 3 validates its effectiveness in RawEdit, while Table 13 (page 25) further demonstrates its applicability to other meta-editors, including RLEdit, UltraEdit, and StableEdit.

3. The third term

The third term is the standard ridge regularization adopted by existing meta-learning-based editors, including MALMEN, RLEdit, UltraEdit, StableEdit, and EMSEdit, and remains unchanged in RawEdit.

In summary, RawEdit (1) is the first to introduce HAR into meta-editing, using answer-token representations to construct historical Gram matrices; (2) directly uses the raw output gradient as the desired residual, eliminating the rGIRr_{\mathrm{GIR}} construction; and (3), most importantly, removes the hypernetwork transformations. We hope these clarifications help readers better understand which aspects of RawEdit build upon existing methods and which constitute its methodological contributions.

[1] Massive editing for large language models via meta learning, ICLR 2024.
[2] EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing, WWW 2026.
[3] Towards lifelong model editing via simulating ideal editor, ICML 2025.
[4] Hippocampal-like sequential editing for continual knowledge updates in large language models. NeurIPS 2025.

Dear Reviewer cafY,

We sincerely appreciate your positive assessment of our work and your recognition of its contributions. In particular, we are grateful that you found our investigation into whether hypernetwork transformations benefit sequential model editing informative, and recognized the value of our component analysis and the simplicity and competitiveness of RawEdit. We also appreciate your recognition that the contrasting MEND results appropriately establish the boundary of our findings.

Thank you also for your constructive questions, which have helped us further clarify and improve the paper. As the reviewer-author discussion period is approaching its deadline, for your convenience, we briefly summarize our responses:

  1. Historical Gram matrix storage (W1&Q1): We clarified that the 14 scalars in Table 11 refer only to learnable parameters, and additionally reported the storage overhead of historical Gram matrices across all four models in Appendix D.1.

  2. Distinctions from existing methods (W2&Q2):We explicitly clarified how RawEdit's residual construction and history-aware regularization differ from MEMIT/AlphaEdit and MALMEN, and summarized its methodological contributions.

  3. Mechanistic explanation (W3&Q3): We provided a conceptual explanation of when learned transformations may offer limited benefits, considering both sequential versus non-sequential editing regimes and different update formulations. We also discussed the contrasting MEND results in Appendix B.4 to clarify the scope of our hypothesis.

Given the importance of revealing the degeneration phenomenon of hypernetwork transformations in existing sequential meta-editing and its implications for future research, we hope our findings and the proposed RawEdit can provide useful insights to the community. If our clarifications have addressed your concerns and you also believe that our work can provide useful insights for future research on sequential model editing, we would greatly appreciate it if you would consider giving our work a higher rating.

Thank you again for your time and constructive feedback. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.