- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses two challenges in memory-based model editing: synonymous reformulations that are lexically distant from the original query, and unrelated queries that are lexically similar to the original query. Building upon the memory and routing architecture of HoReN, we introduce SEAL, which employs an offline-trained four-token soft prompt to better decouple routing representations from scoring. During inference, queries are first processed via the prompted router; if a query is classified as unrelated, the model reverts to the original input—processed by the frozen base model—to ensure that the prompt does not inadvertently alter the base model's output. Our experiments span four backbone models and four benchmarks, with further scaling to 10,000 edits on the Llama architecture.
Critical strengths
- **The problem has practical significance, and the diagnosis and method are clearly connected. **The paper examines surface distance, representation distance, routing score, and final results simultaneously, rather than just reporting overall performance (Figures 3–6; Appendices E, F.2).
- **The generated branch design is targeted and verified. ** After fixing the route, removing prompt from the base path improves WikiBigEdit Locality from 0.552 to 0.852 (Table 8). For inputs that correctly return to the base path, there is also a direct structural basis for restoring the original output distribution.
Critical weaknesses
The primary issue with this paper is that its diagnosis and proposed solutions are derived from combinations of existing, well-known methods.
First, the paper’s diagnosis regarding model editing—specifically that a single edit can introduce perturbations to related knowledge—has already been addressed in a dedicated study presented at ACL 2026. That work demonstrates that such edits cause a distinct increase in uncertainty, which current benchmarks fail to capture (e.g., when tested using prompt sets generated via techniques like rephrasing).
Diagnosing Hidden Instabilities in Model Editing via Uncertainty Quantification. ACL 2026.
Second, the proposed method is essentially a modification of established model editing techniques; it incorporates an external component at the input stage, drawing inspiration from SERAC:
Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model. ICML 2022.
Consequently, I have reservations regarding the paper's innovation. Furthermore, given the inherent limitations of existing model editing benchmarks—a topic the paper itself touches upon—the following issues arise:
- The metrics used do not equate to the "complete success rate" claimed in the main text. Equation 12 in Appendix D.2 calculates the average token-wise match after truncating or padding sequences to a reference length. This approach allows for partial credit for "partially correct" answers and may overlook subsequent generated content. While the absence of teacher forcing is a positive aspect, the paper should also report the "complete answer accuracy" and the "full-output retention rate." Moreover, the loss in reliability is not always negligible; for instance, on the Mistral/CounterFact dataset, the reliability score drops from 0.997 to 0.911.
- The strongest alternative configuration is not sufficiently explored. In Table 10, the "BGE Router + prompt" configuration achieves an OP (Output Performance) of 0.962 on CounterFact—outperforming the primary SEAL configuration (0.911)—while avoiding the need to inject prompts into the LLM or recompute the base branch. This highlights the framework's versatility, yet the authors fail to provide a compelling justification for why the primary configuration is preferable. The analysis lacks cross-benchmark comparisons, evaluations on 10k-scale datasets, latency assessments, and a comparison against a simple, trainable router that utilizes the same triplet-based supervision.
- Distance-based correlation is insufficient to establish the primary cause of difficulty. In Figures 4 and 11, the data is pooled across four different datasets; the observed relationship between "distance" and routing failure may, in fact, be confounded by factors such as the specific benchmark, sequence length, prompt templates, or the presence of distractor sentences. While continuous binning and cross-model analysis are helpful, they do not replace control analysis performed within the same dataset; furthermore, ZsRE-hard involves modifications to both surface-level rewrites and distractors.
Questions for authors
See Weakness
7 comments
Part 2/5
W3. Stricter metrics and the Reliability change
The token-level score of Eq. 12 follows the standard EasyEdit protocol (Wang et al., 2024b) used by the compared methods, which keeps our results comparable with prior work. We agree, though, that it can give partial credit. To examine whether this inflates our results, we re-scored all Llama-3.1-8B evaluation outputs of Table 1 with two stricter metrics:
- Complete-answer EM (Reliability and Generality). A query counts as correct only if the generated answer exactly matches the complete target answer , i.e., the dataset label of the corresponding edit.
- Complete-output retention (Locality). An unrelated query counts as preserved only if the edited model's output string is identical to that of the unedited base model. We newly generated these reference outputs for this purpose.
Following the main evaluation protocol, the generation length is set by the reference label length.
| Benchmark | Rel. (complete-answer EM) | Gen. (complete-answer EM) | Loc. (complete-output retention) | OP (strict) | SEAL OP (token-level, Table 1) | HoReN OP (token-level, Table 1) |
|---|---|---|---|---|---|---|
| ZsRE | 0.994 | 0.969 | 0.999 | 0.987 | 0.991 | 0.979 |
| ZsRE-hard | 0.962 | 0.918 | 0.999 | 0.959 | 0.968 | 0.751 |
| WikiBigEdit | 0.949 | 0.785 | 0.850 | 0.856 | 0.869 | 0.776 |
| CounterFact | 0.968 | 0.850 | 0.919 | 0.910 | 0.911 | 0.364 |
Under the strict metrics, every score stays within about two points of its token-level counterpart, and OP changes by at most 0.013. The token-level metric therefore does not inflate SEAL's results.
A complete-answer match can never score higher than the token-level score of the same output, so HoReN's token-level scores in Table 1 are upper bounds on its strict scores. SEAL's strict Generality and Locality exceed even these upper bounds on every hard-regime benchmark. On CounterFact, for instance, SEAL reaches 0.850/0.919, against HoReN's upper bounds of 0.478/0.194. SEAL's strict OP also exceeds HoReN's token-level OP on all four benchmarks.
Regarding Reliability, the change on Mistral-7B/CounterFact (0.997 → 0.911) is the largest in our tables. In the same setting, however, Generality and Locality more than double (0.368 → 0.755 and 0.335 → 0.819), and OP rises from 0.447 to 0.823. On Llama-3.1-8B at k, SEAL's largest Reliability drop is 0.030.
Part 3/5
W4. The BGE configuration and a trainable-router baseline
We agree that the BGE configuration (BGE Router+, App. F.5.2) is attractive. Because BGE is used only for routing, the Sentinel Prompt never enters the LLM, and Decoupled Two-Stage Inference is unnecessary. We nevertheless regard the hidden-state configuration as the primary instantiation, for two reasons.
It requires no additional model. Routing reuses the LLM's own hidden states before the edit layer, as in GRACE, WISE, MELO and HoReN. The BGE configuration instead adds a separate 335M-parameter encoder (BGE-large-en-v1.5), which must run on every query and maintain its own embedding store.
SEAL is a framework, and the BGE result is evidence for it rather than a competitor to it. The same Sentinel Prompt objective improves the Hopfield router (Table 1), a GRACE-style router (Table 9) and the BGE Router (Table 10). The choice of router is orthogonal, and SEAL improves each router it is paired with. Without , even a strong contrastively trained retriever still fails on CounterFact (Gen./Loc. 0.678/0.689); with it reaches 0.949/0.941.
As requested, we now report the large-scale, efficiency and cross-benchmark analyses for the BGE configuration as well.
Large scale. For the BGE Router on CounterFact at k, we reuse the CounterFact Sentinel Prompt of Table 10 without retraining:
| Method | Rel. | Gen. | Loc. | OP |
|---|---|---|---|---|
| BGE Router | 0.991 | 0.662 | 0.419 | 0.611 |
| BGE Router+ (SEAL) | 0.989 | 0.915 | 0.787 | 0.889 |
At 10k edits, the SEAL framework still yields a consistent gain on the BGE Router.
Inference efficiency (Table 11 extended, WikiBigEdit, k; average ms per query):
| Method | Edit | Para. | Unrel. | Mean | Extra params |
|---|---|---|---|---|---|
| Base model | 74.509 | 74.542 | 68.367 | 72.473 | 0 |
| HoReN | 81.915 | 81.766 | 75.135 | 79.606 | 18.43M |
| SEAL | 82.315 | 85.241 | 92.684 | 86.747 | 18.45M |
| BGE Router | 89.088 | 88.465 | 83.422 | 86.992 | ≈353M (incl. 335M frozen BGE) |
| BGE Router+ | 89.188 | 88.545 | 83.473 | 87.069 | ≈353M (incl. 335M frozen BGE) |
Relative to HoReN, SEAL adds 9.0% mean latency and only 16,384 parameters (0.089%); its extra cost falls mainly on queries routed to the base-model path, which are recomputed without . The BGE configuration avoids this recomputation, and adds only 0.08 ms to the BGE Router. Its mean latency is nevertheless slightly higher than both HoReN's (+9.4%) and SEAL's (+0.4%), because the BGE encoder runs on every query, and it adds a 335M-parameter frozen encoder, over 18× SEAL's extra parameters.
Cross-benchmark transfer. In the primary configuration, a Sentinel Prompt trained on ZsRE-hard and applied without retraining improves OP over HoReN on CounterFact (0.364 → 0.581) and on WikiBigEdit (0.776 → 0.814), as reported in our response to Reviewer S8xy (W4 / Q3). For the BGE Router, we evaluate how a Sentinel Prompt trained on WikiBigEdit (k) transfers to CounterFact and ZsRE-hard without retraining:
| Target | BGE Router (OP) | BGE Router + WikiBigEdit (OP) |
|---|---|---|
| CounterFact | 0.763 | 0.749 |
| ZsRE-hard | 0.938 | 0.969 |
Transfer is not consistent for the BGE Router: the transferred improves ZsRE-hard by 0.031 but lowers CounterFact by 0.014.
Part 1/5
We thank Reviewer QeDC for the careful review. We address each concern below. Unless noted otherwise, results use Llama-3.1-8B with T=1k.
W1. Relation to Gu et al. (ACL 2026)
We thank the reviewer for the pointer. We respectfully note that Gu et al. (2026) address a different problem in a different class of editors, and their findings do not overlap with ours.
Gu et al. study parameter-update editors, such as least-squares methods like ROME and MEMIT, which write each edit directly into the shared model weights. They show that such updates perturb pretrained knowledge, which surfaces as increased post-edit uncertainty under semantic perturbations.
Memory-based editors, the class we study, work differently (Sec. 2, App. A.1). Each edit is stored in an external memory, or a lightweight edit-specific module, attached to the frozen backbone. A router then sends each incoming query along one of two paths:
Because the backbone weights are never overwritten, the weight-level interference measured by Gu et al. does not arise. Whether pretrained knowledge is preserved depends entirely on whether the router makes the correct decision.
This is where our diagnosis applies. We identify routing failures driven by the surface distance between the edit query and its paraphrase and unrelated queries. When the paraphrase query is lexically distant from the edit query (larger ds(xi,x~i)), or the unrelated query is lexically near it (smaller ds(xi,x^i)), the hidden-state representation distance dr used by the router follows the same trend. Paraphrase queries therefore receive lower routing scores and unrelated queries receive higher ones, so the two routing-score distributions overlap. This makes it difficult for the router to send paraphrase queries to the edit-specific path while rejecting unrelated queries to the base-model path, and hence to achieve Generality and Locality simultaneously (Sec. 2.2, Sec. 4.3, App. E).
This chain explains why WISE, GRACE, MELO and HoReN all reach their best OP on ZsRE and degrade on the three hard-regime benchmarks (Table 1). It holds across 4 backbones, 3 layers and 4 editors, and within each individual benchmark (W5).
The two works also differ in what they deliver. Gu et al. propose an evaluation protocol. Our diagnosis directly yields a remedy, which lifts CounterFact OP from 0.364 to 0.911 and ZsRE-hard OP from 0.751 to 0.968, with gains that persist at 10k edits (Table 2).
W2. Relation to SERAC
SERAC (Mitchell et al., 2022b) is a well-known early memory-based editor. It is already included in all our experiments through its SERAC-style DEFER instantiation (Sec. 4.1, App. A.1). We respectfully disagree, however, that SEAL is a modification of SERAC. The two methods differ in their components, in what the added component does, and in how it is trained.
Components. SERAC attaches two separately trained models: a scope classifier that decides whether a query is in scope, and a counterfactual model that generates answers for in-scope queries. SEAL uses neither. Beyond the edit memory and router common to memory-based editors, it adds only a 4-token Sentinel Prompt to the single backbone model. With 16.4K parameters, the Sentinel Prompt is several orders of magnitude smaller than a dedicated scope-classifier network.
Role of the added component. The Sentinel Prompt makes no routing decision and generates nothing. It only reshapes the hidden-state representations used by the router, so that paraphrase and unrelated queries become easier for the router to distinguish. Edited answers still come from the edit values, and all other answers come from the frozen backbone. Using prompt tuning to reshape routing representations has no counterpart in SERAC.
Training. Both methods use hard negatives, but they obtain them differently. SERAC takes them passively, as out-of-scope examples that are semantically similar to the edit. SEALSYN uses only edit-side data, which matches the deployment setting, and constructs its hard negatives actively to target the failure mode we identify. Its unrelated queries keep the edit query's template and replace only the subject with a confusable entity, i.e., they have small ds(xi,x^i). SEALSYN additionally constructs hard positives, paraphrase queries with large ds(xi,x~i), which SERAC does not use (App. H).
Results. Empirically, SERAC-style DEFER performs poorly on all four benchmarks:
SEAL's OP is higher by 0.69–0.94 on every benchmark (Table 1).