acceptodds
Under review as a conference paper at ICLR 2027

Factual Access States in Diffusion Language Models: From Mechanism to Knowledge Editing and Retrieval

Abstract

Factual recall is a lookup: some representation inside the network serves as the key, and what that key is made of decides which queries can retrieve the fact, what an edit must cover, and what a retriever can read. We call it the factual access state. In autoregressive (AR) models the subject position enriches attributes largely independently of the relation, and relation-specific selection happens later, at the readout position. In masked diffusion language models (DLMs) we find the state is assembled differently, and much earlier: in the first attention layer the wording of the relation is routed into the position of whichever entity is visible and the early feed-forward layers look the fact up on this fused representation. The state therefore varies with the relation's surface form and with the query direction, so the queries about one fact occupy a region of state space rather than a point, and an operation on the fact succeeds only for queries that enter it. We treat the state as an interface to a DLM's factual knowledge, and put it to two uses. Writing to the state gives a knowledge editor: from one declarative statement, Multi-view Edit fits one shared value per direction across that region and installs it in the early-MLP block, preserving source-prompt reliability while substantially improving paraphrase and reverse question answering, where AR editors gain near zero. Reading it suggests a complementary use: the state at each entity position can serve as a relation-specific retrieval vector with no training, returning both pieces of evidence for two-entity queries on which pooled, prompted and lexical retrievers almost never return both.

Primary area:
foundation or frontier models, including LLMs
Keywords:
masked diffusion language models, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cHKuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper argues that masked diffusion language models form an early, relation-dependent factual access state at visible entity positions. It supports this account with causal interventions in two models, then uses the proposed state for multi-view knowledge editing and per-entity evidence retrieval.

Critical strengths

The causal interventions give the proposed mechanism more support than attention visualizations alone. The editing and retrieval experiments also test whether the mechanism leads to useful applications; the reported improvements over the included baselines are substantial.

Critical weaknesses

The editing result remains limited on the natural question-transfer setting. On QAEdit, only 17.6% of LLaDA and 23.8% of Dream source-prompt answers are fully correct after Multi-View editing; performance on paraphrased questions is lower. This weakens the practical claim that a declarative edit can reliably be accessed through later questions.

The retrieval comparison does not fully isolate the benefit of the proposed representation. The proposed method uses known entity spans, makes one ranking per entity, and alternates results within a two-item budget. The principal baselines score the whole query. The entity-conditioned control is useful, but it still does not compare against a baseline given the same per-entity ranking procedure. In the natural evaluation, retrieval is also limited to each question’s supplied distractor context.

Questions for authors

How does the method compare with BM25 and dense retrievers when each is given the same entity spans, per-entity rankings, and alternating two-result budget? Would the retrieval gain persist with automatically detected spans and an open corpus? This comparison would materially affect my assessment of the retrieval contribution.

5 comments

Response to Reviewer cHKu (Part 1/2)

We sincerely thank the reviewer for the careful and fair assessment. We ran all the retrieval comparisons you asked for and report them below.

W1. Editing remains limited on QAEdit

We agree, and the paper states it as a limitation (Conclusion, App. G): free-generation transfer is improved, not solved. Two points of context:

  • Scope of the claim. All QAEdit numbers are on the 2,272 examples that the unedited model answers fully wrongly, with one declarative statement as the only edit input and free decoding at test time. The claim is relative: under identical conditions, Multi-View raises the share of at-least-partially-correct answers over the matched Single-View editor from 26.2% to 32.5% on LLaDA-8B and from 38.1% to 44.6% on Dream-7B.
  • Part of the remaining gap comes from decoding, and it can be reduced by combining the correction of Park & Jo (2026) with our method. Adding the partial-reveal views of Park & Jo (2026) to our view family, with everything else fixed, doubles exact match on LLaDA-8B: 17.6% → 35.0% on source prompts and 12.6% → 26.7% on paraphrased questions (weighted score 0.250 → 0.430 and 0.187 → 0.352; App. D.7). The Introduction (§1) now gives the exact-match figures alongside the partial-credit ones, and the composed result is reported in App. D.7 (Table 17).

Response to Reviewer cHKu (Part 2/2)

W2 + Q1. Does the retrieval comparison isolate the representation?

Thank you for this important point. We ran the three comparisons you asked for.

(1) Same entity spans, per-entity rankings, alternating budget of two. Each baseline is told where the two entities are, but not which relation belongs to which. For each entity, we form a sub-query: the original query with the other entity's name replaced by “the other”. Each baseline encodes it in its usual way and ranks all 5,528 cards once per entity; the two rankings then fill the budget of two in turn, exactly as for our method. Results are All@2 (%) on the controlled benchmark, with global search.

MethodQuery unitAdjacentCrossed
LLaDA access state, L31 (ours)whole query, one encoding41.9438.08
Dream access state, L27 (ours)whole query, one encoding38.2821.78
BM25whole query (Table 5)1.381.01
BM25per-entity sub-query, gold spans11.8310.75
All baselines, rangewhole query (Table 5)0.61–3.060.00–2.50
All baselines, rangeper-entity sub-query, gold spans4.0–14.21.9–13.5

Under this protocol, BM25 rises from 1.38 / 1.01 to 11.83 / 10.75. The per-entity procedure lifts the baselines by about ten points, but they stay below the access state. The remaining gap comes from the vector that is read at the entity span.

(2) Open corpus. In the natural setting, we now rank over all 106,329 sentences instead of each question's distractor context. Results are Both@2 (%): distractor context (Table 20) → full corpus.

Method2WikiMultiHopQAHotpotQA
Dream access state, L0 (ours)37.37 → 33.341.82 → 39.7
LLaDA access state, L0 (ours)33.17 → 27.540.82 → 35.3
BM2526.93 → 25.725.15 → 22.8
PromptReps, hybrid13.16 → 9.611.34 → 6.9
DiffRetriever, hybrid (LLaDA)13.60 → 4.319.21 → 13.7
Whole-query pooling (LLaDA)10.13 → 4.716.44 → 8.8

The dense baselines fall to 4.3–9.6 and 6.9–13.7. The access state remains the best method on both datasets.

(3) Automatically detected spans. We replaced the gold spans with spans from an off-the-shelf spaCy NER model, without tuning.

SettingGold spansspaCy spans
Controlled, LLaDA, adjacent / crossed (All@2)41.94 / 38.0833.1 / 32.7
Controlled, Dream, adjacent / crossed (All@2)38.28 / 21.7831.1 / 18.8
Natural, Dream L0, 2Wiki / HotpotQA (Both@2)37.37 / 41.8231.6 / 31.9
Natural, LLaDA L0, 2Wiki / HotpotQA (Both@2)33.17 / 40.8228.1 / 30.7

The access state keeps 75–86% of its gold-span performance. On the controlled benchmark, it stays above the baselines even when those are given gold spans (at most 14.2 / 13.5), and in the natural setting it stays above BM25 (26.93 / 25.15).

In the revision: (i) Table 5 gains the per-entity baseline rows. (ii) In §4.2, the sentence “…which is what the state binds” is replaced by a more precise statement: the advantage is binding each entity to its relation in one encoding of the undecomposed query. (iii) The open-corpus and detected-span results are added to App. F.2 (Table R7) and App. F.1 (Table R6), with a pointer from §4.2. (iv) App. G no longer lists automatically detected spans and open-corpus retrieval as untested.

Thank you again for your time and for the valuable suggestions, which clearly improved the paper. We would be glad to clarify anything further.

Thank you for the authors' detailed responses, and I will keep my score.

Reviewer_FiJNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how masked diffusion language models (DLMs) look up facts and contrasts them with autoregressive (AR) models. On LLaDA-8B, mask-based causal tracing locates the lookup in layer-0 attention and MLPs 1-8, and restricting that attention to subject-to-relation edges recovers most of its effect. The paper calls the resulting relation-fused representation the "factual access state." MULTI-VIEW EDIT fits one shared value per direction over masked views of a statement and, on LLaDA-8B and Dream-7B, improves paraphrase (CounterFact) and reverse-question (BAKE) generalization over matched DLM variants. The same per-entity states serve as training-free retrieval vectors and retrieve both pieces of evidence for two-entity queries more often than whole-query baselines.

Critical strengths
  1. The editing method is well controlled. Multi-View improves paraphrase generalization on both DLMs (Table 2), and a budget-matched variant shows that the choice of views, not their number, drives the gain. This is the main basis for my positive rating, and it does not depend on the mechanistic framing.

  2. The tracing is adapted to DLMs. Corruption uses the model's own [MASK] operator, and layer and block restorations separate the early lookup from a late "conduit" stage, so the localization does not inherit assumptions from AR tracing. A separate retrieval check on Dream-7B-Instruct is consistent with it: removing layer-0 relation messages, but not matched random ones, drops within-entity relation accuracy to chance.

Critical weaknesses
  1. The DLM-versus-AR contrast that frames the paper is never tested in a controlled way. The paper neither traces nor edits an AR model, so its claim that the DLM state "is assembled differently, and much earlier" rests on prior literature. The abstract and Fig. 1 ("key = entity alone") also drop the introduction's condition that the relation follows the subject. Fig. 1's own example, "The capital of France is [MASK]," puts the relation first, so even a causal model's subject position sees it. On the editing side, Table 9 copies AR results from other models and edit pools ("scale references rather than controlled comparisons," App. A.4). These numbers put ROME near ceiling on CounterFact paraphrase, which sits uneasily with the introduction's claim that AR edits do not reliably reach paraphrases. BIRD gains little on reverse QA despite its reverse objective, which partly supports the contrast. However, the DLM reverse gain comes mainly from an object-keyed edit (Table 3) that the paper never tries in an AR model. Without a controlled AR baseline, I would ask the authors to present the DLM-specific lookup as a hypothesis and to limit the abstract's "AR editors gain near zero" to reverse QA. Qwen2.5-7B, Dream-7B's AR base model (Ye et al., 2025), is a natural control. I suggest tracing it with an AR-suited corruption such as subject-embedding noise, reporting how its subject-position states depend on relation wording, and running ROME on it with the same edit pools, with and without an object-keyed reverse edit.

  2. Two missing controls leave the mechanism and retrieval claims open to simpler explanations. In the attention surgery, relation tokens stay unmasked in the corrupted run (Sec. 2.1), so their layer-0 values are identical across runs, and the restored subject-to-relation message adds subject information only through the clean-run attention weights. The rise in the object's probability could therefore reflect a type-level prior rather than a subject-specific lookup. In retrieval, the method ranks once per gold entity span, while the baselines rank once per query. The entity-conditioned control narrows the gap but still scores baselines with one whole-query vector, so the remaining advantage may reflect "what the state binds" (p.9 l.434-435) or simply per-entity querying. Neither point touches the editing results. To separate these readings, I would like to see the restored effect on type-matched distractor objects, and per-entity baselines on the same spans, such as BM25 over each entity's clause and, for crossed queries, the baseline encoders' entity-span states.

Questions for authors
  1. Remark 1 predicts that family-averaged keys remove the single-view paraphrase attenuation, yet in Fig. 4, paraphrase trigger strength barely changes between single-view and family keys while the editing effect more than doubles. App. E.1 assigns this change to the value, but Sec. 3.1.1 and E.4 still credit key coverage. Could the authors reconcile these passages with Fig. 4, or test the split directly: how much of the paraphrase gain remains when the full-family value is written along the single-view key, and vice versa?

  2. How many facts does Table 1 trace, and is attention renormalized after surgery?

  3. Fig. 2's full-block effect peaks far below the indirect effect the text reports for layers 0-8. Does Fig. 2 use Eq. 1's normalization?

  4. The introduction's 63.2% LLaDA reverse-QA result comes from the development pool (Table 10). Please report held-out results for all methods, not only Multi-Binary.

4 comments

Response to Reviewer FiJN (Part 1/3)

We sincerely thank the reviewer for the thorough and insightful review. We ran every control you suggested, and they made the paper clearly stronger.

W1. The DLM-vs-AR contrast was not tested in a controlled way

Thank you for this valuable suggestion. In the submission, the AR side of the contrast rested on published results (GPT-J and LLaMA-2, Table 9) and on prior analyses of AR recall (Meng et al., 2022; Geva et al., 2023); the focus of the paper is the analysis of the DLM mechanism and its comparison with the AR one. Following your suggestion, the revision now adds experiments on Qwen2.5-7B (App. B.7, Tables R2–R4) as direct evidence for this claim.

(a) AR control.

We ran all three parts on Qwen2.5-7B, the AR base model of Dream-7B.

1. Does the subject state depend on the relation? For subject-first prompts, we replace the states at the subject positions (layers 0–8) with the states of the same subject taken from a prompt with a different relation, and measure how much of the recall effect survives.

Qwen2.5-7B (AR)Dream-7BLLaDA-8B
Cosine to the original state (layer 8)1.0000.8740.950
Recall effect retained99.9%85.9%85.0%
Top-1 answer unchanged99%81%77%
Recall effect retained, swap up to layer 2499.9%74.7%69.5%

In Qwen, the swap changes nothing; in both DLMs the state differs, and part of the effect is lost. The AR subject state does not depend on the relation; the DLM state does.

2. Tracing with subject-embedding noise. We follow the original ROME protocol (3σ noise on the subject embeddings, 10 samples, ±5-layer windows).

Qwen (n = 234)Dream (n = 224)LLaDA (n = 197)
Peak single-token restoration: first / middle / last subject token0.14 / 0.35 / 0.700.21 / 0.34 / 0.380.20 / 0.22 / 0.31
MLP only, last subject token, window centred at L4 / L80.58 / 0.530.27 / 0.140.24 / 0.23

Qwen reproduces the known AR picture. In Qwen, the effect concentrates at the last subject token; in the DLMs it is spread over the subject tokens. This agrees with App. D.3, where every subject position carries a usable key.

3. ROME on Qwen2.5-7B (layer 5), compared with our editor on the same BAKE pool. The LLaDA-8B and Dream-7B rows are from Table 3.

ModelEditEdit inputRel.Para.R-QAR-Judge
Qwen2.5-7B (AR)No edit–18.718.613.33.5
Qwen2.5-7B (AR)ROME, forward editone statement97.492.515.312.9
Qwen2.5-7B (AR)ROME, forward + object-keyed reverse editstatement + a separate reverse fact96.892.091.5*17.2
LLaDA-8BNo edit–––22.3530.59
LLaDA-8BSingle-Forwardone statement95.5373.0322.5238.88
LLaDA-8BMulti-Binary (ours)one statement98.4984.5663.2455.60
Dream-7BNo edit–––20.2618.15
Dream-7BSingle-Forwardone statement94.9388.4320.1332.13
Dream-7BMulti-Binary (ours)one statement97.6596.2583.5755.75

* In-sample: we used the R-QA prompt itself as the reverse fact, so the reverse edit is fitted on the prompt it is tested on.

  • The forward edit is near ceiling on paraphrase, in line with Table 9, and moves reverse QA by only 2.0 points, as Single-Forward does on the DLMs (+0.2 / −0.1, Table 3).
  • An object-keyed edit does install the reverse direction in the AR model, but not from the statement: it needs a second, separately constructed reverse fact and a second edit (two facts, two edits). Under a causal mask, the statement offers no position where the object is visible and the subject is predicted.
  • Multi-Binary receives only one declarative statement. It derives the reverse view from it with the model's own masking operator and installs both directions from that single statement. On the same edit pool, it gains +40.9 points on reverse QA and +25.0 on reverse judgment over no edit on LLaDA-8B, and +63.3 and +37.6 on Dream-7B. On reverse judgment, the probe seen at edit time by neither model, Qwen gains +13.7 (3.5 → 17.2).

In the revision, the three results are added as App. B.7 (Tables R2–R4) and referred to from §1, §2.3 and §3.2.2, so the AR side of the contrast now rests on a controlled comparison. In §2.3 and App. B.5, we no longer say that an AR model “cannot express a reverse association natively”. We say instead what the control shows: an AR editor can install the reverse direction, but only from a separately constructed reverse fact and a second edit, whereas the DLM editor obtains both directions from one statement.

Response to Reviewer FiJN (Part 2/3)

(b) The “relation follows the subject” condition was dropped.

You are right. The AR statement holds for subject-first prompts, where the causal mask hides the relation from the subject position. We state this condition in the Introduction but have dropped it in the abstract and in Fig. 1. The abstract now reads “In autoregressive (AR) models, when the relation follows the subject, the subject position enriches attributes largely independently of the relation…”. In Fig. 1, the AR panel is relabelled “key = entity and its prefix” and uses a subject-first example (“The Eiffel Tower is located in [MASK]”).

(c) “AR editors gain near zero” is too broad.

We agree and have narrowed both claims. The abstract now says “…on reverse question answering, published AR editors gain under three points”. In the Introduction (§1), the statement that AR edits are not reliably reachable is restricted to reverse questions, and we state explicitly that AR ROME/MEMIT are near the ceiling on CounterFact paraphrase prompts (Table 9).

(d) Present the contrast as a hypothesis.

With the control in (a), the AR side is now measured on Qwen2.5-7B rather than taken from prior literature. In §2.3 and App. B.7, we state the contrast in the scope the control covers (subject-first prompts; one AR model, the base model of Dream-7B) and cite Meng et al. (2022) and Geva et al. (2023) as consistent prior evidence.

W2. Two missing controls

(a) Could the layer-0 effect be only a type-level prior?

We ran the control you requested on both models (LLaDA-8B: 734 facts; Dream-7B: 858 facts). We restore only the layer-0 subject→relation edges at the subject positions and compare the correct object with a type-matched distractor, i.e., another object of the same relation.

LLaDA-8BDream-7B
IE of the correct object+0.215 ± 0.026+0.959 ± 0.014
IE of the type-matched distractor−0.026 ± 0.005−0.028 ± 0.008
Gain in log p(correct) − log p(distractor) [95% CI]+2.31 [2.13, 2.50]+6.27 [6.08, 6.46]

The correct object rises; the distractor does not, it falls slightly. A type-level prior would raise all objects of the relation together. Replacing the single distractor with all objects of the relation gives the same result (+2.32 / +6.29). The signal restored at layer 0 is therefore specific to the fact. The control is added as App. B.6 (Table R1) and referred to from §2.3.

(b) Per-entity baselines for retrieval.

We ran both requested controls on the controlled benchmark (global search over all 5,528 cards, same budget of two). The full table is in our response to Reviewer cHKu.

  • Crossed queries. Both entities are named first, and the relations are assigned by reference, so there is no per-entity clause. We therefore gave every baseline a per-entity sub-query: the original query with the other entity's name replaced by “the other”, encoded in the baseline's usual way, ranked once per entity and merged round-robin. BM25 then reaches 10.75 and the baselines as a group 1.9–13.5, against 38.08 (LLaDA) and 21.78 (Dream). Per-entity querying alone does not explain the gap.
  • Adjacent queries, BM25 on each entity's clause. BM25 on the clause reaches 91.88 (baselines as a group 57.5–94.8; access state 41.94 / 38.28). Here each clause is already a complete entity–relation sub-query, so the query is handed over decomposed. With the sub-queries above, which do not decompose the query, BM25 reaches 11.83.

The advantage of the access state is thus the binding of each entity to its relation inside one encoding of the undecomposed query. We do not claim it over a retriever that receives a decomposed query. In the revision, §4.2 states the advantage in this form (replacing “which is what the state binds”), the per-entity baselines are added to Table 5, and the clause control is reported in App. F.1.

Response to Reviewer FiJN (Part 3/3)

Q1. Is the paraphrase gain carried by the key or by the value?

We ran the swap test you proposed (LLaDA-8B, 1,000 facts). S denotes the single-view construction and F the full-view family; key and value are swapped independently.

Success (%)S key / S valueS key / F valueF key / S valueF key / F valueShare of the gain from the value (log-odds) [95% CI]
Paraphrase53.169.854.072.591% [89, 93]
Reverse QA34.038.536.041.672% [66, 77]
Reverse judgment39.443.340.343.586% [79, 92]

Your reading of Fig. 4 is correct: the view family works mainly through the value. Writing the full-family value along the single-view key recovers most of the paraphrase gain (53.1 → 69.8, out of 72.5), whereas the family key with the single-view value recovers almost none (54.0). Supervising one shared value on all views makes it correct when it is read from many keys, which is the mechanism of §3.1.3; averaging the key adds a smaller, consistent part. This agrees with Fig. 4, where the paraphrase trigger is unchanged (0.609 → 0.624) while Δlogit more than doubles.

The key matters on the other axis, direction: a forward-only edit triggers at 0.08 on reverse queries, and installing a reverse key raises this to 0.56 (Fig. 4).

In the revision: §3.1.1 and App. E.4 are reworded to agree with App. E.1 and Fig. 4, Remark 1 is qualified accordingly (Prop. 1 describes the key-side component), and the table is added to App. D.4 (Table R5), with a pointer from §3.2.4.

Q2. Number of facts in Table 1; renormalisation

Table 1 uses n = 80 facts: the first 80 CounterFact facts that pass the filters of the tracing protocol. Yes, attention is renormalised: once the other edges are zeroed, the remaining attention weights of each subject token are renormalised to sum to one. Both are now in the caption of Table 1.

Q3. Does Fig. 2 use the normalisation of Eq. 1?

Yes. The gap comes entirely from the unit of restoration. Fig. 2 plots IE(ℓ, i) as in Eq. 1: the state at one token position is restored, and each curve averages over all tokens of that role and over all facts. The figures quoted in the text (e.g., 0.936 for layers 0–8) restore all subject tokens jointly. Subjects average 4.3 tokens, so restoring one of them while the others are still [MASK] recovers much less. Fig. 2 uses n = 194 facts under the same protocol. We now state this in the caption of Fig. 2 and in §2.2.

Q4. The 63.2% in the Introduction comes from the development pool

Agreed, and thank you for catching this. The Introduction quoted the development-pool figure. In the revision, §1 reports the held-out one: on LLaDA-8B, reverse question answering reaches 61.0% on the 1,000 held-out edits of Table 10, instead of 63.2%. The development pool was used to select one setting only, the write position (App. D.3), and the held-out split reproduces the ranking of that sweep.

Thank you again for your time and for the valuable suggestions, which clearly improved the paper. We would be glad to clarify anything further.

Thank you for the additional controls. The type-matched test and key/value swap address important concerns, and I maintain my weak-accept rating.

Reviewer_4LukOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the knowledge editing of diffusion language models (DLMs). The idea is mostly inspired by the ROME paper: first define the indirect effect (IE) for DLM by using mask as corruptions. The authors use IE to identify the key for factual recall in DLM is the factual access state, which is the residual stream in the entity position and contains the early representation of relations via first-layer attention. Comparing with the key for an autoregressive (AR) model, this factual access state is more sensitive to the hidden states of relation tokens, thus could be unrobust to relation paraphrasing. To resolve this issue, the authors modified ROME by partly masking relation tokens when computing the target value vector of the factual recall. They also augment the promp sets by adding reverse-order paraphrasing. They update the down-projection in specific MLP layer using the computed value vector under the rule of linear key-value associative memory with L2L_2 loss. As a byproduct, the authors also propose a retrievel method for DLM based on the cosine similarity between the representations of the entity in the query and the evidence cards.

Critical strengths

The experments are quite comprehensive -- not only the experiments for method evaluations, but also the ones that validate the concept of factual access state. I think this paper is a natural adaptation of ROME to DLMs.

Critical weaknesses

The paper is extremly hard to read -- some wordings are very confusing. For example, it takes me quite a while to understand the line 132 "What survives is an early block that is jointly, but only jointly, sufficient." It's simply saying restoring first-layer attention and early-layer MLP reach high IE and it could not be achieved by restoring any single of the these two. Another example is at line 225, the design principle of the knowledge editing method. Actually, I think this principle feels quite specialized for ROME-like methods but there's no surronding context on ROME. If authors want readers to better understand this principle, I would suggest adding some statement like the proposed method is more like a data-augmented version of ROME.

Questions for authors
  1. Is there a reason for not paraphrasing relation and adding them to the views? I feel this might also work as the current relation view augmentation.

  2. Could you explain why the second peak in Figure 2 is an artifact of the sliding window? Why would it disappear when decreasing w=1?w=1?

  3. In the definition of factual access state, line 184, why would the writing and reading operations upon it being stated in its definition?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Luk

We sincerely thank the reviewer for the careful reading and the very helpful suggestions. Each point is answered below.

W1. Readability (l.132 and l.225)

Thank you for pointing to the exact sentences. You are right, and we have rewritten them.

  • l.132 (§2.2) now reads: “Restoring layer-0 attention together with MLPs 1–8 recovers almost the full effect (IE = 0.943), whereas neither does so alone (0.238 and 0.781).”
  • l.225: §3.1 now opens with a short primer on ROME/MEMIT (the key–value view of an MLP layer and the rank-one update), and the design principle is stated in those terms. We gratefully adopt your framing as the intuition: Multi-View Edit can be read as ROME whose key and value are fitted over an augmented set of views of the edit statement. The primer also states two points where it goes beyond ordinary data augmentation (§3.1, App. A.2): (i) the views need no external data: they are produced from the one statement by the model's own masking operator (relation occlusion, prefix/suffix context); (ii) the reverse view is not one more sample of the same query: it installs a second, object-side association, which is what makes reverse questions answerable.

Beyond these two places, we will revise the whole paper to make it easier to follow. Concretely, we will (i) split and shorten the dense sentences throughout, (ii) define each term before it is used and give a one-line intuition before each formal statement, and (iii) add the background on ROME-style editing wherever the method builds on it. We take this concern seriously and thank you for raising it so concretely.

Q1. Why not add relation paraphrases as views?

Thank you for the suggestion. Paraphrase views are fully compatible with our method: they would simply be one more view type supervised on the same shared value, exactly like the partial-reveal views in App. D.7. The two are orthogonal, so adding paraphrases on top of our views should further improve transfer to unseen rewordings. We left them out of the submission for two reasons:

  • No extra resources. The edit is given as one declarative statement, and we wanted the editor to need no paraphrase resource at edit time. Masked views are produced by the model's own corruption operator, so they are free and come from the same distribution the model is queried under (App. E.4).
  • Paraphrases are what we test on. Our goal is generalisation from one edit: the model is edited with one statement and then tested on rewordings it has not seen. If paraphrases were also given at edit time, the paraphrase score would no longer measure generalisation. This follows the standard protocol: ROME, too, edits with a single prompt and is then evaluated on paraphrases.

We now state this in §3.1.2.

Q2. Why is the second peak in Fig. 2 a window artifact?

With w = 10, each point in Fig. 2 restores ten consecutive layers at the subject positions. By layers 18–25, the fact has already been retrieved and is only being carried towards the [MASK] position, so restoring ten such layers at once adds up many small, redundant contributions. With w = 1 (Fig. 3), no late layer exceeds ≈ 0.03; only layer-0 attention stands out (≈ 0.23).

Cross-peak patching confirms this (App. B.2): restoring the early block while blocking the late one destroys the effect (0.946 → 0.019), whereas restoring the late block with the early one blocked leaves it intact (1.012). The late layers carry the fact; they do not retrieve it. We agree that “artifact” was too terse and have reworded the sentence accordingly (§2.2).

Q3. Why does Definition 1 state the write and read operations?

We agree: the definition should only say what the state is. We improved the readability of this definition in the revision (§2.3): Definition 1 now ends after “d ∈ {fwd, rev} records whether e is the subject or the object of the fact”. The sentences on where §3 writes and where §4 reads are moved to a remark after the definition.

Thank you again for your time and for the valuable suggestions. We would be glad to clarify anything further.

Thanks for the response and the revision. I just read the revised paper and it is easier to read now. I'd like to keep my score of 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We sincerely thank all reviewers and the AC for their time and constructive feedback. We have uploaded a revised version:

  • All changes are shown in blue.
  • New tables are numbered R1–R7. All original tables, figures and appendix sections keep their numbers, so the references in our responses still apply.
  • The main text stays within 10 pages.

The main additions are a controlled AR comparison on Qwen2.5-7B (App. B.7), a type-matched control and a key/value swap test (App. B.6, D.4), and per-entity, open-corpus and detected-span retrieval controls (Table 5, App. F). We also improved readability and made several claims more precise. Details are in our individual responses, and we are happy to clarify anything further.

AuthorsOpenReview ↗

Summary of the discussion

Summary of the discussion

We thank the AC and all reviewers for their time and the constructive discussion. For convenience, we briefly summarise where it stands.

  • Reviewer 4Luk (3) found the paper hard to read. We rewrote the passages pointed out and added a ROME/MEMIT primer (§2.2, §2.3, §3.1). The reviewer confirmed that the revised paper "is easier to read now".
  • Reviewer FiJN (3) asked for a controlled AR comparison and two controls. We added all of them (App. B.6, B.7, D.4; Tables R1–R5) and narrowed the related claims. The reviewer stated that the type-matched test and the key/value swap "address important concerns".
  • Reviewer cHKu (2 → 3) asked for per-entity baselines, open-corpus retrieval and automatically detected spans. We added all three (Table 5, App. F; Tables R6–R7), and the reviewer raised the rating to weak acceptance.

All changes are in blue in the revised PDF. We will carry out the remaining readability improvements in the final version. Thank you again for handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.