acceptodds
Under review as a conference paper at ICLR 2027

Make: Distributionally Robust Knowledge Editing under Multi-Agent Context Shift

Abstract

Knowledge editing for large language models is usually developed and evaluated in isolated settings. In multi-agent systems, role instructions and peer messages change the retrieval context, and thus edits that succeed on clean prompts can fail during interaction. Editing every agent is one fix, but its cost grows with the number of roles. We instead study a setting in which only one designated agent is edited, and ask whether this single intervention can still improve system-level behavior. We identify the main failure mode as interaction-induced representation shift: multi-agent context changes the internal representation used to retrieve the edited fact and can weaken a standard edit. We propose MAKE (Multi-Agent Knowledge Editing), a two-stage framework for multi-agent knowledge editing. Adversarial Context Mining collects deployment-relevant contexts, and Wasserstein-robust multi-key editing enforces the target association on clean and contextual keys while minimizing a tractable upper bound on worst-case off-support activation. We also derive an off-support sensitivity bound for unseen contextual deviations. Experiments on two LLM families and three representative multi-agent architectures show large drops from isolated edit success to final system success for standard editors. MAKE improves multi-agent edit success and conflict robustness while maintaining competitive generalization and preservation under comparable computational cost. It also remains stable under sequential edits, achieving limited degradation of general-purpose capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge Editing, LLMs-based Multi-agent

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_UymEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge editing in multi-agent systems (MAS) and proposes MAKE. MAKE edits one designated agent in two stages. It first mines adversarial contexts, then applies Wasserstein distributionally robust optimization (DRO) to limit the representation shifts that agent interaction causes.

Critical strengths
  1. The problem is realistic and under-explored. Uncoordinated edits to individual agents, or representation shifts from interaction, can make the whole system decide wrongly.
  2. Experiments cover several interaction topologies, including Debate, Hierarchical, and Pipeline.
  3. The ablations measure how upstream prompts, round dynamics, and peer pressure in high-conflict settings lower system-level success downstream.
Critical weaknesses
  1. The baseline comparison is unfair. MEMIT, AlphaEdit, and CAKE receive only a single greedy context and no clean query keys. Their scores collapse as a result (e.g., SAES below 3%), which inflates MAKE’s advantage.
  2. MAKE evaluates deterministic greedy trajectories under the test setting in advance and writes them into the parameters as anchors. Since the anchors match the evaluation setup, it is unclear how the method handles open-ended queries or interaction paths it has not seen.
  3. Most metrics rely on string matching. A semantic judgment (e.g., LLM-as-a-Judge) and a fluency or perplexity measure would make the results more convincing. The overall average also combines local node metrics with the much lower system-level metrics (MAES, MAG), which may hide weak system-level performance. Reporting the two groups separately would be clearer.
Questions for authors
  1. How can a single-node edit reach high MAES in multi-round or high-conflict settings? There, unedited downstream or peer agents may reject or override the new fact based on their own parameters.
  2. If MEMIT and AlphaEdit get the same multi-context support sets and clean query keys as MAKE, does the gap remain? Or do the gains come mainly from multi-context data augmentation rather than from DRO?
  3. MAKE relies on pre-mined adversarial contexts and greedy anchors (cgc_g) that match the evaluation setup. How does it perform on out-of-distribution queries, under stochastic decoding, or on dynamic communication topologies where no precomputed trajectories exist?
7 comments
AuthorsOpenReview ↗

Response for Q1/ Q2/ W1

We are grateful for your recognition of the problem's importance and your suggestions on the evaluation.

Response for Q1

A single-role edit can change the system answer through the messages that downstream agents receive. MAKE uses interaction-conditioned keys to help the edited role provide the new fact even when upstream messages favor the old answer.

1. Propagation of Edited Knowledge through Agent Communication

The edited role's output becomes evidence for downstream agents. With their original parameters, these agents can use the new fact in the received messages to generate their answers. This communication mechanism allows a targeted parameter update to influence the final system response.

2. System-Level Evaluation of Single-Role Editing

We measure this effect directly with MAES. The multi-round experiment tracks final-system success over repeated communication (Figure 4, Appendix B.3), and the peer-pressure experiment measures both the edited role's resistance and final-system success under conflicting messages (Figure 8, Appendix B.7). These results support the effectiveness of selective editing through downstream interaction.

Response for Q2 / W1

Additional contextual support improves the baseline editors, and MAKE retains an advantage when they receive the same contextual inputs. The MAKE ablation also identifies a contribution from DRO.

1. Baseline Comparison with Clean and Multi-Key Inputs

The main-experiment adaptation uses the complete greedy input to the edited role as a single contextual constraint (Appendix A.4.6). To test the effect of this choice, we evaluated every method with a clean-query key, a greedy-interaction key, and six keys from the clean query, greedy anchor, and four selected contexts. Each editor retains its update layers and target-value optimization. The full MAKE/MAKE w/o DRO pair additionally shares identical numerical keys and targets.Tables A–C report Qwen2.5-7B/CounterFact results averaged equally across the three architectures. All five metrics are percentages, and Avg. is their arithmetic mean.

Table A. Editing with the clean-query key

MethodSAESMAESACRMAGMAPAvg.
MEMIT99.2028.6773.8715.7725.3348.57
AlphaEdit99.0029.7374.8715.4325.5348.91
CAKE99.0029.5374.7315.9725.4748.94
MAKE w/o DRO98.8033.8078.2719.7025.3351.18
MAKE99.2034.1379.0019.7325.6051.53

Table B. Editing with the greedy anchor

MethodSAESMAESACRMAGMAPAvg.
MEMIT55.1047.6761.2318.8326.6041.89
AlphaEdit64.0754.9771.9321.4726.4747.78
CAKE66.2054.1774.6722.3325.7748.63
MAKE w/o DRO69.6742.0064.6032.1722.5746.20
MAKE87.4055.7383.3339.1325.8358.28

Table C. Multi-key editing with the same contextual inputs

MethodSAESMAESACRMAGMAPAvg.
MEMIT94.3355.1382.4022.8025.1055.95
AlphaEdit94.0755.0779.7323.2325.4355.51
CAKE94.0754.4779.4023.3025.5755.36
MAKE w/o DRO92.6747.6780.0038.1325.9056.87
MAKE94.0456.5384.0840.0726.1060.16

Tables A–C examine whether MAKE’s advantage comes from the baselines lacking clean-query keys or multiple contexts. The conventional editors achieve approximately 99% SAES with clean-query editing in Table A. Compared with greedy-context editing in Table B, adding the clean query and selected contexts in Table C restores baseline SAES to 94.07–94.33% while improving MAES. Yet MAKE still achieves 56.53% MAES and 40.07% MAG, exceeding the strongest baseline values of 55.13% and 23.30%. Thus, providing the same contexts improves the baselines but does not eliminate MAKE’s advantage in system-level editing success and generalization on related queries.

2. Ablation of the DRO Contribution

Holding the layer, keys, and targets fixed in Table C, retaining DRO raises MAKE's MAES from 47.67% to 56.53% (+8.86%). The original Llama/CounterFact ablation likewise reports gains of 10.5% in MAES and 14.7% in ACR (Table 2). That table also compares full MAKE, MAKE using only the clean key and greedy anchor, and MAKE using randomly selected contexts, with MAES of 35.1%, 12.4%, and 19.3%, respectively. Together, these controls distinguish the contributions of contextual support and DRO.

AuthorsOpenReview ↗

Response for Q3 and W2

Response for Q3 and W2

The frozen MAKE update operates on messages generated in the current interaction. We evaluate it on unseen trajectories and on changes to queries, decoding, and communication connections.

1. Generalization to Unseen Peer Messages

The greedy anchor provides an editing context. At inference, the updated role processes incoming messages directly, with no matching-trajectory lookup or further patch optimization. Table 9 in Appendix B.6 tests the same frozen patch on three independently sampled trajectories excluded from its editing contexts. The Llama/CounterFact average MAES changes from 33.33% to 33.31%, with 100% valid-trajectory coverage.We evaluate Llama-3.1-8B on the CounterFact dataset. We report MAES and Coverage as percentages and calculate Drop as Greedy-MAES minus Unseen-MAES, expressed as %. We average the unseen outcomes within each edit before aggregating across edits. Coverage measures the proportion of valid held-out trajectories.

Table D. Held-out peer-message evaluation

ArchitectureGreedy-MAES ↑Unseen-MAES ↑Drop ↓Coverage ↑
Pipeline56.2055.001.20100.00
Debate21.6024.20−2.60100.00
Hierarchical22.2020.731.47100.00
Average33.3333.310.02100.00

Table D shows that the same frozen patches achieve an average MAES of 33.31% on unseen peer messages, compared with 33.33% on greedy trajectories, with absolute changes below 2.60% across all three architectures and 100% valid-trajectory coverage. This demonstrates that MAKE’s effectiveness extends beyond the greedy trajectories used during editing: the updated role can process newly generated interaction messages without reoptimizing the patch. These results support MAKE’s use of interaction contexts to construct edits that remain effective on unseen peer messages.


2. Robustness to Query, Decoding, and Topology Changes

We further evaluated Qwen2.5-7B/CounterFact under four changes, reusing each method's frozen patch throughout. MAKE uses multi-key inputs and the baseline adaptations use the greedy context, following Appendix A.4.6.

  • Out-of-distribution queries. The two CounterFact paraphrases of each edited fact are evaluated with greedy decoding.
  • Stochastic decoding (peers). Upstream peers use temperature 0.7 and top-p 0.9. The edited role and final aggregation remain greedy.
  • Stochastic decoding (all roles). Every role uses the same sampling settings.
  • Dynamic communication topologies. Specified message connections are independently removed with probability 0.3, with fixed roles and greedy decoding. All methods receive the same removed connections for each edit and repetition.

The sampling and topology conditions use three repetitions per edit, averaged within each edit before aggregation. We average the results equally across the three architectures and report MAES as a percentage. We highlight the highest MAES in each column in bold.

Table E. MAES under out-of-distribution queries, stochastic decoding, and dynamic communication topologies

MethodOriginal ↑Out-of-distribution queries ↑Stochastic decoding (peers) ↑Stochastic decoding (all roles) ↑Dynamic communication topologies ↑
MEMIT47.6718.1735.5132.8743.69
AlphaEdit55.0021.3343.1340.1650.11
CAKE54.2022.1743.2240.3150.42
MAKE56.5332.3750.7646.8751.02

Table E shows that, without reoptimizing patches, MAKE achieves the highest MAES under query paraphrases, stochastic decoding by peers, stochastic decoding by all roles, and dynamic communication topologies, exceeding the strongest baselines by 10.20%, 7.54%, 6.56%, and 0.60%, respectively. These changes produce inputs and communication processes that differ from the greedy trajectories used during editing, MAKE retains its advantage. This demonstrates effectiveness beyond predetermined greedy interactions, consistent with MAKE’s design of constructing robust updates from multiple interaction contexts.

AuthorsOpenReview ↗

Response for W3 [Part 1/2]

Response for W3 [Part 1/2]

MAKE leads under semantic MAES and the average of final-system metrics. Below, Llama and Qwen denote Llama-3.1-8B and Qwen2.5-7B.

1. Semantic Validation with an LLM Judge

We re-scored the same answers with Qwen2.5-14B-Instruct, hiding method names. The judge receives the question, target, and complete answer, accepts equivalent wording, and distinguishes assertions from negation or quotation (Appendix A.4.7). We average four datasets and three architectures equally and will release the judge prompt. Scoring the same complete answers lets us examine whether the method ranking persists under semantic evaluation.

Table F. Original and semantic MAES (%)

MethodLlama: Original ↑Llama: Judge ↑Qwen: Original ↑Qwen: Judge ↑
FT38.2312.8027.5522.50
MEND29.8812.1814.7312.94
ROME21.3310.3145.3530.38
MEMIT32.7313.1838.0027.74
AlphaEdit40.5015.4445.0836.35
CAKE40.4315.3546.0536.30
CrispEdit40.9513.9646.1834.12
MAKE46.5816.2048.0039.97

Table F shows that, when the same responses are evaluated semantically using an LLM-as-a-Judge, MAKE retains the highest MAES on both models, exceeding the strongest baseline, AlphaEdit, by 0.76% and 3.62%, respectively. Although absolute scores decrease for all methods, MAKE’s advantage persists, indicating that its gains extend beyond target-string occurrence to whether the final answer expresses the intended fact. This provides complementary evidence that MAKE improves system-level editing success beyond the original matching rules.

2. Fluency, Consistency, and Perplexity

We score complete CounterFact answers across three architectures. Fluency is 100(H2+2H3)/3100(H_2+2H_3)/3 using bigram/trigram entropy. Consis is TF-IDF cosine similarity to official Wikipedia snippets, scaled by 100 using the supplied vocabulary and IDF weights. Both scores average architectures equally. PPL uses external GPT-2 (124M), with answer-token losses pooled across architectures and averaged per token before exponentiation.

Table G. Answer quality on CounterFact

MethodLlama: Fluency ↑Llama: Consis ↑Llama: PPL ↓Qwen: Fluency ↑Qwen: Consis ↑Qwen: PPL ↓
FT48814.5216.2634011.3530.59
MEND49013.4615.9535810.3633.67
ROME48812.1615.6835114.5830.32
MEMIT48814.0915.7736513.2933.08
AlphaEdit48514.0816.5834114.0931.40
CAKE48314.1116.4734014.2631.33
CrispEdit48214.3218.3928914.0529.93
MAKE49414.5916.6836614.4728.46

Table G shows that MAKE achieves the highest Fluency on both models, the highest Consis on Llama, and the lowest PPL on Qwen. Together with the semantic MAES results in Table F, these findings show that MAKE’s advantages extend to expressing the edited fact while maintaining answer quality. This is consistent with MAKE’s design: using interaction contexts to improve editing effectiveness during multi-agent communication, while constraining changes on background keys to limit disruption to the model’s existing behavior.

Reviewer_6GBaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies parametric knowledge editing when the edited LLM is one role inside a multi-agent system (MAS) and the other agents are left unedited. It argues that role instructions and peer messages shift the MLP key used to retrieve the edited fact ("interaction-induced representation shift"), and gives simple identities showing that a rank-one ROME edit is rescaled by η(c) under a shifted key and is blind to shift directions outside its row space. The proposed method MAKE has two phases: (i) run the upstream agents, sample 20 contexts, and keep the greedy trajectory plus the 4 contexts with the largest key-shift norm; (ii) solve a closed-form multi-key constrained write over the clean key and the mined contextual keys, with the background second-moment matrix C inflated by a scalar ridge term derived from a 2-Wasserstein upper bound. Experiments on Llama-3.1-8B and Qwen2.5-7B, four editing datasets, and three hand-built MAS topologies (Pipeline, Debate, Hierarchical) report five metrics (SAES/MAES/ACR/MAG/MAP). MAKE obtains the best average, with large gains on clean-prompt success and conflict robustness, and smaller gains on final-system success. Additional experiments cover sequential editing (200 edits), multi-round interaction, ablations, cost, role location, held-out peer messages, and peer-pressure levels.

Critical strengths
  1. The problem framing is timely and well motivated. Evaluating edits at the system output of a MAS rather than on a clean prompt is a real gap, and the five-metric protocol (especially ACR, which isolates the edited agent under conflicting peer messages, and MAES, which measures survival through aggregation) is a useful decomposition that others could reuse.
  2. The method is cheap and closed-form. Phase II reduces to an (m+1)×(m+1) Gram inverse and recovers ROME at m=0, so the approach is easy to implement and amortizes well (Table 8). The ablation in Table 2 does show that the contextual keys, not just extra constraints, drive MAES/ACR gains (random contexts 19.3 MAES vs 35.1).
Critical weaknesses
  1. The baselines are non-standard adaptations whose failure appears to be an artifact of the adaptation, so the headline comparison is not sound. Appendix A.4.6 states that MEMIT, AlphaEdit and CAKE extract their key and target value from the full multi-agent context Cadv[0], "preferentially locating the subject in the anchor's query segment", while SAES is evaluated on the clean prompt. The result is SAES of 2.7/2.9/2.9 on CounterFact for Llama (Table 1), essentially the pre-edited 1.1, whereas these methods report ~99% efficacy on Llama-3-8B CounterFact in their own papers. Tellingly, their MAES (30.9/33.5/33.2) is far above their SAES, the opposite of the paper's narrative. So three of the strongest baselines are being shown with a mis-matched key, and ROME is a hybrid (clean key, contextual value). The two baselines that should be mandatory are missing: (a) the standard clean-prompt configuration of each editor, and (b) MEMIT/AlphaEdit/CAKE given exactly MAKE's multi-key support K (clean + greedy + 4 mined keys), which these batch editors support natively. Without (b), one cannot tell whether the gain comes from the Wasserstein-inflated covariance or simply from feeding contextual keys to any locate-then-edit method. The "-DRO" ablation is the closest proxy, and it already shows that most of the Avg gain (50.0 of 56.0) is the multi-key support alone.
  2. The system-level gains, which are the paper's actual claim, are small, single-seed, and not statistically supported, and the sequential-editing comparison is confounded by the rank-8 truncation. Table 4 lists 1 seed. On Qwen (Table 5) MAKE's MAES is 56.5 vs 55.6 (ROME), 60.2 vs 60.1 (CrispEdit), 37.3 vs 36.9 (AlphaEdit), 38.0 vs 37.8 (CrispEdit): all within one binomial standard error at N=2000. On Llama, CounterFact MAES rises only from 33.4 to 36.5. The abstract's "+11.6%" is driven by SAES and ACR, which are edited-agent metrics, not system metrics. For Figure 3, Appendix A.4.1 says every baseline update is truncated to rank-8 factors stored in fp16; truncating AlphaEdit's null-space-projected update by SVD destroys the null-space property, which plausibly explains why AlphaEdit degrades after 200 edits here when its own paper shows stability over thousands. The theory is also thin relative to the framing: Theorems 2.1/2.2 are algebraic identities, Prop. 2.3 is the chain rule, and Prop. 3.1 (Minkowski + Young + ‖U‖op ≤ ‖U‖F) reduces the "Wasserstein DRO" to adding ϵ*·sδ·I to C with a heuristic ϵ*, which the appendix itself calls "an empirical calibration rule, not a confidence-region guarantee".
Questions for authors

Clarification (answers could change my score):

  1. Please report MEMIT/AlphaEdit/CAKE in their standard clean-prompt configuration on your MAS protocol, and with MAKE's exact multi-key support K (clean + greedy + 4 mined keys, same vg*). How does MAKE compare to the latter?
  2. Is the rank-8 factorization applied to MEMIT/AlphaEdit/CAKE's multi-layer updates in Figures 3 and 6? If so, please rerun the sequential-editing study with dense (un-truncated) updates and with edits accumulated in the weights rather than looked up per edit index (Eq. 24).
  3. Please provide multi-seed means and standard errors (or at least bootstrap CIs over edits) for MAES/MAG in Tables 1 and 5. Which differences survive?
  4. How is "+11.6% over the strongest baseline" and "30.04%" (Sec. 1) computed? From Table 1 and Table 5 I get +12.2 and +5.4 on Avg. What setting does Figure 1 show (dataset, architecture, editor used for "Edit all roles"), and why does editing all roles reduce MAES relative to editing one?
  5. Table 3 reports MEND Avg. 50.9 vs ROME 38.4, but Table 1 has MEND below ROME on every dataset. Which subset and metrics is Table 3 averaging?
  6. Sensitivity to ϵ*: if Cϵ* = C + λI with λ tuned on a dev split, does the "-DRO" gap in Table 2 remain? What is the magnitude of ϵ*·sδ relative to the diagonal of C in practice?
  7. The role prompts (Figs. 10–12) tell the edited role "You have CURRENT information" and tell peers that the edited role may be more current. How much of MAES/ACR depends on this prompt design? Please report results with neutral role prompts, and also with the trivial baseline of inserting the new fact into the edited agent's system prompt (no parameter edit).
  8. The held-out test (B.6) samples at temperature 0.3 from the same MAS. Does a patch mined under one architecture transfer to another architecture or to a different peer model?
  9. Why is CoRE (Park et al., 2025), which targets context-robust editing, not a baseline?

Suggestions (not affecting score): drop the "adversarial" label for top-m selection by shift norm; state Fig. 1's setting in the caption; move metric definitions to the main text.

12 comments
AuthorsOpenReview ↗

Response for Q1 and W1

Thank you for your detailed assessment and constructive suggestions.

Response for Q1 and W1

MAKE retains an advantage when conventional editors receive the same contextual inputs, and the comparison between MAKE and MAKE w/o DRO using identical layers, keys, and target values isolates the contribution of DRO.

1. Baseline Comparison with Clean and Multi-Key Support

We evaluated MEMIT, AlphaEdit, CAKE, MAKE w/o DRO, and MAKE on Qwen2.5-7B/CounterFact with three input conditions. Tables A and B use one clean-query or greedy-interaction key per edited layer. Table C uses six inputs shared by all methods, comprising the clean query, greedy anchor, and four selected contexts. Each editor extracts keys at its own update layers and optimizes targets at its own representation. Each patch is active only for the designated role.We average the results equally across the three architectures and report them as percentages. We calculate Avg. as the mean of SAES, MAES, ACR, MAG, and MAP.

Table A. Editing with the clean-query key

MethodSAESMAESACRMAGMAPAvg.
MEMIT99.2028.6773.8715.7725.3348.57
AlphaEdit99.0029.7374.8715.4325.5348.91
CAKE99.0029.5374.7315.9725.4748.94
MAKE w/o DRO98.8033.8078.2719.7025.3351.18
MAKE99.2034.1379.0019.7325.6051.53

Table B. Editing with the greedy anchor

MethodSAESMAESACRMAGMAPAvg.
MEMIT55.1047.6761.2318.8326.6041.89
AlphaEdit64.0754.9771.9321.4726.4747.78
CAKE66.2054.1774.6722.3325.7748.63
MAKE w/o DRO69.6742.0064.6032.1722.5746.20
MAKE87.4055.7383.3339.1325.8358.28

Table C. Multi-key editing with the same contextual inputs

MethodSAESMAESACRMAGMAPAvg.
MEMIT94.3355.1382.4022.8025.1055.95
AlphaEdit94.0755.0779.7323.2325.4355.51
CAKE94.0754.4779.4023.3025.5755.36
MAKE w/o DRO92.6747.6780.0038.1325.9056.87
MAKE94.0456.5384.0840.0726.1060.16

Tables A–C show the complementary contributions of interaction contexts and Wasserstein DRO. Clean-query editing in Table A gives the conventional baselines high SAES but low MAES; greedy-context editing in Table B improves MAES while reducing SAES. Combining clean and contextual keys in Table C improves performance across both conditions. With the same contextual inputs, MAKE achieves 56.53% MAES and 40.07% MAG, exceeding the strongest baseline values of 55.13% and 23.30%. Moreover, with identical layers, keys, and target values, retaining DRO raises MAKE’s MAES from 47.67% to 56.53%. Thus, interaction contexts help editing accommodate multi-agent inputs, while Wasserstein DRO provides additional system-level gains beyond those obtained from the contexts alone.

2. Isolating DRO under Identical Editing Constraints

Full MAKE and MAKE w/o DRO use identical layers, keys, and targets. In Table C, retaining DRO improves MAES by 8.86%. The original Llama/CounterFact ablation likewise reports gains of 10.5% in MAES and 14.7% in ACR (Table 2). These comparisons identify an additional contribution from DRO with contextual support fixed.

AuthorsOpenReview ↗

Response for W2 and Q2

Response for theory question of W2: The theory is also thin relative to the framing: Theorems 2.1/2.2 are algebraic identities....

The analysis explains why clean-prompt edits can fail during interaction and derives the correction used by MAKE.

1. Failure Analysis under Context Shift

Peer messages can change the retrieved value even when the edit retains its full write amplitude. Theorems 2.1 and 2.2 separate backbone shift from edit attenuation and identify shift components outside the update's fitted directions. For such a component δ⊥\delta_\perp, ΔWδ⊥=0\Delta W\delta_\perp=0, and an exact clean-key fit leaves a residual contribution Wδ⊥W\delta_\perp. The diagnostic on 100 Llama/CounterFact edits finds larger edit attenuation among failed contextual queries (Figure 5(a)). These results motivate fitting interaction-conditioned keys alongside the clean key.

2. Propagation Analysis through Downstream Agents

Proposition 2.3 uses the probability chain rule to connect local retrieval with downstream retention, allowing dependence between agents. The product of conditional stage-success probabilities gives the probability that the entire path stays correct. It equals final success when errors are absorbing and the last stage determines the answer. With downstream repair, it is a lower bound whenever a fully correct path suffices for final success.

3. Derivation of the Robust Regularizer

Proposition 3.1 bounds worst-case expected squared changes in layer output over a Wasserstein ball, assuming a nominal distribution with finite second moment dominated by CC. For sδ>0s_\delta>0, choosing parameter β⋆=ϵ⋆/sδ\beta^\star=\epsilon^\star/s_\delta yields

\qquad C_{\epsilon^\star}=C+\epsilon^\star s_\delta I_{d_m}.$$ The correction adds $\epsilon^\star s_\delta\lVert U\rVert_F^2$ to the nominal objective. With positive-definite $C_{\epsilon^\star}$ and nonsingular $\Gamma_K=K^\top C_{\epsilon^\star}^{-1}K$, **Eq. (15)** gives the unique minimizer and minimizes the energy factor in **Corollary 3.2**. The Gram stabilization in **Eq. (79)** converges to this solution as its coefficient tends to zero. ### 4. Empirical Calibration of the Wasserstein Radius MAKE sets $\epsilon^\star=s_\delta\sqrt{\widehat d_{\mathrm{eff}}/m}$ from each edit's observed shifts (**Eq. (14)**). **Theorem 2.4** bounds effective dimension under low-dimensional contextual factors and a locally affine key map. The radius is an empirical calibration (**Appendix C.5**), with the perturbation bound applying within the chosen ball. A confidence guarantee for coverage of future key distributions requires an additional statistical result. We will clarify this scope beside **Eq. (14)** and present **Eqs. (4)–(5)** as diagnostic decompositions, with a reference to numerical stabilization in **Appendix C.6**. # Response for Q2 and W2 related Q2 **Rank-8 factors provide a common, compact update format for efficient role-isolated evaluation. We additionally evaluated dense, untruncated updates accumulated directly in model weights.** ### 1. Independent and Cumulative Editing Protocols **Eq. (24)** describes independent-edit batching, where each fact is edited from the original model and its patch is selected by edit index. Sequential editing in **Section 4.2** and **Figure 6** instead computes each update from the previously edited weights, $$W_t=W_{t-1}+\Delta W_t(W_{t-1}).$$ All queries at a checkpoint use the same cumulative weights $W_t$ for the designated role. Role-based weight switching keeps the other roles at their original parameters. ### 2. General-Capability Preservation with Untruncated Updates We evaluated Llama-3.1-8B on the CounterFact dataset under the Pipeline architecture after 0, 10, 50, 100, and 200 edits. We removed rank truncation while retaining MAKE's target clamps and update-magnitude limits. We report the average Macro-F1 across SST, MRPC, CoLA, RTE, MMLU, and NLI, using the same samples across methods and checkpoints. **Table D. Six-task Macro-F1 after cumulative editing with untruncated updates** | Method | 0 edits | 10 edits | 50 edits | 100 edits | 200 edits | |:---|---:|---:|---:|---:|---:| | MEMIT | **64.2** | 64.0 | 63.7 | 58.9 | 54.0 | | AlphaEdit | **64.2** | 63.9 | 63.2 | 63.2 | 62.9 | | CAKE | **64.2** | 64.1 | 63.3 | 63.4 | 63.2 | | **MAKE** | **64.2** | **64.5** | **65.0** | **65.2** | **64.5** | ||||||| In **Table D**, MAKE maintains six-task Macro-F1 at **64.5 after 200 edits**, compared with **64.2 initially**, using untruncated updates accumulated directly in model weights. This shows that its general-capability preservation in this setting does not depend on rank-8 truncation or per-edit patch retrieval. We will include these results in the revised manuscript and clarify that **Eq. (24)** applies to independent-edit evaluation.
AuthorsOpenReview ↗

Response for Q3 and W2 related Q3

Response for Q3 and W2 related Q3

1. Edit-Level Bootstrap for Uncertainty Estimation

Given the cost of full multi-seed evaluation during rebuttal, we used the edit-level paired bootstrap suggested in your question on historical per-edit records. The procedure uses 10,000 resamples, with the same sampled edit IDs across methods and architectures within each dataset. Generalization queries remain grouped with their edit, preserving within-edit MAG aggregation. Dataset scores are averaged equally, and percentile endpoints define the historical 95% intervals.

2. 95% Confidence Intervals for MAES and MAG

Tables E and F present the means and 95% confidence intervals for MAES and MAG. Each cell reports mean [95% confidence interval] in percent. Results are averaged equally across the three architectures for each dataset.

Table E. MAES, MAG, and 95% confidence intervals on Llama-3.1-8B

MethodCounterFact MAESCounterFact MAGzsRE MAESzsRE MAGRippleEdits MAESRippleEdits MAGUniEdit MAESUniEdit MAG
FT31.3 [30.1, 32.6]21.6 [20.6, 22.6]59.8 [58.5, 61.1]54.9 [53.5, 56.3]17.3 [16.0, 18.6]14.5 [13.3, 15.7]44.5 [43.0, 46.0]8.1 [7.1, 9.1]
MEND28.2 [27.0, 29.4]23.9 [23.0, 24.9]45.1 [43.6, 46.6]41.8 [40.3, 43.3]17.1 [16.0, 18.2]13.2 [12.3, 14.1]29.1 [27.8, 30.5]6.4 [5.7, 7.2]
ROME10.5 [9.7, 11.4]7.9 [7.3, 8.5]43.3 [41.7, 44.9]33.8 [32.3, 35.4]5.8 [5.1, 6.5]4.6 [4.1, 5.2]25.7 [24.3, 27.1]4.6 [4.0, 5.3]
MEMIT30.9 [29.7, 32.1]2.6 [2.3, 2.9]57.2 [55.7, 58.7]29.0 [27.5, 30.5]10.5 [9.7, 11.4]4.4 [3.9, 4.9]32.3 [30.8, 33.8]3.7 [3.1, 4.3]
AlphaEdit33.5 [32.3, 34.8]3.0 [2.6, 3.4]61.8 [60.4, 63.2]29.9 [28.4, 31.5]23.5 [22.4, 24.7]6.4 [5.8, 7.0]43.2 [41.7, 44.7]3.8 [3.2, 4.4]
CAKE33.2 [32.0, 34.5]3.2 [2.8, 3.6]62.4 [61.0, 63.8]29.9 [28.4, 31.4]23.2 [22.1, 24.4]6.6 [6.0, 7.2]42.9 [41.4, 44.4]3.6 [3.0, 4.2]
CrispEdit33.4 [32.2, 34.7]33.0 [32.0, 34.0]60.9 [59.5, 62.3]58.3 [56.9, 59.7]28.3 [27.1, 29.6]22.1 [21.0, 23.2]41.2 [39.7, 42.7]9.0 [7.9, 10.1]
MAKE36.5 [35.3, 37.8]33.4 [32.6, 34.2]65.5 [64.2, 66.8]55.0 [53.6, 56.4]32.2 [31.0, 33.5]23.0 [22.0, 24.0]52.1 [50.7, 53.5]7.8 [7.1, 8.5]

Table F. MAES, MAG, and 95% confidence intervals on Qwen2.5-7B

MethodCounterFact MAESCounterFact MAGzsRE MAESzsRE MAGRippleEdits MAESRippleEdits MAGUniEdit MAESUniEdit MAG
FT32.4 [31.2, 33.6]16.0 [15.2, 16.8]42.3 [40.8, 43.8]36.0 [34.6, 37.5]7.7 [7.0, 8.4]6.7 [6.0, 7.4]27.8 [26.4, 29.3]4.2 [3.5, 4.9]
MEND19.3 [18.2, 20.4]12.1 [11.4, 12.8]25.3 [23.8, 26.8]22.7 [21.2, 24.2]1.8 [1.4, 2.2]4.9 [4.3, 5.5]12.5 [11.4, 13.7]3.4 [2.8, 4.0]
ROME55.6 [54.3, 57.0]37.1 [36.0, 38.2]54.7 [53.3, 56.2]48.5 [47.0, 50.0]33.7 [32.4, 35.0]18.0 [17.0, 19.1]37.4 [35.9, 39.0]5.5 [4.8, 6.3]
MEMIT47.7 [46.4, 49.0]18.8 [18.0, 19.7]49.7 [48.3, 51.1]35.1 [33.6, 36.6]24.0 [22.9, 25.1]9.1 [8.4, 9.9]30.6 [29.2, 32.0]3.9 [3.3, 4.6]
AlphaEdit55.0 [53.7, 56.3]21.5 [20.6, 22.4]59.8 [58.5, 61.2]43.2 [41.7, 44.7]36.9 [35.5, 38.3]13.2 [12.3, 14.1]28.6 [27.1, 30.2]4.5 [3.8, 5.2]
CAKE54.2 [52.9, 55.5]22.3 [21.4, 23.2]59.6 [58.2, 61.0]43.7 [42.3, 45.2]36.5 [35.1, 37.9]13.3 [12.4, 14.2]33.9 [32.4, 35.5]4.5 [3.8, 5.2]
CrispEdit52.2 [50.9, 53.6]39.8 [38.7, 40.9]60.1 [58.6, 61.6]53.5 [52.0, 55.0]34.6 [33.2, 36.0]23.3 [22.0, 24.6]37.8 [36.2, 39.4]6.4 [5.5, 7.4]
MAKE56.5 [55.2, 57.8]40.1 [39.1, 41.1]60.2 [58.7, 61.7]53.7 [52.2, 55.2]37.3 [36.0, 38.7]24.1 [23.1, 25.1]38.0 [36.5, 39.6]6.6 [5.9, 7.4]

In Tables E and F, we report 95% confidence intervals for MAES and MAG using a 10,000-resample edit-level paired bootstrap across three architectures. On Llama-3.1-8B, MAKE's system-level improvements are statistically significant: its MAES interval on CounterFact is 36.5 [35.3, 37.8], which does not overlap with AlphaEdit 33.5 [32.3, 34.8] or MEMIT 30.9 [29.7, 32.1]. MAKE consistently yields the highest mean score across all four datasets for both MAES and MAG.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Response for Q1 and W1

We appreciate your valuable feedback and support of our work.

Response for Q1 and W1

MAKE improves final-system editing success, with gains that vary across backbones and architectures. We report these gains separately from the overall score to make the system-level contribution explicit.

1. System-Level Performance across Backbones and Architectures

In Tables 1 and 5, MAKE achieves the highest architecture-averaged MAES on all four datasets for both backbones. Against the strongest MAES baseline on each dataset, the average gains are 0.4% on Qwen2.5-7B and 4.4% on Llama-3.1-8B. Against CrispEdit, the strongest baseline by four-dataset mean MAES on both backbones, the gains are 1.8 and 5.6%, respectively. The architecture-specific results show where these gains occur. Across the 12 architecture–dataset settings, MAKE leads outright in 9 and ties for first in 1 on Llama (Table 6). On Qwen, MAKE leads outright in 4 and ties for first in 1 (Table 7), with comparisons including all baselines such as FT. Thus, the architecture-averaged advantage coexists with variation across individual settings.

2. Metric Breakdown of the Overall Improvement

To show how much of the aggregate improvement reaches the system output, we separate MAES, MAG, and MAP, which evaluate final answers, from SAES and ACR, which evaluate the edited role (Appendix A.2). The following breakdown compares MAKE with each backbone's strongest baseline by five-metric average. Each contribution is one fifth of the metric's four-dataset mean difference.

Table A. Contribution of Individual Metrics to the Overall Improvement

Contribution to the overall gain (pp)Qwen (relative to ROME)Llama (relative to CrispEdit)
SAES+2.410 (44.4%)+8.415 (68.6%)
ACR+1.630 (30.0%)+2.410 (19.7%)
MAES+0.530 (9.8%)+1.125 (9.2%)
MAG+0.770 (14.2%)−0.160 (−1.3%)
MAP+0.085 (1.6%)+0.470 (3.8%)
System-level subtotal (MAES + MAG + MAP)+1.385 (25.5%)+1.435 (11.7%)
Total gain across all five metrics+5.425+12.260

Table A shows that system-level metrics contribute 25.5% of the overall improvement on Qwen and 11.7% on Llama. To assess system-level performance independently of SAES and ACR, we also compare the mean of MAES, MAG, and MAP. MAKE exceeds the strongest baseline under this aggregation on both backbones, achieving 32.7% compared with 30.4% on Qwen and 32.2% compared with 29.8% on Llama. Thus, MAKE’s advantage remains when the edited-agent metrics are excluded: the improvement extends to the system’s final answers, rather than being confined to the designated agent. This supports MAKE’s goal of improving system-level editing through an update to a single agent. We will present this breakdown and the MAES margins in Section 4.1.

3. Context-Selection Ablation for System-Level Success

Table 2 ablation tests whether interaction-aware context selection improves system responses. Both variants retain the clean key and greedy anchor and select four further contexts from the same pool. Replacing uniform random selection with selection by context shift raises MAES from 19.3% to 35.1% (+15.8%) and ACR from 71.8% to 90.1%, while SAES changes from 97.3% to 97.5%. This isolates the benefit of context selection for performance during interaction.

AuthorsOpenReview ↗

Response for Q2 and W2

Response to Q2 and W2

MAKE achieves higher system-level editing success than multi-key editing baselines using the same mined contexts, and the ablation shows the additional benefit of Wasserstein DRO.

1. Ablation of Wasserstein DRO

On Llama-3.1-8B/CounterFact, retaining DRO raises MAES from 24.6% to 35.1% (+10.5%) and ACR from 75.4% to 90.1% (+14.7%) (Table 2). The two MAKE variants use identical contexts, keys, target values, edited layers, and other solver settings. Their difference is the correction ϵ⋆sδI\epsilon^\star s_\delta I added to the background matrix CC (Eqs. 14–15).

2. Comparison with Multi-Key Editing Baselines

We evaluated MEMIT, AlphaEdit, CAKE, MAKE w/o DRO, and MAKE on the same 2,000 CounterFact edits with Qwen2.5-7B across the three architectures. Tables B–D compare editing with the clean query, the greedy interaction context, and the clean query together with the greedy anchor and four selected contexts, respectively. In Table D, all methods receive the same contextual inputs; each method extracts keys at its own update layers and retains its update rule. Baseline regularization follows Appendix A.4.6, including MEMIT’s covariance weight of 15,000 and the ℓ2\ell_2 coefficient of 10 for AlphaEdit and CAKE.

All metrics are percentages averaged equally across the three architectures, and Avg. is the five-metric mean. Tables B and C use one key per edited layer, while Table D uses six. Table C’s single greedy key differs from the paper’s Greedy Only ablation, which retains both the clean key and greedy anchor.

Table B. Editing with the Clean-Query Key

MethodSAESMAESACRMAGMAPAvg.
MEMIT99.2028.6773.8715.7725.3348.57
AlphaEdit99.0029.7374.8715.4325.5348.91
CAKE99.0029.5374.7315.9725.4748.94
MAKE w/o DRO98.8033.8078.2719.7025.3351.18
MAKE99.2034.1379.0019.7325.6051.53

Table C. Editing with the Greedy Interaction Key

MethodSAESMAESACRMAGMAPAvg.
MEMIT55.1047.6761.2318.8326.6041.89
AlphaEdit64.0754.9771.9321.4726.4747.78
CAKE66.2054.1774.6722.3325.7748.63
MAKE w/o DRO69.6742.0064.6032.1722.5746.20
MAKE87.4055.7383.3339.1325.8358.28

Table D. Multi-Key Editing with the Same Mined Contexts

MethodSAESMAESACRMAGMAPAvg.
MEMIT94.3355.1382.4022.8025.1055.95
AlphaEdit94.0755.0779.7323.2325.4355.51
CAKE94.0754.4779.4023.3025.5755.36
MAKE w/o DRO92.6747.6780.0038.1325.9056.87
MAKE94.0456.5384.0840.0726.1060.16

Tables B–D show why MAKE jointly constrains clean and contextual keys. Clean-query editing gives the three conventional baselines approximately 99% SAES but less than 30% MAES. Greedy-context editing improves MAES while reducing SAES; combining clean and contextual keys brings both above 94% SAES and 54% MAES, supporting joint editing across isolated and interaction conditions.With the same mined contexts in Table D, MAKE achieves 56.53% MAES and 40.07% MAG, exceeding the strongest baseline values of 55.13% and 23.30%. Its advantage therefore extends to system-level generalization and is not fully explained by additional contexts.With identical layers, keys, targets, and other solver settings, retaining Wasserstein DRO improves MAES by 8.86%, ACR by 4.08%, and MAG by 1.94%. This identifies an additional contribution from regularizing the update over nearby background-key distributions beyond fitting the selected keys.

3. Theoretical Basis of the Robust Regularizer

Proposition 3.1 derives the correction from an upper bound on the worst-case expected squared change in layer output over a Wasserstein ball of background-key distributions. Corollary 3.2 bounds sensitivity outside the fitted key span. These results justify the layer-level regularizer; system-level editing success additionally depends on how downstream agents use the edited response (Section 2.3).

AuthorsOpenReview ↗

Response for Q3

Response for Q3

MAKE calibrates its radius from each edit's observed context shifts. Across both backbones and all four benchmarks, unseen peer messages reduce average MAES by 0.66%.

1. Adaptive Radius Calibration from Context Shifts

To adapt regularization to each edit, MAKE measures the shifts from the clean key to the greedy anchor and four selected contextual keys (m=5m=5). Their empirical covariance Σ^δ\widehat\Sigma_\delta and effective dimension d^eff\widehat d_{\mathrm{eff}} determine the radius (Eqs. 6, 10, and 14),

sδ=tr⁡(Σ^δ)dm,ϵ⋆=sδd^effm.s_{\delta}=\sqrt{\frac{\operatorname{tr}(\widehat{\Sigma}_{\delta})}{d_{m}}},\quad \epsilon^{\star}=s_{\delta}\sqrt{\frac{\widehat{d}_{\mathrm{eff}}}{m}}. This rule uses the dispersion and effective dimension of observed shifts to set the correction ϵ⋆sδI\epsilon^\star s_\delta I, without validation-set tuning. The radius is fixed before unseen messages are generated. The Wasserstein ball is centered on the nominal background-key distribution P0P_0, and Appendix C.5 identifies the radius rule as empirical calibration.

2. Unseen-Context Evaluation across Models and Benchmarks

To test robustness beyond the editing contexts, we compare each frozen patch on its greedy trajectory and three independently generated unseen peer-message trajectories. The patch, radius, and edited role's greedy decoding are fixed. Unseen-MAES averages the three outcomes within each edit before averaging over edits.The original Llama/CounterFact test reports an average decline of 0.02%, from 33.33% to 33.31%, with 100% trajectory coverage (Table 9, Appendix B.6). The expanded evaluation covers both backbones, four datasets, and three architectures. Each cell below reports Greedy-MAES / Unseen-MAES / Δ\Delta, where Δ=Greedy−Unseen\Delta=\mathrm{Greedy}-\mathrm{Unseen} in percentage points.

Table E. Greedy and unseen MAES across models, datasets, and architectures

BackboneBenchmarkPipelineDebateHierarchical
Llama-3.1-8BCounterFact56.20 / 55.00 / 1.2021.60 / 24.20 / -2.6022.20 / 20.73 / 1.47
Llama-3.1-8BzsRE80.53 / 77.89 / 2.6457.62 / 57.75 / -0.1356.40 / 54.27 / 2.13
Llama-3.1-8BRippleEdits46.22 / 47.81 / -1.5925.22 / 24.27 / 0.9522.64 / 21.16 / 1.48
Llama-3.1-8BUniEdit63.84 / 62.47 / 1.3743.00 / 42.33 / 0.6739.40 / 40.73 / -1.33
Qwen2.5-7BCounterFact71.91 / 66.85 / 5.0651.21 / 54.69 / -3.4845.00 / 43.80 / 1.20
Qwen2.5-7BzsRE70.80 / 69.87 / 0.9356.11 / 52.49 / 3.6250.40 / 50.27 / 0.13
Qwen2.5-7BRippleEdits56.22 / 56.78 / -0.5630.84 / 32.84 / -2.0023.24 / 21.30 / 1.94
Qwen2.5-7BUniEdit52.80 / 50.27 / 2.5329.80 / 29.13 / 0.6731.40 / 31.93 / -0.53

Table E extends unseen peer-message evaluation to both models, all four datasets, and three architectures. With the editing patches and radii unchanged, mean MAES changes from 46.03% to 45.37%, a decline of only 0.66%, with absolute changes within 3% in 21 of 24 settings. This shows that MAKE’s effectiveness extends beyond the greedy trajectories used during editing: the same updates remain effective on peer messages not used to construct them. These results provide empirical evidence across models and datasets for calibrating the radius from observed context shifts. The peer-pressure experiment further evaluates MAKE under stronger conflicting messages (Figure 8, Appendix B.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.