- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies knowledge editing in multi-agent systems (MAS) and proposes MAKE. MAKE edits one designated agent in two stages. It first mines adversarial contexts, then applies Wasserstein distributionally robust optimization (DRO) to limit the representation shifts that agent interaction causes.
Critical strengths
- The problem is realistic and under-explored. Uncoordinated edits to individual agents, or representation shifts from interaction, can make the whole system decide wrongly.
- Experiments cover several interaction topologies, including Debate, Hierarchical, and Pipeline.
- The ablations measure how upstream prompts, round dynamics, and peer pressure in high-conflict settings lower system-level success downstream.
Critical weaknesses
- The baseline comparison is unfair. MEMIT, AlphaEdit, and CAKE receive only a single greedy context and no clean query keys. Their scores collapse as a result (e.g., SAES below 3%), which inflates MAKE’s advantage.
- MAKE evaluates deterministic greedy trajectories under the test setting in advance and writes them into the parameters as anchors. Since the anchors match the evaluation setup, it is unclear how the method handles open-ended queries or interaction paths it has not seen.
- Most metrics rely on string matching. A semantic judgment (e.g., LLM-as-a-Judge) and a fluency or perplexity measure would make the results more convincing. The overall average also combines local node metrics with the much lower system-level metrics (MAES, MAG), which may hide weak system-level performance. Reporting the two groups separately would be clearer.
Questions for authors
- How can a single-node edit reach high MAES in multi-round or high-conflict settings? There, unedited downstream or peer agents may reject or override the new fact based on their own parameters.
- If MEMIT and AlphaEdit get the same multi-context support sets and clean query keys as MAKE, does the gap remain? Or do the gains come mainly from multi-context data augmentation rather than from DRO?
- MAKE relies on pre-mined adversarial contexts and greedy anchors () that match the evaluation setup. How does it perform on out-of-distribution queries, under stochastic decoding, or on dynamic communication topologies where no precomputed trajectories exist?
7 comments
Response for Q3 and W2
Response for Q3 and W2
The frozen MAKE update operates on messages generated in the current interaction. We evaluate it on unseen trajectories and on changes to queries, decoding, and communication connections.
1. Generalization to Unseen Peer Messages
The greedy anchor provides an editing context. At inference, the updated role processes incoming messages directly, with no matching-trajectory lookup or further patch optimization. Table 9 in Appendix B.6 tests the same frozen patch on three independently sampled trajectories excluded from its editing contexts. The Llama/CounterFact average MAES changes from 33.33% to 33.31%, with 100% valid-trajectory coverage.We evaluate Llama-3.1-8B on the CounterFact dataset. We report MAES and Coverage as percentages and calculate Drop as Greedy-MAES minus Unseen-MAES, expressed as %. We average the unseen outcomes within each edit before aggregating across edits. Coverage measures the proportion of valid held-out trajectories.
Table D. Held-out peer-message evaluation
| Architecture | Greedy-MAES ↑ | Unseen-MAES ↑ | Drop ↓ | Coverage ↑ |
|---|---|---|---|---|
| Pipeline | 56.20 | 55.00 | 1.20 | 100.00 |
| Debate | 21.60 | 24.20 | −2.60 | 100.00 |
| Hierarchical | 22.20 | 20.73 | 1.47 | 100.00 |
| Average | 33.33 | 33.31 | 0.02 | 100.00 |
Table D shows that the same frozen patches achieve an average MAES of 33.31% on unseen peer messages, compared with 33.33% on greedy trajectories, with absolute changes below 2.60% across all three architectures and 100% valid-trajectory coverage. This demonstrates that MAKE’s effectiveness extends beyond the greedy trajectories used during editing: the updated role can process newly generated interaction messages without reoptimizing the patch. These results support MAKE’s use of interaction contexts to construct edits that remain effective on unseen peer messages.
2. Robustness to Query, Decoding, and Topology Changes
We further evaluated Qwen2.5-7B/CounterFact under four changes, reusing each method's frozen patch throughout. MAKE uses multi-key inputs and the baseline adaptations use the greedy context, following Appendix A.4.6.
- Out-of-distribution queries. The two CounterFact paraphrases of each edited fact are evaluated with greedy decoding.
- Stochastic decoding (peers). Upstream peers use temperature 0.7 and top-p 0.9. The edited role and final aggregation remain greedy.
- Stochastic decoding (all roles). Every role uses the same sampling settings.
- Dynamic communication topologies. Specified message connections are independently removed with probability 0.3, with fixed roles and greedy decoding. All methods receive the same removed connections for each edit and repetition.
The sampling and topology conditions use three repetitions per edit, averaged within each edit before aggregation. We average the results equally across the three architectures and report MAES as a percentage. We highlight the highest MAES in each column in bold.
Table E. MAES under out-of-distribution queries, stochastic decoding, and dynamic communication topologies
| Method | Original ↑ | Out-of-distribution queries ↑ | Stochastic decoding (peers) ↑ | Stochastic decoding (all roles) ↑ | Dynamic communication topologies ↑ |
|---|---|---|---|---|---|
| MEMIT | 47.67 | 18.17 | 35.51 | 32.87 | 43.69 |
| AlphaEdit | 55.00 | 21.33 | 43.13 | 40.16 | 50.11 |
| CAKE | 54.20 | 22.17 | 43.22 | 40.31 | 50.42 |
| MAKE | 56.53 | 32.37 | 50.76 | 46.87 | 51.02 |
Table E shows that, without reoptimizing patches, MAKE achieves the highest MAES under query paraphrases, stochastic decoding by peers, stochastic decoding by all roles, and dynamic communication topologies, exceeding the strongest baselines by 10.20%, 7.54%, 6.56%, and 0.60%, respectively. These changes produce inputs and communication processes that differ from the greedy trajectories used during editing, MAKE retains its advantage. This demonstrates effectiveness beyond predetermined greedy interactions, consistent with MAKE’s design of constructing robust updates from multiple interaction contexts.
Response for W3 [Part 1/2]
Response for W3 [Part 1/2]
MAKE leads under semantic MAES and the average of final-system metrics. Below, Llama and Qwen denote Llama-3.1-8B and Qwen2.5-7B.
1. Semantic Validation with an LLM Judge
We re-scored the same answers with Qwen2.5-14B-Instruct, hiding method names. The judge receives the question, target, and complete answer, accepts equivalent wording, and distinguishes assertions from negation or quotation (Appendix A.4.7). We average four datasets and three architectures equally and will release the judge prompt. Scoring the same complete answers lets us examine whether the method ranking persists under semantic evaluation.
Table F. Original and semantic MAES (%)
| Method | Llama: Original ↑ | Llama: Judge ↑ | Qwen: Original ↑ | Qwen: Judge ↑ |
|---|---|---|---|---|
| FT | 38.23 | 12.80 | 27.55 | 22.50 |
| MEND | 29.88 | 12.18 | 14.73 | 12.94 |
| ROME | 21.33 | 10.31 | 45.35 | 30.38 |
| MEMIT | 32.73 | 13.18 | 38.00 | 27.74 |
| AlphaEdit | 40.50 | 15.44 | 45.08 | 36.35 |
| CAKE | 40.43 | 15.35 | 46.05 | 36.30 |
| CrispEdit | 40.95 | 13.96 | 46.18 | 34.12 |
| MAKE | 46.58 | 16.20 | 48.00 | 39.97 |
Table F shows that, when the same responses are evaluated semantically using an LLM-as-a-Judge, MAKE retains the highest MAES on both models, exceeding the strongest baseline, AlphaEdit, by 0.76% and 3.62%, respectively. Although absolute scores decrease for all methods, MAKE’s advantage persists, indicating that its gains extend beyond target-string occurrence to whether the final answer expresses the intended fact. This provides complementary evidence that MAKE improves system-level editing success beyond the original matching rules.
2. Fluency, Consistency, and Perplexity
We score complete CounterFact answers across three architectures. Fluency is using bigram/trigram entropy. Consis is TF-IDF cosine similarity to official Wikipedia snippets, scaled by 100 using the supplied vocabulary and IDF weights. Both scores average architectures equally. PPL uses external GPT-2 (124M), with answer-token losses pooled across architectures and averaged per token before exponentiation.
Table G. Answer quality on CounterFact
| Method | Llama: Fluency ↑ | Llama: Consis ↑ | Llama: PPL ↓ | Qwen: Fluency ↑ | Qwen: Consis ↑ | Qwen: PPL ↓ |
|---|---|---|---|---|---|---|
| FT | 488 | 14.52 | 16.26 | 340 | 11.35 | 30.59 |
| MEND | 490 | 13.46 | 15.95 | 358 | 10.36 | 33.67 |
| ROME | 488 | 12.16 | 15.68 | 351 | 14.58 | 30.32 |
| MEMIT | 488 | 14.09 | 15.77 | 365 | 13.29 | 33.08 |
| AlphaEdit | 485 | 14.08 | 16.58 | 341 | 14.09 | 31.40 |
| CAKE | 483 | 14.11 | 16.47 | 340 | 14.26 | 31.33 |
| CrispEdit | 482 | 14.32 | 18.39 | 289 | 14.05 | 29.93 |
| MAKE | 494 | 14.59 | 16.68 | 366 | 14.47 | 28.46 |
Table G shows that MAKE achieves the highest Fluency on both models, the highest Consis on Llama, and the lowest PPL on Qwen. Together with the semantic MAES results in Table F, these findings show that MAKE’s advantages extend to expressing the edited fact while maintaining answer quality. This is consistent with MAKE’s design: using interaction contexts to improve editing effectiveness during multi-agent communication, while constraining changes on background keys to limit disruption to the model’s existing behavior.
Response for Q1/ Q2/ W1
Response for Q1
A single-role edit can change the system answer through the messages that downstream agents receive. MAKE uses interaction-conditioned keys to help the edited role provide the new fact even when upstream messages favor the old answer.
1. Propagation of Edited Knowledge through Agent Communication
The edited role's output becomes evidence for downstream agents. With their original parameters, these agents can use the new fact in the received messages to generate their answers. This communication mechanism allows a targeted parameter update to influence the final system response.
2. System-Level Evaluation of Single-Role Editing
We measure this effect directly with MAES. The multi-round experiment tracks final-system success over repeated communication (Figure 4, Appendix B.3), and the peer-pressure experiment measures both the edited role's resistance and final-system success under conflicting messages (Figure 8, Appendix B.7). These results support the effectiveness of selective editing through downstream interaction.
Response for Q2 / W1
Additional contextual support improves the baseline editors, and MAKE retains an advantage when they receive the same contextual inputs. The MAKE ablation also identifies a contribution from DRO.
1. Baseline Comparison with Clean and Multi-Key Inputs
The main-experiment adaptation uses the complete greedy input to the edited role as a single contextual constraint (Appendix A.4.6). To test the effect of this choice, we evaluated every method with a clean-query key, a greedy-interaction key, and six keys from the clean query, greedy anchor, and four selected contexts. Each editor retains its update layers and target-value optimization. The full MAKE/MAKE w/o DRO pair additionally shares identical numerical keys and targets.Tables A–C report Qwen2.5-7B/CounterFact results averaged equally across the three architectures. All five metrics are percentages, and Avg. is their arithmetic mean.
Table A. Editing with the clean-query key
Table B. Editing with the greedy anchor
Table C. Multi-key editing with the same contextual inputs
Tables A–C examine whether MAKE’s advantage comes from the baselines lacking clean-query keys or multiple contexts. The conventional editors achieve approximately 99% SAES with clean-query editing in Table A. Compared with greedy-context editing in Table B, adding the clean query and selected contexts in Table C restores baseline SAES to 94.07–94.33% while improving MAES. Yet MAKE still achieves 56.53% MAES and 40.07% MAG, exceeding the strongest baseline values of 55.13% and 23.30%. Thus, providing the same contexts improves the baselines but does not eliminate MAKE’s advantage in system-level editing success and generalization on related queries.
2. Ablation of the DRO Contribution
Holding the layer, keys, and targets fixed in Table C, retaining DRO raises MAKE's MAES from 47.67% to 56.53% (+8.86%). The original Llama/CounterFact ablation likewise reports gains of 10.5% in MAES and 14.7% in ACR (Table 2). That table also compares full MAKE, MAKE using only the clean key and greedy anchor, and MAKE using randomly selected contexts, with MAES of 35.1%, 12.4%, and 19.3%, respectively. Together, these controls distinguish the contributions of contextual support and DRO.