acceptodds
Under review as a conference paper at ICLR 2027

Self-Evolving Multi-Agent Systems via Decentralized Memory

Abstract

Self-evolving multi-agent systems (MAS) have emerged as a promising route to LLM agents that continually improve from experience, with persistent memory at their foundation. However, existing designs almost exclusively adopt a *centralized* repository shared across agents, incurring communication and coordination overhead, raising privacy concerns, and collapsing agent diversity. We propose **DecentMem**, a *decentralized* memory framework in which each agent maintains its own dual-pool memory—an exploitation pool of consolidated past trajectories and an exploration pool of LLM-generated candidates for unseen contexts. The two pools are reweighted online based on stage-wise feedback from an LLM-as-a-judge. Theoretically, we prove that this design guarantees global reachability of the solution space and achieves cumulative regret, matching the stochastic bandit lower bound up to constants. In practice, across three MAS frameworks (AutoGen, DyLAN, AgentNet), three Qwen3 backbones (4B/8B/14B), two Gemma4 backbones (E2B/E4B), and five benchmarks spanning math, code, QA, and embodied tasks, DecentMem improves average accuracy by up to 23.8% over the strongest centralized memory baseline and by up to 52.5% over the no-memory baseline, while reducing token usage by up to 49%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving multi-agent systems, Decentralized memory, Multi-agent systems, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_mZAvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DecentMem, a decentralized memory framework for LLM-based multi-agent systems. Each agent maintains an exploitation pool for retrieving historical experiences and an exploration pool for generating new experiences. An online router dynamically balances the two based on stage-wise feedback from an LLM judge. The paper provides theoretical guarantees on global reachability and logarithmic regret, and evaluates the framework across multiple MAS architectures, LLM backbones, and benchmarks.

Critical strengths
  1. Meaningful problem motivation. The paper addresses an interesting problem of memory management in multi-agent systems. Maintaining agent-specific experiences rather than a shared memory repository is a reasonable approach to preserving role specialization and reducing unnecessary memory retrieval and coordination overhead.

  2. Extensive empirical evaluation. The experiments cover three MAS frameworks, five LLM backbones, and diverse benchmarks. The reported improvements in task performance, token efficiency, and behavioral diversity are promising. The additional ablations on routing strategies and memory ownership also provide useful evidence for the proposed design.

Critical weaknesses
  1. Limited novelty and insufficient justification of the memory design. The distinction between the exploitation and exploration pools appears somewhat artificial. According to Appendix F, exploration essentially corresponds to solving a task without retrieving historical memory, making the framework resemble an adaptive retrieval or context-selection mechanism rather than a fundamentally new memory architecture. It is unclear why two separate memory pools are necessary instead of a single memory repository with an adaptive retrieval policy. More importantly, the practical memory management mechanisms are insufficiently explained: How are memory prototypes constructed and consolidated? How are conflicting, redundant, or outdated memories handled as the repository grows? How exactly do retrieved collaboration trajectories influence agent coordination beyond prompt injection? Similarly, although behavioral diversity is measured, it remains unclear whether the improvement comes from decentralized memory itself or from role-specific prompting and other design choices.

  2. Insufficient baselines and questionable connection between theory and implementation. The evaluation lacks essential single-agent and compute-matched Best-of-N or self-consistency baselines, making it difficult to establish the benefits of multi-agent collaboration over simpler inference strategies. In particular, the reported Qwen3-4B performance on AIME24/25 appears unusually low and warrants clarification of the inference configuration, including thinking mode, generation budget, and evaluation protocol. Comparisons with simpler memory retrieval methods and standard bandit algorithms (e.g., UCB) would also help establish the value of the proposed router. Furthermore, the theoretical regret guarantee relies on strong assumptions about the reward function and routing dynamics, whose validity in the actual implementation is unclear. The stage-wise feedback compares scores from consecutive collaboration stages, which may have different objectives, raising concerns about whether the resulting updates reliably reflect the effectiveness of the selected memory pool. Overall, the theoretical analysis seems insufficiently connected to the practical algorithm.

Questions for authors

See Critical Weaknesses

4 comments

W1

It is unclear why two separate memory pools are necessary instead of a single memory repository with an adaptive retrieval policy. More importantly, the practical memory management mechanisms are insufficiently explained:

We thank the reviewer for this point. The two pools are intended to implement two distinct search policies: the E-pool proposes actions conditioned on historically relevant experience, whereas the X-pool generates fresh, history-independent proposals from the LLM prior. As illustrated by the global-reachability analysis, the former performs a similarity-based local walk, while the latter provides a teleportation mechanism that can reach regions not induced by retrieved experience. With a single adaptive retrieval or context-selection mechanism, even if the system decides to propose a new action after searching the shared repository, that proposal can still be partly influenced by the retrieved relevant information. By explicitly separating the two pools, DecentMem allows the exploration branch to generate genuinely new proposals without conditioning on historical retrieval.

Additionally, G-Memory is an example of a single adaptive retrieval method. We conduct extensive experiments comparing DecentMem with G-Memory, and the results show that DecentMem outperforms G-Memory, further supporting the benefit of explicitly separating exploitation and exploration.

How are memory prototypes constructed and consolidated? /How are conflicting, redundant, or outdated memories handled as the repository grows?

We thank the reviewer for raising these implementation details. Each memory piece is represented as z=(ξ,r⋆)z=(\xi,r^\star), where ξ\xi is the context prototype and r⋆=(rtrajectory,rcomment)r^\star=(r_{\text{trajectory}},r_{\text{comment}}) records the action/cooperation trajectory together with the agent’s rationale. New experiences are first stored in the X-pool and, after the task finishes, consolidated into the E-pool.

As the repository grows, the E-pool is compressed and capped at 100 memory prototypes rather than growing indefinitely. Redundant or overlapping experiences are reduced through this compression. During retrieval, we use Top-K similarity with a threshold. If no sufficiently relevant memory is found, the router falls back to exploration. We will make these implementation details clearer in the revision.

For coordination, the stored trajectory includes task decomposition, cooperation history, and which agent handled each sub-task. When a compatible trajectory is retrieved, its previous allocation can be reused as a coordination prior for the next stage. In the current implementation, this information is passed through the agent context.

it remains unclear whether the improvement comes from decentralized memory itself or from role-specific prompting and other design choices.

We thank the reviewer for this point. G-Memory serves as a centralized-memory baseline under the same role-specific agent setup, which helps control for the effects of role-specific prompting and retrieval. Since DecentMem still outperforms G-Memory in most comparisons, the results suggest that the improvement cannot be explained by role-specific prompting alone and is associated with the decentralized memory design.

AuthorsOpenReview ↗

Continued from the previous comment

W2

The evaluation lacks essential single-agent and compute-matched Best-of-N or self-consistency baselines including thinking mode, generation budget, and evaluation protocol.

We thank the reviewer for raising this point. We additionally evaluate Gemma4-E2B under single-agent and self-consistency settings and compare them with DecentMem using the same backbone. DecentMem consistently outperforms both baselines. We also note that the no-memory DyLAN setting can sometimes perform worse than the single-agent baseline, as dynamic multi-agent coordination without persistent memory or feedback from the LLM-as-a-judge can lead to less effective cooperation.

FrameworkMethodAIME25&24MBPP-PlusALFWorldBBH
DyLANSingle agent21.6754.6257.9332.17
Self-consistency N=925.0058.1363.1738.63
DecentMem28.3366.7969.5343.47

We thank the reviewer for pointing this out. We have added more details on the inference and evaluation settings for reproducibility. All Qwen3 and Gemma4 models are evaluated in non-thinking mode (enable_thinking=False). For generation, we follow the recommended decoding configurations for each model family: for Qwen3, we use temperature (=0.7), top-(p=0.8), and top-(k=20); for Gemma4, we use temperature (=1.0), top-(p=0.95), and top-(k=64).

For the generation budget, we use the same maximum output length of 32798 tokens for each model generation across all compared memory methods under the same backbone and MAS framework. The prompts used for task solving, decomposition, and agent interaction are already provided in the Appendix.

For evaluation, Qwen3 models are run locally through Ollama and Gemma4 models through Hugging Face Transformers. All memory methods and the no-memory baseline use the same backbone, benchmark examples, generation settings, and evaluation procedure within each experimental setting. We evaluate performance using the task-specific success metric, primarily accuracy, and use the same Qwen-32B LLM-as-a-judge across both Qwen- and Gemma-based agents for stage-wise feedback.

Comparisons with simpler memory retrieval methods and standard bandit algorithms (e.g., UCB) would also help establish the value of the proposed router.

The simpler memory retrieval baseline corresponds to the exploitation-only setting in Table 3. We further compare our router with UCB, and the results are shown in the following table. The overall trend remains consistent with the results reported in the draft. In particular, the online router can better adapt to different task preferences between exploitation and exploration, making it more effective across different tasks.

RoutingQwen3-8B BBHQwen3-8B AIMEGemma4-E4B BBHGemma4-E4B AIME
Exploit74.2726.6757.2330.00
UCB66.1330.0055.2338.33
Online76.1936.6759.6943.33

Furthermore, the theoretical regret guarantee relies on strong assumptions about the reward function and routing dynamics, whose validity in the actual implementation is unclear. The stage-wise feedback compares scores from consecutive collaboration stages, which may have different objectives, raising concerns about whether the resulting updates reliably reflect the effectiveness of the selected memory pool. Overall, the theoretical analysis seems insufficiently connected to the practical algorithm.

We thank the reviewer for raising this point. We will clarify that the regret result is a conditional guarantee under the stated assumptions, with Assumption 2 requiring the routing fixed point to coincide with the optimum. Our stochastic-approximation analysis already allows bounded zero-mean fluctuations through the martingale-difference noise term. Regarding stronger systematic or highly non-stationary deviations, we will acknowledge them as a limitation of the current analysis and an important direction for future work.

Thank you for the detailed response and additional experiments. The new single-agent, self-consistency, and UCB comparisons, along with the clarified implementation details, address several of my concerns.

However, I remain somewhat concerned about the novelty of the dual-pool design compared to simpler adaptive retrieval strategies, as well as the gap between the theoretical assumptions and the practical routing algorithm. In particular, the reliability of comparing feedback across different collaboration stages remains unclear.

  1. Novelty of the dual-pool design compared to simpler adaptive retrieval strategies
RoutingQwen3-8B BBHQwen3-8B AIME
Single Adaptive Retrieval73.2931.67
DecentMem76.1936.67

We thank the reviewer for raising this point. Additionally, we conduct a comparison between DecentMem and Single Adaptive Retrieval using Qwen3-8B under the AgentNet framework. In Single Adaptive Retrieval, each agent maintains a single memory pool and first retrieves relevant historical memories. If the retrieval similarity falls below the threshold of 0.8, following the same setting as DecentMem, the agent generates a new memory piece for exploration. As the results show, DecentMem consistently outperforms Single Adaptive Retrieval. Notably, the performance gap is larger on the mathematical reasoning benchmark (AIME) than on BBH. One possible explanation is that mathematical reasoning tasks may require more exploration of alternative solution strategies, whereas Single Adaptive Retrieval can still be influenced by previously retrieved memories through the Transformer's attention mechanism, potentially constraining its ability to explore novel solutions. In contrast, DecentMem explicitly separates exploitation and exploration, allowing the exploration branch to generate new proposals without conditioning on previously retrieved historical memories.

From an engineering perspective, with a single adaptive retrieval or context-selection mechanism, even if an agent decides to generate a new proposal after searching the memory pool, its generation may still be partially influenced by previously retrieved information. This is because the retrieved memory has already entered the model's context and can affect subsequent token generation through the Transformer's attention mechanism.

By explicitly separating the two memory pools, DecentMem implements a mechanism similar to a two-stage decision system. In the first stage, the agent searches the exploitation pool to identify relevant historical experiences. If no useful memory is found, it proceeds to the second stage, where the exploration branch operates with a fresh context that excludes previously retrieved memories, allowing it to generate genuinely new proposals without being influenced by the historical information considered in the first stage. Hence, DecentMem separates not only memory storage and retrieval policies, but also the information available to the model during exploitation and exploration.

  1. The gap between the theoretical assumptions and the practical routing algorithm

To further address the gap between the theoretical assumptions and the practical routing algorithm, we have made two revisions.

First, we explicitly incorporate the actual initialization α0=0.5\alpha_0=0.5 into our theoretical analysis by introducing a localization condition, allowing the convergence analysis to account for the practical starting point.

Second, regarding the assumptions on the reward function, the stage-wise reward is obtained from LLM-as-a-judge feedback, while rE(α)r_E(\alpha) and rX(α)r_X(\alpha) denote the corresponding expected rewards under each routing probability. Since these reward functions are induced by the underlying LLM and task distribution and are not available in closed form, we clarify that the theoretical convergence guarantee holds if the reward functions and the resulting routing dynamics satisfy the stated stability and localization assumptions. As these conditions cannot be guaranteed for arbitrary LLM-induced reward functions, we explicitly acknowledge this as a limitation in the revised version.

The corresponding assumptions and detailed proofs are provided in Appendix A.2 (Assumption 2 and Theorem 4, pp. 15–18) of the latest revised version.

  1. The reliability of comparing feedback across different collaboration stages
FrameworkMethodAIME25&24
AgentNetDecentMem23.33
10%21.67
First stage 10%23.33
Second stage 10%20.00
Third Stage 10%20.00

We thank the reviewer for raising this point. We further conduct a robustness test by introducing 10% incorrect stage-level scores from the LLM-as-a-judge, considering all stages together as well as the first, second, and third stages separately, using Qwen3-4B under the AgentNet framework. As the results show, intermediate reasoning stages play an important role in mathematical reasoning tasks, as incorrect feedback at the middle stage has a greater impact on final accuracy. Overall, the results demonstrate that DecentMem remains relatively robust to incorrect stage-level feedback.

Reviewer_xoZkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DecentMem, a decentralized dual‑pool memory framework for self‑evolving LLM multi‑agent systems, where each agent maintains a private exploitation pool for consolidated past trajectories and an exploration pool for novel candidate strategies, with online weight re‑weighting guided by LLM‑as‑a‑judge stage‑level feedback. Empirical evaluations across multi‑agent frameworks, LLM backbones, and benchmarks show DecentMem improves task accuracy over centralized‑memory baselines.

Critical strengths
  1. This paper is well-structured and well-written.

  2. Experimental results across multiple multi‑agent frameworks and model backbones demonstrate that agent‑private memory preserves behavioral diversity, yields better task accuracy and reduces token overhead compared with state‑of‑the‑art centralized memory baselines.

Critical weaknesses
  1. These theoretical guarantees concerning regret and reachability are built upon idealized mathematical assumptions and have not yet been validated in real-world scenarios where LLM agents operate amidst noise.
Questions for authors
  1. To what extent do the idealized assumptions for the regret bound hold under noisy real‑world LLM‑agent outputs?

  2. How robust is DecentMem’s overall performance to incorrect stage‑level scores from the external LLM‑as‑a‑judge?

  3. What obstacles prevent evaluating DecentMem on high‑stakes benchmarks such as legal or medical reasoning tasks?

  4. How does the memory storage overhead scale with increasing agent numbers and task volumes?

3 comments

W1 and Q1

To what extent do the idealized assumptions for the regret bound hold under noisy real‑world LLM‑agent outputs?

We thank the reviewer for raising this point. Our stochastic-approximation formulation already incorporates a martingale-difference noise term with bounded variance, capturing stochastic fluctuations in the one-step routing updates induced by random pool selection and binary stage-wise feedback. For example, even at the same routing probability αk\alpha_k, the router may stochastically select different pools and observe different binary improvement signals, leading to different one-step updates around the same expected routing direction. Therefore, bounded zero-mean stochasticity is compatible with our analysis, while systematic or highly non-stationary errors may fall outside the stated assumptions. We will clarify this scope and acknowledge robustness to such stronger real-world noise as an important direction for future work.

Q2

incorrect stage‑level scores from the external LLM‑as‑a‑judge

We run a robustness test with different percentages of incorrect stage-level scores from the LLM-as-a-judge using Qwen3-4B. The results show that DecentMem is relatively robust to a low percentage of incorrect scores, while larger fluctuations appear when the proportion of incorrect scores becomes high. Regarding different multi-agent frameworks, the more deterministic frameworks are also more robust to incorrect judge feedback.

FrameworkMethodAIME25&24BBH
AgentNetDecentMem23.3359.43
10%21.6754.43
30%15.0032.44
AutoGenDecentMem21.6763.80
10%21.6760.52
30%16.6755.43

Q3

What obstacles prevent evaluating DecentMem on high‑stakes benchmarks such as legal or medical reasoning tasks?

We further evaluate DecentMem on two medical reasoning benchmarks, MedQA and MedMCQA. MedQA contains multiple-choice questions from medical licensing examinations and focuses on clinical knowledge and reasoning, while MedMCQA covers a wider range of medical subjects through questions from medical entrance and professional examinations.

For each benchmark, we randomly sample 500 examples and use Qwen3-4B as the backbone, following the same multi-agent and memory settings as in the main experiments. The results are shown below. The overall trend remains consistent with the main results, with DecentMem outperforming the centralized-memory and no-memory baselines on both datasets.

FrameworkMethodMedQAMedMCQA
AgentNetNo memory45.450.4
MetaGPT51.456.4
ChatDev49.453.4
G-Memory52.059.0
DecentMem54.662.6
AutoGenNo memory52.456.0
MetaGPT54.667.4
ChatDev56.663.4
G-Memory61.065.4
DecentMem65.673.0

Q4

How does the memory storage overhead scale with increasing agent numbers and task volumes?

Each agent’s E-pool is compressed and capped at 100 memory prototypes, so the storage cost per agent has a fixed upper bound and does not grow indefinitely with task volume. As the number of agents increases, total memory storage therefore grows approximately linearly with the number of agents, i.e., O(M)O(M) for MM agents under a fixed per-agent cap.

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of DecentMem

The author has addressed most of my concerns. I will maintain my rating of 3: Weak acceptance.

Reviewer_EQovOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that the existing mainstream architecture of centralized shared memory is a wrong choice for self-evolving MASs with LLMs due to its inability to preserve the role-complementary specializations of the individual agents.

In this work, the authors propose DecentMem, where each agent holds a private dual-memory structure: the exploitation pool (E-pool) filled with historical trajectories and the exploration pool (X-pool) filled with LLM-generated candidate trajectories. An online router reweights the contribution of both pools to search according to LLM-as-a-judger feedback at each stage of the process, after which the newly-formed pool is consolidated to E-pool.

From a theoretical standpoint, the search performed by each individual agent is modeled as a graph random walk with teleportation. It is claimed that (i) the graph has global reachability and (ii) it guarantees the O(log T) cumulative regret in correspondence with the lower bound for the stochastic bandit. On the empirical side, for three different MAS frameworks (AutoGen, DyLAN, AgentNet), five different LLM backbones (Qwen3-4B/8B/14B, Gemma4-E2B/E4B), and five benchmarks (AIME25 & AIME24, MBPP-Plus, ALFWorld, BBH), DecentMem shows the best performance in 14 out of 15 cells, reaching up to +23.8% improvement in comparison to the most effective centralized baseline (G-Memory) and -49% in terms of token budget usage with growing margins of stochasticity of coordination.

Critical strengths
  1. First strength is the proposition that “gains grow with stochasticity” is explicitly separated from just being a claim. In Table 5, it is shown that both centralized-with-separated-routing and decentralized-with-shared-routing are inferior to DecentMem, highlighting the role of agent private memories over agent specific routing. This shifts the main point from “our system wins” into an explicit decomposition of mechanisms, which is one of the major points supporting the acceptability.

  2. Second is the explicit support for the claim regarding homogenization is shown by direct evidence from the mechanics. Behavioral diversity (see Table 4) reveals that DecentMem increases the behavior diversity between agents approximately twofold compared to centralizing schemes, especially for the more dynamic environment.

Critical weaknesses
  1. I think the claim set of the theory extends beyond the demonstrable scope. The manuscript focuses on “order-optimal O(log T) regret, matching the bandit lower bound” as one of its key contributions. However, the proof only shows that the weight of the router α converges to a fixed point at the usual speed. That the fixed point in question is the optimal α is presumed (Assumption 2), and the experimental setup does not fit the standard bandit framework; thus, the quoted lower bound is not really a matching one. The theorem on global reachability holds true, but has little power—in essence, any exploration term with full support would do. However, the theorem does give a proper convergence statement; it is just wrongly presented as an optimal one. This exaggeration is important because theoretical results comprise one of three announced contributions.

  2. The major tables do not have error bars. Tables 1–2 contain single numbers, with no standard deviations or significance statements, even though the Appendix suggests that seed variance could be rather high (±3.3 on AIME). Some of the smaller per-benchmark “wins” could be due to variance, and most of the average gains come from BBH, while centralized memory beats ALFWorld sometimes. But, overall, the trend seems to hold, and the bolded best/second-best claims a certain accuracy which is currently lacking empirical grounds.

Questions for authors
  1. Does the Qwen-32B judge help only DecentMem’s router, or do the baselines get the same feedback? If only DecentMem, how much of the gain comes from the judge rather than from decentralization?

  2. Can you show the router actually converges to the best α, instead of assuming it? If not, would you restate the theorem as convergence to the router’s fixed point rather than regret-optimality?

  3. The E-pool keeps growing with no eviction (Eq. 8). Do retrieval cost and the 49% token savings hold up well past 200 tasks? Could you add mean±std and a simple paired test for at least the headline cells of Tables 1–2?

2 comments

W1 and Q2:

Can you show the router actually converges to the best α, instead of assuming it? If not, would you restate the theorem as convergence to the router’s fixed point rather than regret-optimality?

We thank the reviewer for this clarification. The theorem is intended to show that αt\alpha_t converges to α⋆\alpha^\star at the stated rate, with Assumption 2 ensuring that this fixed point coincides with the optimal routing probability. We agree that our broader routing setting does not directly fit the standard stochastic-bandit framework, so the previous wording of “matching the lower bound” and “order-optimal” was too strong. We will therefore revise the claim to state that, under the given conditions (Assumption 1 and 2), our method converges to the optimal routing α⋆\alpha^\star and achieves logarithmic regret, comparable to the rates obtained in classical stochastic-bandit settings. Under these conditions, this implies that the router progressively approaches the optimal routing probability α⋆\alpha^\star while the average regret per routing decision vanishes as T increases.

Q1

Does the Qwen-32B judge help only DecentMem’s router

We thank the reviewer for raising this point. No. All methods receive feedback from the same Qwen-32B judge, and the judge setting is kept consistent across all methods. Therefore, the performance gains of DecentMem are not due to privileged access to the judge feedback.

W2 and Q3

The E-pool keeps growing with no eviction (Eq. 8). Do retrieval cost and the 49% token savings hold up well past 200 tasks?

We thank the reviewer for raising this point. Although Eq. 8 describes consolidation into the E-pool, in our implementation the E-pool is compressed and capped at 100 memory prototypes rather than growing indefinitely. We applied the same memory-capacity constraint to the centralized baselines to ensure a fair comparison of retrieval and token costs. Therefore, the reported token savings are measured under the same bounded-memory setting across methods. We will clarify this implementation detail in the revision.

Could you add mean±std and a simple paired test for at least the headline cells of Tables 1–2?

Additionally, we run multiple seeds and conduct a simple paired test between DecentMem and the strongest centralized baseline, G-Memory, using Qwen3-4B under the AgentNet framework. The results show p<0.05p<0.05, indicating that DecentMem significantly outperforms G-Memory.

MethodAIME25&24MBPP-PlusALFWorldBBH
G-Memory10.00 ± 3.3363.18 ± 1.1452.14 ± 0.6148.14 ± 0.96
DecentMem23.33 ± 1.6767.53 ± 0.9064.45 ± 0.7559.43 ± 0.72
Paired pp< 0.01< 0.01< 0.001< 0.001

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of DecentMem

Reviewer_F5ySOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DecentMem, a memory framework for LLM-based multi-agent systems in which each agent maintains a private exploitation pool of historical collaboration experiences and a temporary exploration pool of newly generated experiences. At each stage, a probabilistic router selects historical retrieval or generation without historical retrieval, with fallback to exploration when retrieval similarity is insufficient. An LLM judge provides stage-level feedback to update routing weights, and exploration memories are subsequently consolidated into the exploitation pool. The paper presents conditional reachability and logarithmic routing-regret analyses. Experiments across three multi-agent frameworks, five LLM backbones, and mathematical reasoning, code generation, BBH, and embodied tasks report accuracy improvements and token savings over the tested memory baselines.

Critical strengths

1: The exploration–exploitation framing is intuitive and useful. Solving a problem often requires both drawing on past experience and trying new approaches. The dual-pool design provides a concrete way to combine these behaviors and adjust their use as experience accumulates.

2: The paper raises an interesting question about memory ownership in self-improving multi-agent systems. Whether agents should share experiences or maintain private memories is a meaningful design choice that could affect specialization and collaboration over time. I find this perspective valuable, although the experiments do not yet provide sufficient evidence for the proposed explanation of why decentralized memory helps.

Critical weaknesses

1: The paper does not establish that shared memory causes agents to lose their specialization. Table 4 shows that DecentMem produces more different outputs, but different outputs do not necessarily mean complementary expertise. Table 5 shows a performance benefit from private memory, but does not explain whether this comes from preserving specialization or simply retrieving more relevant information. A shared memory system could also give each agent different, role-specific experiences.

2: The paper does not show that exploration produces useful new knowledge. Appendix F describes ordinary LLM prompting without historical retrieval. Theorem 1 assumes that the LLM can generate every feasible strategy with nonzero probability; it does not show that the proposed mechanism discovers useful strategies within a realistic budget. The experiments do not track whether exploration-generated memories improve later tasks. Moreover, Eq. (8) stores all exploration memories without specifying a check for correctness or usefulness.

Questions for authors

1: Does the centralized baseline in Table 5 use role-specific memory retrieval? What evidence shows that DecentMem’s gains come from preserving complementary expertise, rather than simply retrieving more relevant memories? In particular, do agents using shared memory actually become less specialized as experience accumulates?

2: What evidence shows that X-pool generates useful new strategies and that reusing them improves later tasks? Are these memories checked for correctness or usefulness before being added to the E-pool in Eq. (8)? If all generated memories are retained, how does the system avoid accumulating and reusing incorrect experiences?

4 comments

W1 and Q1

Does the centralized baseline in Table 5 use role-specific memory retrieval?

We thank the reviewer for raising these points. Regarding role-specific retrieval, this is already captured by G-Memory, one of our strongest centralized baselines, which retrieves relevant memories for individual agents. Overall, DecentMem still outperforms G-Memory in most comparisons.

What evidence shows that DecentMem’s gains come from preserving complementary expertise, rather than simply retrieving more relevant memories?

In our design, each memory prototype also contains role-related information, including the functional role or persona of the agent involved in each sub-task. Therefore, retrieving relevant memories is also closely related to retrieving agent-specific experiences.

In particular, do agents using shared memory actually become less specialized as experience accumulates?

Regarding whether shared memory makes agents less specialized, Appendix Table 4 shows that centralized-memory methods have substantially lower behavioral diversity. This suggests that agents using shared memory tend to produce more similar actions, whereas DecentMem maintains more diverse behaviors, consistent with stronger agent specialization on the same task.

W2 and Q2

What evidence shows that X-pool generates useful new strategies and that reusing them improves later tasks?

We thank the reviewer for raising this point. Table 3 already compares Exploitation Only with our full method, where the higher accuracy of DecentMem suggests that the X-pool contributes useful strategies. We further conduct an experiment in which the online router is used during the first half of the tasks, followed by exploitation only in the second half. The higher accuracy in this setting shows that strategies generated through the X-pool remain useful for later tasks.

Qwen3-8B

Routing StrategyAgentNet BBHAgentNet AIMEAutoGen BBHAutoGen AIME
Exploitation Only (α=1\alpha=1)74.2726.6784.2928.33
Online → Exploit (first 50% online, last 50% Exploitation Only)75.2330.0084.5631.67

Gemma4-E4B

Routing StrategyAgentNet BBHAgentNet AIMEAutoGen BBHAutoGen AIME
Exploitation Only (α=1\alpha=1)57.2330.0065.3733.33
Online → Exploit (first 50% online, last 50% Exploitation Only)57.1436.6766.8636.67

Are these memories checked for correctness or usefulness before being added to the E-pool in Eq. (8)? If all generated memories are retained, how does the system avoid accumulating and reusing incorrect experiences?

Regarding the accumulation and reuse of incorrect experiences, the LLM-as-a-judge evaluates stage-wise outcomes based on correctness and solution quality, and uses this feedback to update the online routing weights. if a newly generated memory from the exploration pool receives negative feedback from the LLM-as-a-judge, the agent discards this memory rather than consolidating it for future reuse.

When exploitation is selected, only the Top-K most relevant memories above a similarity threshold are retrieved; otherwise, the system falls back to exploration. This reduces the reuse of irrelevant or mismatched experiences and also limits their negative effect even when they are retrieved.

We further run a robustness test with different percentages of incorrect stage-level scores from the LLM-as-a-judge using Qwen3- 4B. The results show that DecentMem is relatively robust to a low percentage of incorrect scores

FrameworkMethodAIME25&24BBH
AgentNetDecentMem23.3359.43
10%21.6754.43
AutoGenDecentMem21.6763.80
10%21.6760.52

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of DecentMem

Thank you authors for the additional experiments and for addressing my questions. I found the experiment that switches from online routing to exploitation halfway through the task sequence helpful. The results strengthen the empirical case for DecentMem, and I see the decentralized memory design as a worthwhile contribution. I am raising my rating from weak rejection to weak acceptance.

We sincerely thank the reviewer for their constructive feedback and for acknowledging our rebuttal and clarifications.

Kind regards,

The authors of DecentMem

Reviewer_puaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a memory framework, named DecentMem, in which each agent maintains a private exploitation pool containing past collaboration experience and a temporary exploration pool containing newly generated experience. An agent-specific router chooses between retrieval and exploration, updates its routing weight using improvements in stage-level LLM-judge scores, and consolidates exploratory experience after each task. The paper also presents reachability and logarithmic-regret analyses.

Critical strengths
  1. The idea proposed in this paper is interesting and reasonable: to avoid agent homogenization and improve overall performance of multi-agent systems by decentralizing the agents' memories.

  2. Based on the experimental results, this method appears to outperform multi-agent systems with centralized memory.

Critical weaknesses
  1. Theorem 2 states that Assumption 1 alone is sufficient for logarithmic cumulative regret. However, the corresponding result in Appendix A.2, Theorem 4, additionally requires Assumption 2. This additional assumption should be explicitly included in the theorem statement rather than introduced only in the proof, as the current presentation may mislead readers into believing that Assumption 1 alone is sufficient for the theorem to hold. Moreover, there is a substantive discrepancy: Assumption 2 requires the router’s limiting mean dynamics to have an equilibrium at the reward maximizer and to be locally contractive around it. These properties do not follow from the strict concavity of the reward function, and the paper does not establish that the proposed update rule satisfies them. Besides, local stability does not by itself justify convergence from the actual initialization or the unconditional mean-square error bound used in the proof. The main theorem should explicitly state all required conditions, and the paper should either establish their applicability to the proposed router or narrow the claim to an appropriately qualified conditional result.
  2. The paper invokes a classical stochastic multi-armed bandit lower bound to argue that its logarithmic regret is order-optimal. However, the connection between that setting and the paper’s routing problem is not established. In a standard two-armed stochastic bandit, each arm has a fixed expected reward, so the expected reward of selecting the exploitation arm with probability α\alpha is linear in α\alpha. Here, both rE(α)r_E(\alpha) and rX(α)r_X(\alpha) may depend on the routing probability, and their mixture is assumed to be strictly concave with an interior optimum. The classical lower bound therefore does not automatically apply to this problem class. Even if the logarithmic upper bound is established, the optimality claim requires an applicable lower bound or a valid reduction that preserves the feedback model and regret comparator.
Questions for authors
  1. Theorem 2 invokes only Assumption 1, whereas Theorem 4 in Appendix A.2 additionally requires Assumption 2. What is the intended scope of the guarantee? Can the authors establish that the proposed routing update satisfies the required alignment between its equilibrium and the reward maximizer, and explain why the stated local stability condition suffices for convergence from the actual initialization and the mean-square error bound in Equation (22)?

  2. Which stochastic-bandit lower bound applies to the proposed routing problem, where the expected pool rewards depend on the routing probability and the optimal mixture is interior? Can the authors provide an applicable lower-bound argument or a reduction that preserves the feedback model and regret comparator? Without such a connection, should the result be stated as a conditional logarithmic upper bound rather than an order-optimal guarantee?

5 comments

W1 and Q1

Theorem 2 invokes only Assumption 1, whereas Theorem 4 in Appendix A.2 additionally requires Assumption 2

We thank the reviewer for pointing this out. We agree that Assumption 2 should be stated explicitly in both Theorems 2 and 4. Assumption 1 gives a unique reward maximizer α⋆\alpha^\star, while Assumption 2 additionally requires the mean routing dynamics to have an equilibrium at α⋆\alpha^\star, be locally stable around it, and satisfy the stated localization condition from the actual initialization.

routing updates alignment reward function/the mean-square error bound in Equation (22)

We thank the reviewer for pointing out these gaps. We have revised the theoretical analysis accordingly. First, we no longer treat the equilibrium alignment and local contraction of the routing dynamics as consequences of the strict concavity of the reward. Instead, we state them explicitly in Assumption 2 through h(α⋆)=0h(\alpha^\star)=0 and the local contraction condition h(α)(α−α⋆)≤−c(α−α⋆)2h(\alpha)(\alpha-\alpha^\star)\le -c(\alpha-\alpha^\star)^2.

Second, to address convergence from the actual initialization α0=0.5\alpha_0=0.5, we introduce an explicit localization condition, Pr⁡(αk∉U)≤CU/(k+1)\Pr(\alpha_k\notin U)\le C_U/(k+1). We then extend the proof to control the routing process both inside and outside the locally stable neighborhood, including the projection case. This directly addresses the reviewer’s concern about the unconditional mean-square error bound. Specifically, we derive Vk+1≤(1−2cηk)Vk+Cηk2,V_{k+1}\le (1-2c\eta_k)V_k+C\eta_k^2, where Vk=E[(αk−α⋆)2]V_k=\mathbb{E}[(\alpha_k-\alpha^\star)^2], and prove E[(αk−α⋆)2]=O(1/k).\mathbb{E}[(\alpha_k-\alpha^\star)^2]=O(1/k). This then yields O(log⁡T)O(\log T) cumulative regret.

The corresponding assumptions and detailed proof are provided in Appendix A.2 (Assumption 2 and Theorem 4, pp. 15–18) of the latest version, with the mean-square recursion and O(1/k)O(1/k) convergence proof given on pp. 17–18.

Finally, the main theorem will explicitly states all required assumptions and the diminishing update schedule. In our setting, the stage-wise reward is obtained from LLM-as-a-judge feedback, while rE(α)r_E(\alpha) and rX(α)r_X(\alpha) denote the corresponding expected rewards under each routing probability. Since these reward functions are induced by the underlying LLM and task distribution and are not available in closed form, we will narrow the theoretical result in the revised version to a conditional guarantee under the stated stability and localization assumptions.

W2 and Q2

stochastic-bandit lower bound cannot applied to the proposed routing problem

We thank the reviewer for highlighting this distinction. We acknowledge that the connection to the classical stochastic-bandit lower bound requires more careful qualification in our broader routing setting, where (rE(α))(r_E(\alpha)) and (rX(α))(r_X(\alpha)) may depend on α\alpha and the optimum can be interior. We will therefore revise the claim to state that, under the given conditions, our method achieves a logarithmic regret rate comparable to those of classical stochastic-bandit methods. Under these conditions, this implies that the router progressively approaches the optimal routing probability (α⋆)(\alpha^\star) while the average regret per routing decision vanishes as T increases.

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of DecentMem

Since it is the last day of the rebuttal period, we would like to kindly ask whether you have any remaining concerns regarding our responses. Many thanks again for your time and valuable feedback.

Kind regards,

The authors of DecentMem

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.