acceptodds
Under review as a conference paper at ICLR 2027

Wired Together: Learning Persistent Relationships through Reward-Modulated Social Plasticity

16 views

Abstract

Coordination in multi-agent systems is typically specified per task, by a fixed topology, a dialogue protocol, or a planner that assigns work. We study a second axis of design: with whom agents have a relationship. We model multi-agent systems as an adaptive social network whose directed bonds evolve online through reward-modulated social plasticity: social co-activity marks a pair as eligible, and later outcomes consolidate or weaken the bond. The learned graph feeds back into learning, through reward diffusion and bond-gated experience sharing, and into inference, through social deliberation. In WIRE, an embodied environment of five chambers with increasing coordination demands, relational state helps most when it shapes learning. With RL fine-tuning, relational coupling improves cooperative completion in three of four tested configurations, while at inference time social plasticity achieves similar cooperative completion to a centralised orchestrator but higher task return, suggesting benefits on more challenging cooperative behaviour. Different social interaction regimes shape relationships differently: communication builds the strongest bonds, whereas observation or imitation leads to the highest cooperative completion. When studying long-term cooperation, we find that retaining both episodic memory and bonds across different phases of learning favours exploitation of previously successful partners, whereas retaining only bonds balances exploitation and exploration. In this way, the bond graph acts as a compact distillation of social experience while remaining plastic enough to rewire toward new collaborators. Learned relationships thus offer a decentralised complement to orchestration and provide an explicit, inspectable record of which inter-agent relationships have been reinforced through interaction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Multi-Agent Reinforcement Learning, Agent Coordination, Decentralized Learning, Social Learning, Hebbian Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Y1MaOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a persistent directed bond graph for multi-agent coordination. Bonds are updated online through reward-modulated social plasticity and are used during RL fine-tuning for reward diffusion and experience sharing, or during inference for social deliberation. The method is evaluated in WIRE, a five-stage embodied curriculum, with experiments on performance, social cues, and partner transfer.

Critical strengths

The paper explores an interesting and useful design direction: maintaining an explicit, inspectable relationship state across episodes rather than rebuilding coordination from scratch for each task.

Critical weaknesses
  • The learned bond does not clearly distinguish interaction frequency from useful cooperation. Equation 7 allows repeated co-firing to strengthen a bond without reward, while the eligibility trace credits recent co-actors based on correlation rather than counterfactual contribution. Consistently, Table 2 and Figure 8 show that communication produces the strongest bonds but relatively poor cooperative completion. It is therefore unclear whether WW measures valuable relationships or mainly repeated interaction.

  • The source and reliability of the reported gains remain unclear. Reward diffusion, experience sharing, and social deliberation are not separately ablated; the main results lack significance tests and clear per-condition seed counts; and the evaluation uses only a self-built curriculum that enforces co-location. The reversed Gemma-E4B MAPPO result in Table 1 further suggests that the mechanism is not consistently beneficial.

Questions for authors
  • Can the authors separate the effects of interaction frequency and successful contribution, for example by ablating the association floor and adding a counterfactual contribution test?

  • What are the separate contributions of reward diffusion, experience sharing, and social deliberation? Why does the Gemma-E4B MAPPO setting degrade?

  • How many independent seeds were used, are the main differences statistically reliable, and does the method still work when co-location is not enforced?

Reviewer_bQx9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a persistent relationship graph for embodied multi-agent collaboration. Relationships are updated through interactions and rewards, influencing social decision-making at inference time and supporting reward diffusion and experience sharing during reinforcement learning. Experiments in WIRE show that relational state can improve collaboration in some learning settings. A key contribution is treating relationships between agents as a form of system state that can accumulate and be reused.

Critical strengths
  1. The explicit relationship graph compresses interaction experience into relational state that can be retained and reused, while making connections between agents easier to understand.
  2. Sequential chambers connect early interactions with subsequent collaboration, providing a concrete setting for evaluating the role of relational state.
  3. The paper reports both improvements and failures, helping clarify the scope and limitations of the relational mechanism.
Critical weaknesses
  1. The sources of performance gains need to be separated more clearly. The RL setting introduces both relationship-weighted reward diffusion and experience sharing. The main comparisons do not sufficiently distinguish benefits from learned relational structure from those due to reward mixing or additional shared samples.
  2. The empirical advantage is not consistent. Collaborative performance declines under some models and learning settings, while cooperative milestone completion is broadly comparable to centralized orchestration. Per-agent cooperative milestone completion is low, and evaluation is limited to a single custom environment, leaving limited evidence for reliable performance on difficult cooperative tasks.
Questions for authors
  1. With matched learning and interaction budgets, controlled reward diffusion, and equal numbers of shared samples, how much additional benefit does the learned relational structure provide over uniform weights or simple partner preferences?
  2. Do the main advantages persist when only task rewards are counted, more independent runs are conducted, and evaluation extends to another environment?
Reviewer_fJPwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose reward-modulated social plasticity for learning persistent inter-agent relationships. The proposed system maintains a weighted, directed social graph whose bonds evolve through co-firing signals and temporal eligibility traces. An association floor supports bond formation through repeated co-firing, while reward-gated consolidation, failure-gated decay, and homeostatic decay determine subsequent changes. The graph persists across episodes and conditions social deliberation through advisory directives. During RL fine-tuning, it also supports reward diffusion and bond-gated experience sharing. The agents use the MindForge cognitive stack, with per-agent LoRA policies updated online through IPPO or MAPPO while the base model remains frozen.

The authors introduce WIRE, a five-chamber embodied environment built on Craftium, and organize the evaluation around three research questions. RQ1 compares social plasticity with no-graph baselines and, for zero-shot Gemma-E4B, a centralised orchestration baseline matched on estimated inference FLOPs. The social-plasticity condition reports higher mean cooperative completion in three of four RL fine-tuning configurations, with a decrease for Gemma-E4B MAPPO. Zero-shot Gemma-E4B achieves similar cooperative completion to centralised orchestration and a higher mean task return, while both Qwen backbones show lower cooperative completion than their no-graph baselines. Cooperative completion measures agent-level milestone credits, and task return in this comparison includes communication rewards.

RQ2 examines how social interaction regimes shape learned bonds. Communication produces the strongest bonds, while Spatial engagement achieves the highest cooperative completion among the tested regimes. RQ3 examines how memory and learned relational state carry social experience into long-term cooperation. The authors bring three previously successful pairs into a six-agent team and compare full transfer, memory only, bonds only, and fresh start, with memory and bonds continuing to update during Phase B. Bonds only achieves the highest mean task return, while full transfer produces the strongest partner preference, measured by the share of messages addressed to the former Phase-A partner.

Critical strengths
  • The proposed social graph provides an explicit representation of persistent relationships, and Appendix G demonstrates how this representation can be examined alongside agent behaviour. Figure 10 shows bond consolidation associated with milestone rewards and the persistence of pair ordering across episodes, while Figures 11–15 relate bond trajectories and orchestrator assignments to messages, joint milestones, and combat. The examples include successful coordination and failures in translating communicated intent into physical execution. This makes the graph useful for examining how relationships develop and where coordination breaks down, and gives concrete support to the paper’s claim that relational state is inspectable.

  • RQ2 provides a substantive finding about the relationship between bond strength and cooperative performance. The authors use the same reward schedule for communication, observation, and imitation, and report task return excluding the rewards associated with these social acts.Communication produces stronger bonds and lower cooperative completion than Spatial engagement, while the combined-cue conditions show communication dominating the single social-action slot. These results identify the choice of social interaction regime as an important design decision and show why stronger bonds alone provide an incomplete assessment of useful cooperation.

Critical weaknesses
  • The proposed social-plasticity rule leaves uncertainty about how reliably learned bonds support useful cooperation. The authors acknowledge this distinction in RQ2 and Table 10, where stronger bonds are associated more closely with communication than with physical cooperation, while Appendix G illustrates both successful joint execution and failures to translate coordinated intent into progress. Co-firing and subsequent rewards can strengthen a bond even when the interaction contributed little to the outcome, and that bond can then influence help requests, reward diffusion, and experience sharing. This creates a risk of reinforcing relationships with limited coordination value, which may constrain the method’s applicability when frequent interaction is a poor indicator of useful cooperation.
Questions for authors
  • In RQ1, social plasticity improves Gemma-E4B under IPPO but reduces cooperative completion under MAPPO. Beyond the critic-target mismatch acknowledged in Appendix E.1, did you identify observations from these runs that help explain this difference? This would help clarify the conditions under which coupling relational state into learning is beneficial.
1 comment

Thank you for the detailed response and additional experiments. Your replies and evaluations to my questions and the other reviewers have helped clarify my concerns. The corrected per-agent adapter address my question on the MAPPO results. My remaining concern for relative bond strength would still correlate to interaction frequency, and they determine whose rewards and experiences each agent learns from e.g. in the embodied settings that the paper motivates on, where a unhelpful agent that communicates frequently could therefore gain influence that then diverges on communication and physical contribution. However, learning relationships between agents using their framework is an interesting direction that future research can build on, and I will therefore maintain my original evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to reviewer bQx9 - Part 4

  1. Critical Weakness 2c: "Per-agent cooperative milestone completion is low [...]"

Cooperative milestones are the hardest part of WIRE by design: each one needs agents to act together, such as two agents breaking an anvil at the same time, cooperating in combat, or pressing the switch to open a teammate’s door. Embodied coordination of this kind remains difficult for open-weight LLM agents [3, 4], so absolute rates are low in every condition, including both centralised orchestrators (3.9–4.7% per agent).

  1. Critical Weakness 2d: "[...] evaluation is limited to a single custom environment" / "[...] and evaluation extends to another environment?"

The rationale for introducing WIRE and testing on it was to study relationships that persist and are reused: WIRE is a single sequential curriculum whose chambers build on one another, so bonds formed in one chamber carry into the next, where cooperation becomes more demanding, whereas existing cooperative benchmarks evaluate collaboration in separate, self-contained scenarios (Appendix A).

To extend the evaluation beyond WIRE, we are working on adding AgentWorld [4], a third-party benchmark on a persistent 2D MMO world with specific agent roles (for example, a miner supplies ore that a smith turns into an axe) and a vast open-ended setting, where agents spread over a large map and interact only by messaging and handing over items. This new environment lets us test social plasticity in a setting where agents have heterogeneous roles; the effects of collaboration could be shown faster, as agents’ actions are higher-level actions instead of discrete ones, and for future extension, the number of agents could scale even more than WIRE. However, as released, AgentWorld is not long-horizon across evolving tasks: each task is scored on its own and the team is reset per task, so bonds learned in one task are discarded before they can matter, and all messages go through a global chat, so no message has a recipient that a bond could inform. We therefore need to adapt it for continual learning: teams play chained tasks while keeping their identities, bonds and memory, and agents message one teammate at a time.

We are working on characterising, quantitatively and qualitatively, the contribution of social plasticity in AgentWorld; below are the preliminary results (Appendix I, Table 16 in the manuscript). Agents play a chain of four tasks. The success rate is the percentage of the four tasks completed in a run, as judged by the benchmark’s own verifiers (mean ± SD across runs).

ConditionSuccess rate %
Gemma-E4B16.7 ± 14.4
Gemma-E4B + Social plasticity25.0 ± 0.0

With social plasticity, agents complete more of the chained tasks (25.0% vs. 16.7%), and every run completes a task, suggesting that bonds carried across tasks support coordination beyond WIRE.

  1. Question 2: "Do the main advantages persist when only task rewards are counted, more independent runs are conducted [...]?"

Each seed is 3 episodes of up to 1,000 steps. The zero-shot rows of Table 1 use 6 seeds each, the scaling experiments 3 seeds per team-size × budget cell (90 runs in total), and the RL runs 3 seeds, and the new RL runs 2 seeds so far, with more running. The scaling gains hold up: +2.0 points pooled over cells (p = .0015), higher or tied in all 15 cells (sign test p = .002), and +2.9 points at 600 tokens (Holm p = .033).

Task rewards only. In Table 1, task return includes communication rewards. Solo and cooperative completion count milestones and never included message pay, and the scaling runs have no message pay at all. We also recomputed task return without message pay for every pair of rows in Table 1 of the manuscript. Social plasticity’s difference from the baseline changes by at most 14 points in any row: for example, +27 with and +27 without message pay for zero-shot Gemma-E4B. The task-return differences therefore do not come from the communication reward.

References

[3] Lică, M. T., Shirekar, O., Colle, B., and Raman, C. MindForge: Empowering embodied agents with theory of mind for lifelong cultural learning. NeurIPS, 2025.

[4] Shu, R., Zhang, Y., Cho, Y. M., Yang, J. M., Yuan, Y., Zheng, W., Guntuku, S. C., Ungar, L., Yu, Z., and Zhang, R. AgentWorld: Benchmarking long-horizon collaboration of multi-agent LLMs. arXiv:2609.31590, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.