acceptodds
Under review as a conference paper at ICLR 2027

BISKY: Dynamic Bipartite Coordination for Open Multi-Agent Energy Systems

Abstract

Multi-agent reinforcement learning is used to coordinate distributed energy resources that are coupled through the power grid and energy market. During deployment, the set of active agents may change, and agents that remain active may also change their operating patterns. Centralized approaches provide coordination but often require global information and a fixed population. Independent learners accommodate population changes but cannot use knowledge learned by other agents to improve coordination. In this paper, we propose BISKY, a decentralized framework that shares knowledge through a dynamic bipartite graph. The two graph partitions represent active agents and coordination channels for recurring operating behaviors. Each agent forms a compact routing signature from its recent local trajectory and connects to the channel that matches its current behavior. Channels combine compact updates from their connected agents and return the resulting knowledge for local use without requiring global observations or a centralized critic. By storing coordination knowledge in behavioral channels rather than tying it to agent identities, BISKY preserves and reuses this knowledge as the active agent set and individual operating behaviors change. We evaluate BISKY in peer-to-peer energy trading and electric-vehicle charging with fixed and changing active agent sets. In the closed setting, BISKY reaches about 95% of the cost savings, 88% of the grid reduction, and 88% of the electric-vehicle charging reward achieved by MAPPO, whose centralized critic uses joint system information. Results show that behavior-based channels improve coordination without requiring global information or a fixed agent population.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multi-agent reinforcement learning, decentralized coordination, peer-to-peer energy trading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t74rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BISKY addresses multi-agent energy coordination when agents enter, leave, or change their behavior. Each agent trains its own PPO actor and critic while keeping raw trajectories local. A dynamic bipartite graph connects agents to behavior channels, which aggregate either belief-model adapter updates or policy gradients. Channels let newcomers reuse knowledge from agents with similar operating patterns. The paper evaluates peer-to-peer energy trading and EV charging, reporting stronger open-population performance than the evaluated baselines and ablations suggesting that dynamic routing and retained knowledge matter most under population changes.

Critical strengths
  1. Knowledge stored outside an agent's private model can remain useful when membership changes, while agents keep control of their own policies. This is a coherent approach to reconciling local learning with experience sharing. The belief variant's validation on separate complete episodes also provides a practical safeguard against harmful transfers. This combination makes the framework potentially useful beyond a fixed team.
  2. The evaluation contains meaningful evidence of coordination under changing participation. The two environments couple agents through different mechanisms: market interactions and electrical constraints. Population schedules are paired across methods, and the EV experiments disable centralized action projection, so an external optimizer does not automatically repair infeasible joint actions.
Critical weaknesses
  1. The paper does not establish when locally learned knowledge is compatible across agents. Behavioral similarity determines whom to share with, but does not by itself make model updates transferable. In the belief variant, adapters share an initialization while their underlying predictors remain local; the description does not establish whether those predictors remain aligned. The policy variant similarly merges gradients from local policies without specifying a common evaluation point. I cannot determine whether channels combine compatible updates or rely on local validation to reject mismatched ones. A precise account of parameter alignment and synchronization could resolve this concern, but it is needed to assess the proposed sharing mechanism.
  2. The results do not clearly connect the proposed mechanism to the reported advantage. Table 1 reports one BISKY entry without identifying which of its two sharing variants was evaluated. Table 2 provides variant-specific ablations, but leaves the environment and return normalization unspecified. The reader therefore cannot connect these promising component effects to the main cross-environment comparison. This matters because the gain over the strongest open baseline in energy trading is modest, making the ablation evidence particularly important.
Questions for authors
  1. What parameters are shared, frozen, or synchronized before merging? Please define the reference for the adapter updates in Equation (10) and the policy parameters at which the gradients in Equation (11) are computed.
  2. Which sharing variant produced each Table 1 result? Which environment and normalization formula produced Table 2, and how do its runs relate to the main comparison?
2 comments

We thank the reviewer for the careful review and constructive feedback. We agree that behavioral similarity alone does not guarantee the compatibility of model updates, and we have clarified the parameter alignment and knowledge-sharing procedures in Section 3.

Belief variant (Eq. 10): Shared parameters and reference point.

In the belief-sharing variant, only the lightweight adapters are shared. Each agent's main forecasting model is trained on its own local data and is never synchronized with other agents. Thus, we do not assume that the underlying forecasting models are aligned.

All adapters share the same initialization, θinit\theta_{\mathrm{init}}, and each agent computes its update relative to this reference:

Δθi=θi−θinit.\Delta\theta_i = \theta_i - \theta_{\mathrm{init}}.

The channel merges these updates, and each agent evaluates the merged update on its own held-out episodes before accepting it. Specifically, the merged update is applied to the agent's current adapter according to Eq. (10), and the agent retains it only if its local forecasting error does not increase; otherwise, αi=0\alpha_i=0.

The shared initialization provides a common parameter reference, but does not guarantee transferability across different local predictors. The local validation step addresses this issue by rejecting merged updates that worsen the receiving agent's forecasting performance. Table 2 supports its importance: removing validation reduces the normalized return to 60.1 in the open setting, below the no-sharing baseline of 68.3, compared with 89.5 for the full belief-sharing variant.

Policy variant (Eq. 11): Gradient evaluation and synchronization.

In the policy-sharing variant, all agents assigned to a channel use the same channel policy parameters θc\theta_c, including the actor and critic, to select actions based on their own local observations.

Each member computes its PPO gradient at the common channel parameters θc\theta_c, using its own locally collected trajectories. The channel merges these gradients according to Eq. (11) and updates θc\theta_c according to Eq. (12). Thus, all gradients within a channel are evaluated at the same parameter point, rather than being directly merged from independently parameterized policies. Raw trajectories are never pooled.

When an agent joins or switches channels, it adopts the policy parameters θc\theta_c of its newly assigned channel. Unlike a single globally shared policy, BISKY maintains separate channel policies for agents with similar operating behaviors, allowing knowledge to remain available as channel membership changes.

We have also corrected an inaccurate statement in Section 3. In the policy-sharing variant, agents select actions using their assigned channel policy; the previous description incorrectly stated that the channel policy does not select actions.

Clarification of Tables 1 and 2.

We have revised the captions of Tables 1 and 2 and Appendix C.2 to clarify the sharing variants, evaluation environments, and normalization.

In Table 1, P2P energy trading uses policy merging, with additional belief-adapter merging in the closed setting. EV charging uses belief-adapter merging.

Table 2 reports ablation results from P2P energy trading only, covering both belief-sharing and policy-sharing variants. These experiments use five independent random seeds from runs separate from those reported in Table 1.

The normalized scores use a random policy as the zero reference and SharedPPO, a shared actor-critic trained on pooled agent data, as the 100 reference, except for the open belief-sharing setting, which uses a different reference policy as specified in Appendix C.2. Scores are calculated from each method's return at the end of training and averaged across seeds. SharedPPO is included only as a reference because pooling raw observations violates our locality requirement.

Regarding the relatively small performance difference from POAM in open P2P trading, we have revised the Introduction and Section 4 to describe BISKY as achieving comparable performance.

Summary of update compatibility. Behavioral similarity determines which agents share knowledge, but it does not itself guarantee that their updates are compatible. In the belief variant, the common adapter initialization provides a shared reference, and local validation determines whether an agent accepts a merged update. In the policy variant, all agents within a channel compute gradients at the same channel parameters θc\theta_c.

In both variants, raw observations and trajectories remain local, and no centralized critic is required.

Thank you for clarifying the parameter synchronization and experimental settings, which resolves my earlier questions. I still have a concern about the closed-to-open comparisons, since Table 1 uses different P2P sharing configurations and Table 2 uses different normalization references for the belief-sharing scores. Similar scores therefore do not, by themselves, show that the same configuration retains its performance when the population changes. This leaves the performance-retention claim insufficiently supported, so I maintain my original rating.

Reviewer_yeorOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces BISKY, a method that applies MARL to open multi-agent energy systems where agents can join, leave, or change behavior. It uses a dynamic bipartite graph to connect agents with similar operating behaviors and share knowledge through belief-adapter or policy-gradient merging, without requiring a centralized critic. The authors present experiments on peer-to-peer energy trading and EV charging that show improvements over decentralized MARL baselines, particularly with changing agent populations.

Critical strengths
  • BISKY is evalauted on two energy coordination tasks, peer-to-peer energy trading and EV charging, and generally outperforms the decentralized baselines in the open setting. The ablation studies do suggest that dynamic routing and knowledge sharing contribute to its performance.
  • I also find the paper well written with a clear motivation. I do believe the problem is relevant and important.
Critical weaknesses
  • The paper to me feels largely incremental. It combines existing techniques, including similarity-based clustering, federated model aggregation, TIES merging, and independent PPO. Adding the bipartite graph to organize knowledge sharing among agents is interesting, as it is a way to determine which agents should share knowledge and link them. However, I think the graph essentially performs behavior-based clustering followed by knowledge aggregation within clusters. It is unclear what the bipartite graph formulation contributes beyond existing clustering-based or role-based MARL approaches. The random and static routing ablations are useful, but I would have liked to see comparisons with simpler alternatives, such as a single shared channel or online clustering with the same merging procedure. And even so, the improvements over POAM in peer-to-peer energy trading appear relatively small and within overlapping standard deviations.
  • I also find some of the experimental results difficult to interpret. Table 1 does not clearly specify whether BISKY uses belief sharing or policy sharing, or I fail to understand it. Table 2 reports normalized returns without clearly specifying the environment or how the normalization is computed. This makes it difficult to connect the ablation results to the main results and understand which variant is responsible for the improvements.
Questions for authors
  1. Could you please clarify what the dynamic bipartite graph formulation adds beyond behavior-based clustering and knowledge aggregation? How does BISKY differ from existing clustering-based or role-based MARL approaches?
  2. The improvements over POAM in peer-to-peer energy trading are relatively small, with overlapping standard deviations. Are these improvements statistically significant esp across different random seeds?
1 comment

We thank the reviewer for the constructive feedback and for recognizing the importance of the problem and the clarity of the paper.

1. What BISKY adds beyond clustering. We agree that behavior-based grouping and model merging are established techniques, and we do not claim novelty in either component individually. BISKY's contribution lies in maintaining reusable behavior channels that dynamically connect agents with similar operating patterns, allowing shared knowledge to persist as agent membership changes. New agents can reuse knowledge learned by similar agents, and existing agents can switch channels as their behavior changes. Channel knowledge is preserved as membership changes, provided the channel remains active.

The bipartite graph represents the dynamic relationships between agents and these shared knowledge channels. Agent nodes represent the changing population, channel nodes maintain reusable knowledge, and dynamic edges determine channel membership. This separates knowledge storage from individual agents and allows knowledge sharing to adapt as agents join, leave, or change behavior.

This also distinguishes BISKY from role-based MARL methods such as ROMA. In ROMA, learned roles condition agents' policies within a centralized training framework. In BISKY, channels explicitly store and merge model knowledge across dynamically assigned agents, without requiring a centralized critic.

2. Simpler alternatives. We agree that a single shared channel is an important baseline and that random routing does not provide the same comparison. We have therefore added a single-channel baseline to Table 2, using the same merging procedure as BISKY. Its normalized scores are 40.4 ± 7.7 (closed) and 25.6 ± 8.5 (open), compared with 93.1 and 92.3 for BISKY's policy-sharing variant. The single-channel baseline also performs worse than random routing (90.4 and 75.5), indicating that merging all agents into one channel is substantially less effective than maintaining multiple channels.

The existing ablations further show the importance of behavior-based routing and persistent channel knowledge. In the open setting, BISKY achieves 89.5 (belief) and 92.3 (policy), compared with 70.4 and 75.5 for random routing, 73.1 and 78.3 for static routing, and 68.5 and 72.4 without retaining channel knowledge.

Together, these results show that BISKY's performance depends not only on model merging, but also on behavior-based routing and persistent channel knowledge.

3. Clarification of Tables 1 and 2. We have revised the captions of Tables 1 and 2 and Appendix C.2 to clarify the variants, evaluation settings, and normalization.

In Table 1, P2P trading uses policy merging, with additional belief-adapter merging in the closed setting. EV charging uses belief-adapter merging.

Table 2 reports ablation results for P2P trading only, covering both belief-sharing and policy-sharing variants. These results use five independent random seeds from runs separate from those in Table 1. The normalized scores use a random policy as the zero reference and SharedPPO, trained on pooled agent data, as the 100 reference, except for the open belief-sharing setting, which uses a different reference policy. SharedPPO is included only as a reference because pooling raw observations does not satisfy our locality requirement.

4. Statistical significance versus POAM. In the open P2P setting, the performance differences between BISKY and POAM are within the variability across random seeds. We therefore do not claim that BISKY significantly outperforms POAM in this setting. We have revised the Introduction and Section 4 to state that BISKY achieves comparable performance.

However, the two methods operate under different information assumptions. The POAM implementation used in our comparison is trained to predict teammates' observations or actions and requires the maximum team size to be specified in advance (Wang et al., 2024; Appendix C, NAHT/POAM). In contrast, BISKY is designed for a setting where each agent's raw observations remain local and the population size is not known in advance (Section 1). Thus, BISKY achieves comparable performance under more restrictive information-sharing assumptions, with lower reward variability across the evaluated seeds (standard deviation 12.4 versus 31.3).

Our results support the following claim: BISKY enables agents to reuse knowledge learned by behaviorally similar agents as they join, leave, or change behavior, without sharing raw local data or requiring a centralized critic.

Reviewer_3ietOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a Multiagent Reinforcement Learning approach focused on allowing learning how to coordinate and form charging/trading strategies in decentralized energy systems. The heart of the approach is to have an encoded routing signature that allows to identify certain patterns for each of the agents and then facilitate coordination. The approach is validated in seemingly realistic power grid simulations and overperforms baseline MARL methods. Some work could be done improving the written manuscript, and discussing better how to bridge the gap between simulation and a real deployment (more below), but in general I would say it's a valid incremental advance of the SOTA.

The way the experimental results are presented is not very helpful, Figure 2 with the rewards is purely "debugging" information and completely irrelevant to assess performance because what really matters as performance is only cost and energy used. I would move those graphs to the appendix (or if there is enough space, leave it just for the ablation analysis). I would instead would like to have seen how the two relevant metrics actually progress over time during learning. Especially in the "open" environment. Likewise, the rewards can be removed from table 1.

There is clear omission of related literature that is a bit "older". The present proposal is a bit more clever way of defining the agent neighborhood but there has been significant body of work using neighborhood-based learning for RL techniques in very similar coordinating charging applications. Representative work in the following but there are really A LOT of work in the area in the 2013-2020 timeframe (also there is some related work on learning for traffic systems):

Silva, Felipe Leno, et al. "Coordination of electric vehicle charging through multiagent reinforcement learning." IEEE Transactions on Smart Grid 11.3 (2019): 2347-2356.

Critical strengths

Paper focuses in a domain of practical concern and usefulness and use seemingly realistic simulations to validate the method. It shows a good number of MARL baseline techniques (tho it could have done a better job exploring non-learning baselines).

Approach can probably generalize to other domains and represent a genuine incremental advance for MARL learning methods.

Critical weaknesses

The manuscript is really verbose and the proposed approach is a bit hard to follow. There should be an algorithm describing the method and the description in text could be significantly compacted to be more direct and precise.

There are some significant gaps on how the problem would be described in a real deployment and how it's seen in the simulation (more in the questions).

Questions for authors

The routing signature is composed of features that are averaged across time steps, two questions about the routing signature:

(i) it seems really hard to generalize this to other domains, how would be the signature defined in another domain in a way that the feature wouldn't be included in the observation and still it would reveal the pattern of the agent?

(ii) It seems like there is a gap between how this would be implemented in practice. How would you identify that the routing signatures have to be recomputed? They are features calculated over time, but if you have a system long running that suddenly change those averages would not represent the current behavior unless you refresh it, and it would also not be useful if you refresh it all the time, so how to find the right balance? Furthermore, you would need a constant connection link to be transferring this info across all the agents all the time, this sounds very unrealistic in a practical deployment.

2 comments

We thank the reviewer for the positive assessment and the constructive questions regarding deployment and the routing mechanism.

1. Cost and grid metrics. We agree that cost savings and grid reduction are the primary practical metrics. We have replaced the reward curves in Figure 2 with cost-savings and grid-reduction curves over training in the open setting, with markers where agents join or leave. The original reward curves are now in Appendix D.

We retain reward in Table 1 as a complementary metric because the comparison provides additional information. In the closed setting, IPPO and InforMARL reach rewards close to BISKY's (338.7 and 339.2 vs. 358.1) but much less grid reduction (14.1% and 16.6% vs. 23.2%). Reward alone would make them appear nearly as effective as BISKY, whereas grid reduction reveals a substantial difference in system-level performance. For EV charging, the reward is the SustainGym objective itself (charging profit minus carbon cost minus constraint violations), so it is the right metric there.

2. Earlier neighborhood-based work. Thank you for highlighting Silva et al. (2020). We have added it to Section 2. We agree that neighborhood-based coordination has been extensively studied and do not claim novelty in grouping agents with similar characteristics.

BISKY differs in how shared knowledge is organized and reused. Rather than associating shared knowledge with particular agents or fixed neighborhoods, BISKY maintains persistent behavior channels that aggregate updates from agents with similar behavioral signatures. Agents can dynamically select or switch channels as their behavior changes, and new agents can reuse knowledge accumulated by existing channels. Thus, our contribution lies primarily in behavior-based knowledge organization and dynamic reuse rather than neighborhood identification alone.

3. Algorithm Clarification. We have added Algorithm 1 to Section 3 to clarify the overall procedure. It describes local data collection, signature updates, channel selection or creation, and the aggregation of member updates within each channel.

4. Question 1: Generalization of routing signatures. The signature does not require information beyond an agent's local observations. Its purpose is to summarize behavioral patterns across time rather than introduce additional observable features.

For example, two households with and without rooftop solar may appear identical in terms of solar generation at midnight. However, their generation patterns become distinguishable when observations are aggregated over several days. BISKY combines historical statistics, such as typical demand and solar generation, with recent operating conditions, such as battery state and energy consumption, to characterize agent behavior.

Similarly, routing signatures can be defined for other domains using domain-relevant observations. For example, in traffic signal control, a signature could summarize queue lengths and signal phase durations over historical and recent windows. Although we have not evaluated BISKY in other domains, its routing and channel-selection mechanisms do not depend on energy-specific features.

5. Question 2: Signature updates and communication. BISKY does not require a separate change-detection mechanism to trigger signature recomputation. Signatures are updated periodically at routing rounds using historical and recent observations. Since the signatures consist of aggregated statistics, their computation is inexpensive. In our P2P setup, routing occurs once per PPO update, every few episodes (corresponding to a few days of operation). Importantly, recomputing a signature does not necessarily trigger channel switching: an agent switches only if another channel matches its signature better than its current channel by a fixed margin. By combining a historical window (the last four days) with recent observations, the signature can capture persistent behavioral changes, such as a new battery or seasonal changes, while reducing the effect of short-term fluctuations, such as an unusual day.

Regarding communication, BISKY does not require direct agent-to-agent exchanges or continuous communication links. Each agent communicates with the channel infrastructure at the routing intervals described above, submitting its signature and model update without transmitting raw local observations. In a practical deployment, channels could be maintained by a shared market or coordination platform.

We have clarified these deployment considerations in the Conclusion. We also acknowledge that our current evaluation is simulation-based and does not explicitly model communication delays, network failures, or real-world deployment overhead. Evaluating these effects remains important future work.

Thank you for the further clarifications. In general I keep my already-positive assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.