acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Intent Signals for Intra-Step Multi-Agent Coordination

Abstract

Decentralized agents must typically select actions at the same instant from local observations, so no agent can see its teammates’ current-step decisions. Centralized training with decentralized execution (CTDE) stabilizes learning, but it does not remove this execution-time ambiguity. Most existing communication protocols transmit encoded trajectories or multi-step forecasts, so each recipient still has to guess what others will do now. We study a two-stage policy that runs inside a single environment timestep while preserving simultaneous execution. Each agent first emits a compact signal trained with a reconstruction objective to retain both its local view and a draft action distribution; agents then form executed actions by conditioning on the collected signals. Task-driven policy gradients are withheld from the message module so that the signals remain semantically stable and can serve as reliable inputs to the joint policy, allowing competing intents to be reconciled before the environment transitions. On fully cooperative suites and team-versus-team tasks against learning opponents, this design consistently improves over recent communication-based MARL methods.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, multi-agent communication, decentralized execution, coordination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6UQKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents PreACT, a framework for cooperative multi-agent reinforcement learning that uses self-supervised intent signals to enable coordination within a single environment timestep. The framework adopts a two-stage policy: agents first generate and broadcast messages from local observations and policy representations, then condition their final actions on the collected messages while preserving simultaneous execution. The communication module is trained through observation reconstruction and action-distribution prediction, with task-driven policy gradients withheld to promote semantic stability. Experiments on Multi-Agent Particle Environments, Level-Based Foraging, and Google Research Football demonstrate competitive coordination and task performance relative to the evaluated baselines.

Critical strengths
  1. The experiments are extensive, covering diverse scenarios and a wide range of baselines, with evaluations of several aspects of the method.
  2. The paper focuses on a well-motivated problem that is relevant to practical multi-agent coordination.
Critical weaknesses

Scalability: The method relies on team-wide broadcasting, which may limit its scalability. In the current implementation, every agent sends messages to the entire team, without learning when to communicate, whom to communicate with, or how much information to send. In larger teams, many messages may be irrelevant to the current decision, increasing the communication burden. Moreover, the experiments are limited to simulated environments, so the method’s applicability to real-world settings remains unverified.

Evaluation: I have some concerns about whether the messages maintain consistent semantics throughout training. The messages are 16-dimensional continuous vectors. Although the paper examines their semantics through a decoder and visualizations, it does not clarify what particular dimensions or regions of the message space represent. Additional evidence showing whether similar messages retain the same meaning across different stages of training would help support this claim.

Questions for authors
  1. How are the tentative actions in Figure 7 computed? Are they obtained from the message decoder or from a policy evaluated without teammates’ messages?
  2. How are the learned opponents evaluated across methods? Are teams tested against a common set of fixed opponents, or only against their own co-trained opponents?
Reviewer_wBvPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PreACT, a communication method for multi-agent reinforcement learning. Each agent first generates a message from its local observation and policy representation, then uses the collected messages to determine its action within the same environment timestep. The message module is trained to reconstruct local observations and predict the action distribution of the policy after communication, without receiving direct policy gradients. The aim is to learn stable, action-relevant messages that improve coordination. The authors evaluate PreACT on MPE, LBF, and GRF tasks and report improved rewards over the selected baselines. Ablation studies examine the message training objective, input design, and robustness to communication disruptions.

Critical strengths
  1. The combination of observation reconstruction and action distribution prediction provides a reasonable way to train messages that contain both perceptual and behavioral information. This gives the communication module an explicit objective related to coordination.

  2. The results, particularly on GRF counterattack_hard are promising. PreACT shows a clear increase in reward after approximately 5M environment steps and achieves substantially higher rewards than the reported baselines within the evaluated training budget. This provides evidence that the proposed design can benefit coordination in this task.

Critical weaknesses
  1. The experimental comparisons are incomplete and difficult to assess quantitatively. The paper does not provide a comprehensive table summarizing the final episodic returns or success rates across tasks and methods. Some baselines are also missing from individual plots in Figures 2 and 3 without a clear explanation. For example, SMPE is absent from the GRF comparisons. These results should be included where applicable, or the reasons for their omission should be explained. The baseline selection also needs stronger justification given recent developments in this area. Relevant methods such as MATCH and CAIC are discussed in the related work but are not evaluated. Together, these gaps make it difficult to assess the extent of the improvement over existing methods.

  2. The training budgets and the interpretation of the learning curves require further justification. In Figure 3, the LBF tasks use 5M environment steps, whereas the GRF tasks use approximately 10M. Different tasks can reasonably require different budgets, but the paper does not explain how these budgets were selected, and several curves have not clearly reached a stable plateau. In GRF counterattack_hard, PreACT shows a substantial reward increase after approximately 5M steps. This is a positive result, but it also raises the question of whether other methods might show similar improvements with longer training. Extending training for the compared methods under a common budget within each task would help distinguish faster learning from better long-term performance. The current results demonstrate an advantage within the reported budget, but do not establish whether this advantage persists after longer training.

Questions for authors
  1. In Figure 1, is MsgDec intended to predict the final action probability distribution produced after communication, using only the agent's own message? If so, 'Predicted Action Probability Distributio' would be a more accurate label than 'Predicted Draft Action'.

  2. How were the training budgets selected for each task? Do the authors have longer training runs for counterattack_hard or other tasks where the curves are still improving? These results would help determine whether the observed advantage reflects faster learning, better long-term performance, or both.

  3. What exactly does the '50% stochastic delay' setting in Figure 4 represent? Please clarify the delay duration, how delayed messages are handled, and whether delays are introduced during training, evaluation, or both. Also, does Table 3 include message transmission and synchronization costs? Since the paper discusses the latency associated with sequential coordination, a direct comparison of the complete decision latency against SeqComm would help establish the practical benefit of the proposed protocol.

Reviewer_2cQSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a two-stage communication framework for multi-agent coordination. The problem that the paper aims to solve is that, in practice, agents have to choose an action at the same time as others, so their actions can’t be conditioned on the other agents’ action. The two stage protocol aims to resolve this: in the first stage, the agent share an “intent” message which is then used as conditioning for actual action generation in the second stage. The message generator is trained using a reconstruction loss, rather than from policy gradient. As such, it is aimed to be policy independent.

Critical strengths

The algorithm is compared with several related baselines. In general, the simulation studies show that the algorithm converges faster or at better returns than the baseline. The general design of the method is sound (and in line with many other approaches in this line of work). The simulations also cover a variety of benchmark environments (continuous, grid world, etc.) with useful ablations.

Critical weaknesses

My main concern is the novelty of the actual work and the lack of comparison with some very related papers. The actual problem the paper aims to solve.. simultaneous action execution through intent sharing, is not new, and has received a lot of attention. There are many papers on learned message generation for MARL, many of which encode latent information that serve as intent recognition. Many of these papers also rely on reconstruction type loss, rather than receiving policy gradients. So although the specific architecture is new, the general approach is very common.

More crucially, there are no comparisons with some very direct baselines. For example, TarMAC which is one of the more commonly used baseline in learned message generation, is very close to the second stage of this paper. TarMAC does use reward gradients but is a very natural baseline. Similarly, MASIA is a very common baseline in this line of work that’s missing.

Questions for authors

How does the proposed method compare with MASIA and TarMAC?

The ablations seem to suggest that the major gain comes from reconstructing the observations more than intent generation. If that is indeed the case, then it contradicts the paper’s own framing.

If gradient separation (i.e., not using policy gradients for the learned message function) is the main claim, then it warrants more experiments. For example, comparisons with TACTIC which solves a similar problem without policy gradients. The ablation compares reward-only message training with reconstruction-only. It is missing the natural middle ground: reconstruction loss plus policy gradients. That is what most auxiliary-loss communication methods do, and it is the variant needed to show that isolation itself matters.

Were same hyper parameters used for the proposed method across environments? If not, why. And was similar hyperprameter tuning allowed for baselines?

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.