acceptodds
Under review as a conference paper at ICLR 2027

From Influence to Utility: Reference-Aligned Message Scheduling for Frozen Multi-Agent Policies

Abstract

A learned communication policy can generate more messages than a deployment link can carry. Selecting among them requires distinguishing a useful receiver response from a merely large one. We study this problem with the actor and communication stack frozen. CA-RAPU evaluates each candidate-induced policy shift against an action reference from other senders addressing the same receiver, then combines positive progress with influence, delivery reliability, confidence, and cost. The analysis characterizes the ambiguity of unsigned influence and gives sufficient conditions under which reference error preserves the selected set. Across 20 independently trained stacks in an assignment-controlled Spread task, CA-RAPU improves return over unsigned KL by 0.0277 [0.0153, 0.0401] at identical traffic. When each method calibrates its own transmission threshold, the net-return gain is 0.132 [0.112, 0.153]. Brackets give paired 95% confidence intervals. Redistributing peer support reverses the comparison with KL even when queue size stays fixed. Reference alignment thus helps allocate a frozen policy's messages when peers provide informative evidence about receiver actions.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, communication scheduling, receiver influence, reference alignment, constrained communication

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vjpyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CA-RAPU, a training-free message scheduling method for MARL systems with frozen policies and communication networks. The method determines which messages to transmit under bandwidth constraints by measuring how each message changes the receiver's action distribution and whether that change aligns with a reference constructed from other agents' messages.

The method is primarily evaluated on a modified Simple Spread benchmark with simulator-generated noisy target-location clues, alongside a smaller exploratory CommNet Predator–Prey experiment. Results show improvements over KL-based scheduling in the modified Spread environment, but mixed results in CommNet.

Critical strengths

Practical motivation: Addresses communication constraints in pretrained MARL systems without requiring policy retraining.

Critical weaknesses

W1. Limited generalizability due to highly task-specific assumptions. The method requires a meaningful reference derived from other agents' messages. In the primary experiment, the simulator explicitly provides noisy estimates of other agents' target locations, which are encoded and decoded to construct this reference. This is a highly customized communication setting, unlike conventional MARL methods that exchange learned latent representations without explicit target-location information. It remains unclear how CA-RAPU can be applied to general pretrained communication policies where such information is unavailable. The CommNet experiment provides insufficient evidence to establish broad applicability.

W2. Insufficient justification and ablation of the encoder-decoder architecture. The proposed experimental architecture uses an encoder to transform two-dimensional target coordinates into two-dimensional messages, followed by a decoder that reconstructs target coordinates. The motivation and necessity of this architecture are unclear, particularly since transmitting the original coordinates directly would be a simpler alternative. The paper lacks a direct comparison against raw-coordinate transmission without the encoder-decoder, making it difficult to understand the benefit of this design and its relationship to the proposed scheduling mechanism.

W3. Insufficient comparisons with established communication-efficiency methods. Most primary baselines are heuristic variations or paper-specific learned comparators rather than established MARL communication algorithms. Although the paper discusses attention-based, belief-based, and learned communication scheduling methods, it provides limited empirical comparisons with them. In particular, Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (NeurIPS 2019) is relevant prior work that is not cited nor compared against. While full comparisons with jointly trained communication algorithms may require adapting the experimental protocol, stronger baselines incorporating established uncertainty-, variance-, attention-, or belief-based message-selection principles are necessary to establish the proposed method's advantages.

W4. Insufficient benchmark diversity and limited evidence of generalization. The experimental evaluation relies predominantly on one heavily modified Simple Spread benchmark, with three- and four-agent variants representing essentially the same task. The additional CommNet Predator–Prey experiment covers only six trained policies and produces mixed results. CA-RAPU underperforms KL-based scheduling on five of six policies under fixed communication capacity, and the planned larger experiment was not conducted. Consequently, the experimental evidence is insufficient to demonstrate the effectiveness of the proposed method across diverse MARL tasks and communication architectures.

Questions for authors
  1. General applicability: How can CA-RAPU construct a meaningful action reference for conventional MARL communication architectures where messages are learned latent representations rather than explicitly decodable task information? What happens when simulator-generated target clues are unavailable?

  2. Encoder-decoder necessity: Why is an encoder-decoder required when the underlying information consists of two-dimensional coordinates? Could the authors provide an ablation comparing against direct coordinate transmission without encoding?

  3. Stronger baselines: How does CA-RAPU compare with established communication-efficiency methods? Could relevant message-selection heuristics from these methods be evaluated under the same frozen-policy setting?

  4. Broader evaluation: Could the authors demonstrate effectiveness on additional established MARL benchmarks and communication architectures, particularly those using naturally learned latent messages without task-specific reference information?

  5. Communication overhead and deployment assumptions: The method assumes a centralized scheduler with access to candidate messages and receiver-policy states. How are these obtained in a distributed MARL system? Would gathering this information incur additional communication costs, and how would these affect the reported bandwidth savings?

Reviewer_qmNKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies communication efficiency in multi-agent reinforcement learning. Existing influence-based methods usually evaluate message importance by measuring how much a message changes the receiver's policy. However, the authors argue that a large policy change does not necessarily improve task return.

Therefore, the paper proposes CA-RAPU. It studies how to select more useful messages from those already generated under limited communication bandwidth. The method keeps the agent policies and communication models frozen. It uses messages from other senders to construct an action reference. This reference helps determine whether the policy change caused by a candidate message is beneficial to the task. The method then ranks and selects messages based on policy influence, message confidence, delivery reliability, and communication cost.

The authors conduct experiments in a modified MPE Simple Spread environment. They also use a CommNet Predator–Prey experiment to examine the transfer of the method. The results show that CA-RAPU achieves higher average returns under a fixed communication budget.

Critical strengths

(1) The research problem is clear, and the method has some novelty.

The paper considers the direction of the policy change caused by a message when scheduling communication. It does not select messages only based on the magnitude of their influence. I think this is a reasonable idea.

In addition, CA-RAPU does not require retraining the original agent policies or communication models. Instead, it selects from messages that have already been generated. This design has some practical value. It is especially useful when policies have already been trained but communication conditions change during deployment.

(2) The experiments are carefully designed and support the effectiveness of the method in controlled settings.

The paper compares the task returns of different message selection methods. It considers both fixed communication budgets and adaptive communication rates.

In the fixed-traffic experiments, the authors use 20 independently trained policies for comparison. Table 1 shows that CA-RAPU improves the average return over unsigned receiver KL by 0.0277.

The authors also change the distribution of peer messages. The results show that the effectiveness of the reference direction depends on the quality and distribution of peer information. The related ablation experiments, confidence intervals, and additional analyses in the appendix are also relatively complete.

Critical weaknesses

(1) The method relies on task-specific reference information. Its applicability to general multi-agent communication tasks is still unclear.

The main idea of CA-RAPU is to construct a reference direction from messages sent by other agents. It then checks whether the policy change caused by a candidate message agrees with this direction.

However, the main experiments use a modified Simple Spread task. The messages mainly contain noisy coordinate information about target landmarks. The reference adapter uses this information to calculate a target direction and then maps it to action scores.

This design is suitable for navigation tasks. However, messages in more general multi-agent communication tasks may contain high-dimensional features, hidden intentions, or other representations. These representations may be difficult to interpret directly. In such tasks, constructing a task-related action reference from other agents' messages may be difficult.

In addition, the authors conduct a CommNet Predator–Prey experiment in Appendix L. Under a fixed communication capacity, CA-RAPU has a negative average return difference compared with unsigned receiver KL. This result does not show a clear advantage.

Therefore, I think the paper provides good evidence that CA-RAPU works in tasks with clear target directions and redundant peer information. However, the current results do not sufficiently show that the method applies to different types of multi-agent communication policies. This limits the generality of the contribution.

(2) The actual performance improvement over existing methods is limited and needs further discussion.

In Table 1, CA-RAPU improves the average task return over unsigned receiver KL by 0.0277. Compared with CA-RAPU's average return of about 2.70, the improvement is only around 1%.

The task success rate also increases from 13.16% to 15.29%, an absolute improvement of about 2.13 percentage points. The performance difference compared with the action-distribution L2 method is not statistically significant.

In addition, CA-RAPU requires extra computation to construct the reference direction. Appendix M reports the median CPU latency per scheduling decision with six candidate messages. It is about 3.87 ms for cached CA-RAPU, compared with 2.75 ms for unsigned receiver KL.

Since CA-RAPU is designed for communication-constrained deployment, the authors should discuss whether the observed performance gains justify the additional computational cost.

In the experiments with adaptive transmission thresholds, CA-RAPU achieves a clearer advantage in net return. However, it also sends more messages on average than the KL method. This result shows a better trade-off between task return and communication cost. It does not necessarily mean that the method achieves the same level of improvement under an equal communication budget.

I think the paper should distinguish these two types of improvements more clearly and discuss their practical value.

Questions for authors

(1) About the applicability of the method

In Spread, CA-RAPU constructs the reference direction from peer messages containing coordinate information. How should such a reference adapter be designed for tasks without clear action semantics?

In Appendix L, CA-RAPU does not show a clear advantage in the fixed-capacity CommNet Predator–Prey experiment. What do the authors think is the main reason? Could this be related to the difficulty of obtaining a reliable task direction from peer messages?

(2) About the practical significance of the performance improvement

In the fixed-traffic experiments, CA-RAPU improves the average return over unsigned receiver KL by about 0.0277. However, it also requires additional computation for reference construction.

How do the authors view the trade-off between this performance improvement and the computational cost? In practical deployment, when would CA-RAPU be preferable to the less computationally expensive KL-based scheduler?

Also, could the authors further explain the different advantages of fixed-traffic message selection and adaptive transmission thresholds in terms of net return?

Reviewer_VG8sOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for message scheduling between frozen MLP encoders in communication channel where only a subset of message can be communicated. The proposed scheduler scores candidate messages based on the change in receiver action distribution as measured by KL and a signed progress term toward a reference action based on other messages to receiver. On synthetic experiments, they report minor gains over unsigned KL baseline at equal traffic.

Critical strengths

Many statistics and comparisons are reported including null/negative results.

Critical weaknesses
  • Unclear and confusing writing with nonstandard, vague phrases that make the paper hard to read.
  • The paper's focus seems outside the scope of ICLR and unrelated to learning. A communications venue seems like a better fit
  • The gains seem marginal at best with only +0.0277 improvement over the baseline
Questions for authors
  • What does CA-RAPU stand for? It is never defined?
  • What is the main contribution of the paper and how is it related to learning representations?
  • How is the theory connected to the experimental results? Which experiments test the theory?
  • Have you tried other models and datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.