acceptodds
Under review as a conference paper at ICLR 2027

Does Reward Select Interaction? Identifying and Steering Information Pathways in Multi-Agent Policies

Abstract

Multi-robot remote estimation requires allocating each robot's limited transmission quota between cached measurements and source-discovery notices. We propose a local transmission scheduler with a fixed sampler–relay covering rule. An additive score combines proxies for Gauss–Markov uncertainty and delivery probability with timeliness slack; a threshold defers low-score data. The cross-entropy method optimizes the score weights and threshold offline to minimize estimation mean squared error. A finite-horizon analysis of an isolated delivery motivates the local features and the ranking-and-threshold structure. On 200 held-out instances with 12 sources, 30 robots, and a 330-frame team budget, the scheduler reduces mean squared error by 0.0162 (95% confidence interval: [0.0089, 0.0235]), increases timely information coverage by 3.49 percentage points, and reduces mean transmissions from 326.4 to 310.6 frames relative to covering without scheduling. Independent retraining at a 360-frame budget reproduces the scheduling benefit.

Primary area:
reinforcement learning
Keywords:
Multi-agent reinforcement learning, shortcut learning, causal intervention, information pathway

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HZEuOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
2
Presentation
2(−1)
Soundness
1(−1)
Read review
Paper summary

This paper examines whether high team performance implies that a multi-agent policy relies on peer information. In a flocking task, native IPPO achieves polarization of 0.977 but drops to 0.339 when a shared reference is removed. The authors formalize this shortcut through policy classes that omit or admit observation channels and derive an (N−1/2N^{-1/2}) performance bound for peer-omitting policies under independent references and idealized dynamics. They introduce SCIA, which audits frozen policies using channel decorrelation, zero ablation, and action influence. They also propose SI-PO, which randomizes reference directions during training without modifying the reward or PPO update. Experiments with scripted controls, NEAT, IPPO, and several task variants evaluate the resulting information dependence.

Critical strengths
  1. Proposition 2 formalizes why return maximization need not select peer interactions, while Proposition 3 identifies conditions under which peer-omitting policies cannot achieve high polarization.

  2. SCIA provides a systematic intervention-based diagnostic that distinguishes channel availability, correspondence, and action influence. Seven hand-specified controls clarify why zero ablation alone can be misleading.

Critical weaknesses
  1. Peer shuffling preserves empirical input marginals but may disrupt the conditional relationship between peer observations, local states, and physically feasible trajectories. Performance degradation may therefore reflect observation inconsistency as well as dependence on genuine peer information.

  2. Proposition 3 relies on independent local frames, decoupled rotation-equivariant dynamics, and a polarization-only objective. The practical simulator includes coupled dynamics and additional reward terms. The direct experiment supports the random-heading prediction but does not establish the full optimization-gap result for the original training objective.

  3. Both SI-PO and zero-reference training induce peer dependence in the main alignment task. However, zero-reference training achieves higher peer correspondence and reference-loss polarization. The paper does not establish a consistent advantage from preserving the reference channel.

  4. In the speed-cue task, SI-PO's speed-matching advantage over zero-reference training is not established. The retained speed code is also barely used.

  5. In goal reaching, the SI-PO analogue fails its predefined performance criterion, whereas the zero-reference analogue succeeds. The successful pathway-switching results therefore do not establish that port preservation is beneficial across tasks.

Questions for authors
  1. What is the precise causal quantity identified by SCIA? How can the authors distinguish dependence on genuine peer information from sensitivity to physically inconsistent observations introduced by cross-world shuffling? Could physically consistent or state-matched interventions provide a useful control?

  2. Under what conditions does SI-PO offer an advantage over zero-reference training? The speed-cue experiment does not establish the intended information-preservation benefit.

  3. The matched mixture encounters shortcut-incompatible conditions in only one third of training worlds, whereas SI-PO and zero-reference training encounter them throughout training. Could the mixture's failure reflect reduced exposure rather than an intrinsic limitation of the mixed objective?

  4. How reliable is SI-PO across independent training runs? The initial five-seed reference-loss criterion failed and passed only after adding five seeds. Could the authors report the per-seed success rate and perform an independently specified replication?

  5. How broadly do the theoretical and diagnostic conclusions extend beyond the controlled flocking setting? How would the analysis change with shared environmental dynamics, alternative coordination signals, or observation channels that cannot be cleanly separated into reference and peer statistics?

Reviewer_LU32OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether agents coordinate by using their peers or following a shared reference. The authors introduce SCIA, which scramble the observation channel to distinguish these behaviors, and tests if training references encourages peer use. Their results shown that standard IPPO method can align well yet fails without when reference was removed, and just by training with independent or no reference quite solve the problem.

Critical strengths

The motivation is interesting for the field of multi-agent reinforcement learning.

Overall, the reference experiments on figure 1 make the paper's motivation convincing. The agents look well coordinated until the reference disappears, even though they still observe their peers. Together with the low peer-correspondence effect in table 2, leading to an interpretation on why high team performance alone is insufficient evidence of peer-based coordination.

I found the comparison between removing and scrambling inputs ok. The examples with redundant channels on table 3 shows how removing either input alone can miss the pathway that the policy uses. In the same way, similar patter in learned policies appears in figure 3, which make this distinction empirically relevant.

The experiments are convincing, and the number of seeds per experiment is appropriated.

Critical weaknesses

As I mentioned earlier as a strength, the results are convincing in the tested settings. However, I am unsure how well the findings extend to the other forms of coordination. The additional tasks broaden the evaluation, although they still use the same simulator and retain much of the original observation structure of section 4.3. This leaves me uncertain about how well SCIA and SI-PO would transfer to tasks where cooperation involves different roles or longer sequences of interdependent actions.

SI-PO achieves a higher mean return in the main task, but zero-port produces stronger peer dependence and better alignment after reference loss. Given the lack of and established benefit in the speed-cue task and weaker navigation results on tables 7 and 8, I am unsure when SI-PO should be preferred over the simples zero-port approach.

As a minor issue, personally I don't like the way the individual confidence intervals are presented within the text. I know the intention, but it makes the text polluted and more difficult to read. For my perspective, just using the average value would be enough.

Questions for authors

How much do SCIA and SI-PO depend on having clearly separated peer and reference inputs? Could you give a concrete example of how they would apply to a different coordination task, such as Overcooked? In particular, what would count as the peer information and the shortcut cue, and how would you intervene on them?

An experiment outside the current task family would strengthen the evidence for generality. I see this as a suggestion for further validation, rather than a request for a complete new benchmark during the rebuttal, since the time may not be enough.

Reviewer_MSwxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Whether it is, or is not actually the case, this paper reads like it was mostly written by an LLM and is not reviewable. I am an expert on the field and have close to no idea what it is talking about. There may be an interesting idea here, but the paper would require so much rewriting that I am neither able to parse it nor even able to begin to suggest how to improve it. I am sorry that this is rude, but I think it is also rude to ask reviewers to spend time on something this unpolished.

Critical strengths

Read the summary.

Critical weaknesses

Read the summary.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.