acceptodds
Withdrawn from ICLR 2027

Stationary Alone, Drifting Together: Adapting to Interaction-Mediated Nonstationarity in Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

External conditions can change how agents interact through shared infrastructure without changing their dynamics in isolation. This drift persists with frozen partners and vanishes for a lone agent. We formalise this interaction-mediated nonstationarity as a family of partially observable Markov games indexed by severity and introduce three public benchmarks for routing, voltage control and cooperative lifting, preserving their original rewards. We propose PACT (Peer-Action Coupling Tracking), a wrapper for unmodified multi-agent learners. Each agent predicts disturbances from peers' broadcast actions and the infrastructure layout using an online model whose size is independent of fleet size. A trust level controls compensation or steering, with zero trust recovering the host exactly. In a linearised feeder, predictive compensation with exact estimates raises the instability threshold without bound as trust approaches its maximum. In a routing model, excessive trust causes oscillations that make the route predicted to be faster become slower. Experiments against representative baselines demonstrate consistent improvements over the host learners across all benchmarks.

Primary area:
reinforcement learning
Keywords:
Multi-agent Reinforcement Learning, Interaction-Mediated Non-stationarity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NebfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers a special case of non-stationarity in multi-agent reinforcement learning where only the interaction couplings between agents drift, while an agent's isolated dynamics remain stationary. They propose a model for describing Markov games under such non-stationarity, and propose a wrapper which adapts a stationary RL method to this setting. The method is validated on routing, voltage control, and collaborative lifting environments.

Critical strengths
  • The experimental evaluation is thorough and well-designed. It contains a sufficient number of baselines, metrics, and problem environments.

  • The proposed method is widely applicable, as it can be bolted on top of arbitrary stationary RL methods.

Critical weaknesses
  • The related work section is too dense. It describes a large number of related works, in more detail than I believe is necessary, all contained in a single large paragraph. The section would benefit from a leaner, more directed exposition.

  • Definition 3.1 is not sufficiently precise. The main objects presented are largely described in prose. Given that this defines the main class of problems studied in the paper, the definitions should be pinned down mathematically, specifying domains, co-domains, along with any regularity conditions assumed. The paragraph following this definition seems to carve out a special exception for the routing problem, implying this formalisation is not sufficient for modelling the desired class to begin with. This exception is also justified using forward references, which further obscures the exposition.

  • Table 1(a) is missing some information. The caption claims that PACT with g≡0g \equiv 0 recovers its host, but it does not state which host this is. Moreover, the g≡0g \equiv 0 row does not seem to match any of the other hosts shown in that table. In general, it is not clear which host PACT wraps in each of the experiments.

Questions for authors
  • PACT seems to wrap only a single method per application problem. How was this chosen, and how does wrapping generally affect the remaining host learners?

  • PACT assumes each agent has full access to its peer's actions during execution. This is often not the case in larger MARL environments, and many of the baseline methods considered assume only local agent information during execution. Could PACT be generalised to respect these constraints?

Reviewer_ZLoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interaction-mediated nonstationarity in MARL and proposes PACT, which estimates changing coupling effects using peer actions and predefined structural information. The method is evaluated on multiple MARL benchmarks, with theoretical analysis on compensation and stability.

Critical strengths
  1. The paper considers an interesting and practically meaningful type of nonstationarity that is not sufficiently addressed in existing MARL works.

  2. PACT is relatively simple and can be applied to existing MARL algorithms. The theoretical analysis is useful, and the results in MAPDN and VMAS are promising.

Critical weaknesses

Weakness 1: Limited Generality and Evaluation Fairness

PACT relies on predefined coupling information, broadcast peer actions, and environment-specific measurements and compensation rules. Although the paper evaluates different domains, the environments are modified using coupling structures that are also provided to PACT. Therefore, it is unclear how well the method works when such information is inaccurate or unavailable.

Also, many baselines do not have access to the same coupling information. This makes it difficult to separate the contribution of the proposed method from the advantage of additional structural information. Appendix B provides detailed HP analysis for PACT, but the tuning procedures and computational budgets for baselines are not clearly reported. I think these issues make the experimental comparisons less convincing.


Weakness 2: Concerns about Appendix D and Empirical Improvements

The authors provide extensive theoretical analysis of individual baselines in Appendix D. However, many arguments mainly show information limitations rather than directly explaining the observed performance gaps. I think some conclusions are stronger than what the theoretical results support.

There are also some concerns about the application of the theoretical results. For example, Lemma D.1(iii) is used for URB in Section D.21, although it considers action corruption in feeder and lifting tasks. Also, Lemma D.1(ii) characterizes a regression target but does not directly guarantee convergence to an optimal policy. These arguments need more justification.

Finally, in URB (Table 1), most of PACT's improvement already appears at sigma = 0. This makes it unclear how much improvement actually comes from adapting to nonstationarity.


Questions for authors
  1. How robust is PACT when the coupling structure is inaccurate or partially unknown?

  2. Were comparable HP tuning and computational budgets used for all baselines? Can the authors compare with baselines using the same coupling information?

  3. How do the theoretical results in Appendix D explain the actual performance gaps between algorithms?

  4. In URB, how much of the improvement comes from nonstationarity adaptation rather than general routing preference?

Reviewer_Wy6POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces and formalizes a distinct category of nonstationarity in multi-agent reinforcement learning termed interaction-mediated nonstationarity. In this setting, individual agents' private dynamics remain stationary in isolation, while an exogenous time-varying driver rescales the coupling gains through which agents affect one another over shared physical infrastructure. Consequently, the nonstationarity vanishes for a lone agent on the medium yet persists even when all peer policies are frozen. To benchmark this problem class, the authors instantiate three physics-grounded environments across urban traffic routing, active voltage control in power distribution feeders, and cooperative robotic lifting under supply-rail droop, alongside an additional evaluation on Multi-Agent MuJoCo.

Methodologically, the paper proposes PACT (Peer-Action Coupling Tracking), a lightweight decentralized wrapper around unmodified host MARL learners. Utilizing a declared infrastructure coupling operator WW and broadcast peer actions, each agent aggregates peer loads into rr low-dimensional coupling channels and estimates the time-varying coupling gains (r+1r+1 parameters, independent of fleet size nn) online via recursive least squares (RLS) with a forgetting factor. Scaled by a covariance-based directional confidence and a trust hyperparameter gg, PACT either directly inverts additive/multiplicative action distortions or steers policy logits away from high-cost candidates. Theoretically, the authors establish RLS tracking bounds, show that persistent excitation is upper-bounded by policy indecision, bound return degradation via the compensation residual, and analyze the closed-loop collective feedback induced by decentralized compensation—proving that predictive compensation scales the stability threshold by 1/(1−g)1/(1-g) on a linearized feeder, whereas excessive trust induces route-flapping oscillations in congestion routing. Extensive experiments against 20+ baselines across nonstationary RL, robust RL, agent modeling, and disturbance observers demonstrate consistent performance and stability gains.

Critical strengths

(1) Crisp Problem Formulation and Insightful Closed-Loop Theoretical Analysis: Separating interaction-mediated drift (C) from co-learner drift (A) and exogenous single-agent drift (B) via two testable properties (vanishing alone, persisting under frozen partners) is both conceptually clean and practically grounded in shared-infrastructure control. More importantly, the theoretical analysis goes well beyond standard black-box tracking + Performance Difference Lemma bounds:

  • Proposition C.12 & Proposition D.4 rigorously explain why cause-based predictive compensation (using broadcast peer loads) fundamentally outperforms effect-based disturbance observers (like ESO) in coupled physical loops: a one-step-late observer on an agent's own residual shifts the characteristic polynomial and actually lowers the collective instability threshold below that of an uncompensated fleet (Θc=(2−l)/(2−l+gl)<1)(\Theta_c= (2-l)/(2−l+gl)<1) whereas PACT raises it by 1/(1−g)1/(1−g).
  • Lemma C.7 & Proposition C.8 formalize the multi-agent dual-control dilemma, showing that channel excitation is bounded by policy entropy/covariance, which justifies both declaring the structural operator WW and using directional prediction confidence ψiTPiψi\psi_i^TP_i\psi_i rather than tr(Pi)tr(P_i).

(2) Exceptional Depth of Baseline Analysis (Appendix D) and Empirical Rigor: Rather than merely reporting empirical numbers, Section D provides a mathematical dissection of over 20 baselines across 10+ subfields, explicitly identifying the exact information-theoretic floor (Lemma D.1) or structural mismatch where each family fails (e.g., why knowing the exogenous driver A(t)A(t) in LCPO/RMA still leaves the peer-load variance Var(x_i(t)|\mathcal{H}_i) uncompensated, and why uniform mean-field aggregation in MF-Q/MF-AC fails under heterogeneous spatial weights Wi,jW_{i,j}. This makes the empirical gains on MAPDN and VMAS highly convincing and transparent.

Critical weaknesses

(1) Strong Structural Priors and Benchmark-Wrapper Alignment ("White-Box" Coupling Geometry): PACT's dramatic sample efficiency and parameter compactness (r+1=4r+1=4 scalars) rely on two strong domain assumptions: (i) exact knowledge of the infrastructure coupling operator WW (e.g., exact linearized DistFlow reactances Sm[i,j]S_m[i,j] in MAPDN and exact spatial decay scale λ\lambda and hardware class gains recvcls(i)\text{recv}_{\text{cls}(i)} in VMAS), and (ii) immediate, noiseless access to all peers' executed actions ajex(t−1)a_j^{\text{ex}}(t-1) and an explicit, uncontaminated per-step sensor for relative excess yi(t)y_i(t) (e.g., observing both aisenta_i^{\text{sent}} and aiexa_i^{\text{ex}} separately). Crucially, because the authors also designed the interaction-mediated harm injections on the stock URB, MAPDN, and VMAS benchmarks (Section 3.1), the benchmark harm definitions (Eqs. 7–10) and PACT's basis construction Λi(⋅;W)\Lambda_i(\cdot; W) share the exact same operator WW and channel structure below saturation (Proposition C.2(i)). While URLS tests removing WW, URLS must fit O(n)O(n) raw parameters; there is no evaluation of PACT under misspecified or approximate operators W^≠W\hat{W} \neq W (e.g., inaccurate line impedances or unmodeled cross-class leakage) or noisy excess measurements.

(2) Marginal Empirical Impact on URB and Lack of Environment Details for Multi-Agent MuJoCo: On the URB routing benchmark (Table 1), where the true harm (a max over link volume-to-capacity ratios, Eq. 5–6) lies outside PACT's linear additive surrogate class, the harm itself only degrades host learners by 1.3%−3.8%1.3\% - 3.8\%. PACT (fixed gg) reduces travel time from 3.4013.401 to 3.3693.369 (−0.032-0.032), but 0.0240.024 (75%75\%) of this improvement already occurs at σ=0\sigma=0 simply because Eq. (16) biases logits toward shorter free-flow travel times fftk\text{fft}_k. Moreover, simple vanilla IQL (3.1603.160) and Greedy (3.0443.044) substantially outperform the IPPO host and PACT (3.3693.369) at σ=3\sigma=3, and learned gg (3.3973.397) provides virtually no gain over g≡0g \equiv 0 (3.4013.401). Section 5.4 introduces a fourth experiment on Multi-Agent MuJoCo (Ant-4x2 at σ=3\sigma=3, Fig. 3), but the paper never defines what the shared medium (M), coupling operator (W), exogenous driver (D), or harm map (H) actually are for Ant-4x2, nor is it mentioned in Section 3.1 or Appendix A.

(3) Breakdown of Unbiased Identification and Exact Compensation under Physical Saturation: PACT's core theoretical guarantees (Proposition C.2, Theorem 4.1, and Corollary C.10) explicitly assume that the system operates strictly below saturation—meaning neither the disturbance clip (qsatq_{\text{sat}}, δmax⁡\delta_{\max}) nor the actuator capability box (qˉi\bar{q}_i, clipbox\text{clip}_{\text{box}}) is active[cite: 4, 5, 6, 21, 26]. Furthermore, Section 4 claims that because the regression target yiy_i is measured against the sent action aisenta_i^{\text{sent}}, compensation does not bias identification[cite: 5]. However, at high severity (σ=3\sigma = 3), compensating via aisent=aicmd/(1−c^i)a_i^{\text{sent}} = a_i^{\text{cmd}} / (1 - \hat{c}_i) or qisent=qicmd−gisˉiy^iq_i^{\text{sent}} = q_i^{\text{cmd}} - g_i \bar{s}_i \hat{y}_i actively pushes aisenta_i^{\text{sent}} toward the actuator saturation limits[cite: 6]. Once the outer actuator clip activates, the observed excess yiy_i conflates peer coupling harm with hardware saturation, introducing severe non-zero bias (bˉ>0\bar{b} > 0 in Theorem C.4) that is driven by the agent's own compensation[cite: 4, 5, 22]. The paper does not analyze how frequently saturation occurs at σ=3\sigma=3 or how much estimation bias it induces.

(4)Monotonicity with respect to severity in Proposition C.1(v):

The proposition claims that the magnitude of interaction-mediated harm is non-decreasing in the severity parameter σ\sigma. However, the proof treats the coupling load xi(t)x_i(t) as fixed, whereas this load depends on other agents' previously executed actions, which themselves depend on σ\sigma.Consequently, the claimed monotonicity need not hold when comparing complete closed-loop trajectories under different severity levels. For example, consider a two-agent unsaturated feeder model with qiex(t)=0.5−σqjex(t−1)q_i^{\mathrm{ex}}(t)=0.5-\sigma q_j^{\mathrm{ex}}(t-1) and initial executed actions equal to 0.5. At t=2t=2, the reaction magnitude is 0$.5\sigma|1-\sigma|$, which is not monotonic in σ\sigma. The proposition should explicitly restrict monotonicity to fixed coupling loads and driver values, or provide additional assumptions guaranteeing closed-loop monotonicity.

Questions for authors

(1) Robustness to Operator Misspecification (W^≠W\hat{W} \neq W): In MAPDN and VMAS, PACT uses the exact same coupling matrix WW (DistFlow SmS_m or distance kernel WijW_{ij}) that generates the benchmark harm. In real power grids or robotic systems, line reactances XbX_b or actuator coupling parameters are only known approximately. How does PACT perform on MAPDN and VMAS if the declared operator W^\hat{W} given to PACT is perturbed by multiplicative noise (e.g., W^ij=Wij(1+ξij)\hat{W}_{ij} = W_{ij}(1 + \xi_{ij}) with 10%–30%10\%\text{--}30\% error) compared to URLS and ESO?

(2) Specification of Multi-Agent MuJoCo (Ant-4x2): Could you specify the four declared objects (M, W, D, H) and the excess signal yiy_i used for the Ant-4x2 experiment in Section 5.4? Why does the interaction-mediated harm only slow down early sample efficiency on Ant while MAPPO/HAPPO/HATRPO eventually reach the exact same asymptotic return (∼5200\sim 5200) without knowing the coupling?

(3) URB Learned Trust (gpolg^{\text{pol}}) and Host Selection: On URB (Table 1a), why does PACT, learned g (3.3973.397 at σ=3\sigma=3) underperform PACT, fixed g (3.3693.369) and barely improve upon PACT, g=0 (3.4013.401)? Furthermore, given that IQL achieves 3.1603.160 at σ=3\sigma=3 (much better than the PPO host at 3.4013.401), does wrapping IQL with PACT yield meaningful drift-specific reductions on URB, or is the drift effect on URB inherently too small due to route-choice avoidance?

(4) Broadcast vs Local Sensing Feasibility: On MAPDN, utilities often avoid real-time all-to-all inverter broadcast of qjex(t−1)q_j^{\text{ex}}(t-1) due to communication latency/reliability, which is why local Volt/VAr droop curves (IEEE 1547) are used in the first place. How sensitive is PACT's stability margin enhancement (Proposition C.12) to a 1-step or 2-step communication delay in receiving peer broadcasts qjexq_j^{\text{ex}}?

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback and believe that all their concerns can be addressed. However, we have withdrawn the paper to undertake additional work that may not be feasible to complete within the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.